<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Akash Mathur on Medium]]></title>
        <description><![CDATA[Stories by Akash Mathur on Medium]]></description>
        <link>https://medium.com/@akash-mathur?source=rss-eb7bcac979b6------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*eD5Xy7puiDthVIoiOnSJEQ.jpeg</url>
            <title>Stories by Akash Mathur on Medium</title>
            <link>https://medium.com/@akash-mathur?source=rss-eb7bcac979b6------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 13:00:04 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@akash-mathur/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[LLM Evaluation: A Practical Guide for Developers]]></title>
            <link>https://akash-mathur.medium.com/llm-evaluation-a-practical-guide-for-developers-fb25608e920b?source=rss-eb7bcac979b6------2</link>
            <guid isPermaLink="false">https://medium.com/p/fb25608e920b</guid>
            <category><![CDATA[llm-evaluation]]></category>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[llm-as-a-judge]]></category>
            <category><![CDATA[synthetic-data]]></category>
            <dc:creator><![CDATA[Akash Mathur]]></dc:creator>
            <pubDate>Fri, 19 Sep 2025 19:45:45 GMT</pubDate>
            <atom:updated>2025-09-19T19:45:45.038Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pC-FQAz6Tb0KBwwE.png" /></figure><p>Evaluating <strong>Large Language Models (LLMs)</strong> is a crucial process for ensuring their quality, safety, and reliability. As LLMs become more integrated into our daily lives, from customer service chatbots to creative writing assistants, it’s essential to have robust methods to measure their performance.</p><p>The evaluation process is multifaceted, combining a variety of techniques to provide a comprehensive assessment. There are primarily 3 wide methods :</p><p>One method involves using <strong>standardized benchmarks</strong>, which are datasets designed to test specific capabilities like reasoning, factual knowledge, and common sense. Secondly, <a href="https://arxiv.org/html/2406.18403v1"><strong>human evaluation</strong></a> plays a vital role. This involves having human experts or a crowd of annotators directly assess the model’s outputs for qualities such as coherence, helpfulness, and style. Finally, a more recent and increasingly popular approach is <a href="https://arxiv.org/abs/2411.15594"><strong>LLM as a judge</strong></a>, where a powerful LLM is used to score the outputs of other LLMs, offering a scalable and often insightful way to compare different models’ performances.</p><p>These three approaches, when used together, provide a holistic framework for understanding and improving LLMs. Now, before we start coding, let’s do a bit of the ground work.</p><h3>What to Evaluate?</h3><p>LLM evaluation primarily focuses on two key areas: <strong>Response Evaluation</strong> and <strong>Retrieval Evaluation</strong>.</p><h4>Response Evaluation</h4><p>This area of evaluation assesses the quality of the generated answers, which can be challenging since the output is not a single number. The metrics used for this include:</p><ul><li><strong>Correctness:</strong> Measures if the generated answer matches a known reference answer.</li><li><strong>Semantic Similarity:</strong> Determines if the generated answer has a similar meaning to the reference answer.</li><li><strong>Faithfulness:</strong> Checks if the answer is based entirely on the retrieved context, identifying if there is any <strong>hallucination</strong>.</li><li><strong>Context Relevancy:</strong> Evaluates if the retrieved context is relevant to the user’s query.</li><li><strong>Answer Relevancy:</strong> Assesses whether the final answer directly addresses the query.</li><li><strong>Guideline Adherence:</strong> Verifies if the generated answer follows specific rules or guidelines.</li></ul><h4>Retrieval Evaluation</h4><p>This focuses on measuring the quality of the documents or sources retrieved by the LLM, independent of the final response. It involves:</p><ul><li><strong>Dataset Generation:</strong> Creating question and context pairs from a text corpus.</li><li><strong>Retrieval Metrics:</strong> Evaluating the retrieved results for a given set of questions using standard ranking metrics like <strong>mean-reciprocal rank (MRR)</strong> and <strong>hit-rate</strong> to see if the most relevant documents are being retrieved.</li></ul><h3>How to Evaluate?</h3><p>Frameworks like Ragas and Deepeval have emerged as essential tools for LLM evaluation, providing a structured, programmatic way to assess models, especially in complex applications like Retrieval-Augmented Generation (RAG). These frameworks abstract away much of the complexity by offering pre-built evaluation metrics and allowing for deep customization to fit specific use cases.</p><h4><a href="https://docs.ragas.io/en/stable/">Ragas</a></h4><p>Ragas is an open-source framework designed to evaluate RAG pipelines. It helps developers measure the performance of both the retrieval and generation components of a RAG system.</p><p>Ragas provides a set of evaluation <a href="https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/">metrics</a> that can be used to measure the performance of your LLM application.</p><h4><a href="https://github.com/confident-ai/deepeval">Deepeval</a></h4><p>Deepeval is another popular open-source framework that approaches LLM evaluation with a <strong>unit testing</strong> approach, allowing developers to write tests for their LLM applications. It provides a comprehensive library of built-in metrics, which can be easily integrated into a testing workflow.</p><p>One of Deepeval’s most flexible features is its <strong>G-Eval</strong> metric. G-Eval uses Chain-of-Thought (CoT) reasoning to evaluate LLM outputs based on any custom criteria you define. Instead of a single prompt, you can provide the LLM judge with a series of structured evaluation steps. This allows for a granular, human-like assessment of an LLM’s output, enabling you to test for highly specific criteria like tone, style, or adherence to complex instructions.</p><h4><a href="https://arxiv.org/abs/2411.15594"><strong>LLM-as-a-judge</strong></a></h4><p>A highly capable <strong>SoTA LLM</strong> can be instructed to act as an evaluation judge by using a carefully crafted prompt. This is a core part of the <strong>LLM-as-a-judge</strong> methodology and allows for a scalable, automated way to evaluate model outputs against a set of criteria.</p><p>You can explicitly define the metrics to be evaluated, such as <strong>helpfulness, relevance, accuracy, etc.</strong> The judge then uses its advanced reasoning capabilities to analyze the generated response and provide a score based on these instructions along with a change of thoughts (CoT) reasoning.</p><p>However, to note, the quality of this evaluation is highly dependent on the quality of the prompt.</p><p>We’ve now covered the foundational concepts of LLM evaluation. It’s time to transition from theory to practice by exploring the code that brings these principles to life. Let’s dive into some practical examples to see how these evaluation methods work in action.</p><h3>Let’s roll up the sleeves!</h3><p>In this article, we’re going to be leveraging the <a href="https://docs.ragas.io/en/stable/">Ragas</a> framework for our evaluations. We’re also going to explore a few more powerful retrieval systems that can potentially improve the quality of our generations, everything <strong>open-source</strong>.</p><p>If you want to follow along, download this notebook from either <a href="https://github.com/akashmathur-2212/Deep-Learning-with-PyTorch-HuggingFace/blob/main/llm-evaluation/evaluation_with_ragas_langchain.ipynb">GitHub</a> or <a href="https://drive.google.com/file/d/1ZvHk4ckZtf4DtzJPC7zysrwEpqmCrQ6L/view?usp=sharing">Colab</a>. Without further ado, let’s begin!</p><p><strong>Data Ingestion</strong></p><p>First things first, let&#39;s load the data. This code uses the <strong>DoclingLoader</strong> to read the contents of a PDF file llm-as-judge.pdf, and prepares the document for use within an LLM application.</p><p>Then it breaks down a large document into smaller, more manageable pieces. It uses a <strong>RecursiveCharacterTextSplitter</strong> to divide the contents of the PDF into chunks, with each chunk limited to a maximum of 512 characters.</p><pre>from langchain.vectorstores import FAISS<br>from langchain_community.embeddings import HuggingFaceBgeEmbeddings<br>from langchain.text_splitter import RecursiveCharacterTextSplitter<br>from langchain.chat_models import init_chat_model<br>from langchain_docling import DoclingLoader<br>from langchain.prompts import ChatPromptTemplate<br>from operator import itemgetter<br>from langchain.schema.output_parser import StrOutputParser<br>from langchain.schema.runnable import RunnableLambda, RunnablePassthrough<br>from langchain.output_parsers import ResponseSchema<br>from langchain.output_parsers import StructuredOutputParser<br><br><br>source = &quot;../data/llm-as-judge.pdf&quot;  # file path or URL<br>loader = DoclingLoader(file_path=source)<br>base_docs = loader.load()<br><br>text_splitter = RecursiveCharacterTextSplitter(chunk_size=512)<br>docs = text_splitter.split_documents(base_docs)</pre><p><strong>Setting up the Embedding Model</strong></p><p>We will be using BAAI/bge-small-enwhich is responsible for converting text into vectors. These vectors are a numerical representation that captures the semantic meaning of the text.</p><p><strong>Building the Vector Store and Retriever</strong></p><p>Next, it uses the embeddings to transform the document chunks into these vectors. It then stores them in a <strong>vector store</strong> using the efficient FAISS library. Finally, it creates a <strong>retriever</strong> from this store, which is configured to fetch the top-K most relevant documents whenever a user submits a query.</p><pre>model_name = &quot;BAAI/bge-small-en&quot;<br>model_kwargs = {&quot;device&quot;: &quot;mps&quot;}<br>encode_kwargs = {&quot;normalize_embeddings&quot;: True}<br>embeddings = HuggingFaceBgeEmbeddings(<br>    model_name=model_name, model_kwargs=model_kwargs, encode_kwargs=encode_kwargs<br>)<br><br>vectorstore = FAISS.from_documents(docs, embeddings)<br>base_retriever = vectorstore.as_retriever(search_kwargs={&quot;k&quot; : 5})</pre><p><strong>Setting up the Open-source LLMs</strong></p><p>We’ll set up two separate Large Language Models (LLMs) to serve different purposes in our evaluation pipeline.</p><p>By using <a href="https://ollama.com/"><strong>Ollama</strong></a>, I could run these models easily on my MacBook M1 Max, 32GB which avoids the costs and potential privacy concerns associated with cloud-based APIs.</p><p>Here’s the breakdown of what each model is doing:</p><ul><li><strong>question_generation_llm:</strong> We will use a smaller model, gemma3:1b, to serve as the <strong>question-answering </strong>model. This model will be used to generate responses that we will then evaluate by the larger LLM.</li><li><strong>answer_generation_llm:</strong> We will use a larger, more capable model, gemma3:4b. This LLM&#39;s primary role is to create high-quality, <em>ground truth/reference</em> answers that our evaluation metrics will use to assess the quality of the responses from the first model.</li></ul><p><strong>A Quick Note on Model Selection</strong></p><p>While we’re using smaller, open-source models like Gemma 3 for this demonstration, it’s crucial to understand that for real-world, high-stakes evaluations, the <strong>answer_generation_llm</strong> should ideally be a <strong>SoTA (State-of-the-Art) model</strong> such as <strong>GPT-5</strong> or <strong>4.1</strong> or a highly capable open-source alternative like <strong>DeepSeek-V3.1</strong>. Using a top-tier model for generating reference answers ensures the highest quality and reliability of your ground truth data, making your overall evaluation more accurate.</p><pre>question_generation_llm = init_chat_model(model=&quot;gemma3:1b&quot;, model_provider=&quot;ollama&quot;, temperature=0) # your Q&amp;A model<br>answer_generation_llm = init_chat_model(model=&quot;gemma3:4b&quot;, model_provider=&quot;ollama&quot;, temperature=0) # your SOTA model like GPT-4 latest version or Deepseek</pre><p><strong>Setting Up our Basic Question Answer Chain</strong></p><p>Now we can instantiate our basic RAG chain!</p><pre>retrieval_augmented_qa_chain = (<br>    {&quot;context&quot;: itemgetter(&quot;question&quot;) | base_retriever, &quot;question&quot;: itemgetter(&quot;question&quot;)}<br>    | RunnablePassthrough.assign(context=itemgetter(&quot;context&quot;))<br>    | {&quot;response&quot;: prompt | question_generation_llm, &quot;context&quot;: itemgetter(&quot;context&quot;)}<br>)<br><br>result = retrieval_augmented_qa_chain.invoke({&quot;question&quot; : question})</pre><h4>Ground Truth Synthetic Data Creation</h4><p>Now that our documents are prepared, we can generate a <strong>synthetic ground truth dataset</strong> to evaluate our models. This process leverages a powerful LLM to automatically create high-quality question-and-answer pairs directly from our document chunks.</p><p>For this step, the best practice is to use a <strong>SoTA model</strong> to ensure the highest quality of the generated data. While we’re using <strong>gemma3:4b</strong> for our demonstration, ideally, you would use a model like GPT-5 or a similar high-performance alternative. This process can be resource-intensive and take a significant amount of time, as the model needs to generate questions and corresponding answers for each of the document chunks.</p><p>The core idea is straightforward: we will use a framework like LangChain to programmatically create questions from our source contexts and have the LLM provide a perfect answer for each one. The resulting dataset will serve as our <strong><em>gold-standard</em></strong> ground truth, providing a reliable benchmark for our evaluation.</p><p><strong>Question generation</strong></p><p>It uses a detailed <strong>prompt</strong> to instruct the LLM to act as an expert and create a single, specific question based on a provided context. The code also forces the LLM to format its output as <strong>JSON</strong> with a specific schema. Finally, it sends the prompt to the question_generation_llm, gets the raw response, and then <strong>parses</strong> it to extract the question into a structured format.</p><p>We now extend the question-generation process to build our synthetic dataset. This code iterates through several document chunks, and for each one, it prompts our LLM to create a relevant question. The final output is a list of question-context pairs, ready to be used as our ground truth data.</p><pre>from tqdm import tqdm<br><br>question_schema = ResponseSchema(<br>    name=&quot;question&quot;,<br>    description=&quot;a question about the context.&quot;<br>)<br><br>question_response_schemas = [<br>    question_schema,<br>]<br><br>question_output_parser = StructuredOutputParser.from_response_schemas(question_response_schemas)<br>format_instructions = question_output_parser.get_format_instructions()<br><br>bare_prompt_template = &quot;{content}&quot;<br>bare_template = ChatPromptTemplate.from_template(template=bare_prompt_template)<br><br>qa_template = &quot;&quot;&quot;\<br>You are an AI Expert. For each context, create a question that is specific to the context. Avoid creating generic or general questions.<br><br>question: a question about the context.<br><br>Format the output as JSON with the following keys:<br>question<br><br>context: {context}<br>&quot;&quot;&quot;<br>prompt_template = ChatPromptTemplate.from_template(template=qa_template)<br><br>qac_triples = []<br><br># taking only 5 samples to demonstrate<br>for text in tqdm(docs[3:7]):<br>  messages = prompt_template.format_messages(<br>      context=text,<br>      format_instructions=format_instructions<br>  )<br>  response = question_generation_chain.invoke({&quot;content&quot; : messages})<br>  try:<br>    output_dict = question_output_parser.parse(response.content)<br>  except Exception as e:<br>    continue<br>  output_dict[&quot;context&quot;] = text<br>  qac_triples.append(output_dict)</pre><p><strong>Answer generation</strong></p><p>Now, we will be generating high-quality <strong>answers</strong> for the questions we just created.</p><p>Similar to the previous step, it defines a structured output format, this time for an “answer” in JSON. It then uses a detailed prompt to instruct the <strong>answer-generation LLM</strong> (ideally SoTA model, butgemma3:4bin our case) to provide an answer based on a given question and its original context.</p><p>By passing both the question and the context to the LLM, we ensure that the generated answer is directly grounded in the source material and is accurate, which is essential for creating a reliable ground truth dataset. The final output is a structured answer that can be used later to evaluate our question-answering model.</p><pre>answer_schema = ResponseSchema(<br>    name=&quot;answer&quot;,<br>    description=&quot;an answer to the question&quot;<br>)<br><br>answer_response_schemas = [<br>    answer_schema,<br>]<br><br>answer_output_parser = StructuredOutputParser.from_response_schemas(answer_response_schemas)<br>format_instructions = answer_output_parser.get_format_instructions()<br><br>qa_template = &quot;&quot;&quot;\<br>You are an AI Expert. For each question and context, create an answer.<br><br>answer: a answer about the context.<br><br>Format the output as JSON with the following keys:<br>answer<br><br>question: {question}<br>context: {context}<br>&quot;&quot;&quot;<br><br>prompt_template = ChatPromptTemplate.from_template(template=qa_template)<br><br>for triple in tqdm(qac_triples):<br>  messages = prompt_template.format_messages(<br>      context=triple[&quot;context&quot;],<br>      question=triple[&quot;question&quot;],<br>      format_instructions=format_instructions<br>  )<br>  response = answer_generation_chain.invoke({&quot;content&quot; : messages})<br>  try:<br>    output_dict = answer_output_parser.parse(response.content)<br>  except Exception as e:<br>    continue<br>  triple[&quot;answer&quot;] = output_dict[&quot;answer&quot;]</pre><p><strong>Evaluation Dataset</strong></p><p>Now, we can take the synthetic <strong>question-context-answer</strong> triples we’ve generated and organize them into a structured dataset for evaluation using Ragas.</p><p>An evaluation dataset is a homogeneous collection of <a href="https://docs.ragas.io/en/stable/concepts/components/eval_sample/">data samples</a> designed to assess the performance and capabilities of an AI application. In Ragas, evaluation datasets are represented using the EvaluationDataset class, which provides a structured way to organize and manage data samples for evaluation purposes.</p><pre>import pandas as pd<br>from datasets import Dataset<br><br>ground_truth_qac_set = pd.DataFrame(qac_triples)<br>ground_truth_qac_set[&quot;context&quot;] = ground_truth_qac_set[&quot;context&quot;].map(lambda x: str(x.page_content))<br>ground_truth_qac_set = ground_truth_qac_set.rename(columns={&quot;answer&quot; : &quot;ground_truth&quot;})<br><br>eval_dataset = Dataset.from_pandas(ground_truth_qac_set)</pre><h3>Evaluation Using Ragas</h3><p>Now we can evaluate using Ragas!</p><p>The set-up is: we simply need to create a dataset with our generated answers and our contexts, and then evaluate using the framework.</p><pre>from ragas.metrics import (<br>    answer_relevancy,<br>    faithfulness,<br>    context_recall,<br>    context_precision,<br>    answer_correctness,<br>    answer_similarity<br>)<br><br>from ragas import evaluate<br><br>def create_ragas_dataset(rag_pipeline, eval_dataset):<br>  rag_dataset = []<br>  for row in tqdm(eval_dataset):<br>    answer = rag_pipeline.invoke({&quot;question&quot; : row[&quot;question&quot;]})<br>    rag_dataset.append(<br>        {&quot;question&quot; : row[&quot;question&quot;],<br>         &quot;answer&quot; : answer[&quot;response&quot;].content,<br>         &quot;contexts&quot; : [context.page_content for context in answer[&quot;context&quot;]],<br>         &quot;reference&quot; : row[&quot;ground_truth&quot;]<br>         }<br>    )<br>  rag_df = pd.DataFrame(rag_dataset)<br>  rag_eval_dataset = Dataset.from_pandas(rag_df)<br>  return rag_eval_dataset<br><br>def evaluate_ragas_dataset(ragas_dataset):<br>  result = evaluate(<br>    dataset = ragas_dataset,<br>    metrics=[<br>        context_precision,<br>        faithfulness,<br>        answer_relevancy,<br>        context_recall,<br>        answer_correctness,<br>        answer_similarity<br>    ],<br>    llm=answer_generation_llm,<br>    embeddings=embeddings<br>  )<br>  return result</pre><p>Let&#39;s create our dataset first, and finally evaluate how it did!</p><pre>basic_qa_ragas_dataset = create_ragas_dataset(retrieval_augmented_qa_chain, eval_dataset)<br><br>basic_qa_result = evaluate_ragas_dataset(basic_qa_ragas_dataset)</pre><h3>Evaluating Basic to Complex Retrievers</h3><p>Now we can test how changing our Retriever impacts our Ragas evaluation!</p><p>We’ll build this simplecreate_qa_chainto create standardized QA chains where the only different component will be the retriever.</p><pre>def create_qa_chain(retriever):<br>  created_qa_chain = (<br>    {&quot;context&quot;: itemgetter(&quot;question&quot;) | retriever,<br>     &quot;question&quot;: itemgetter(&quot;question&quot;)<br>    }<br>    | RunnablePassthrough.assign(<br>        context=itemgetter(&quot;context&quot;)<br>      )<br>    | {<br>         &quot;response&quot;: prompt | question_generation_llm,<br>         &quot;context&quot;: itemgetter(&quot;context&quot;),<br>      }<br>  )<br><br>  return created_qa_chain</pre><p>We have already created a basic QA retriever, so let’s create more complex ones!</p><p><strong>Parent Document Retriever</strong></p><p>One of the easier ways we can imagine improving a retriever is to embed our documents into small chunks, and then retrieve a significant amount of additional context that “surrounds” the found context.</p><p>The basic outline of this retrieval method is as follows:</p><ol><li>Obtain User Question</li><li>Retrieve child documents using Dense Vector Retrieval</li><li>Merge the child documents based on their parents. If they have the same parents, they become merged.</li><li>Replace the child documents with their respective parent documents from an in-memory store.</li><li>Use the parent documents to augment generation.</li></ol><pre>from langchain.retrievers import ParentDocumentRetriever<br>from langchain.storage import InMemoryStore<br><br>parent_splitter = RecursiveCharacterTextSplitter(chunk_size=1500)<br>child_splitter = RecursiveCharacterTextSplitter(chunk_size=200)<br><br>base_docs_text = [x.page_content for x in base_docs]<br>vectorstore = FAISS.from_texts(base_docs_text, embeddings)<br>store = InMemoryStore()<br><br>parent_document_retriever = ParentDocumentRetriever(<br>    vectorstore=vectorstore,<br>    docstore=store,<br>    child_splitter=child_splitter,<br>    parent_splitter=parent_splitter,<br>)<br><br>parent_document_retriever.add_documents(base_docs)<br><br>parent_document_retriever_qa_chain = create_qa_chain(parent_document_retriever)</pre><p>Now, create the Ragas dataset and evaluate it:</p><pre>pdr_qa_ragas_dataset = create_ragas_dataset(parent_document_retriever_qa_chain, eval_dataset)<br><br>pdr_qa_result = evaluate_ragas_dataset(pdr_qa_ragas_dataset)</pre><p><strong>Ensemble Retrieval (Hybrid/Fusion Search)</strong></p><p>Next, let’s look at ensemble retrieval. The basic idea is as follows:</p><ol><li>Obtain User Question</li><li>Hit the Retriever Pair</li><li>Retrieve Documents with BM25 Sparse Vector Retrieval</li><li>Retrieve Documents with Dense Vector Retrieval Method</li><li>Collect and “fuse” the retrieved docs based on their weighting using the Reciprocal Rank Fusion algorithm into a single ranked list.</li><li>Use those documents to augment our generation.</li><li>Ensure your `weights` list, i.e., the relative weighting of each retriever, sums to 1.</li></ol><pre>from langchain.retrievers import BM25Retriever, EnsembleRetriever<br><br>rec_text_splitter = RecursiveCharacterTextSplitter(chunk_size=512, chunk_overlap=75)<br>rec_docs = text_splitter.split_documents(base_docs)<br><br>bm25_retriever = BM25Retriever.from_documents(rec_docs, k=2)<br><br>vectorstore = FAISS.from_documents(rec_docs, embeddings)<br>faiss_retriever = vectorstore.as_retriever(search_kwargs={&quot;k&quot;: 3})<br><br>ensemble_retriever = EnsembleRetriever(retrievers=[bm25_retriever, faiss_retriever], weights=[0.75, 0.25])<br><br>ensemble_retriever_qa_chain = create_qa_chain(ensemble_retriever)</pre><p>Now, create the Ragas dataset and evaluate it:</p><pre>ensemble_qa_ragas_dataset = create_ragas_dataset(ensemble_retriever_qa_chain, eval_dataset)<br><br>ensemble_qa_result = evaluate_ragas_dataset(ensemble_qa_ragas_dataset)<br><br>ensemble_qa_result.to_pandas()</pre><h3>Let’s Compare!</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lQ4cVqa7yTxO2GT71PK-Qw.png" /></figure><p>We can also zoom in on each result and find specific information about each of the questions and answers by converting the evaluation result into a pandas dataframe usingto_pandas().</p><p><strong>Note:</strong> While I was using the Ollama model locally for evaluation, I observed a couple of `TimeoutError()`, maybe that’s the reason I got inconsistent values likenan.</p><p>You have now seen a complete, end-to-end process for building a synthetic ground-truth dataset from a document and then evaluating it using Ragas. This hands-on walkthrough has demonstrated how to move from foundational theory to practical application of modern evaluation frameworks.</p><h3>Conclusion</h3><p>Evaluating LLMs is no longer a simple task of checking for a single correct answer. Instead, it is a sophisticated, multi-faceted discipline that requires a combination of automated tools and human-in-the-loop insights. We’ve explored the importance of different evaluation methodologies, from using traditional benchmarks to leveraging <strong>LLM-as-a-judge</strong> frameworks.</p><p>The ability to generate a synthetic ground-truth dataset, complete with questions and accurate answers, is a powerful technique that allows for robust, scalable evaluation, even when human-labeled data is scarce. This process is critical for building trustworthy and reliable LLM applications, especially in domains where accuracy and faithfulness are paramount.</p><h3>Next Steps: The Path to a Golden Dataset</h3><p>While we have successfully created a synthetic dataset, our work is not yet complete. Before we can use this data for production-grade applications, we must first validate it to create a <strong>“golden” dataset</strong>. This ensures that the questions and answers we generated are of the highest quality and truly represent accurate ground truth. Relying on an unvalidated synthetic dataset can lead to misleading evaluation results, a classic case of “garbage in, garbage out.”</p><p>The next crucial step is to perform a thorough quality check on a representative number of samples from our newly created dataset. This process should involve:</p><ul><li><strong>Manual Review:</strong> A subject-matter expert or human evaluator will manually review a statistically significant sample of the generated question-and-answer pairs.</li><li><strong>Verification:</strong> The reviewer will verify that the generated questions are relevant, clear, and specific to the context, and that the answers are factually accurate, faithful to the source, and well-structured.</li><li><strong>Refinement:</strong> Any inaccuracies or poorly formatted examples discovered during this review will be corrected or removed, ensuring that our golden dataset is a reliable and trustworthy benchmark.</li></ul><p>By taking the time to validate our synthetic data, we create a robust, high-quality golden dataset. This validated dataset will be the foundation for all future evaluations, allowing us to confidently assess and improve our LLM application with a reliable source of truth.</p><p><em>Before you go, if you enjoyed reading this article:</em></p><p>👉 <em>Be sure to </em><strong><em>clap</em></strong><em> and </em><strong><em>follow</em></strong><em> me, and let me know if any feedback.</em></p><p>👉<em>I built versatile Generative AI applications, covered advanced RAG concepts, and serverless AWS architectures for Big Data processing. You’re welcome to take a look at the below GitHub repo and star⭐it.</em></p><ul><li><em>⭐ Deep Learning Playground with </em><strong><em>PyTorch</em></strong><em> &amp; </em><strong><em>Hugging Face</em></strong><em> </em><a href="https://github.com/akashmathur-2212/Deep-Learning-with-PyTorch-HuggingFace"><em>Repository</em></a></li><li><em>⭐ </em><strong><em>Generative AI</em></strong><em> applications </em><a href="https://github.com/akashmathur-2212/LLMs-playground"><em>Repository</em></a></li><li><em>⭐ </em><strong><em>Advanced RAG</em></strong><em> applications </em><a href="https://github.com/akashmathur-2212/LLMs-playground/tree/main/LlamaIndex-applications/Advanced-RAG"><em>Repository</em></a></li><li><em>⭐ Serverless </em><strong><em>AWS</em></strong><em> architectures </em><a href="https://github.com/akashmathur-2212/aws-serverless-workflows"><em>Repository</em></a></li></ul><p>👉<em>Follow me: </em><a href="https://www.linkedin.com/in/akashmathur22/"><strong><em>LinkedIn</em></strong></a> <strong><em>| </em></strong><a href="https://github.com/akashmathur-2212"><strong><em>GitHub</em></strong></a> <strong><em>| </em></strong><a href="https://akash-mathur.medium.com/"><strong><em>Medium</em></strong></a><strong><em> | </em></strong><a href="https://akashmathur-2212.github.io/"><strong><em>Portfolio</em></strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fb25608e920b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploy a Serverless ML Inference using FastAPI, AWS Lambda, and API Gateway]]></title>
            <link>https://aws.plainenglish.io/deploy-a-serverless-ml-inference-using-fastapi-aws-lambda-and-api-gateway-32b0b9a40236?source=rss-eb7bcac979b6------2</link>
            <guid isPermaLink="false">https://medium.com/p/32b0b9a40236</guid>
            <category><![CDATA[api]]></category>
            <category><![CDATA[fastapi]]></category>
            <category><![CDATA[lambda]]></category>
            <category><![CDATA[deployment]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Akash Mathur]]></dc:creator>
            <pubDate>Sat, 17 Feb 2024 11:15:01 GMT</pubDate>
            <atom:updated>2024-02-19T02:24:45.512Z</atom:updated>
            <content:encoded><![CDATA[<h3>Deploy a Serverless ML Inference Using FastAPI, AWS Lambda, and API Gateway</h3><h4>Harnessing the Power of Scalable Infrastructure for Seamless ML Deployment</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*3720cNYenmVypeqq.jpg" /><figcaption>Scalable Infrastructure for Seamless ML Deployment</figcaption></figure><h3>1. Introduction</h3><p>Transitioning machine learning (ML) models from proof of concept to production is a crucial step for data scientists, yet it often poses significant challenges. One of the main hurdles lies in deploying locally trained models to the cloud for scalable inference and integration into various applications. However, with the right approach, this process can be streamlined, minimizing effort and maximizing efficiency.</p><p><a href="https://fastapi.tiangolo.com/"><strong>FastAPI</strong></a><strong> </strong>stands out as a modern, high-performance web framework tailored for API development with Python. Known for its user-friendly interface and scalability, it has become a preferred choice among developers.</p><p><a href="https://aws.amazon.com/pm/lambda/?gclid=CjwKCAiArLyuBhA7EiwA-qo80OxFb0ZdK5B1AxqDm0iny6hT0oCkq6T0DF6v4ofdMp2-BFTt9RImjBoCuWMQAvD_BwE&amp;trk=5cc83e4b-8a6e-4976-92ff-7a6198f2fe76&amp;sc_channel=ps&amp;ef_id=CjwKCAiArLyuBhA7EiwA-qo80OxFb0ZdK5B1AxqDm0iny6hT0oCkq6T0DF6v4ofdMp2-BFTt9RImjBoCuWMQAvD_BwE:G:s&amp;s_kwcid=AL!4422!3!651612776783!e!!g!!aws%20lambda!19828229697!143940519541"><strong>AWS Lambda</strong></a> offers a direct pathway to expose models and deploy ML applications using your preferred open-source framework, providing flexibility and cost-effectiveness.</p><p>This blog post demonstrates a straightforward approach to deploying and running serverless ML inference, exposing your ML model using FastAPI, Docker, Lambda, and API Gateway.</p><p><strong>1.1 Introduction to FastAPI</strong></p><p><a href="https://fastapi.tiangolo.com/"><strong>FastAPI</strong> </a>emerges as a standout choice in this area, offering a modern, high-performance web framework for building APIs with Python. It excels in developing serverless applications, especially for RESTful microservices and ML inference at scale. With built-in features such as automatic API documentation, FastAPI simplifies the deployment of high-performance inference APIs. Combining the strengths of established frameworks like Flask and Django with its innovations, FastAPI is built on the Starlette asynchronous web framework, ensuring efficient management of concurrent requests and high traffic loads.</p><p>Leveraging FastAPI you can effortlessly scale out and manage complex business logic, testing locally and hosting on Lambda, then exposing through a unified API gateway, seamlessly integrating the power of an open-source web framework into Lambda without requiring extensive code refactoring.</p><p><strong>1.2 Why FastAPI</strong></p><ul><li><strong>High Performance</strong>: FastAPI harnesses asynchronous capabilities and type hints to optimize web framework performance.</li><li><strong>Ease of Use</strong>: With a declarative approach and auto-generated documentation, FastAPI simplifies API development and maintenance.</li><li><strong>Type Safety</strong>: Leveraging type hints, FastAPI ensures robust data validation and serialization, enhancing code reliability and readability.</li><li><strong>Compatibility</strong>: Seamlessly integrating with popular Python libraries and frameworks such as <a href="https://docs.pydantic.dev/latest/">Pydantic</a> enhances developer productivity.</li><li><strong>Scalability</strong>: FastAPI’s asynchronous architecture enables it to handle high loads and concurrent requests effectively.</li></ul><h3>2. Setting Up the Project</h3><p>Before diving into development, establishing the project environment is crucial. This involves installing FastAPI along with any necessary dependencies for our API. Utilizing virtual environments ensures a secure development setup.</p><p><strong>2.1 Creating the Environment</strong></p><p>Creating a virtual environment is fundamental for project development, offering numerous benefits such as dependency isolation, reproducibility, and collaboration facilitation. It guarantees that project dependencies are contained, permits precise package version specification, and ensures a tidy development environment.</p><p>To Create a Virtual Environment:</p><ol><li>Open the command prompt.</li><li>Navigate to the project directory using the cd command.</li></ol><p>Create a virtual environment using the command:</p><pre>python -m venv myenv</pre><p>Activate the virtual environment with:</p><pre>myenv\Scripts\activate</pre><p><strong>2.2 Installing dependencies and libraries</strong></p><p>After creating and activating the environment, the next step is to install the necessary dependencies for your project. Dependencies are additional packages or libraries that your project relies on to function properly. To install dependencies, you can use a package manager such as pip, or you can use the following command to install all at once:</p><pre>pip install -r requirements.txt</pre><p>By installing dependencies, you ensure that your project has access to the required functionality and libraries. It’s important to keep track of the dependencies and their versions to maintain consistency and avoid compatibility issues.</p><h3>3. Preparing the Model</h3><p><strong>3.1 Quick Overview of the Diabetes Prediction Model</strong></p><p>The integrated prediction model within this API is a classification model tailored for diabetes<strong> </strong>prediction. This <a href="https://www.kaggle.com/datasets/mathchi/diabetes-data-set">dataset </a>is originally from the National Institute of Diabetes and Digestive and Kidney Diseases. The objective is to predict based on diagnostic measurements whether a patient has diabetes.</p><p>Utilizing input features, it delivers binary predictions for diabetes status, categorizing them as either 1 (indicating a positive prediction) or 0 (representing a negative prediction).</p><p><strong>3.2 Storing the ML Components into Pickle Files</strong></p><p>The classification model, feature pipeline,<strong> </strong>and all other ML components were saved as pickle files. This approach ensures model persistence, enabling effortless saving and loading while preserving its state and parameters, thereby enhancing portability and ease of deployment.</p><p>To save the trained model, the following code snippet was executed in the notebook:</p><pre>import pandas as pd<br>import numpy as np<br>import pickle<br>from sklearn.ensemble import RandomForestClassifier<br>from sklearn.metrics import f1_score, accuracy_score, roc_auc_score<br><br># Read data into pandas dataframe<br>df=pd.read_csv(&#39;.\\data\\diabetes.csv&#39;)<br><br>#spliting data into  80% train and  20% test<br>train, test = train_test_split(df, test_size=0.2, random_state=42)<br>print(f&#39;Train: {train.shape}, Test: {test.shape}&#39;)<br><br># create features and targets from the train data<br>X_train = train.drop(columns=[&#39;Outcome&#39;])<br>y_train = train[&#39;Outcome&#39;].copy()<br><br># create features and targets from test data<br>X_test = test.drop(columns=[&#39;Outcome&#39;])<br>y_test = test[&#39;Outcome&#39;].copy()<br><br>#model train<br>model = RandomForestClassifier(n_estimators=50, n_jobs=1,random_state=8)<br>model.fit(X_train, y_train)<br><br># create predictions<br>y_pred = model.predict(X_test)<br><br>#evaluate<br>accuracy_score(y_pred, y_test), f1_score(y_pred, y_test)<br><br>#save all ML components to pickle<br>model_filename = &#39;./artifacts/model.pkl&#39;<br>with open(model_filename, &#39;wb&#39;) as model_file:<br>    pickle.dump(model, model_file)<br></pre><p>To save all ML components, including the model, preprocessing steps, and any other relevant objects, here&#39;s a general approach to saving all components to a single pickle file:</p><pre>import pickle<br><br># Define your model and preprocessing steps<br>model = model # Your trained machine learning model<br>preprocessor = preprocessor  # Your preprocessing steps (e.g., scaler, encoder, etc.)<br><br># Other relevant objects<br>other_objects = {<br>    &#39;key1&#39;: value1,<br>    &#39;key2&#39;: value2,<br>    # Add any other relevant objects here<br>}<br><br># Combine all components into a dictionary<br>ml_components = {<br>    &#39;model&#39;: model,<br>    &#39;preprocessor&#39;: preprocessor,<br>    **other_objects<br>}<br><br># Save all components to a pickle file<br>with open(&#39;ml_components.pkl&#39;, &#39;wb&#39;) as f:<br>    pickle.dump(ml_components, f)</pre><h3>4. Building the API</h3><p>Let’s install FastAPI and Uvicorn. Uvicorn is a dependency of FastAPI and is the library that will build and manage the server.</p><blockquote><strong>Uvicorn </strong>is the server that is used to run and test our APIs. Uvicorn is an ASGI web server implementation for Python.</blockquote><pre>pip install fastapi uvicorn</pre><h4><strong>4.1 Designing the API</strong></h4><p>The most important part of this repository is the <a href="https://github.com/akashmathur-2212/Serverless-Deployment-Using-FastAPI-AWS">Serverless-Deployment-Using-FastAPI-AWS</a> directory. It contains the code and the resources that are going to be used for model serving.</p><ul><li>data- Training data</li><li>artifacts- This directory contains model_endpoint or model.pickle, and all the ML components necessary that make up our serverless endpoint.</li><li>Dockerfile – Dockerfile to build the Docker image that Lambda will use, the Lambda function code that uses FastAPI to handle inference requests and route them to the correct endpoint, and the model artifacts of the model that we want to deploy.</li><li>requirements.txt – The code for the Lambda function and its dependencies specified.</li><li>app.py – Contains the code that implements a Lambda function that wraps a FastAPI endpoint around an existing API gateway</li><li>utils.py – This includes an inference script that loads the necessary model artifacts so that the model can be passed to the app.py that will then expose it as an endpoint.</li></ul><h4><strong>4.2 API Endpoints</strong></h4><ol><li><strong><em>Root Endpoint</em>:</strong> This endpoint serves as the entry point, welcoming users with a friendly message — “Welcome to the Diabetes API”.</li><li><strong><em>Health Check Endpoint</em></strong>: To ensure the API’s reliability, a health check endpoint is important. We’ll implement a straightforward endpoint to monitor the API’s status, affirming its operational state.</li><li><strong><em>Model Information Endpoint</em></strong>: Providing transparency, this endpoint furnishes details about the underlying model. It fetches the model’s name, parameters, and both categorical and numerical features utilized during training.</li><li><strong><em>Individual Prediction Endpoint</em></strong>: Catering to single predictions, this endpoint accepts input parameters and delivers predictions for individual data records. It’s a handy tool for users seeking predictions on a singular data point.</li><li><strong><em>Batch Prediction Endpoint</em></strong>: Streamlining predictions for multiple data points, this endpoint accepts a list of input records. It leverages our trained ML model to process the data and returns predictions for each input, facilitating bulk predictions.</li><li><strong><em>Data Upload Endpoint</em></strong>: Addressing real-world scenarios, this endpoint empowers users to upload their data for prediction. Accepting data files or JSON/CSV formatted data, the API processes the input and delivers predictions using our ML model.</li></ol><h4><strong>4.3 Integrating ML Components Into API</strong></h4><p>To make accurate predictions, we need to integrate a trained ML model with all other components into our API.</p><p>We can load the trained model directly, or we can leverage the <a href="https://fastapi.tiangolo.com/advanced/events/?source=post_page-----e329ca3a89c6--------------------------------#lifespan-function">Lifespan Events</a> functionality that FastAPI offers for better optimization.</p><p><strong>4.3.1 Lifespan Events</strong></p><p>In FastAPI, a lifespan function can indeed help in loading machine learning models across multiple users by allowing for efficient initialization and cleanup tasks. Lifespan functions are special asynchronous functions that can be registered to execute when the application starts up (startup event) or shuts down (shutdown event). Leveraging these functions, you can preload machine learning models during startup, ensuring they are readily available to handle incoming requests from multiple users.</p><p>Here’s how a lifespan function can assist in loading machine learning models:</p><ol><li><strong>Startup Initialization</strong>: You can define a startup event handler to load the machine learning model into memory when the FastAPI application starts up. This ensures that the model is loaded only once, regardless of the number of concurrent users or requests.</li><li><strong>Efficient Resource Management</strong>: By preloading the model during startup, you avoid the overhead of loading the model for each individual request. This improves the overall performance and responsiveness of your application, as users don’t experience delays associated with model loading.</li><li><strong>Shared Model Across Users</strong>: Since the model is loaded during startup and remains in memory throughout the application’s lifecycle, it can be shared across multiple users and requests. This eliminates the need to reload the model for each user, leading to better resource utilization and scalability.</li><li><strong>Auto Shutdown</strong>: Additionally, you can define a shutdown event handler to perform cleanup tasks, such as releasing resources associated with the model, when the application shuts down. This ensures proper resource management and prevents memory leaks.</li></ol><p>Read more about it from the documentation:</p><p><a href="https://fastapi.tiangolo.com/advanced/events/?source=post_page-----e329ca3a89c6--------------------------------#lifespan-function">Lifespan Events - FastAPI</a></p><pre>import uvicorn<br>from fastapi import FastAPI, Request, File, UploadFile<br>from fastapi.responses import JSONResponse<br>from utils import load_pickle, make_prediction, process_label, process_json_csv, output_batch, return_columns<br>import pandas as pd<br>import pickle<br>from contextlib import asynccontextmanager<br>from typing import List<br>from pydantic import BaseModel<br>from mangum import Mangum<br><br><br>###### load model with Lifespan Events #######<br># model prediction<br>def diabetes_predictor(x: dict) -&gt; dict:<br>    with open(&quot;./artifacts/model.pkl&quot;, &#39;rb&#39;) as model_file:<br>        model = pickle.load(model_file)<br><br>    data_df = pd.DataFrame(x, index=[0])<br>    prediction = model.predict(data_df)<br><br>    if(prediction[0]&gt;0.5):<br>        prediction=&quot;Diabetic&quot;<br>    else:<br>        prediction=&quot;Non Diabetic&quot;<br>    <br>    return {&#39;prediction&#39;: prediction}<br><br><br># Life Span Management<br>ml_models = {}<br><br>@asynccontextmanager<br>async def ml_lifespan_manager(app: FastAPI):<br>    ml_models[&quot;diabetes_predictor&quot;] = diabetes_predictor<br>    yield<br>    ml_models.clear()<br><br><br># Create an instance of FastAPI<br>app = FastAPI(lifespan=ml_lifespan_manager, debug=True)</pre><p><strong>4.3.2 Loading ML Components without Lifespan Events</strong></p><p>For this example, we are using just 1 ML component i.e. ML Model, but a bigger application will have other components as well. We can load them at the start.</p><pre>import pickle<br><br>model = load_pickle(&quot;./artifacts/ml_components.pkl&quot;)<br><br># Access individual components<br>model = ml_components[&#39;model&#39;]<br>preprocessor = ml_components[&#39;preprocessor&#39;]<br>other = ml_components[&#39;other&#39;]</pre><p><strong>4.3.3 Handling Data Preprocessing</strong></p><p>Before making predictions, it’s essential to preprocess the data appropriately. This involves performing various preprocessing steps, such as scaling, encoding, and handling missing values, to ensure that the input data meets the requirements of our machine-learning model. We will follow these steps for data handling and preprocessing.</p><p><strong><em>A. Collecting and Validating Inputs</em></strong></p><p>To improve your models in FastAPI, you can leverage Pydantic’s powerful features for data validation, serialization, and schema generation.</p><p>Pydantic allows you to define data models with type annotations, making it easy to validate incoming request data. By specifying data types, constraints, and default values, you can ensure that your API handles input data correctly and gracefully handles validation errors.</p><p>Pydantic models can be used to generate JSON Schema representations of your data models. This can be useful for validating JSON data against a schema or for generating documentation for clients consuming your API.</p><ul><li><strong><em>Individual Prediction Endpoint</em></strong>: For individual predictions, input data is directly passed into the endpoint as variables, a Pydantic BaseModel is employed to validate input ensuring seamless integration and straightforward processing.</li><li><strong><em>Batch Prediction Endpoint</em></strong>: When handling batch predictions, a Pydantic BaseModel is employed to validate structured batch input. This ensures consistency and accuracy across multiple data points, enhancing the reliability of predictions.</li><li><strong><em>Data Upload Endpoint</em></strong>: Users have the option to upload JSON or CSV files for prediction via the data upload endpoint. These files undergo careful validation for format and structure compatibility to handle potential errors effectively.</li></ul><pre># Input Data Validation<br>class ModelInput(BaseModel):<br>    Pregnancies : int<br>    Glucose : int<br>    BloodPressure : int<br>    SkinThickness : int<br>    Insulin : int<br>    BMI : float<br>    DiabetesPedigreeFunction : float<br>    Age : int<br><br><br># Input for Batch prediction<br>class ModelInputs(BaseModel):<br>    all: List[ModelInput]<br><br>    def return_dict_inputs(cls):<br>        return [dict(input) for input in cls.all]</pre><p><strong><em>B. Feature Engineering</em></strong></p><p>After collecting and validating the data, the next step is to create the features that were used to train the model. This involves transforming and manipulating the raw data to extract or create the same input variables for the prediction model.</p><p>It will be done using the preprocessor you just loaded from the pickle.</p><p><strong><em>C. Handling Upload File types</em></strong></p><p>The function below is used to process JSON and CSV files into formats that can be converted into data frames and used for prediction.</p><pre>def process_json_csv(contents, file_type, valid_formats):<br><br>    # Read the file contents as a byte string<br>    contents = contents.decode()  # Decode the byte string to a regular string<br>    new_columns = return_columns() # return new_columns<br>    # Process the uploaded file<br>    if file_type == valid_formats[0]:<br>        data = pd.read_csv(StringIO(contents)) # read csv files<br>    elif file_type == valid_formats[1]:<br>        data = pd.read_json(contents) # read json file<br>    <br>    dict_new_old_cols = dict(zip(data.columns, new_columns)) # get dict of new and old cols<br>    data = data.rename(columns=dict_new_old_cols) # rename colums to appropriate columns<br>    return data</pre><p><strong><em>D. Making a Prediction</em></strong></p><p>Now, we will have to make a prediction using the API. The <strong>make_prediction </strong>function was created to achieve this purpose.</p><pre>def make_prediction(data, model):<br>    new_columns = return_columns() <br>    dict_new_old_cols = dict(zip(data.columns, new_columns)) # create a dict of original columns and new columns<br>    data = data.rename(columns=dict_new_old_cols)<br>    # feature_engineering_encoding(data) # create new features<br>    # make prediction<br>    label = model.predict(data) # make a prediction<br>    probs = model.predict_proba(data) # predit sepsis status for inputs<br>    return label, probs.max()</pre><p><strong><em>E. Formatting outputs</em></strong></p><p>The outputs when a post request is made to API need to be formatted in a proper format. The function <strong>output_batch</strong> takes care of that.</p><pre>def output_batch(data1, labels):<br>    data_labels = pd.DataFrame(labels, columns=[&#39;Predicted Label&#39;]) # convert label into a dataframe<br>    data_labels[&#39;Predicted Label&#39;] = data_labels.apply(process_label, axis=1) # change label to understanding strings<br>    results_list = [] # create an empty lits<br>    x = data1.to_dict(&#39;index&#39;) # convert  datafram into dictionary<br>    y = data_labels.to_dict(&#39;index&#39;) # convert  datafram into dictionary<br>    for i in range(len(y)):<br>        results_list.append({i:{&#39;inputs&#39;: x[i], &#39;output&#39;:y[i]}}) # append input and labels<br><br>    final_dict = {&#39;results&#39;: results_list}<br>    return final_dict</pre><p><strong>4.3.4 Define routes and functions</strong></p><p>Building endpoints for FastAPI involves defining routes and functions that handle HTTP requests. Each endpoint corresponds to a specific URL path and HTTP method (e.g., GET, POST, PUT, DELETE). Here’s a general guide on how to build endpoints in FastAPI.</p><p>Use the app object to define routes using the @app.route() decorator. Each route corresponds to a specific URL path and HTTP method.</p><pre># Create an instance of FastAPI<br>app = FastAPI(lifespan=ml_lifespan_manager, debug=True)<br><br><br># Root endpoint<br>@app.get(&#39;/&#39;)<br>def index():<br>    return {&#39;message&#39;: &#39;Welcome to the Diabetes API&#39;}<br><br><br># Health check endpoint<br>@app.get(&quot;/health&quot;)<br>def check_health():<br>    return {&quot;status&quot;: &quot;ok&quot;}<br><br><br># Model information endpoint<br>@app.post(&#39;/model-info&#39;)<br>async def model_info():<br>    model_name = ml_models[&quot;diabetes_predictor&quot;].__class__.__name__ # get model name <br>    model_params = ml_models[&quot;diabetes_predictor&quot;].get_params() # get model parameters<br><br>    # model_name = model.__class__.__name__ # get model name <br>    # model_params = model.get_params() # get model parameters<br><br>    model_information =  {&#39;model info&#39;: {<br>            &#39;model name &#39;: model_name,<br>            &#39;model parameters&#39;: model_params<br>            }<br>            }<br>    return model_information # return model information<br><br><br># Single Prediction endpoint <br>@app.post(&quot;/predict&quot;)<br>async def predict(model_input: ModelInput):<br><br>    data = dict(model_input)<br><br>    return ml_models[&quot;diabetes_predictor&quot;](data)<br><br><br># Batch prediction endpoint<br>@app.post(&#39;/predict-batch&#39;)<br>async def predict_batch(inputs: ModelInputs):<br>    # Create a dataframe from inputs<br>    data = pd.DataFrame(inputs.return_dict_inputs())<br>    labels, probs = make_prediction(data, model) # Get the labels<br>    response = output_batch(data, labels) # output results<br>    return response<br><br><br># Upload data endpoint<br>@app.post(&quot;/upload-data&quot;)<br>async def upload_data(file: UploadFile = File(...)):<br>    file_type = file.content_type # get the type of the uploaded file<br>    valid_formats = [&#39;text/csv&#39;, &#39;application/json&#39;] # create a list of valid formats API can receive<br>    if file_type not in valid_formats:<br>        return JSONResponse(content={&quot;error&quot;: f&quot;Invalid file format. Must be one of: {&#39;, &#39;.join(valid_formats)}&quot;}) # return an error if file type is not included in the valid formats<br>    <br>    else:<br>        contents = await file.read() # read contents in file<br>        data= process_json_csv(contents=contents,file_type=file_type, valid_formats=valid_formats) # process files  <br>        labels, probs = make_prediction(data, model) # Get the labels<br>        response = output_batch(data, labels) # output results<br><br>    return response</pre><h3>5. Testing the API</h3><p>Testing APIs across various endpoints is essential to ensure their functionality, reliability, and performance. Run this command on the terminal:</p><pre>uvicorn app:app --reload </pre><ol><li>Open your browser at <a href="http://127.0.0.1:8000/predict">http://127.0.0.1:8000</a>. Now go to <a href="http://127.0.0.1:8000/docs">http://127.0.0.1:8000/docs</a>. You will see the automatic interactive API documentation (provided by <a href="https://github.com/swagger-api/swagger-ui">Swagger UI</a>)</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z14sE1OVTFaYVTY64q6QJw.png" /><figcaption>Interactive API by <a href="https://github.com/swagger-api/swagger-ui">Swagger UI</a></figcaption></figure><p>2. Test the health check endpoint by sending a GET request to the health URL or /health. Verify that the response indicates that the API is running correctly.</p><p>3. Test the model information endpoint by sending a get request to the model information URL/model-info</p><p>4. Test the <strong>Individual Prediction Endpoint</strong> URL/predictdesigned for single predictions, this endpoint accepts input parameters and returns predictions for individual data records.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pl8jU4bvoekO3vSYSpkI3w.png" /><figcaption>Individual Prediction Endpoint</figcaption></figure><p>5. Testing the <strong>Batch Prediction Endpoint</strong> URL/predict-batchpredictions for multiple data points, this endpoint accepts a list of input records and returns predictions for each input. Testing this endpoint involves sending requests with batches of input data and verifying that the API processes the data correctly and returns accurate predictions for each input record.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*La87LCdsgZfR7RfezQ5gfw.png" /><figcaption>Batch Prediction Endpoint for 2 Samples</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tuJ_HqtFl05PzPmNDTjWqg.png" /><figcaption>Request Body</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X2CaNEL84YyMvCqvfWp_0Q.png" /><figcaption>Response Body Sample 1</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5PfbmHDnTl13gCzHmyfq-w.png" /><figcaption>Response Body Sample 2</figcaption></figure><p>6. Testing the <strong>Data Upload Endpoint </strong>URL/upload-datafor real-world scenarios, this endpoint allows users to upload data for prediction in JSON or CSV format. I tested using the api_upload_test.csv file located on this path ./data/api_upload_test.csvin the <a href="https://github.com/akashmathur-2212/Serverless-Deployment-Using-FastAPI-AWS">GitHub Repository</a>.</p><p>Testing this endpoint involves uploading sample data files and validating that the API processes the data correctly and returns accurate predictions using the machine learning model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-VFsboFHgXGyOxJQ4R4dRQ.png" /><figcaption>Data Upload Endpoint</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3FGDK4ShbPcQrdC7BdD8UQ.png" /><figcaption>Response Body</figcaption></figure><p>By thoroughly testing each of these API endpoints, you can ensure the overall functionality, reliability, and performance of your API, providing users with a seamless experience when interacting with it.</p><p>Now that the API is up and running on your local machine, it’s currently accessible only to you. However, to make it available to the public, it needs to be deployed to the cloud.</p><p>To achieve this, we’ll utilize AWS services, which provide robust infrastructure and scalable solutions for hosting applications. Let’s leverage the power of AWS to take your API to the next level and reach a broader audience.</p><h3>Deployment Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/716/0*pWviM-ZT35Y1ZKyr.png" /><figcaption>Deployment Architecture</figcaption></figure><p>When considering which AWS service to utilize for deployment, several crucial questions should arise:</p><ol><li><strong>Scaling</strong>: How can the hosting environment scale up seamlessly as traffic increases?</li><li><strong>Traffic Distribution</strong>: How can traffic be efficiently load-balanced across multiple instances?</li><li><strong>Updates Management</strong>: What is the process for performing rolling updates for the application?</li><li><strong>Security and Patching</strong>: How are operating systems and security patches managed?</li><li><strong>Cost Optimization</strong>: How can costs be optimized, considering that you pay for server uptime regardless of API usage?</li></ol><p>Given these considerations, serverless computing with AWS Lambda emerges as an optimal solution for hosting APIs. AWS Lambda addresses all these concerns effectively compared to EC2:</p><ul><li><strong>Auto-scaling</strong>: It automatically scales up or down in response to incoming traffic, ensuring that your API can handle any level of demand without manual intervention.</li><li><strong>Traffic Distribution</strong>: It integrates seamlessly with AWS API Gateway, which provides built-in capabilities for distributing incoming traffic across multiple Lambda instances.</li><li><strong>Rolling Updates</strong>: By decoupling code deployment from infrastructure management, AWS Lambda enables effortless rolling updates without disrupting service availability.</li><li><strong>Security and Patching</strong>: AWS manages the underlying infrastructure, including security and patching, ensuring that your API remains secure and up-to-date without additional maintenance overhead.</li><li><strong>Cost-effectiveness</strong>: With AWS Lambda, you only pay for the compute time consumed by your API requests, making it highly cost-effective compared to traditional server-based hosting models where you pay for server uptime regardless of usage.</li></ul><p>But, the problem with running FastAPI on AWS Lambda is that Lambda doesn&#39;t understand the FastAPI ASGI interface like@app.get(‘/’), @app.post(‘/’) etc. Instead, lambda understands something known as a lambda handler.</p><p>A lambda handler refers to the entry point of your AWS Lambda function. It’s a function in your code that Lambda invokes when the function is triggered. This lambda handler is the entry point for your Lambda function. When you create a Lambda function, you specify the name of the function and the name of the handler function.</p><pre>def lambda_handler(event, context):<br>       pass</pre><p>Here, event contains data that&#39;s passed into the function when it&#39;s invoked, and context provides runtime information.</p><p>In our example, when we trigger an AWS Lambda function through an API Gateway HTTP request, the event object that is passed to the Lambda function typically contains information about the HTTP request. This information is formatted as a JSON object and includes details such as the HTTP method, headers, query parameters, path parameters, and the request body.</p><p>To unpack the data and route it to the relevant FastAPI path or function, we can leverage the capabilities of the Mangum library.</p><h4>Magnum</h4><p><a href="https://mangum.io/"><strong>Magnum </strong></a>is a Python library designed to provide support for ASGI (Asynchronous Server Gateway Interface) and serves as a <em>bridge</em> between ASGI applications/frameworks and AWS Lambda, enabling developers to harness the power of serverless computing for their asynchronous web applications.</p><p>Magnum allows you to deploy ASGI applications as serverless functions on AWS Lambda. This integration enables you to leverage the scalability and pay-as-you-go pricing model of Lambda for your asynchronous web applications. This allows for efficient event-driven handling of incoming requests, maximizing performance and minimizing resource consumption.</p><p>Magnum integrates seamlessly with AWS API Gateway too, allowing you to expose your ASGI applications as HTTP endpoints accessible via API Gateway. This enables you to build RESTful APIs or web applications using ASGI frameworks and deploy them as serverless functions on Lambda.</p><h4><strong>Implementation</strong></h4><p>You can install it in your virtual environment using pip install mangumand wrap your entire app with the magnum handler.</p><pre># Create an instance of FastAPI<br>app = FastAPI(lifespan=ml_lifespan_manager, debug=True)<br><br># Mangum wrapper for AWS deployment<br>handler = Mangum(app)</pre><ol><li><strong>Create DockerFile: </strong>We have to package all our dependencies into a Docker container, for that we have to create the blueprint using DockerFile.</li></ol><pre>FROM public.ecr.aws/amazonlinux/amazonlinux:2023<br><br>WORKDIR /app<br><br># Install software<br>RUN yum update -y &amp;&amp; \<br>    yum install -y python39 &amp;&amp; \<br>    python3 -m ensurepip --upgrade &amp;&amp; \<br>    python3 -m pip install --upgrade setuptools<br><br>COPY ./requirements.txt /app/requirements.txt<br>COPY ./artifacts /app/artifacts<br>COPY ./app.py /app/app.py<br>COPY ./utils.py /app/utils.py<br><br>RUN python3 -m pip install -r requirements.txt<br><br>RUN echo &quot;Image Built&quot;<br><br># Set the CMD to your handler (could also be done as a parameter override outside of the Dockerfile) #&quot;python3&quot;, &quot;-u&quot;<br>CMD [&quot;app.handler&quot;]</pre><p><strong>2. Create AWS ECR Repository</strong>: Log in to the AWS Management Console, navigate to the Amazon ECR service, and create a new repository to store your Docker images.</p><p><strong>3. Build and Tag Docker Image</strong>: In your terminal, navigate to the root directory of your project and build the Docker image. Tag the image with the URI of your ECR repository.</p><pre>docker build -t ml_serving_fastapi .<br><br>docker tag ml_serving_fastapi:latest &lt;ecr-repo-path&gt;:latest<br><br>docker push &lt;ecr-repo-path&gt;:latest</pre><p><strong>4. Configure Lambda Container Image</strong>: Select “Container image” as the runtime type, and choose the ECR repository and image you pushed.</p><p><strong>5. Expose your API to the Internet: </strong>Below are the are 2 approaches but they differ in their complexity and feature set. Here’s a breakdown:</p><p><strong>A. Lambda Function URLs:</strong></p><ul><li><strong>Simple and fast setup:</strong> Easily add an HTTPS endpoint directly to your Lambda function with minimal configuration.</li><li><strong>Limited features:</strong> No support for request validation, caching, custom domains, throttling, advanced authorization, or integration with other AWS services.</li><li><strong>Best for:</strong> Simple microservices, webhooks, testing, and quick prototypes.</li><li><strong>Response time:</strong> Up to 15 minutes.</li></ul><p>You can simply create it from the Configuration tab:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jF2E2PnrwAwvrj_Oa58ItQ.png" /></figure><p><strong>B. API Gateway:</strong></p><ul><li><strong>Flexible and powerful:</strong> Create RESTful APIs, WebSocket APIs, and manage multiple Lambda functions together.</li><li><strong>Rich features:</strong> Request validation, caching, custom domains, throttling, integration with other AWS services, and more.</li><li><strong>More complex setup:</strong> Requires creating an API and configuring integrations with your Lambda functions.</li><li><strong>Best for:</strong> Production-ready APIs, secure access control, managing complex interactions with multiple services.</li><li><strong>Response time:</strong> Typically 30 seconds, but can be impacted by additional features.</li></ul><p>To create an API Gateway that interacts with your AWS Lambda function, follow my earlier tutorial:</p><p><a href="https://aws.plainenglish.io/deploying-llms-serverless-magic-with-lambda-sagemaker-dlc-and-api-gateway-1bf99517d43e">Deploying LLMs: Serverless Magic with Lambda, SageMaker DLC, and API Gateway</a></p><h3>Conclusion</h3><p>In conclusion, deploying a serverless machine learning inference solution using FastAPI and AWS offers a powerful combination of flexibility, scalability, and efficiency. Leveraging the robust capabilities of FastAPI for building high-performance APIs and the seamless scalability of AWS Lambda for executing code in response to incoming requests, developers can create and deploy ML models with ease.</p><p>By harnessing the agility of serverless computing, we can efficiently handle varying workloads, reduce operational overhead, and deliver real-time predictions to users at scale. With this approach, deploying ML inference becomes not only accessible but also cost-effective.</p><p>Refer to the complete code on Github:</p><p><a href="https://github.com/akashmathur-2212/Serverless-Deployment-Using-FastAPI-AWS">GitHub - akashmathur-2212/Serverless-Deployment-Using-FastAPI-AWS: Deploy a Serverless ML Inference using FastAPI and AWS Lambda</a></p><p>To refer to other AWS Serverless workflows, refer to this repo:</p><p><a href="https://github.com/akashmathur-2212/aws-serverless-workflows">GitHub - akashmathur-2212/aws-serverless-workflows: aws-serverless-analytics-automation</a></p><p><em>Thank you for reading this article, I hope it added some pieces to your knowledge stack! Before you go, if you enjoyed reading this article:</em></p><p>👉 <em>Be sure to </em><strong><em>clap</em></strong><em> and </em><strong><em>follow</em></strong><em> me, and let me know if any feedback.</em></p><p>👉<em>I built versatile Generative AI applications using the Large Language Model (LLM), covered advanced RAG concepts, and serverless AWS architectures for Big Data processing. You’re welcome to take a look at the repo and star⭐it.</em></p><ul><li><em>⭐ </em><strong><em>Generative AI</em></strong><em> applications </em><a href="https://github.com/akashmathur-2212/LLMs-playground"><em>Repository</em></a></li><li><em>⭐ </em><strong><em>Advanced RAG</em></strong><em> applications </em><a href="https://github.com/akashmathur-2212/LLMs-playground/tree/main/LlamaIndex-applications/Advanced-RAG"><em>Repository</em></a></li><li><em>⭐ Serverless </em><strong><em>AWS</em></strong><em> architectures </em><a href="https://github.com/akashmathur-2212/aws-serverless-workflows"><em>Repository</em></a></li></ul><p>👉<em>Follow me: </em><a href="https://www.linkedin.com/in/akashmathur22/"><strong><em>LinkedIn</em></strong></a> <strong><em>| </em></strong><a href="https://github.com/akashmathur-2212"><strong><em>GitHub</em></strong></a> <strong><em>| </em></strong><a href="https://akash-mathur.medium.com/"><strong><em>Medium</em></strong></a><strong><em> | </em></strong><a href="https://akashmathur-2212.github.io/"><strong><em>Portfolio</em></strong></a></p><h3>In Plain English 🚀</h3><p><em>Thank you for being a part of the </em><a href="https://plainenglish.io/"><strong><em>In Plain English</em></strong></a><em> community! Before you go:</em></p><ul><li>Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></li><li>Follow us: <a href="https://twitter.com/inPlainEngHQ"><strong>X</strong></a><strong> | </strong><a href="https://www.linkedin.com/company/inplainenglish/"><strong>LinkedIn</strong></a><strong> | </strong><a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong>YouTube</strong></a><strong> | </strong><a href="https://discord.gg/in-plain-english-709094664682340443"><strong>Discord</strong></a><strong> | </strong><a href="https://newsletter.plainenglish.io/"><strong>Newsletter</strong></a></li><li>Visit our other platforms: <a href="https://stackademic.com/"><strong>Stackademic</strong></a><strong> | </strong><a href="https://cofeed.app/"><strong>CoFeed</strong></a><strong> | </strong><a href="https://venturemagazine.net/"><strong>Venture</strong></a><strong> | </strong><a href="https://blog.cubed.run/"><strong>Cubed</strong></a></li><li>More content at <a href="https://plainenglish.io/"><strong>PlainEnglish.io</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=32b0b9a40236" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/deploy-a-serverless-ml-inference-using-fastapi-aws-lambda-and-api-gateway-32b0b9a40236">Deploy a Serverless ML Inference using FastAPI, AWS Lambda, and API Gateway</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Management in LlamaIndex: Smart Tracking and Debugging of Document Changes]]></title>
            <link>https://akash-mathur.medium.com/data-management-in-llamaindex-smart-tracking-and-debugging-of-document-changes-7b81c304382b?source=rss-eb7bcac979b6------2</link>
            <guid isPermaLink="false">https://medium.com/p/7b81c304382b</guid>
            <category><![CDATA[document-management]]></category>
            <category><![CDATA[tracking]]></category>
            <category><![CDATA[llamaindex]]></category>
            <category><![CDATA[retrieval-augmented]]></category>
            <category><![CDATA[debugging]]></category>
            <dc:creator><![CDATA[Akash Mathur]]></dc:creator>
            <pubDate>Fri, 02 Feb 2024 07:36:01 GMT</pubDate>
            <atom:updated>2024-02-17T11:21:15.787Z</atom:updated>
            <content:encoded><![CDATA[<h4>Efficient Document Tracing and Debugging</h4><h3>Welcome to the Advanced RAG 📚Learning Series!</h3><p>Dive deeper into the fascinating world of Retrieval-Augmented Generation with this comprehensive series of articles. This series delves into cutting-edge techniques and strategies to elevate your understanding and mastery of RAG applications. Explore the following articles to enhance your skills and <strong>Stay tuned </strong>🔔 for more articles in this series as we continue to delve deeper into the world of Advanced RAG and unlock its boundless potential.</p><p>Don’t miss out on any discoveries! Bookmark🏷️ this article and check back often for the latest installments in this exciting learning series.</p><p>Topics covered so far:</p><ol><li><a href="https://akash-mathur.medium.com/advanced-rag-optimizing-retrieval-with-additional-context-metadata-using-llamaindex-aeaa32d7aa2f">Optimizing Retrieval with Additional Context &amp; MetaData using LlamaIndex</a></li><li><a href="https://akash-mathur.medium.com/advanced-rag-enhancing-retrieval-efficiency-through-evaluating-reranker-models-using-llamaindex-3f104f24607e">Enhancing Retrieval Efficiency through Rerankers using LlamaIndex</a></li><li><a href="https://akash-mathur.medium.com/advanced-rag-query-augmentation-for-next-level-search-using-llamaindex-d362fed7ecc3">Query Augmentation for Next-Level Search using LlamaIndex</a></li><li>Smart Tracking and Debugging of Document Changes using LlamaIndex (you are here!)</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/0*WotPOtXwYvC9pJsO.jpg" /><figcaption>Document Tracking &amp; Debugging</figcaption></figure><p>In the world of handling private information, dealing with the constant changes in our data sources can be tricky. As we explore more ways to use RAG-based applications with different types of documents, questions arise about making sure it understands new content, managing changes in existing documents, and not giving outdated information from deleted documents.</p><p>The challenge lies in adapting to the ever-shifting nature of our private data sources efficiently. This is where LlamaIndex document management steps in to provide a smart solution.</p><p>In this blog post, we’ll take a closer look at how LlamaIndex tackles the complexities of changing data sources, ensuring accurate and quick information retrieval even as things keep evolving.</p><h3>Say ‘No’ to Reindexing Fatigue</h3><p>Indexing data repeatedly can increase costs in RAG-based applications. Here’s why:</p><ul><li><strong>Token Consumption: </strong>RAG applications rely on tokens to process and interact with data. Every time you reload, re-embed, and re-index your data, you consume tokens. This can accumulate quickly, especially with large private data sources.</li><li><strong>Computational Resources: </strong>Reindexing involves significant computational power to process and structure the data. Frequent reindexing demands more resources from your infrastructure, leading to higher operating costs.</li><li><strong>Time &amp; Efficiency:</strong> Reindexing takes time! Each iteration means your application is unavailable and users might experience delays. This impacts overall efficiency and user experience.</li></ul><p>To mitigate these challenges and control costs, it’s essential to implement efficient indexing strategies to strike a balance between maintaining data freshness and managing resource utilization effectively.</p><p>Let’s uncover the features that make LlamaIndex stand out in managing and adapting to the constant changes in your data.</p><ol><li><a href="#bc0a">Document Management in LlamaIndex</a></li></ol><p>1.1 <a href="#c445">Load Data</a></p><p>1.2 <a href="#1981">Refresh</a></p><p>1.2.1 <a href="#0f11">Dissecting the </a><a href="#0f11">docstore.json</a></p><p>1.3 <a href="#80e7">Delete</a></p><p>1.4 <a href="#349f">Update</a></p><p>2. <a href="#e731">Tracing and Debugging</a></p><p>2.1 <a href="#0306">Llama Debug Handler</a></p><p>2.2 <a href="#a0d3">Callback Management</a></p><p>3. <a href="#ec52">Key Takeaways</a></p><h3>Document Management in LlamaIndex</h3><p>LlamaIndex relies on three pivotal data structures — index struct, doc store, and vector store to proficiently manage documents.</p><p><strong>Index Struct</strong></p><p>The index struct is a fundamental data structure that serves as an organized and searchable reference to the documents within LlamaIndex. It contains information about the location and characteristics of documents, enabling rapid and efficient retrieval.</p><p><strong>Doc Store (Document Store)</strong></p><p>The docstore is a storage system specifically designed for holding the actual documents or content. It stores the raw text, metadata, or any other relevant information associated with each document.</p><p>This data structure acts as the repository for the content that the index struct points to, ensuring that the actual document details are readily available when needed.</p><blockquote>As soon as you persist your indexes to any form of storage, the docstore.json file will be generated.</blockquote><p><strong>Vector Store</strong></p><p>When we employ a vector index, a vector store comes into play. A vector index involves representing documents as vectors in a multi-dimensional space, allowing for similarity comparisons and advanced search functionalities.</p><p>The vector store holds these vector representations, facilitating efficient similarity searches and retrieval based on the spatial relationships between documents.</p><h3>Defining Documents</h3><p>Documents can either be created automatically via data loaders or constructed manually. By default, all of the <a href="https://docs.llamaindex.ai/en/stable/module_guides/loading/connector/root.html">data loaders</a> return Document objects through the load_data function.</p><p><strong>Significance of </strong><strong>doc_id and </strong><strong>ref_doc_id</strong></p><p>Connecting a docstoreto the ingestion, pipeline makes document management possible. The doc_id is used to enable the efficient refreshing of documents in the index. When using SimpleDirectoryReader, you can automatically set the doc doc_id to be the full path to each document.</p><p><strong>Note</strong>: the ID can also be set through the node_id or id_ property on a Document object.</p><p>By using either document.doc_id or node.ref_doc_id as a reference point, the ingestion pipeline actively checks for duplicate documents. Here’s how it operates:</p><ol><li>It maintains a map linking doc_id to document_hash.</li><li>When a duplicate doc_id is identified, and hash has been altered, the document undergoes re-processing.</li><li>If the hash remains unchanged, the document is skipped in the pipeline.</li></ol><h3>Document Management in Action</h3><p>You can follow alongside by clicking on the <a href="https://github.com/akashmathur-2212/LLMs-playground/tree/main/LlamaIndex-applications/Advanced-RAG/document_tracking_debugging">Link to the GitHub code repository</a></p><h4>1. Load Data</h4><p><strong>Accessing Google Drive from Google Colab</strong></p><p>You can use the drive module from google.colab to mount your entire Google Drive to Colab by:</p><pre>from google.colab import drive<br>drive.mount(&#39;/content/gdrive&#39;)<br><br>!ls &#39;/content/gdrive/MyDrive/document_management/&#39;<br><br><br>files_path = &#39;/content/gdrive/MyDrive/document_management/&#39;<br>index_persist_dir = &#39;./storage/document_management/&#39;<br><br>try:<br>    storage_context = StorageContext.from_defaults(persist_dir=index_persist_dir)<br>    index = load_index_from_storage(storage_context)<br>    print(&#39;loading from disk&#39;)<br>except:<br>    documents = SimpleDirectoryReader(files_path, filename_as_id=True).load_data()<br>    index = VectorStoreIndex.from_documents(documents, service_context=service_context)<br>    index.storage_context.persist(persist_dir=index_persist_dir)<br>    print(&#39;persisting to disk&#39;)</pre><p><strong>1.1 Checking File Contents: </strong>Let’s list the contents of a specific directory in Google Drive to confirm the presence of files to be indexed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/276/1*FIBhVOBe3epb5yD70Nomjg.png" /><figcaption>Original Files</figcaption></figure><p><strong>1.2 Setting Paths</strong>: Defines two key paths:</p><ul><li>files_path: The location of the documents to be indexed.</li><li>index_persist_dir: The directory where the index will be stored for future use.</li></ul><p><strong>1.3 Loading or Creating Index: </strong>It attempts to load an existing index from the storage directory to save processing time. If no index is found, it proceeds to create a new one by reading the documents from the files_path using a SimpleDirectoryReader. Then, it creates a VectorStoreIndex from those documents, preparing for efficient search and retrieval. Finally, persists (saves) the newly created index to the storage path for future use.</p><p>Let’s do a sanity check to ensure what documents the index has ingested. Each index has a ref_doc_info property which is a kind of mapping to the original doc IDs that we input.</p><pre>print(&#39;ref_docs ingested: &#39;, len(index.ref_doc_info))<br>print(&#39;number of input documents: &#39;, len(documents))<br><br>ref_docs ingested:  4<br>number of input documents:  4</pre><p>So, the index is properly inserted in all our documents.</p><p>Let’s load the index that we persisted/stored. You can also store it in any of the Vector Database and load it from there. Then, double-check to ensure the original number of documents are stored in the Memory or Vector Database.</p><pre>index = load_index_from_storage(StorageContext.from_defaults(persist_dir=index_persist_dir),<br>                                service_context=service_context)<br><br>print(&#39;ref_docs ingested: &#39;, len(index.ref_doc_info))<br>print(&#39;number of input documents: &#39;, len(documents))<br><br>**********<br>Trace: index_construction<br>**********<br>ref_docs ingested:  4<br>number of input documents:  4</pre><p><strong><em>Question: </em></strong><em>Let’s check a random question related to Andrew Huberman’s sleep podcast for which there is NO information present in the Vector DataStore yet.</em></p><pre>query_input = &quot;How music can help in sleep?&quot;<br>response = index.as_query_engine().query(query_input)<br><br>from llama_index.response.pprint_utils import pprint_response<br>pprint_response(response, show_source=True)</pre><p><strong><em>Response:</em></strong></p><p><em>Examine the below response and identify the top nodes&#39; content with low </em><strong><em>similarity scores</em></strong><em>. The nodes&#39; text pertains to the Neural Network, while the query is related to sleep. This situation is indicative of </em><strong><em>hallucination</em></strong><em>, where the model is fabricating an answer unrelated to the query.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/691/1*DNHcqfdA0JZD3P6xfnSycw.png" /><figcaption>Response 1</figcaption></figure><p>Also, let’s confirm by checking thedoc_id of the documents present in VectorStore. You will see that there is NO andrew_sleep.txt document present yet.</p><pre>for doc_num in range(len(documents)):<br>  print(f&quot;document-{doc_num} --&gt; {documents[doc_num].id_}&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/1*3DitaJs_eEhAMroi3tzpsQ.png" /><figcaption>There is no andrew_sleep.txt</figcaption></figure><p>If you want to debug further i.e. what was sent to LLM as a prompt and what it returned. Look at the <a href="#e731">Tracing and Debugging</a> section.</p><h4>2. Refresh</h4><p>By assigning a unique doc_id to each document during the data loading process, you gain the ability to seamlessly update your index through the use of the refresh() function.</p><blockquote>This function <strong><em>selectively </em></strong>updates documents that share the same doc_id but have different text contents. Furthermore, any documents absent from the index entirely will be seamlessly inserted during the refresh process.</blockquote><p>As an additional benefit, the refresh() function provides a boolean list, offering insights into which documents from the input have been successfully refreshed within the index.</p><p>This straightforward approach ensures efficient and targeted updates to your index based on unique document identifiers.</p><p>Let’s add andrew_huberman_sleep.txt and see how it responds:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/258/1*_kJgaut-W4HF2dwwoIDqPw.png" /><figcaption>Added a New Document</figcaption></figure><pre># reading files again<br>documents = SimpleDirectoryReader(files_path, filename_as_id=True).load_data()<br>print(f&quot;Loaded {len(documents)} docs&quot;)<br><br>&lt;&lt;Loaded 5 docs&gt;&gt;</pre><pre># Run refresh_ref_docs method to check for document updates<br>refreshed_docs = index.refresh_ref_docs(documents)<br><br>print(refreshed_docs, &quot;\n&quot;)<br>print(&#39;Number of newly inserted/refreshed docs: &#39;, sum(refreshed_docs))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/354/1*xqb2zpygvFVbkH2iLw2L6A.png" /><figcaption>Number of newly inserted/refreshed docs</figcaption></figure><p>Since, we have added the new document andrew_huberman_sleep.txt, the number of newly inserted/refreshed docs equal to 1</p><p><strong><em>Question: </em></strong><em>Let’s again ask the earlier question related to Andrew Huberman’s sleep podcast.</em></p><pre>query_input = &quot;How music can help in sleep?&quot;<br>response = index.as_query_engine().query(query_input)<br><br>from llama_index.response.pprint_utils import pprint_response<br>pprint_response(response, show_source=True)</pre><p><strong><em>Response: </em></strong><em>Voila! Examine the response and identify the top nodes’ content with </em><em>higher similarity scores. Now, the nodes’ text pertains to the sleep podcast text, which is very well correlated with our query.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/575/1*ikuXMQ5p_t9s0f52tT_VGQ.png" /><figcaption>Response 2</figcaption></figure><p>Also, let’s confirm by checking doc_id each of the documents present. You will see that andrew_sleep.txt the document is present <strong>now</strong>.</p><pre>for doc_num in range(len(documents)):<br>  print(f&quot;document-{doc_num} --&gt; {documents[doc_num].id_}&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/1*C0iuxiJK2CYCOnbFEFs4EA.png" /><figcaption>document-0 andrew_sleep.txt</figcaption></figure><p>Let’s look at the index.ref_doc_info to find out the doc_id and RefDocInfo</p><pre>info_dict = index.ref_doc_info<br>info_dict</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8dChN3thFe6vheiINNDzdg.png" /><figcaption>ref_doc_info</figcaption></figure><p><strong>Insights</strong>: Every document (identified by doc_id) has a RefDocInfo that holds all the parsed nodes from that document. For example, the document evaluation_metrics.txt has been parsed into 5 nodes. Customizing metadata for each document is possible and can be tailored to your specific use case. For additional guidance, refer to this <a href="https://docs.llamaindex.ai/en/stable/module_guides/loading/documents_and_nodes/usage_documents.html">comprehensive guide</a>.</p><h4><strong>Dissecting the </strong><strong>docstore.json</strong></h4><p>For indexes, by leveraging the docstore.json, you gain the capability to view the documents you’ve inserted. The output displays entries with ingested doc ids as keys, along with their corresponding node_ids indicating the nodes they were split into. Additionally, the original metadata dictionary of each input document can also be tracked for comprehensive visibility into the stored information.</p><p>The <strong>docstore.json</strong>maintains a map linking doc_id to document_hash. When a duplicate doc_id is identified, and hash has been altered, the document undergoes re-processing. If the hash remains unchanged, the document is skipped in the pipeline. When you update/delete any part of this document, the document_hash will be updated with the next index refresh.</p><p>It contains reference document information (ref_doc_info), metadata providing additional details about each document, and the actual textual content or data that the model uses for retrieval and generation. These components collectively enable the RAG model to effectively navigate and utilize the stored documents during the retrieval process.</p><p>Let’s look at each of the components:</p><ol><li><strong>Ref_doc_info</strong></li></ol><ul><li>It refers to the reference document information associated with each document in the docstore.</li><li>It typically includes details about how the document is tokenized, its embeddings, and any other information necessary for retrieval.</li><li>This information is crucial for the model to effectively retrieve relevant content during the generation phase.</li></ul><pre>data[&#39;docstore/ref_doc_info&#39;]</pre><p>Same as index.ref_doc_info</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d74wkEOuZGh3G-YtkkNbtQ.png" /><figcaption>docstore/ref_doc_info</figcaption></figure><p>2. <strong>metadata</strong></p><ul><li>It encompasses additional information about each document beyond its textual content.</li><li>This could include attributes such as the document’s title, author, publication date, or any other relevant metadata.</li><li>Metadata is valuable for providing context and additional details about the documents stored in the docstore.</li></ul><pre>data[&#39;docstore/metadata&#39;]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pAdK8GN5Ta7u6soRvZRgNw.png" /><figcaption>document_hash for evaluation_metrics.txt document</figcaption></figure><p>3. <strong>data</strong></p><ul><li>It refers to the actual content of the documents. This could include the original text or any other textual information that the model uses for retrieval and generation.</li></ul><pre>data[&#39;docstore/data&#39;]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EWyyWmn7gObImntGv5Bzgw.png" /><figcaption>docstore/data</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GNQVLuTN-N4jiNi60YmN9g.png" /><figcaption>docstore/data</figcaption></figure><h4>3. <strong>Delete</strong></h4><p>Deleting a document from the index is a straightforward process. It can be done by specifying the doc_id. Upon deletion, all nodes associated with the specified document are removed.</p><pre>index.delete_ref_doc(&quot;/content/gdrive/MyDrive/document_management/imbalanced_data.txt&quot;, delete_from_docstore=True)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/225/1*aYcq1KTk5CsGepq65Ck0bA.png" /><figcaption>Deleted a Document</figcaption></figure><p>Let’s double-check:</p><pre>for doc_num in range(len(documents)):<br>  print(f&quot;document-{doc_num} --&gt; {documents[doc_num].id_}&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/641/1*JWicmA9j0zZqiV2N382YhQ.png" /><figcaption>documents with their ID</figcaption></figure><p><strong><em>Question: </em></strong><em>Let’s again ask the question related to the deleted document imbalanced_data.txt.</em></p><pre>query_input = &quot;What are the various techniques mentioned in the document to handle imbalanced data?&quot;<br>response = index.as_query_engine().query(query_input)</pre><p><strong><em>Response: </em></strong><em>Examine the response and identify the top nodes’ content (surprisingly we have a slightly higher similarity score) but the nodes’ text pertains to the evaluation metric MAE which is coming from evaluation_metrics.txt. This situation is indicative of hallucination, where the model is fabricating an answer unrelated to the query.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/570/1*pTdJdzVwoVNCXY7qo33R4g.png" /><figcaption>Response 3</figcaption></figure><h4>4. <strong>Update</strong></h4><p>To modify information within an index, you can perform an update on a document with the same doc_id if it is already present. This allows you to seamlessly incorporate changes to the document, ensuring that the updated information is reflected within the index.</p><p>Now, I have updated(appended) the text in neural_network.txt with the content fromimbalanced_data.txt.</p><pre>documents = SimpleDirectoryReader(files_path, filename_as_id=True).load_data()<br>print(f&quot;Loaded {len(documents)} docs&quot;)<br><br>&lt;&lt;Loaded 4 docs&gt;&gt;<br><br># Run refresh_ref_docs method to check for document updates<br>refreshed_docs = index.refresh_ref_docs(documents)<br><br>print(refreshed_docs, &quot;\n&quot;)<br>print(&#39;Number of newly inserted/refreshed docs: &#39;, sum(refreshed_docs))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/356/1*Ta3dvLsiRHgYh1iEWvo7tQ.png" /><figcaption>refreshed docs</figcaption></figure><p><strong><em>Question: </em></strong><em>Let’s again ask the question related to the imbalanced_data whose text has been updated in the neural_network.txt.</em></p><pre>query_input = &quot;What is SMOTE technique mentioned in the document to handle imbalanced data?&quot;<br>response = index.as_query_engine().query(query_input)<br><br>from llama_index.response.pprint_utils import pprint_response<br>pprint_response(response, show_source=True)</pre><p><strong><em>Response: </em></strong><em>Better response!</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/715/1*Rcx8A3PWFCdaSTowBNr_gw.png" /><figcaption>Response 4</figcaption></figure><p><strong>Note</strong>: You can try updating the document towards the start, middle, or end and compare the responses.</p><h3>Tracing and Debugging</h3><p>Let’s debug further using basic logging.This can be done anywhere in your application like this:</p><pre>import logging<br>import sys<br><br>logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)<br>logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))</pre><h4>Llama Debug Handler</h4><p>Now, we will learn how we can debug our queries in LlamaIndex more efficiently. Each query undergoes distinct stages within LlamaIndex before delivering the final response. Understanding these steps, including the time taken by each subprocess, can significantly improve the debugging process.</p><p>Beyond queries, data ingestion in LlamaIndex experiences various stages before becoming available for querying. The key advantages of effective debugging are:</p><ol><li>Verify the time consumed by each subprocess easily.</li><li>Identify processes that contribute more or less time during query operations.</li><li>Examine the values held by these subprocesses at different stages, both during Data Ingestion and Query operations.</li></ol><p>What makes this debugging process even more accessible is that there’s no need for an external library or tool. LlamaIndex itself provides a debugging solution through a feature known as the Llama Debug Handler.</p><p>Let’s explore how to leverage this feature in code to gain a deeper understanding of its capabilities.</p><h4><strong>Callback Management</strong></h4><p>LlamaIndex offers a robust callback system designed to facilitate debugging, monitoring, and tracing the library’s internal processes. Beyond logging event-related data, we can monitor the duration and frequency of each occurrence. Additionally, a trace map of events is maintained, providing callbacks the flexibility to utilize this data as needed.</p><p>For instance, the default behavior of LlamaDebugHandler involves printing the trace of events after most operations. Obtaining a straightforward callback handler can be achieved as follows:</p><pre>from llama_index.callbacks import CallbackManager, LlamaDebugHandler<br><br>import nest_asyncio<br>nest_asyncio.apply()<br><br># Using the LlamaDebugHandler to print the trace of the underlying steps<br># regarding each sub-process for document ingestion part <br>llama_debug = LlamaDebugHandler(print_trace_on_end=True)<br>callback_manager = CallbackManager([llama_debug])<br><br># ServiceContext<br># Service context contains information about which LLM we <br># are going to use for response generation and which <br># embedding model to create embeddings<br>service_context = ServiceContext.from_defaults(llm=llm,<br>                                               embed_model=embed_model,<br>                                               callback_manager=callback_manager<br>                                               )</pre><p>Let’s check out the logs to debug what happens when we query our document using LlamaIndex.</p><pre>query_input = &quot;How music can help in sleep?&quot;<br>response = index.as_query_engine().query(query_input)</pre><p>After running this piece of code you’ll see what happens when you query your data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/328/1*HtbiyyBzD5gLn8FoDoKTKg.png" /><figcaption>Tracing the sub-processes</figcaption></figure><p>Isn’t it great! Every part of the process is shown as logs for you to debug if there is any issue in your case.</p><p>Llama Debug Handler not only shows the time for each process. In fact, at any time of the process, you can check and debug every process and find out what is going on here.</p><p>For example, If you want to check how many LLM calls you have made until the file is running, You can do that simply by adding this line of code at the end of the above code.</p><pre># Print info on the LLM calls during the list index query<br>print(llama_debug.get_event_time_info(CBEventType.LLM))<br><br># Output on the terminal will be like this<br>EventStats(total_secs=1.509315, average_secs=1.509315, total_count=1)</pre><p>If you want to check what was sent to LLM and what LLM returned, You can do that by simply adding this line of code at the bottom.</p><pre># Print info on llm inputs/outputs - returns start/end events for each LLM call<br>event_pairs = llama_debug.get_llm_inputs_outputs()<br>print(event_pairs[0][0]) # Shows what was sent to LLM<br>print(event_pairs[0][1].payload.keys()) # What other things you can debug by simply passing the key<br>print(event_pairs[0][1].payload[&quot;completion&quot;]) # Shows the LLM response it generated.</pre><p>Let’s look at the Payloadsent to LLM regarding our Sleep/Music query:</p><pre># Look at the Payload prompt sent to LLM<br># It has no mention of Sleep/Music etc.<br>event_pairs[0][0].payload[&#39;formatted_prompt&#39;]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IBW5gUIpnscuH7AdLF3FEA.png" /><figcaption>Payload (Prompt)</figcaption></figure><p>Let’s look at the Response(Completion)from the LLM :</p><pre># clearly a case of halluciniation<br>print(event_pairs[0][1].payload[&quot;completion&quot;]) # Shows the LLM response it generated.</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8UpPps8d5TcQ3oyqfm71KQ.png" /><figcaption>Response(Completion)</figcaption></figure><p>Llama Debug Handler not only contains information about the current state of LLM but there’s a lot more that you can do with it. For example, You can debug the following other parts the same way we did for the LLM one.</p><ul><li>CBEventType.LLM</li><li>CBEventType.EMBEDDING</li><li>CBEventType.CHUNKING</li><li>CBEventType.NODE_PARSING</li><li>CBEventType.RETRIEVE</li><li>CBEventType.SYNTHESIZE</li><li>CBEventType.TREE</li><li>CBEventType.QUERY</li></ul><p>Refer to <a href="https://docs.llamaindex.ai/en/stable/module_guides/observability/callbacks/root.html">LlamaIndex </a>for more information.</p><h3>Key Takeaways</h3><ol><li>LlamaIndex relies on three pivotal data structures — index struct, doc store, and vector store (if utilizing a vector index) — to proficiently manage documents.</li><li><strong>Constantly Refreshing Data: </strong>LlamaIndex showcases its adaptability by efficiently indexing and continuously refreshing data, making it a versatile and dynamic document management tool.</li><li><strong>Enhancing Efficiency:</strong> Leveraging document management tools like LlamaIndex proves invaluable for organizing and analyzing large datasets, contributing to time savings and increased operational efficiency.</li><li><strong>Time and Token Savings: </strong>Saving and loading indexes can save time and tokens when working with document management systems.</li><li><strong>Importance of Regular Updates</strong>: The significance of regularly updating and managing document indexes for staying current and relevant.</li><li><strong>Caution in Deletion</strong>: While LlamaIndex offers the capability to update and insert documents based on content changes or new doc IDs, caution is advised when deleting from the doc store to prevent unintended consequences.</li></ol><p>Refer to the complete code on Github:</p><p><a href="https://github.com/akashmathur-2212/LLMs-playground/tree/main/LlamaIndex-applications/Advanced-RAG/document_tracking_debugging">LLMs-playground/LlamaIndex-applications/Advanced-RAG/document_tracking_debugging at main · akashmathur-2212/LLMs-playground</a></p><p>To refer to other advanced RAG methods, refer to this repo:</p><p><a href="https://github.com/akashmathur-2212/LLMs-playground/tree/main/LlamaIndex-applications/Advanced-RAG?source=post_page-----d362fed7ecc3--------------------------------">LLMs-playground/LlamaIndex-applications/Advanced-RAG at main · akashmathur-2212/LLMs-playground</a></p><h3>Reference</h3><ol><li><a href="https://docs.llamaindex.ai/en/stable/examples/ingestion/document_management_pipeline.html">https://docs.llamaindex.ai/en/stable/examples/ingestion/document_management_pipeline.html</a></li><li><a href="https://docs.llamaindex.ai/en/stable/understanding/tracing_and_debugging/tracing_and_debugging.html#basic-logging">https://docs.llamaindex.ai/en/stable/understanding/tracing_and_debugging/tracing_and_debugging.html#basic-logging</a></li><li><a href="https://docs.llamaindex.ai/en/stable/module_guides/loading/documents_and_nodes/usage_documents.html">https://docs.llamaindex.ai/en/stable/module_guides/loading/documents_and_nodes/usage_documents.html</a></li></ol><p><em>Thank you for reading this article, I hope it added some pieces to your knowledge stack! Before you go, if you enjoyed reading this article:</em></p><p>👉 <em>Be sure to </em><strong><em>clap</em></strong><em> and </em><strong><em>follow</em></strong><em> me, and let me know if any feedback.</em></p><p>👉<em>I built versatile Generative AI applications using the Large Language Model (LLM), covered advanced RAG concepts, and serverless AWS architectures for Big Data processing. You’re welcome to take a look at the repo and star⭐it.</em></p><ul><li><em>⭐ </em><strong><em>Generative AI</em></strong><em> applications </em><a href="https://github.com/akashmathur-2212/LLMs-playground"><em>Repository</em></a></li><li><em>⭐ </em><strong><em>Advanced RAG</em></strong><em> applications </em><a href="https://github.com/akashmathur-2212/LLMs-playground/tree/main/LlamaIndex-applications/Advanced-RAG"><em>Repository</em></a></li><li><em>⭐ Serverless </em><strong><em>AWS</em></strong><em> architectures </em><a href="https://github.com/akashmathur-2212/aws-serverless-workflows"><em>Repository</em></a></li></ul><p>👉<em>Follow me: </em><a href="https://www.linkedin.com/in/akashmathur22/"><strong><em>LinkedIn</em></strong></a> <strong><em>| </em></strong><a href="https://github.com/akashmathur-2212"><strong><em>GitHub</em></strong></a> <strong><em>| </em></strong><a href="https://akash-mathur.medium.com/"><strong><em>Medium</em></strong></a><strong><em> | </em></strong><a href="https://akashmathur-2212.github.io/"><strong><em>Portfolio</em></strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7b81c304382b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Advanced RAG: Query Augmentation for Next-Level Search using LlamaIndex]]></title>
            <link>https://akash-mathur.medium.com/advanced-rag-query-augmentation-for-next-level-search-using-llamaindex-d362fed7ecc3?source=rss-eb7bcac979b6------2</link>
            <guid isPermaLink="false">https://medium.com/p/d362fed7ecc3</guid>
            <category><![CDATA[query]]></category>
            <category><![CDATA[retrieval-augmented]]></category>
            <category><![CDATA[search]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[llamaindex]]></category>
            <dc:creator><![CDATA[Akash Mathur]]></dc:creator>
            <pubDate>Thu, 18 Jan 2024 11:05:03 GMT</pubDate>
            <atom:updated>2024-02-02T07:40:18.826Z</atom:updated>
            <content:encoded><![CDATA[<h4>Using Open Source LLM <a href="https://huggingface.co/HuggingFaceH4/zephyr-7b-alpha">Zephyr-7b-alpha</a> and BGE Embeddings <a href="https://huggingface.co/BAAI/bge-large-en-v1.5">bge-large-en-v1.5</a></h4><h3>Welcome to the Advanced RAG 📚Learning Series!</h3><p>Dive deeper into the fascinating world of Retrieval-Augmented Generation with this comprehensive series of articles. This series delves into cutting-edge techniques and strategies to elevate your understanding and mastery of RAG applications. Explore the following articles to enhance your skills and <strong>Stay tuned </strong>🔔 for more articles in this series as we continue to delve deeper into the world of Advanced RAG and unlock its boundless potential.</p><p>Don’t miss out on any discoveries! Bookmark🏷️ this article and check back often for the latest installments in this exciting learning series.</p><p>Topics covered so far:</p><ol><li><a href="https://akash-mathur.medium.com/advanced-rag-optimizing-retrieval-with-additional-context-metadata-using-llamaindex-aeaa32d7aa2f">Optimizing Retrieval with Additional Context &amp; MetaData using LlamaIndex</a></li><li><a href="https://akash-mathur.medium.com/advanced-rag-enhancing-retrieval-efficiency-through-evaluating-reranker-models-using-llamaindex-3f104f24607e">Enhancing Retrieval Efficiency through Rerankers using LlamaIndex</a></li><li>Query Augmentation for Next-Level Search using LlamaIndex (you are here!)</li><li><a href="https://akash-mathur.medium.com/data-management-in-llamaindex-smart-tracking-and-debugging-of-document-changes-7b81c304382b">Smart Tracking and Debugging of Document Changes using LlamaIndex</a></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2mlluIXS05R7a03CvwxW4g.png" /></figure><p>In the realm of information retrieval, Retriever-Augmented Generation (RAG) models have marked a paradigm shift, empowering large language models (LLMs) to generate contextually rich and accurate responses. However, the path to unlocking RAG’s full potential often lies beyond the limits of its default query-retrieval-generation framework.</p><p>This article delves into the transformative power of advanced query transformation techniques designed to <em>bridge the gap between initial user prompts and the most relevant information within vast databases</em>.</p><p><strong>The Challenge of Retrieval Misalignment</strong></p><p>At the heart of query transformations lies a fundamental challenge: user-generated prompts often lack the precise language or structure that aligns seamlessly with the wording of relevant documents. This misalignment can hinder retrieval efforts, leading to suboptimal responses from even the most sophisticated LLMs. Query transformations address this challenge by strategically modifying queries before the retrieval stage, enhancing their relevance, and guiding the LLM towards better information extraction.</p><p><strong>Problem with Zero-Shot Challenges</strong></p><p>Recent research has shed light on the benefits of breaking down complex queries into smaller, more manageable steps, a technique particularly effective for queries that require knowledge augmentation. However, fully zero-shot dense retrieval systems, where relevance labels are absent, continue to pose significant challenges. Advanced query transformations emerge as a promising approach to address these challenges, offering innovative strategies to navigate this challenge.</p><blockquote>The idea behind query transformations is that the retriever may not consider a user’s initial prompt to retrieve semantically similar documents. However, it will modify the query to increase its relevance to our sources before retrieving and feeding them to the language model.</blockquote><p>There are many techniques for enhancing RAG, creating the additional challenge of knowing when to apply each. In this article, we will analyze <strong>5</strong> powerful query transformation techniques and will see how they can help to bridge the retrieval gap and perform <strong>next-level search</strong>.</p><ol><li><a href="#52e2">Hypothetical Document Embeddings (HyDE)</a></li><li><a href="#ce5d">Sub-Question Query Engine</a></li><li><a href="#351c">Router Query Engine</a></li><li><a href="#45a5">Single-Step Query Decomposition</a></li><li><a href="#46d4">Multi-Step Query Decomposition</a></li></ol><p><strong>Knowledge and Action, Hand in Hand</strong></p><p>This journey won’t end with theoretical insights alone. Alongside each technique, you’ll find references to a dedicated GitHub repository, providing the code samples and implementation details.</p><p>Let’s dive into it.</p><h3>Open Source LLM and Embedding</h3><p><strong>LLM</strong>: Throughout this exploration, we’ll harness the power of <a href="https://huggingface.co/HuggingFaceH4/zephyr-7b-alpha">Zephyr-7b-alpha</a>, a state-of-the-art open-source LLM renowned for its remarkable capabilities in understanding and generating text.</p><p><strong>Embeddings</strong>: We will use BGE embeddings (<a href="https://huggingface.co/BAAI/bge-large-en-v1.5">bge-large-en-v1.5</a>), a general-purpose embedding model to enable effective semantic search and knowledge extraction.</p><p>This embedding model ranks 5th in the <a href="https://huggingface.co/spaces/mteb/leaderboard">MTEB Embedding Benchmark</a>. Also, check out their repo <a href="https://github.com/FlagOpen/FlagEmbedding">https://github.com/FlagOpen/FlagEmbedding</a></p><p>Let’s jump into the code.</p><pre>import logging<br>import sys<br><br>logging.basicConfig(stream=sys.stdout, level=logging.INFO)<br>logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))<br><br>import json<br>import torch<br>from pathlib import Path<br>import pandas as pd<br>pd.set_option(&quot;display.max_colwidth&quot;, -1)<br><br>from copy import deepcopy<br><br># transformers<br>from transformers import BitsAndBytesConfig<br><br># llama_index<br>from llama_index.prompts import PromptTemplate<br>from llama_index.llms import HuggingFaceLLM<br>from llama_index import download_loader, Document, VectorStoreIndex, ServiceContext<br>from llama_index.node_parser import SentenceSplitter<br>from langchain.embeddings import HuggingFaceEmbeddings<br><br>from llama_index.indices.query.query_transform import HyDEQueryTransform<br>from llama_index.query_engine.transform_query_engine import TransformQueryEngine<br><br>from IPython.display import Markdown, display<br>from llama_index.response.notebook_utils import display_source_node<br><br>from llama_index.query_engine import RetrieverQueryEngine<br>from IPython.display import Markdown, display, HTML<br>from llama_index.retrievers import VectorIndexRetriever<br><br>from sentence_transformers import SentenceTransformer<br><br>DEVICE = &quot;cuda:0&quot; if torch.cuda.is_available() else &quot;cpu&quot;</pre><pre>#Load Data<br><br>PDFReader = download_loader(&quot;PDFReader&quot;)<br>loader = PDFReader()<br>docs = loader.load_data(file=Path(&quot;QLoRa.pdf&quot;))<br><br># create chunks<br>node_parser = SentenceSplitter(chunk_size=256)<br>nodes = node_parser.get_nodes_from_documents(docs)</pre><pre>#Load Open Source LLM (zephyr-7b-alpha)<br><br>from google.colab import userdata<br><br># huggingface api token<br>hf_token = userdata.get(&#39;hf_token&#39;)<br><br>quantization_config = BitsAndBytesConfig(<br>    load_in_4bit=True,<br>    bnb_4bit_compute_dtype=torch.float16,<br>    bnb_4bit_quant_type=&quot;nf4&quot;,<br>    bnb_4bit_use_double_quant=True,<br>)<br><br><br>def messages_to_prompt(messages):<br>  prompt = &quot;&quot;<br>  for message in messages:<br>    if message.role == &#39;system&#39;:<br>      prompt += f&quot;&lt;|system|&gt;\n{message.content}\n&quot;<br>    elif message.role == &#39;user&#39;:<br>      prompt += f&quot;&lt;|user|&gt;\n{message.content}\n&quot;<br>    elif message.role == &#39;assistant&#39;:<br>      prompt += f&quot;&lt;|assistant|&gt;\n{message.content}\n&quot;<br><br>  # ensure we start with a system prompt, insert blank if needed<br>  if not prompt.startswith(&quot;&lt;|system|&gt;\n&quot;):<br>    prompt = &quot;&lt;|system|&gt;\n\n&quot; + prompt<br><br>  # add final assistant prompt<br>  prompt = prompt + &quot;&lt;|assistant|&gt;\n&quot;<br><br>  return prompt<br><br><br>llm = HuggingFaceLLM(<br>    model_name=&quot;HuggingFaceH4/zephyr-7b-alpha&quot;,<br>    tokenizer_name=&quot;HuggingFaceH4/zephyr-7b-alpha&quot;,<br>    query_wrapper_prompt=PromptTemplate(&quot;&lt;|system|&gt;\n\n&lt;|user|&gt;\n{query_str}\n&lt;|assistant|&gt;\n&quot;),<br>    context_window=3900,<br>    max_new_tokens=256,<br>    model_kwargs={&quot;quantization_config&quot;: quantization_config},<br>    # tokenizer_kwargs={},<br>    generate_kwargs={&quot;temperature&quot;: 0.7, &quot;top_k&quot;: 50, &quot;top_p&quot;: 0.95, &quot;do_sample&quot;:True},<br>    messages_to_prompt=messages_to_prompt,<br>    device_map=&quot;auto&quot;,<br>)</pre><pre>#Load Open Embedding (bge-large-en-v1.5)<br><br>embed_model = HuggingFaceEmbeddings(model_name=&quot;BAAI/bge-large-en-v1.5&quot;)</pre><p>Before applying any query transformation, configure the Index and Retriever</p><pre># ServiceContext<br>service_context = ServiceContext.from_defaults(llm=llm,<br>                                               embed_model=embed_model<br>                                               )<br><br># index<br>vector_index = VectorStoreIndex(<br>    nodes, service_context=service_context<br>)</pre><p>Let’s discuss and apply the above-mentioned query transformations one by one.</p><h3><strong>1. </strong>Hypothetical Document Embeddings (HyDE)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/957/1*w-dn2tNkwjjzQoTrW2wRJA.png" /><figcaption>HyDE — Under the hood</figcaption></figure><p><a href="http://boston.lti.cs.cmu.edu/luyug/HyDE/HyDE.pdf">HyDE</a> (Hypothetical Document Embeddings) is a novel approach to dense retrieval that involves two distinct phases:</p><p><strong>1. Generating a Hypothetical Answer</strong></p><ul><li>Instead of directly searching for relevant documents based on the raw query, HyDE first constructs a hypothetical document that <em>might</em> answer the query.</li><li>This is achieved by utilizing an instruction-following language model, tasked with generating a likely response to the query.</li><li>While this hypothetical document may not be factually accurate in every detail, it serves as a valuable example of what a relevant document could look like, capturing the essence of relevance.</li></ul><p><strong>2. Encoding and Retrieval</strong></p><ul><li>The hypothetical document is then processed by an unsupervised contrastive encoder, which distills its key features into a compact embedding vector.</li><li>Importantly, the encoder’s dense bottleneck acts as a lossy compressor, filtering out irrelevant details.</li><li>This embedding vector is then compared against a database of corpus embeddings, representing actual documents.</li><li>The retrieval process leverages <strong>document-document</strong> similarity encoded within the inner product during contrastive training, enabling the identification of documents that closely align with the hypothetical answers.</li><li>The most similar real documents are retrieved and presented as potential responses to the query, enhancing retrieval accuracy.</li></ul><p>Let’s jump into the code.</p><p>First, we query without transformation. Then, the same query string is used for embedding lookup and also summarization.</p><pre>query_str = &quot;Describe the trade-offs between using BFloat16 as the computation data type and other possible choices. When would you choose one over the other?&quot;<br><br>query_engine = vector_index.as_query_engine()<br>response = query_engine.query(query_str)<br>display(Markdown(f&quot;{response}&quot;))</pre><p>Response:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/899/1*AY3cXRs9hBwHHPzUvvRpvw.png" /><figcaption>Response without HyDe</figcaption></figure><p>Let’s apply HyDe transformation and see the results</p><pre>hyde = HyDEQueryTransform(include_original=True, llm=llm)<br>hyde_query_engine = TransformQueryEngine(query_engine, hyde)<br>response = hyde_query_engine.query(query_str)<br>display(Markdown(f&quot;{response}&quot;))</pre><p>Response:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/903/1*CYTrYn46x1a7s7Jc-gqQnA.png" /><figcaption>Response with HyDe</figcaption></figure><p>Let’s look at the hypothetical document. We use `HyDEQueryTransform` to generate a hypothetical document and use it for embedding lookup.</p><pre>query_bundle = hyde(query_str)<br>hyde_doc = query_bundle.embedding_strs[0]<br>hyde_doc</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/905/1*VBYieDkDGadUd-r-72cvfA.png" /><figcaption>hyde_doc</figcaption></figure><p><strong>Conclusion </strong>— You can see HyDE improves output quality significantly, by hallucinating accurately, thus improving the embedding quality and final output.</p><p><a href="https://github.com/akashmathur-2212/LLMs-playground/blob/main/LlamaIndex-applications/Advanced-RAG/advanced_query_transformations/Advanced_Query_Transformations.ipynb">Link to the GitHub code repository</a></p><h3>2. Sub-Question Query Engine</h3><p><strong>Understanding Traditional Query Engines</strong></p><p>Normal query engines are designed to locate relevant information within vast datasets. They act as intermediaries between users’ questions and stored data. When a user poses a query, the engine carefully analyzes it, pinpoints relevant data, and presents a comprehensive response.</p><p><strong>Limitations of Traditional Query Engines</strong></p><p>While traditional query engines excel at straightforward questions, they often face challenges when confronted with multi-faceted questions spanning multiple documents.</p><p>Simply merging documents and extracting top k elements frequently fails to capture the nuances required for truly informative responses.</p><p><strong>Enter Sub-Question Query Engines</strong></p><p><strong>Decomposition Strategy</strong>: To address this complexity, Sub-Question Query Engines adopt a divide-and-conquer approach. They elegantly decompose complex queries into a series of sub-questions, each targeting specific aspects of the original inquiry.</p><p>The implementation involves defining a Sub-Question Query Engine for each data source. Instead of treating all documents equally, the engine strategically addresses sub-questions specific to each data source. To generate the final response, a top-level Sub-Question Query Engine is then employed to synthesize the results from individual sub-questions.</p><p>Given the initial complex question, we use LLM to generate sub-questions and execute sub-questions on selected data sources. It gathers all sub-responses and then synthesizes the final response.</p><p>Let’s jump into the code.</p><pre>from llama_index.tools import QueryEngineTool, ToolMetadata<br>from llama_index.query_engine import SubQuestionQueryEngine<br>from llama_index.callbacks import CallbackManager, LlamaDebugHandler<br><br>import nest_asyncio<br>nest_asyncio.apply()<br><br># Using the LlamaDebugHandler to print the trace of the sub questions<br># captured by the SUB_QUESTION callback event type<br>llama_debug = LlamaDebugHandler(print_trace_on_end=True)<br>callback_manager = CallbackManager([llama_debug])<br><br># ServiceContext<br>service_context = ServiceContext.from_defaults(llm=llm,<br>                                               embed_model=embed_model,<br>                                               callback_manager=callback_manager<br>                                               )<br><br># vector query engine<br>vector_query_engine = VectorStoreIndex.from_documents(<br>    docs, service_context=service_context, use_async=True<br>).as_query_engine()</pre><p>Construct sub-question query engine and run some queries!</p><pre># setup base query engine as tool<br>query_engine_tools = [<br>    QueryEngineTool(<br>        query_engine=vector_query_engine,<br>        metadata=ToolMetadata(<br>            name=&quot;qlora_paper&quot;,<br>            description=&quot;Efficient Finetuning of Quantized LLMs&quot;,<br>        ),<br>    ),<br>]<br><br>query_engine = SubQuestionQueryEngine.from_defaults(<br>    query_engine_tools=query_engine_tools,<br>    service_context=service_context,<br>    use_async=True,<br>)<br><br>response = query_engine.query(&quot;Describe the trade-offs between using BFloat16 as the computation data type and other possible choices. When would you choose one over the other?&quot;)</pre><p>Below are the Sub-question generated:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UxD7a50UnM5boKgOmFBngQ.png" /></figure><p>Response:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/915/1*EIp2SDeMyt7jfZq6Wv0Xag.png" /><figcaption>Sub-Question Query Engine Response</figcaption></figure><p><a href="https://github.com/akashmathur-2212/LLMs-playground/blob/main/LlamaIndex-applications/Advanced-RAG/advanced_query_transformations/Advanced_Query_Transformations.ipynb">Link to the GitHub code repository</a></p><h3>3. Router Query Engine</h3><p>Now, we will define a router query engine that selects one out of several candidate query engines to execute a query.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Ado23RkTTpNFRO0b" /><figcaption>Router Query Engine</figcaption></figure><p>A Router Query Engine serves as a powerful decision-making module that plays a crucial role in selecting the most appropriate choices based on user queries and metadata-defined options. These routers are versatile modules that can operate independently as “selector modules” or can be utilized as query engines or retrievers on top of other query engines or retrievers.</p><p>Routers excel in various use cases, including selecting the appropriate data source from a diverse range of options and deciding whether to perform summarization or semantic search based on the user query. They can also handle more complex tasks like trying out multiple choices simultaneously and combining the results using multi-routing capabilities.</p><p>We also define a “selector”. Users can easily employ routers as query engines or retrievers, with the router taking on the responsibility of selecting query engines or retrievers to route user queries effectively.</p><p>There are several selectors available, each with some distinct attributes.</p><ol><li>The LLM selectors use the LLM to output a JSON that is parsed and the corresponding indexes are queried.</li><li>The Pydantic selectors (currently only supported by gpt-4 and gpt-3.5 (the default)) use the OpenAI Function Call API to produce pydantic selection objects, rather than parsing raw JSON.</li><li>For each type of selector, there is also the option to select one index to route to, or multiple.</li><li>Then, define the RouterQueryEngine with a desired selector module. Here, we use the LLMSingleSelector, which uses LLM to choose an underlying query engine to route the query to.</li></ol><p>Let’s jump into the code.</p><p>We will define a custom router query engine that selects one out of several candidate query engines to execute a query.</p><pre>from llama_index import VectorStoreIndex, SummaryIndex, SimpleKeywordTableIndex<br><br>service_context = ServiceContext.from_defaults(llm=llm,<br>                                               embed_model=embed_model<br>)<br><br>### Define all the different indexes over same data ###<br><br># vector index<br>vector_index = VectorStoreIndex(<br>    nodes, service_context=service_context<br>)<br><br># summary index<br>summary_index = SummaryIndex(<br>    nodes, service_context=service_context<br>    )<br><br># keyword index<br>keyword_index = SimpleKeywordTableIndex(nodes, service_context=service_context)</pre><p>Next, we define Query Engines for each Index. We then wrap these with QueryEngineTool.</p><pre>summary_query_engine = summary_index.as_query_engine(<br>    response_mode=&quot;tree_summarize&quot;,<br>    service_context=service_context<br>)<br><br>vector_query_engine = vector_index.as_query_engine(service_context=service_context)<br><br>keyword_query_engine = keyword_index.as_query_engine(service_context=service_context)</pre><pre>from llama_index.tools.query_engine import QueryEngineTool<br><br>summary_tool = QueryEngineTool.from_defaults(<br>    query_engine=summary_query_engine,<br>    description=(<br>        &quot;Useful for summarization questions related to Efficient Finetuning QLORA reserach paper&quot;<br>    ),<br>)<br><br>vector_tool = QueryEngineTool.from_defaults(<br>    query_engine=vector_query_engine,<br>    description=(<br>        &quot;Useful for retrieving specific context from QLORA reserach paper related to Efficient Finetuning &quot;<br>    ),<br>)<br><br>keyword_tool = QueryEngineTool.from_defaults(<br>    query_engine=keyword_query_engine,<br>    description=(<br>        &quot;Useful for retrieving specific context from QLORA reserach paper related to Efficient Finetuning &quot;<br>        &quot;using entities mentioned in query&quot;<br>    ),<br>)</pre><p>Then, we will use LLM selectorswhich can use OpenAI or any other LLM to parse generated JSON under the hood to select a sub-index for routing.</p><p><strong>LLMSingleSelector</strong></p><pre># LLMSingleSelector<br><br>from llama_index.query_engine.router_query_engine import RouterQueryEngine<br>from llama_index.selectors.llm_selectors import LLMSingleSelector, LLMMultiSelector<br><br>router_query_engine  = RouterQueryEngine(<br>    selector=LLMSingleSelector.from_defaults(service_context=service_context),<br>    query_engine_tools=[<br>        summary_tool,<br>        vector_tool,<br>        keyword_tool,<br>    ],<br>    service_context=service_context,<br>)<br><br>response = router_query_engine.query(&quot;What is Double Quantization?&quot;)</pre><p>Response:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/915/1*QZ6BYV_sradF7cklR91i9Q.png" /><figcaption>Router Query Engine Response</figcaption></figure><p><strong>LLMMultiSelector</strong></p><p>If we want to route our query to multiple indexes, we can use a multi selector. The multi selector sends to query to multiple sub-indexes, and then aggregates all responses using a summary index to form a complete answer.</p><pre>router_query_engine  = RouterQueryEngine(<br>    selector=LLMMultiSelector.from_defaults(service_context=service_context),<br>    query_engine_tools=[<br>        summary_tool,<br>        vector_tool,<br>        keyword_tool,<br>    ],<br>    service_context=service_context,<br>)<br><br>print(str(response))</pre><p><a href="https://github.com/akashmathur-2212/LLMs-playground/blob/main/LlamaIndex-applications/Advanced-RAG/advanced_query_transformations/Advanced_Query_Transformations.ipynb">Link to the GitHub code repository</a></p><h3>4. Single-Step Query Decomposition</h3><p>Recent studies have demonstrated that LLMs tend to perform better when they break down complex questions into smaller, more manageable steps. In cases where a query is complex, different parts of the knowledge base may be relevant to answer distinct “subqueries” within the overall question. The single-step query transformation acknowledges this and aims to address each subquery independently.</p><p>The single-step query decomposition feature is designed to transform a complicated question into a simpler one, specifically tailored to extract relevant information from the data collection. By breaking down the original question into smaller, more focused subqueries, the model can provide sub-answers that collectively contribute to addressing the complexity of the original question.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WOxTo_06eohiEWHN.png" /><figcaption>Image from LlamaIndex documentation</figcaption></figure><h3>5. Multi-Step Query Decomposition</h3><p>The multi-step query transformation represents an innovative approach known as the self-ask method. This method is rooted in the concept of a language model asking and answering follow-up questions to itself before providing an answer to the original query. The goal is to empower the model to seamlessly combine the information it has learned independently.</p><p>The model connects scattered facts, synthesizes insights, and uncovers relationships that might have remained obscured in a single-step approach.</p><p>Hence, Multi-Step Query Transformations overcome a common limitation of LLMs: the difficulty in combining separate facts to draw new conclusions. By iteratively exploring knowledge, the model uncovers connections that might otherwise remain hidden.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MfVz4-MJPsmKGQis.png" /><figcaption>Image from LlamaIndex documentation</figcaption></figure><p>Let’s jump into the code.</p><pre>from llama_index.indices.query.query_transform.base import StepDecomposeQueryTransform<br>from llama_index.query_engine.multistep_query_engine import MultiStepQueryEngine<br><br># set Logging to DEBUG for more detailed outputs<br>from llama_index.query_engine.multistep_query_engine import (<br>    MultiStepQueryEngine,<br>)<br><br>step_decompose_transform = StepDecomposeQueryTransform(llm=llm, verbose=True)<br>query_engine = vector_index.as_query_engine(service_context=service_context)<br><br>query_engine = MultiStepQueryEngine(<br>    query_engine=query_engine,<br>    query_transform=step_decompose_transform<br>)</pre><p><strong>Note</strong>: While running MultiStepQueryEngine, I was getting the ValueError — Could not load OpenAI model.</p><p>It looks like MultiStepQueryEngine supports <strong>only</strong> OpenAI GPT-4 and GPT-3.5 model as of now. I will keep on looking into this space and will update the code accordingly. I will also keep it as an open question for everyone and please share your response in the comment section.</p><h3>When one is Right for me?</h3><p>Both Sub-Question Query Engine and Single/Multi-step query decomposition in RAG tackle complex queries, but they approach the problem from different angles:</p><p><strong>Sub-Question Query Engine</strong></p><p>It focuses on the <strong><em>divide-and-conquer</em></strong> approach. It decomposes a complex query into a series of smaller, focused sub-questions. Each sub-question is sent to a dedicated Sub-Question Query Engine that retrieves relevant information from its specific data source.</p><p>Hence, it ensures each sub-question gets the appropriate data source, leading to more precise results. It provides comprehensive answers by aggregating insights from various sub-questions to provide a holistic response.</p><p><strong>Single/Multi-step query decomposition</strong></p><p>It focuses on the <strong><em>sequential refinement</em></strong> of the query. It breaks down the complex query into intermediate steps, progressively enriching the search with the retrieved information. Each step searches for relevant documents based on the current query state, updating the query with extracted knowledge.</p><p>Hence, it avoids redundant retrieval by refining the query with each step.</p><h3>Conclusion</h3><p>This exploration of advanced query augmentation is not an end, but an exciting beginning. While the approaches we covered in this article offer transformative powers, the boundaries of information retrieval stretch ever further.</p><p>Future research may delve into hybrid approaches, combining these techniques for even greater synergy. We may witness the rise of personalized transformations, adapting to individual user needs and preferences. Ultimately, the journey towards bridging the retrieval gap is a continuous one, fuelled by innovation and driven by the desire to connect users with the most accurate and insightful information, wherever it may reside.</p><p>Refer to the complete code on Github:</p><p><a href="https://github.com/akashmathur-2212/LLMs-playground/blob/main/LlamaIndex-applications/Advanced-RAG/advanced_query_transformations/Advanced_Query_Transformations.ipynb">LLMs-playground/LlamaIndex-applications/Advanced-RAG/advanced_query_transformations/Advanced_Query_Transformations.ipynb at main · akashmathur-2212/LLMs-playground</a></p><p>To refer to other advanced RAG methods, refer to this repo:</p><p><a href="https://github.com/akashmathur-2212/LLMs-playground/tree/main/LlamaIndex-applications/Advanced-RAG">LLMs-playground/LlamaIndex-applications/Advanced-RAG at main · akashmathur-2212/LLMs-playground</a></p><p><em>Thank you for reading this article, I hope it added some pieces to your knowledge stack! Before you go, if you enjoyed reading this article:</em></p><p>👉 <em>Be sure to </em><strong><em>clap</em></strong><em> and </em><strong><em>follow</em></strong><em> me, and let me know if any feedback.</em></p><p>👉<em>I built versatile Generative AI applications using the Large Language Model (LLM), covered advanced RAG concepts, and serverless AWS architectures for Big Data processing. You’re welcome to take a look at the repo and star⭐it.</em></p><ul><li><em>⭐ </em><strong><em>Generative AI</em></strong><em> applications </em><a href="https://github.com/akashmathur-2212/LLMs-playground"><em>Repository</em></a></li><li><em>⭐ </em><strong><em>Advanced RAG</em></strong><em> applications </em><a href="https://github.com/akashmathur-2212/LLMs-playground/tree/main/LlamaIndex-applications/Advanced-RAG"><em>Repository</em></a></li><li><em>⭐ Serverless </em><strong><em>AWS</em></strong><em> architectures </em><a href="https://github.com/akashmathur-2212/aws-serverless-workflows"><em>Repository</em></a></li></ul><p>👉<em>Follow me: </em><a href="https://www.linkedin.com/in/akashmathur22/"><strong><em>LinkedIn</em></strong></a> <strong><em>| </em></strong><a href="https://github.com/akashmathur-2212"><strong><em>GitHub</em></strong></a> <strong><em>| </em></strong><a href="https://akash-mathur.medium.com/"><strong><em>Medium</em></strong></a><strong><em> | </em></strong><a href="https://akashmathur-2212.github.io/"><strong><em>Portfolio</em></strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d362fed7ecc3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Advanced RAG: Enhancing Retrieval Efficiency through Evaluating Reranker Models using LlamaIndex]]></title>
            <link>https://akash-mathur.medium.com/advanced-rag-enhancing-retrieval-efficiency-through-evaluating-reranker-models-using-llamaindex-3f104f24607e?source=rss-eb7bcac979b6------2</link>
            <guid isPermaLink="false">https://medium.com/p/3f104f24607e</guid>
            <category><![CDATA[retrieval-augmented]]></category>
            <category><![CDATA[reranking]]></category>
            <category><![CDATA[retrieval]]></category>
            <category><![CDATA[cohere]]></category>
            <category><![CDATA[open-source-llm]]></category>
            <dc:creator><![CDATA[Akash Mathur]]></dc:creator>
            <pubDate>Thu, 28 Dec 2023 14:00:13 GMT</pubDate>
            <atom:updated>2024-02-02T07:40:54.529Z</atom:updated>
            <content:encoded><![CDATA[<h3>Advanced RAG: Enhancing Retrieval Efficiency through Rerankers using LlamaIndex🦙</h3><h4>Using Open Source LLM <a href="https://huggingface.co/HuggingFaceH4/zephyr-7b-alpha">Zephyr-7b-alpha</a> and Cohere and BGE Embeddings</h4><h3>Welcome to the Advanced RAG 📚Learning Series!</h3><p>Dive deeper into the fascinating world of Retrieval-Augmented Generation with this comprehensive series of articles. This series delves into cutting-edge techniques and strategies to elevate your understanding and mastery of RAG applications. Explore the following articles to enhance your skills and <strong>Stay tuned </strong>🔔 for more articles in this series as we continue to delve deeper into the world of Advanced RAG and unlock its boundless potential.</p><p>Don’t miss out on any discoveries! Bookmark🏷️ this article and check back often for the latest installments in this exciting learning series.</p><p>Topics covered so far:</p><ol><li><a href="https://akash-mathur.medium.com/advanced-rag-optimizing-retrieval-with-additional-context-metadata-using-llamaindex-aeaa32d7aa2f">Optimizing Retrieval with Additional Context &amp; MetaData using LlamaIndex</a></li><li>Enhancing Retrieval Efficiency through Rerankers using LlamaIndex (you are here!)</li><li><a href="https://akash-mathur.medium.com/advanced-rag-query-augmentation-for-next-level-search-using-llamaindex-d362fed7ecc3#46d4">Query Augmentation for Next-Level Search using LlamaIndex</a></li><li><a href="https://akash-mathur.medium.com/data-management-in-llamaindex-smart-tracking-and-debugging-of-document-changes-7b81c304382b">Smart Tracking and Debugging of Document Changes using LlamaIndex</a></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1020/0*yIp8YcRj7AtZjGzZ.jpg" /><figcaption>Reranking concept</figcaption></figure><p>Advanced RAG architectures need to implement dynamic retrieval. A big downside with naive top-k RAG is the fact that retrieval is static.</p><ul><li>It always retrieves a fixed number (k) regardless of the query</li><li>What if the query requires more context (e.g. summarization) or less context?</li></ul><p>Instead, you can implement dynamic retrieval in two conceptual stages:</p><ul><li><strong>Pre-retrieval:</strong> Decide which retriever is suitable for the use case</li><li><strong>Post-retrieval:</strong> Decide whether to rerank or filter results</li></ul><p>This is <strong>super</strong> important to dynamically prune irrelevant context. It allows us to set a big top-k but still increase precision.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TNcBnYGpgz6D_mX6" /><figcaption>Dynamic Retrieval for RAG</figcaption></figure><p>This blog explains one of the key concepts in building a robust recommendation system focusing on optimizing the retrieval process. Here, we will focus on <strong><em>LLM-powered retrieval and reranking</em></strong> to improve how we find documents.</p><p>When we talk about LLM-powered retrieval, we mean a way to find documents in a smarter way than the usual method using embeddings.</p><blockquote>It helps to create enhancements to document retrieval beyond naive top-k embedding-based lookup and returns more relevant documents than embedding-based retrieval.</blockquote><p>But, it comes with a tradeoff of much higher latency and cost. Hence, to balance things out: we start by quickly finding documents using the usual embedding-based retrieval, and then, we use the smarter LLM-powered method to make sure the best documents show up first as a <strong>reranking </strong>step. It’s like doing a fast search first and then doing a deeper, more accurate search to get the best results.</p><h3>Why Re-ranking?</h3><p>Re-ranking is a step in the retrieval process where the initially retrieved results are further sorted, refined, or reordered based on certain criteria to improve their relevance or accuracy.</p><p>Here’s an example: Imagine you’re searching for information about penguins. When you enter your search query, the system quickly brings up several articles about different types of penguins. These results are the first pass, retrieved based on general relevance.</p><p>Now, let’s say the first few articles are about “Penguins in Antarctica,” but what you really wanted was information about “Penguins in Zoo Habitats.” Re-ranking comes in at this stage. The system can use additional methods, perhaps considering user behavior, specific keywords, or more sophisticated algorithms, to reorder these search results. It might move the articles about zoo habitats higher in the list, pushing down those about Antarctica. This step ensures that the most relevant or helpful information appears at the top, making it easier for you to find what you’re looking for.</p><p>In essence, re-ranking fine-tunes the initially retrieved results, providing a more tailored and relevant set of documents or information based on specific criteria or user preferences.</p><h4>What’s wrong with embedding-based retrieval?</h4><p>There are numerous advantages to using embedding-based retrieval:</p><ol><li>It performs rapid computations of dot products and doesn’t necessitate any model calls during query time.</li><li>Even though not flawless, embeddings can reasonably encode the document and query semantics. There’s a subset of queries where embedding-based retrieval yields highly pertinent outcomes.</li></ol><p>However, despite these merits, embedding-based retrieval can sometimes lack precision and return irrelevant context related to the query. This, in turn, diminishes the overall quality of the RAG system, regardless of the LLM’s quality.</p><p>This challenge isn’t new and has been tackled in recommendation systems through a <strong>two</strong>-<strong>stage </strong>process. The initial stage utilizes <strong>embedding-based</strong> retrieval with a high top-k value, emphasizing recall over precision.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6vUK0hQjaCGdZYgF.png" /><figcaption>Second Stage — <strong>Reranking</strong></figcaption></figure><p>Subsequently, the second stage employs a slightly more computationally intensive process, focusing on higher precision and lower recall to <strong>rerank</strong> the initially retrieved candidates.</p><h3>Implementation(Open Source LLM and Embedding)</h3><p>Essentially, this method employs the LLM to determine the relevance of specific documents or nodes to a provided query. The input prompt includes a group of potential documents, and the LLM’s role involves both identifying the pertinent set of documents and assessing their relevance using an internal metric.</p><p>We will use Open Source LLM <a href="https://huggingface.co/HuggingFaceH4/zephyr-7b-alpha">zephyr-7b-alpha</a> and embedding <a href="https://huggingface.co/hkunlp/instructor-large">hkunlp/instructor-large</a></p><p>Let’s implement it step-by-step:</p><ol><li><strong>Load Data — </strong>Before your chosen LLM can act on your data you need to load it. The way LlamaIndex does this is via data connectors, also called Reader. Data connectors ingest data from different data sources and format the data into Document objects. A Document is a collection of data (currently text, and in the future, images, and audio) and metadata about that data.</li></ol><pre>PDFReader = download_loader(&quot;PDFReader&quot;)<br>loader = PDFReader()<br>docs = loader.load_data(file=Path(&quot;QLoRa.pdf&quot;))</pre><p>2. <strong>Chunking</strong>- We will create nodes by splitting the text into a chunk size of 512. A Node is the atomic unit of data in LlamaIndex and represents a “chunk” of a source Document. Nodes contain <strong>metadata</strong> and <strong>relationship</strong> information with other nodes.</p><pre>node_parser = SimpleNodeParser.from_defaults(chunk_size=512)<br>nodes = node_parser.get_nodes_from_documents(docs)</pre><p><strong>3. Open Source</strong> <strong>LLM and Embedding — </strong>We will use Open Source LLM <a href="https://huggingface.co/HuggingFaceH4/zephyr-7b-alpha">zephyr-7b-alpha</a> and will quantify it for memory and computation. This should run on a T4 GPU in the free tier on Colab.</p><p>In this example, we will use <a href="https://huggingface.co/hkunlp/instructor-large">hkunlp/instructor-large</a>. This is an instruction-finetuned text embedding model that can generate text embeddings tailored to any task (e.g., classification, retrieval, clustering, text evaluation, etc.) <em>by simply providing the task instruction, without any finetuning</em>. Instructor👨‍ ranks at #14 on the <a href="https://huggingface.co/spaces/mteb/leaderboard">MTEB leaderboard</a>!</p><pre>from google.colab import userdata<br><br># huggingface and cohere api token<br>hf_token = userdata.get(&#39;hf_token&#39;)<br><br>quantization_config = BitsAndBytesConfig(<br>    load_in_4bit=True,<br>    bnb_4bit_compute_dtype=torch.float16,<br>    bnb_4bit_quant_type=&quot;nf4&quot;,<br>    bnb_4bit_use_double_quant=True,<br>)<br><br>def messages_to_prompt(messages):<br>  prompt = &quot;&quot;<br>  for message in messages:<br>    if message.role == &#39;system&#39;:<br>      prompt += f&quot;&lt;|system|&gt;\n{message.content}&lt;/s&gt;\n&quot;<br>    elif message.role == &#39;user&#39;:<br>      prompt += f&quot;&lt;|user|&gt;\n{message.content}&lt;/s&gt;\n&quot;<br>    elif message.role == &#39;assistant&#39;:<br>      prompt += f&quot;&lt;|assistant|&gt;\n{message.content}&lt;/s&gt;\n&quot;<br><br>  # ensure we start with a system prompt, insert blank if needed<br>  if not prompt.startswith(&quot;&lt;|system|&gt;\n&quot;):<br>    prompt = &quot;&lt;|system|&gt;\n&lt;/s&gt;\n&quot; + prompt<br><br>  # add final assistant prompt<br>  prompt = prompt + &quot;&lt;|assistant|&gt;\n&quot;<br><br>  return prompt<br><br># LLM<br>llm = HuggingFaceLLM(<br>    model_name=&quot;HuggingFaceH4/zephyr-7b-alpha&quot;,<br>    tokenizer_name=&quot;HuggingFaceH4/zephyr-7b-alpha&quot;,<br>    query_wrapper_prompt=PromptTemplate(&quot;&lt;|system|&gt;\n&lt;/s&gt;\n&lt;|user|&gt;\n{query_str}&lt;/s&gt;\n&lt;|assistant|&gt;\n&quot;),<br>    context_window=3900,<br>    max_new_tokens=256,<br>    model_kwargs={&quot;quantization_config&quot;: quantization_config},<br>    # tokenizer_kwargs={},<br>    generate_kwargs={&quot;temperature&quot;: 0.7, &quot;top_k&quot;: 50, &quot;top_p&quot;: 0.95},<br>    messages_to_prompt=messages_to_prompt,<br>    device_map=&quot;auto&quot;,<br>)<br><br># Embedding<br>embed_model = HuggingFaceInstructEmbeddings(<br>    model_name=&quot;hkunlp/instructor-large&quot;, model_kwargs={&quot;device&quot;: DEVICE}<br>)</pre><p>4. <strong>Configure Index and Retriever — </strong>First, we will be setting up the ServiceContextobject and will be using it to construct an index and query.</p><blockquote>An <strong>Index </strong>is a data structure composed of Document objects, designed to enable querying by an LLM. This means creating a <strong>data structure</strong> that allows for querying the data. For LLMs this nearly always means creating vector embeddings, and other metadata strategies to find contextually relevant data.</blockquote><p><strong>Vector store indexes </strong>are the most common and simple to use. Allowing answering a query over a large corpus of data. Querying a vector store index involves fetching the `top-k` most similar Nodes, and passing those into the `Response Synthesis` module.</p><p><strong>Vector Store Index embeds your documents</strong>. It takes your Documents and splits them up into Nodes. It then creates vector embeddings of the text of every node, ready to be queried by an LLM.</p><p>When you want to search your embeddings, your query is itself turned into a vector embedding, and then a mathematical operation is carried out by VectorStoreIndex to rank all the embeddings by how semantically similar they are to your query.</p><p><strong>Top K Retrieval </strong>Once the ranking is complete, VectorStoreIndex returns the most-similar embeddings as their corresponding chunks of text. The number of embeddings it returns is known as k, so the parameter controlling how many embeddings to return is known as `top_k`. This whole type of search is often referred to as “top-k semantic retrieval” for this reason.</p><pre># ServiceContext<br>service_context = ServiceContext.from_defaults(llm=llm,<br>                                               embed_model=embed_model<br>                                               )<br><br># index<br>vector_index = VectorStoreIndex(<br>    nodes, service_context=service_context<br>)<br><br># configure retriever<br>retriever = VectorIndexRetriever(<br>    index=vector_index,<br>    similarity_top_k=10,<br>    service_context=service_context)</pre><p>You can follow along with this <strong>notebook</strong>:</p><p><a href="https://github.com/akashmathur-2212/LLMs-playground/tree/main/LlamaIndex-applications/Advanced-RAG/reranker_models_evaluation">LLMs-playground/LlamaIndex-applications/Advanced-RAG/reranker_models_evaluation at main · akashmathur-2212/LLMs-playground</a></p><p>5. <strong>Initialize Re-rankers</strong></p><p>We will compare the performance of our Retrieval with 3 rerankers:</p><ul><li><a href="https://txt.cohere.com/rerank/">CohereAI</a></li><li><a href="https://huggingface.co/BAAI/bge-reranker-base">bge-reranker-base</a></li><li><a href="https://huggingface.co/BAAI/bge-reranker-large">bge-reranker-large</a></li></ul><pre># Define all embeddings and rerankers<br>RERANKERS = {<br>    &quot;WithoutReranker&quot;: &quot;None&quot;,<br>    &quot;CohereRerank&quot;: CohereRerank(api_key=cohere_api_key, top_n=5),<br>    &quot;bge-reranker-base&quot;: SentenceTransformerRerank(model=&quot;BAAI/bge-reranker-base&quot;, top_n=5),<br>    &quot;bge-reranker-large&quot;: SentenceTransformerRerank(model=&quot;BAAI/bge-reranker-large&quot;, top_n=5)<br>}</pre><p>6. <strong>Retrieval Comparisons</strong></p><p>A retriever defines how to efficiently retrieve relevant context from an index when given a query. Your retrieval strategy is key to the relevancy of the data retrieved and the efficiency with which it’s done. The most common type of retrieval is “top-k” semantic retrieval, but there are many other retrieval strategies.</p><p><a href="https://docs.llamaindex.ai/en/stable/module_guides/querying/node_postprocessors/root.html"><strong>Node Postprocessors</strong></a>: A node postprocessor takes in a set of retrieved nodes and applies transformations, filtering, or re-ranking logic to them. Node Post-processors are most commonly applied within a query engine after the node retrieval step and before the response.</p><p>Let’s create some helper functions to perform our task:</p><pre># helper functions<br><br>def get_retrieved_nodes(<br>    query_str, reranker<br>):<br>    query_bundle = QueryBundle(query_str)<br><br>    retrieved_nodes = retriever.retrieve(query_bundle)<br><br>    if reranker != &quot;None&quot;:<br>      retrieved_nodes = reranker.postprocess_nodes(retrieved_nodes, query_bundle)<br>    else:<br>        retrieved_nodes<br><br>    return retrieved_nodes<br><br><br>def pretty_print(df):<br>    return display(HTML(df.to_html().replace(&quot;\\n&quot;, &quot;&lt;br&gt;&quot;)))<br><br><br>def visualize_retrieved_nodes(nodes) -&gt; None:<br>    result_dicts = []<br>    for node in nodes:<br>        node = deepcopy(node)<br>        node.node.metadata = None<br>        node_text = node.node.get_text()<br>        node_text = node_text.replace(&quot;\n&quot;, &quot; &quot;)<br><br>        result_dict = {&quot;Score&quot;: node.score, &quot;Text&quot;: node_text}<br>        result_dicts.append(result_dict)<br><br>    pretty_print(pd.DataFrame(result_dicts))</pre><p>Let’s visualize the result for our query:</p><pre>query_str = &quot;What are the top features of QLoRA?&quot;<br><br># Loop over rerankers<br>for rerank_name, reranker in RERANKERS.items():<br>    print(f&quot;Running Evaluation for Reranker: {rerank_name}&quot;)<br><br>    query_bundle = QueryBundle(query_str)<br><br>    retrieved_nodes = retriever.retrieve(query_bundle)<br><br>    if reranker != &quot;None&quot;:<br>      retrieved_nodes = reranker.postprocess_nodes(retrieved_nodes, query_bundle)<br>    else:<br>        retrieved_nodes<br><br>    print(f&quot;Visualize Retrieved Nodes for Reranker: {rerank_name}&quot;)<br>    visualize_retrieved_nodes(retrieved_nodes)</pre><ul><li><strong>No reranker —</strong> The top node has a similarity score of <strong>0.87. </strong>It’s our baseline score.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/887/1*XkZXB7e-ofrIoaWrTownUw.png" /></figure><ul><li><strong>CohereRerank — </strong>The top node has a similarity score of <strong>0.988. </strong>See, how using the Cohere reranker has improved the quality of our retrieved nodes.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/889/1*iyVHaiykODn1he9X35zpcQ.png" /></figure><ul><li><strong>bge-reranker-base — </strong>The top node has a similarity score of <strong>0.72. </strong>It didn&#39;t work up to the mark. Maybe, we should try it with a different embedding model and compare results.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/892/1*aDdw0orn19BpfSijll9g8w.png" /></figure><h3>Evaluation</h3><p>Now, we will use <strong>RetrieverEvaluator</strong> to evaluate the quality of our Retriever.</p><p>We define various evaluation metrics, such as <strong>hit-rate</strong> and <strong>MRR</strong>, which assess the quality of retrieved results against the ground-truth context for each specific question. To simplify the creation of the evaluation dataset initially, we can depend on synthetic data generation methods.</p><ul><li><strong>MRR </strong>stands for <a href="https://en.wikipedia.org/wiki/Mean_reciprocal_rank">Mean Reciprocal Rank</a> and is a rank-aware relevance score in the first 10 results of a ranking. The reciprocal rank of a query response is the multiplicative inverse of the rank of the first correct answer: 1 for first place, 1/2 for second place, and 1/n for the nth place. The mean reciprocal rank is the average of the reciprocal ranks of results for a sample of queries.</li><li><strong>Hit Rate </strong>measures the proportion or percentage of queries for which the retrieved results contain at least one relevant item from the ground truth. For instance, imagine a search engine returning a list of documents in response to a user’s query. The ground truth here refers to the known relevant documents for that query, typically determined by human judgment or labeled data. The hit rate calculates how often the search results include at least one of these relevant documents.</li></ul><h4>Build an Evaluation dataset of (query, context) pairs</h4><p>We can manually curate a retrieval evaluation dataset of questions + node id’s. Llamaindex offers synthetic dataset generation over an existing text corpus with our generate_question_context_pairs function. This uses the LLM to auto-generate questions from each context chunk. Refer <a href="https://docs.llamaindex.ai/en/stable/module_guides/evaluating/usage_pattern_retrieval.html">here</a>.</p><p>Here we build a simple evaluation dataset over the existing text corpus. We will use Zephr-7B LLM to generate Question-Context Pairs.</p><p>The returned result is a EmbeddingQAFinetuneDataset object (containing queries, relevant_docs, and corpus).</p><pre># Prompt to generate questions<br>qa_generate_prompt_tmpl = &quot;&quot;&quot;\<br>Context information is below.<br><br>---------------------<br>{context_str}<br>---------------------<br><br>Given the context information and not prior knowledge.<br>generate only questions based on the below query.<br><br>You are a Professor. Your task is to setup \<br>{num_questions_per_chunk} questions for an upcoming \<br>quiz/examination. The questions should be diverse in nature \<br>across the document. The questions should not contain options, not start with Q1/ Q2. \<br>Restrict the questions to the context information provided.\<br>&quot;&quot;&quot;<br><br># Evaluator<br><br>qa_dataset = generate_question_context_pairs(<br>    nodes, llm=llm, num_questions_per_chunk=2, qa_generate_prompt_tmpl=qa_generate_prompt_tmpl<br>)</pre><pre># helper function for displaying results<br>def display_results(reranker_name, eval_results):<br>    &quot;&quot;&quot;Display results from evaluate.&quot;&quot;&quot;<br><br>    metric_dicts = []<br>    for eval_result in eval_results:<br>        metric_dict = eval_result.metric_vals_dict<br>        metric_dicts.append(metric_dict)<br><br>    full_df = pd.DataFrame(metric_dicts)<br><br>    hit_rate = full_df[&quot;hit_rate&quot;].mean()<br>    mrr = full_df[&quot;mrr&quot;].mean()<br><br>    metric_df = pd.DataFrame({&quot;Reranker&quot;: [reranker_name], &quot;hit_rate&quot;: [hit_rate], &quot;mrr&quot;: [mrr]})<br><br>    return metric_df</pre><p>Llamaindex offers a function to run a RetrieverEvaluator over a dataset in batch mode.</p><pre>query_str = &quot;What are the top features of QLoRA?&quot;<br><br>results_df = pd.DataFrame()<br># Loop over rerankers<br>for rerank_name, reranker in RERANKERS.items():<br>    print(f&quot;Running Evaluation for Reranker: {rerank_name}&quot;)<br><br>    query_bundle = QueryBundle(query_str)<br><br>    retrieved_nodes = retriever.retrieve(query_bundle)<br><br>    if reranker != &quot;None&quot;:<br>      retrieved_nodes = reranker.postprocess_nodes(retrieved_nodes, query_bundle)<br>    else:<br>        retrieved_nodes<br><br>    retriever_evaluator = RetrieverEvaluator.from_metric_names(<br>        [&quot;mrr&quot;, &quot;hit_rate&quot;], retriever=retriever<br>    )<br><br>    eval_results = await retriever_evaluator.aevaluate_dataset(qa_dataset)<br><br>    current_df = display_results(rerank_name, eval_results)<br>    results_df = pd.concat([results_df, current_df], ignore_index=True)</pre><h3>Results</h3><ul><li><strong>WithoutReranker</strong>: This provides the baseline performance for each embedding.</li><li><strong>CohereRerank</strong>: Provided the best results among all others.</li><li><strong>bge-reranker-base</strong>: Less accurate results than <strong>CohereRerank</strong>. Maybe the selected embedding model is not effective for this reranker.</li><li><strong>bge-reranker-large</strong>: Poor results than <strong>bge-reranker-base</strong>. Maybe the selected embedding model is not effective for this reranker.</li></ul><p>The results indicate the importance of rerankers in optimizing the retrieval process. Particularly, CohereRerank has showcased its ability to elevate any embedding to the next level.</p><h3>Summary</h3><p>In this blog post, we’ve demonstrated performance comparison across multiple rerankers using LlamaIndex, which has improved retrieval performance metrics.</p><p>Additionally, we can improve the retrieval process by performing experiments (not limited to) :</p><ol><li>Employ various embedding models</li><li>Employ various reranker models.</li><li>Select additional evaluation metrics like `Accuracy` and `NDCG` (Normalized Discounted Cumulative Gain) etc.</li><li>Select various other LLMs</li></ol><h3><strong>Conclusion</strong></h3><p>Selecting the appropriate embedding for the initial search holds utmost importance; even the most effective reranker can’t compensate for poor basic search outcomes (<strong>bge-rerankers </strong>in our case<strong>).</strong></p><p>Maximizing retriever performance depends on discovering the optimal blend of embeddings and rerankers. This continues to be an active research area to identify the most effective combinations.</p><p>Refer to the complete code on Github:</p><p><a href="https://github.com/akashmathur-2212/LLMs-playground/tree/main/LlamaIndex-applications/Advanced-RAG/reranker_models_evaluation">LLMs-playground/LlamaIndex-applications/Advanced-RAG/reranker_models_evaluation at main · akashmathur-2212/LLMs-playground</a></p><p>To refer to other advanced RAG methods, refer to this repo:</p><p><a href="https://github.com/akashmathur-2212/LLMs-playground/tree/main/LlamaIndex-applications/Advanced-RAG">LLMs-playground/LlamaIndex-applications/Advanced-RAG at main · akashmathur-2212/LLMs-playground</a></p><p><em>Thank you for reading this article, I hope it added some pieces to your knowledge stack! Before you go, if you enjoyed reading this article:</em></p><p>👉 <em>Be sure to </em><strong><em>clap</em></strong><em> and </em><strong><em>follow</em></strong><em> me, and let me know if any feedback.</em></p><p>👉<em>I built versatile Generative AI applications using the Large Language Model (LLM), covered advanced RAG concepts, and serverless AWS architectures for Big Data processing. You’re welcome to take a look at the repo and star⭐it.</em></p><ul><li><em>⭐ </em><strong><em>Generative AI</em></strong><em> applications </em><a href="https://github.com/akashmathur-2212/LLMs-playground"><em>Repository</em></a></li><li><em>⭐ </em><strong><em>Advanced RAG</em></strong><em> applications </em><a href="https://github.com/akashmathur-2212/LLMs-playground/tree/main/LlamaIndex-applications/Advanced-RAG"><em>Repository</em></a></li><li><em>⭐ Serverless </em><strong><em>AWS</em></strong><em> architectures </em><a href="https://github.com/akashmathur-2212/aws-serverless-workflows"><em>Repository</em></a></li></ul><p>👉<em>Follow me: </em><a href="https://www.linkedin.com/in/akashmathur22/"><strong><em>LinkedIn</em></strong></a> <strong><em>| </em></strong><a href="https://github.com/akashmathur-2212"><strong><em>GitHub</em></strong></a> <strong><em>| </em></strong><a href="https://akash-mathur.medium.com/"><strong><em>Medium</em></strong></a><strong><em> | </em></strong><a href="https://akashmathur-2212.github.io/"><strong><em>Portfolio</em></strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3f104f24607e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Advanced RAG: Optimizing Retrieval with Additional Context & MetaData using LlamaIndex]]></title>
            <link>https://akash-mathur.medium.com/advanced-rag-optimizing-retrieval-with-additional-context-metadata-using-llamaindex-aeaa32d7aa2f?source=rss-eb7bcac979b6------2</link>
            <guid isPermaLink="false">https://medium.com/p/aeaa32d7aa2f</guid>
            <category><![CDATA[llamaindex]]></category>
            <category><![CDATA[zephyr]]></category>
            <category><![CDATA[open-source-llm]]></category>
            <category><![CDATA[retrieval-augmented]]></category>
            <category><![CDATA[metadata]]></category>
            <dc:creator><![CDATA[Akash Mathur]]></dc:creator>
            <pubDate>Sat, 23 Dec 2023 11:06:02 GMT</pubDate>
            <atom:updated>2024-02-02T07:41:33.825Z</atom:updated>
            <content:encoded><![CDATA[<h4>Using Open Source LLM <a href="https://huggingface.co/HuggingFaceH4/zephyr-7b-alpha">Zephyr-7b-alpha</a> and Instruct Embeddings <a href="https://huggingface.co/hkunlp/instructor-large">hkunlp/instructor-large</a></h4><h3>Welcome to the Advanced RAG 📚Learning Series!</h3><p>Dive deeper into the fascinating world of Retrieval-Augmented Generation with this comprehensive series of articles. This series delves into cutting-edge techniques and strategies to elevate your understanding and mastery of RAG applications. Explore the following articles to enhance your skills and <strong>Stay tuned </strong>🔔 for more articles in this series as we continue to delve deeper into the world of Advanced RAG and unlock its boundless potential.</p><p>Don’t miss out on any discoveries! Bookmark🏷️ this article and check back often for the latest installments in this exciting learning series.</p><p>Topics covered so far:</p><ol><li>Optimizing Retrieval with Additional Context &amp; MetaData using LlamaIndex (you are here!)</li><li><a href="https://akash-mathur.medium.com/advanced-rag-enhancing-retrieval-efficiency-through-evaluating-reranker-models-using-llamaindex-3f104f24607e">Enhancing Retrieval Efficiency through Rerankers using LlamaIndex</a></li><li><a href="https://akash-mathur.medium.com/advanced-rag-query-augmentation-for-next-level-search-using-llamaindex-d362fed7ecc3#46d4">Query Augmentation for Next-Level Search using LlamaIndex</a></li><li><a href="https://akash-mathur.medium.com/data-management-in-llamaindex-smart-tracking-and-debugging-of-document-changes-7b81c304382b">Smart Tracking and Debugging of Document Changes using LlamaIndex</a></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4VBFahYzpKQuNxu9.png" /><figcaption>An image by the author</figcaption></figure><p><strong>RAG (Retrieval Augmented Generation)</strong> is a technique for augmenting LLM knowledge with additional, often private or real-time, data.</p><p>LLMs are trained on enormous bodies of data but they aren’t trained on <strong>your</strong> data. RAG solves this problem by adding your data to the data LLMs already have access to.</p><p>If you want to build AI applications that can reason about private data or data introduced after a model’s cutoff date, you need to augment the knowledge of the model with the specific information it needs. The process of bringing the appropriate information and inserting it into the model prompt is known as Retrieval Augmented Generation (RAG).</p><p>In RAG, your data is loaded and prepared for queries or “indexed”. User queries act on the index, which filters your data down to the most relevant context. This context and your query then go to the LLM along with a prompt, and the LLM responds.</p><h4><strong>Stages within RAG</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9NR0yxRS6rg50Ihq4a97LA.png" /><figcaption>Stages within RAG</figcaption></figure><p>There are five key stages within RAG, which in turn will be a part of any larger application you build. These are:</p><ol><li><strong>Loading</strong>: this refers to getting your data from where it lives — whether it’s text files, PDFs, another website, a database, or an API — into your pipeline. <a href="https://llamahub.ai/">LlamaHub</a> provides hundreds of connectors to choose from.</li></ol><p><strong>2. Spliting</strong>: If we were to use these large sections, then we’d be inserting a lot of noisy/unwanted context and because all LLMs have a maximum context length, we wouldn’t be able to fit too much other relevant context. So instead, we’re going to split the text within each section into smaller chunks. Intuitively, smaller chunks will encapsulate single/few concepts and will be less noisy compared to larger chunks.</p><blockquote><strong><em>Text splitters </em></strong>break large Documents into smaller chunks. This is useful both for indexing data and for passing it in to a model, since large chunks are harder to search over and won’t in a model’s finite context window. Empirically models struggle to find the relevant context in very long prompts.</blockquote><p><strong>3. Indexing</strong>: this means creating a data structure that allows for querying the data. For LLMs, this nearly always means creating vector embeddings, numerical representations of the meaning of your data, as well as numerous other metadata strategies to make it easy to accurately find contextually relevant data.</p><p><strong>4. Storing</strong>: once your data is indexed you will almost always want to store your index, as well as other metadata, to avoid having to re-index it.</p><p><strong>5. Querying</strong>: for any given indexing strategy there are many ways you can utilize LLMs and LlamaIndex data structures to query, including sub-queries, multi-step queries, and hybrid strategies.</p><p><strong>6. Evaluation</strong>: A critical step in any pipeline is checking how effective it is relative to other strategies, or when you make changes. Evaluation provides objective measures of how accurate, faithful, and fast your responses to queries are.</p><h3>Advanced RAG</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*7PPKqk3Yu3nHEP5m.png" /><figcaption>Parent-Child Chunks Retrieval</figcaption></figure><p>Now we’ll dive into the overview of <strong>ONE </strong>of the advanced RAG techniques related to <strong>Context Enhancement </strong>i.e.<strong> <em>Parent-Child Chunks Retrieval</em></strong>.</p><blockquote><strong>The concept here is to retrieve smaller chunks for better search quality</strong>, <strong>but add up surrounding context for LLM to reason upon</strong>.</blockquote><p>Let’s understand with an example-</p><p>Imagine you have a really big book, and you want to find specific chapters or paragraphs quickly. <em>This Retriever works like a smart librarian who carefully breaks down this book into smaller, understandable chunks and remembers where they came from</em>. When you ask for one of those smaller pieces, it not only gives you that bit but also knows exactly which big section or even the whole book it belongs to. It’s like having a shortcut to the exact spot in the big book, saving time and effort while making sure you understand everything in context.</p><p>It fetches smaller chunks during retrieval first, then references the parent IDs, and returns the bigger chunks.</p><p>Documents are split into a hierarchy of chunks and then the smallest leaf chunks are sent to the index. At the retrieval time, we retrieve k leaf chunks, and if n chunks refer to the same parent chunk, we replace them with this parent chunk and send it to LLM for answer generation.</p><p>In this blog post, we will dive into the implementations of this method in LlamaIndex. It fetches smaller chunks during retrieval first, then if more than <em>n</em> chunks in top <em>k</em> retrieved chunks are linked to the same parent node (larger chunk), we replace the context fed to the LLM by this parent node — works like auto merging a few retrieved chunks into a larger parent chunk, hence the method name. Just to note — the search is performed just within the child nodes index.</p><p>Let’s deeper dive with the help of code.</p><h3>Implementation (Open Source LLM and Embedding)</h3><p>When you first perform retrieval, you may want to retrieve the reference as opposed to the raw text. You can have multiple references point to the same node.</p><p>In this blog, we will explore some different usages of node references:</p><ul><li><strong>Chunk references</strong>: Different chunk sizes refer to a bigger chunk</li><li><strong>Metadata references</strong>: Summaries + Generated Questions referring to a bigger chunk</li></ul><p>We will use Open Source LLM <a href="https://huggingface.co/HuggingFaceH4/zephyr-7b-alpha">zephyr-7b-alpha</a> and embedding <a href="https://huggingface.co/hkunlp/instructor-large">hkunlp/instructor-large</a></p><p>This <a href="https://github.com/akashmathur-2212/LLMs-playground/blob/main/LlamaIndex-applications/Advanced-RAG/parent_child_document_retriever/parent_child_document_retriever_metadata_extraction.ipynb">notebook</a> shows how you can use recursive retrieval to traverse node relationships and fetch nodes based on “references”.</p><p>Let’s implement it step-by-step:</p><p><strong>A. Load Data — </strong>Before your chosen LLM can act on your data you need to load it. The way LlamaIndex does this is via data connectors, also called Reader. Data connectors ingest data from different data sources and format the data into Document objects. A Document is a collection of data (currently text, and in the future, images, and audio) and metadata about that data.</p><pre>PDFReader = download_loader(&quot;PDFReader&quot;)<br>loader = PDFReader()<br>docs = loader.load_data(file=Path(&quot;./data/QLoRa.pdf&quot;))<br><br># combine all the text<br>doc_text = &quot;\n\n&quot;.join([d.get_content() for d in docs])<br>documents = [Document(text=doc_text)]</pre><p><strong>B. Chunking</strong>- We will useSentenceSplitterthat split the text while respecting the boundaries of sentences. This function is used to break down large bodies of text into smaller sections, ensuring that sentences aren&#39;t split in the middle and making it easier to process or analyze text data in manageable portions. We will create an initial set of nodes (chunk size 1024).</p><pre>node_parser = SentenceSplitter(chunk_size=1024)<br><br>base_nodes = node_parser.get_nodes_from_documents(documents)<br># set node ids to be a constant<br>for idx, node in enumerate(base_nodes):<br>    node.id_ = f&quot;node-{idx}&quot;<br><br># print all the node ids corrosponding to all the chunks<br>for node in base_nodes:<br>  print(node.id_)</pre><p><strong>C. Open Source</strong> <strong>LLM and Embedding — </strong>We will use Open Source LLM <a href="https://huggingface.co/HuggingFaceH4/zephyr-7b-alpha">zephyr-7b-alpha</a> and will quantify it for memory and computation. This should run on a T4 GPU in the free tier on Colab.</p><p>In this example, we will use <a href="https://huggingface.co/hkunlp/instructor-large">hkunlp/instructor-large</a>. This is an instruction-finetuned text embedding model that can generate text embeddings tailored to any task (e.g., classification, retrieval, clustering, text evaluation, etc.) <em>by simply providing the task instruction, without any finetuning</em>. Instructor👨‍ ranks at #14 on the <a href="https://huggingface.co/spaces/mteb/leaderboard">MTEB leaderboard</a>!</p><p>Now, we will be setting up the ServiceContextobject, and will be using it to construct an index and query. The input documents will be broken into nodes, and the embedding model will generate an embedding for each node. Then, at query time, the embedding model will be used again to embed the query text.</p><pre>from google.colab import userdata<br><br># huggingface api token for downloading zephyr-7b<br>hf_token = userdata.get(&#39;hf_token&#39;)<br><br>quantization_config = BitsAndBytesConfig(<br>    load_in_4bit=True,<br>    bnb_4bit_compute_dtype=torch.float16,<br>    bnb_4bit_quant_type=&quot;nf4&quot;,<br>    bnb_4bit_use_double_quant=True,<br>)<br><br><br>def messages_to_prompt(messages):<br>  prompt = &quot;&quot;<br>  for message in messages:<br>    if message.role == &#39;system&#39;:<br>      prompt += f&quot;&lt;|system|&gt;\n{message.content}&lt;/s&gt;\n&quot;<br>    elif message.role == &#39;user&#39;:<br>      prompt += f&quot;&lt;|user|&gt;\n{message.content}&lt;/s&gt;\n&quot;<br>    elif message.role == &#39;assistant&#39;:<br>      prompt += f&quot;&lt;|assistant|&gt;\n{message.content}&lt;/s&gt;\n&quot;<br><br>  # ensure we start with a system prompt, insert blank if needed<br>  if not prompt.startswith(&quot;&lt;|system|&gt;\n&quot;):<br>    prompt = &quot;&lt;|system|&gt;\n&lt;/s&gt;\n&quot; + prompt<br><br>  # add final assistant prompt<br>  prompt = prompt + &quot;&lt;|assistant|&gt;\n&quot;<br><br>  return prompt<br><br><br># LLM<br>llm = HuggingFaceLLM(<br>    model_name=&quot;HuggingFaceH4/zephyr-7b-alpha&quot;,<br>    tokenizer_name=&quot;HuggingFaceH4/zephyr-7b-alpha&quot;,<br>    query_wrapper_prompt=PromptTemplate(&quot;&lt;|system|&gt;\n&lt;/s&gt;\n&lt;|user|&gt;\n{query_str}&lt;/s&gt;\n&lt;|assistant|&gt;\n&quot;),<br>    context_window=3900,<br>    max_new_tokens=256,<br>    model_kwargs={&quot;quantization_config&quot;: quantization_config},<br>    # tokenizer_kwargs={},<br>    generate_kwargs={&quot;temperature&quot;: 0.7, &quot;top_k&quot;: 50, &quot;top_p&quot;: 0.95},<br>    messages_to_prompt=messages_to_prompt,<br>    device_map=&quot;auto&quot;,<br>)<br><br># Embedding<br>embed_model = HuggingFaceInstructEmbeddings(<br>    model_name=&quot;hkunlp/instructor-large&quot;, model_kwargs={&quot;device&quot;: DEVICE}<br>)<br><br># set your ServiceContext for all the next steps<br>service_context = ServiceContext.from_defaults(<br>    llm=llm, embed_model=embed_model<br>)</pre><h4><em>1. Baseline Retriever</em></h4><p>Let’s define a baseline retriever that simply fetches the top-k raw text nodes by embedding similarity. But, we have to index our data first:</p><p><strong>a) Indexing</strong> — With our data loaded, we now have a list of Document objects (or a list of Nodes). It’s time to build an Index over these objects so you can start querying them.</p><p>In LlamaIndex terms, an Index is a data structure composed of Document objects, designed to enable querying by an LLM.</p><pre># index<br>base_index = VectorStoreIndex(base_nodes, service_context=service_context)</pre><p><strong>b) Base Retriever- </strong>It simply fetches the top-k raw text nodes by embedding similarity.</p><pre># find top 2 nodes<br>base_retriever = base_index.as_retriever(similarity_top_k=2)<br><br>retrievals = base_retriever.retrieve(<br>    &quot;Can you tell me about the Paged Optimizers?&quot;<br>)<br><br>for n in retrievals:<br>    display_source_node(n, source_length=1500)</pre><p>See the output node-id and similarity score.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/905/1*ZBE_6aG3JYo5uGUn2uZXlA.png" /></figure><p><strong>c) Querying — </strong>Now we’ve loaded our data, and built an index, we’re ready to get to the most significant part of an LLM application: <em>querying!</em></p><p>The most important thing to know about querying is that it is just a prompt to an LLM: so it can be a question and get an answer, or a request for summarization, or a much more complex instruction.</p><p>The basis of all querying is the QueryEngine. The simplest way to get a QueryEngine is to get your index to create one for you, like this:</p><pre># create a query engine<br>query_engine_base = RetrieverQueryEngine.from_args(<br>    base_retriever, service_context=service_context<br>)<br><br># query<br>response = query_engine_base.query(<br>    &quot;Can you tell me about the Paged Optimizers?&quot;<br>)<br></pre><h4>2. Chunk References: Smaller Child Chunks Referring to Bigger Parent Chunk</h4><p>Before, we used big pieces of text, each about <strong><em>1024 </em></strong>characters long, for finding and putting together information. Now, we’ll try something different. Instead of using those big chunks, we’ll break them down into smaller pieces, like making smaller puzzles from a big one.</p><p>Here’s how we’ll do it:</p><p>Imagine each big piece of text is like a big jigsaw puzzle piece. We’ll split that big piece into smaller ones:</p><ul><li>From each big piece, we’ll create 4 slightly bigger pieces, each around 256 characters long.</li><li>Next, we’ll have 2 even bigger pieces, each about 512 characters long.</li></ul><p>And of course, we’ll still keep the original big piece of 1024 characters. So, instead of just having one big piece, we’ll have lots of smaller ones. This helps us find specific things more easily, and when we need to see the bigger picture, we can always refer back to the original big piece.</p><pre>sub_chunk_sizes = [256, 512]<br>sub_node_parsers = [SentenceSplitter(chunk_size=c) for c in sub_chunk_sizes]<br><br>all_nodes = []<br>for base_node in base_nodes:<br>    for n in sub_node_parsers:<br>        sub_nodes = n.get_nodes_from_documents([base_node])<br>        sub_inodes = [<br>            IndexNode.from_text_node(sn, base_node.node_id) for sn in sub_nodes<br>        ]<br>        all_nodes.extend(sub_inodes)<br><br>    # also add original node to node<br>    original_node = IndexNode.from_text_node(base_node, base_node.node_id)<br>    all_nodes.append(original_node)<br><br><br>all_nodes_dict = {n.node_id: n for n in all_nodes}</pre><p>Now, let&#39;s take a look at all the ALL nodes dictionary containing parent and child nodes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/897/1*ldD0-mY-kkRFXOS3B6eqNw.png" /><figcaption>ALL nodes dictionary</figcaption></figure><p>Let’s check one child node and it’s a reference to its parent node. See, this index node (child) is referenced to node-0 (parent).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/1*mldsHmmZ98oTQRQsfSBRfA.png" /></figure><p>Similarly, you will see that these many smaller chunks (IndexNode) are associated with each of the original text chunks(TextNode) for example node-0. All of the smaller chunks reference to the large chunk in the metadata with index_id pointing to the index ID of the larger chunk.</p><p><strong>a) Indexing (from these smaller chunks)</strong></p><pre>vector_index_chunk = VectorStoreIndex(<br>    all_nodes, service_context=service_context<br>)<br><br>vector_retriever_chunk = vector_index_chunk.as_retriever(similarity_top_k=2)</pre><p>When we perform retrieval, we want to retrieve the reference as opposed to the raw text. You can have multiple references point to the same</p><p><strong>b) Recursive Retriever — </strong>When we perform retrieval, we want to retrieve the reference as opposed to the raw text. You can have multiple references point to the same node.</p><pre># Recursive Retriever<br>retriever_chunk = RecursiveRetriever(<br>    &quot;vector&quot;,<br>    retriever_dict={&quot;vector&quot;: vector_retriever_chunk},<br>    node_dict=all_nodes_dict,<br>    verbose=True,<br>)<br><br># query<br>nodes = retriever_chunk.retrieve(<br>    &quot;Can you tell me about the Paged Optimizers?&quot;<br>)<br><br>for node in nodes:<br>    display_source_node(node, source_length=2000)</pre><p><strong>c) Querying — </strong>Let’s start <em>querying!</em></p><pre># Retriever Query Engine<br>query_engine_chunk = RetrieverQueryEngine.from_args(<br>    retriever_chunk, service_context=service_context<br>)<br><br># query<br>response = query_engine_chunk.query(<br>    &quot;Can you tell me about the Paged Optimizers?&quot;<br>)</pre><h3>Metadata References: Summaries and Generated Questions referring to a bigger chunk</h3><p>Now, we will add some additional context that references the source node.</p><p>This additional context includes summaries as well as generated questions. Due to the limited compute, I am only extracting questions, but you can uncomment the summarizer to extract summaries.</p><p>During query time, we retrieve smaller chunks, but we follow references to bigger chunks. This allows us to have more context for synthesis.</p><pre>import nest_asyncio<br>nest_asyncio.apply()<br><br>extractors = [<br>    # SummaryExtractor(summaries=[&quot;self&quot;], llm=llm, show_progress=True),<br>    QuestionsAnsweredExtractor(questions=1, llm=llm, show_progress=True),<br>]<br><br># run metadata extractor across base nodes, get back dictionaries<br>metadata_dicts = []<br>for extractor in extractors:<br>    metadata_dicts.extend(extractor.extract(base_nodes))</pre><p>Let’s add additional metadata to our existing nodes:</p><pre># all nodes consists of source nodes, along with metadata<br>import copy<br><br>all_nodes = copy.deepcopy(base_nodes)<br>for idx, d in enumerate(metadata_dicts):<br><br>    # QnA node<br>    inode_q = IndexNode(<br>        text=d[&quot;questions_this_excerpt_can_answer&quot;],<br>        index_id=base_nodes[idx].node_id,<br>    )<br><br>    # Summary node  <br>    # inode_s = IndexNode(<br>    #     text=d[&quot;section_summary&quot;], index_id=base_nodes[idx].node_id)<br><br>    all_nodes.extend([inode_q]) #, inode_s<br><br># all_nodes_dict<br>all_nodes_dict = {n.node_id: n for n in all_nodes}</pre><p><strong>a) Indexing (with MetaData)</strong></p><pre>vector_index_metadata = VectorStoreIndex(all_nodes, service_context=service_context)<br>vector_retriever_metadata = vector_index_metadata.as_retriever(similarity_top_k=2)</pre><p><strong>b) Recursive Retriever (with MetaData)</strong></p><pre>retriever_metadata = RecursiveRetriever(<br>    &quot;vector&quot;,<br>    retriever_dict={&quot;vector&quot;: vector_retriever_metadata},<br>    node_dict=all_nodes_dict,<br>    verbose=True,<br>)<br><br>nodes = retriever_metadata.retrieve(<br>    &quot;Can you tell me about the Paged Optimizers?&quot;<br>)<br><br>for node in nodes:<br>    display_source_node(node, source_length=2000)</pre><p><strong>c) Querying (with MetaData)</strong></p><pre>query_engine_metadata = RetrieverQueryEngine.from_args(<br>    retriever_metadata, service_context=service_context<br>)<br>response = query_engine_metadata.query(<br>    &quot;Can you tell me about the Paged Optimizers?&quot;<br>)</pre><h3>Storing</h3><p>Once you have data loaded and indexed, you will probably want to store it to avoid the time and cost of re-indexing it. By default, your indexed data is stored only in memory.</p><p>We will use Chroma to store the embeddings from a VectorStorendex, you need to:</p><ul><li>initialize the Chroma client</li><li>create a Collection to store your data in Chroma</li><li>assign Chroma as the vector_store in a StorageContext</li><li>initialize your VectorStoreIndex using that StorageContext</li></ul><p>Here’s what that looks like:</p><pre># initialize client, setting path to save data<br>db = chromadb.PersistentClient(path=&quot;./chroma_db&quot;)<br><br># create collection<br>chroma_collection = db.get_or_create_collection(&quot;QLoRa_knowledge_database&quot;)<br><br># assign chroma as the vector_store to the context<br>vector_store = ChromaVectorStore(chroma_collection=chroma_collection)<br>storage_context = StorageContext.from_defaults(vector_store=vector_store)<br><br># create your index<br>vector_index_metadata_db = VectorStoreIndex(<br>    all_nodes,<br>    storage_context=storage_context,<br>    service_context=service_context<br>)</pre><h3>Compare the Outputs</h3><p>Finally, we can run evaluations on each of the retrievers (either chunk or metadata) to measure the hit rate and MRR on our specific use case and select the optimal retriever.</p><p>Refer to the notebook code on Github:</p><p><a href="https://github.com/akashmathur-2212/LLMs-playground/blob/main/LlamaIndex-applications/Advanced-RAG/parent_child_document_retriever/parent_child_document_retriever_metadata_extraction.ipynb">LLMs-playground/LlamaIndex-applications/Advanced-RAG/parent_child_document_retriever/parent_child_document_retriever_metadata_extraction.ipynb at main · akashmathur-2212/LLMs-playground</a></p><p>To refer to other advanced RAG methods, refer to this repo:</p><p><a href="https://github.com/akashmathur-2212/LLMs-playground/tree/main/LlamaIndex-applications/Advanced-RAG">LLMs-playground/LlamaIndex-applications/Advanced-RAG at main · akashmathur-2212/LLMs-playground</a></p><p><strong>References</strong></p><ul><li><a href="https://pub.towardsai.net/advanced-rag-techniques-an-illustrated-overview-04d193d8fec6">https://pub.towardsai.net/advanced-rag-techniques-an-illustrated-overview-04d193d8fec6</a></li></ul><p><em>Thank you for reading this article, I hope it added some pieces to your knowledge stack! Before you go, if you enjoyed reading this article:</em></p><p>👉 <em>Be sure to </em><strong><em>clap</em></strong><em> and </em><strong><em>follow</em></strong><em> me, and let me know if any feedback.</em></p><p>👉<em>I built versatile Generative AI applications using the Large Language Model (LLM), covered advanced RAG concepts, and serverless AWS architectures for Big Data processing. You’re welcome to take a look at the repo and star⭐it.</em></p><ul><li><em>⭐ </em><strong><em>Generative AI</em></strong><em> applications </em><a href="https://github.com/akashmathur-2212/LLMs-playground"><em>Repository</em></a></li><li><em>⭐ </em><strong><em>Advanced RAG</em></strong><em> applications </em><a href="https://github.com/akashmathur-2212/LLMs-playground/tree/main/LlamaIndex-applications/Advanced-RAG"><em>Repository</em></a></li><li><em>⭐ Serverless </em><strong><em>AWS</em></strong><em> architectures </em><a href="https://github.com/akashmathur-2212/aws-serverless-workflows"><em>Repository</em></a></li></ul><p>👉<em>Follow me: </em><a href="https://www.linkedin.com/in/akashmathur22/"><strong><em>LinkedIn</em></strong></a> <strong><em>| </em></strong><a href="https://github.com/akashmathur-2212"><strong><em>GitHub</em></strong></a> <strong><em>| </em></strong><a href="https://akash-mathur.medium.com/"><strong><em>Medium</em></strong></a><strong><em> | </em></strong><a href="https://akashmathur-2212.github.io/"><strong><em>Portfolio</em></strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aeaa32d7aa2f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Elevating Mistral-7B’s Performance through QLoRA]]></title>
            <link>https://akash-mathur.medium.com/elevating-mistral-7bs-performance-through-qlora-b2504cf7c2fe?source=rss-eb7bcac979b6------2</link>
            <guid isPermaLink="false">https://medium.com/p/b2504cf7c2fe</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[fine-tuning]]></category>
            <category><![CDATA[mistral-7b]]></category>
            <category><![CDATA[quantization]]></category>
            <dc:creator><![CDATA[Akash Mathur]]></dc:creator>
            <pubDate>Fri, 15 Dec 2023 09:24:57 GMT</pubDate>
            <atom:updated>2023-12-17T11:10:54.372Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fILw-w9osGAu-IH0" /></figure><p>In recent advancements within the realm of Natural Language Processing (NLP), fine-tuning pre-trained language models has emerged as a pivotal technique for tailoring models to specific tasks and domains. Leveraging the power of large-scale language models such as the GPT (Generative Pre-trained Transformer) series, fine-tuning facilitates the adaptation of these models to particular contexts, yielding superior performance in various language understanding and generation tasks.</p><h3>🌟A Quick Note on Fine-Tuning LLMs🌟</h3><p>Training LLMs is computationally intensive. Full fine-tuning requires memory not just to store the model, but various other parameters that are required during the training process.</p><blockquote>Even if your computer can hold the model weights, which are now on the order of hundreds of gigabytes for the largest models, you must also be able to allocate memory for optimizer states, gradients, forward activations, and temporary memory throughout the training process.</blockquote><p>These additional components can be many times larger than the model and can quickly become too large to handle on consumer hardware.</p><p>In contrast to full fine-tuning where every model weight is updated during supervised learning, <strong>Parameter-Efficient Fine-Tuning (PEFT) </strong>methods only update a small subset of parameters. Some techniques freeze most of the model weights and focus on fine-tuning a subset of existing model parameters, for example, particular layers or components.</p><p>Other techniques don’t touch the original model weights at all, and instead add a small number of new parameters or layers and fine-tune only the new components. With PEFT, most if not all of the LLM weights are kept frozen.</p><p>As a result, the number of trained parameters is much smaller than the number of parameters in the original LLM. In some cases, just <strong>15–20%</strong> of the original LLM weights. This makes the memory requirements for training much more manageable. In fact, PEFT can often be performed on a single GPU. And because the original LLM is only slightly modified or left unchanged, PEFT is less prone to the catastrophic forgetting problems of full fine-tuning</p><p>Full fine-tuning results in a new version of the model for every task you train on. Each of these is the same size as the original model, so it can create an expensive storage problem if you’re fine-tuning for multiple tasks. With PEFT, you train only a small number of weights, which results in a much smaller footprint overall, as small as megabytes depending on the task. The new parameters are combined with the original LLM weights for inference.</p><p>The PEFT weights are trained for each task and can be easily swapped out for inference, allowing efficient adaptation of the original model to multiple tasks. There are several methods you can use for parameter efficient fine-tuning, each with trade-offs on parameter efficiency, memory efficiency, training speed, model quality, and inference costs.</p><p>We will discuss one of the <em>reparameterization </em>methods which <strong>reduces the number of parameters to train by creating new low-rank transformations of the original network weights.</strong> A commonly used technique of this type is <strong>LoRA (Low-rank Adaptation)</strong>, which we’ll explore in this article.</p><h3>🌟(Q)Low-Rank Adaptation (LoRA)🌟</h3><p>LoRA is a parameter-efficient fine-tuning technique that falls into the re-parameterization category.</p><blockquote>LoRA freezes the pretrained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture, greatly reducing the number of trainable parameters for downstream tasks.</blockquote><p>In transformers, there are two kinds of neural networks; <strong>self-attention and feedforward networks</strong>. The weights of these networks are learned during pre-training. After the embedding vectors are created, they’re fed into the self-attention layers where a series of weights are applied to calculate the attention scores. During full fine-tuning, every parameter in these layers is updated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J8mrQK74FHE5rv4MGY9ovg.png" /><figcaption>Basic NN</figcaption></figure><blockquote>A <a href="https://web.stanford.edu/class/cs168/l/l9.pdf"><em>low-rank approximation</em></a> of a matrix aims to approximate the original matrix as closely as possible, but with a lower rank. The <em>rank</em> of a matrix is a value that gives you an idea of the matrix’s complexity; a lower-rank matrix reduces computational complexity, and thus increases efficiency of matrix multiplications. <em>Low-rank decomposition</em> refers to the process of effectively approximating a matrix <em>A</em> by deriving low-rank approximations of <em>A</em>. Singular Value Decomposition (SVD) is a common method for low-rank decomposition.</blockquote><p>More details for the math people—</p><p>For example, suppose we have an LLM with 7B parameters represented in a weight matrix <em>W</em>. (In reality, the model parameters are, of course, distributed across different matrices in many layers, but for simplicity, we refer to a single weight matrix here). During backpropagation, we learn a <em>ΔW</em> matrix, which contains information on how much we want to update the original weights to minimize the loss function during training.</p><p>The weight update is then as follows:</p><p><em>W</em>updated = <em>W</em> + <em>ΔW</em></p><p>If the weight matrix <em>W</em> contains 7B parameters, then the weight update matrix <em>ΔW</em> also contains 7B parameters, and computing the matrix <em>ΔW</em> can be very compute and memory intensive.</p><p>The LoRA method proposed by <a href="https://arxiv.org/abs/2106.09685">Hu <em>et al.</em></a> replaces to decompose the weight changes, <em>ΔW</em>, into a lower-rank representation. To be precise, it does not require to explicitly compute <em>ΔW</em>. Instead, LoRA learns the decomposed representation of <em>ΔW</em> directly during training which is where the savings are coming from, as shown in the figure below.</p><blockquote>The rank of a matrix is the linear space spanned by its rows or columns; to be precise, it is the maximal number of linearly independent columns (or rows) in a matrix. Low-rank decomposition approximates a given matrix <em>A</em> with lower-rank matrices by a product of two matrices <em>U</em> and <em>V</em>, which can then be further decomposed into matrices of lower dimensions.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/837/1*EIGs1WXp9g2Iy67efbSlcA.png" /><figcaption>With LoRA</figcaption></figure><p>As illustrated above, the decomposition of <em>ΔW</em> means that we represent the large matrix <em>ΔW</em> with two smaller LoRA matrices, <em>A</em> and <em>B</em>. If <em>A</em> has the same number of rows as <em>ΔW</em> and <em>B</em> has the same number of columns as <em>ΔW</em>, we can write the decomposition as <em>ΔW = AB</em>. (<em>AB </em>is the matrix multiplication result between matrices <em>A</em> and <em>B</em>.)</p><p>How much memory does this save? It depends on the rank <em>r</em>, which is a hyperparameter. For example, if <em>ΔW</em> has 10,000 rows and 20,000 columns, it stores 200,000,000 parameters. If we choose <em>A</em> and <em>B</em> with <em>r=8</em>, then <em>A</em> has 10,000 rows and 8 columns, and <em>B</em> has 8 rows and 20,000 columns, that’s 10,000×8 + 8×20,000 = 240,000 parameters, which is about 830× less than 200,000,000.</p><p>Of course, <em>A</em> and <em>B</em> can’t capture all the information that <em>ΔW</em> could capture, but this is by design. When using LoRA, we hypothesize that the model requires <em>W</em> to be a large matrix with full rank to capture all the knowledge in the pretraining dataset. However, when we finetune an LLM, we don’t need to update all the weights and capture the core information for the adaptation in a smaller number of weights than <em>ΔW</em> would; hence, we have the low-rank updates via <em>AB</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1023/1*pLqSHMx2uTKWHKReUEqAJA.png" /><figcaption>QLoRA</figcaption></figure><p>In<strong> QLoRA</strong>, the original model’s parameters are first quantized to lower-bit values based on a user-defined quantization configuration. This makes the model more compact. Subsequently, LoRA is applied to the model’s layers to further optimize for the specific task. This combination in QLoRA allows for fine-tuning on significantly less computational power, which essentially democratizes the ability to fine-tune models.</p><p>Because this model has the same number of parameters as the original, there is little to no impact on inference latency. Researchers have found that applying LoRA to just the self-attention layers of the model is often enough to fine-tune for a task and achieve performance gains. However, in principle, you can also use LoRA on other components like the feed-forward layers. However since most of the parameters of LLMs are in the attention layers, you get the biggest savings in trainable parameters by applying LoRA to these weights matrices.</p><h3>🌟Let’s Start Execution!🌟</h3><p>In this notebook and tutorial, we will fine-tune the <a href="https://github.com/mistralai/mistral-src">Mistral 7B</a> model — which outperforms Llama 2 13B on all tested benchmarks.</p><p>In this notebook, we will load the large model in 4bit using bitsandbytes and use LoRA to train using the PEFT library from Hugging Face 🤗.</p><p>If you get an error like this: OutOfMemoryError: CUDA out of memory, tweak your parameters to make the model less computationally intensive.</p><h4>1. Model</h4><p>Let’s now load Mistral — mistralai/Mistral-7B-v0.1 — using 4-bit quantization and set up the tokenizer. Add padding on the left as it <a href="https://ai.stackexchange.com/questions/41485/while-fine-tuning-a-decoder-only-llm-like-llama-on-chat-dataset-what-kind-of-pa">makes training use less memory</a>.</p><p>For model_max_length, it&#39;s helpful to get a distribution of your data lengths.</p><pre>model_id = &quot;mistralai/Mistral-7B-v0.1&quot;<br><br>bnb_config = BitsAndBytesConfig(<br>    load_in_4bit=True,<br>    bnb_4bit_use_double_quant=True,<br>    bnb_4bit_quant_type=&quot;nf4&quot;,<br>    bnb_4bit_compute_dtype=torch.bfloat16<br>)<br><br>model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=bnb_config, device_map=&quot;auto&quot;)<br><br># Set up the tokenizer. Add padding on the left as it makes training use less memory.<br># https://ai.stackexchange.com/questions/41485/while-fine-tuning-a-decoder-only-llm-like-llama-on-chat-dataset-what-kind-of-pa<br><br>tokenizer = AutoTokenizer.from_pretrained(<br>    model_id,<br>    model_max_length=512,<br>    padding_side=&quot;left&quot;,<br>    add_eos_token=True)<br><br>tokenizer.pad_token = tokenizer.eos_token</pre><p>That’s what the model looks like, notice the self-attention layers which we will utilize during LoRA-</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/658/1*Ova7YmNP2DzZKgBWM4UuRg.png" /><figcaption>self_attn layers before LoRA</figcaption></figure><p>You can check the percentage of trainable model parameters using below:</p><pre>def print_number_of_trainable_model_parameters(model):<br>    trainable_model_params = 0<br>    all_model_params = 0<br>    for _, param in model.named_parameters():<br>        all_model_params += param.numel()<br>        if param.requires_grad:<br>            trainable_model_params += param.numel()<br>    return f&quot;trainable model parameters: {trainable_model_params}\nall model parameters: {all_model_params}\npercentage of trainable model parameters: {100 * trainable_model_params / all_model_params:.2f}%&quot;<br><br>print(print_number_of_trainable_model_parameters(model))</pre><h4>2. Dataset</h4><p>We will leverage the Hugging Face datasets library to preprocess financial data for training a language model. The load_dataset function fetches the financial dataset, and a custom function generate_prompt is defined to create prompt texts based on task instructions and corresponding responses in the dataset. These prompts are then added as a new column called &quot;prompt&quot; to the dataset.</p><p>The dataset is shuffled using a specified seed to randomize the order of examples. Subsequently, the tokenizer function processes the &quot;prompt&quot; column&#39;s text data in batches, tokenizing it for model training.</p><p>Lastly, the dataset is split into training and testing subsets, with 90% of the data allocated for training and 10% for testing, facilitating supervised learning on the language model. The resulting training and testing datasets (train_data and test_data) are prepared for subsequent model training and evaluation, respectively.</p><pre>from datasets import load_dataset<br><br>data = load_dataset(&quot;gbharti/finance-alpaca&quot;, split=&#39;train&#39;)<br><br># Define a function to generate a prompt text based on a data point<br>def generate_prompt(data_point):<br>    &quot;&quot;&quot;<br>    Generate input text based on a prompt, task instruction, (context info.), and answer<br><br>    :param data_point: dict: Data point<br>    :return: dict: tokenized prompt<br>    &quot;&quot;&quot;<br>    # Create a text with just instruction and response<br>    text = &#39;Below is an instruction that describes a task. Write a response that &#39; \<br>           &#39;appropriately completes the request.\n\n&#39;<br>    text += f&#39;### Instruction:\n{data_point[&quot;instruction&quot;]}\n\n&#39;<br>    text += f&#39;### Response:\n{data_point[&quot;output&quot;]}&#39;<br>    return text<br><br># Add the &quot;prompt&quot; column in the dataset by applying the generate_prompt function to each data point<br>text_column = [generate_prompt(data_point) for data_point in data]<br>data = data.add_column(&quot;prompt&quot;, text_column)<br><br># Shuffle the dataset with a specified seed<br>data = data.shuffle(seed=1234)<br><br># Tokenize the &quot;prompt&quot; column using the tokenizer, processing the data in batches<br>data = data.map(lambda samples: tokenizer(samples[&quot;prompt&quot;]), batched=True)<br><br># Split the dataset into training and testing subsets, with 90% for training and 10% for testing<br>data = data.train_test_split(test_size=0.1)<br>train_data = data[&quot;train&quot;]<br>test_data = data[&quot;test&quot;]</pre><h4>3. Setup the PEFT/LoRA model for Fine-Tuning</h4><p>Now we need to set up the LoRA model for fine-tuning with a new layer/parameter adapter. Using PEFT/LoRA, we will be freezing the underlying LLM and only training the adapter. Have a look at the LoRA configuration below. Note the rank (`r`) hyper-parameter, which defines the rank/dimension of the adapter to be trained.</p><p>As we saw earlier the model layers, as we will apply QLoRA to all the self_attnlayers of the model. Those layers are q_proj, k_proj, v_proj, o_proj.</p><p>Now, we define the LoRA config.</p><ul><li>r is the rank of the low-rank matrix used in the adapters, which thus controls the number of parameters trained. A higher rank will allow for more expressivity, but there is a compute tradeoff.</li><li>alpha is the scaling factor for the learned weights. The weight matrix is scaled by alpha/r, and thus a higher value for alpha assigns more</li></ul><p>We will use r=8 and lora_alpha=16 so that we have more emphasis on the new fine-tuned data while also reducing computational complexity.</p><pre>lora_target_modules = [<br>    &quot;q_proj&quot;,<br>    &quot;up_proj&quot;,<br>    &quot;o_proj&quot;,<br>    &quot;k_proj&quot;,<br>    # &quot;down_proj&quot;,<br>    # &quot;gate_proj&quot;,<br>    # &quot;v_proj&quot;,<br>]<br><br>lora_config = LoraConfig(<br>    r=8,<br>    lora_alpha=16,<br>    target_modules=lora_target_modules,<br>    lora_dropout=0.1,<br>    bias=&quot;none&quot;,<br>    task_type=&quot;CAUSAL_LM&quot;<br>)<br><br># Add LoRA adapter layers/parameters to the original LLM to be trained.<br>model = get_peft_model(model, lora_config)<br><br>print(print_number_of_trainable_model_parameters(model))</pre><p>Now, look at the Mistral’s layers after LoRA</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/694/1*q7LUqaVrMCOHCKFuGMu_aw.png" /><figcaption>self_attn layers after LoRA</figcaption></figure><h4>4. Start Training</h4><p>Define training arguments and create a `Trainer` instance. A note on training:</p><blockquote>You can set the max_steps to be high initially, and examine at what step your model’s performance starts to degrade. There is where you’ll find a sweet spot for how many steps to perform. For example, say you start with 500 steps, and find that at around 100 steps the model starts overfitting — the validation loss goes up (bad) while the training loss goes down significantly, meaning the model is learning the training set really well, but is unable to generalize to new datapoints. Therefore, 100 steps would be your sweet spot, so you would use the checkpoint-100 model repo in your <strong>output dir</strong> as your final model.</blockquote><pre>OUTPUT_DIR = &quot;output&quot;<br><br>peft_training_args = transformers.TrainingArguments(<br>    per_device_train_batch_size=1,<br>    gradient_accumulation_steps=4,<br>    num_train_epochs=1,<br>    learning_rate=2e-4, # Want a small lr for finetuning<br>    fp16=True,<br>    save_total_limit=3,<br>    logging_steps=1,<br>    output_dir=OUTPUT_DIR,<br>    max_steps=5,<br>    optim=&quot;paged_adamw_8bit&quot;,<br>    lr_scheduler_type=&quot;cosine&quot;,<br>    warmup_ratio=0.05<br>)<br><br>trainer = transformers.Trainer(<br>    model=model,<br>    train_dataset=train_data,<br>    eval_dataset=test_data,<br>    args=peft_training_args,<br>    data_collator=transformers.DataCollatorForLanguageModeling(tokenizer, mlm=False),<br>)<br>model.config.use_cache = False # silence the warnings. Please re-enable for inference!<br>trainer.train()</pre><p><strong>Save model</strong></p><p>After your model is finished training, you can save your model to a directory and push it to HF <a href="https://huggingface.co/akash2212/mistral_7b_finetuned">Space</a>.</p><pre>model.save_pretrained(&quot;output_dir&quot;)<br><br>from huggingface_hub import notebook_login<br>notebook_login()<br><br>model.push_to_hub(&quot;akash2212/mistral_7b_finetuned&quot;, use_auth_token=True)<br>tokenizer.push_to_hub(&quot;akash2212/mistral_7b_finetuned&quot;, use_auth_token=True)</pre><h4>5. Inference Time!</h4><p>Lets load the specified pre-trained weights and configurations for inference tasks.</p><pre>PEFT_MODEL = &quot;akash2212/mistral_7b_finetuned&quot;<br><br>config = PeftConfig.from_pretrained(PEFT_MODEL)<br>model = AutoModelForCausalLM.from_pretrained(<br>    config.base_model_name_or_path,<br>    return_dict=True,<br>    quantization_config=bnb_config,<br>    device_map=&quot;auto&quot;,<br>    trust_remote_code=True,<br>)<br>tokenizer = AutoTokenizer.from_pretrained(config.base_model_name_or_path)<br>tokenizer.pad_token = tokenizer.eos_token<br><br># Load the Lora model<br>model = PeftModel.from_pretrained(model, PEFT_MODEL)</pre><p>The below function generates a response or completion for a given query by preparing a prompt, tokenizing it, feeding it into the model, and returning the generated completion as text.</p><pre>def get_completion(query: str, model, tokenizer) -&gt; str:<br><br>  prompt_template = &quot;&quot;&quot;<br>  Below is an instruction that describes a task. Write a response that appropriately completes the request.<br>  ### Question:<br>  {query}<br><br>  ### Answer:<br>  &quot;&quot;&quot;<br>  prompt = prompt_template.format(query=query)<br><br>  encodeds = tokenizer(prompt, return_tensors=&quot;pt&quot;, add_special_tokens=True)<br><br>  model_inputs = encodeds.to(DEVICE)<br><br><br>  generated_ids = model.generate(**model_inputs, max_new_tokens=1000, do_sample=True, pad_token_id=tokenizer.eos_token_id)<br>  decoded = tokenizer.batch_decode(generated_ids)<br>  return (decoded[0])<br><br><br>result = get_completion(query=&quot;Pay off car loan entirely or leave $1 until the end of the loan period?&quot;, model=model, tokenizer=tokenizer)<br>print(result)</pre><h4>🌟Additional Notes🌟</h4><ol><li><strong>Balancing LoRA Hyperparameters: R and Alpha</strong></li></ol><p>As the <a href="https://arxiv.org/abs/2106.09685">original LoRA paper</a> outlines, LoRA introduces an additional scaling coefficient for applying the LoRA weights to the pretrained weights during the forward pass. The scaling involves the rank parameter r, which we discussed earlier, as well as another hyperparameter α (alpha) that is applied as follows:</p><pre>scaling = alpha / r<br>weight += (lora_B @ lora_A) * scaling</pre><p>As we can see in the code formula above, the larger the influence of the LoRA weights.</p><p>In this experiment, I used <em>r=8</em> and <em>alpha=16</em>, which resulted in a 2-fold scaling. Choosing alpha as two times <em>r</em> is a common rule of thumb when using LoRA for LLMs, but I was curious if this still holds for larger <em>r</em> values. In other words, “alpha = 2×rank” really seems to be a sweet spot.</p><p>Choosing alpha as two times as large as <em>r</em> may often result in the best outcomes, but it may also not hurt to experiment with different ratios.</p><p>2. <strong>Enable LoRA for More Layers</strong></p><p>The tables above showed experiments where LoRA was only enabled for select weight matrices, i.e., the Key and Value weight matrices in each transformer layer. In addition, we can also enable LoRA for the Query weight matrices, the other linear layers between the multihead attention blocks, and the linear output layer.</p><p>If we enable LoRA for all these additional layers, we increase the number of trainable parameters which comes with a larger memory requirement but can increase the modeling performance noticeably. It might be worthwhile exploring the other combinations in future experiments.</p><p>To conclude, here’s a simplified explanation of how LoRA works</p><ol><li><strong>Initial Model</strong>: Start with a large pre-trained model (e.g., Llama 2, Mistral, etc.).</li><li><strong>Low-Rank Matrix</strong>: Introduce low-rank approximations for the matrices that will be used to adapt the model for the specific task at hand. Low-rank matrices (adapters) are typically formed for all linear layers of the model, but this can vary based on the model architecture and the task.</li><li><strong>Transform Layers</strong>: Instead of directly modifying the original weights of the model, LoRA applies a transformation using the low-rank matrices to the outputs of affected layers.</li><li><strong>Fine-tuning</strong>: During the fine-tuning process, only the parameters in the low-rank matrices are updated. The rest of the model’s parameters are kept fixed. Again, updating only low-rank matrices allows for fine-tuning on smaller, cheaper GPUs.</li><li><strong>Prediction</strong>: For making predictions, the adapted layers are used in conjunction with the original pre-trained model. The low-rank adapted layers act as a kind of “add-on” to the existing architecture, adjusting its behavior for the specific task.</li></ol><h4>🌟Some benefits of LoRA🌟</h4><ol><li><strong>Efficiency</strong>: Because it only updates a small subset of parameters, fine-tuning is faster and requires less computational power. According to the LoRA paper, it can reduce the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times.</li><li><strong>Specialization</strong>: The low-rank adaptation allows the model to specialize in a particular task without a complete overhaul of the original weights.</li><li><strong>Preservation</strong>: By keeping the bulk of the model fixed, LoRA helps preserve the generalization capabilities of the original pre-trained model while still enabling task-specific adaptations.</li></ol><p>Refer to the notebook code on Github:</p><p><a href="https://github.com/akashmathur-2212/LLMs-playground/tree/main/mistral-finetune-using-LoRA">LLMs-playground/mistral-finetune-using-LoRA at main · akashmathur-2212/LLMs-playground</a></p><p><em>Thank you for reading this article, I hope it added some pieces to your knowledge stack! Before you go, if you enjoyed reading this article:</em></p><ul><li><em>Be sure to </em><strong><em>clap</em></strong><em> and </em><strong><em>follow</em></strong><em> me, and let me know if any feedback.</em></li><li><em>I built versatile </em><a href="https://github.com/akashmathur-2212/LLMs-playground"><em>applications</em></a><em> using the Large Language Model (LLM) and serverless </em><a href="https://github.com/akashmathur-2212/aws-serverless-workflows"><em>AWS architectures</em></a><em> for Big Data processing. You’re welcome to take a look at the repo and star⭐it.</em></li><li><em>Follow me: </em><a href="https://www.linkedin.com/in/akashmathur22/"><strong><em>LinkedIn</em></strong></a> <strong><em>| </em></strong><a href="https://github.com/akashmathur-2212"><strong><em>GitHub</em></strong></a> <strong><em>| </em></strong><a href="https://akash-mathur.medium.com/"><strong><em>Medium</em></strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b2504cf7c2fe" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Vector Database vs. Indexing- Path to Efficient Data Handling]]></title>
            <link>https://akash-mathur.medium.com/vector-database-vs-indexing-path-to-efficient-data-handling-382cc1207491?source=rss-eb7bcac979b6------2</link>
            <guid isPermaLink="false">https://medium.com/p/382cc1207491</guid>
            <category><![CDATA[vector-search]]></category>
            <category><![CDATA[retrieval-augmented]]></category>
            <category><![CDATA[vector]]></category>
            <category><![CDATA[indexing]]></category>
            <category><![CDATA[vector-database]]></category>
            <dc:creator><![CDATA[Akash Mathur]]></dc:creator>
            <pubDate>Sun, 10 Dec 2023 11:36:43 GMT</pubDate>
            <atom:updated>2023-12-15T08:56:13.174Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6F4xJ_hyFxrSOvfk.jpg" /></figure><p>In the realm of digital information and creative applications, the terms ‘<strong>vector database</strong>’ and ‘<strong>vector library</strong>’ often emerge, each encompassing distinct functionalities and serving unique purposes. While both deal with vectors, their roles diverge significantly. Understanding the nuances between these entities is crucial in harnessing their potential across various fields.</p><h3>But why Vector search?</h3><p>Vector search has gained prominence due to its efficacy in handling high-dimensional data across all the fields of artificial intelligence.</p><p>Traditional methods of searching and indexing struggle when dealing with high-dimensional data due to the curse of dimensionality, where distances between points lose meaning and computational costs soar exponentially with dimensionality.</p><p>That’s where Vector Databases and Vector Libraries come in. Vector search, often facilitated by techniques like approximate nearest neighbor search, addresses these challenges by representing data points as vectors in a continuous space, enabling efficient similarity searches, clustering, recommendation systems, and retrieval tasks. This approach allows for the rapid and accurate retrieval of similar or relevant items from vast datasets, making it invaluable in scenarios where traditional search methods falter.</p><h3>Vector Library</h3><p>Vector libraries focus on storing vector embeddings in indexes stored in memory. They have specific features:</p><ol><li><strong>Storing Only Vectors</strong>: They keep the vector embeddings only, not the original objects they came from.</li><li><strong>Immutable Data</strong>: Once you build the index with your data, you can’t change it. No adding, deleting, or modifying afterward without rebuilding.</li><li><strong>Query Limitation</strong>: Many libraries need you to import all data before building the index. You can’t query during import, which might be a problem for big datasets.</li></ol><p>When you query a vector library, you get back <em>vectors </em>and <em>object IDs</em>. But the real information is in the <em>objects</em>, not just their <em>IDs</em>. So, to get the full picture, you’d need to keep the objects stored separately and match IDs to objects.</p><p>Examples like Facebook <a href="https://engineering.fb.com/2017/03/29/data-infrastructure/faiss-a-library-for-efficient-similarity-search/">Faiss</a>, Spotify <a href="https://github.com/spotify/annoy">Annoy</a>, and others use the ANN algorithm for similarity search. Each has its way of implementing it — Faiss with clustering, Annoy with trees, and <a href="https://github.com/google-research/google-research/tree/master/scann">ScaNN </a>with vector compression.</p><p>Choosing one depends on what your application needs and how you measure performance.</p><blockquote>I have written an article on Vector Indexing and ANN with AWS Serverless Architecture</blockquote><blockquote>Read more about it on <a href="https://akash-mathur.medium.com/unlocking-faiss-for-efficient-search-vector-indexing-and-ann-with-serverless-architecture-5b2b59ead20f">Medium</a></blockquote><blockquote>Code Repository <a href="https://github.com/akashmathur-2212/Recommendation-System-Playground/tree/main/faiss-similarity-search">GitHub</a></blockquote><h3>Vector Database</h3><p>Vector databases go beyond indexing and approximate nearest-neighbor search algorithms. Vector databases are specifically designed to manage vector embeddings and offer several advantages:</p><p>1. <strong>Data management:</strong> Vector databases offer well-known and easy-to-use features for data storage, like inserting, deleting, and updating data. This makes managing and maintaining vector data easier than using a standalone vector <em>index</em> like FAISS, which requires additional work to integrate with a storage solution.</p><p>2. <strong>Metadata storage and filtering:</strong> Vector databases can store metadata associated with each vector entry. Users can then query the database using additional metadata filters for finer-grained queries.</p><p>3. <strong>Scalability:</strong> Vector databases are designed to scale with growing data volumes and user demands, providing better support for distributed and parallel processing. Standalone vector indices may require custom solutions to achieve similar levels of scalability (such as deploying and managing them on Kubernetes clusters or other similar systems).</p><p>4. <strong>Real-time updates:</strong> Vector databases often support real-time data updates, allowing for dynamic changes to the data, whereas standalone vector indexes may require a full re-indexing process to incorporate new data, which can be time-consuming and computationally expensive.</p><p>5. <strong>Backups and collections:</strong> Vector databases handle the routine operation of backing up all the data stored in the database. Pinecone also allows users to selectively choose specific indexes that can be backed up in the form of “collections,” which store the data in that index for later use.</p><p>6. <strong>Ecosystem integration:</strong> Vector databases can more easily integrate with other components of a data processing ecosystem, such as ETL pipelines (like Spark), analytics tools (like <a href="https://www.tableau.com/">Tableau</a> and <a href="https://segment.com/">Segment</a>), and visualization platforms (like <a href="https://grafana.com/">Grafana</a>) — streamlining the data management workflow. It also enables easy integration with other AI related tools like <a href="https://python.langchain.com/en/latest/index.html">LangChain</a>, <a href="https://gpt-index.readthedocs.io/">LlamaIndex</a> and <a href="https://openai.com/blog/chatgpt-plugins">ChatGPT’s Plugins</a>.</p><p>7. <strong>Data security and access control:</strong> Vector databases typically offer built-in data security features and access control mechanisms to protect sensitive information, which may not be available in standalone vector index solutions.</p><h3>Vector Database Features</h3><h4><strong>Filtering</strong></h4><p>Every vector stored in the database also includes metadata. In addition to the ability to query for similar vectors, vector databases can also filter the results based on a metadata query. To do this, the vector database usually maintains two indexes: a vector index and a metadata index.</p><p>The filtering process can be performed either before or after the vector search itself, but each approach has its challenges that may impact the query performance:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/715/1*DwYWRzxXPgpabIiLgtOuoQ.png" /><figcaption>Pre and Post Filtering</figcaption></figure><p>· <strong>Pre-filtering:</strong> In this approach, metadata filtering is done before the vector search. While this can help reduce the search space, it may also cause the system to overlook relevant results that don’t match the metadata filter criteria. Additionally, extensive metadata filtering may slow down the query process due to the added computational overhead.</p><p>· <strong>Post-filtering:</strong> In this approach, the metadata filtering is done after the vector search. This can help ensure that all relevant results are considered, but it may also introduce additional overhead and slow down the query process as irrelevant results need to be filtered out after the search is complete.</p><p>To optimize the filtering process, vector databases use various techniques, such as leveraging advanced indexing methods for metadata or using parallel processing to speed up the filtering tasks. Balancing the trade-offs between search performance and filtering accuracy is essential for providing efficient and relevant query results in vector databases.</p><h4><strong>Database Operations</strong></h4><p>Unlike vector indexes, vector databases are equipped with a set of capabilities that makes them better qualified to be used in high-scale production settings. Let’s take a look at an overall overview of the components that are involved in operating the database.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/794/1*l5BkkfbezNyjo5i1Ji_JGQ.png" /><figcaption>Database Components</figcaption></figure><p><strong><em>Performance and Fault tolerance</em></strong></p><p>Performance and fault tolerance are tightly related. The more data we have, the more nodes that are required — and the bigger chance for errors and failures. As is the case with other types of databases, we want to ensure that queries are executed as quickly as possible even if some of the underlying nodes fail.</p><p>To ensure both high performance and fault tolerance, vector databases use sharding and replication apply the following:</p><p>1. <strong>Sharding</strong> — partitioning the data across multiple nodes. There are different methods for partitioning the data — for example, it can be partitioned by the similarity of different clusters of data so that similar vectors are stored in the same partition. When a query is made, it is sent to all the shards and the results are retrieved and combined.</p><p>2. <strong>Replication</strong> — creating multiple copies of the data across different nodes. This ensures that even if a particular node fails, other nodes will be able to replace it.</p><p><strong><em>Monitoring</em></strong></p><p>To effectively manage and maintain a vector database, we need a robust monitoring system that tracks the important aspects of the database’s performance, health, and overall status. Some aspects of monitoring a vector database include the following:</p><p>1. <strong>Resource usage</strong> — monitoring resource usage, such as CPU, memory, disk space, and network activity, enables the identification of potential issues or resource constraints that could affect the performance of the database.</p><p>2. <strong>Query performance</strong> — query latency, throughput, and error rates may indicate potential systemic issues that need to be addressed.</p><p>3. <strong>System health</strong> — overall system health monitoring includes the status of individual nodes, the replication process, and other critical components.</p><p><strong><em>Access-control</em></strong></p><p>Access control is the process of managing and regulating user access to data and resources. It is a vital component of data security, ensuring that only authorized users can view, modify, or interact with sensitive data stored within the vector database.</p><h3>Which one is right for me?</h3><p>Vector databases and vector libraries are both technologies that enable vector similarity search, but they differ in functionality and usability.</p><p>· <strong>Vector databases</strong> can store and update data, handle various types of data sources, perform queries during data import, and provide user-friendly and enterprise-ready features.</p><p>· <strong>Vector libraries</strong> can only store data, handle vectors only, require importing all the data before building the index, and require more technical expertise and manual configuration.</p><p>Some vector databases are built on top of existing libraries, such as <a href="https://faiss.ai/">Faiss</a>. This allows them to take advantage of the existing code and features of the library, which can save time and effort in development.</p><p>Determining whether a vector database or a vector library is right for you depends on your specific needs and the context in which you intend to use these tools. Whether it’s about managing and querying large volumes of data points or utilizing predefined vector graphics and operations — will help you determine which option aligns better with your needs.</p><h3>🌟Bonus! 🌟</h3><p>As you have come this far, let me share the best Vector Databases and libraries you can choose from. The choice between the two depends on the specific requirements and scale of the application.</p><h4>Best Vector Databases &amp; Libraries</h4><p>· <a href="https://www.elastic.co/elasticsearch/vector-database">Elasticsearch</a> — A distributed search and analytics engine that supports various types of data.</p><p>· <a href="https://faiss.ai/">Faiss</a> — A library for efficient similarity search and clustering of dense vectors. It contains algorithms that search in sets of vectors of any size, up to ones that possibly do not fit in RAM.</p><p>· <a href="https://milvus.io/">Milvus</a> — An open-source vector database that can manage trillions of vector datasets and supports multiple vector search indexes and built-in filtering.</p><p>· <a href="https://qdrant.tech/">Qdrant</a> — A vector similarity search engine and vector database. It provides a production-ready service with a convenient API to store, search, and manage data points.</p><p>· <a href="https://www.trychroma.com/">Chroma</a> — An AI-native open-source embedding database. It is simple, feature-rich, and integrable with various tools and platforms for working with embeddings.</p><p>· <a href="https://opensearch.org/">OpenSearch</a> — A community-driven, open source fork of Elasticsearch and Kibana following the license change in early 2021. It includes a vector database functionality that allows you to store and index vectors and metadata, and perform vector similarity search using k-NN indexes.</p><p>· <a href="https://weaviate.io/">Weaviate</a> — An open-source vector database that allows you to store data objects and vector embeddings from your favorite ML-models, and scale seamlessly into billions of data objects.</p><p>· <a href="https://vespa.ai/">Vespa</a>— A fully featured search engine and vector database. It supports vector search (ANN), lexical search, and search in structured data, all in the same query. Integrated machine-learned model inference allows you to apply AI to make sense of your data in real time.</p><p>· <a href="https://github.com/pgvector/pgvector">pgvector</a> — An open-source extension for PostgreSQL that allows you to store and query vector embeddings within your database. It is built on top of the Faiss library, which is a popular library for efficient similarity search of dense vectors.</p><p>· <a href="https://vald.vdaas.org/">Vald</a> — A highly scalable distributed fast approximate nearest neighbor dense vector search engine. Vald is designed and implemented based on the Cloud-Native architecture. It uses the fastest ANN Algorithm NGT to search neighbors.</p><p>· <a href="https://cassandra.apache.org/">Apache Cassandra</a> — An open-source NoSQL distributed database trusted by thousands of companies. Vector search is coming to Apache Cassandra in its 5.0 release, which is expected to be available in late 2023 or early 2024.</p><p>· <a href="https://github.com/google-research/google-research/tree/master/scann">ScaNN</a> (Scalable Nearest Neighbors, Google Research) — A library for efficient vector similarity search, which finds the k nearest vectors to a query vector, as measured by a similarity metric.</p><p><a href="https://www.pinecone.io/">Pinecone</a> — A vector database that is designed for machine learning applications. It is fast, scalable, and supports a variety of machine learning algorithms. Pinecone is built on top of Faiss, a library for efficient similarity search of dense vectors.</p><p><em>Thank you for reading this article, I hope it added some pieces to your knowledge stack! Before you go, if you enjoyed reading this article:</em></p><ul><li><em>Be sure to </em><strong><em>clap</em></strong><em> and </em><strong><em>follow</em></strong><em> me, and let me know if any feedback.</em></li><li><em>I built versatile </em><a href="https://github.com/akashmathur-2212/LLMs-playground"><em>applications</em></a><em> using the Large Language Model (LLM) and serverless </em><a href="https://github.com/akashmathur-2212/aws-serverless-workflows"><em>AWS architectures</em></a><em> for Big Data processing. You’re welcome to take a look at the repo and star⭐it.</em></li><li><em>Follow me: </em><a href="https://www.linkedin.com/in/akashmathur22/"><strong><em>LinkedIn</em></strong></a> <strong><em>| </em></strong><a href="https://github.com/akashmathur-2212"><strong><em>GitHub</em></strong></a> <strong><em>| </em></strong><a href="https://akash-mathur.medium.com/"><strong><em>Medium</em></strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=382cc1207491" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Comprehensive Guide to Customize your Llama2 ChatBot with LlamaIndex and Streamlit]]></title>
            <link>https://akash-mathur.medium.com/comprehensive-guide-to-customize-your-llama2-chatbot-with-llamaindex-and-streamlit-76bbd041eafc?source=rss-eb7bcac979b6------2</link>
            <guid isPermaLink="false">https://medium.com/p/76bbd041eafc</guid>
            <category><![CDATA[vector-database]]></category>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[retrieval-augmented]]></category>
            <category><![CDATA[chatbots]]></category>
            <dc:creator><![CDATA[Akash Mathur]]></dc:creator>
            <pubDate>Tue, 28 Nov 2023 19:28:31 GMT</pubDate>
            <atom:updated>2023-11-29T12:06:13.175Z</atom:updated>
            <content:encoded><![CDATA[<h3>Comprehensive Guide to Customize your Llama2 ChatBot using LlamaIndex and Streamlit</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8iuWWjIwkiDX9Rzv.png" /><figcaption>An image by Author</figcaption></figure><h3>Problem Statement</h3><p>In today’s information-driven landscape, extracting valuable insights locked within numerous data sources (APIs, PDFs, documents, CSV, SQL, etc.) presents a formidable challenge. The wealth of knowledge residing in these documents often remains untapped due to the sheer volume and the time-consuming nature of manual extraction.</p><h3>Solution</h3><p><a href="https://www.llamaindex.ai/">LlamaIndex </a>is a simple, flexible data framework for connecting<br>custom data sources to large language models. It provides the key tools to augment your LLM applications with data</p><blockquote>LLMs come pre-trained on huge amounts of publicly available data like Wikipedia, mailing lists, textbooks, source code and more. However, they are not trained on <strong>your</strong> data, which may be private or specific to the problem you’re trying to solve. It’s behind APIs, in SQL databases, or trapped in PDFs and slide decks.</blockquote><p>LlamaIndex solves this problem by connecting to these data sources and adding your data to the data LLMs already have. This is often called <strong>Retrieval-Augmented Generation (RAG)</strong>. RAG enables you to use LLMs to query your data, transform it, and generate new insights. You can ask questions about your data, create chatbots, build semi-autonomous agents, and more.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TD7yyNVxHZkbnS9e9rwm3Q.png" /></figure><h3>Use Case</h3><p>In this blog, we will see the use case of <strong>Q&amp;A</strong> on the dataset (multiple PDFs) and <strong>Chatbot </strong>which can handle multiple back-and-forth queries and answers, getting clarification or answering follow-up questions. In the end, we will deploy our solution to Streamlit.</p><p>LlamaIndex gives you the tools to build knowledge-augmented chatbots. Throughout this blog, you’ll notice sections are arranged in the order you’ll perform these steps while building your app -</p><ol><li>Models</li><li>Loading</li><li>Indexing</li><li>Storing</li><li>Querying</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r6eRtCwpaJvhOrbx2xihug.png" /></figure><p>Let’s deep dive into each section through our use case:</p><h3>Models</h3><h4>LLMs</h4><p>One of the first steps when building an LLM-based application is which LLM to use. LLMs are a core component of LlamaIndex. They can be used as standalone modules or plugged into other core LlamaIndex modules (indices, retrievers, query engines).</p><p>Let’s look at how LLMs are used at multiple different stages of your pipeline:</p><ol><li>During <strong>Indexing,</strong> you may use an LLM to determine the relevance of data (whether to index it at all) or you may use an LLM to summarize the raw data and index the summaries instead.</li></ol><p>2. During <strong>Querying</strong> LLMs can be used in two ways:</p><ul><li>During <strong>Retrieval</strong> (fetching data from your index) LLMs can be given an array of options (such as multiple different indices) and make decisions about where best to find the information you’re looking for. An agentic LLM can also use <em>tools</em> at this stage to query different data sources.</li><li>During <strong>Response Synthesis</strong> (turning the retrieved data into an answer) an LLM can combine answers to multiple sub-queries into a single coherent answer, or it can transform data, such as from unstructured text to JSON or another programmatic output format.</li></ul><p>Usually, you will instantiate an LLM and pass it to a ServiceContext, which you then pass to other stages of the pipeline.</p><blockquote><strong>Quick Note on Tokenization</strong></blockquote><blockquote>By default, LlamaIndex uses a global tokenizer for all token counting. This defaults to cl100k from tiktoken, which is the tokenizer to match the default LLM gpt-3.5-turbo.</blockquote><blockquote>If you change the LLM, you may need to update this tokenizer to ensure accurate token counts, chunking, and prompting.</blockquote><p>In this example, we will use Open Source <a href="https://huggingface.co/meta-llama/Llama-2-7b-chat-hf">meta-llama/Llama-2–7b-chat-hf</a> as our LLM and will quantify it for memory and computation. This should run on a T4 GPU in the free tier on Colab.</p><pre>import torch<br>from transformers import BitsAndBytesConfig<br>from llama_index.prompts import PromptTemplate<br>from llama_index.llms import HuggingFaceLLM<br><br># huggingface api token for downloading llama2<br>hf_token = &quot;your-hugging-face-token&quot;<br><br>quantization_config = BitsAndBytesConfig(<br>    load_in_4bit=True,<br>    bnb_4bit_compute_dtype=torch.float16,<br>    bnb_4bit_quant_type=&quot;nf4&quot;,<br>    bnb_4bit_use_double_quant=True,<br>)<br><br>llm = HuggingFaceLLM(<br>    model_name=&quot;meta-llama/Llama-2-7b-chat-hf&quot;,<br>    tokenizer_name=&quot;meta-llama/Llama-2-7b-chat-hf&quot;,<br>    query_wrapper_prompt=PromptTemplate(&quot;&lt;s&gt; [INST] {query_str} [/INST] &quot;),<br>    context_window=3900,<br>    model_kwargs={&quot;token&quot;: hf_token, &quot;quantization_config&quot;: quantization_config},<br>    tokenizer_kwargs={&quot;token&quot;: hf_token},<br>    device_map=&quot;auto&quot;,<br>)</pre><h4><strong>Embeddings</strong></h4><p>Embeddings are used in LlamaIndex to represent your documents using a sophisticated numerical representation. Embedding models take text as input and return a long list of numbers used to capture the semantics of the text. These embedding models have been trained to represent text this way and help enable many applications, including search.</p><p>When calculating the similarity between embeddings, there are many methods to use (dot product, cosine similarity, etc.). By default, LlamaIndex uses cosine similarity when comparing embeddings.</p><p>There are many embedding models to pick from. By default, LlamaIndex uses <em>text-embedding-ada-002</em> from OpenAI.</p><p>In this example, we will use <a href="https://huggingface.co/hkunlp/instructor-large">HuggingFaceInstructEmbeddings</a>. This is an instruction-finetuned text embedding model that can generate text embeddings tailored to any task (e.g., classification, retrieval, clustering, text evaluation, etc.) <em>by simply providing the task instruction, without any finetuning</em>. Instructor👨‍ ranks at #14 on the <a href="https://huggingface.co/spaces/mteb/leaderboard">MTEB leaderboard</a>!</p><p>We will useSentenceSplitterthat split the text while respecting the boundaries of sentences. This function is used to break down large bodies of text into smaller sections, ensuring that sentences aren&#39;t split in the middle and making it easier to process or analyze text data in manageable portions.</p><p>The most common usage for an embedding model will be setting it in the ServiceContextobject, and then using it to construct an index and query. The input documents will be broken into nodes, and the embedding model will generate an embedding for each node. Then, at query time, the embedding model will be used again to embed the query text.</p><pre>embed_model = HuggingFaceInstructEmbeddings(<br>    model_name=&quot;hkunlp/instructor-large&quot;, model_kwargs={&quot;device&quot;: DEVICE}<br>)<br><br>text_splitter = SentenceSplitter(chunk_size=1024, chunk_overlap=20) # using the default chunk_# values as they work just fine<br><br>service_context = ServiceContext.from_defaults(llm=llm,<br>                                               embed_model=embed_model,<br>                                               text_splitter=text_splitter<br>                                               )</pre><h3>Loading</h3><p>Before your chosen LLM can act on your data you need to load it. The way LlamaIndex does this is via data connectors, also called Reader. Data connectors ingest data from different data sources and format the data into Document objects. A Document is a collection of data (currently text, and in the future, images, and audio) and metadata about that data.</p><h4>Loading using SimpleDirectoryReader</h4><p>The easiest reader to use is our SimpleDirectoryReader, which creates documents out of every file in a given directory. It is built into LlamaIndex and can read a variety of formats including Markdown, PDFs, Word documents, images, etc.</p><pre>from llama_index import SimpleDirectoryReader<br><br># Load data<br>documents = SimpleDirectoryReader(&#39;./sample_data/pdfs&#39;).load_data()<br>len(documents)</pre><h3>Indexing</h3><p>With your data loaded, you now have a list of Document objects (or a list of Nodes). It’s time to build an Index over these objects so you can start querying them.</p><p>In LlamaIndex terms, an Index is a data structure composed of Document objects, designed to enable querying by an LLM.</p><h4>Parsing Documents into Nodes</h4><p>Under the hood, indexers split your Document into Node objects, which are similar to Documents (they contain text and metadata) but have a relationship to their parent Document.</p><blockquote><strong>Node</strong> corresponds to a chunk of text from a Document. LlamaIndex takes in Document objects and internally parses/chunks them into Node objects</blockquote><p>The way in which your text is split up can have a large effect on the performance of your application in terms of accuracy and relevance of results returned. The defaults work well for simple text documents, so depending on what your data looks like you will sometimes want to modify the default ways in which your documents are split up.</p><blockquote>Remember, a ServiceContext is a simple bundle of configuration data passed to many parts of LlamaIndex.</blockquote><p>Indexes have a .from_documents() method which accepts an array of Document objects and will correctly parse and chunk them up.</p><pre>vector_index = VectorStoreIndex.from_documents(documents, service_context=service_context)</pre><p>When you want to search your embeddings, your query is itself turned into a vector embedding, and then a mathematical operation is carried out VectorStorendexto rank all the embeddings by how semantically similar they are to your query.</p><p><strong><em>Reference</em></strong>: Colab <a href="https://github.com/akashmathur-2212/LLMs-playground/blob/main/LlamaIndex-applications/llama2-multi-documents-chatbot/chatbot.ipynb">Notebook</a></p><h3>Storing</h3><p>Once you have data loaded and indexed, you will probably want to store it to avoid the time and cost of re-indexing it. By default, your indexed data is stored only in memory.</p><h4>Persisting to disk</h4><p>The simplest way to store your indexed data is to use the built-in .persist() method of every Index, which writes all the data to disk at the location specified. This works for any type of index.</p><p>Under the hood, LlamaIndex also supports swappable <strong>storage components</strong> that allows you to customize:</p><ul><li><strong>Document stores</strong>: where ingested documents (i.e., Node objects) are stored,</li><li><strong>Index stores</strong>: where index metadata are stored,</li><li><strong>Vector stores</strong>: where embedding vectors are stored.</li><li><strong>Graph stores</strong>: where knowledge graphs are stored (i.e. for KnowledgeGraphIndex).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/471/1*NQgqk99wiOLLwBkGIlhVlg.png" /></figure><h4>Using Vector Stores</h4><p>A VectorStorendex is by far the most frequent type of Index you’ll encounter. The VectorStorendex takes your Documents and splits them up into Nodes. It then creates vector embeddings of the text of every node, ready to be queried by an LLM. In this example, we’ll be using <a href="https://www.trychroma.com/">Chroma</a>, an open-source vector store.</p><pre># check if storage already exists<br>if not os.path.exists(&quot;./storage&quot;):<br>    # load the documents and create the index<br>    documents = SimpleDirectoryReader(&quot;data&quot;).load_data()<br>    vector_index = VectorStoreIndex.from_documents(documents)<br>    # store it for later<br>    vector_index.storage_context.persist()<br>else:<br>    # load the existing index<br>    storage_context = StorageContext.from_defaults(persist_dir=&quot;./storage&quot;)<br>    vector_index = load_index_from_storage(storage_context)</pre><h4>Using Chroma</h4><p>To use Chroma to store the embeddings from a VectorStorendex, you need to:</p><ul><li>initialize the Chroma client</li><li>create a Collection to store your data in Chroma</li><li>assign Chroma as the vector_store in a StorageContext</li><li>initialize your VectorStoreIndex using that StorageContext</li></ul><p>Here’s what that looks like:</p><pre>import chromadb<br>from llama_index import VectorStoreIndex, SimpleDirectoryReader<br>from llama_index.vector_stores import ChromaVectorStore<br>from llama_index.storage.storage_context import StorageContext<br><br># # To make ephemeral client that is a short lasting client or an in-memory client<br># db = chromadb.EphemeralClient()<br><br># initialize the Persistent client, setting path to save data<br>db = chromadb.PersistentClient(path=&quot;./chroma_db&quot;)<br><br># create collection<br>chroma_collection = db.get_or_create_collection(&quot;bank_earnings_database&quot;)<br><br># assign chroma as the vector_store to the context<br>vector_store = ChromaVectorStore(chroma_collection=chroma_collection)<br>storage_context = StorageContext.from_defaults(vector_store=vector_store)<br><br># create your index<br>vector_index = VectorStoreIndex.from_documents(<br>    documents,<br>    storage_context=storage_context,<br>    service_context=service_context<br>)</pre><p><strong><em>Reference</em></strong>: Colab <a href="https://github.com/akashmathur-2212/LLMs-playground/blob/main/LlamaIndex-applications/llama2-multi-documents-chatbot/chatbot.ipynb">Notebook</a></p><h3>Querying</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/496/1*gk_qw6RQJ8hrh_OgFsW0ZA.png" /><figcaption>Querying</figcaption></figure><h4>Query Engine</h4><p>Now you’ve loaded your data, built an index, and stored that index for later, you’re ready to get to the most significant part of an LLM application: <em>querying!</em></p><p>The most important thing to know about querying is that it is just a prompt to an LLM: so it can be a question and get an answer, or a request for summarization, or a much more complex instruction.</p><p>The basis of all querying is the QueryEngine. The simplest way to get a QueryEngine is to get your index to create one for you, like this:</p><pre># create a query engine and query<br>query_engine = vector_index.as_query_engine()<br>response = query_engine.query(&quot;Enter your query&quot;)<br>print(response)</pre><h4><strong>Stages of querying</strong></h4><p>However, there is more to querying than initially meets the eye. Querying consists of three distinct stages:</p><ul><li><strong>Retrieval</strong> is when you find and return the most relevant documents for your query from your Index. The most common type of retrieval is top_k semantic retrieval, but there are many other retrieval strategies.</li><li><strong>Postprocessing</strong> is when the Nodes retrieved are optionally reranked, transformed, or filtered, for instance by requiring that they have specific metadata such as keywords attached.</li><li><strong>Response synthesis</strong> is when your query, your most relevant data and your prompt are combined and sent to your LLM to return a response.</li></ul><p><strong>Node Postprocessors</strong></p><p>It also supports advanced Node filtering and augmentation that can further improve the relevancy of the retrieved Node objects. This can help reduce the time/number of LLM calls/costs or improve response quality. For example:</p><ul><li><strong>KeywordNodePostprocessor</strong>: filters nodes by required_keywords and exclude_keywords.</li><li><strong>SimilarityPostprocessor</strong>: filters nodes by setting a threshold on the similarity score (thus only supported by embedding-based retrievers)</li></ul><p>The full list of node postprocessors is documented in the <a href="https://docs.llamaindex.ai/en/stable/api_reference/node_postprocessor.html">Node Postprocessor Reference</a>.</p><p><strong>Response Synthesis</strong></p><p>It synthesizes a response given the retrieved Node. You can see how to <a href="https://docs.llamaindex.ai/en/stable/module_guides/deploying/query_engine/response_modes.html">specify different response modes</a>.</p><pre># configure retriever<br>retriever = VectorIndexRetriever(<br>    index=vector_index,<br>    similarity_top_k=6,<br>)<br><br># configure node postprocessors<br>s_processor = SimilarityPostprocessor(similarity_cutoff=0.79)<br>k_processor = KeywordNodePostprocessor(<br>    exclude_keywords=[&quot;environmental&quot;]<br>)<br><br># configure response synthesizer<br>response_synthesizer = get_response_synthesizer(service_context=service_context)<br><br>query_engine = RetrieverQueryEngine(<br>    retriever=retriever,<br>    node_postprocessors=[s_processor, k_processor],<br>    response_synthesizer=response_synthesizer<br>)</pre><p>Before we start querying, let’s understand quickly the <a href="https://docs.llamaindex.ai/en/stable/module_guides/deploying/query_engine/response_modes.html">Response Modes</a> Llamaindex supports:</p><pre># compact<br>query_engine = vector_index.as_query_engine(response_mode=&quot;compact&quot;)<br>response = query_engine.query(&quot;What is the Q2FY24 Net revenue for HDFC Bank?&quot;)<br>pprint_response(response, show_source=True)<br><br># refine<br>query_engine = vector_index.as_query_engine(response_mode=&quot;refine&quot;)<br>response = query_engine.query(&quot;What is the Q2FY24 Net revenue for HDFC Bank?&quot;)<br>pprint_response(response, show_source=True)<br><br># tree_summarize<br>query_engine = vector_index.as_query_engine(response_mode=&quot;tree_summarize&quot;)<br>response = query_engine.query(&quot;What is the Q2FY24 Net revenue for HDFC Bank?&quot;)<br>pprint_response(response, show_source=True)<br><br># no_text<br>&quot;&quot;&quot;<br>Only runs the retriever to fetch the nodes that would have been sent <br>to the LLM, without actually sending them. <br>&quot;&quot;&quot;<br>query_engine = vector_index.as_query_engine(response_mode=&quot;no_text&quot;)<br>print(response.source_nodes)</pre><h4>Chat Engine</h4><p>So far you have only been working with the query engine. Llama index also supports a chat engine. So how do they differ?</p><blockquote>A chat engine is a high-level interface for having a conversation with your data. A chat is a multiple back and forth instead of a single question and answer with your data.</blockquote><p>By <em>keeping track of the conversation history</em>, it can answer questions with the past context in mind.</p><p>Chat engine has several <a href="https://docs.llamaindex.ai/en/stable/module_guides/deploying/chat_engines/root.html#modules">chat modes</a>. We will use the “<strong>condense question</strong>” mode.</p><p>The chat history is analyzed and the user message is rewritten to be a query for the index. The response is returned after reading the response from the query engine. So let’s use the condensed question to see how it will remember the history of the chat.</p><pre>chat_engine = vector_index.as_chat_engine(chat_mode=&quot;condense_question&quot;)<br>response = chat_engine.chat(&quot;Can you provide important highlights of ICICI Bank&#39;s earnings report?&quot;)<br>print(response)<br><br>chat_engine.reset()<br></pre><p><strong><em>Reference</em></strong>: Colab <a href="https://github.com/akashmathur-2212/LLMs-playground/blob/main/LlamaIndex-applications/llama2-multi-documents-chatbot/chatbot.ipynb">Notebook</a></p><h3>Putting It All Together Into a Streamlit App‍🚀</h3><p>You’ve loaded your data, indexed it, stored your index, and queried your index. Now we will build and deploy a custom web app using a simple interface using <a href="https://streamlit.io/">Streamlit</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cxiOdPkANmF0_IavCJkm0A.png" /><figcaption>Streamlit UI</figcaption></figure><p>Refer to the streamlit code on Github:</p><p><a href="https://github.com/akashmathur-2212/LLMs-playground/blob/main/LlamaIndex-applications/llama2-multi-documents-chatbot/app.py">LLMs-playground/LlamaIndex-applications/llama2-multi-documents-chatbot/app.py at main · akashmathur-2212/LLMs-playground</a></p><h3>🌟Bonus! 🌟</h3><p>As you have come this far, let me share a couple of things:</p><ol><li>If you want to use <strong>LangChain </strong>to build a similar Chatbot. Refer to this repo:</li></ol><p><a href="https://github.com/akashmathur-2212/LLMs-playground/tree/main/LangChain-applications/mistral_7B-multiPDF-chatbot">LLMs-playground/LangChain-applications/mistral_7B-multiPDF-chatbot at main · akashmathur-2212/LLMs-playground</a></p><p>2. If you want to use <strong>LangChain </strong>to chat with <strong>CSV</strong>, Refer to this repo:</p><p><a href="https://github.com/akashmathur-2212/LLMs-playground/tree/main/LangChain-applications/llama2-chat-with-CSV">LLMs-playground/LangChain-applications/llama2-chat-with-CSV at main · akashmathur-2212/LLMs-playground</a></p><p>3. Let me also show how we can use LLMs to automate metadata extraction to improve our retrieval process!</p><h3>Automated Metadata Extraction for Better Retrieval</h3><p>Automated Metadata Extraction in LlamaIndex refers to the process of automatically identifying and extracting key information from documents to create descriptive metadata. This metadata serves as tags or labels that help categorize and organize documents, enabling more efficient indexing and retrieval.</p><p>We can use LLMs to automate metadata extraction with Metadata Extractor modules-</p><ol><li><strong>SummaryExtractor</strong> — automatically extracts a summary over a set of Nodes</li><li><strong>QuestionsAnsweredExtractor</strong> — extracts a set of questions that each Node can answer</li><li><strong>TitleExtractor</strong> — extracts a title over the context of each Node</li><li><strong>EntityExtractor</strong> — extracts entities (i.e. names of places, people, things) mentioned in the content of each Node</li></ol><p>Then you can chain the Metadata Extractors with the node parser. Refer to the notebook below:</p><p><a href="https://github.com/akashmathur-2212/LLMs-playground/blob/main/LlamaIndex-applications/llama2-multi-documents-chatbot/chatbot.ipynb">LLMs-playground/LlamaIndex-applications/llama2-multi-documents-chatbot/chatbot.ipynb at main · akashmathur-2212/LLMs-playground</a></p><p>Congratulations on building your first LLM-powered Chatbot! 🎉👏🎊</p><p>If you enjoyed reading this article comment “Hell Yes!” in the comment section and let me know if any feedback.</p><p>You’re welcome to take a look at the repo and star⭐it.</p><p><a href="https://github.com/akashmathur-2212/LLMs-playground/tree/main/LlamaIndex-applications/llama2-multi-documents-chatbot">LLMs-playground/LlamaIndex-applications/llama2-multi-documents-chatbot at main · akashmathur-2212/LLMs-playground</a></p><p>Feel free to follow me on <a href="https://akash-mathur.medium.com/">Medium</a>, and <a href="https://github.com/akashmathur-2212">GitHub</a>, or say Hi on <a href="https://www.linkedin.com/in/akashmathur22/">LinkedIn</a>. I am excited to discuss across AI, ML, NLP, and MLOps areas!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=76bbd041eafc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Big Data Processing Using Distributed Maps and AWS Step Functions]]></title>
            <link>https://aws.plainenglish.io/big-data-processing-using-distributed-maps-and-aws-step-functions-2cfa6128039d?source=rss-eb7bcac979b6------2</link>
            <guid isPermaLink="false">https://medium.com/p/2cfa6128039d</guid>
            <category><![CDATA[parallel-processing]]></category>
            <category><![CDATA[aws-step-functions]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <dc:creator><![CDATA[Akash Mathur]]></dc:creator>
            <pubDate>Sat, 11 Nov 2023 11:20:54 GMT</pubDate>
            <atom:updated>2023-11-13T01:40:17.588Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*CBXrOxpGzKV9o2j7.jpg" /></figure><p>Navigating large datasets often involves the challenge of parallelizing tasks for individual elements within the dataset. While Apache Spark is a common industry solution for such scenarios, there are situations where its use may not be feasible or optimal as incorporating Spark introduces complexities such as addressing concerns within User Defined Functions (UDF).</p><p>In such cases, a compelling option to explore is using <strong>AWS Step Function — Distributed Map</strong>. When dealing with large datasets or performing repetitive tasks on multiple items, orchestrating and parallelizing workflows becomes crucial.</p><h3>Problem Statement</h3><p>I have to transform a large-scale dataset and the transformation involves applying a complex function to each element in a specific column. The current approach using df[column].apply() is sequential and struggles to scale efficiently with large datasets.</p><h3>Solution</h3><p><strong>Exciting News</strong>: AWS Step Functions now introduces the <em>Distributed Map</em>, unlocking even more possibilities for efficiently handling large-scale parallel workloads.</p><p>The standard map state in Step Functions, while powerful, was limited to handling 40 parallel iterations at a time. Scaling up to process thousands or more items in parallel required intricate workarounds.</p><p>Enter the Distributed Map state! It is designed to seamlessly coordinate expansive parallel workloads in your serverless applications. Whether iterating through millions of logs, images, or .csv files stored in Amazon S3, the Distributed Map state can now launch up to ten thousand parallel workflows for robust data processing.</p><p>You have the flexibility to process data using any service API supported by Step Functions, but the common and efficient choice is invoking Lambda functions and in this example also, we will write our data processing logic inside the lambda.</p><p>In this article, I’ll share an illustrative example of a post-processing task where I applied operations to each individual row item. Specifically, I worked with textual data present in each row of a CSV file. The approach involved invoking a Lambda function asynchronously, serving as the post-processing function, across every row item within the CSV file. The mechanism facilitating this is the distributed map, effectively iterating over a CSV file residing in an S3 bucket.</p><h3>Understanding Distributed Maps</h3><p>Before we step ahead in actual implementation, let’s understand a bit about <em>Distributed Maps.</em> It allows you to apply the same operation to each element in a collection concurrently. This is particularly useful when dealing with tasks like processing data rows in a CSV file, analyzing images in a batch, or invoking a Lambda function for multiple inputs simultaneously.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*mWOcr5Vaw172zY-G.png" /><figcaption>How does a Map function work (in general)?</figcaption></figure><p>Let’s dive into the key components and steps involved in utilizing Distributed Maps with AWS Step Functions.</p><h3>Key Components:</h3><h4>1. State Machine</h4><p>A State Machine in AWS Step Functions is a visual representation of a workflow. It consists of states that define the different steps or actions to be taken. In the context of Distributed Maps, the state machine includes the Map state.</p><h4>2. Map State</h4><p>The Map state is the star of the show when it comes to distributed operations. It enables parallel execution of a set of operations on each element in an array, making it efficient for processing large datasets.</p><h4>3. Input Data</h4><p>The input data is the collection or array on which the Map state operates. In the case of processing a CSV file, each row would be an element in the array.</p><h4>4. Task States</h4><p>Task states represent individual operations or tasks that need to be performed on each element. For example, invoking a Lambda function or processing data.</p><h3>Steps to Implement Distributed Maps:</h3><p>1. <strong>Lambda Function</strong>: Firstly, create a Lambda function that performs the post-processing on an individual row. Ensure this Lambda function accepts a row as input and returns the processed result.</p><pre># Lambda Function: post-pocessing Function<br>import json<br><br>def lambda_handler(event, context):<br>    # Assuming event contains the row data<br>    row_data = event[&#39;column_name&#39;]<br>    <br>    # Perform your post-processing operations on row_data<br>    cleaned_text = re.sub(r&#39;[^A-Za-z\s]&#39;, &#39;&#39;, row_data)<br><br>    # Remove extra whitespaces<br>    cleaned_text = re.sub(r&#39;\s+&#39;, &#39; &#39;, cleaned_text).strip()<br>    <br>    processed_result = f&quot;Processed data: {cleaned_text}&quot;<br><br>    return {<br>        &#39;statusCode&#39;: 200,<br>        &#39;body&#39;: json.dumps(processed_result)<br>    }</pre><p>2. <strong>AWS Step Functions State Machine Definition</strong>: Now, create an AWS Step Functions state machine to orchestrate the distributed map.</p><p><strong>a)</strong> Navigate to the Step Functions page and select Create state machine.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*aqM7JEoRMGSTKDJQ.png" /></figure><p><strong>b)</strong> In the visual editor, I search and select the Map component on the left-side pane, and I drag it to the workflow area. On the right side, I configure the component. I chose Distributed as the Processing mode and Amazon S3 as the Item Source.</p><p>Distributed maps are natively integrated with S3. I entered the name of the bucket and the prefix path where the CSV file is stored.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GVqxS3negxQZ-7qoQ7LoRQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y6kQgWZrdbPt5p5opmZoMA.png" /></figure><p>On the Runtime Settings section, I choose <em>Standard</em> for Child workflow type. I also may decide to restrict the <em>Concurrency </em>limit. It helps to ensure we operate within the concurrency quotas of the downstream services (Lambda) for a particular account or Region.</p><p>By default, the output of my sub-workflows will be aggregated as state output, up to 256KB. To process larger outputs, I may choose to Export map state results to Amazon S3.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*34ON3Cpj_xgK5ype5mSLPw.png" /></figure><p><strong>c)</strong> Finally, I define what post-processing task to do for each row item. In this demo, I want to invoke a Lambda function for each file in the S3 bucket. The function exists already (Step-1).</p><p>I search for and select the Lambda invocation action on the left-side pane. I drag it to the distributed map component. Then, I use the right-side configuration panel to select the actual Lambda function to invoke: post-processing-function:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kpSppHGyQ8W0kf8hskuJzw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_hF2aFULCd2vFnwrurt0Sw.png" /></figure><p><strong>d) </strong>Select Next. I select Next again on the Review generated code page (not shown here).</p><p>On the Specify state machine settings page, I enter a Name for my state machine and the IAM Permissions to run. Then, I select Create state machine.</p><p><strong>e)</strong> Now I am ready to start the execution. On the State machine page, I select the new workflow and select Start execution. I can optionally enter a JSON document to pass to the workflow. In this demo, the workflow does not handle the input data as it’s provided by the input state. I leave it as-is, and I select Start execution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GFzuzcEo9H-J1XBK4tQQCg.png" /></figure><p><strong>Final State Machine and its Definition</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/372/1*WrfKX1hrSMPadKQ1h9Jjag.png" /><figcaption>State Machine</figcaption></figure><pre>{<br>  &quot;Comment&quot;: &quot;A description of my state machine&quot;,<br>  &quot;StartAt&quot;: &quot;Text-File-Processing&quot;,<br>  &quot;States&quot;: {<br>    &quot;Text-File-Processing&quot;: {<br>      &quot;Type&quot;: &quot;Map&quot;,<br>      &quot;ItemProcessor&quot;: {<br>        &quot;ProcessorConfig&quot;: {<br>          &quot;Mode&quot;: &quot;DISTRIBUTED&quot;,<br>          &quot;ExecutionType&quot;: &quot;STANDARD&quot;<br>        },<br>        &quot;StartAt&quot;: &quot;Post-Processing Lambda&quot;,<br>        &quot;States&quot;: {<br>          &quot;Post-Processing Lambda&quot;: {<br>            &quot;Type&quot;: &quot;Task&quot;,<br>            &quot;Resource&quot;: &quot;arn:aws:states:::lambda:invoke&quot;,<br>            &quot;OutputPath&quot;: &quot;$.Payload&quot;,<br>            &quot;Parameters&quot;: {<br>              &quot;FunctionName&quot;: &quot;arn:aws:lambda:us-east-1:XXXXXX:function:step-function-name:$LATEST&quot;,<br>              &quot;Payload.$&quot;: &quot;$&quot;<br>            },<br>            &quot;Retry&quot;: [<br>              {<br>                &quot;ErrorEquals&quot;: [<br>                  &quot;Lambda.ServiceException&quot;,<br>                  &quot;Lambda.AWSLambdaException&quot;,<br>                  &quot;Lambda.SdkClientException&quot;,<br>                  &quot;Lambda.TooManyRequestsException&quot;<br>                ],<br>                &quot;IntervalSeconds&quot;: 1,<br>                &quot;MaxAttempts&quot;: 3,<br>                &quot;BackoffRate&quot;: 2<br>              }<br>            ],<br>            &quot;End&quot;: true<br>          }<br>        }<br>      },<br>      &quot;ItemReader&quot;: {<br>        &quot;Resource&quot;: &quot;arn:aws:states:::s3:getObject&quot;,<br>        &quot;ReaderConfig&quot;: {<br>          &quot;InputType&quot;: &quot;CSV&quot;,<br>          &quot;CSVHeaderLocation&quot;: &quot;FIRST_ROW&quot;<br>        },<br>        &quot;Parameters&quot;: {<br>          &quot;Bucket&quot;: &quot;bucke-name&quot;,<br>          &quot;Key&quot;: &quot;config.csv&quot;<br>        }<br>      },<br>      &quot;Label&quot;: &quot;Text-File-Processing&quot;,<br>      &quot;End&quot;: true<br>    }<br>  }<br>}</pre><p>“‘Voila!’ You’ve just witnessed a glimpse of parallel execution, accelerating data transformation. Now, let’s explore a bit more as we continue with this article.</p><p>The end-to-end code for this module is saved to my <em>GitHub </em><a href="https://github.com/akashmathur-2212/aws-serverless-workflows/tree/main/Step-Function-Distributed-Map"><em>repository</em></a><em>.</em></p><h3>Use Cases of Distributed Maps with AWS Step Functions</h3><ol><li><strong>Data Processing Pipelines</strong>: Distribute processing tasks across a large dataset, allowing parallel execution for faster data transformation.</li><li><strong>Batch Image/Video Processing</strong>: Process and analyze a collection of images or videos concurrently, benefiting applications like computer vision.</li><li><strong>Log Analysis</strong>: Analyze logs from distributed systems concurrently, identifying patterns or anomalies in real time.</li><li><strong>Parallel API Invocations</strong>: Invoke external APIs concurrently for tasks like geocoding, weather information retrieval, or language translation.</li><li><strong>Database Record Processing</strong>: Process and update records in a database concurrently, improving throughput for large-scale operations.</li></ol><p>Overall, Distributed Maps are a powerful tool for orchestrating large-scale parallel workloads, but they should be used with care and awareness of the potential disadvantages.</p><h3><strong>Disadvantages</strong></h3><ol><li><strong>Limited input and output sizes</strong>: Distributed Maps have a limit of 100 million items for input and 10GB for output. This may be a limitation for some workloads.</li><li><strong>Limited downstream service capabilities:</strong> The downstream service that you use to process the results of a Distributed Map must be able to handle the input and output sizes. For example, Lambda has a limit of 256MB for input and 1024MB for output.</li></ol><p>In addition to these disadvantages, it is important to note that Distributed Maps are still a relatively new feature in Step Functions, and there is less documentation and community support for them than for other Step Functions states.</p><p>Here are some tips for mitigating the disadvantages of Distributed Maps:</p><ul><li><strong>Use Distributed Maps judiciously</strong>: Only use Distributed Maps for workloads that truly require large-scale parallelism. For smaller workloads, consider using other Step Function states, such as Task or Parallel.</li><li><strong>Design your workflow for failure</strong>: Handle failure cases and retries gracefully. You may want to consider using a state machine pattern such as the retry loop pattern.</li><li><strong>Optimize your workflow for performance:</strong> Consider using features such as the tolerated failure threshold to reduce the performance impact of failures.</li><li><strong>Estimate the cost of your workflow:</strong> Use the Step Functions pricing calculator to estimate the cost of running your workflow with Distributed Maps.</li><li><strong>Choose the right downstream service</strong>: Make sure that the downstream service that you use to process the results of a Distributed Map can handle the input and output sizes.</li></ul><p>In conclusion, <em>Distributed Maps</em> introduce a powerful paradigm for orchestrating large-scale parallel workloads with unparalleled ease and efficiency. Whether you are processing extensive datasets, analyzing logs, or orchestrating complex workflows, the distributed map state provides a scalable and cost-effective solution.</p><p>The ability to seamlessly scale up to 10,000 parallel executions empowers developers to tackle data-intensive tasks with confidence. Leveraging serverless architecture, this feature optimizes costs by ensuring resources are consumed only during execution, aligning with the pay-as-you-go model.</p><p>While the advantages of increased throughput and simplified orchestration are evident, it’s crucial to consider concurrency limits, cold start overhead, and potential complexities in dependency management for certain use cases. Careful design and testing are essential to harness the full potential of Distributed Maps with AWS Step Functions.</p><p>Congratulations! 🎉👏🎊</p><p>You now understand how to leverage the Distributed Map state and orchestrate large-scale parallel workloads for efficient data processing at an unprecedented scale.</p><p>If you enjoyed reading this article comment “Hell Yes!” in the comment section and let me know if any feedback.</p><p>Feel free to follow me on <a href="https://akash-mathur.medium.com/">Medium</a>, and <a href="https://github.com/akashmathur-2212">GitHub</a>, or say Hi on <a href="https://www.linkedin.com/in/akashmathur22/">LinkedIn</a>. I am excited to discuss across AI, ML, NLP, and MLOps areas!</p><h3><a href="https://plainenglish.io/">PlainEnglish.io</a> 🚀</h3><p><em>Thank you for being a part of the In Plain English community! Before you go:</em></p><ul><li><em>Be sure to </em><strong><em>clap</em></strong><em> and </em><strong><em>follow</em></strong><em> the writer</em></li><li><em>Learn how you can also </em><a href="https://plainenglish.io/blog/how-to-write-for-in-plain-english"><strong><em>write for In Plain English</em></strong></a><em>️</em></li><li><em>Follow us: </em><a href="https://twitter.com/inPlainEngHQ"><strong><em>X</em></strong></a><strong><em> | </em></strong><a href="https://www.linkedin.com/company/inplainenglish/"><strong><em>LinkedIn</em></strong></a><strong><em> | </em></strong><a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong><em>YouTube</em></strong></a><strong><em> | </em></strong><a href="https://discord.gg/in-plain-english-709094664682340443"><strong><em>Discord</em></strong></a><strong><em> | </em></strong><a href="https://newsletter.plainenglish.io/"><strong><em>Newsletter</em></strong></a></li><li><em>Visit our other platforms: </em><a href="https://stackademic.com/"><strong><em>Stackademic</em></strong></a><strong><em> | </em></strong><a href="https://cofeed.app/"><strong><em>CoFeed</em></strong></a><strong><em> | </em></strong><a href="https://venturemagazine.net/"><strong><em>Venture</em></strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2cfa6128039d" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/big-data-processing-using-distributed-maps-and-aws-step-functions-2cfa6128039d">Big Data Processing Using Distributed Maps and AWS Step Functions</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>