Introduction

In the contemporary landscape of artificial intelligence, the need for systems capable of efficiently processing and interpreting diverse data types is paramount. Multi-modal Retrieval-Augmented Generation (RAG) is a pioneering approach that addresses this need by integrating the strengths of retrieval and generative models to handle text, tables, and images. This system enhances the capability to extract meaningful insights from complex documents, such as PDFs, which often contain a mixture of textual, tabular, and visual information.

This report provides an extensive overview of multi-modal RAG systems, focusing on their architecture, capabilities, and implementation. We will delve into the specifics of a Python-based multi-modal RAG system designed to summarize and process diverse data types and discuss its potential applications, benefits, and challenges.

Understanding Multi-Modal RAG Systems

What is RAG?

Retrieval-augmented generation (RAG) is an AI (Artificial Intelligence) framework that combines retrieval mechanisms with generative models to produce more accurate and contextually relevant outputs. In a typical RAG setup, a retrieval component searches for relevant information from a large corpus of documents or data, which a generative model then uses to produce responses or summaries. This combination allows for more informed and context-aware generation, improving the output quality.

Multi-Modal RAG

Multi-modal RAG extends this concept by incorporating multiple data modalities, such as text, images, and tables. The ability to process and integrate information from distinct data sources makes multi-modal RAG systems highly versatile and applicable to various use cases. These systems can understand and generate outputs considering the full spectrum of available information, leading to more comprehensive and accurate results.

Critical Components of a Multi-Modal RAG System

Data Ingestion and Processing: The system needs to handle various data types, extracting and preprocessing information from text, tables, and images.

Retrieval Mechanism: This component searches for relevant data from a corpus or database, using embeddings and other search algorithms to find the most pertinent information.

Generative Model: A large language model (LLM) like GPT-4o, Gemini-vision pro generates summaries, answers, or other outputs based on the retrieved information.

Integration and Output: The retrieved data and generative outputs are integrated to provide a coherent and contextually appropriate response.

System Architecture

The system uses Python and integrates various libraries and tools to handle different data modalities. Key components include:

Large Language Models (LLMs): Two LLMs are used — gemini-1.0-pro-vision-001 and GPT-4 omni.

Embeddings: Used to represent text, tables, and images in a vector space for efficient retrieval.

PostgreSQL: A database system for storing and managing tabular data.

Milvus: A vector database used for storing and retrieving high-dimensional vectors.

Implementation

The implementation revolves around a multi-modal retrieval-augmented generation (RAG) system that processes and summarizes data from PDF files, including text, tables, and images.

The Multimodal class is designed to handle multiple data modalities, including text, tables, and images. It integrates with a large language model (LLM) for vision and text processing and provides various utility functions to summarize and process different data types.

PDF Extraction

To extract and classify distinct types of content from PDF files, facilitating efficient data processing and summarization. Here is a concise overview:

The file processing module is to dissect a PDF document into its constituent elements, including text, tables, and images, to facilitate further analysis and processing.

Its key features encompass taking a PDF file path and extracting various elements using a chunking strategy, wherein the document is divided into manageable chunks based on headings or titles. The module extracts and stores images for subsequent processing if image extraction is enabled.

Moreover, it identifies and extracts tables while converting them into a structured format. The module also incorporates character limits, such as max_characters, to control the chunk size, ensuring efficient data processing. Ultimately, the output of this module is a categorized list of elements, including texts, tables, and images, extracted from the PDF document, ready for further analysis and manipulation.

Summarization

The implementation includes distinct summarization processes for text, tables, and images, each tailored to its data type. Text Summarization returns the input texts as summaries, serving as a direct passthrough. Table Summarization employs a templated prompt to query with GPT-4o, which summarizes the tables, and this process allows for batch processing with a set concurrency limit to manage multiple tables efficiently. Image Summarization involves encoding images in base64 format, using Gemini-vision-pro to generate detailed descriptions, and returning both the encoded image and its summary, thus providing a comprehensive approach to handling and interpreting visual data.

Storing Data in Milvus

Preparation for data insertion involves processing and converting summarized elements (texts, tables, images) into suitable formats. Each component is encapsulated in a Document object with relevant metadata. Connecting to Milvus involves establishing a server connection by providing details such as host, port, and collection name derived from the selected file. The storage process entails inserting these documents into the Milvus vector database, where embeddings for each document are computed and stored. This enables efficient similarity search and retrieval, optimizing the system’s ability to fetch relevant information based on user queries.

Storing Tabular data in PostgreSQL

Adding table data to PostgreSQL involves converting the table into a DataFrame, mapping its data types to SQL-compatible types, and generating an SQL query to create the table in the database. The data is then inserted row-by-row, and the transaction is committed to ensure persistence. Synchronizing with PostgreSQL entails pushing the table data to the database, retrieving it, and displaying it in a DataFrame format.

Using Multi-Vector-Retriever

A MultiVectorRetriever object is created with the following components:

• vectorstore: Reference to the Milvus vector database containing the stored embeddings.
• docstore: An in-memory document store to hold the original documents for quick access.
• id_key: A unique identifier key for each document.

Generating Responses

When handling queries, the system employs a tailored workflow depending on whether the query relates to text, tables, or images.

For text-related queries, it begins by searching for semantically matching text summaries stored as documents in Milvus. The relevant text documents are fetched based on their similarity to the query, and the results are displayed using Streamlit’s text display functions. The language model then generates a response, shown below the text, explaining based on the retrieved data.

For table-related queries, the process involves searching Milvus for relevant table summaries. If a matching summary is found, the corresponding full table data is fetched from PostgreSQL, ensuring the response is based on actual data. The results, including text and table data, are displayed in Streamlit, with the language model’s response explaining the query results in context.

For image-related queries, the retriever searches Milvus for image summaries. The relevant summaries are fetched, and the images are displayed using Streamlit’s image display, which shows decoded base64 images. The vision model then generates a detailed description of the image, which is displayed below the image, explaining the query results based on the visual data.

Conclusion

The implementation of the multi-modal retrieval-augmented generation (RAG) system efficiently processes and responds to diverse queries related to text, tables, and images from PDF documents. The system ensures accurate and contextually relevant responses by leveraging advanced search capabilities in Milvus for semantic matching and integrating powerful language and vision models such as GPT-4o and Gemini-vision pro. The system retrieves and presents relevant summaries for text-related queries, supplemented by detailed explanations from a language model. Table-related queries benefit from a robust mechanism that fetches summaries from Milvus and synchronizes with PostgreSQL to utilize actual data, providing comprehensive insights. Image-related queries are handled through encoding, retrieval, and detailed description generation, ensuring users receive thorough explanations of visual content. This multi-modal approach enhances the user experience by offering tailored responses that cater to different data types, demonstrating the system’s versatility and effectiveness. Overall, the implementation highlights a seamless integration of retrieval and generation technologies, providing an efficient tool for extracting and summarizing information from complex documents.

About LangGraph

LangGraph is a programming library designed for graph-based processing and analysis of natural language. It provides tools for constructing, manipulating, and querying graphs where nodes represent linguistic elements (words, phrases, or sentences) and edges capture various relationships between these elements (such as syntactic dependencies, semantic similarities, or co-occurrences). By leveraging graph theory, LangGraph facilitates advanced NLP tasks, including text summarization, semantic search, information extraction, and knowledge representation. Its flexible and efficient framework supports integration with other NLP libraries and can handle complex language models, making it a valuable resource for researchers and developers in computational linguistics.

About Data Visualization using LangGraph

Data visualization using LangGraph involves orchestrating a multi-agent system to analyze data and create visual representations efficiently. LangGraph dynamically routes tasks to specialized agents for data analysis and chart generation by employing components like prompt templates and AI models. This approach ensures seamless data processing and visualization integration, leveraging LangGraph’s capabilities to manage complex workflows and produce insightful visual data representations.

Functionality and Features

LangGraph is a powerful library for advanced NLP using graph-based methods. Its key features include:

1. Graph Construction:- Create graphs from text data with nodes for words, phrases, or sentences and edges for relationships.
2. Graph Manipulation: Modify graphs by adding, removing, merging, or splitting nodes and edges.
3. Graph Querying: Perform complex queries and graph traversal to extract patterns.
4. NLP Integration: Integrate with NLP tools and support machine learning models for tasks like named entity recognition.
5. Visualization: Visualize graphs with customizable options to understand text structure.
6. Analysis: Use centrality measures and graph metrics for text analysis.
7. Applications: Text summarization, semantic search, information extraction, and knowledge representation.

Procedures for Utilizing LangGraph

1. Models Utilized

• Large Language Model(LLM): Azure Open AI GPT-4o

2. Tools Used

This function connects to a specified database and extracts information about a particular table, saving its contents as a CSV file. It verifies the existence of the schema and table, then retrieves the table’s data into a pandas DataFrame. The function outputs the first five rows, table dimensions, column data types, memory usage, number of unique values per column, and statistical data descriptions. If any errors occur, it returns an appropriate message.

This code snippet defines a tool for executing Python code within a REPL (Read-Eval-Print Loop) environment. The ‘python_repl’ function takes a string of Python code as input, runs it, and returns the output. If an error occurs during execution, it captures and returns the error message. Upon successful execution, it provides a formatted response, including the executed code and its output. This tool is helpful for dynamically running and debugging Python code snippets, making it easier to visualize results and perform on-the-fly computations.

3. Node Description

• Node 1(data_analyser_node):- In this snippet, `data_analyser_agent` is created using the `create_agent` method from `utils,` with `get_table_info` as its tool and `data_analyser_prompt` as the system prompt. `data_analyser_node` is defined using `functools.partial` to create a reusable function for this agent, specifying it as the “data_analyser” node. This node, referred to as node 1, is responsible for analyzing table data by invoking the `get_table_info` tool within the agent framework.

• Node 2 (chart_generator_node):- In this snippet, a `chart_generator_agent` is created using the `create_chain_runnable` method from `utils,` with `chart_generator_prompt` as the system prompt. This agent is designed to generate charts based on the instructions provided. The `chart_generator_node` is defined using `functools.partial` to create a reusable function, specifying the `chart_generator_agent` as the chain and naming it “chart_generator.” This node will process the given state and generate charts, integrating with the rest of the workflow to visualize data as required.

4. Final Multi-Agent graph structure

This graph illustrates a multi-agent workflow that begins with a `ChatPromptTemplate,` which sets up the prompt for the conversation. It then utilizes the `AzureChatOpenAI` model to process the input, followed by the `JsonOutputFunctionsParser` to parse the AI’s output and determine the next steps. ChatPrompt, AzureChat, and JsonOutputFunction comprise the LLM Chain that makes up the supervisor agent. The workflow routes tasks to either the `data_analyser` node, which analyzes data using the `data_analyser_agent`, or the `chart_generator` node, which generates charts using the `chart_generator_agent`. The process continues routing tasks until the `FINISH` condition is met, concluding at the `__end__` node. This setup demonstrates how an AI-driven supervisor manages the flow between data analysis and chart generation.

5. Results

Conclusion

The provided code leverages `LangGraph` for multi-agent workflows and integrates with Datwhiz to visualize tables. The `GraphUtils` class orchestrates the creation of agents and chains, enabling them to process tasks and communicate results efficiently. By compiling a state graph with a supervisor node to manage task routing, this setup ensures that agents can dynamically report and delegate work. This combination enhances workflow management and table visualization, streamlining complex processes within any ecosystem.

Problem Statement

We aim to compare the effectiveness of the LlamaIndex and LangChain approaches in building knowledge graphs. This involves examining how well each method organizes information and helps us find answers to questions. By comparing LlamaIndex and LangChain, we want to understand which is better at creating useful knowledge graphs.

Steps Involve for LlamaIndex

1.LLM Model and Embedding Model Used

• LLM model — anthropic.claude-v2
• Embedding model — titan-embed

2. Reading PDF Files Using LlamaIndex and Converting Them to Document Type

• PDFReader extracts text from each page of PDF and converts it to document type.
• Only document-type chunks can be passed to the knowledge graph generator `KnowledgeGraphIndex.`
• Following open-domain sustainability reports were used during the experimentation.
• ‘Pepsico_AnnualReport_2021’
• ‘Pepsico_SustainabilityReport_2022’
• ‘Pepsico_SustainabilityReport_2020’
• ‘Pepsico_SustainabilityReport_2021’
• ‘Nestle_SustainabilityReport_2022’
• ‘KimberlyClark_AnnualReport_2020’

3. Changing the Default LLM Model and Embedding Model

• Changing the LLM model and embedding the model to ours because LlamaIndex uses OpenAI GPT models by default.
• Added custom chunk_size (number of characters into text chunk) and chunk_overlap (how many common characters are between two chunks) to ensure consistency for both LlamaIndex and LangChain. The same chunk creator was used for both frameworks.

4. Creating the Knowledge Graph for the Documents Using Llamaindex

KnowledgeGraphIndex in LlamaIndex works by extracting knowledge triplets from unstructured text and organizing them into a graph structure that can be queried. Here’s a detailed explanation of how it works:

• Document Loading: The first step involves loading the documents from which the knowledge graph will be built.
• LLM Definition: The next step is to define the Language Model (LLM) because it uses the LLM model to create a knowledge graph.
• Knowledge Graph Index Creation: The KnowledgeGraphIndex is created from the documents. This takes in the documents, max_triplets_per_chunk, and the service context (in which our llm model and embedding model are defined) as arguments. During this process, the LLM extracts knowledge triplets from the text. These triplets are then organized into a graph structure.

5. Ingesting Graphs into Opensearch Vector Database

OpenSearch Vector Database is a specialized storage system for vector embeddings, optimized for similarity searches. It efficiently organizes and retrieves data based on the similarity of their vector representations.

• Setting Up Variables: Setting up variables that will be used to initialize the OpensearchVectorClient. These include the endpoint (the URL of the Opensearch server), the index (the name of the index in Opensearch in which data will be stored), the dimension of the vectors (dimension of embedding for each word), the text field (the name of the field in the index that contains the text data), the embedding field (the name of the field in the index that includes the vector embeddings), and the username and password for authentication.
• Initializing the OpensearchVectorClient: The OpensearchVectorClient is initialized with the previously set variables. This client is used to interact with the OpenSearch server.
• Initializing the OpensearchVectorStore: The OpensearchVectorStore is initialized with the OpensearchVectorClient. The OpensearchVectorStore is a vector store that uses OpenSearch as its backend. It is used to store and retrieve vector embeddings.
• Setting Up the StorageContext: The StorageContext is initialized with the OpensearchVectorStore as its vector store. The StorageContext in LlamaIndex is a container that holds the document, index, and vector stores. In this case, the vector store is set to the OpensearchVectorStore, and the document store and index store are set to their default values.
• Ingesting Data Using VectorStoreIndex: A VectorStoreIndex is a type of index in LlamaIndex that uses a vector store for storing and retrieving vector embeddings. VectorStoreIndex requires the documents(representing the list of documents to be stored), the storage_context(in which vector store is defined), and service_context(service context contains the services that are used by the index, such as the LLM model, embedding model, and chunk creator) as input parameters.

6. Creating OpensearchVectorClient Instance for Index Name in Which Data is Stored

• Setting Up variables: setting up variables that will be used to initialize the OpensearchVectorClient. These include the endpoint (the URL of the Opensearch server), the index (the name of the index in Opensearch in which data is stored), the dimension of the vectors (dimension of embedding for each word), the text field (the name of the field in the index that contains the text data), the embedding field (the name of the field in the index that includes the vector embeddings), and the username and password for authentication.
• Initializing the OpensearchVectorClient: The OpensearchVectorClient is initialized with the previously set variables. This client is used to interact with the OpenSearch server.
• Initializing the OpensearchVectorStore: The OpensearchVectorStore is initialized with the OpensearchVectorClient. The OpensearchVectorStore is a vector store that uses OpenSearch as its backend. It is used to store and retrieve vector embeddings.
• VectorStoreIndex.from_vector_store: creates index object taking vector_store as input, which is initialized with the index name in which data is stored.

7. Creating Q&A Query Engine with Custom Prompt Using LlamaIndex

llama_index_qa function will take the question as input and retrieve graph nodes and edges from the vector store according to the question. Then, these retrieved graph nodes and edges will be passed to the prompt as context, and then it asks LLM to generate an answer for the question by passing a modified prompt as input to LLM.

Steps Involve for LangChain

1. LLM Model and Embedding Model Used

• LLM model — anthropic.claude-v2
• Embedding model — titan-embed

2. Reading PDF Files Using Langchain and Converting Them to Documents Type

• PyPDFLoader extracts and converts text from each PDF page into a document type.
• Open-domain sustainability reports were used during the experiment.
• ‘Pepsico_AnnualReport_2021’
• ‘Pepsico_SustainabilityReport_2022’
• ‘Pepsico_SustainabilityReport_2020’
• ‘Pepsico_SustainabilityReport_2021’
• ‘Nestle_SustainabilityReport_2022’
• ‘KimberlyClark_AnnualReport_2020’

3. Converting Loaded Documents into Chunks

• Using RecursiveCharacterTextSplitter, same as llama-index, with the same chunk_size and chunk_overlap for converting documents into chunks.
• The RecursiveCharacterTextSplitter is designed to split text into smaller, semantically meaningful chunks and then combine them into larger ones until they reach a specific size, with some overlap to maintain context between chunks. The number of characters measures the chunk size, and the chunk overlap is the maximum overlap between chunks.

4. Creating an Instance of a Knowledge Graph Generator Using LangChain

• Using an LLM model, the GraphIndexCreator class in LangChain creates graph indexes from text. Specifically, when you instantiate a GraphIndexCreator and pass in an LLM, it configures the class to use that LLM model to parse text and extract knowledge graph triples.
• So, creating the GraphIndexCreator initializes the graph creation tool and allows you to call the language model and build graphs using text. The graphs can then be used to power things like question answering.
• In summary, it sets up a system to automate building knowledge graphs from text using the power of large language models. The LLM processes the text, and GraphIndexCreator prompts it appropriately and organizes the output into graph structures.

5. Storing All the Nodes and Edges of the Graph into a Vector Database Called OpenSearch

• Setting up authentication and configuration to interact with Amazon OpenSearch Service (AOS) from LangChain’s OpenSearchVectorSearch. This includes setting the service name as ‘es’, specifying the AWS region for AOS, retrieving temporary AWS credentials using boto3, creating an AWS4Auth object for authentication to AOS, and using OpenSearchVectorSearch to index documents and enable vector similarity search. Additionally, it configures the connection details to point to the AOS endpoint, including HTTP authentication, timeout, SSL usage, certificate verification, connection class, bulk indexing batch size, and index name.
• OpenSearchVectorSearch ingests the graph data into OpenSearch by taking all the specified documents and variables, enabling efficient storage and retrieval of vector embeddings within the OpenSearch backend.

6. Creating an Instance of OpenSearch Vector Database for Index Name in Which Data is Stored

The instance of the OpenSearch vector database for the index name is created using the OpenSearchVectorSearch object. This enables similarity search over documents stored in Amazon OpenSearch Service. The process involves connecting to OpenSearch using credentials, specifying the embedding function for encoding text, defining the OpenSearch domain endpoint, setting the index name for the stored documents, and utilizing AWS Sigv4 authentication along with RequestsHttpConnection for HTTP requests.

7. Creating Q&A Query Chain with Custom Prompt Using LangChain

langchain_qa function will take the question as input and retrieve graph nodes and edges from the vector store according to the question. Then, these retrieved graph nodes and edges will be passed to the prompt as context, and then it asks llm to generate an answer for the question by passing a modified prompt as input to llm.

Plotting Knowledge Graph as Image

The created function takes an LLM result object as input, extracts graph data from it, constructs a directed graph using the NetworkX library and visualizes it using Matplotlib. The graph represents a knowledge graph generated by the llama-index system. Each node represents a concept, and edges represent relationships between concepts. The function displays the graph with labeled nodes and edges, visually representing the knowledge structure.

Hallucination Evaluation

Evaluates the quality of a language model’s answer. It takes the question, the model’s generated answer, and the context from which the answer was derived. It then measures the answer against the context to determine if it contains any hallucinated information. The function returns a score indicating the degree of hallucination, with a score above 0.5, meaning that the answer contains no hallucinated content.

Implemented Code

Results

Conclusion

The comparative analysis of LlamaIndex and LangChain for knowledge graph construction reveals nuanced insights into their strengths and weaknesses. LlamaIndex, utilizing the anthropic.claude-v2 LLM and titan-embed model demonstrate strong document processing capabilities and knowledge graph generation capabilities. It excels in extracting and organizing knowledge triplets, making it highly effective for creating structured and queryable knowledge graphs. On the contrary, LangChain, with its similar setup, showcases efficiency in chunking documents and generating graph indexes, offering a streamlined approach to embedding and vector similarity search in OpenSearch. Ultimately, the choice between LlamaIndex and LangChain will depend on specific project requirements, but both frameworks provide potent tools for advancing knowledge graph technology.

In natural language processing (NLP), the Retrieval Augmented Generation (RAG) paradigm represents a significant advancement, offering a hybrid approach that seamlessly integrates retrieval-based and generative methods. By combining the strengths of both approaches, RAG aims to overcome the limitations of traditional generative models, such as lack of context awareness and difficulty in producing relevant responses to diverse queries. At the heart of RAG lies the integration of a powerful retrieval mechanism, which leverages a large-scale database to gather relevant information that informs the subsequent generation of text responses.

In this article, we delve into evaluating RAG models using visualization methods. We aim to leverage the Chroma database for querying relevant documents and utilize UMAP projection to visualize embeddings in a two-dimensional space.

Objective: Assessing Contextual Relevance in RAG Models

The primary objective of this evaluation is to assess the effectiveness of RAG models in generating contextually relevant responses by visualizing the relationship between queries, retrieved documents, and generated text embeddings. Traditional evaluation metrics may provide insights into the fluency and coherence of generated text, but they cannot often assess the relevance of generated responses in relation to the input queries and retrieved documents.

Methodology: A Step-by-Step Approach

Step 1: Data Collection

We first identify relevant sources containing textual information in PDF format to collect a set of PDF documents for retrieval. Next, we systematically acquire these documents, ensuring comprehensive coverage of the desired content. The curated collection of PDFs serves as the knowledge base for retrieving information in response to user queries.

Step 2: Chunking Strategies

Chunking refers to dividing large text documents into smaller, manageable chunks for processing. There are several strategies for chunking text data, including:

Sentence-Level Chunks: This strategy involves dividing the text into chunks based on sentence boundaries. Each chunk contains one or more complete sentences. This approach ensures that the context is preserved within each chunk.

The choice of chunking strategy depends on factors such as the nature of the text data, the desired granularity of chunks, and the computational resources available for processing.

Step 3: Generating Embeddings of the Chunks

Once the text data is chunked into smaller segments, we generate embeddings for each chunk. Embeddings are dense vector representations of text data that capture semantic information about the text. There are several techniques for generating embeddings of text data, including:

Sentence Transformer Embeddings: We can also use techniques such as Universal Sentence Encoder (USE) to generate embeddings for entire sentences. This approach is instrumental when dealing with sentence-level chunks.

• BAAI/gte-large-en: This is a larger version of the above model, offering improved performance in machine translation compared to the base version.
• Sentence-transformers/all-MiniLM-L12-v2 and all-MiniLM-L6-v2: These models are trained on massive text corpora and specialize in generating sentence embeddings. These embeddings can be used for downstream tasks like semantic search, clustering, and text classification.

The choice of embedding technique depends on factors such as the size of the text data, the computational resources available, and the specific requirements of the task.

Step 4: Creating a Chroma Collection

Once we have generated embeddings for the text chunks, we create a Chroma collection to store these embeddings. Chroma is a database system optimized for storing and querying high-dimensional embeddings. To create a Chroma collection, we perform the following steps:

• Initialize Chroma Client: We initialize a Chroma client object to connect to the Chroma database.
• Define Collection Name: We specify a name for the Chroma collection that will store the embeddings.
• Add Documents: We add the corresponding embedding to the Chroma collection using the add method for each text chunk. The document ID is used as the key to retrieve embeddings during querying.

By storing the embeddings in a Chroma collection, we can efficiently query the collection to retrieve relevant chunks based on similarity to a given query.

Step 5: Querying the Chroma Collection

Once the Chroma collection is created and populated with embeddings, we can query the collection to retrieve relevant chunks based on a given query. The querying process involves the following steps:

• Input Query: We provide a query to the RAG model, representing the user’s input or question.
• Generate Query Embedding: We generate an embedding for the input query using the same embedding technique for generating chunk embeddings.
• Query Chroma Collection: We query the Chroma collection using the generated query embedding to retrieve the most similar chunks. This is typically done using nearest-neighbor search algorithms such as k-nearest neighbors (KNN) or approximate nearest neighbors (ANN).
• Retrieve Relevant Chunks: The retrieved chunks are then passed to the generator component of the RAG model to generate contextually relevant responses.

By querying the Chroma collection, we can efficiently retrieve relevant chunks based on the semantic similarity between the query and the stored embeddings.

Step 6: Visualization

To visualize the performance of the RAG model, we employ UMAP (Uniform Manifold Approximation and Projection) to project the high-dimensional embeddings into a two-dimensional space. UMAP is a dimensionality reduction technique that preserves the local structure of the data, making it well-suited for visualizing embeddings. We utilize Streamlit, a Python library for creating interactive web applications and developing interactive plots and dashboards for visualization.

The visualization components include:

• Query-Document Visualization: We plot the embeddings of queries and retrieved documents in a two-dimensional space using UMAP. Each point in the plot represents a query or a document, and we use different colors or markers to distinguish between them. Users can interactively explore the spatial relationship between queries and relevant documents.
• Query-Response Visualization: We visualize the embeddings of queries and generated responses in a two-dimensional space. Each point in the plot represents a query or a generated response. Users can analyze how well the generated responses align with the context provided by the queries.

Analysis and Interpretation of Visualization Results

Finally, we analyze the visualizations and evaluation metrics to assess the performance of the RAG model. We identify patterns, trends, and areas for improvement based on the visualization and evaluation results. This analysis provides insights into the effectiveness of the RAG model in generating contextually relevant responses and informs future research directions.

By following this methodology, we aim to provide a comprehensive evaluation of RAG models using visualization methods, thereby facilitating a deeper understanding of their capabilities and limitations in natural language generation tasks.

In natural language processing (NLP), the Retrieval Augmented Generation (RAG) paradigm represents a significant advancement, offering a hybrid approach that seamlessly integrates retrieval-based and generative methods. By combining the strengths of both approaches, RAG aims to overcome the limitations of traditional generative models, such as lack of context awareness and difficulty in producing relevant responses to diverse queries. At the core of RAG lies the integration of a powerful retrieval mechanism, which leverages a large-scale database to gather pertinent information that informs the subsequent generation of text responses.

In this report, we will evaluate RAG models using visualization methods. We aim to utilize the Milvus collection to query relevant documents and utilize UMAP projection to visualize embeddings in a two-dimensional space.

Objective

The primary objective of this evaluation is to assess the effectiveness of RAG models in generating contextually relevant responses by visualizing the relationship between queries, retrieved documents, and generated text embeddings. Traditional evaluation metrics may provide insights into the fluency and coherence of generated text, but they cannot often assess the relevance of generated responses in relation to the input queries and retrieved documents.

Methodology

1. Creating a Milvus Collection

Utilizing the Milvus server, we establish a Milvus collection to house the embeddings. Milvus is a highly efficient database system specially designed for storing and querying high-dimensional embeddings. To initiate the process, we establish a connection with the Milvus server and assign a name to the collection where the embeddings will be stored. This collection acts as a repository for the embeddings extracted from the text chunks.

2. Querying the Milvus Collection

Upon the successful creation of the Milvus collection, we proceed to query it to retrieve pertinent chunks of text based on a given query. The querying process encompasses the following sequential steps:

• Input Query: We furnish the RAG model with a query, which represents the user’s input or inquiry.
• Generating Query Embedding: Employing the same embedding technique utilized for generating embeddings of text chunks, we generate an embedding for the input query. This embedding encapsulates the semantic essence of the query.
• Querying Milvus Collection: Leveraging the generated query embedding, we execute a query on the Milvus collection to retrieve the most akin chunks of text. Typically, this is accomplished using nearest-neighbor search algorithms like k-nearest neighbors (KNN) or approximate nearest neighbors (ANN).
• Retrieving Relevant Chunks: The retrieved text chunks, deemed most similar to the input query based on their embeddings, are subsequently forwarded to the generator component of the RAG model to generate contextually relevant responses. This step ensures the responses are closely aligned with the semantic context provided by the user’s query.

By employing the Milvus collection, we streamline the process of retrieving relevant text chunks, enabling efficient and accurate responses to user queries.

3. Visualization

To provide a comprehensive evaluation of the RAG model’s performance, we utilize UMAP (Uniform Manifold Approximation and Projection) to project the high-dimensional embeddings into a two-dimensional space for visualization purposes. UMAP, a dimensionality reduction technique, is adept at preserving the local structure of the data, rendering it ideal for visualizing embeddings. Additionally, we employ Streamlit, a Python library renowned for creating interactive web applications, to craft interactive plots and dashboards for visualization.

The visualization components encompass:

• Query-Document Visualization: Utilizing UMAP, we plot the embeddings of queries alongside those of retrieved documents in a two-dimensional space. Each point on the plot represents either a query or a document, with different colors or markers distinguishing between them. This visualization empowers users to interactively explore the spatial relationship between queries and the relevant documents retrieved.
• Query-Response Visualization: Similarly, we visualize the embeddings of queries juxtaposed with those of the generated responses in a two-dimensional space. Each point on the plot corresponds to either a query or a generated response. This visualization facilitates an analysis of how well the generated responses align with the contextual information provided by the queries, offering insights into the effectiveness of the RAG model.

Analysis and Interpretation

Subsequently, we conducted a thorough analysis of the visualizations and evaluation metrics to assess the performance of the RAG model. This analysis entails identifying patterns, trends, and potential areas for improvement based on the visualization and evaluation results. By scrutinizing these findings, we gain valuable insights into the RAG model’s efficacy in generating contextually relevant responses, thereby informing future research endeavors in the domain of natural language generation tasks.

Phoenix, an advanced observability platform, provides exceptional insights into LLM operations, enabling quick identification, diagnosis, and rectification of issues. Its integration with LlamaIndex enhances these capabilities further, offering an innovative approach to observability. This combination allows data scientists and developers to gain better insights into their models’ operations, significantly improving LLM functionality and troubleshooting capabilities.

Key Pillars of LLM Observability

The concept of LLM Observability rests on five crucial pillars, each integral for maintaining a high-performance LLM system.

• Evaluation: It plays a pivotal role as it involves utilizing LLM Eval to understand and validate LLM performance comprehensively. This process helps identify and rectify issues such as hallucinations or Q&A problems, ensuring the model’s reliability and accuracy.
• LLM Traces and Spans: It offers a unique insight into LLM applications, capturing spans and traces from common LLM App frameworks like LangChain and LlamaIndex. This data is instrumental in understanding the internal workings of the LLM, aiding in debugging and optimizing its performance.
• Prompt Analysis and Troubleshooting: It enables tracking performance issues using Evals or traditional metrics. By analyzing prompts, one can identify and rectify problems, iterating using live production data to reproduce and address issues as they arise precisely.
• Search and Retrieval: These are essential aspects of LLM observability, particularly in integrating proprietary data into an LLM using techniques like Retrieval-Augmented Generation (RAG). Troubleshooting and evaluating RAG are paramount for maintaining optimal LLM performance and functionality.
• Fine-Tuning: It involves collecting data examples or clusters of problems as data points for export into fine-tuning workflows. This process ensures that the LLM adapts and improves over time, enhancing its performance and effectiveness in various applications.

Enhancing Observability in Large Language Models with Arize

Arize Phoenix is an open-source observability library and platform tailored for experimentation, evaluation, and troubleshooting in machine learning applications. The toolset is adept at ingesting inference data for various machine learning models, including LLMs, computer vision (CV), natural language processing (NLP), tabular datasets, and LLM traces. It empowers AI engineers and data scientists to rapidly visualize their data, assess performance, pinpoint issues and insights, and seamlessly export findings for improvement.

• Comprehensive Observability: Arize offers a holistic solution for enhancing observability across the entire stack of LLMs. By enabling teams to work backward from the output, it helps pinpoint the exact source of issues, thus facilitating efficient troubleshooting and analysis.
• Evaluation and Feedback: Arize helps collect and generate evaluations, including responses and user feedback. This allows teams to identify clusters of bad responses, providing valuable insights for further troubleshooting and analysis, ultimately improving the overall performance of their LLMs.
• Troubleshooting Retrieval Issues: Arize provides tools to troubleshoot bad retrieval, particularly concerning search and retrieval with vector stores. Teams can evaluate whether the system pulled all relevant context and if enough relevant context is available, addressing issues at the root of retrieval problems.
• Detection of Faults: Arize can help catch LLM hallucinations and bad responses. By identifying and analyzing these, teams can uncover trends and areas for improvement, ensuring that their LLMs deliver accurate and reliable results.
• Prompt Optimization: Arize helps uncover bad prompts and prompt templates, offering analysis to understand which templates perform better. This insight allows teams to refine their prompts for improved LLM performance.
• Agent Tracing and Spans: Arize’s agent tracing and spans feature enables teams to trace where the chain failed, providing a detailed understanding of failed calls or issues within spans. This functionality is invaluable for identifying and rectifying issues quickly and effectively.
• Fine-Tuning Support: For teams fine-tuning their LLMs, Arize provides workflows to understand patterns in bad responses. This information allows teams to fine-tune their models more effectively, improving performance on specific phrases or inputs where the model typically provides a bad response.

Traces

Tracing plays a crucial role in understanding the behavior of LLM applications, which often exhibit nondeterministic problems or complexity beyond local reproduction. A trace records the paths followed by requests originating from either applications or end-users as they traverse various steps within a system. This capability is essential for diagnosing performance issues, as it enhances visibility into the health of an application or system, enabling the debugging of challenging behaviors to replicate locally. Tracing down the sequence of events within a request as it progresses through a system simplifies debugging and understanding LLM applications.

Each trace consists of one or more spans; the first span represents the root span, encapsulating the entire request lifecycle. Subsequent spans provide detailed contextual information about the steps in processing a request, offering a comprehensive view of the request’s execution path.

Components in Traces

Traces consist of the following metrics:

Total Traces — Provides us with the total number of traces created to generate an answer by the model.

Total Tokens — It is the total amount of input and output tokens that the LLM model has used to generate the required answer.

Latency P50 — Often referred to in the context of latency or response time, P50 represents the median value in a dataset. In the context of transaction durations or call latencies, the P50 threshold is the value at which 50% of the transactions or calls have durations greater than this threshold. This means that P50 latency is the median latency, indicating that half of the transactions or calls experienced latencies higher than this value. In practical terms, P50 provides a measure of central tendency, helpful in understanding a system or process’s typical or median performance.

Latency P99 — P99, or the 99th percentile, is a latency metric that indicates the threshold below which 99% of requests to an endpoint fall in response time. For example, if the P99 latency for an endpoint is 100 milliseconds, it means that 99% of the requests made to that endpoint are completed in 100 milliseconds or less. P99 is particularly useful because it focuses on the worst-case scenario, helping to identify and address issues that might only affect a small percentage of users but could still be significant in the overall user experience.

Evaluation Metrics — This metric provides us with details about the credibility of our model’s output. Here, we evaluate the model on Hallucination, QA Correctness, and Relevance.

• Hallucination — In generating responses or answers, “hallucination” refers to providing information not supported by the reference text or assuming details that are not explicitly stated or available in the reference text. It essentially means generating content that goes beyond the scope or context of the original information provided. Hallucinations can lead to inaccurate or misleading responses, as the generated content may not be grounded in the facts or details provided in the reference text.
• QA Correctness — QA Correctness refers to evaluating whether a question posed to a system was correctly answered based on the data retrieved by the system. Unlike retrieval evaluations focusing on the chunks of data returned, QA Correctness evaluates the system’s overall performance in providing accurate answers to questions. This evaluation is crucial for assessing the effectiveness of a LLM in detecting issues in Q&A systems, particularly regarding the accuracy and correctness of the answers provided based on the context data retrieved.
• Relevance — A relevance evaluator is critical in determining whether a retrieved document chunk contains relevant information that answers a given query. This evaluation is essential for assessing the effectiveness of retrieval systems, ensuring that the retrieved information is pertinent and can adequately address the specific question posed by the user.

1. NDCG (Normalized Discounted Cumulative Gain) evaluates the effectiveness of ranking retrieved documents, giving more weight to relevant documents appearing higher in the list. This metric is valuable when the order of search results matters.
2. Precision assesses the accuracy of a retrieval system by calculating the ratio of relevant documents among those retrieved, focusing on result relevancy.
3. Hit Rate, commonly used in recommendation systems, indicates the frequency of relevant user recommendations. In retrieval systems, it reflects the success rate of retrieving pertinent documents for queries.

Trace Details

Phoenix provides a comprehensive set of features for understanding and troubleshooting your system, including:

• Application latency tracking, which highlights slow invocations of LLMs, retrievers, and other components.
• Token usage breakdown with LLMs, allowing you to identify the most resource-intensive LLM calls.
• Capture of critical runtime exceptions, such as rate-limiting, as exception events.
• Viewing all retrieved documents during a retriever call, along with their scores and return order.
• Access to embeddings used for retrieval and the underlying embedding model.
• Viewing LLM parameters, such as temperature and system prompts, used during a call to debug.
• Identifying the prompt template used during the prompting step and the variables involved.
• Viewing descriptions and function signatures of tools accessed by your LLM.
• Accessing function selection and function messages in input messages to the LLM when using models with function calls, such as OpenAI or others.

Problem Statement

LlamaIndex offers convenient high-level APIs for building powerful applications with minimal code. However, understanding its internal operations and diagnosing performance issues can be challenging. To address this, we use Phoenix, a tool that enhances observability of LlamaIndex applications. Phoenix visualizes the underlying structure of each query engine call, highlighting problematic execution spans based on latency, token count, and other evaluation metrics. This paper presents the design and implementation of Phoenix, demonstrating its effectiveness in improving the development and maintenance of LlamaIndex applications.

Procedures for Utilizing LlamaIndex

1. Models Utilized

• Large Language Model (LLM): Azure Open AI GPT-4
• Embedding Model: text-embedding-ada-002

2. PDF Processing with LlamaIndex

• Utilizing PDFReader to extract text from each PDF page and convert it to a document type.
• Only document-type chunks are compatible with the vector database.
• Experimentation involved open domain sustainability reports:

1. Pepsico_AnnualReport_2021
2. Pepsico_SustainabilityReport_2022
3. Pepsico_SustainabilityReport_2020
4. Pepsico_SustainabilityReport_2021
5. Nestle_SustainabilityReport_2022
6. KimberlyClark_AnnualReport_2020

3. Customization of LLM and Embedding Models in LlamaIndex

• Modification of the LLM and embedding models to our custom versions, as llama-index defaults to OpenAI GPT models.
• Introduction of custom chunk size (number of characters per text chunk) and chunk overlap (common characters between two chunks) parameters.
• Utilization of the same chunk creator for both llama-index functions.

4. Ingesting Documents into OpenSearch Vector Database

• The OpenSearch vector database is a specialized storage system designed for vector embeddings, offering optimized similarity searches. It efficiently organizes and retrieves data based on the similarity of their vector representations. Leveraging OpenSearch Service’s vector database capabilities enables the implementation of semantic search, Retrieval Augmented Generation (RAG) with LLMs, recommendation engines, and search for rich media.

a) Setting Up Variables

• Initialize variables required to initialize the OpenSearchVectorClient. These variables include the endpoint (URL of the OpenSearch server), the index (name of the index in OpenSearch where data will be stored), the vector dimensions (dimension of embedding for each word), the text field (name of the field in the index containing text data), the embedding field (name of the field in the index containing vector embeddings), and the username and password for authentication.

b) Initializing the OpenSearchVectorClient

• Initialize the OpenSearchVectorClient using the previously set variables. This client is used to interact with the OpenSearch server.

c) Initializing the OpenSearchVectorStore

• Initialize the OpenSearchVectorStore with the OpenSearchVectorClient. The OpenSearchVectorStore is a vector store that utilizes OpenSearch as its backend, which is used for storing and retrieving vector embeddings.

d) Setting Up the StorageContext

• nitialize the StorageContext with the OpensearchVectorStore as its vector store. The StorageContext in LlamaIndex is a container holding the document, index, and vector stores. The vector store is set to the OpensearchVectorStore, while the document store and index store are set to their default values.

e) Ingesting Data using VectorStoreIndex

• Utilize VectorStoreIndex, a LlamaIndex type that uses a vector store for storing and retrieving vector embeddings. VectorStoreIndex requires documents (representing the list of documents to be stored), the storage_context (in which the vector store is defined), and service_context (containing the services used by the index, such as the LLM model, embedding model, and chunk creator) as input parameters.

5. Establishing an OpenSearchVectorClient Instance for the Target Index

a) Setting Up Variables

Variables are configured for initializing the OpenSearchVectorClient. These include the endpoint (URL of the OpenSearch server), the index (name of the index in OpenSearch where data is stored), the vector dimension (embedding dimension for each word), the text field (field name in the index containing text data), the embedding field (field name in the index containing vector embeddings), and the username and password for authentication.

b) Initializing the OpenSearchVectorClient

The OpenSearchVectorClient is initialized with the configured variables, allowing interaction with the OpenSearch server.

c) Initializing the OpenSearchVectorStore

Initialization of the OpenSearchVectorStore is performed using the OpenSearchVectorClient. This store utilizes OpenSearch as its backend and is used for storing and retrieving vector embeddings.

d) Creating the VectorStoreIndex

The VectorStoreIndex.from_vector_store method is used to create an index object, taking the vector_store as input, which was initialized with the index name where the data is stored.

6. Creating a Query Engine with Custom Prompt Using LlamaIndex

• The process involves a retriever function that takes a question as input and retrieves document chunks from the vector store based on the question’s similarity search.
• These retrieved document chunks are then passed to the prompt as context.
• Subsequently, the prompt asks the LLM to generate an answer for the question by passing the modified prompt as input to the LLM.

7. Phoenix Traces:

• Giving the user a prompt to open the Phoenix UI.
• LLM trace observability aims to simplify the troubleshooting of LLM frameworks and enhance the visibility of your LLM system.
• Arize Phoenix aims to enhance development tools for close-to-code LLM span and trace debugging, prioritizing the alignment of AI systems with desired functionality over troubleshooting distributed system timing.

8. Phoenix Eval

a) Retrieving QA Data and Retrieved Documents

• Fetching QA Data

The function get_qa_with_reference fetches question-answer pairs along with reference data from the Phoenix instance. It communicates with the Phoenix API endpoint provided in the px. Client constructor to retrieve this data. The returned data is stored in a DataFrame named queries_df.

• Fetching Retrieved Documents

The function get_retrieved_documents is to retrieve documents that were retrieved or accessed during the process. It communicates with the Phoenix API endpoint using the px.Client constructor to fetch the data. The retrieved documents are stored in a data frame named retrieved_documents_df.

b) Initializing Evaluation Model

• Initializing an evaluation model configured to utilize OpenAI’s GPT-4 model for natural language processing tasks. It sets up the necessary parameters for authentication.

c) Initializing Evaluators: Three evaluators are initialized

• hallucination_evaluator: Evaluates hallucination, i.e., whether the model generates responses containing fabricated or irrelevant information.
• qa_correctness_evaluator: Evaluates the correctness of question-answer pairs generated by the model.
• relevance_evaluator: Evaluates the relevance of retrieved documents to the queries.

d) Running Evaluations

• For hallucination and QA correctness evaluation, the run_evals() function is called with the DataFrame containing question-answer pairs (queries_df) and a list of evaluators (hallucination_evaluator and qa_correctness_evaluator). Explanations for evaluation results are provided.
• The run_evals() function is called for relevance evaluation with the DataFrame containing retrieved documents (retrieved_documents_df) and the relevance_evaluator. Explanations for evaluation results are also provided.

e) Logging Evaluations

• Evaluation results for hallucination and QA correctness are logged using px.log_evaluations() with SpanEvaluations for span-level evaluation results.
• Evaluation results for relevance are logged using px.log_evaluations() with DocumentEvaluations for document-level evaluation results.

Example

Let us consider an example with a user question, an output generated by the model, and the traces created by the Phoenix.

This methodology follows a chain process, where user questions trigger retrieval of answers from a vector database using similarity search. Through this approach, we can assess the chain’s performance in terms of metrics such as Hallucination and QA correctness as evaluation metrics. Additionally, we can track the processing time for each question and record the total token usage necessary for executing this chain.

Spans

A span is a fundamental unit of work or operation within an application akin to a time segment. It tracks specific operations a request undergoes, providing a detailed account of the activities conducted during its execution. Each span comprises essential components such as a name, time-related data, structured log messages, and additional metadata known as attributes, which collectively offer insights into the operation being tracked. Notably, spans can be nested, as indicated by the presence of a parent span ID, allowing them to represent sub-operations within a larger task. This nesting capability enhances the granularity with which operations are captured, enabling a more comprehensive understanding of the work performed in an application. Upon creation, a span is classified into several categories, including Chain, Retriever, Reranker, LLM, Embedding, Agent, or Tool, based on the nature of the operation it represents.

• Chain: It represents a pivotal point or link between various steps within the application’s workflow. It denotes sequences of calls, including those to an LLM model, a tool, or a data preprocessing step. Essentially, a Chain span serves as a starting point or a connector that facilitates the execution flow between different components or stages of the application. For instance, it could signify initiating a request to an LLM application or the mechanism through which context is transferred from one operation, such as data retrieval, to another, such as an LLM call. By serving as a linkage mechanism, the Chain span aids in tracing the execution path of an application, thereby enabling a clear visualization of the interactions and dependencies among its various components or steps.
• Retriever: It is a vital component within the data processing pipeline, tasked with retrieving the most pertinent context based on a user query or chat message. While it can be constructed on top of indexes, it can also exist as an independent entity. In query engines and Chat Engines, the retriever is a foundational element for retrieving relevant context. In the context of spans, as described in the provided sources, a Retriever span represents a step-in data retrieval. It signifies a call to a vector store or database, which is crucial for systems that retrieve information from extensive data sets based on queries or criteria. The retriever’s primary function is to efficiently fetch the most relevant documents or information for subsequent application steps, such as input for a LLM for generating responses or further analysis. In the broader landscape of LLM applications, particularly those involving Retrieval-Augmented Generation (RAG), the retriever is critical in ensuring the quality and relevance of processed information. By adeptly retrieving pertinent documents, the system significantly enhances the accuracy and utility of generated outputs, underscoring the retriever’s indispensability in LLM application architectures.
• Embedding: Commonly known as vector embedding, embedding acts as a numerical representation of the semantics or meaning contained within the text. It is critical in LLMs and their applications. It enables the mathematical comparison of two pieces of text with similar meanings, even when the actual text varies significantly. This mathematical relationship enables semantic search, where a user’s query terms can be matched with text related to the meaning rather than keyword matching. There are various types of embeddings, each differing in efficiency, effectiveness, and computational cost. For example, LlamaIndex employs text-embedding-ada-002 by default, the default embedding used by OpenAI. Different LLMs may utilize distinct embeddings to suit their specific requirements.

In the LLM Traces and Observability context, an embedding within a span denotes a call made to an LLM (Language Model) to generate an embedding vector. This vector represents information in a dense format within a high-dimensional space, where similar items are positioned closer together. Within a span — a unit of work or operation in a trace — an “Embedding” span indicates that during the execution of a request or operation, a call was made to an LLM to obtain such an embedding. This process may involve retrieving a dense vector representation of text for further processing steps, such as similarity comparison or information retrieval. The span tracks this operation, including its start and end times, and may include additional metadata in its attributes, such as the model’s name used for generating the embedding or the input text.

• Chunking: It involves retrieving document chunks from the vector database. These chunks are selected based on a specified top k value and then forwarded to the LLM for answer generation.
• LLM: It is a sophisticated computational system capable of completing text, engaging in chat-like interactions with users, and executing instructions. Its primary function is to comprehend and generate human-like text based on the input it receives. LLMs find applications in various fields, including chatbots, text completion tasks, etc. In performance analysis, an LLM span represents a call to an LLM system, such as those provided by OpenAI or Llama. This span signifies a request made to the Language Learning Model, which could involve generating text, answering questions, or performing any other task within the capabilities of LLMs.

Span Lookup

Query Span

The Query Span serves as a container for the user’s query, encompassing details about the input question and its corresponding output. Evaluation metrics such as Hallucination and QA Correctness play a crucial role, with a score of 1 indicating a hallucinated answer and 0 denoting incorrect QA Correctness. Additionally, Attributes within the Query Span consist of key-value pairs designed to hold metadata, facilitating the annotation of spans with pertinent information related to the monitored operation.

Retrieve Span

The Retrieve span is responsible for fetching the documents relevant to a user query. It retrieves the top k most relevant documents and their respective relevance scores. Additionally, it provides information on the time taken to retrieve these documents, offering insights into the efficiency of the retrieval process.

Embedding Span

The Embedding span is a crucial component in retrieving and converting query text to embeddings within a system. It encompasses the embedding model for retrieving documents and converting text into embeddings. Additionally, the Embedding span includes attributes such as the dimensional vector employed and the time taken to complete the embedding process.

LLM

LLM leverages context retrieved from the vector database to generate answers based on the provided prompt. Using GPT-3.5 Turbo as the underlying model, the LLM processes the prompt to generate answers formatted according to the specified requirements.

The prompt template offers a detailed overview of the template utilized within an application, including the input and output variables employed and the format of the resulting output. On the other hand, invocation parameters aid in comprehending additional parameters like temperature and the model utilized during the result-generation process.

In this article, we explore the integration of Phoenix with the Llama Index to enhance the evaluation of Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) applications. We aim to address the challenges associated with LLM evaluation and provide a structured approach for utilizing the Llama Index effectively.

Problem Statement

Llama Index offers convenient high-level APIs for building powerful applications with minimal code. However, understanding the performance of a Large Language Model (LLM) model and the Retrieval Augmented Generation (RAG) application can be challenging. To address this, we use Phoenix, a tool that evaluates Llama Index applications. Existing LLM evaluation lacks rigor (focusing only on models, not prompt templates) and faces integration issues across development and deployment, hindering efficient processing and flexibility. Phoenix LLM Evals solves this by offering pre-tested and custom templates, enabling efficient batch processing, and ensuring seamless integration within Llama Index workflows.

Procedures for Utilizing the Llama Index

Models Utilized

• Large Language Model (LLM): anthropic.claude-v2
• Embedding Model: titan-embed

PDF Processing with Llama Index

1. Utilizing PDFReader to extract text from each PDF page and convert it to a document type.
2. Only document-type chunks are compatible with the vector database.

Experimentation involved open domain sustainability reports:

• ‘Pepsico_AnnualReport_2021’
• ‘Pepsico_SustainabilityReport_2022’
• ‘Pepsico_SustainabilityReport_2020’
• ‘Pepsico_SustainabilityReport_2021’
• ‘Nestle_SustainabilityReport_2022’
• ‘KimberlyClark_AnnualReport_2020’

Customization of LLM and Embedding Models in Llama Index

• Modification of the LLM and embedding models to our custom versions, as llama-index defaults to OpenAI GPT models.
• Introduction of custom chunk creator, chunk_size (number of characters per text chunk), and chunk_overlap (common characters between two chunks) parameters.

Ingesting Documents into OpenSearch Vector Database

• The OpenSearch vector database is a specialized storage system designed for vector embeddings, offering optimized similarity searches. It efficiently organizes and retrieves data based on the similarity of their vector representations. Leveraging OpenSearch Service’s vector database capabilities enables the implementation of semantic search, Retrieval Augmented Generation (RAG) with LLMs, recommendation engines, and search for rich media.

a) Setting Up Variables

• Initialize variables required to initialize the OpenSearchVectorClient. These variables include the endpoint (URL of the OpenSearch server), the index (name of the index in OpenSearch where data will be stored), the vector dimensions (dimension of embedding for each word), the text field (name of the field in the index containing text data), the embedding field (name of the field in the index containing vector embeddings), and the username and password for authentication.

b) Initializing the OpenSearchVectorClient

• Initialize the OpenSearchVectorClient using the previously set variables. This client is used to interacting with the OpenSearch server.

c) Initializing the OpenSearchVectorStore

• Initialize the OpenSearchVectorStore with the OpenSearchVectorClient. The OpenSearchVectorStore is a vector store that utilizes OpenSearch as its backend, which is used for storing and retrieving vector embeddings.

d) Setting Up the StorageContext

• Initialize the StorageContext with the OpensearchVectorStore as its vector store. The StorageContext in Llama Index is a container holding the document, index, and vector stores. The vector store is set to the OpensearchVectorStore, while the document and index stores are set to their default values.

e) Ingesting Data using VectorStoreIndex

• Utilize VectorStoreIndex, a Llama Index type that uses a vector store for storing and retrieving vector embeddings. VectorStoreIndex requires documents (representing the list of documents to be stored), the storage_context (in which the vector store is defined), and service_context (containing the services used by the index, such as the LLM model, embedding model, and chunk creator) as input parameters.

Establishing an OpenSearchVectorClient Instance for the Target Index

a. Setting Up Variables

Variables are configured for initializing the OpenSearchVectorClient. These include the endpoint (URL of the OpenSearch server), the index (name of the index in OpenSearch where data is stored), the vector dimension (embedding dimension for each word), the text field (field name in the index containing text data), the embedding field (field name in the index containing vector embeddings), and the username and password for authentication.

b. Initializing the OpenSearchVectorClient

The OpenSearchVectorClient is initialized with the configured variables, allowing interaction with the OpenSearch server.

c. Initializing the OpenSearchVectorStore

Initialization of the OpenSearchVectorStore is performed using the OpenSearchVectorClient. This store utilizes OpenSearch as its backend and is used for storing and retrieving vector embeddings.

d. Creating the VectorStoreIndex

The VectorStoreIndex.from_vector_store method is used to create an index object, taking the vector_store as input, which was initialized with the index name where the data is stored.

Creating a Query Engine with Custom Prompt Using Llama Index

• The process involves a retriever function that inputs a question and retrieves document chunks from the vector store based on the question’s similarity search.
• These retrieved document chunks are then passed to the prompt as context.
• Subsequently, the prompt asks the LLM to generate an answer for the question by passing the modified prompt as input to the LLM.

Enhancing LLM Evaluation with Arize Phoenix

Phoenix LLM Evaluation refers to assessing the performance and effectiveness of Large Language Models (LLMs) using the Phoenix observability library. This evaluation methodology involves utilizing Phoenix’s features and tools specifically designed for evaluating LLM-based applications. By leveraging Phoenix, users can conduct rigorous evaluations, analyze model behavior, diagnose performance issues, and optimize the development and maintenance of LLM applications. Phoenix LLM Evaluation aims to provide a streamlined and efficient approach to evaluating LLM models, ensuring transparency, reproducibility, and scalability in the evaluation process.

LLM Evaluation

Large Language Model (LLM) evaluation involves systematically assessing the performance and capabilities of LLM applications. It measures language generation quality, response relevance, bias detection, interpretability, and adherence to task requirements. Through standardized metrics, benchmark datasets, and interpretability tools, LLM evaluation provides insights into model strengths and weaknesses. It includes human assessments, automated metrics calculation, interpretability analysis, and comparison against benchmarks. By rigorously evaluating LLM models, stakeholders can make informed decisions on model improvements and deployment strategies for real-world applications.

Components in Phoenix LLM Evaluation

a) Pre-tested Eval Templates: Offers a collection of ready-made evaluation templates for common use cases, such as RAG relevance, toxicity, and code generation.

b) Custom Dataset Support: Supports custom datasets and evaluation creation, allowing for tailored evaluations based on specific use cases.

c) Fast On-Batch Processing: Optimized for high-throughput evaluations, ensuring efficient utilization of API keys and resources.

d) Benchmark Scripts and Datasets: Provides benchmark scripts and datasets for rigorous evaluation of models and prompts.

e) LangChain and LlamaIndex Integration: Enables evaluation directly within LangChain and LlamaIndex callback systems.

f) One-Click Explanations: Facilitates instant explanation of evaluation outcomes without requiring additional configuration.

g) Span Level Evaluation: Conducts evaluations at the granular level of individual spans within the LLM application, providing greater insight into performance issues.

h) Data Science Rigor: Employs data science principles to test model and template combinations, improving evaluation reliability.

i) Extensibility: Encourages modification and creation of custom evaluation templates to meet unique use cases.

j) Open Source: Available as an open-source project, fostering collaboration and innovation among the developer community.

How Phoenix LLM Evaluation Works

a) Data Preparation: You would prepare a dataset containing questions, their corresponding references, and the LLM-generated answers. This could involve formatting the data as needed for efficient evaluation.

b) Defining Evaluation Prompt Template: Create a prompt template specific to the task. For example, if you want to measure factual accuracy, you might create a prompt asking whether certain statements made by the LLM are true or false, given the reference material.

c) Defining Evaluation LLM Model: Choose an appropriate LLM model for performing the evaluation. Responses do not have to be the same as the original LLM generating the responses; it can be a separate model specifically trained for evaluation purposes.

d) Final Label Generation and Explanation: After evaluating the LLM-generated answers against the reference materials, assign a final label indicating the quality of the response. This label may reflect various aspects such as factual correctness, relevancy, coherence, etc., depending on the chosen evaluation metric(s). Optionally, provide explanations for the assigned labels. These explanations help understand why the response received a particular score, thus improving transparency and enabling further analysis.

Type of LLM Evaluations

a) Retrieval (RAG) Relevance

A relevance evaluator plays a critical role in determining whether a retrieved document chunk contains relevant information that answers a given query. This evaluation is essential for assessing the effectiveness of retrieval systems, ensuring that the retrieved information is pertinent and can adequately address the specific question posed by the user.

Here’s what it does:

Input: It takes two pieces of information:

1. Question: The user’s specific inquiry.
2. Reference text: retrieved documents containing potentially relevant information.

Processing: It analyzes both the question and the reference text, focusing on the specific details of the question.

Output: It outputs a single word:

1. “relevant”: If the reference text contains information that could potentially answer the question.
2. “unrelated”: If the reference text contains no relevant information for the specific question.

b) Hallucination Evaluation

The Hallucination Evaluation task involves presenting a query, a reference text, and a generated answer. The answer is generated based on the reference text, and the task is to determine if the answer contains false information (hallucination) by comparing it to the reference text. In the context of generating responses or answers, “hallucination” refers to providing information that is not supported by the reference text or assuming details that are not explicitly stated or available in the reference text. It essentially means generating content that goes beyond the scope or context of the original information provided. Hallucinations can lead to inaccurate or misleading responses, as the generated content may not be grounded in the facts or details provided in the reference text.

Here’s what it does:

Input: It takes this information:

1. Question: The user’s specific inquiry.
2. Reference text: retrieved documents containing potentially relevant information.
3. Answer: LLM generated an answer based on the reference.

Processing: It analyzes the question, answer, and reference text, focusing on the specific details of the question and reference text.

Output: It outputs a single word:

1. “factual”: If the answer to the question is correct relative to the reference text and does not contain made-up information.
2. “hallucinated”: If the answer provides factually inaccurate information to the query based on the reference text.

c) QA Correctness

It refers to evaluating whether a question posed to a system was correctly answered based on the data retrieved by the system. Unlike retrieval evaluations focusing on the chunks of data returned, QA correctness evaluates the system’s overall performance in providing accurate answers to questions. This evaluation is crucial for assessing the effectiveness of a Large Language Model (LLM) in detecting issues in Q&A systems, particularly regarding the accuracy and correctness of the answers provided based on the context data retrieved.

Here’s what it does:

Input: It takes three pieces of information:

1. Question: The user’s specific inquiry.
2. Reference text: Retrieved documents containing potentially relevant information.
3. Answer: The response provided to the question.

Processing: It compares the answer provided with the question and analyzes whether the information in the answer aligns with the details provided in the reference text.

Output: It outputs a single word:

1. “correct”: If the question is correctly and fully answered by the provided answer based on the reference text.
2. “incorrect”: If the question is incorrect or only partially answered by the provided answer compared to the reference text.

d) Toxicity Evaluation

Toxicity evaluation refers to assessing written or verbal content to determine whether it contains elements considered toxic. In AI and natural language processing, toxicity evaluation typically involves analyzing text data to identify language that may be harmful, offensive, hateful, demeaning, or threatening towards individuals or groups.

Here’s what it does:

Input: It takes a single piece of information:

1. Text: The written content to be examined for toxicity.

Processing: It analyzes the text, focusing heavily on the concept of toxicity as defined by making hateful statements, demeaning or disparaging others, using inappropriate language, or threatening physical or emotional violence.

Output: It outputs a single word:

1. “toxic”: If the text meets the definition of toxicity by containing hateful statements, demeaning language, inappropriate language, or threats of violence.
2. “non-toxic”: If the text contains no words, sentiments, or meaning, that could be considered toxic.

e) Code Generation Evaluation

Code Generation Evaluation involves the assessment of code produced through an automated generation process. It aims to determine the correctness and readability of the generated code. The evaluation examines whether the code is clear, concise, and easy to understand, and identifies whether the code is unnecessarily complex or convoluted for the task it aims to accomplish. The evaluation process typically involves reviewing the generated code against a specific query or task and providing a judgment on its readability, marking it as “readable” or “unreadable” based on established criteria.

Here’s what it does:

Input: It takes a single piece of information:

1. Question: The user’s specific inquiry.
2. Code: LLM Generated code

Processing: It analyzes the provided code, focusing on readability, simplicity, and clarity.

Output: It outputs a single word:

1. “readable”: If the code is readable, focusing on simplicity and clarity.
2. “unreadable”: If the code is unreadable or needlessly complex for what it’s trying to accomplish.

f) Summarization Evaluation

Summarization Evaluation involves assessing the quality of a generated summary in comparison to the original document. It aims to determine if the summary effectively captures the key information and is concise, coherent, and independent relative to the content of the original text. The evaluation process typically involves examining the summary and original document and judging whether the summary provides a comprehensive overview of the main points without unnecessary details, maintains logical coherence, and can be understood independently of the original document. The evaluation results help assess the performance of automated summarization systems and ensure that the generated summaries meet the desired standards of quality and usefulness.

Here’s what it does:

Input: It takes two pieces of information:

1. Summary: A summary of the document.
2. Original Document: The full text of the document.

Processing: It compares the summary text to the original document, focusing on whether the summary is comprehensive, concise, coherent, and independent relative to the original document.

Output: It outputs a single string:

1. “good”: If the summary is comprehensive, concise, coherent, and independent relative to the original document.
2. “bad”: If the summary is not comprehensive, concise, coherent, and independent relative to the original document.

g) Reference (Citation) Link Evaluation

Reference (Citation) Link Evaluation involves assessing the relevance and accuracy of a provided reference link in response to a query or question. It aims to determine whether the linked content adequately addresses the user’s inquiry and whether the information within the link is trustworthy and aligned with the context of the conversation or question the user poses.

Here’s what it does:

Input: It takes two pieces of information:

1. Question: The user’s specific inquiry.
2. Documentation URL Text: Extracted text of reference link.

Processing: It analyzes the documentation text to determine if it adequately addresses the questions posed by the customer in the conversation. It evaluates whether the information provided in the documentation aligns with the inquiries raised by the customer.

Output: It outputs a single word:

1. “correct”: If the documentation text answers the question the customer had in the conversation, even if it’s in a general way or partially.
2. “incorrect”: If the text does not answer the question in the conversation or doesn’t contain information that would allow you to answer the specific question.

h) AI vs. Human (Groundtruth) Evaluation

AI vs. Human (Groundtruth) Evaluation compares answers produced by artificial intelligence models against those provided by humans, typically experts or individuals with domain knowledge. It aims to determine how AI-generated responses align with the substance and completeness of human-ground truth answers.

Here’s what it does:

Input: It takes four pieces of information:

1. Question: The user’s specific inquiry.
2. Ground Truth Answer: The definitive and accurate response from an expert or trusted source.
3. AI Answer: The response generated by an artificial intelligence model in relation to the user’s specific inquiry.

Processing: It compares the AI-generated answer to the human ground truth answer, focusing on whether the AI answer correctly matches, in substance, the human answer. If the AI answer correctly answers the question or contains the main idea of the human answer, it is considered “correct”. If the AI answer diverges from or does not contain the main idea of the human answer, it is considered “incorrect”.

Output: It outputs a single word:

1. “correct”: If the AI answer correctly matches the human ground truth answer or contains the human answer’s main idea.
2. “incorrect”: If the AI answer diverges from or does not contain the main idea of the human ground truth answer.

i) SQL code evaluation

SQL code evaluation is the process of analyzing and determining the correctness of SQL queries in accordance with given criteria. It involves assessing whether SQL queries adhere to the syntax rules of the SQL language and accurately retrieve or manipulate data based on the provided database schema and the question’s requirements. The evaluation ensures that SQL queries are syntactically and semantically correct, producing accurate results without errors or deviations from the intended functionality.

Here’s what it does:

Input: It takes three pieces of information:

1. Question: The user’s specific inquiry.
2. Database Schema: The structure and organization of a SQL database, defining tables, fields, and relationships
3. SQL query: The query is generated based on the database schema.

Processing: It evaluates the SQL query against the database schema, considering syntax and semantic correctness. The goal is to determine whether the SQL query accurately reflects the information in the database schema and effectively addresses the user’s inquiry.

Output: It outputs a single word:

1. “correct”: If the SQL query is based on the database schema or assumes information available in the question.
2. “incorrect”: If the SQL query provides factually inaccurate information or does not align with the database schema relative to the question.

Conclusion

Integrating Phoenix with the Llama Index provides a complete framework for evaluating LLM models and RAG applications. By leveraging the capabilities of Phoenix, we can conduct rigorous evaluations, diagnose performance issues, and optimize the development and maintenance of LLM applications. This ensures transparency, reproducibility, and scalability in the evaluation process, contributing to the advancement of natural language processing and machine learning technologies.