<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Shivam Solanki on Medium]]></title>
        <description><![CDATA[Stories by Shivam Solanki on Medium]]></description>
        <link>https://medium.com/@Shivam_Solanki?source=rss-89c3d96ab56b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*ubQSO-BhOpXtq3c0</url>
            <title>Stories by Shivam Solanki on Medium</title>
            <link>https://medium.com/@Shivam_Solanki?source=rss-89c3d96ab56b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 22:03:33 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@Shivam_Solanki/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[JudgeIt: Evaluate Query-Rewrite Accuracy in Multi-Turn Conversations using LLM-as-a-judge]]></title>
            <link>https://medium.com/towards-generative-ai/judgeit-evaluate-query-rewrite-accuracy-in-multi-turn-conversations-using-llm-as-a-judge-2a222abace2b?source=rss-89c3d96ab56b------2</link>
            <guid isPermaLink="false">https://medium.com/p/2a222abace2b</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[generative-ai-use-cases]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[evaluation]]></category>
            <dc:creator><![CDATA[Shivam Solanki]]></dc:creator>
            <pubDate>Fri, 11 Oct 2024 05:33:49 GMT</pubDate>
            <atom:updated>2024-10-14T19:37:26.095Z</atom:updated>
            <content:encoded><![CDATA[<h4>JudgeIt: LLM-Judge Framework for Query Rewrite Evaluation in conversation memory</h4><p>You can star the GH Repo for <a href="https://github.com/ibm-ecosystem-engineering/JudgeIt-LLM-as-a-Judge">Judgeit here</a>.</p><p>Generative AI pipelines often require precise evaluation, whether it’s for Query Rewriting, RAG, or Text-to-SQL tasks. <a href="https://github.com/ibm-ecosystem-engineering/JudgeIt-LLM-as-a-Judge">JudgeIt</a> is a powerful automated framework that leverages large language models (LLMs) to evaluate the quality of these outputs. This blog will dive into how to use JudgeIt for multi-turn query rewrite evaluation and how to use the service as an application.</p><p>Cover blog for more on intro to the framework</p><p><a href="https://medium.com/@KunalSavvy/judgeit-the-most-accurate-framework-to-automate-evaluation-of-gen-ai-34785d3cff6e">JudgeIT — The Most Accurate Framework to Automate Evaluation of Gen AI</a></p><h4>Introduction to the JudgeIt Framework</h4><p><a href="https://github.com/ibm-ecosystem-engineering/JudgeIt-LLM-as-a-Judge">JudgeIt</a> is designed to automate and streamline the evaluation of generated content by comparing it to a “golden” reference. By leveraging LLMs as judges, JudgeIt provides an efficient way to evaluate outputs from generative AI models across a variety of tasks, including:</p><p><strong>RAG Evaluations</strong>: Compare generated text with reference (golden) text.<br><strong>Multi-turn Query Rewriting</strong>: Evaluate how well queries are rewritten based on a multi-turn conversation.<br><strong>Text-to-SQL Conversions</strong>: Assess the accuracy of converting natural language queries into SQL code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Woxlpovtvm4OfXmuCg5ILg.png" /><figcaption>JudgeIt Flow</figcaption></figure><p>This level of automation allows developers to batch-evaluate large datasets, saving time and ensuring a standardized evaluation process so that the Data Scientists can iterate on multiple pipeline configurations for improving accuracy.</p><h4>Multi-Turn Query Rewrite Evaluation</h4><p>Multi-turn queries are common in conversational AI, where the context of a conversation evolves over several turns. Query rewriting ensures that a question is rephrased to be self-contained and understandable without prior context.</p><blockquote>For example:<br>1. Previous Question: “What is the capital of France?”<br>2. Previous Answer: “Paris.”<br>3. Current Question: “What’s the population?”</blockquote><p>In this case, the rewritten query could be: “<em>What is the population of Paris?</em>”</p><p>JudgeIt evaluates these rewritten queries by comparing the model-generated rewrite against a “golden” or correct version using a binary scoring system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fc-K3NUprKGyE2W-ADfFrw.png" /><figcaption>Query Rewrite dataset example</figcaption></figure><p><strong>Evaluation Components:<br></strong>- Previous Question<br>- Previous Answer<br>- Current Question<br>- Golden Rewritten Current Question<br>- Model Rewritten Current Question</p><h4>Establishing Reliability</h4><p>To ensure the reliability of JudgeIt, the framework was benchmarked against human evaluators, who manually evaluated all rewritten queries. The results of these human evaluations were then compared to JudgeIt’s automated evaluations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/457/1*hvr4rbyYHwWyER8uiZPlng.png" /><figcaption>Reliability testing results</figcaption></figure><p>Key performance metrics from this comparison:<br>- Accuracy: 0.92<br>- Precision: 0.96<br>- Recall: 0.93<br>- F1 Score: 0.95</p><p>These metrics demonstrate that the LLM-based evaluation is highly reliable, closely matching human judgment in evaluating query rewrites.</p><h3>How to Use JudgeIt for Multi-Turn Evaluation</h3><p>There are two flavors to use the JudgeIt service — as a framework, and as an application.</p><h4>Configuring Your Input File</h4><p>Each type of LLM Judge accepts an Excel or CSV input file. Sample files for each evaluation type are available in the repository under JudgeIt-LLM-as-a-Judge/Framework/data/input. You can use these sample files to create and test your input.</p><p>The input file should contain the following columns:</p><p><em>| previous_question | previous_answer | current_question | golden_rewritten_question | rewritten_question |</em></p><p><strong>Note</strong>: Your input file may contain additional columns, which will not affect the evaluation but will be preserved in the output file.</p><h4><strong>Framework Approach</strong></h4><p>One approach to using JudgeIt is through the JudgeIt Python framework, which includes various Python modules designed for different LLM evaluation tasks.</p><p>Configure your parameters by editing the config.ini file. Below is an example configuration:</p><pre><br> [Default]<br> home_dir = /&lt;home_directory&gt;/JudgeIt-LLM-as-a-Judge/<br> model_id = meta-llama/llama-3–70b-instruct<br> input_file_name = Framework/data/input/sample_multi_turn_input.xlsx<br> output_file_name = Framework/data/output/sample_multi_turn_output.xlsx<br> judge_type = multi_turn_eval<br><br>[WML_CRED]<br> wml_url = https://us-south.ml.cloud.ibm.com<br> api_key = &lt;ibm_cloud_api_key&gt;<br> project_id = &lt;watsonx.ai_project_id&gt;</pre><p>Key configuration details:</p><ul><li><strong>home_dir</strong>: Path to the folder where the repository is stored.</li><li><strong>model_id</strong>: The watsonx.ai model ID used for evaluation.</li><li><strong>input_file_name</strong>: Path to the input file for the evaluation (sample files are provided in the repository).</li><li><strong>output_file_name</strong>: Path where the output file will be saved.</li><li><strong>judge_type</strong>: Specifies the type of LLM evaluation (e.g., multi_turn_eval).</li><li><strong>wml_url</strong>: Your watsonx.ai URL (e.g., https://&lt;your_region&gt;.ml.cloud.ibm.com).</li><li><strong>api_key</strong>: IBM Cloud API key for authentication.</li><li><strong>project_id</strong>: watsonx.ai project ID.</li></ul><p>Run the evaluation by executing the following command:</p><pre>python main.py</pre><p>The evaluation results will be displayed in the terminal, and a copy will be saved in /JudgeIt-LLM-as-a-Judge/Framework/data/output.</p><h4>JudgeIt App Approach</h4><p>Another approach to using JudgeIt is through the JudgeIt App which provides you with an intuitive UI where you can select the evaluation task, and either pass one example or the dataset for batch evaluation.</p><h4>Prerequisites</h4><p>The following prerequisites are required to run the JudgeIt App:</p><ol><li><a href="https://github.com/ibm-ecosystem-engineering/JudgeIt-LLM-as-a-Judge/blob/main/JudgeIt-Testing-Document.md#judgeit-rest-service">JudgeIt Backend REST Service</a> is up and running</li><li><a href="https://nodejs.org/en">Node.js</a> v18 or higher</li></ol><h4>Steps to run the JudgeIt App</h4><p>1. Change the directory into the <a href="https://github.com/ibm-ecosystem-engineering/JudgeIt-LLM-as-a-Judge">JudgeIt</a> App</p><pre>cd JudgeIt-LLM-as-a-Judge/JudgeIt-App</pre><p>2. Copy env file to .env</p><pre>cp env .env</pre><p>3. Configure your parameters in .env. Make sure `NEXT_PUBLIC_LLM_JUDGE_API_KEYvalue matches with the value assigned in the backend service.</p><p>4. Install dependencies</p><pre>npm install</pre><p>5. Run the development server</p><pre>npm run dev</pre><p>6. Open <a href="http://localhost:3000/">http://localhost:3000</a> with your browser to access the JudgeIt App.</p><p>A. Click Batch Evaluation mode</p><p>B. Then select the Multi-turn Query Rewrite Evaluation option button</p><p>C. Upload a CSV/Excel file containing the following columns:</p><p><em>| previous_question | previous_answer | current_question | golden_rewritten_question | rewritten_question |</em></p><p>D. Click on the SUBMIT button and observe the results for the number of correct and incorrect rewritten queries in a multi-turn conversation pipeline.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FhaY7_M3m6l8%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DhaY7_M3m6l8&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FhaY7_M3m6l8%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/c6c81ba278ae7bfc1cfa061d2952ba2c/href">https://medium.com/media/c6c81ba278ae7bfc1cfa061d2952ba2c/href</a></iframe><p>1. <strong>Input Data</strong>: Provide the framework with a dataset that contains multiple-turn conversations. The dataset should include the original and rewritten queries.<br>2. <strong>Model Configuration</strong>: Select the LLM model to be used for evaluating the rewritten queries. Currently, it has been tested with Llama3–70B, Mistral-Large and Llama3.1–70B.<br>3. <strong>Run the Evaluation</strong>: JudgeIt compares the generated rewrites with the golden rewrites and produces evaluation metrics.</p><h4>Conclusion</h4><p>JudgeIt provides a powerful and automated way to evaluate multi-turn query rewrites, RAG outputs, and Text-to-SQL conversions. Its use of LLM judges ensures that evaluations are fast, scalable, and reliable, reducing the reliance on manual human review. With high accuracy, precision, and recall metrics, JudgeIt offers a robust solution for AI engineers working with generative AI pipelines, allowing them to refine and improve their models with greater confidence.</p><p>If you’re working with multi-turn conversations or complex generative tasks, consider incorporating JudgeIt into your workflow to streamline evaluation processes and ensure high-quality outputs from your LLM pipelines.</p><p>Follow <a href="https://medium.com/towards-generative-ai">Towards Generative AI</a> for more content related to the latest in AI advancement.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2a222abace2b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/towards-generative-ai/judgeit-evaluate-query-rewrite-accuracy-in-multi-turn-conversations-using-llm-as-a-judge-2a222abace2b">JudgeIt: Evaluate Query-Rewrite Accuracy in Multi-Turn Conversations using LLM-as-a-judge</a> was originally published in <a href="https://medium.com/towards-generative-ai">Towards Generative AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automating NL2SQL Execution Accuracy Measurements: QueryCraft]]></title>
            <link>https://medium.com/@Shivam_Solanki/querycraft-automating-nl2sql-execution-accuracy-measurements-5d7ab8398232?source=rss-89c3d96ab56b------2</link>
            <guid isPermaLink="false">https://medium.com/p/5d7ab8398232</guid>
            <category><![CDATA[finetune-llm]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[text-to-sql]]></category>
            <category><![CDATA[nlsql]]></category>
            <category><![CDATA[genai]]></category>
            <dc:creator><![CDATA[Shivam Solanki]]></dc:creator>
            <pubDate>Mon, 10 Jun 2024 13:06:11 GMT</pubDate>
            <atom:updated>2024-08-30T13:27:25.606Z</atom:updated>
            <content:encoded><![CDATA[<h3>Automating Text-To-SQL Execution Accuracy Measurements: QueryCraft</h3><h4>Step 6B. A Step dedicated to capturing evaluation of Execution Accuracy in Text-To-SQL</h4><p>The frontier between natural language and database querying is rapidly advancing with the integration of Large Language Models (LLMs) into SQL generation tasks. Central to this integration is the robust evaluation of the generated queries. Enter the Execution Evaluator — a cornerstone component within the <a href="https://github.com/ibm-ecosystem-engineering/SuperKnowa-QueryCraft">SuperKnowa QueryCraft</a>, designed to meticulously assess the practical accuracy of SQL queries generated from text descriptions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*74aXsk-MEdO4IznmGxdN6w.png" /><figcaption>Highlighted Eval Framework of the QueryCraft pipeline</figcaption></figure><h4>Introducing the Execution Evaluator</h4><p>The Execution Evaluator is a sophisticated module poised to evaluate the precision of SQL queries crafted from natural language inputs within the NL2SQL pipeline. The module accomplishes this by analyzing the execution accuracy of queries against a database, providing a quantitative assessment that transcends mere syntactic correctness.</p><h4>Why Execution Accuracy Matters</h4><p>Execution accuracy transcends the confines of syntactic validation, venturing into the realm where queries are gauged based on their semantic integrity. This accuracy ensures the SQL queries, once executed, yield results that faithfully align with the intended user queries — essentially embodying the true essence of what was asked.</p><p>Let’s dissect the essential methods and understand their roles in evaluating execution accuracy.</p><h4>A Peek Into the Execution Evaluator’s Mechanics</h4><p>The following is the input of the Execution Evaluator service of <a href="https://github.com/ibm-ecosystem-engineering/QueryCraft/tree/main">QueryCraft</a>:</p><ul><li>Golden query dataset, which contains the natural language question and SQL query pairs</li><li>Database against which the service will run the queries</li></ul><p>Once you provide these two essential information to run the Execution Evaluator service on your database, it can generate the following outputs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/947/1*-FdJfAu6jAOzhS6fcrpBnQ.png" /><figcaption>Input-Output of Execution evaluator service</figcaption></figure><p><strong>Inference Accuracy</strong>: An evaluation that measures the query’s accuracy pre-query correction.<br><strong>Post-Processing Accuracy</strong>: An assessment that encapsulates the accuracy after the query correction process has refined the generated SQL.</p><h4>Working principle of our Execution Evaluator Module:</h4><p>Now that we understand the inputs and outputs of the Execution Evaluator module, let’s dive deep into the logic behind the scenes to calculate execution accuracy.</p><p><strong>Error Handling</strong>:<br>A fundamental part of evaluation is error handling. The error_handling function catches common SQL errors such as missing columns, syntax mistakes, and ambiguous names, categorizing them into a human-readable form. This function enhances the evaluator&#39;s robustness by gracefully capturing and reporting SQL execution issues.</p><p><strong>Query Reformatting</strong>:<br>The reformat_query function addresses a specific normalization need. It replaces table-specific wildcards liket1.* with a generic `*`. This normalization is necessary because different SQL generators may have varying conventions for representing queries, and the evaluator must compare like with like.</p><p><strong>SQL Validation</strong>:<br>Before comparing results, we must ensure the query is executable. The isValidSQL function attempts to run the SQL command. If it executes without error, the query is considered valid, signifying it has passed the first gate of evaluation.</p><p><strong>Unordered Row Comparison</strong>:<br>The unorder_row function sorts the elements within each row irrespective of their order, converting the row into a normalized tuple. This is particularly useful for comparing query results where the order of columns is not fixed, but their contents matter.</p><p><strong>Quick Rejection</strong>:<br>The quick_rej method provides a swift pathway to reject non-matching results. It compares sorted versions of result rows, either respecting their order (for ordered queries) or as unordered sets (for queries where the order is irrelevant).</p><p><strong>Permutation Generation</strong>:<br>For more nuanced cases, get_constraint_permutation and permute_tuple engage in a sophisticated match-check by considering all possible permutations of the result sets, accommodating scenarios where the columns might be returned in a different order but still represent the same data.</p><p><strong>Multiset Equality</strong>:<br>multiset_eq checks whether two lists contain the same elements, including the same number of duplicates, which is crucial for SQL queries where the result may include repeating rows.</p><p><strong>Comprehensive Result Comparison</strong>:<br>Finally, the result_eq function is the centerpiece that orchestrates the comparison process, employing all the aforementioned helpers to ascertain whether two result sets are equivalent, accounting for both the presence and order of rows and columns as necessary.</p><p><strong>Evaluation Calls</strong>:<br>- eval_exec_match_sqlite and eval_exec_match_db2 serve as wrappers to apply the evaluator logic to SQLite and IBM DB2 databases, respectively.<br>- query_processing cleans and reformats the queries and model outputs, preparing them for evaluation.<br>- formaterAndCaller_sqlite and formaterAndCaller_db2 are the higher-level functions that format the queries, call the evaluation functions, and collect the scores.</p><h4>Conclusion</h4><p>The Execution Evaluator stands as a critical pillar within the NL2SQL pipeline, ensuring the usability and accuracy of the generated SQL queries. By incorporating comprehensive error handling, flexible result comparison, and extensive normalization, it offers a thorough examination of the model’s output.</p><p>The Execution Evaluator guarantees that the final output is not just syntactically correct but also contextually accurate, capable of interacting with databases to fetch the correct data, thus validating the practical application of the generated queries. Therefore, the Execution Evaluator forms the foundation of the <a href="https://medium.com/towards-generative-ai/querycraft-evaluation-framework-for-nl2sql-generation-8c5f461e7b05">Evaluation framework</a> of our NL2SQL pipeline.</p><p>In essence, this evaluator transcends beyond a mere correctness check — it endows the NL2SQL pipeline with the reliability and robustness necessary for real-world applications, marking a significant milestone in the field of natural language to database querying.</p><p>The next step is to analyze the generated queries using our Query Analysis dashboard. Learn more about it in the <a href="https://medium.com/towards-generative-ai/exploring-llm-performance-in-text-to-sql-generation-with-query-analysis-dashboard-ccd547a020fd">next blog</a>.</p><p>Follow <a href="https://medium.com/towards-generative-ai">Towards Generative AI</a> for more on the latest advancements in AI.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5d7ab8398232" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Evaluation Framework for NL2SQL Generation: QueryCraft]]></title>
            <link>https://medium.com/towards-generative-ai/querycraft-evaluation-framework-for-nl2sql-generation-8c5f461e7b05?source=rss-89c3d96ab56b------2</link>
            <guid isPermaLink="false">https://medium.com/p/8c5f461e7b05</guid>
            <category><![CDATA[text-to-sql]]></category>
            <category><![CDATA[nlsql]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[text-2-sql]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Shivam Solanki]]></dc:creator>
            <pubDate>Mon, 10 Jun 2024 13:01:10 GMT</pubDate>
            <atom:updated>2025-10-20T20:34:23.005Z</atom:updated>
            <content:encoded><![CDATA[<h3>Evaluation Framework for Text-To-SQL Generation: QueryCraft</h3><h4>Step 6. Elevating Text-To-SQL with QueryCraft’s Evaluation Framework</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P8zzZZPu7_gYLYoQHitjhA.png" /><figcaption>Highlighted Eval framework of the QueryCraft pipeline</figcaption></figure><p>Accurate evaluation is just as crucial as the initial model training when refining the capabilities of large language models (LLMs) for NL2SQL tasks. We understand this need and have crafted an innovative evaluation framework in <a href="https://github.com/ibm-ecosystem-engineering/QueryCraft/tree/main">QueryCraft</a> to rigorously assess and refine our NL2SQL pipeline. Our framework consists of three pivotal components: <strong>Query Correction, Execution Evaluation, </strong>and<strong> the Query Analysis Dashboard.</strong></p><h4>1. Query Correction: Polishing SQL to Perfection</h4><p>The path to perfect SQL generation is fraught with syntactic snares. Our Query Correction service is the polish that smoothes out these rough edges. By rectifying syntax-level inaccuracies not caught by the LLM model, we enhance the query’s suitability for subsequent evaluation phases, ensuring the SQL output is ready to face real-world databases.</p><p>You need to use the LLM to generate inference (SQL queries) on your golden dataset (containing natural language and SQL pairs). This serves as the input to the Query Correction service (as shown in the image below).</p><p>In our case, this is a CSV file with the columns question (natural language question), context (database schema), query (SQL query), and model_op (Model output or generated query). The service works its magic, tweaking and tuning the SQL until it’s structurally sound. Please read our detailed blog on the <a href="https://medium.com/@abhilashamangal/614cd395a85b">Query Correction service</a> to learn how it works.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mdo2kMcF7cCq1fvjR1fEBg.png" /><figcaption>Query Correction Service</figcaption></figure><h4>2. Execution Evaluation: The Litmus Test of SQL Usability</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/732/1*xr4Cm8EalcnAtNF8_W8pMw.gif" /><figcaption>Execution Evaluator</figcaption></figure><p>Accuracy in SQL generation is meaningless without executable queries that yield the correct results. Our Execution Evaluation service steps in as the litmus test, analyzing the NL2SQL results based on raw inference and the refined output post-query correction.</p><p>This service computes execution accuracy by running the generated queries against the database and comparing the results with those generated by the golden queries. The metrics include exact matches, partial matches, and an insightful breakdown of errors, giving ML engineers the feedback necessary to iterate and improve.</p><h3>A Peek Into the Execution Evaluator’s Mechanics</h3><p>The following is the input of the Execution Evaluator service of <a href="https://github.com/ibm-ecosystem-engineering/QueryCraft/tree/main">QueryCraft</a>:</p><ul><li>Golden query dataset, which contains the natural language question and SQL query pairs</li><li>Database against which the service will run the queries</li></ul><p>Once you provide these two essential information to run the Execution Evaluator service on your database, it can generate the following outputs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/947/0*RfM9LESj52QCh5yD.png" /></figure><p>Input-Output of Execution evaluator service</p><p><strong>Inference Accuracy</strong>: An evaluation that measures the query’s accuracy pre-query correction.<br><strong>Post-Processing Accuracy</strong>: An assessment that encapsulates the accuracy after the query correction process has refined the generated SQL.</p><h3>Working principle of our Execution Evaluator Module:</h3><p>Now that we understand the inputs and outputs of the Execution Evaluator module, let’s dive deep into the logic behind the scenes to calculate execution accuracy.</p><p><strong>Error Handling</strong>:<br>A fundamental part of evaluation is error handling. The error_handling function catches common SQL errors such as missing columns, syntax mistakes, and ambiguous names, categorizing them into a human-readable form. This function enhances the evaluator&#39;s robustness by gracefully capturing and reporting SQL execution issues.</p><p><strong>Query Reformatting</strong>:<br>The reformat_query function addresses a specific normalization need. It replaces table-specific wildcards liket1.* with a generic `*`. This normalization is necessary because different SQL generators may have varying conventions for representing queries, and the evaluator must compare like with like.</p><p><strong>SQL Validation</strong>:<br>Before comparing results, we must ensure the query is executable. The isValidSQL function attempts to run the SQL command. If it executes without error, the query is considered valid, signifying it has passed the first gate of evaluation.</p><p><strong>Unordered Row Comparison</strong>:<br>The unorder_row function sorts the elements within each row irrespective of their order, converting the row into a normalized tuple. This is particularly useful for comparing query results where the order of columns is not fixed, but their contents matter.</p><p><strong>Quick Rejection</strong>:<br>The quick_rej method provides a swift pathway to reject non-matching results. It compares sorted versions of result rows, either respecting their order (for ordered queries) or as unordered sets (for queries where the order is irrelevant).</p><p><strong>Permutation Generation</strong>:<br>For more nuanced cases, get_constraint_permutation and permute_tuple engage in a sophisticated match-check by considering all possible permutations of the result sets, accommodating scenarios where the columns might be returned in a different order but still represent the same data.</p><p><strong>Multiset Equality</strong>:<br>multiset_eq checks whether two lists contain the same elements, including the same number of duplicates, which is crucial for SQL queries where the result may include repeating rows.</p><p><strong>Comprehensive Result Comparison</strong>:<br>Finally, the result_eq function is the centerpiece that orchestrates the comparison process, employing all the aforementioned helpers to ascertain whether two result sets are equivalent, accounting for both the presence and order of rows and columns as necessary.</p><p><strong>Evaluation Calls</strong>:<br>- eval_exec_match_sqlite and eval_exec_match_db2 serve as wrappers to apply the evaluator logic to SQLite and IBM DB2 databases, respectively.<br>- query_processing cleans and reformats the queries and model outputs, preparing them for evaluation.<br>- formaterAndCaller_sqlite and formaterAndCaller_db2 are the higher-level functions that format the queries, call the evaluation functions, and collect the scores.</p><p>The output of the Query correction service serves as the input of the Execution evaluator service (as shown in the image below). This input is a CSV file with the following columns: question (natural language question), context (database schema), query (SQL query), model_op (Model output or generated query), and optionally, model_op1 for the query correction output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/945/1*RCQREy4A-26cYaV1_7n1pg.png" /><figcaption>Execution Evaluator</figcaption></figure><h4>3. Query Analysis Dashboard: Insights at a Glance</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/678/1*Nu-lzIewv9tVH2JKma98-g.gif" /><figcaption>Query Analysis Dashboard</figcaption></figure><p>Evaluation data is only as good as the insights it offers. Our Query Analysis Dashboard encapsulates this ideology by serving as a one-stop visualization tool for examining generated queries, categorizing inaccuracies, and benchmarking the results across multiple LLMs. It’s an invaluable resource for identifying areas for model improvement with features that include:</p><p><strong>Query Type Analysis</strong>: Analyze where your model excels and where it stumbles.<br><strong>Error Analysis</strong>: Pinpoint and categorize the exact nature of errors in query generation.<br><strong>Result Analysis</strong>: Assess the overall success rate of query execution.<br><strong>Execution Accuracy Metrics</strong>: Measure the precision of your model’s output before and after query correction.</p><p>The output of the Execution evaluation service is fed into the Query analysis dashboard. This is a CSV file containing the evaluation service results with columns like question, context, query, model_op, and optionally, model_op1, error_type, and result.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g1VIMBpHUQgxePPewSoCQg.png" /><figcaption>Query Analysis dashboard service</figcaption></figure><p>Please read our detailed blog on the <a href="https://medium.com/@abhilashamangal/ccd547a020fd">Query Analysis Dashboard</a> to learn more about it.</p><h4>Conclusion</h4><p>In conclusion, <a href="https://github.com/ibm-ecosystem-engineering/QueryCraft/tree/main">QueryCraft’s</a> evaluation framework empowers developers to refine their NL2SQL pipeline rapidly. By incorporating Query Correction, Execution Evaluation, and the Query Analysis Dashboard, our framework provides a comprehensive suite of tools to identify and rectify errors, measure success, and gain valuable insights into model behavior.</p><p>This multifaceted approach ensures that generated SQL queries are not just syntactically correct but also functionally accurate and executable against real-world databases. The Query Analysis Dashboard’s clear visualization capabilities further accelerate the development process by allowing ML engineers to pinpoint areas for improvement and benchmark the performance of various large language models.</p><p><a href="https://github.com/ibm-ecosystem-engineering/QueryCraft/tree/main">QueryCraft’s</a> evaluation framework is a powerful asset for anyone developing or refining NL2SQL models. By leveraging its capabilities, you can ensure that your models generate grammatically correct SQL and queries that deliver the desired results on real databases.</p><p><strong>Ready to take your </strong>NL2SQL<strong> models to the next level?</strong> Explore <a href="https://github.com/ibm-ecosystem-engineering/QueryCraft/tree/main">QueryCraft’s</a> evaluation framework today and unlock the full potential of your LLMs!</p><p>For a deeper dive into the specific functionalities of each evaluation component, please refer to our detailed blogs on <a href="https://medium.com/towards-generative-ai/improving-text-to-sql-accuracy-with-query-correction-service-614cd395a85b">Query Correction</a>, <a href="https://medium.com/@Shivam_Solanki/text-to-sql-execution-accuracy-5d7ab8398232">Execution Evaluation</a>, and the <a href="https://medium.com/@abhilashamangal/ccd547a020fd">Query Analysis Dashboard</a>.</p><p>Follow <a href="https://medium.com/towards-generative-ai">Towards Generative AI</a> for more on the latest advancements in AI.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8c5f461e7b05" width="1" height="1" alt=""><hr><p><a href="https://medium.com/towards-generative-ai/querycraft-evaluation-framework-for-nl2sql-generation-8c5f461e7b05">Evaluation Framework for NL2SQL Generation: QueryCraft</a> was originally published in <a href="https://medium.com/towards-generative-ai">Towards Generative AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[QueryCraft: How to Ingest your own Data for Text-To-SQL]]></title>
            <link>https://medium.com/towards-generative-ai/querycraft-how-to-ingest-your-own-data-for-nl2sql-a9e0276e8e8a?source=rss-89c3d96ab56b------2</link>
            <guid isPermaLink="false">https://medium.com/p/a9e0276e8e8a</guid>
            <category><![CDATA[generative-ai-use-cases]]></category>
            <category><![CDATA[nlsql]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Shivam Solanki]]></dc:creator>
            <pubDate>Mon, 10 Jun 2024 12:22:33 GMT</pubDate>
            <atom:updated>2024-08-30T13:28:53.798Z</atom:updated>
            <content:encoded><![CDATA[<h4>Step 1. Data Ingestion in QueryCraft to Streamline SQL Generation</h4><p>Welcome to a deeper dive into the <a href="https://github.com/ibm-ecosystem-engineering/SuperKnowa-QueryCraft/blob/main/document/Step%201.%20Data%20Ingestion.md">Data Ingestion step</a> of the QueryCraft pipeline, an essential component of the <a href="https://github.com/ibm-ecosystem-engineering/SuperKnowa-QueryCraft/tree/main">SuperKnowa-QueryCraft framework</a>. QueryCraft offers an innovative approach to adapting large language models (LLMs) for the generation of SQL queries from natural language, simplifying complex SQL scripting for developers and business analysts.</p><h4>Why Data Ingestion Matters</h4><p>Data ingestion is the first critical step in setting up a robust Text2SQL pipeline. It involves importing your data into a suitable environment where the LLM can interact with it. This process not only feeds the necessary data into the system but also sets the stage for all subsequent operations, from context retrieval to query generation and corrections.</p><h4>How QueryCraft Handles Data Ingestion</h4><p><a href="https://github.com/ibm-ecosystem-engineering/QueryCraft/tree/main">QueryCraft</a> provides flexible options for data ingestion, accommodating</p><ol><li>delimited file formats and</li><li>SQLite databases.</li></ol><p>This versatility ensures that users can integrate <a href="https://github.com/ibm-ecosystem-engineering/QueryCraft/tree/main">QueryCraft</a> seamlessly into any of their existing tech stacks without the need for extensive modifications.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*zdAestcW7euXGn9geYhpsg.gif" /><figcaption>Configuring Data Ingestion in QueryCraft</figcaption></figure><p><strong>I. Delimited File Ingestion:<br></strong>For those utilizing cloud databases like Db2 on IBM Cloud, <a href="https://github.com/ibm-ecosystem-engineering/QueryCraft/tree/main">QueryCraft</a> simplifies the ingestion of delimited files. Users can directly upload their data files, which are then ingested into the database to be utilized throughout the <a href="https://github.com/ibm-ecosystem-engineering/QueryCraft/tree/main">QueryCraft</a> pipeline. This approach not only ensures that all data necessary for generating and refining SQL queries is readily available but also streamlines the setup process for subsequent modules.</p><p>The overall input and output of this service is shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/830/1*WfR_fnEKxFms7AUiynGCgg.png" /><figcaption>Input and Output of Data Ingestion service.</figcaption></figure><p>Here are the steps that you can follow for Data Ingestion to Db2 on IBM Cloud:</p><p><em>Prerequisites (First time activity):</em></p><ul><li>Access to IBM Cloud. You can create a free account: <a href="https://cloud.ibm.com/registration">https://cloud.ibm.com/registration</a></li><li>Access to a DB2 database. You can provision a free instance: <a href="https://cloud.ibm.com/catalog/services/db2">https://cloud.ibm.com/catalog/services/db2</a></li><li>Service credentials for the DB2 database. Get the db2 credentials from the IBM cloud by following the steps here: https://cloud.ibm.com/docs/Db2onCloud?topic=Db2onCloud-getting-started</li></ul><p>The <a href="https://github.com/ibm-ecosystem-engineering/SuperKnowa-QueryCraft/blob/main/code/db2_Ingestion.py">db2_Ingestion</a> module offers a streamlined method for inserting data from CSV or any delimiter file into db2 to fine-tune NL2SQL pipelines.</p><p>1. First, set the following credentials in the expertConfig.ini file under the Db2_credentials section:</p><ul><li><strong>dsn_database</strong>: Name of the database.</li><li><strong>dsn_uid</strong>: User ID for the database.</li><li><strong>dsn_pwd</strong>: Password for the database.</li><li><strong>dsn_hostname</strong>: Hostname or IP address of the database server.</li><li><strong>dsn_port</strong>: Port number of the database server.</li><li><strong>dsn_protocol</strong>: Protocol used for communication.</li><li><strong>dsn_driver</strong>: Driver used for database connection.</li></ul><p>2. If you don’t have delimited files for your database, which also contains a golden query dataset, you can use a file from the input/dataset folder.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/286/1*R8Sh3vQNzEyCL-qM1-luAQ.png" /><figcaption>Provided example dataset in QuerCraft</figcaption></figure><p>3. Now specify the file path, including the file name, in the <a href="https://github.com/ibm-ecosystem-engineering/SuperKnowa-QueryCraft/blob/main/simpleConfig.ini">simpleConfig</a> file under the DataIngestion section. Additionally, indicate the table name that needs to be created in the db2 database. If you are using the salary.csv, TheHistoryofBaseball is the right schema. Ensuring the right schema is important as the Golden query dataset contains this information in the column db_id. This is required to run the context retriever and the execution evaluation service.</p><pre>#Relative path (from home_dir) of csv file to be ingested in db2 table<br>#CSV file for Loading<br>#filename = ../input/datasets/people.csv<br>filename = input/datasets/salary.csv<br>#Schema name - Database<br>schema_name = TheHistoryofBaseball<br># Table name for CSV data<br>table_name= querycraft_db2_test</pre><p>If the user needs to import a file specifying the delimiter for files other than CSV, the user can adjust the delimiter from the <a href="https://github.com/ibm-ecosystem-engineering/SuperKnowa-QueryCraft/blob/main/expertConfig.ini">expertConfig</a> file:</p><p>delimiter=,</p><p><strong>Usage:</strong></p><p>Run the Data Ingestion module of the QueryCraft pipeline using the <a href="https://github.com/ibm-ecosystem-engineering/SuperKnowa-QueryCraft/blob/main/runQueryCraft.sh">runQueryCraft.sh</a>, file with the dataIngestion option after setting the <a href="https://github.com/ibm-ecosystem-engineering/SuperKnowa-QueryCraft/blob/main/simpleConfig.ini">simpleConfig</a> file to insert salary.csv into the querycraft_db2_test table in db2.</p><p>sh runQueryCraft.sh</p><p>Enter the name of the component you want to run:</p><p>dataIngestion</p><p>You can validate the successful execution of the dataIngestion module from the DB2 UI as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*CaqA69s2UDw0fRhFex3JrA.png" /><figcaption>Db2 UI validation for data ingestion</figcaption></figure><p><strong>II. SQLite Ingestion:<br></strong>Alternatively, users can opt for SQLite ingestion if they prefer a more lightweight or localized database solution. This method supports uploading a database dump in `.sqlite` format, making it ideal for portable or standalone applications of the QueryCraft pipeline.</p><p><strong>Setting the Stage for Success</strong></p><p>Once data is ingested, QueryCraft utilizes this foundation to perform a range of tasks:</p><ul><li><strong>Context Retrieval</strong>: Extracting relevant schema and metadata to understand the database structure better.</li><li><strong>Fine-Tuning</strong>: Adjusting the LLM’s parameters to optimize SQL query generation based on the specific characteristics of the ingested data.</li><li><strong>Inference and Correction</strong>: Generating SQL queries from natural language and refining them to ensure accuracy and efficiency.</li></ul><h4>The Impact of Efficient Data Ingestion</h4><p>Effective data ingestion not only enhances the performance of the SQL generation process but also significantly reduces the time and effort required to prepare data for complex queries. By automating and simplifying this initial step, <a href="https://github.com/ibm-ecosystem-engineering/QueryCraft/tree/main">QueryCraft</a> allows users to focus more on strategic tasks such as data analysis and decision-making rather than on the technical nuances of data preparation.</p><h4>Conclusion</h4><p>In the realm of database management and query generation, the ease and efficiency of setting up an NL2SQL pipeline are paramount. QueryCraft’s approach to data ingestion exemplifies its commitment to making advanced AI tools accessible and practical for a broad audience. Whether you are a seasoned database professional or a business analyst seeking to leverage the power of LLMs, <a href="https://github.com/ibm-ecosystem-engineering/QueryCraft/tree/main">QueryCraft</a> provides a user-friendly and powerful toolset to transform natural language queries into precise SQL commands.</p><p>For those interested in exploring further, QueryCraft&#39;s full capabilities extend beyond data ingestion to offer a comprehensive suite of tools designed to enhance every aspect of SQL query generation and optimization.</p><p>Explore more about <a href="https://github.com/ibm-ecosystem-engineering/SuperKnowa-QueryCraft/tree/main">QueryCraft</a> and stay tuned for our next post on the intricacies of <a href="https://medium.com/@sahil_desai/07b5f4b51273">Context Retrieval</a> in the QueryCraft pipeline.</p><p>Follow <a href="https://medium.com/towards-generative-ai">Towards Generative AI</a> for more on the latest advancements in AI.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a9e0276e8e8a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/towards-generative-ai/querycraft-how-to-ingest-your-own-data-for-nl2sql-a9e0276e8e8a">QueryCraft: How to Ingest your own Data for Text-To-SQL</a> was originally published in <a href="https://medium.com/towards-generative-ai">Towards Generative AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[LLM Evaluation Toolkit for RAG Pipelines]]></title>
            <link>https://medium.com/towards-generative-ai/llm-evaluation-toolkit-for-rag-pipelines-9c37eb543614?source=rss-89c3d96ab56b------2</link>
            <guid isPermaLink="false">https://medium.com/p/9c37eb543614</guid>
            <category><![CDATA[evaluation]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[mlflow]]></category>
            <category><![CDATA[rag-evaluation]]></category>
            <dc:creator><![CDATA[Shivam Solanki]]></dc:creator>
            <pubDate>Mon, 21 Aug 2023 15:02:06 GMT</pubDate>
            <atom:updated>2023-08-21T15:02:06.968Z</atom:updated>
            <content:encoded><![CDATA[<h4>Introducing a novel SuperKnowa framework for evaluating LLM-based RAG pipelines and automatically generating leaderboard</h4><p>In the rapidly evolving field of Natural Language Processing (NLP), Retriever-Augmented Generation (RAG) pipelines have emerged as a powerful approach to question-answering tasks. We have covered the different building blocks of the RAG pipeline in detail in our previous <a href="https://medium.com/towards-generative-ai/improving-rag-retrieval-augmented-generation-answer-quality-with-re-ranker-55a19931325">blogs</a>. The ability to evaluate these models effectively is crucial for ongoing development and research. In this blog post, we’ll explore an evaluation framework that we have specifically developed for RAG pipelines, offering insights into its structure, purpose, and key components.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Rq2AKaorE0EIXbF9.jpeg" /><figcaption>Evaluation pipeline</figcaption></figure><h4>Purpose and Need for Evaluation</h4><p>Evaluating RAG pipelines is essential for several reasons:</p><p><strong>Model Comparison</strong>: Understanding how different models perform under similar conditions.<br><strong>Performance Tuning</strong>: Identifying areas for improvement and optimizing model parameters.<br><strong>Insights and Analysis</strong>: Gaining insights into how models are functioning, what they are getting right, and where they are going wrong.<br><strong>Collaboration and Sharin</strong>g: Facilitating collaboration between researchers, data scientists, and practitioners by standardizing the evaluation process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*jtONdViLXpYCH7qu" /><figcaption>Leaderboard</figcaption></figure><p>Our evaluation framework for RAG pipelines includes a comprehensive toolkit designed to make the evaluation process more streamlined and insightful. Here’s an overview of our <a href="https://github.com/ibm-ecosystem-engineering/SuperKnowa/tree/main/5.%20LLM%20Model%20Evaluations/I.%20LLM%20Eval%20Toolkit">evaluation toolkit</a>:</p><p>1. <a href="https://github.com/ibm-ecosystem-engineering/SuperKnowa/blob/main/5.%20LLM%20Model%20Evaluations/I.%20LLM%20Eval%20Toolkit/Eval_Package/config.json"><strong>Configurable JSON File</strong></a>: Allows users to specify various parameters, settings, and configurations.<br>2. <a href="https://github.com/ibm-ecosystem-engineering/SuperKnowa/blob/main/5.%20LLM%20Model%20Evaluations/I.%20LLM%20Eval%20Toolkit/mlflow_package/mlflow_utils_v2.py"><strong>MLFlow Integration</strong></a>: Offers capabilities for tracking experiments, sharing insights, and collaborating across teams.<br>3. <a href="https://github.com/ibm-ecosystem-engineering/SuperKnowa/blob/main/5.%20LLM%20Model%20Evaluations/I.%20LLM%20Eval%20Toolkit/mlflow_package/streamlit_app.py"><strong>Visualization Tools</strong></a>: Provides interactive dashboards and visualizations for in-depth analysis.<br>4. <a href="https://github.com/ibm-ecosystem-engineering/SuperKnowa/blob/main/5.%20LLM%20Model%20Evaluations/I.%20LLM%20Eval%20Toolkit/Eval_Package/eval_script.py"><strong>Customizable Evaluation Metrics</strong></a>: Supports a wide range of evaluation metrics like BLEU, METEOR, ROUGE, and more.<br>5. <a href="https://github.com/ibm-ecosystem-engineering/SuperKnowa/tree/main/5.%20LLM%20Model%20Evaluations"><strong>RAG Model Evaluation</strong></a>: A detailed guide for evaluating the RAG pipeline using various datasets, models, and evaluation metrics.</p><h4>Datasets Used for Evaluation</h4><p>Evaluating the SuperKnowa Model requires diverse and representative datasets. Here’s an overview of the datasets utilized:</p><p>1. <strong>IBM Test Data</strong>: Enterprise-related questions and answers on IBM products.<br>2. <strong>CoQA Data</strong>: Conversational question-answering dataset.<br>5. <strong>QuAC Data</strong>: Questions on academic content.<br>6. <strong>TidyQA Data</strong>: General knowledge questions and answers.</p><h4>Models Utilized for Comparison</h4><p>To gauge the RAG Model’s effectiveness, it is compared against various other models, including but not limited to:</p><p><strong>COGA</strong>: IBM trained foundation model fine-tuned for Conversational Question Answering<br><a href="https://huggingface.co/google/flan-t5-xxl"><strong>FlanT5-XXL</strong>:</a> A larger variant of the FlanT5 model.<br><a href="https://huggingface.co/bigscience/bloom"><strong>Bloom</strong></a>: BigScience Large Open-science Open-access Multilingual Language Model</p><h4>Evaluation Metrics and Scores</h4><p>A range of evaluation metrics is employed to provide a comprehensive assessment of the SuperKnowa Model:</p><p><strong>BLEU Score</strong>: Measures the similarity between a generated sentence and a reference sentence.<br><strong>METEOR Score</strong>: Considers synonyms, stemming, and more.<br><strong>ROUGE Score</strong>: Measures the overlap between a generated summary and a reference summary.<br><strong>SentenceSim Score</strong>: Compares semantic similarity between sentences.<br><strong>SimHash Score</strong>: Used for duplicate detection.<br><strong>Perplexity Score</strong>: Measures how well the model predicts the sample.<br><strong>BLEURT Score</strong>: Evaluates text generation quality.<br><strong>F1 Score</strong>: Harmonic mean of precision and recall.<br><strong>BERT Score</strong>: Considers contextual embeddings from BERT.</p><h4>Experiment Parameters</h4><p>The evaluation process is controlled by various parameters which can be specified in the <a href="https://github.com/ibm-ecosystem-engineering/SuperKnowa/blob/e84c4461aac444099eac067c9dbbe794a0540f61/5.%20LLM%20Model%20Evaluations/I.%20LLM%20Eval%20Toolkit/Eval_Package/config.json">config file</a>. These parameters may include:</p><p><strong>Model Name</strong>: Specifies the model being evaluated.<br><strong>Temperature</strong>: Controls randomness in model output.<br><strong>Top P</strong>: Influences the selection of top predictions.<br><strong>Dataset</strong>: Identifies the dataset used for evaluation.</p><h4>Introduction to MLFLow Package</h4><p>The <a href="https://github.com/ibm-ecosystem-engineering/SuperKnowa/tree/e84c4461aac444099eac067c9dbbe794a0540f61/5.%20LLM%20Model%20Evaluations/I.%20LLM%20Eval%20Toolkit/mlflow_package">MLFlow package</a> serves as an essential utility within the RAG pipeline evaluation framework. It’s designed to enhance and extend MLFlow’s functionalities, providing specific methods and functions tailored to the evaluation process’s needs. Whether you’re managing experiments, logging metrics, or visualizing results, this script offers valuable capabilities.</p><h4>Key Functions and Utilities</h4><p>Here’s a look at some of the key functions and utilities within the <a href="https://github.com/ibm-ecosystem-engineering/SuperKnowa/tree/e84c4461aac444099eac067c9dbbe794a0540f61/5.%20LLM%20Model%20Evaluations/I.%20LLM%20Eval%20Toolkit/mlflow_package">MLFlow package</a> script:</p><ol><li><a href="https://github.com/ibm-ecosystem-engineering/SuperKnowa/blob/e84c4461aac444099eac067c9dbbe794a0540f61/5.%20LLM%20Model%20Evaluations/I.%20LLM%20Eval%20Toolkit/mlflow_package/mlflow_utils_v2.py">Experiment Management</a>: Functions to create, list, and manage MLFlow experiments.<br>2. <a href="https://github.com/ibm-ecosystem-engineering/SuperKnowa/blob/e84c4461aac444099eac067c9dbbe794a0540f61/5.%20LLM%20Model%20Evaluations/I.%20LLM%20Eval%20Toolkit/mlflow_package/mlflow_utils_v2.py">Metric Logging</a>: Methods for logging metrics, parameters, and artifacts in a standardized manner.<br>3. <a href="https://github.com/ibm-ecosystem-engineering/SuperKnowa/blob/e84c4461aac444099eac067c9dbbe794a0540f61/5.%20LLM%20Model%20Evaluations/I.%20LLM%20Eval%20Toolkit/mlflow_package/mlflow_utils_v2.py">Model Tracking</a>: Tools to log and manage models, including versioning and comparison.<br>4. <a href="https://github.com/ibm-ecosystem-engineering/SuperKnowa/blob/e84c4461aac444099eac067c9dbbe794a0540f61/5.%20LLM%20Model%20Evaluations/I.%20LLM%20Eval%20Toolkit/mlflow_package/streamlit_app.py">Result Visualization</a>: Utilities to visualize results using various plots and charts.<br>5. Run Management: Functions to start, monitor, and end MLFlow runs.</li></ol><h4>Benefits and Usage of the MLFlow Package</h4><p>The MLFlow package offers several benefits that align with the evaluation framework’s goals:</p><p><strong>Standardization</strong>: Provides a consistent approach to experiment management, metric logging, and result visualization.<br><strong>Extensibility</strong>: Can be customized and extended to fit specific evaluation needs and scenarios.<br><strong>Integration</strong>: Seamlessly integrates with the existing RAG pipeline evaluation framework, enhancing MLFlow’s capabilities.<br><strong>Efficiency</strong>: Simplifies repetitive tasks, saving time and effort.</p><h3>Evaluation package step-by-step guide</h3><h4>Step1: Setup Evaluation package for the RAG pipeline</h4><p>Follow the steps below to install the best evaluation package for LLM based RAG pipeline.</p><ol><li>Get the code base: Clone the <a href="https://github.com/ibm-ecosystem-engineering/SuperKnowa">repo</a>, cd to the directory <a href="https://github.com/ibm-ecosystem-engineering/SuperKnowa/tree/main/5.%20LLM%20Model%20Evaluations/I.%20LLM%20Eval%20Toolkit/Eval_Package">5. LLM Model Evaluations/I. LLM Eval Toolkit/Eval_Package</a></li><li>Install dependencies</li></ol><pre>pip3 install -r setup.txt</pre><p>3. Set up the <a href="https://github.com/ibm-ecosystem-engineering/SuperKnowa/blob/main/5.%20LLM%20Model%20Evaluations/I.%20LLM%20Eval%20Toolkit/Eval_Package/config.json">JSON configuration file</a> for evaluating models. The first part is the model config file containing values of your API key, the name of the model, etc.</p><pre>{<br>  &quot;model&quot;: {<br>    &quot;watsonxai_token&quot;: &quot;Bearer Your-API-Key&quot;,<br>    &quot;model_name&quot;: &quot;google/flan-t5-xxl&quot;<br>  },<br>  ...<br>}</pre><p>The second part is the data config file containing parameters related to the input data and evaluation.</p><pre>{<br>  ...<br>  &quot;data&quot;: {<br>    &quot;data_path&quot;: &quot;path/to/dataset&quot;,<br>    &quot;question&quot;: &quot;instruction&quot;,<br>    &quot;context&quot;: &quot;input&quot;,<br>    &quot;idea_answer&quot;: &quot;output&quot;,<br>    &quot;q_num&quot;: 5<br>  },<br>  ...<br>}</pre><p>Finally, configure the result config part to define the file where the evaluation results will be saved.</p><p>4. Run the <a href="https://github.com/ibm-ecosystem-engineering/SuperKnowa/blob/main/5.%20LLM%20Model%20Evaluations/I.%20LLM%20Eval%20Toolkit/Eval_Package/eval_script.py">evaluation script</a></p><pre>python3 eval_script.py</pre><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2F6FV33YFa0HxuDxjPv9%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2Fv1.Y2lkPTc5MGI3NjExYjJod3pnZG1hbjloano0cmhxZ3JrbDJsbXgxM2t5enF5eXI3cWZvaSZlcD12MV9pbnRlcm5hbF9naWZfYnlfaWQmY3Q9Zw%2F6FV33YFa0HxuDxjPv9%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2F6FV33YFa0HxuDxjPv9%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="244" frameborder="0" scrolling="no"><a href="https://medium.com/media/2098f7cc12fba278c14b820aa932031c/href">https://medium.com/media/2098f7cc12fba278c14b820aa932031c/href</a></iframe><h4>Step2: Infuse MLFlow package in the RAG pipeline</h4><p>These steps will quickly guide you through the process of infusing MLFlow package to easily manage and track evaluation experiments for your RAG pipeline.</p><p>1. Install dependencies:</p><pre>python3 setup.py install</pre><p>2. Import the library</p><pre>import mlflow_utils</pre><p>3. Set the path of your evaluation directory: Set the path of the directory which contains the evaluation results from the RAG pipeline</p><pre>path=&quot;set the path of your evalution result DIR&quot;<br>mlflow_utils.run_mlflow(path)</pre><p>4. Start the MLFLow dashboards locally</p><pre>mlflow ui</pre><p>You can observe the leaderboard on <a href="http://127.0.0.1:5000/">http://127.0.0.1:5000</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jQvxd7o0ep2tlgyQ.png" /><figcaption>Evaluation Leaderboard</figcaption></figure><h3>Conclusion and Recommendations</h3><p>The SuperKnowa Model evaluation package offers a robust and insightful process to understand the model’s performance, strengths, and areas for improvement. By employing various datasets, models, evaluation metrics, and experiment parameters, this evaluation package serves as a comprehensive tool for researchers, data scientists, and practitioners. Here are some recommendations for those looking to explore further:</p><p><strong>Key takeaways include:</strong></p><p><strong>Versatility</strong>: The ability to evaluate the RAG pipeline across different domains and contexts.<br> <strong>Comparative Analysis</strong>: Understanding how the LLM Model stacks against other state-of-the-art models.<br> <strong>In-Depth Insights</strong>: Gaining granular insights into model behavior and performance.<br> <strong>Scalability</strong>: Adapting the evaluation process to different models and tasks.</p><p>The SuperKnowa Model evaluation within the RAG pipelines framework exemplifies a structured and methodical approach to model assessment. Explore the provided notebook, datasets, and guides to delve deeper into the evaluation process and harness the full potential of the SuperKnowa Model.</p><p>Thanks to <a href="https://medium.com/u/c90aacc01308">Sahil Desai</a> for developing the evaluation package of our RAG pipeline. The full code and implementation details can be found on the <a href="https://github.com/ibm-ecosystem-engineering/SuperKnowa/tree/main">GitHub Repo</a>.</p><p>Follow <a href="https://medium.com/towards-generative-ai">Towards Generative AI</a> for more content related to the latest in AI advancement.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9c37eb543614" width="1" height="1" alt=""><hr><p><a href="https://medium.com/towards-generative-ai/llm-evaluation-toolkit-for-rag-pipelines-9c37eb543614">LLM Evaluation Toolkit for RAG Pipelines</a> was originally published in <a href="https://medium.com/towards-generative-ai">Towards Generative AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding In-Context Learning for Language Models]]></title>
            <link>https://medium.com/towards-generative-ai/understanding-in-context-learning-for-language-models-7086747b8512?source=rss-89c3d96ab56b------2</link>
            <guid isPermaLink="false">https://medium.com/p/7086747b8512</guid>
            <category><![CDATA[nlg]]></category>
            <category><![CDATA[question-answering]]></category>
            <category><![CDATA[prompt-engineering]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[in-context-learning]]></category>
            <dc:creator><![CDATA[Shivam Solanki]]></dc:creator>
            <pubDate>Fri, 11 Aug 2023 17:51:09 GMT</pubDate>
            <atom:updated>2023-08-11T18:48:49.729Z</atom:updated>
            <content:encoded><![CDATA[<h4>In-context learning for RAG (Retrieval Augmented Generation) pipeline</h4><p>As AI continues to evolve, the ability of models to learn and understand context is becoming increasingly important. In the field of Natural Language Processing (NLP), one of the most exciting developments is the capacity for Language Models (LMs) to grasp and generate responses based on context. This concept, known as “In-context Learning,” is revolutionizing the way we interact with AI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/370/0*epr0TzFpJaZNixdI" /><figcaption>In-context Learning (Source — Papers with Code)</figcaption></figure><p><strong>What is In-Context Learning?</strong></p><p>In-context learning is the ability of an AI model to generate responses or make predictions based on the specific context provided to it. This context is usually in the form of preceding text or a prompt. In the realm of LMs, this context comes in the form of a text passage or a conversation history, which the model uses to generate appropriate responses.</p><p>For instance, if you feed the model a context about DataOps and ask it a question within that context, the model will generate a response that fits within the given context.</p><p><strong>How does In-context learning fits in the RAG pipeline?</strong></p><p>At a high level, RAG involves two key steps. First, given an input prompt, it uses a retriever to fetch relevant documents from a corpus of knowledge. Then, it feeds these documents, along with the original prompt, to a Large Language model which generates a response. This allows the model to pull in relevant information from the document to pass it as a context generating its response. The In-context learning component of the RAG pipeline is shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7foxs_a6YBLECyc3_XSEkg.png" /><figcaption>RAG pipeline</figcaption></figure><p><strong>Implementing In-Context Learning with a Language Model</strong></p><p>To illustrate how in-context learning works, let’s look at a simple Python implementation. The script prompts the user for a context and a question based on that context, and then uses an AI model to generate an answer to the question.</p><p>Here’s a snippet of the Python code used:</p><pre>def get_input(prompt=&quot;Please enter your question: &quot;):<br> return input(prompt)<br>def get_context(prompt=&quot;Please enter the context: &quot;):<br> return input(prompt)<br>def process_watsonx_request():<br> question = get_input()<br> context = get_context()<br> #…</pre><p>You can find the full script on <a href="https://github.com/ibm-ecosystem-engineering/SuperKnowa/tree/main/4.%20In-context%20learning%20using%20LLM">GitHub</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YvLoK8pUrT41-eoGnJwa3g.png" /></figure><p><strong>Seeing It In Action</strong></p><p>Here’s an example of how it works:</p><pre>&gt;&gt;&gt; process_bam_request()<br>Please enter the context: DataOps is a collaborative data management discipline that focuses on end-to-end data management and the elimination of data silos…<br>Please enter your question: What are the benefits of DataOps?</pre><p>And the script will generate an answer like:</p><p>“DataOps offers several benefits including decreasing the cycle time in deploying analytical solutions, lowering data defects, reducing the time required to resolve data defects, and minimizing data silos.”</p><p><strong>Conclusion</strong></p><p>In-context learning for LLMs is a powerful tool in the field of AI. It allows models to generate more relevant and accurate responses by considering the context in which a question or prompt is given. This leads to more natural and meaningful interactions between humans and AI, opening up exciting possibilities for the future of AI communication.</p><p>Follow <a href="https://medium.com/towards-generative-ai">Towards Generative AI</a> for more on the latest in AI.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7086747b8512" width="1" height="1" alt=""><hr><p><a href="https://medium.com/towards-generative-ai/understanding-in-context-learning-for-language-models-7086747b8512">Understanding In-Context Learning for Language Models</a> was originally published in <a href="https://medium.com/towards-generative-ai">Towards Generative AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Improving RAG (Retrieval Augmented Generation) Answer Quality with Re-ranker]]></title>
            <link>https://medium.com/towards-generative-ai/improving-rag-retrieval-augmented-generation-answer-quality-with-re-ranker-55a19931325?source=rss-89c3d96ab56b------2</link>
            <guid isPermaLink="false">https://medium.com/p/55a19931325</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[nlg]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[generative-ai-use-cases]]></category>
            <category><![CDATA[colbert]]></category>
            <dc:creator><![CDATA[Shivam Solanki]]></dc:creator>
            <pubDate>Fri, 04 Aug 2023 16:30:05 GMT</pubDate>
            <atom:updated>2023-08-27T02:49:17.256Z</atom:updated>
            <content:encoded><![CDATA[<h4>Implementing the Re-ranker algorithm in the RAG pipeline</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/250/0*t-lP7v9-d-VnzcNF.png" /><figcaption>Ranking model. Credit — <a href="https://en.wikipedia.org/wiki/Learning_to_rank">https://en.wikipedia.org/wiki/Learning_to_rank</a></figcaption></figure><p>Artificial intelligence has significantly improved over the years, and AI models have become masterminds at answering questions based on a given context.</p><p>Human-like Q&amp;A accuracy is advantageous when researching and extracting relevant information from a large set of documents corpus, like IBM’s Watson Discovery (Elastic Search, Solr, etc.), which allows us to cast a broad net to capture snippets of text pertinent to a specific query. In the previous <a href="https://medium.com/towards-generative-ai/unleashing-the-power-of-the-information-retriever-in-the-retrieval-augmented-generation-pipeline-a782c7287e9b">blog</a>, we learned about adding the information retrieval component to the RAG pipeline. However, to achieve a much higher level of accuracy, we need a way to rank these responses and fine-tune them even more accurately. Enter Re-ranker.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D03PwPL_sFSJrH77hmpx1g.png" /><figcaption>Architecture diagram — RAG</figcaption></figure><p><strong>What is Re-ranker?</strong></p><p>The drawn-out name for Re-ranker is “<strong>learning-to-rank approach</strong>,” and it’s precisely what it sounds like. The Re-ranker component sifts through the responses provided by a retriever (like the Watson Discovery, Elastic Search, etc.) and ranks them based on their relevance. ColBERT (Contextualized late interaction over BERT) based DrDecr model is trained with a deep learning technique using a large dataset for ranking passages in response to a given query.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/666/0*iSueoDzSZlK3LIpC.png" /><figcaption>ColBERT re-ranker (Source: <a href="http://www.semanticscholar.org/">www.semanticscholar.org</a>)</figcaption></figure><p>The detailed implementation of the DrDecr model can be found in:</p><p><a href="https://arxiv.org/pdf/2112.08185.pdf">https://arxiv.org/pdf/2112.08185.pdf</a></p><p><strong>Why is Re-ranker useful?</strong></p><p>We need re-ranking because the first-stage retriever may be flawed. It may rank some irrelevant documents high, while some relevant documents might get lower scores. Thus, not all top-k documents are relevant, and not all relevant documents are in the top-k. Re-ranker refines these results and brings up the most relevant answers.</p><p><strong>Implementing the Re-ranker</strong></p><p>To experiment with the re-ranking in practice, let me guide you through an example using Watson Discovery as a retriever and a ColBERT (Contextualized late interaction over BERT) based DrDecr model as a re-ranker.</p><p>The first step is the setup, and for this, we will be importing the required libraries, including the ColBERTReranker, from the PrimeQA components.</p><pre>from primeqa.components.reranker.colbert_reranker import ColBERTReranker</pre><p>Next, we load the pre-trained re-ranker model.</p><pre>reranker = ColBERTReranker(model=&quot;DrDecr.dnn&quot;)<br>reranker.load()</pre><p>With the model now in place, we can use it to predict and re-rank documents relevant to a given query. Implementation of re-ranker will take place right after retrieval of data from Watson Discovery, i.e.,</p><ol><li>The user question/query is processed through Watson Discovery to get retrieving results.</li></ol><pre>def process_discovery_retriever(question):<br>    global string_unicode<br><br>    projects = discovery.list_projects().get_result()<br>    for project in projects[&#39;projects&#39;]:<br>        if (project[&#39;name&#39;] == project_name):<br>            project_id = project[&#39;project_id&#39;]<br><br>    collections = discovery.list_collections(project_id = project_id).get_result()<br>    for collection in collections[&#39;collections&#39;]:<br>        # print(collection)<br>        if (collection[&#39;name&#39;] == collection_name):<br>            collection_id = collection[&#39;collection_id&#39;]<br><br><br>    query_result = discovery.query(project_id=project_id, query=question).get_result()<br>    return query_result</pre><p>2. It is then re-ranked by the pre-trained re-ranking model.</p><pre># Run ColBERT Reranker<br>from primeqa.components.reranker.colbert_reranker import ColBERTReranker<br>model_name_or_path = &quot;DrDecr.dnn&quot;<br>max_reranked_documents = 2<br>reranker = ColBERTReranker(model=model_name_or_path)<br>reranker.load()<br><br>reranked_results = reranker.predict(queries= [question], documents = [results], max_num_documents=max_reranked_documents)<br><br>print(reranked_results)<br><br>reranked_results_to_display = [result[&#39;document&#39;] for result in reranked_results[0]]<br>df = pd.DataFrame.from_records(reranked_results_to_display, columns=[&#39;rank&#39;,&#39;document_id&#39;,&#39;title&#39;,&#39;text&#39;])<br>print(&#39;======================================================================&#39;)<br>print(f&#39;QUERY: {question}&#39;)<br>display( HTML(df.to_html()) )</pre><p>The <a href="https://github.com/ibm-ecosystem-engineering/SuperKnowa/blob/main/3.%20Re-ranker/Watson%20Discovery/Re-ranker.ipynb">example notebook</a> attached to this blog demonstrates these steps for a question like “What is Cloud Pak for Data?”</p><p><strong>Analyzing the results</strong></p><p>The <a href="https://github.com/ibm-ecosystem-engineering/SuperKnowa/blob/main/3.%20Re-ranker/Watson%20Discovery/Re-ranker.ipynb">notebook</a> shows the results in two separate stages:</p><p>1. After retrieval but before re-ranking.</p><p>2. After re-ranking.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Smoyd51Wo-2StfSQ3PqI4g.png" /><figcaption>Before and After Results</figcaption></figure><p>You will notice that the relevance of the responses improves significantly after the re-ranking process. In the before column, you can observe that the document is talking about Palantir for Cloud Pak for Data (CP4D) and not CP4D. On the other hand, you can observe the sentence — “ IBM Cloud Pak for Data is a cloud-native solution that enables data scientists, data engineers and…………..” in the after re-ranker implementation. Therefore, this example demonstrates the improvement in the performance of the retrieval pipeline.</p><p><strong>Impact of implementing Re-ranker</strong></p><p>The accuracy of the RAG (Retrieval Augmented generation) pipeline is highly dependent on the result of the retriever/re-ranker. For example, if you provide the results from the Before column in the previous section to an LLM (Large Language Model) and ask “What is Cloud Pak for Data?”, it will provide you with a wrong answer and describe what is Palantir for Cloud Pak for Data instead. This type of retriever-reranker pipeline can be easily implemented for many purposes including but not limited to RAG.</p><p><strong>Conclusion</strong></p><p>As we push for AIs to provide more accurate and human-like answers, tools like Watson Discovery, Elastic Search, and re-ranking strategies are pivotal. Adding a re-ranker to our pipeline allows us to use better the sheer amount of information we can leverage in large documents.</p><p>While the retriever (Watson Discovery, Elastic Search, etc.) helps us to zone in on the appropriate context, the re-ranker sharpens our results and ensures that the most pertinent information is a priority.</p><p>Whether you’re building an intelligent Q&amp;A system or striving to improve your existing AI’s ability to answer questions, it’s worth considering adding a re-ranking model to your stack. In this era of information overload, it’s about more than just having the information. It’s about having the correct information at the right time.</p><p>The full implementation details can be found in this <a href="https://github.com/ibm-ecosystem-engineering/SuperKnowa/blob/main/3.%20Re-ranker/Watson%20Discovery/Re-ranker.ipynb">notebook</a> on the <a href="https://github.com/ibm-ecosystem-engineering/SuperKnowa/tree/main">GitHub Repo</a>.</p><p>The next stage in the RAG pipeline is to implement In-context learning to augment the knowledge and create a prompt for LLM. Learn more about it in our The next stage in the RAG pipeline is to implement a Re-ranker to get the most relevant document. Learn more about it in our The next stage in the RAG pipeline is to implement a Re-ranker to get the most relevant document. Learn more about it in our <a href="https://medium.com/@Shivam_Solanki/understanding-in-context-learning-for-language-models-7086747b8512">next blog</a>.</p><p>Follow <a href="https://medium.com/towards-generative-ai">Towards Generative AI</a> for more content related to the latest in AI advancement.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=55a19931325" width="1" height="1" alt=""><hr><p><a href="https://medium.com/towards-generative-ai/improving-rag-retrieval-augmented-generation-answer-quality-with-re-ranker-55a19931325">Improving RAG (Retrieval Augmented Generation) Answer Quality with Re-ranker</a> was originally published in <a href="https://medium.com/towards-generative-ai">Towards Generative AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unleashing the Power of the Information Retriever in the Retrieval Augmented Generation Pipeline]]></title>
            <link>https://medium.com/towards-generative-ai/unleashing-the-power-of-the-information-retriever-in-the-retrieval-augmented-generation-pipeline-a782c7287e9b?source=rss-89c3d96ab56b------2</link>
            <guid isPermaLink="false">https://medium.com/p/a782c7287e9b</guid>
            <category><![CDATA[nlg]]></category>
            <category><![CDATA[retrieval]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <dc:creator><![CDATA[Shivam Solanki]]></dc:creator>
            <pubDate>Mon, 24 Jul 2023 19:13:58 GMT</pubDate>
            <atom:updated>2023-08-18T23:15:11.661Z</atom:updated>
            <content:encoded><![CDATA[<h4>Exploring how an information retriever enhances generation capabilities in the RAG pipeline using different knowledge sources</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*u3pnm0Vrg19GDR81.png" /><figcaption>Information Retrieval. Credit — <a href="https://tonellotto.github.io/"><strong>Nicola Tonellotto Homepage</strong></a></figcaption></figure><p>Artificial intelligence is transforming the way we process and understand text. One popular use case in the Large Language Model (LLM) field is the Retrieval Augmented Generation (RAG). Central to the RAG pipeline is an Information retriever, which fetches relevant documents from a knowledge source to augment the context for generating responses. So, we will focus on the Information retrieval component of the RAG pipeline shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HdOHG5PBopknVidjdO4itg.png" /><figcaption>RAG Pipeline</figcaption></figure><h4>Understanding the Information Retriever</h4><p>An information retriever serves as a bridge between the input and an extensive knowledge source. It scans a corpus of documents, ranks them according to their relevance to the input, and retrieves the most pertinent documents. The retrieved documents are then passed to a Large Language model, which generates a response based on both the input and the additional context provided by the retrieved documents.</p><h4>1. Information Retrieval using Apache Solr</h4><p>The first example of a neural retriever we’ll look at uses Solr as the document store. The solr_retriever function initializes a connection to a Solr instance. It sends a query to the Solr instance and retrieves the top matching documents.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OHUhYqvEEUE4-FoYmcaTUw.png" /><figcaption>Solr Retriever</figcaption></figure><p>The above query is a GET request sent to a Solr instance, which is an open-source search platform. Let’s break down the different components:</p><ul><li><a href="http://150.239.171.68:8983/solr/superknowa/select%60">http://000.000.000.00:0000/solr/superknowa/select</a>: This is the URL of the Solr service. <a href="http://150.239.171.68:8983/solr/superknowa/select%60">000.000.000.00:0000</a> is the IP address and port where Solr is running, solris the application context (this is usually ‘solr’), superknowais the name of the collection being queried, and select is the request handler that Solr uses to search the collection.</li><li>q=’+question+: This is the query parameter, where question is the search term that will be used to search the Solr collection. The +question+ is a way to concatenate the variable question into the URL string in Python.</li><li>q.op=AND: This is the default operator for the query. When the operator is AND, a document will only match the query if it contains all of the terms. You can adjust this operator to get the best results depending on the use case.</li><li>wt=json: This parameter determines the response format. In this case, json is specified, so the response will be in JSON format.</li></ul><p>So, in simple terms, this Solr query is searching for documents in the superknowa collection that contains all of the terms in the question string and it will return the results in JSON format.</p><h4>2. Information Retrieval using Elasticsearch</h4><p>Next, we have an implementation of a neural retriever using Elasticsearch, a popular open-source search and analytics engine. The elastic_retriever function connects to an Elasticsearch instance and retrieves documents based on a given query similar to the Solr query explained in the previous section.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_931VIbwTGNJB0AdKu-AKg.png" /><figcaption>ES Retriever</figcaption></figure><h4>3. Information Retrieval using Watson Discovery</h4><p>Finally, we introduce a neural retriever that uses Watson Discovery as the knowledge source. The process_discovery_retriever function initializes a connection to a Watson Discovery instance. It takes a query, sends it to the Watson Discovery instance, and retrieves the top matching entities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x9lVJazOr5N35eaLosfafQ.png" /><figcaption>Watson Discovery Retriever</figcaption></figure><h4>Conclusion</h4><p>Neural retrievers, whether built with Apache Solr, Elasticsearch, or Watson Discovery, are vital to the Retrieval Augmented Generation pipeline. They enable the model to augment its context with relevant information from a vast knowledge source, leading to more nuanced and insightful responses. By understanding and leveraging the power of Information retrievers, we can push the boundaries of AI communication and create more meaningful interactions with AI models.</p><p>The next stage in the RAG pipeline is to implement a Re-ranker to get the most relevant document. Learn more about it in our <a href="https://medium.com/towards-generative-ai/improving-rag-retrieval-augmented-generation-answer-quality-with-re-ranker-55a19931325">next blog</a>.</p><p>Thanks to <a href="https://medium.com/u/f30e8889ba31">Abhilasha Mangal</a> for building the Knowledge Retrieval component of our RAG pipeline. And check our <a href="https://github.com/ibm-ecosystem-engineering/SuperKnowa/tree/main">GitHub</a> for full code implementation.</p><p>Follow <a href="https://medium.com/towards-generative-ai">Towards Generative AI</a> for more technical content related to AI.</p><p><a href="https://medium.com/towards-generative-ai/newsletters/synapses-last-week-in-genai">Subscribe</a> to the 3 min <a href="https://medium.com/towards-generative-ai/newsletters/synapses-last-week-in-genai">newsletter</a> to learn about 3 most impactful things in Generative AI every week.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a782c7287e9b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/towards-generative-ai/unleashing-the-power-of-the-information-retriever-in-the-retrieval-augmented-generation-pipeline-a782c7287e9b">Unleashing the Power of the Information Retriever in the Retrieval Augmented Generation Pipeline</a> was originally published in <a href="https://medium.com/towards-generative-ai">Towards Generative AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[OpenAI vs Watson NLP: Sentiment Analysis]]></title>
            <link>https://medium.com/towards-generative-ai/openai-vs-watson-nlp-sentiment-analysis-80196b94d4a?source=rss-89c3d96ab56b------2</link>
            <guid isPermaLink="false">https://medium.com/p/80196b94d4a</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[watson-nlp]]></category>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[nlg]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Shivam Solanki]]></dc:creator>
            <pubDate>Mon, 10 Jul 2023 19:05:10 GMT</pubDate>
            <atom:updated>2023-07-11T16:04:51.605Z</atom:updated>
            <content:encoded><![CDATA[<h4>Exploring the Pros and Cons of OpenAI and Watson NLP for Sentiment Analysis Tasks</h4><p>In the last blog on <a href="https://medium.com/@Shivam_Solanki/openai-vs-watson-nlp-which-is-better-for-your-enterprise-aae5be4fdd5">OpenAI vs Watson NLP: Which is Better for your enterprise</a>, we have seen a high-level overview of the comparison between OpenAI and Watson NLP for various NLP tasks. Now, we will dive deeper into comparing the two for Sentiment analysis.</p><h3>Sentiment Analysis</h3><p>Sentiment analysis is available in both Watson NLP and OpenAI out of the box using pre-trained models. The performance of the Sentiment classification model is comparable in both Watson NLP and OpenAI.</p><p>Here is an output from the OpenAI sentiment classification tasks when all the sentences in the document were properly numbered.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/724/1*xeZk9OYgdvJzy1pPIyip0A.png" /><figcaption>OpenAI sentiment output 1</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/911/1*0S62Nu46Am1fBmHxhKJ_RQ.png" /><figcaption>Watson NLP sentiment output</figcaption></figure><p>However, the major difference lies in the usage pattern. If we pass the raw document with the following prompt:</p><pre>Classify the sentiment &quot;Text...&quot;</pre><p>The output is Sentiment: Neutral . So, in order to extract the sentiment for each sentence, the next logical step is to change the prompt to</p><pre>Classify the sentiment in all the sentences &quot;Text...&quot;</pre><p>The output is</p><pre><br>Sentiment 1: Neutral <br>Sentiment 2: Negative <br>Sentiment 3: Positive</pre><p>You can also increase the maximum token length to 3700 (the token length also includes <strong>366</strong> tokens in the prompt) to get the sentiment for each sentence in the document. Therefore, with this prompt:</p><pre>Extract Sentiment from each sentence in the following text with confidence score:\n{text}</pre><p>You will get sentiment for 24 sentences in a document containing 30 sentences. This exercise was explained to show how difficult it can be to customize the prompt and parameters to get the desired output (table shown below).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/789/1*1U4j0WdCEXQWQU6HjV4WTw.png" /><figcaption>Sentiment output variations</figcaption></figure><p>Additionally, you cannot process large documents and extract sentiment for individual sentences directly. The Watson NLP, on the other hand, has been developed for enterprises so it provides the capability to handle large documents easily and you can extract sentence-level sentiment directly using the run method on the raw text:</p><pre>sentiment_model = watson_nlp.load(watson_nlp.download(&#39;sentiment-aggregated_cnn-workflow_en_stock&#39;))<br>sentiment_result = sentiment_model.run(text)</pre><h4>Targeted Sentiment Analysis</h4><p>In our testing, we have found that the Watson NLP library works better for targeted sentiment analysis as compared to OpenAI.</p><p>Let’s consider this example text: “Terrible service and management, but management was kind enough to help with our request.”</p><p>OpenAI extracts the sentiment as</p><pre>&#39;Sentiment towards service: Negative &#39;, &#39;Sentiment towards management: Mixed&#39;</pre><p>using the following prompt</p><pre>Extract Targeted Sentiment from each sentence in the following text:\n{text}</pre><p>You can also extract the confidence score for the targeted sentiment using another prompt</p><pre>Extract Targeted Sentiment from each sentence in the following text with the confidence score:\n{text}</pre><pre>&#39;Sentence 1: Terrible service and management: Sentiment - Negative, Confidence Score - High &#39;,<br>&#39;Sentence 2: Management was kind enough to help with our request: Sentiment - Positive, Confidence Score - High&#39;</pre><p>The targeted sentiment output for the same sentence using the Watson NLP library is shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/341/1*It46rLFnKyb64PPEK3gOnQ.png" /><figcaption>Targeted Sentiment Output</figcaption></figure><p>You can observe that the Watson NLP works much better for targeted sentiment analysis as it can identify the three targets and their sentiment every single time they are mentioned in the document using a single line of code:</p><pre>tsa_model = watson_nlp.download_and_load(&#39;targets-sentiment_sequence-bert_multi_stock&#39;)<br>tsa_predictions = tsa_model.run(syntax_prediction)</pre><p>Therefore, Watson NLP is much better for targeted sentiment analysis and also better for sentiment analysis on large documents using the pre-trained model.</p><p>Now for customizing the models, as I have explained in my previous blog OpenAI doesn’t provide documentation or examples at this time to fine-tune or custom-train the base models for any other downstream tasks besides binary classification. Therefore, Watson NLP is the clear winner in terms of customizability with lots of options both in fine-tuning a pre-trained model as well as custom training a Deep Learning or Machine Learning based model</p><h3>Conclusion</h3><p>We have seen how the NLP tasks for understanding the voice of the customer like Sentiment analysis can be carried out both using OpenAI and the Watson NLP. However, in terms of customizability, rate limit, and token limit, Watson NLP is the clear winner for all of these tasks.</p><p>If you want to read about our comparative analysis for the task Emotion and Tone analysis, you can read this <a href="https://medium.com/@Shivam_Solanki/openai-vs-watson-nlp-emotion-and-tone-analysis-feb35cbb3cc1">blog</a>.</p><p><a href="https://developer.ibm.com/blogs/cheatsheet-for-watson-libraries/">Learn more about IBM Watson libraries with this list of links to help you find the information you need.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=80196b94d4a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/towards-generative-ai/openai-vs-watson-nlp-sentiment-analysis-80196b94d4a">OpenAI vs Watson NLP: Sentiment Analysis</a> was originally published in <a href="https://medium.com/towards-generative-ai">Towards Generative AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[OpenAI vs Watson NLP: Emotion and Tone Analysis]]></title>
            <link>https://medium.com/towards-generative-ai/openai-vs-watson-nlp-emotion-and-tone-analysis-feb35cbb3cc1?source=rss-89c3d96ab56b------2</link>
            <guid isPermaLink="false">https://medium.com/p/feb35cbb3cc1</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[watson]]></category>
            <category><![CDATA[nlg]]></category>
            <dc:creator><![CDATA[Shivam Solanki]]></dc:creator>
            <pubDate>Mon, 10 Jul 2023 17:41:43 GMT</pubDate>
            <atom:updated>2023-07-10T17:41:43.452Z</atom:updated>
            <content:encoded><![CDATA[<h4>Analyzing the Strengths and Weaknesses of OpenAI and Watson NLP in Emotion and Tone Analysis</h4><p>In the last blog on <a href="https://medium.com/@Shivam_Solanki/openai-vs-watson-nlp-sentiment-analysis-80196b94d4a">OpenAI vs Watson NLP: Sentiment Analysis</a>, we have seen the comparison between OpenAI and Watson NLP for Sentiment analysis. Now, we will compare the two for the tasks for Emotion and Tone Analysis.</p><p>Emotion and Tone analysis is a process of evaluating the emotional tone or attitude conveyed by a piece of text. It involves analyzing the language used and identifying the emotions and attitudes that are conveyed by the words and phrases.</p><p>Emotion and Tone analysis typically involves a <strong>more nuanced evaluation of the emotional content of text compared to sentiment analysis</strong>, which often focuses on identifying whether the overall sentiment of the text is positive, negative, or neutral. In tone analysis, the emphasis is on identifying the specific emotions and attitudes that are expressed, such as anger, happiness, sadness, sarcasm, or irony.</p><p>Emotion and Tone analysis can be useful in a variety of contexts, such as understanding the tone of customer feedback, evaluating the tone of political speeches or media coverage, or assessing the tone of written communication in a workplace or academic setting. It can also be used in combination with sentiment analysis to provide a more comprehensive understanding of the emotional content of a piece of text.</p><h3>Tone Analysis</h3><p>The tone analysis in OpenAI provides only two tones positive and negative which is no different from sentiment classification.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hSo-n2kgA4hEzlRQWRzxQg.png" /><figcaption>OpenAI Tone analysis output</figcaption></figure><p>The Watson NLP on the other hand provides granular level tones; Excited, satisfied, sad, frustrated, polite, impolite, and sympathetic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/423/1*rkE_NyUxaDsSuDIrRd0_Rw.png" /><figcaption>Watson NLP tone output</figcaption></figure><p>Therefore, Watson NLP is a clear winner for the Tone Analysis task as the other factors around customizability, rate, and token limits are the same as the other NLP tasks described in the previous blog: <a href="https://medium.com/@Shivam_Solanki/openai-vs-watson-nlp-sentiment-analysis-80196b94d4a">OpenAI vs Watson NLP: Sentiment Analysis</a>.</p><h3>Emotion Classification</h3><p>Emotion classification can also be done out of the box using a pre-trained model both in the OpenAI and the Watson NLP library.</p><p>OpenAI output</p><pre>Emotion: Joy (confidence score: 0.9)</pre><p>Watson NLP output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/917/1*mBDKuhLouM6V2ciCMaS67w.png" /><figcaption>Watson NLP emotion output</figcaption></figure><p>The only difference in emotion classification is again the customizability, rate, and token limits in which the Watson NLP is the winner.</p><h3>Conclusion</h3><p>We have seen how the NLP tasks Emotion and Tone analysis can be carried out both using OpenAI and the Watson NLP. However, in terms of customizability, rate limit, and token limit, Watson NLP is the clear winner for both Emotion and Tony analysis tasks.</p><p>If you want to read about our comparative analysis for the tasks like Entity, Keywords, and PII extraction, you can read this blog.</p><p><a href="https://developer.ibm.com/blogs/cheatsheet-for-watson-libraries/">Learn more about IBM Watson libraries with this list of links to help you find the information you need.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=feb35cbb3cc1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/towards-generative-ai/openai-vs-watson-nlp-emotion-and-tone-analysis-feb35cbb3cc1">OpenAI vs Watson NLP: Emotion and Tone Analysis</a> was originally published in <a href="https://medium.com/towards-generative-ai">Towards Generative AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>