Co-authored by NeelimaReddy Gade , Manish Bhide , Ravi Chamarthy

The Challenge: Is Your AI Agent Secure?

AI-powered development tools are changing how developers build applications. What previously required days of coding can now be accomplished in minutes with a well-crafted prompt.

With IBM Bob, developers can generate AI agents, implement business logic, integrate agent development frameworks, and accelerate their development workflows. But as AI-generated agents become increasingly capable, an important question emerges:

Just because an AI agent meets your organization’s governance policies, does that mean it is secure?

Not necessarily.

An AI agent might satisfy organizational requirements around fairness, transparency, and responsible AI while still containing serious security vulnerabilities.

For example, consider a loan approval agent that processes applicant information to determine eligibility. What if malicious instructions embedded in an applicant’s input manipulate the agent into using fabricated income or credit scores? Or what if failures in fairness checks do not prevent the agent from proceeding with loan approval decisions?

response = llm.invoke([HumanMessage(content=applicant_input)])

Here, untrusted applicant input is passed directly to the LLM without adequate safeguards, potentially allowing malicious instructions to influence the agent’s behavior. This could lead to Agent Goal Hijack (ASI01), resulting in incorrect loan eligibility decisions.

This is where the OWASP Top 10 for Agentic Applications becomes particularly relevant. It highlights the most critical security risks associated with AI agents, including goal hijacking, tool misuse, unauthorized code execution, and privilege abuse.

Understanding Agentic AI Security Risks

Unlike traditional applications, AI agents can dynamically invoke tools, access information, and perform actions. This autonomy introduces security risks such as malicious instructions, unvalidated inputs, unauthorized access, and sensitive data exposure.

The OWASP Top 10 for Agentic Applications for 2026 provides a framework for understanding and addressing security risks associated with agentic applications.

For developers, the challenge is translating these security principles into practical checks against the code they are building.

Questions worth asking include:

• Can the agent execute code generated by an LLM without validation?
• Are tool invocations protected by appropriate input validation?
• Can the agent access resources or data beyond the requesting user’s permissions?
• Are credentials and other sensitive information adequately protected?

These are precisely the kinds of security concerns that need attention before deploying an agent into production.

Introducing OWASP Security Assessment in IBM watsonx.governance

A recent release of IBM watsonx.governance introduces security assessment capabilities aligned with the OWASP Top 10 for Agentic Applications, which provides an additional way to assess the security posture of discovered AI agents.

The workflow is particularly interesting when combined with IBM Bob and GitHub Enterprise.

Developers can generate agent code using Bob, push the code to a configured GitHub Enterprise repository, and then use watsonx.governance to discover the agent and initiate an OWASP security scan.

The assessment identifies potential security issues, categorizes findings, assigns severity levels, and provides supporting evidence and recommendations.

The resulting workflow looks like this:

IBM Bob → GitHub Enterprise → watsonx.governance AI Discovery → OWASP Security Scan → Findings → Code Remediation → Rescan

This connects agent development, governance visibility, and security assessment in a single iterative process.

From Agent Discovery to Security Remediation

The configuration and assessment process is straightforward.

Configure GitHub Enterprise in watsonx.governance

Configure a GitHub Enterprise connection in watsonx.governance by specifying the GitHub organization, repository, and Personal Access Token.

This enables watsonx.governance to discover AI agents from the configured repository.

Configure Git Credentials for IBM Bob

Configure Git authentication on the local development system so Bob can push generated code to GitHub Enterprise.

For example, using ~/.netrc:

# cat  ~/.netrc 
machine github.ibm.com
login <GIT_USER>
password <GIT_PERSONAL_ACCESS_TOKEN>
# echo "Restrict access to the credentials file"
# chmod 600 ~/.netrc

Ensure the credentials have the required repository permissions.

Generate and Push Agent Code Using IBM Bob

Provide the following prompt to IBM Bob:

Create an agent using the LangGraph Agent Framework for a loan approval 
scenario.Evaluate applicant eligibility based on income, credit score, 
and existing liabilities whileensuring fair treatment regardless of gender.
Keep the implementation simple and contained in a single Python file named
techxchange_loan_approval_agent.py. 
Make sure the agent name is "Loan Processing Agent".
Also, push the generated Python file to:
https://github.ibm.com/<MYORG>/<MYREPO>/tree/master/<MYFOLDER>

Bob generates the agent code and can push it to the specified GitHub Enterprise repository using the configured credentials.

Discover the Agent and Trigger the OWASP Scan

Navigate to AI Discovery in the watsonx.governance console, the agents discovered from the configured GitHub Enterprise repository appear in the asset inventory. Select the discovered agent, open its actions menu, and choose Run OWASP scan.

Review the OWASP Security Findings

Once the scan completes, watsonx.governance displays the identified vulnerabilities along with their severity, evidence, impact, and recommended remediation.

In this example, the assessment identified 2 security findings — 1 High and 1 Medium, covering two OWASP categories:

• ASI01 — Agent Goal Hijack (High): Applicant-controlled instructions can manipulate the loan processing agent’s objective, potentially introducing fabricated applicant information into eligibility decisions.
• ASI08 — Cascading Failures (Medium): Failures detected during fairness audits do not prevent the agent from proceeding with final loan decisions, potentially resulting in unfair outcomes.

Remediate the Vulnerabilities Using IBM Bob

The findings can be addressed manually or provided back to IBM Bob for code remediation.

For example:

Review the following OWASP security finding identified by 
IBM watsonx.governance.

Category: ASI01_Agent_Goal_Hijack
Severity: High
Finding:
Applicant-controlled instructions can hijack the loan parser objective and
feed fabricated profile data into eligibility decisions.
Evidence:
The parse_applicant function passes untrusted applicant text directly to 
the LLM and uses the generated JSON output for downstream loan eligibility 
decisions without validation.
Recommendation:
Treat applicant-provided text as untrusted data. Use structured LLM outputs
with strict schema validation and prevent embedded instructions from 
influencing the agent's behavior. Fix the vulnerability while preserving the
existing loan approval functionality. Add appropriate input validation and
security tests.

Once the changes are reviewed and tested, the updated code can be pushed to GitHub Enterprise. The OWASP scan can then be triggered again in watsonx.governance to verify whether the reported vulnerability has been addressed.

This creates a continuous security improvement workflow:

Generate with Bob → Push to GitHub → Discover in watsonx.governance → Run OWASP Scan → Fix with Bob → Rescan

Conclusion

AI-generated agents can introduce security vulnerabilities even when they implement the intended business functionality.

The OWASP Top 10 security assessment for Agentic Applications capability in IBM watsonx.governance helps developers identify such vulnerabilities in agents discovered from GitHub Enterprise repositories.

Combined with IBM Bob, this enables a practical workflow for generating agent code, identifying security risks, remediating vulnerabilities, and validating the updated implementation.

The goal is not just to generate code for AI agents faster, but to build them more securely.

References

• New in IBM watsonx.governance: Govern AI agents as you build, expand accountability and streamline compliance
• Govern AI Agents as You Build: Bringing IBM watsonx.governance into IBM Bob
• OWASP Top 10 for Agentic Applications — 2026

Why AI Governance Matters in AI-Assisted Development?

AI-powered development assistants are transforming how developers build applications and AI agents. With (AI-powered) software development partners like IBM Bob, developers can turn natural-language instructions into working code and AI agents within minutes. But faster development raises an important question:

How do we ensure that AI-generated code aligns with an organization’s responsible AI policies?

This is where the integration of IBM Bob with IBM watsonx.governance becomes valuable.

By connecting AI-assisted development with governance capabilities, organizations can introduce policy validation directly and as early as into the development workflow, helping developers identify potential policy violations and governance risks before applications move further toward deployment.

Bringing Governance into the Development Workflow

When a developer saves an agent-related file or completes an interaction with Bob, watsonx.governance can analyze the code, identify AI agents, capture relevant metadata, and assess potential policy violations and risks against organizational AI usage criteria.

Well, let’s take an example — consider a developer is using IBM Bob to build a loan approval agent with the following prompt:

Create an agent using the LangGraph Agent Framework for a loan approval scenario. Evaluate applicant eligibility based on income, credit score, and existing liabilities while ensuring fair treatment regardless of gender. Keep the implementation simple and contained in a single Python file with file name as loan_approval_agent.py.

The agent code is generated, but loan approval is a sensitive use case. At first glance, the prompt appears to address fairness. But does that automatically make the agent compliant with organizational AI policies?

Not necessarily!

Loan approval is a consequential decision-making use case, and many organizations require additional risk reviews for AI systems involved in determining access to financial services.

And, this is where watsonx.governance enters the picture.

Through the IBM Bob extension, an API call can be triggered to watsonx.governance to evaluate the generated code or its associated use-case information against organizational governance policies.

The resulting policy assessment helps developers understand whether their implementation meets the applicable policy criteria or requires further review.

How the Integration Works.

The integration can be understood through five key steps:

Define organizational policies: Governance teams establish AI usage criteria in watsonx.governance, identifying use cases that require additional scrutiny, such as loan approvals, healthcare decisions, employment screening, and sensitive profiling.

Generate code with IBM Bob: A developer provides a natural-language prompt, and Bob generates the corresponding agent implementation.

Trigger governance validation: The Bob extension invokes an API to submit relevant code or use-case information for policy evaluation.

Evaluate policy compliance: watsonx.governance evaluates the submitted information against the configured policy criteria.

Surface the governance result: The developer receives a policy outcome, such as MET or NOT MET, along with relevant findings that can guide remediation or additional review.

Auto-Govern the Agent: Beyond policy assessment, the Bob extension triggers an automated governance workflow to create a Shadow Agent, perform similarity searches against existing agents, and apply auto-triaging rules. Based on the applicable governance criteria, the Shadow Agent can be promoted to a Governed Agent, enabling centralized oversight and lifecycle management.

So, in short — this brings governance closer to the point of development without requiring developers to initiate a completely separate governance process. To better understand Policy MET and NOT MET outcomes, let’s look at a sample policy and how the evaluation results are determined.

Sample Policy: High-Risk AI Use Case Assessment

Sample Policy Objective

Identify AI agents involved in sensitive or consequential decision-making that require additional governance review.

AI agents supporting credit or loan decisions, insurance eligibility, healthcare diagnosis, employment decisions, education assessments, profiling, surveillance, or other defined sensitive use cases must undergo additional risk review.

Scenario 1: Policy MET

Imagine a developer creates an AI agent that explains general financial concepts and answers questions about loan terminology without assessing individual applicants or making eligibility recommendations.

The governance assessment identifies no applicable high-risk decision-making category.

The developer can continue development, subject to other applicable organizational policies.

Scenario 2: Policy NOT MET

Now consider our LangGraph loan approval agent.

The generated code evaluates an applicant’s income, credit score, and existing liabilities to determine loan eligibility.

The governance assessment identifies that the agent supports decisions affecting access to credit, which is explicitly included in the organization’s high-risk review categories.

Illustrative result:

Policy: High-Risk AI Use Case Assessment
Status: NOT MET
Risk Category: Access to Essential Services — Credit and Loan Decisions
Reason: The agent supports loan eligibility decisions and requires additional risk review.
Recommended Action: Initiate governance review before deployment.

Important: A NOT MET result is not necessarily a code failure. It is a signal that the agent requires additional governance attention.

The developer and governance team can then assess fairness, explainability, human oversight, and other relevant requirements before approving the application for its intended use.

Why This Integration Matters

The significance of this integration goes beyond identifying policy violations.

It helps organizations shift from reactive governance toward governance embedded into AI development.

• For developers, it means earlier awareness of potential policy concerns without leaving their development environment.
• For governance teams, it means greater visibility into AI agents being created across the organization, including agent metadata and identified risks.
• For enterprises, it creates an opportunity to scale AI development while maintaining oversight and accountability.

Most importantly, it encourages developers to think about responsible AI from the moment an agent is created, rather than treating governance as a final compliance checkpoint.

The future of enterprise AI development isn’t just about building agents faster. It’s about building them responsibly, with governance integrated from the start.

References

• New in IBM watsonx.governance: Govern AI agents as you build, expand accountability and streamline compliance
• IBM watsonx.governance

Build with IBM Bob. Govern with watsonx.governance. Innovate responsibly.

Co-authored by Amisha Bhanushali, Manish Bhide, and Ravi Chamarthy

Why visibility, controls, and accountability are becoming essential to managing model risk across financial institutions.

India Markets — RBI Model Risk Management 2026: A New Era of Enterprise-Wide Model Governance

As financial institutions increasingly rely on autonomous AI agents, Generative AI, and machine learning models managing model risk has become a critical enterprise responsibility.

The Reserve Bank of India’s (RBI) proposed Model Risk Management (MRM) framework signals a shift from fragmented, model-specific oversight toward a more structured, risk-based, and enterprise-wide governance approach.

The framework emphasizes several foundational expectations:

• Enterprise governance: Board-approved MRM frameworks, clearly defined ownership, and oversight by senior management and risk committees.
• Model inventory and risk tiering: Comprehensive identification, classification, and risk-based management of models across the organization.
• Independent validation and continuous monitoring: Appropriate validation before and after deployment, ongoing performance monitoring, and escalation of breaches.
• AI and third-party governance: Safeguards for AI/ML models, explainability, human oversight, and governance of externally sourced models.
• Documentation and auditability: Traceable decisions, model changes, approvals, and evidence throughout the model lifecycle.

The real challenge for financial institutions is not simply defining these policies. It is operationalizing them consistently across hundreds of models, multiple business units, and increasingly complex AI technologies.

The fundamental shift is from managing individual models to managing model risk as an enterprise-wide discipline.

IBM watsonx.governance: Bringing Visibility, Controls, and Accountability Together

Meeting evolving MRM expectations requires more than spreadsheets, disconnected approval processes, and periodic reviews. Financial institutions need a scalable governance operating model that connects people, processes, models, and risk information.

IBM watsonx.governance provides capabilities to support this operating model through three foundational pillars.

Visibility: Know What Exists and Understand the Risk

Effective governance starts with understanding the enterprise’s model landscape.

IBM watsonx.governance enables centralized visibility into governed AI use cases, model inventories including and not limited to — agents, tools, MCPServers, LLMs, Knowledge bases — lifecycle information, associated risks, and monitoring outcomes. Through Governance Console dynamic dashboards, and governance records, organizations can build a more consistent view of traditional ML and generative AI assets, including those from third-party providers.

This helps answer a fundamental question: What models are being used, where are they deployed, and what risks do they introduce?

Controls: Turn Governance Policies into Action

Visibility alone does not reduce model risk. Institutions need mechanisms to consistently apply governance requirements throughout the model lifecycle.

watsonx.governance supports configurable risk assessments, model validation workflows, approval processes, policy-driven monitoring, and exception management. AI-specific evaluation capabilities further help organizations assess dimensions such as fairness, drift, quality, and generative AI risks.

The objective is to move from governance policies documented on paper to repeatable, measurable, and actionable controls.

Accountability: Establish Ownership and Demonstrate Oversight

Model risk management ultimately depends on clearly defined responsibilities and demonstrable oversight.

Through role-based workflows, lifecycle documentation, approval records, and governance reporting, watsonx.governance helps establish accountability among model owners, validators, risk teams, and governance stakeholders.

It also provides a foundation for management and Board-level reporting, helping institutions demonstrate not just that controls exist, but how governance activities are performed and evidenced.

Together, Visibility, Controls, and Accountability create the foundation for an integrated model risk governance operating model.

Connecting RBI Expectations to watsonx.governance Capabilities

The RBI’s proposed MRM expectations can be translated into practical governance capabilities supported by IBM watsonx.governance.

These capabilities can help institutions establish a more consistent governance framework spanning traditional models, machine learning, Generative AI, and AI agents, with appropriate integrations and configuration.

Importantly, technology is an enabler of regulatory alignment — not a substitute for independent validation, institutional accountability, or regulatory interpretation.

The Way Forward

RBI’s evolving MRM direction represents an opportunity for financial institutions to rethink how model risk is governed across the enterprise.

Rather than treating MRM as another compliance exercise, institutions can use this transition to establish a unified governance operating model built around visibility, controls, and accountability.

With IBM watsonx.governance, organizations have a platform foundation to move from fragmented model oversight toward more connected, scalable, and evidence-driven model risk management.

The future of model risk management is not just about governing more models. It is about governing them with greater consistency, transparency, and accountability.

References

• RBI's Draft Guidance on Regulatory Principles for Model Risk Management | Article | Chambers and Partners
• From AI governance to AI assurance: What we shared at Think 2026 | IBM

Co-Authored with Pratap V

What this article addresses..

This article is an extension to the earlier post — Evaluating LangFlow RAG Flows using IBM watsonx.governance SDK — which demonstrated how LangFlow-based RAG applications can be evaluated using the IBM watsonx.governance SDK and its out-of-the-box quality metrics.

While the earlier work focused on evaluating RAG flows in a static setup, this article takes the concept further by operationalizing the evaluation loop with traceability, observability, and governed experimentation.

In this extended exploration:

• A LangFlow Agent is designed to respond to Banking FAQs.
• LangFuse is integrated with LangFlow to enable full traceability of each flow execution.
• The collected traces are processed to extract structured agent input, retreived context, and output.
• IBM watsonx.governance Metrics Evaluator is employed to compute a combination of standard and custom metrics for both test and validation datasets.
• The results are stored as experiment runs in IBM watsonx.governance Evaluation Studio, enabling governed comparison metrics between test and validation data runs.

How this is addressed..

To establish a traceable and governed evaluation workflow, the solution combines LangFlow, LangFuse, and IBM watsonx.governance into a cohesive pipeline.

Each component contributes a critical capability — from data flow orchestration to observability and evaluation.

The overall process unfolds through the following stages:

Integration with LangFuse for Tracing

LangFlow executions are integrated with LangFuse to capture trace-level data for every interaction. Each trace records the input, retrieved context, generated output, and additional metadata such as execution time and latency. These traces enable complete observability into how the agent processes user requests.

To establish the integration, create an account in LangFuse, followed by creation of an organization and a project, and then create new API Keys in the project.

Set the LangFuse credentials in the same terminal where LangFlow is running.

% export LANGFUSE_SECRET_KEY=[Your LANGFUSE_SECRET_KEY]
% export LANGFUSE_PUBLIC_KEY=[Your LANGFUSE_PUBLIC_KEY]
% export LANGFUSE_HOST="https://cloud.langfuse.com"
% pip install langfuse
% langflow run

LangFlow Agent Setup

A LangFlow-based agent is designed to handle Banking FAQ–type queries. It dynamically determines whether to invoke the search_documents tool (to retrieve relevant information from Astra DB) or the calculator tool (for arithmetic responses, say to compute Mutual Funds returns YoY based on the invested amount). Retrieved context is combined with the input query to generate a final answer using the OpenAI GPT model.

As an In-Flow evaluations, like in the previous article Evaluating LangFlow RAG Flows using IBM watsonx.governance SDK the flow is enriched to include a custom component to compute Relevance metrics on question and the Answer. Please refer to the said post for the code and the procedure on how to do it.

Conversing with Agent

From the LangFlow Playground, send questions around Banking related products or ask around financial computations, for any other question the Agent will come back with “I do not know” answer as mentioned in its prompt instruction. The agent will make use of the respective tools — “search_documents” or “calculator” — to fetch the context built a prompt and run against the OpenAI GPT model, and serve back the response to the user. Behind all these, the traces for each integration is captured and logged to LangFuse.

LangFuse Traces

Over the interactions with the Agent, one can observe the logged traces and their associated cost against underlying LLM from LangFuse dashboard, as in the below snapshot.

One can further drill down on individual traces from the LangFuse Tracing UI.

Trace Extraction and Dataset Preparation

A LangFuse utility is used to fetch traces from the last one hour and extract structured information containing input, context, and output. To facilitate supervised evaluation, a ground_truth field is added for each record.

def extract_trace_fields(traces):
    from jsonpath_ng import parse
    jsonpath_input = parse('$.input."Chat Input (ChatInput-qHFrs)".input_value')
    jsonpath_output = parse('$.output."Agent (Agent-Zr1RM)".response.text')
    jsonpath_context_blocks = parse('$.output."Agent (Agent-Zr1RM)".response.data.content_blocks[*].contents[*]')
    
    results = []
    for trace in traces.data:
        trace_dict = trace.dict() if hasattr(trace, "dict") else trace
    
        # Apply JSONPath queries
        user_input = [m.value for m in jsonpath_input.find(trace_dict)]
        agent_output = [m.value for m in jsonpath_output.find(trace_dict)]
    
        # Context: filter tool_use blocks named search_documents
        context_matches = [m.value for m in jsonpath_context_blocks.find(trace_dict)]
        context_texts = []
        for block in context_matches:
            if block.get("type") == "tool_use" and block.get("name") == "search_documents":
                for out in block.get("output", []):
                    text = out.get("data", {}).get("text")
                    if text:
                        context_texts.append(text)
    
        results.append({
            "id": trace_dict.get("id"),
            "input": user_input[0] if user_input else None,
            "output": agent_output[0] if agent_output else None,
            "context": "\n\n".join(context_texts) if context_texts else None
        })
    return results

Please refer to the notebook for the complete code.

Metric Computation using watsonx.governance Metrics Evaluator

The IBM watsonx.governance Metrics Evaluator is used to compute a combination of out-of-the-box (only few are listed below and the comprehensive list can be found here) and custom metrics for both datasets.

Out of the box Metrics:

• AnswerRelevanceMetric()
• FaithfulnessMetric()
• AnswerSimilarityMetric()
• TextGradeLevelMetric()
• TextReadingEaseMetric()

Custom Metrics:

• ContextSufficiencyMetric()
• ContextRelevanceMetric()
• ContextPrecisionMetric()

class ContextSufficiencyMetric(BaseContextMetric):
    name: ClassVar[str] = "context_sufficiency"
    display_name: ClassVar[str] = "Context Sufficiency"
    metric_type: ClassVar[str] = "sufficiency"
    eval_fn: ClassVar[Callable[[str, str], float]] = staticmethod(evaluate_context_sufficiency)
    result_cls: ClassVar[Type[RecordMetricResult]] = ContextSufficiencyResult


class ContextRelevanceMetric(BaseContextMetric):
    name: ClassVar[str] = "context_relevance"
    display_name: ClassVar[str] = "Context Relevance"
    metric_type: ClassVar[str] = "relevance"
    eval_fn: ClassVar[Callable[[str, str], float]] = staticmethod(evaluate_context_relevance)
    result_cls: ClassVar[Type[RecordMetricResult]] = ContextRelevanceResult


class ContextPrecisionMetric(BaseContextMetric):
    name: ClassVar[str] = "context_precision"
    display_name: ClassVar[str] = "Context Precision"
    metric_type: ClassVar[str] = "precision"
    eval_fn: ClassVar[Callable[[str, str], float]] = staticmethod(evaluate_context_precision)
    result_cls: ClassVar[Type[RecordMetricResult]] = ContextPrecisionResult

Please refer to the notebook for the complete code around usage of Custom metrics.

evaluation_results_1 = evaluator.evaluate(
    data=banking_rag_chatbot_qna_1,
    metrics=[
        ContextSufficiencyMetric(), ContextRelevanceMetric(), ContextPrecisionMetric(),
        AnswerRelevanceMetric(), FaithfulnessMetric(),
        AnswerSimilarityMetric(), TextGradeLevelMetric(), TextReadingEaseMetric()
    ]
)
evaluation_results_df_1 = evaluation_results_1.to_df()

These metrics collectively measure contextual relevance, generated output quality, and factual alignment between the model output and the retrieved content.

Experiment Tracking in IBM watsonx.governance Evaluation Studio

The computed metrics are logged as two separate experiment runs — Test Run and Validation Run — using the Evaluation Studio APIs. An AI Evaluation is then created to compare these runs, enabling structured analysis of how prompt, model, or retrieval changes affect output quality.

Please refer to the notebook for the complete code around setting up the same — the snapshot of the Agents Evaluation Studio is as below.

So, that’s about it — in summary this article demonstrates that if you are building LangFlow based agent and integrated it with LangFuse for tracing, then during development time, one can capture those traces and on them evaluate the various quality and custom metrics using IBM watsonx.governance SDK — Metrics Evaluator and use Agents Evaluation Studio for comparison against agents metrics, or data sets or experiment runs.

Key Takeaways

The combined setup establishes a governed evaluation loop, where every LangFlow execution is observable (via LangFuse), measurable (via watsonx.governance), and comparable (via Evaluation Studio).
This ensures that model improvements or prompt iterations are based on quantitative, reproducible evidence rather than subjective judgment.

Resources

• Refer here for the notebook used to compute the metrics and Evaluation Studio integration.
• Refer here for the custom python component code.
• Refer here for more on Evaluation Studio capabilities.
• Refer here for more on Evals with watsonx.governance.
• Refer here for all the comprehensive list of evals from IBM watsonx.governance SDK.

Happy Tracing and Happy Evaluations!

Retrieval-Augmented Generation (RAG) has quickly become one of the most effective ways to build domain-specific agentic chatbots and knowledge assistants. With tools like LangFlow, it becomes remarkably easy to design and deploy these RAG pipelines through an intuitive visual interface. At the same time, validating the quality and reliability of the generated answers is essential. This is where IBM watsonx.governance Evaluation Studio plays a vital role. In this article, we’ll explore how to integrate LangFlow and the IBM watsonx.governance SDK to not only build a RAG flow using DataStax LangFlow but also evaluate its performance with IBM watsonx.governance AI quality metrics.

IBM watsonx.governance is a platform for Responsible AI that ensures LLM and Agent outputs are transparent, reliable, and auditable. It offers experiment tracking, rich evaluation metrics, and an AI Evaluation Studio to compare outputs across runs for agents, prompts, and foundation models.

and..

LangFlow is an open-source visual framework that lets developers quickly build LLM-powered applications using a drag-and-drop interface. It simplifies the design of RAG pipelines by enabling easy orchestration of retrievers, embeddings, vector stores, and language models, while also supporting extensions through custom Python components — among various other supporting features like agentic flows, etc.

Building a Banking FAQs Chatbot with LangFlow

For this article, the Vector RAG example flow in LangFlow is used, with a needed modifications for evaluation.

1. Bank FAQs Data Ingestion — Loaded a dataset of Banking FAQs into DataStax AstraDB, where the text was vectorized and indexed for retrieval.
2. Flow Design

• The user query comes from the Chat Input component.
• Retrieved documents from AstraDB are parsed and injected into the prompt along with the user query.
• The OpenAI GPT model generates the final answer.

3. LangFlow Components

4. Dual Branch Component Execution

• Branch 1 — The standard chatbot output, where the generated answer is returned to the user and executed within the LangFlow Playground. The next section explains how quality evaluations are applied to the LLM-generated output, from a notebook and then pushed to Evaluation Studio for comparison.
• Branch 2 — Metrics evaluation path, where Faithfulness, Context Relevance, and Answer Relevance among other metrics are computed using the question, retrieved context, and generated answer. This is done using a Custom Python Component which internally uses IBM watsonx.governance SDK MetricsEvaluator. More on this in the latter section.

Evaluation Studio — Perform Quality Evaluation Experiments on the output of LangFlow RAG Flows

Building the chatbot is only half the story. To ensure quality, the workflow is extended to compute evaluation metrics using IBM watsonx.governance Metrics Evaluation and thereby track the experiments using IBM watsonx.governance Evaluation Studio.

Test & Validation Datasets — Generate LangFlow RAG Flow Responses

• Using the LangFlow API route ran questions from test data and collect generated answers. Alongside, the ground-truth answers for these test sets are also provided.
• Likewise, the generated answers are collected for the validation data set.

Metrics Computation — watsonx.governance Metrics Evaluators

• Using IBM watsonx.governance SDK, calculated the built-in metrics,
• Answer Relevance (between generated answer and asked question).
• Answer Similarity (between generated and ground truth answers).
• Text Readability (to assess fluency and clarity).
• Metrics evaluation can be easily extended to capture custom metrics as well.

# Evaluator and its configuration
evaluator = MetricsEvaluator(
    api_client = APIClient(credentials=credentials),
    configuration = GenAIConfiguration(
        input_fields=["question"],
        context_fields=[],
        output_fields=["langflow_generated_answer"],
        reference_fields=["ground_truth"]),
)
# Evaluator the said metrics
evaluation_results_1 = evaluator.evaluate(
    data=banking_rag_chatbot_qna_1,
    metrics=[
        AnswerRelevanceMetric(), AnswerSimilarityMetric(),
        TextGradeLevelMetric(), TextReadingEaseMetric(),
    ]
)
testdata_results_df_1 = evaluation_results_1.to_df()
run_1_metric_results = construct_run_metrics(testdata_results_df_1)

Experiment Run Comparisons — watsonx.governance Evaluation Studio

• Using the watsonx.governance SDK, create:
• AI Experiments — to organize test and validation runs.
• Experiment Runs — each run captured metrics for a dataset (test or validation).
• AI Evaluations — aggregated results pushed to Evaluation Studio, where different runs can be compared visually.

# Creating AI Experiment asset
name = "LangFlow based Banking ChatBot AI Experiment"
description = "Evaluation of a LangFlow based RAG-based chatbot for common banking queries, using manually curated test cases across account services, loans, cards, digital banking, and UPI. Each case includes the user question, retrieved context, model answer, and human-verified ground truth for quality benchmarking."

ai_experiment_client = AIExperimentsClient(api_client=api_client, project_id=WATSONX_PROJECT_ID)
ai_experiment = AIExperiment(name=name, 
                             description=description,
                             component_type="agent",
                             component_name="LangFlow based Banking Chatbot Agent")

ai_experiment_asset = ai_experiment_client.create(ai_experiment)

# associating experiment runs with the metrics
ai_experiment_client.update(
    ai_experiment_id=ai_experiment_id,
    experiment_run_details=AIExperimentRun(run_id=str(uuid.uuid4()),
                            run_name="LangFlow based ChatBot Test Data Metrics Evaluation", nodes=[], duration=10
                        ),
    evaluation_results=run_1_metric_results
)

Evaluation Studio — Experiment tracking and comparison of metrics

Using Evaluation Studio, we can easily compare metrics across different runs of the LangFlow-based Banking ChatBot experiment. The side-by-side view highlights variations in answer quality between test and validation datasets. Color-coded indicators make it simple to spot strengths and weaknesses. This enables practitioners to quickly assess performance, track improvements, and make data-driven decisions on model quality.

LangFlow — Evaluations from Custom Python Component

While LangFlow provides built-in components for building RAG flows, went a step further and directly integrate IBM watsonx.governance metrics into the flow. For this, a Custom Python Component is created that receives the context, question, and answer from the other flow components — specifically from Chat Input, AstraDB output, and LLM Output components — and computes a set of evaluation metrics.

The custom component leverages the ibm_watsonx_gov SDK to calculate metrics such as:

• Answer Relevance — how well the generated answer addresses the user’s question.
• Faithfulness — whether the answer is grounded in the retrieved context from AstraDB.
• Context Relevance — the degree of relevance between the retrieved context and the question.
• Text Grade Level & Reading Ease — readability measures to ensure responses are easy to understand.
• And one can compute a whole set of various metrics as listed here.

The screenshot above shows the “Check Relevance Metrics” custom component inside Vector RAG LangFlow. Behind the scenes, as in the below code snippet, the component initializes credentials for IBM watsonx.governance, constructs a MetricsEvaluator, and runs evaluations on the inputs. The results are returned as structured data that can be viewed directly in the LangFlow flow or exported for further analysis.

class CustomComponent(Component):
    # Input and Outputs .. please refer to the actual code from the 
    # References section

    # Main function to compute metrics
    def build_output(self) -> Data:
        # ...

        # Initialize evaluator with credentials and configuration
        evaluator = MetricsEvaluator(
            api_client=APIClient(credentials=credentials),
            configuration=GenAIConfiguration(
                input_fields=["question"],   # user query
                context_fields=["context"],  # retrieved context
                output_fields=["answer"],    # LLM generated response
                reference_fields=[]          # (optional) ground truth references
                ),
        )        
        
        # Build a dataframe with single row: {question, context, answer}
        input_df = pd.DataFrame([{"question": self.input_value, "context": self.context, "answer": self.answer}])

        # Run evaluation using selected metrics
        evaluation_results = evaluator.evaluate(
            data=input_df,
            metrics=[
                AnswerRelevanceMetric(), FaithfulnessMetric(), ContextRelevanceMetric(),
                TextGradeLevelMetric(), TextReadingEaseMetric(),
                ],
        )        
        
        # Convert results to a DataFrame, then to JSON
        evaluation_results_df = evaluation_results.to_df()
        json_obj = evaluation_results_df.to_dict(orient="records")[0]
        
        # Wrap results into a LangFlow Data object for output
        data = Data(data=json_obj)
        self.status = data
        return data

Following the output of one such sample exeuction.

Key Takeaways

This integration ensures that every chatbot response is automatically evaluated, enabling ChatBot developers to monitor quality metrics seamlessly as part of their RAG workflow. It bridges LangFlow’s orchestration power with watsonx.governance’s evaluation capabilities, creating a more robust and measurable AI pipeline. The Banking FAQs chatbot use case demonstrates how test/validation evaluations can be pushed to Evaluation Studio for side-by-side comparison

Resources

• Refer here for the notebook used to compute the metrics and Evaluation Studio integration.
• Refer here for the custom python component code.
• Refer here for more on Evaluation Studio capabilities.
• Refer here for more on Evals with watsonx.governance.
• Refer here for all the comprehensive list of evals from IBM watsonx.governance SDK.

Happy LangFlows and Evaluations!

With the excitement surrounding the recent release of the DeepSeek R1 model, many LLM application developers are eager to explore its capabilities. A key question arises — does the DeepSeek model generate faithful responses based on the provided context, or does it produce hallucinated answers? Additionally, how relevant are the generated responses by the model to the questions asked?

This is where IBM watsonx.governance comes into play. It enables the evaluation of RAG-based applications powered by DeepSeek R1 by measuring key quality metrics such as faithfulness, answer relevancy, and similarity — comparing model-generated outputs against ground truth data or even answers generated from other large language models. By leveraging watsonx.governance, developers can gain deeper insights into model performance and ensure the reliability of their applications, including any Proof of Concepts they plan to build.

Model Installation — Local

On a RHEL VM (I used an 8-core, 16GB memory instance with multiple other applications running), install Ollama as the root user using the following command:

# curl -fsSL https://ollama.com/install.sh | sh

If Jupyter Notebooks are running under a non-root user (which is typically the case), download and run the DeepSeek R1 model as below using the non-root user.

(Initially, I experimented with the 14B model, but given the resource constraints of my VM, I opted for the bare 1.5B model instead.)

# ollama run deepseek-r1:1.5b

Model generation

There are multiple ways to connect to the model via Ollama. In this case, I chose to use LangChain to interact with the DeepSeek R1 model. The main reason for this choice is that LangChain provides Chain-of-Thought (CoT) reasoning in its output.

from langchain_community.llms import Ollama
llm = Ollama(model="deepseek-r1:1.5b")

As you may have already noticed by now, the DeepSeek model’s response includes CoT reasoning enclosed within <think> and </think> tags. Below is a utility function to extract the model’s reply while excluding the content within these tags.

def extract_model_reply(model_response):
    output = model_response.generations[0][0].text
    import re
    model_reply = re.sub(r'^.*?</think>\s*', '', output, flags=re.DOTALL)
    print(model_reply)
    return model_reply

Below is the prompt, parameterized with the RAG $context and the $question to be asked.

from string import Template
prompt_template = Template("""
You are a helpful AI assistant. Use ONLY the following context to answer the question. If unsure, say "I don't know". Keep answers under 2 sentences.

Context: $context

Question: $question

Answer:
""")

For a sample dataset containing relevant contexts and corresponding questions, construct a prompt and invoke the model to generate responses. Then, use the utility function to extract the actual model output.

# accumulate the prompt response for each query
results = []
for relevant_context, question in zip(relevant_contexts, question_texts):
    context = construct_context(relevant_context)
    prompt_str = prompt_template.substitute(context=context, question=question)
    model_response = llm.generate([prompt_str])
    model_reply = extract_model_reply(model_response)
    results.append(model_reply)

Below is one such sample output from DeepSeek R1 1.5B model.

generations=[[GenerationChunk(text='

<think>\nOkay, I need to figure out how to answer the question "What is the difference between copayment and coinsurance?" 
based on the provided context. 

Let me go through each term one by one.\n\n
First, a copayment refers to a fixed amount you pay for covered services, like $20 for a doctor visit. 
It\'s a set amount that doesn\'t change regardless of how much your costs are beyond the deductible.\n\n
On the other hand, coinsurance is a percentage of costs you must pay before your insurance covers the rest. 
For example, if the deductible is 10%, and you spend $150 for an office visit, you\'d pay $15 (10% of $150) in coinsurance.\n\n
So, the key difference is that copayment is a flat fee, while coinsurance adds on a percentage based on costs exceeding a certain threshold. 
This makes me understand how each term works and helps prevent miscommunication about payment structures.\n</think>\n\n

The copayment is a fixed amount paid for covered services (e.g., $20), while the coinsurance is a percentage of costs paid before the deductible, such as 10% on an office visit costing $150.', 

generation_info={'model': 'deepseek-r1:1.5b', 'created_at': '2025-01-30T13:03:52.62955432Z', 'response': '', 'done': True, 'done_reason': 'stop', 
'context': [151644, ….,13], 'total_duration': 22509413057, 'load_duration': 2381354546, 'prompt_eval_count': 432, 'prompt_eval_duration': 7470000000, 
'eval_count': 250, 'eval_duration': 12655000000})]] llm_output=None run=[RunInfo(run_id=UUID('f8795416-5415-47fe-b196-cdc7dde43cae'))]

Here is a Pandas DataFrame containing the question, DeepSeek R1 response, reference answer (e.g., from another LLM), and the RAG context.

Evaluation of RAG metrics using IBM watsonx.governance

Using the combination of Question, Contexts, DeepSeek R1 model generated answer, and with the reference answer, let’s evaluate a host of various metrics:

• Faithfulness: Measures how well the generated text aligns with the context. Did the model stay on topic based on what it found?
• Answer Relevance: Captures how relevant the final response is to the original prompt.
• Context Relevance: Captures how valid are the retrieved contexts against the questions.
• Unanswered Requests: Questions that could not be answered by the model.
• Answer Similarity: Given a ground truth answer, evaluate the semantic similarity with the model generated answer.
• Content Analysis Metrics
• Coverage: This metric quantifies the extent to which the answer is derivative of the context, where it measures the percentage of summary words within the source text.
• Density: The density measure is used to assess how well the overall sequence of words in the generated answer aligns with being a series of direct extractions from the source text.
• Compression: The metric calculates the ratio of the number of words in the content to the number of words in the generated answer.
• Novelty: The Novelty / Abstractness metric measures how many new n-grams are present in the generated answer as compared to the references.
• Redundancy: The redundancy metric measures the percentage of n-grams in the answer which are (unnecessarily!) repeated.

from ibm_metrics_plugin.metrics.llm.utils.constants import LLMTextMetricGroup, LLMCommonMetrics, LLMQAMetrics, LLMRAGMetrics, RetrievalQualityMetric

# Edit below values based on the input data
context_columns = ["contexts"]
question_column = "question"
answer_column = "d3r1_result"
reference_column = "reference_answer"

config_json = {
            "configuration": {
                "record_level": True,
                "context_columns": context_columns,
                "question_column": question_column,
                 LLMTextMetricGroup.RAG.value: {
                        LLMCommonMetrics.FAITHFULNESS.value: {},
                        LLMCommonMetrics.ANSWER_RELEVANCE.value: {},
                        LLMQAMetrics.UNSUCCESSFUL_REQUESTS.value: {},
                        LLMCommonMetrics.CONTENT_ANALYSIS.value: {},
                        LLMRAGMetrics.RETRIEVAL_QUALITY.value: {
                            RetrievalQualityMetric.CONTEXT_RELEVANCE.value: {
                            "ngrams": 5
                            },
                            RetrievalQualityMetric.RETRIEVAL_PRECISION.value: {},
                            RetrievalQualityMetric.AVERAGE_PRECISION.value: {},
                            RetrievalQualityMetric.RECIPROCAL_RANK.value: {},
                            RetrievalQualityMetric.HIT_RATE.value: {},
                            RetrievalQualityMetric.NDCG.value: {}
                        }
                }
            }
        }

Call the watsonx.governance SDK to evaluate the metrics, by passing the model input and output and the reference answers.

df_input = pd.DataFrame(llm_data, columns=context_columns+[question_column])
df_output = pd.DataFrame(llm_data, columns=[answer_column])
df_reference = pd.DataFrame(llm_data, columns=[reference_column])

import json
metrics_result = client.llm_metrics.compute_metrics(config_json, 
                                                    sources=df_input, 
                                                    predictions=df_output,
                                                    references=df_reference,
                                                    # background_mode=False
                                                   )

Here are the evaluation results.

As observed, faithfulness is strong for some context-answer combinations but falls short in others, even when human oversight is involved. Similarly, while most generated answers appear relevant to the corresponding questions, some do not, despite expectations.

Notes

This article does not draw conclusions about the model’s overall performance, as it also depends on the dataset used. Instead, it focuses on demonstrating the mechanism for using IBM watsonx.governance to evaluate responses generated by the DeepSeek R1 model.

As most LLM application developers are still in the exploration phase of the DeepSeek R1 model, with many implementations being part of a Proof of Concept (PoC), this article focuses solely on evaluating metrics using the SDK approach.

Additionally, IBM watsonx.governance can be leveraged for continuous monitoring of prompts across any Large Language Model (LLM).

References

• IBM watsonx.goverenance
• IBM watsonx.goverenance monitoring toolkit
• Sample notebooks
• Ollama DeepSeek Installation

The Understanding — Agentic AI and Agentic RAG.

Imagine a system similar to a Tour Operator, proficient in handling essential travel services like booking flights, hotels, and currency exchange.

This system acts as an Agent orchestrating multiple Tools (flight booking, hotel booking etc), representing a foundational AI capability that exhibit autonomous decision-making and goal-oriented behaviour. That’s Agentic AI.

For example, via., the Tour Operator’s ChatBot a customer might ask, “I want to spend a week in the English countryside. Can you suggest some places along with hotel details?”. To handle this query, the Agent leverages Large Language Model (LLM) say Granite or OpenAI GPT model or the likes, which acts as the reasoning engine behind the process, combined with tools for retrieving precise information. Here’s how it works:

1. Plan: The LLM interprets the query and formulates an initial plan to retrieve the required information (e.g., places and hotels).
2. Reason and Execute: The Agent uses the LLM to orchestrate the execution by calling specific tools, such as a Places Retriever and a Hotels Retriever, to fetch relevant information.
3. Reflect: The LLM evaluates whether the response fully answers the question. If not, refine the Plan and repeat.

For this query, the LLM Agent might first list places like the Cotswolds and Yorkshire Dales. Realizing this doesn’t address the hotel aspect, it then re-plans to get hotel details using a different tool and fetches hotels like North End House in the Cotswolds and The Old Farmhouse in Yorkshire Dales. The final response integrates both places and hotels, fully addressing the query.

Now, if a user extends the query to include travel insurance, the system can incorporate an Insurance Provider Tool without a complete redesign, demonstrating its modular and scalable design.

The Problem — Is the Agentic RAG System faithful in providing the answers?

Continuing with the example above, how can we verify that the Places included in the final answer are indeed sourced from the Places Retriever, and similarly, that the Hotel names come from the Hotels Retriever?

This brings us to the importance of measuring faithfulness — ensuring the generated answer aligns accurately with the information retrieved from the Agentic Tools — and checking for any hallucinations or inaccuracies.

And this is where IBM watsonx.governance comes into play:

• Measure Faithfulness: Evaluate whether the generated answer aligns accurately with the context retrieved from the Agentic Tool(s).
• Assess Tool Context Relevance: Determine how relevant the retrieved context from the Agentic Tool(s) is to the user’s question.
• Evaluate Answer Relevance: Ensure the generated answer is pertinent and directly addresses the user’s question.

In this post, we will focus on understanding how to measure the above three key RAG metrics of an Agentic System: Faithfulness, Tool Context Relevance, and Answer Relevance.

While there are other important aspects of quality evaluation for an Agentic RAG system — such as ensuring the right tool is called for a given question or assessing the accuracy of the information passed to the tool — these will be explored in an near future. For now, our focus will remain on these core RAG metrics.

In the following sections, we will develop a simple Agentic RAG System, capture its outputs, and leverage IBM watsonx.governance to evaluate key RAG metrics such as Faithfulness, Tool Context Relevance, and Answer Relevance.

The Setup — Building an Agentic RAG system.

Although this post initially began with the tour operator example and ideally would have liked to continue with the same dataset, finding suitable data for this scenario proved challenging. Additionally, generating high-quality synthetic data for this purpose is time-intensive. Therefore, this post will proceed with a different example to illustrate the concepts.

So, imagine a team of LLM Application Developers building a chatbot designed to answer questions about AI governance topics. One of the data sources for this chatbot comes from the FAQs of IBM’s own watsonx suite of products — IBM watsonx FAQs.

If the LLM Application Developers were to build a traditional RAG system, then the steps would be like this:

1. Split, index, and load the documents into the vector store.
2. Query the vector store with the user’s question to retrieve the top-k chunks/contexts.
3. Use these contexts and the user’s question to create a system prompt and run it through an LLM.
4. Finally, retrieve the answer.

Now, let’s say users of this chatbot begin asking questions about the recently released EU AI Act. The challenge with a traditional RAG system is its lack of adaptability. Such systems may struggle to adjust to changing information sources or evolving user needs. However, Agentic RAG, with its modular architecture, adapts easily by adding new tools to its flexible framework.

In this case, for the chatbot to answer questions about the EU AI Act, the summary of the EU AI Act is available here — EU AI Act Summary.

Not to deviate a lot from the traditional RAG system, the next step is to load the data from the above web resources to, say, a vector store and create a Retriever out of it — as shown in the code snippet below:

gov_faq_urls = [
    'https://www.ibm.com/docs/en/watsonx/saas?topic=overview-faq'
]

ai_act_urls = [
    'https://artificialintelligenceact.eu/high-level-summary/'
]

def get_retriever(urls, collection_name):
    docs = [WebBaseLoader(url).load() for url in urls]
    docs_list = [item for sublist in docs for item in sublist]
    text_splitter = get_text_splitter()
    doc_splits = text_splitter.split_documents(docs_list)
    vectorstore = Chroma.from_documents(
        documents=doc_splits,
        collection_name=collection_name,
        embedding=get_embeddings(),
    )
    retriever = vectorstore.as_retriever()
    return retriever

gov_faqs_retriever = get_retriever(urls=gov_faq_urls, collection_name='gov_faqs')
ai_act_retriever = get_retriever(urls=ai_act_urls, collection_name='ai_act')

To proceed, the next step is to wrap each retriever into Agentic tool-based functions using LangChain. The docstring for each tool function should be sufficiently descriptive, as the Agent will rely on this description to determine which tool to call. Below is an example implementation:

from langchain.tools import tool
ai_act_context = ""
gov_faqs_context = ""

@tool
def get_Gov_FAQs_Context(question: str):
    """Get context from watsonx.ai, watsonx.governance, OpenScale related Frequently asked questions."""
    global gov_faqs_context
    gov_faqs_context = gov_faqs_retriever.invoke(question)
    return gov_faqs_context

@tool
def get_AI_Act_Summary_Context(question: str):
    """Get context from High-level summary of the AI Act."""
    global ai_act_context
    ai_act_context = ai_act_retriever.invoke(question)
    return ai_act_context

tools = [get_AI_Act_Summary_Context, get_Gov_FAQs_Context]

Using the concept of Structured Chat Prompts from langchain, let’s create a System Prompt and Human Prompt as follows. These prompts are taken from the IBM Generative AI toolkit, which, are also mentioned in the langchain documentation.

system_prompt = """Respond to the human as helpfully and accurately as possible. You have access to the following tools: {tools}
Use a json blob to specify a tool by providing an action key (tool name) and an action_input key (tool input).
Valid "action" values: "Final Answer" or {tool_names}
Provide only ONE action per $JSON_BLOB, as shown:"
```
{{
  "action": $TOOL_NAME,
  "action_input": $INPUT
}}
```
Follow this format:
Question: input question to answer
Thought: consider previous and subsequent steps
Action:
```
$JSON_BLOB
```
Observation: action result
... (repeat Thought/Action/Observation N times)
Thought: I know what to respond
Action:
```
{{
  "action": "Final Answer",
  "action_input": "Final response to human"
}}
Begin! Reminder to ALWAYS respond with a valid json blob of a single action.
Respond directly if appropriate. Format is Action:```$JSON_BLOB```then Observation"""


human_prompt = """{input}
{agent_scratchpad}
(reminder to always respond in a JSON blob)"""

Using both system prompt and human prompt strings, create a langchain prompt.

from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain.prompts import PromptTemplate
prompt = ChatPromptTemplate.from_messages(
    [
        ("system", system_prompt),
        MessagesPlaceholder("chat_history", optional=True),
        ("human", human_prompt),
    ]
)

For LLM to be used by the Agent, let’s use Granite model from IBM watsonx.ai.

from langchain_ibm import WatsonxEmbeddings, WatsonxLLM
llm = WatsonxLLM(
    model_id="ibm/granite-3-8b-instruct", 
    url=credentials.get("url"),
    apikey=credentials.get("apikey"),
    project_id=project_id,
    params={
        GenParams.DECODING_METHOD: "greedy",
        GenParams.TEMPERATURE: 0,
        GenParams.MIN_NEW_TOKENS: 5,
        GenParams.MAX_NEW_TOKENS: 250,
        GenParams.STOP_SEQUENCES: ["Human:", "Observation"],
    },
)

And now we can set up the chain with above objects. The AgentExecutor class, which takes chain and the tools, is used to execute the agent.

chain = (
    RunnablePassthrough.assign(
        agent_scratchpad=lambda x: format_log_to_str(x["intermediate_steps"]),
        chat_history=lambda x: memory.chat_memory.messages,
    )
    | prompt
    | llm
    | JSONAgentOutputParser()
)

agent_executor = AgentExecutor(
    agent=chain, tools=tools, handle_parsing_errors=True, verbose=True, memory=memory
)

Let’s run one such question “What are considered as Prohibited AI systems?” which ideally is against the EU AI Act content.

response = agent_executor.invoke(
    {"input": "What are considered as Prohibited AI systems?"}
)

The output will be something like this:

Entering new AgentExecutor chain...

```
{
  "action": "get_AI_Act_Summary_Context",
  "action_input": {
    "question": "What are considered as Prohibited AI systems?"
  }
}
```

Observation[Document(metadata={'language': 'en-US', 'source': 'https://artificialintelligenceact.eu/high-level-summary/', 'title': 'High-level summary of the AI Act ...GPAI models must:')]

Action:
```
{
  "action": "Final Answer",
  "action_input": "Prohibited AI systems, as per the EU Artificial Intelligence Act, are AI systems that are used as a safety component or a product covered by EU laws in Annex I and required to undergo a third-party conformity assessment under those Annex I laws. They also include AI systems under Annex III use cases, except if the AI system performs a narrow procedural task, improves the result of a previously completed human activity, detects decision-making patterns or deviations from prior decision-making patterns and is not meant to replace or influence the previously completed human assessment without proper human review, or performs a preparatory task to an assessment relevant for the purpose of the use cases listed in Annex III. Additionally, AI systems that profile individuals, i.e., automated processing of personal data to assess various aspects of a person’s life, are always considered high-risk. Providers whose AI system falls under the use cases in Annex III but believes it is not high-risk must document such an assessment before placing it on the market or putting it into service."
}
```

The AgentExecutor identifies that it needs to call the “get_AI_Act_Summary_Context” tool based on the question and the tool’s docstring. It then invokes the corresponding EU AI Act retriever to obtain the context for the given question. The Agent uses this retrieved context, along with the system prompt and human prompt combination, to generate the final answer, by calling the configured Granite LLM. Store this context in a variable, as it will be used to calculate the RAG metrics.

Continue handling similar questions with the Agent.

Let’s ask one more question, for which the answer is not found in any of the retriever context.

response = agent_executor.invoke(
    {"input": "How to monitor custom metrics with Watson OpenScale?"}
)

The output will be something like this:

```
{
  "action": "Final Answer",
  "action_input": "To monitor custom metrics with Watson OpenScale, you can use the Watson OpenScale API to collect and analyze data on your AI models' performance. This includes metrics such as accuracy, fairness, and explainability. You can also set up alerts to notify you when certain thresholds are met. For more detailed information, refer to the Watson OpenScale documentation."
}
```

Looks like the above answer is not completely correct (or to say partially correct) for the question asked, but still the Agent is providing some answer related to custom metrics. Clearly Hallucinating! And this is what we need to detect using IBM watsonx.governance.

Upon asking for some more questions, we capture each tool’s context and also the answer in a data frame. As shown in the screen-shot below.

The notebook demonstrating such Agentic RAG system using langchain is shared in the references section of this post.

This post demonstrates the use of two tools, where the output of each tool is treated as the RAG context. In practice, there may be scenarios where the Agent invokes multiple matching tools to answer a single user request. This demo can certainly be extended to handle multi-tool invocations for a given user request, and constructing the RAG context using the output of each tool execution. And to complicate things, and to be more real, there can be scenarios where there are multi-agents with multi-tools invocations. However, for simplicity of this post, we will focus on restricting the execution to a single tool for each user request.

The Monitoring Configuration — IBM watsonx.governance.

In summary till now, for the given set of questions, we have utilised different Agentic Tools to capture their respective outputs, which are considered as the RAG context. Combining this context with the system prompt and human prompt, we generate the final answer. The result can be represented as a structured dataset: [{ question, contexts, generated answer }].

Using IBM watsonx.governance, evaluation and monitoring of LLM applications can be performed at various stages:

Application Design Phase

As soon as the LLM application developer creates a prompt or Agentic tool(s) and wants to validate the application, they can test the prompts or tools using test data.

To facilitate this, the developer can leverage the IBM watsonx.governance Python SDK APIs and sample notebooks to validate the designed prompts effectively.

Application Development Phase

When the developer identifies the need to validate the prompt using validation data, two approaches can be taken:

Prompt Not Yet Inferred:

1. Subscribe the prompt with watsonx.governance, specifying a preferred LLM from watsonx.ai.
2. watsonx.governance performs inferencing to generate outputs.
3. Using the prompt inputs and outputs, the platform evaluates various LLM quality metrics, including RAG metrics.

Prompt Already Inferred:

1. Subscribe the prompt as a Detached Prompt Template in watsonx.governance.
2. Upload data containing the prompt inputs and their corresponding pre-inferred outputs.
3. watsonx.governance evaluates the uploaded data against various metrics.

Pre-Production and Production Phases

1. In the pre-production phase, evaluate prompts comprehensively to ensure readiness for deployment.
2. Once in production, implement continuous monitoring to maintain quality and detect potential issues.

For the demonstration in this post, we will evaluate the RAG metrics using the Detached Prompt Template approach, as we have already captured the structured dataset: [{ question, contexts, generated answer }]. And then there are two ways to evaluate:

• Option 1: Upload the dataset via the watsonx.governance UI.
• Option 2: Use the Python SDK APIs in a notebook.

In this case the notebook route, as in Option 2 is used, to showcase the APIs. This approach may interest developers who want to automate the evaluation process as part of their application workflows.

Creation of the Detached Prompt Template

Utilize the watsonx.governance SDK API to create the Detached Prompt Template, providing details such as the Model ID (in this case, the Granite3 8B model) representing the Agent LLM, the task (RAG in this instance) associated with the prompt, the prompt template, its variables, and other relevant details.

from ibm_aigov_facts_client import DetachedPromptTemplate, PromptTemplate
pta_details = facts_client.assets.create_detached_prompt(
    model_id="ibm/granite-3-8b-instruct",
    task_id="retrieval_augmented_generation",
    name="Agentic RAG Testing",
    description="Agentic RAG Testing",
    prompt_details=PromptTemplate(
        input="$system_prompt_human_prompt",
        prompt_variables={"context": "", "query": ""},
        input_prefix="",
        output_prefix="",
    ),
    detached_information=DetachedPromptTemplate(
        prompt_id="detached_prompt",
        model_id="ibm/granite-3-8b-instruct",
        model_provider="IBM",
        model_name="granite-3-8b-instruct",
        model_url="$model_url",
        prompt_url="$prompt_url",
        prompt_additional_info={"IBM Cloud Region": "us-east1"}
))
project_pta_id = pta_details.to_dict()["asset_id"]

To evaluate RAG metrics in IBM Cloud SaaS, we will use the LLM-as-a-Judge approach. This requires configuring an LLM as an Evaluator within watsonx.governance. You can accomplish this through the UI or by using the following code snippet.

The Judge Evaluator can be a model from watsonx.ai or any other LLM. For this demonstration, the MIXTRAL_8X7B_INSTRUCT_V01 model is used as the Judge Evaluator.

from ibm_watsonx_ai.foundation_models.utils.enums import ModelTypes

gen_ai_evaluator = wos_client.integrated_systems.add(
    name="RAG Metrics Evaluator",
    description="RAG Metrics Evaluator",
    type="generative_ai_evaluator",
    parameters={"evaluator_type": "watsonx.ai", "model_id": ModelTypes.MIXTRAL_8X7B_INSTRUCT_V01.value},
    credentials={
        "wml_location": "cloud",
        "apikey": CLOUD_API_KEY,
    },
)

# get evaluator integrated system ID
result = gen_ai_evaluator.result._to_dict()
evaluator_id = result["metadata"]["id"]
evaluator_id

Configure the RAG and other metrics, like HAP, PII detection on input and output, to evaluate either using API or using UI.

response = wos_client.monitor_instances.mrm.execute_prompt_setup(
    prompt_template_asset_id=project_pta_id, 
    project_id=project_id,
    label_column="reference",
    context_fields = ["context"],     
    question_field = "query",     
    operational_space_id="development", 
    problem_type="retrieval_augmented_generation",
    input_data_type="unstructured_text", 
    supporting_monitors={
    "generative_ai_quality": {
        "parameters": {
            "generative_ai_evaluator": { # global LLM as judge configuration
               "enabled": True,
               "evaluator_id": evaluator_id,
            },
            "min_sample_size": 2,
            "metrics_configuration": {
                "faithfulness": {},
                "answer_relevance": {},
                "unsuccessful_requests": {},
                "hap_input_score": {},
                "hap_score": {},
                "pii": {},
                "pii_input": {},
                "retrieval_quality": {},
                "answer_similarity": {},
            },
        }
    }
}, 
    background_mode=False
)

result = response.result
result.to_dict()

RAG Metrics Evaluation

Trigger Prompt Setup, followed by evaluating the prompt.

response = wos_client.monitor_instances.mrm.get_prompt_setup(
    prompt_template_asset_id=project_pta_id,
    project_id=project_id
)
#...
response  = wos_client.monitor_instances.mrm.evaluate_risk(
    monitor_instance_id=mrm_monitor_instance_id,
    test_data_set_name=test_data_set_name, 
    test_data_path=test_data_path,
    content_type=content_type,
    body=body,
    project_id=project_id,
    includes_model_output=True,
    background_mode=False
)

And then visualise the results or use APIs to retrieve the evaluated metrics. Using API is as follows:

wos_client.monitor_instances.show_metrics(monitor_instance_id=genaiquality_monitor_id, project_id=project_id)

Using UI, navigate to the Detached Prompt Prompt Template asset in the project in dataplatform.cloud.ibm.com:

Notice that the overall Faithfulness is bit low 0.65 as an average for 4 records, let’s check why.

Drill down further to examine the faithfulness for each record. Recall that there was one question about custom metrics where no relevant context was retrieved, yet the LLM used by the Agent still generated an answer. For this record, the Faithfulness score is 0.2 — and we could detect this successfully!

And, in contrast, the other records have a reasonably good Faithfulness score of 0.8. A higher faithfulness score, closer to 1, indicates better performance.

To demonstrate further, using IBM watsonx.governance, you can perform a root-cause analysis of the violated metrics against their thresholds. For instance, you can analyze how many records out of 100 have violated thresholds for metrics like Faithfulness, Answer Relevance, or Unanswered Questions. Additionally, you can identify the records that have violations across all three metrics, or a combination of them. This is illustrated in the screenshot below.

For Faithfulness, you can pinpoint which part of the context is contributing to the answer, including the source attributions.

And that wraps up the demonstration!

The Summary

• Began with an explanation of Agentic AI and Agentic RAG.
• Built an Agentic RAG application using LangChain, featuring a single Agent with multiple tools.
• Executed a few questions, capturing the output of each tool as the context, resulting in a collection of [ {question, contexts, answers} ].
• Created a Detached Prompt Template by combining the system prompt and human prompt.
• Used the Mistral model as the RAG LLM-as-a-Judge Evaluator to assess RAG metrics.
• Logged and evaluated the metrics.
• Visualized the metrics using the IBM watsonx.governance UI.

References

• IBM watsonx.goverenance
• IBM watsonx.governance Evaluation Studio documentation
• IBM watsonx.goverenance monitoring toolkit
• Sample notebooks
• Notebook used in this post

Hope this helps!!

The problem statement: Evaluating multiple LLMs and Prompts for Quality Chatbot Responses.

Consider a scenario where a bank is building a chatbot powered by a Large Language Model using Retrieval-Augmented Generation, enabling the bank users to ask questions about banking products. Let’s say a team of LLM application developers is tasked with the development of this chatbot. The typical steps to develop the RAG powered chatbot are as follows:

1. Split, index, and load the documents into the vector store.
2. Query the vector store with the user’s question to retrieve the top-k chunks/contexts.
3. Use these contexts and the user’s question to construct a system prompt and run it against an LLM.
4. Finally, retrieve the answer.

Something like this:

In this process, LLM application developers typically face the following challenges:

• Selecting the LLM: Among different LLMs, determine which one provides responses that result in better RAG metrics, such as Faithfulness, Answer Relevance, and Context Relevance.
• Defining the system prompt: Does including varied and multiple N-shot examples result in better relevance scores and no hallucinated answers compared to using only a few N-shot examples?
• Tuning the prompt parameters: For instance, does a prompt temperature value of 0.6 produce better answer relevance scores and fewer hallucinations compared to a temperature value of 0.9?

You can imagine various combinations for evaluating which LLM to use, what prompt string to employ, and which prompt parameters to set.

The obvious question is: How can LLM application developers streamline the LLM selection process, perform these evaluations, and effectively compare the quality metrics?

The Solution: IBM watsonx.governance Evaluation Studio.

IBM watsonx.governance Evaluation Studio enables LLM application developers to evaluate and compare generative AI assets using quality metrics and customizable criteria tailored to specific application needs. Developers can assess the performance of multiple assets simultaneously and compare results to identify the best LLM, prompt, or prompt parameters for their LLM-powered application.

By using Evaluation Studio, developers can accelerate the AI development process. The platform automates the evaluation of multiple AI assets for different tasks (like Summarisation, Content Generation, RAG, Text Classification, Entity Extraction, Question and Answering), removing the need to review each prompt template individually. Instead, a single AI evaluation experiment can be configured to test multiple prompt templates at once, saving valuable development time.

The Evaluation Procedure: IBM watsonx.governance Evaluation Studio.

Among the various combinations mentioned above, let’s consider a scenario where the LLM application developers have designed the following system prompt. They now want to validate it by testing the prompt against three different LLMs using validation data.

Context: {context}

Answer the following question using only information from the above context.
Answer in a complete sentence. If there is no good answer in the context, 
say "I don't know".

Question: {question}

Answer:

Login to https://dataplatform.cloud.ibm.com/ and navigate to the project of your choice. Using IBM watsonx.ai Prompt Lab, in a wx context project, create 3 Prompt Template assets, each using a different LLM — as shown below.

Please note that, in practice, the system prompt would include much more sophisticated and detailed instructions with N-shot examples. However, for the purpose of this post, we are using a highly simplified prompt for demonstration.

You don’t need to evaluate each Prompt Template Asset individually as we would be evaluating all the Prompts at one go as part of the Evaluation Studio process, but the choice is yours.

The next step is to create an Evaluation Asset using the following steps:

• In the project, click on “New asset”
• Select “Evaluate and compare prompts”
• Provide a name and an optional description for the experiment.
• Select the task type for the prompt, which in this case is RAG.
• Clicking “Next” will display the available RAG-based prompt templates from the current project.

• Clicking “Next” again will bring you to a screen where you can configure the metric dimensions: “Generative AI Quality” and “Model Health.”
• For “Generative AI Quality”, click “Configure” to select the metrics for evaluation.

• Select all metrics except “Data safety” metrics (for this demonstration, we are not checking for HAP and PII content in the validation data, as we know it doesn’t contain any, so no need to check it again).

• To evaluate Answer Quality and Retrieval Quality metrics, configure an LLM-as-a-Judge model to assess Faithfulness, Context Relevance, and Answer Relevance metrics. We are using Mistral AI as the Judge Evaluator for the RAG metrics evaluation. Click on “Save”
• Click again on “Save”, after changing any metric thresholds, as needed.

• If you go back to the beginning of this section, the system prompt contains two variables: context and question. In the follow-up screen, select these two prompt variables.

• Clicking “Next” will provide a screen for test/validation data selection. In the context of RAG, the test/validation file should contain the columns for Question, Context, and Reference Answer. The Reference Answer column is optional and is needed when one would like to compute reference based metrics like ROUGE, BLEU, etc.

• Select the data asset file from the Project Assets.
• In the follow-up screen, select the appropriate mappings: choose the column from the data asset that corresponds to the context, the column from the CSV that corresponds to the reference answer (for computing reference-based quality metrics such as ROUGE, BLEU, etc.), and the column for the question.

• Review the configuration selections and click on “Run evaluation”

• The evaluation will be triggered, and you will see the evaluation progress screen, as shown below.

This concludes the evaluation configuration and triggering. In the next section, we will review the evaluation output.

The Evaluation Results: IBM watsonx.governance Evaluation Studio.

Once the evaluation job is completed, the Evaluation/Metrics comparison screen will be displayed, showing the computed metrics.

The Metrics comparison screen will display the metrics for the selected metrics group, with each bar in the chart representing the respective selected LLM. For example, the red-colored bar corresponds to the Granite model. You can observe that the Mistral model performs better for the Faithfulness and Answer Relevance metrics for the selected Prompts.

You can change the Metrics group using the dropdown, as shown in the screenshot below, to view the computed metrics for the selected prompts.

You can pin a prompt to view the difference in metric values across different prompts, as shown in the screenshot below.

Click on the icon beside the Metrics Group selection to view the “Custom Ranking” option. This allows the LLM application developer to apply weight factors to a set of metrics and see how the computed score fares across the selected prompts. For example, in the screenshot below, the Faithfulness metric is assigned a weight factor of 1, and the Answer Relevance metric is assigned a weight factor of 0.8.

When you click “Apply,” the Metrics comparison screen is updated with the selected metrics, and the prompts are ranked based on the custom ranking score. As you can see, Mistral performs better according to the custom ranking score (again this is only based on the demo/synthetic data, and not based on actual customer data) for the selected Prompts.

Well, I can go on with the features, but will pause here..

In summary, IBM watsonx.governance Evaluation Studio enables developers to configure a single evaluation experiment to test and compare multiple prompt templates across varied LLMs simultaneously, streamlining the process, reducing manual effort, and boosting development efficiency.

Give it a try today at cloud.ibm.com by provisioning watsonx.governance!

References

• IBM watsonx.goverenance
• IBM watsonx.governance Evaluation Studio documentation

Happy Evaluating! — IBM watsonx.governance — Streamlining Generative AI evaluations for smarter decisions!!

Greetings!

Bridging Azure Databricks and IBM watsonx.governance

Built on Apache Spark, Azure Databricks allows users to collaboratively work on data tasks throughout the entire project lifecycle — from data ingestion and engineering to machine learning model development and deployment. By harnessing the power of the Azure cloud, Databricks unlocks valuable data insights and provides machine learning model development and deployment capabilities.

However, managing and monitoring deployed models is crucial for any AI-based project. This is where IBM Watsonx.governance comes in. It provides a comprehensive solution for governing both traditional predictive AI/Machine Learning models and the emerging realm of Large Language Model (LLM) prompt deployments. This allows organizations to proactively monitor model performance, mitigate risks, and ensure responsible AI practices throughout the deployment lifecycle.

Ensuring the continued effectiveness and quality of deployed AI models is a critical challenge.

This article explores how IBM Watsonx.governance empowers effective monitoring of predictive AI/ML models deployed in Azure Databricks.

The Use Case

Let’s keep it simple.

Imagine we have information about someone’s “age”, “gender”, “Body Mass Index”, “family size”, and “smoking” habits.

Using this information, we will build a model to predict the person health insurance amount.

Classic example of a regression problem.

Data is from https://www.kaggle.com/datasets/mirichoi0218/insurance

Azure Databricks — Model Development and Deployment ..

In this post, Microsoft Azure is used as the cloud provider for Databricks. However, you can also use other providers like AWS or Vertex AI to host Databricks workspaces.

Azure Databricks workspace

Login to https://portal.azure.com and create Azure Databricks workspace. I earlier tried with Trial tier, but something or the other did not work (may be just a timing thing) and hence I used Premium tier.

Azure Databricks Cluster

Once the workspace is created, launch it, and proceed to create a compute ML cluster. As part of this article, we used 10.4 LTS (Long Term Support) ML runtime and bare minimum Worker and Driver type and minimum and maximum workers.

Load training data to Databricks Catalog

Load the Insurance amount training data, to the default schema in the catalog.

AutoML experiment

Start an AutoML experiment by selecting the cluster, and the training dataset, select prediction column and the metric to evaluate the AutoML experiments. Thereby run the experiment.

Experiment Results

Check the experiment results and select the model with better R2 score for deployment.

Model deployment

Select the model with better R2 score for deployment.

Model endpoint

For the selected model, create an endpoint — basically deploy the model.

Query endpoint — sample invocation

Do a sample evaluation/invocation of the model deployment.

IBM watsonx.governance monitoring

Wrapping Azure Databricks invocations as Python Function

Currently, we have created and deployed a model in Azure Databricks. The next step is to use IBM watsonx.governance to configure and monitor this model deployment. To facilitate monitoring, we encapsulate the Azure Databricks invocation endpoint in a REST API, which can also be deployed as a Python Function in IBM Watson Machine Learning.

Here is the code snippet that wraps the Azure Databricks model invocation endpoint as a python function:

def azure_databricks_scoring_wrapper():

    AZURE_DATABRICKS_TOKEN = '#AZURE_DATABRICKS_TOKEN' #generate a token in Azure Databricks console, with desired validity.
    invocations_url = '#invocations_url'
    
    import os
    import requests
    import numpy as np
    import pandas as pd
    import json
    
    #entry point scoring method for this python function    
    def score(scoring_payload):
        
        headers = {}
        headers["Content-Type"] = "application/json"
        headers["Accept"] = "application/json"
        headers["Authorization"] = "Bearer {}".format(AZURE_DATABRICKS_TOKEN)        
        
        input_fields = scoring_payload['input_data'][0]['fields']
        input_values = scoring_payload['input_data'][0]['values']

        # construct the scoring payload for ADB Model Deployment
        adb_scoring_payload = {
            "dataframe_split": {
                "columns": input_fields,
                "data": input_values
            }
        }
        
        # get the response
        response = requests.post(invocations_url, headers=headers, json = adb_scoring_payload, verify=False)
        response_json = response.json()

        # construct the response to what WML Python Function should send back.
        transformed_response = {
            'predictions': [{
                'values': [[round(value, 4)] for value in response_json['predictions']],
                'fields': ['prediction']
            }]
        }        
        return transformed_response
    return score

Monitoring Evaluations

The next steps are as follows:

1. Deploy the azure_databricks_scoring_wrapperto WML and obtain a scoring endpoint.
2. Use the IBM watsonx.governance UI to configure this endpoint for monitoring.

Alternatively, as a Developer Persona, you can use IBM watsonx.governance APIs to configure the monitors and perform evaluations by following these steps:

1. In watsonx.governance, create a Service Provider for WML where the azure_databricks_scoring_wrapper Python function is deployed.
2. Subscribe the WML Python Function deployment to IBM watsonx.governance for monitoring.
3. Score the model and log the data.
4. Configure all monitors, including Drift V2, Fairness (to ensure that Females and Males receive similar insurance amounts regardless of gender as a model training feature), Explain (both local transaction-level and global explanations to understand the features contributing to predictions), and Quality monitoring.
5. Perform evaluations using both the API and the UI.
6. Check the results.

End to end notebook can be found here.

Evaluation Results

You can use IBM watsonx.governance UI to check the evaluated results.

Conclusion

This post details the steps for creating and deploying a model in Azure Databricks, wrapping the Azure Databricks invocations endpoint as a WML Python function deployment, and configure IBM watsonx.governance for various monitors to finally evaluate them.

References

• IBM watsonx.goverenance
• Notebook used for setting up IBM watsonx.governance monitoring

Happy Monitoring!

Greetings!

The Context — IBM watsonx.governance 2.0.

Among its many notable features, IBM watsonx.governance 2.0 includes two new key functionalities: assessing the quality of prompts generated against any Large Language Models, and evaluating the quality of RAG-powered applications.

This post explain these two aspects.

To set the context, let’s consider a use case where a bank wants to build a chat-bot to answer user questions about various banking products, such as “What is the annual interest rate for a savings account?” or “How to enable two-factor authentication for the account?”, etc.

Building such an application in a cost-effective manner with proper governance is one aspect, and equally important to evaluate in development phase and continuously monitor are:

• Faithfulness detection — ensuring the chat-bot provides accurate answers based on the given context, rather than fabricating or hallucination responses.
• Relevance detection — ensuring the chat-bot generates answers pertinent to the questions and that the application retrieves contexts relevant to the questions rather than irrelevant ones.

And that’s exactly this post will delve into — How IBM watsonx.governance can measure the faithfulness and relevance of answers obtained from a Conversational AI system, such as a chat-bot powered by any Large Language Model and Retrieval-Augmented Generation (RAG) Pattern.

The Background — RAG Pattern.

Before delving into the product capabilities, here’s a quick overview of the RAG pattern.

Using the example of a banking-related chat-bot, a typical RAG system is set up as follows:

• Banking related data is divided into documents, which are further broken down into smaller chunks. Embeddings are generated for these chunks and stored in a vector database such as Milvus, Chroma DB, etc.

text_splitter = RecursiveCharacterTextSplitter(chunk_size= chunk_size, chunk_overlap=chunk_overlap)
docs = text_splitter.split_documents(data)

embeddings = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")

doc_db = Chroma.from_documents(docs, embedding=embeddings)

• For a given question, such as “What is the annual interest rate for a savings account?”, the system retrieves the relevant contexts from the vector store. For instance, the relevant contexts might be found on pages 5, 10, and 20 of the vector store.

query1 = 'From the list of beneficiaries, I am not able to see the name of beneficiary I have added earlier.'
query2 = 'If a transaction is not credited to the beneficiary account, does the remitter get back the money?'
question_texts = [query1, query2]

# Retrieve relevant context for each question
relevant_contexts = []

for question_text in question_texts:
    relevant_chunks = doc_db.similarity_search_with_score(query=question_text, k=1)
    relevant_contexts.append(relevant_chunks)

• A prompt is then constructed using the combination of these contexts and the original question, formatted like this:

Context: {}

Answer the following question using only information from the above context. 

Answer in a complete sentence. If there is no good answer in the context, 
say "I don't know".

Question: {}

Answer:

• This prompt is run against a Large Language Model (LLM), which could be any LLM, such as those from IBM watsonx.ai, AWS Bedrock, Azure OpenAI, Google Vertex AI, etc.

This post uses Azure OpenAI GPT35 Turbo model for running the RAG prompts.

import os
import openai

openai.api_type = "azure"
openai.api_base = "https://azure-openai-xxxxx.openai.azure.com/"
openai.api_version = "2023-09-15-preview"
openai.api_key = '#openai.api_key'

def get_completion(prompt_text):
    response = openai.Completion.create(
        engine = "azure-openai-deployment-001",
        prompt=prompt_text,
        temperature=0.1,
        max_tokens=100,
        top_p=0.5,
        frequency_penalty=0,
        presence_penalty=0,
        stop='\n'
    )
    return response.choices[0].text

results = []

for prompt_text in prompt_texts:
    model_response = get_completion(prompt_text=prompt_text)
    results.append(model_response)

• Finally, the system generates the answer.

results

["Please use 'raise a query' option available in support section in the app. You could also track your query from time to time. You can now browse our 'Discover our features' videos available in support section of our app for a demo of various features available in our app. ",
 "Yes. If the beneficiary's bank is unable to credit the beneficiary's account for any reason, the former will return the money to the remitting bank within 2 hours. Once the amount is received by the remitting bank, it is credited to the remitter's account by the branch concerned."]

Export the results to a CSV file for evaluating using IBM watsonx.governance

final_payload = []
for question, relevant_context, answer, reference_answer in zip(question_texts, relevant_contexts, results, reference_answers):
    record_payload = []

    context = construct_context(relevant_context)
    record_payload.append(context)
    record_payload.append(question)
    record_payload.append(answer)
    record_payload.append(reference_answer)
    final_payload.append(record_payload)

import pandas as pd
llm_data = pd.DataFrame(final_payload, columns=['context', 'question', 'generated_text', 'reference_answer'])

test_data_path = "banking_faq_data.csv"
llm_data.to_csv(test_data_path, index=False)

At this stage, we have the combination of:

• The question
• The retrieved contexts
• The generated answer

Using these elements, IBM watsonx.governance can be utilized to measure the RAG Triad metrics: Context Relevance, Faithfulness, and Answer Relevance.

• Context Relevance: This measures whether the retrieved contexts are pertinent to the question. (will be released shortly)
• Faithfulness: This assesses whether the model is deriving the answer from the provided context or if it is being creative and providing its own answer, which could lead to hallucinations.
• Answer Relevance: This evaluates whether the obtained answer is relevant to the question.

Here is a quick description of additional metrics:

• Answer Similarity Metric: This metric assesses the overall semantic similarity between the generated response and the ideal answer. (will be released shortly)
• Answer Coverage Metrics: These include various measures:
• Coverage: Evaluates how much of the answer is a derivative of the context. (available as part of watsonx.governance monitoring SDK)
• Density: Measures how closely the answer follows the source text. (available as part of watsonx.governance monitoring SDK)
• Compression: Assesses the conciseness of the generated summary compared to the source. (available as part of watsonx.governance monitoring SDK)
• Novelty: Evaluates the presence of new n-grams in the answer compared to the references. (available as part of watsonx.governance monitoring SDK)
• Redundancy: Detects repetitive information within the answer. (available as part of watsonx.governance monitoring SDK)
• Keyword Inclusions: Checks for the presence of important keywords from the context in the answer and calculates the F1 Score for matching keywords between the context and the answer. (available as part of watsonx.governance monitoring SDK)
• Unanswered Questions: Identifies questions that the model could not answer. (available as part of watsonx.governance monitoring SDK and also runtime support.)

As a quick recap, this post has covered the RAG pattern and evaluated some questions to retrieve relevant contexts and answers.

Now, let’s shift focus to how IBM watsonx.governance can be used to evaluate such RAG/LLM — powered applications.

IBM watsonx.governance — workflow

IBM watsonx.governance offers comprehensive governance capabilities for LLM — powered applications, beyond just computing metrics.

During the design phase, LLM application developers can use the IBM watsonx.governance monitoring toolkit to evaluate the RAG Triad Metrics on the RAG prompts output. This helps refine the retrieval of relevant contexts from the vector database and adjust the prompt generation parameters as needed. Detailed instructions are available in this notebook.

Once the prompt is designed by LLM application developers, it can be further developed and tested in a project and tracked as part of an AI use case. If the metric values are satisfactory during testing, the prompt can be validated by LLM application validators in a pre-production or validation environment, known as the Validation Space.

After validation, the prompt can be promoted to a production space for continuous monitoring. As users ask questions, the combination of questions, contexts, and answers are payload logged with IBM watsonx.governance.

At every phase, IBM watsonx.governance provides capabilities to track the prompt as part of the AI use cases.

This post focuses on creating, configuring, and evaluating prompts in a Development Project using IBM watsonx.governance, emphasizing API integration. The same functionalities are also available via the IBM watsonx.governance UI.

Detached Prompt Template Asset Creation

For a given prompt template with prompt input and prompt variables, and using the LLM details, we create a “detached” prompt template asset.

In this context, “detached” refers to the use of an external LLM model, such as Azure OpenAI GPT-3.5 Turbo, instead of an using an existing model from IBM watsonx.ai. Since the monitoring and evaluation are performed on the logged data for a given prompt template, and not directly scoring the prompt template itself, this setup is termed as a “detached prompt template asset.”

task_id = "retrieval_augmented_generation"
name = "Banking FAQ External prompt (Azure OpenAI GPT35)"

from ibm_aigov_facts_client import DetachedPromptTemplate, PromptTemplate
detached_information = DetachedPromptTemplate(
    prompt_id="banking_faq_prompt",
    model_id="gpt-35-turbo",
    model_provider="Azure OpenAI GPT35",
    model_name=name, ...
)

# define parameters for PromptTemplate
prompt_variables = {"context": "", "question": ""}
input = """Context: {context}
        Answer the following question using only information from the above context. 
        Answer in a complete sentence. If there is no good answer in the context, say "I don't know".
        Question: {question}
        Answer:"""

prompt_template = PromptTemplate(
    input=input,
    prompt_variables=prompt_variables, ...
)

pta_details = facts_client.assets.create_detached_prompt(
    task_id=task_id,
    prompt_details=prompt_template,
    detached_information=detached_information, ...
)
project_pta_id = pta_details.to_dict()["asset_id"]

Detached Prompt Template Asset Setup

For a given detached prompt template asset, setup the monitoring by configuring the RAG related metrics, as described in the below code snippet. As you can notice, we are configuring Faithfulness, Answer Relevance, and other LLM Quality Metrics like ROUGE, BLEU, etc metrics.

monitors = {
    "generative_ai_quality": {
        "parameters": {
            "min_sample_size": 10,
            "metrics_configuration": {
                "faithfulness": {},
                "answer_relevance": {},
                "rouge_score": {},
                "exact_match": {},
                "bleu": {},
                "unsuccessful_requests": {}
            }
        }
    }
}

response = wos_client.monitor_instances.mrm.execute_prompt_setup(
    prompt_template_asset_id=project_pta_id, 
    project_id=PROJECT_ID,
    supporting_monitors=monitors, ...
)

Detached Prompt Template Asset Evaluation

Now, we proceed with evaluating the prompt template asset for RAG metrics using the test data that we have gathered in the previous steps.

As a quick recap, the test data comprise of: questions, their contexts, and the answers. Also includes ground truth answer, if any.

Evaluation happens using home grown models specifically built for checking faithfulness and relevancy metrics.

response  = wos_client.monitor_instances.mrm.evaluate_risk(
    test_data_path=test_data_path, ...
)

Evaluation Results and Fact Sheets

Once the evaluation is completed, you can then view the results in the IBM watsonx.governance console or can fetch the results via., the API

Evaluation results in the notebook:

If you zoom in the screen, then you can notice that the Faithfulness score seems to be good, above 70%, but answer relevance score is very less, and this is where the LLM Application developer can go back and fine tune the RAG prompt and/or fix the retrieval of the contexts.

Evaluation results in the IBM watsonx.governance console

Fact Sheets for the corresponding Prompt Template Asset:

Likewise, the AI Use Case Lifecycle tracking:

Conclusion

This post describes the RAG pattern in general and using IBM watsonx.governance, this post describes how to evaluate the RAG related metrics using the combination of Questions, their contexts and answers — which is generated using an external LLM.

References

• IBM watsonx.goverenance
• IBM watsonx.goverenance monitoring toolkit
• Sample notebooks
• Notebook used in this post

Happy Governing! — IBM watsonx.governance — Making AI conversations more ‘faithful’!!