Mirage

How I Built Mirage: An AI Forensic Investigator Using SigNoz, OpenTelemetry, MCP, and RAG

Artificial Intelligence has become a core component of modern applications, from customer support assistants to coding copilots. While these systems are getting smarter, one question continues to bother developers:

Can we actually trust every AI response?

During the Agents of SigNoz Hackathon, I wanted to explore a problem that traditional observability platforms don’t solve. Tools like SigNoz are excellent at telling us if an application is healthy — whether services are running, latency is increasing, or errors are occurring. But they don’t answer a different class of questions:

• Why did the AI generate this answer?
• Was the response supported by retrieved knowledge?
• Did the model hallucinate?
• What evidence influenced its decision?

To answer these questions, I built Mirage, an AI Forensic Investigation Platform that combines OpenTelemetry, SigNoz, Model Context Protocol (MCP), and a custom Trust Engine to investigate every AI interaction.

The Problem I Wanted to Solve

For the demo, I built a Northwind Support RAG Agent. The agent was intentionally limited to answering questions only from the Northwind support knowledge base.

During testing, I asked a simple question:

“Do you have any idea about AWS?”

Since AWS documentation doesn’t exist in the knowledge base, the expected behavior was for the agent to refuse the question.

Instead, it confidently explained AWS.

The response looked convincing, but it wasn’t supported by any retrieved documents.

This is a classic AI hallucination.

Interestingly, nothing was technically “broken.”

• The API responded successfully.
• Vector search completed.
• The language model generated an answer.
• No exceptions were thrown.

From an infrastructure perspective, everything was healthy.

Yet the AI still produced an incorrect answer.

That became the motivation behind Mirage.

Building the Investigation Pipeline

Rather than building another chatbot dashboard, I wanted Mirage to reconstruct the complete execution path behind every AI response.

The workflow looks like this:

User Query
      │
Northwind Support RAG Agent
      │
OpenTelemetry Instrumentation
      │
SigNoz
      │
Model Context Protocol (MCP)
      │
Mirage Trust Engine
      │
Investigation Report

Every stage of the AI pipeline emits OpenTelemetry spans, allowing Mirage to analyze not only what the AI answered but also how it arrived at that answer.

Instrumenting the Agent with OpenTelemetry

One of the first things I implemented was end-to-end instrumentation.

Instead of tracing only HTTP requests, I created spans for every stage of the RAG pipeline:

• User Query
• Document Retrieval
• Vector Search
• LLM Response Generation
• Trust Evaluation

Each span is exported to SigNoz using OpenTelemetry.

This gave me complete visibility into how an AI request moves through the system.

Using SigNoz Beyond Traditional Monitoring

Initially, I integrated SigNoz simply to visualize traces.

As Mirage evolved, I realized SigNoz could become much more than a monitoring dashboard.

Instead of using it only for observability, I treated it as the primary source of forensic evidence.

Mirage retrieves data from SigNoz through the Model Context Protocol (MCP), allowing it to access:

• Distributed traces
• Logs
• Metrics
• Services
• Alerts

This telemetry is then correlated with the AI’s retrieval evidence.

For example, if the Trust Engine detects a hallucination, Mirage can immediately reconstruct:

• Which traces belong to the interaction
• Which service processed the request
• Whether retrieval actually happened
• Which documents were retrieved
• Whether the generated answer was grounded

This transforms observability data into explainable AI investigations.

Building the Trust Engine

Traditional observability focuses on infrastructure health.

Get Bavyasakthivel’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Mirage introduces a Trust Engine that focuses on AI reliability.

For every AI interaction, the Trust Engine evaluates multiple signals, including:

• Retrieval quality
• Grounding evidence
• Missing context
• Hallucinations
• Weak retrieval
• Honest refusals
• Semantic failures

Rather than producing a simple pass/fail result, Mirage generates:

• Trust Score
• Investigation Verdict
• Root Cause Analysis
• Supporting Evidence
• Suggested Resolution

This makes debugging AI systems much more practical than manually reading traces.

Evidence Instead of Assumptions

One of my favorite parts of the project is the Evidence Locker.

Instead of simply saying:

“The AI hallucinated.”

Mirage shows why.

The Evidence Locker displays:

• Retrieved documents
• Similarity scores
• Grounding evidence
• Missing context

Developers can immediately verify whether the response was actually supported by the retrieved knowledge.

If the evidence doesn’t justify the answer, Mirage marks the interaction as Untrusted.

Replaying AI Decisions

Another feature I enjoyed building was Case Replay.

Instead of looking at isolated logs, Mirage reconstructs the complete AI execution timeline.

Developers can replay:

• User Query
• Retrieval
• LLM Generation
• Trust Evaluation
• Final Verdict

This makes AI debugging feel much closer to debugging a distributed application.

What I Learned

Before building Mirage, I viewed observability as something primarily used for debugging infrastructure.

This project changed that perspective.

I realized observability can also explain AI behavior.

OpenTelemetry captures what happened.

SigNoz stores the evidence.

MCP retrieves the investigation context.

The Trust Engine explains whether the AI’s reasoning can actually be trusted.

Together, they create something much more valuable than monitoring — they create accountability.

Conclusion

Building Mirage taught me that the future of AI observability isn’t just about monitoring APIs or tracking latency.

As AI systems become more autonomous, developers also need visibility into why models make decisions, whether those decisions are supported by evidence, and how to investigate failures when they aren’t.

By combining OpenTelemetry, SigNoz, MCP, and a custom Trust Engine, Mirage transforms AI debugging into an evidence-driven forensic investigation.

Traditional observability answers:

“Is my application healthy?”

Mirage extends that by answering:

“Can I trust my AI?”