Evaluating RAG in Production: What We Learned Beyond “The Answer Looks Good”
What we learned about evaluating retrieval quality, safe abstention, and AI behavior in production.
I prefer this over “Is your RAG system actually working?” because “Can you prove it?” gets directly at the enterprise problem: moving from subjective confidence to measurable evidence.
The story should start with a very relatable problem
We had a working RAG system.
We tested it exactly the way most teams initially test RAG:
Ask a question → read the answer → check the citation → “Looks good.”
And for development, that’s useful.
But once we started thinking about production, we realized something uncomfortable:
We couldn’t consistently explain what “good” meant.
A fluent answer could still be based on poor retrieval.
Relevant context could still result in an irrelevant answer.
And sometimes the knowledge simply didn’t contain the answer at all.
That forced us to stop thinking about evaluation as “give the response a score” and start thinking about the different failure modes in the RAG pipeline.
1. We first had to define what we were actually measuring
Instead of asking:
“Was this a good answer?”
we started separating the problem:
Context relevance
Did retrieval bring back information that was actually useful for the question?
Answer relevance
Did the generated response actually address what the user asked?
Answerability
Was there enough evidence in the retrieved knowledge to answer reliably?
That distinction matters because each points to a different problem.
Poor context relevance might indicate a retrieval problem.
Good context with a poor answer may indicate a generation problem.
Insufficient context may mean the system shouldn’t answer at all.
This gave us much more useful information than a single “quality score.”
2. One of our most interesting results was a non-answer
Initially, it is easy to think of this as failure:
“I don’t have enough information in the available knowledge to answer that reliably.”
The system didn’t answer.
Bad result?
Not necessarily.
If the knowledge base genuinely doesn’t contain the answer, generating a confident response is actually the worse outcome.
So safe abstention became a successful production behavior, not automatically an evaluation failure.
That changed an important assumption:
A production RAG system shouldn’t be optimized to answer every question. It should be optimized to answer when the evidence supports an answer.
This is one of the strongest ideas in the article.
3. Then agents made evaluation more complicated
Once we introduced AI agents, not every conversation followed the same path.
One execution might be:
User → Agent → RAG → LLM → Answer
Another might be:
User → Agent → Tool → Business System → Result
And another could combine both.
That raised a surprisingly important question:
Should a tool-only agent execution receive a RAG quality score?
We decided it shouldn’t.
Get Suresh Pawar’s stories in your inbox
Join Medium for free to get updates from this writer.
If no retrieval happened, there is no RAG behavior to evaluate.
This eventually led us to separate two concepts in the architecture:
Agent Run History
What happened?
Which agent ran? Which tool was invoked? What was the execution outcome?
Online Evaluation
Was the RAG behavior good?
Was the retrieved context relevant? Was the answer relevant? Was there enough evidence? Did the system abstain appropriately?
That separation made both systems cleaner.
Observability tells us what the AI did. Evaluation tells us how well a particular AI behavior performed.
That’s a much stronger architecture insight than simply explaining DeepEval metrics.
4. Then we discovered measuring AI quality has its own cost
Once online evaluation worked, the natural temptation was:
Evaluate everything.
More evaluation means more visibility.
But production evaluation isn’t free.
Depending on the evaluation approach, it can introduce additional model calls, which means:
additional tokens → additional latency → additional cost
So we faced another architecture trade-off.
Do we evaluate 100% of production traffic for maximum visibility?
Or evaluate a representative sample and reduce operational overhead?
We chose to make online evaluation sampling configurable.
That way evaluation becomes an operational policy, rather than a hard-coded assumption.
This is an excellent place to mention DeepEval naturally:
We incorporated DeepEval into the evaluation layer, but the bigger architecture decision wasn’t which evaluation framework to use. It was deciding what should be evaluated, when it should be evaluated, and what the resulting signals actually mean.
That sounds much stronger than “we used DeepEval.”
5. The bigger lesson
We started with:
“Does the answer look good?”
We eventually needed to ask:
Was the right context retrieved?
Was the answer relevant?
Was there enough evidence to answer?
Should the system have abstained?
Was this even a RAG interaction?
What happened during the agent run?
How much evaluation should we run in production?
What does continuously measuring quality cost?
And that’s the evolution I’d make the article about.
Closing thought
I’d finish with:
Production AI quality isn’t about giving every response a score.
It’s about knowing what behavior occurred, what should be measured, what good looks like, and when not answering is actually the correct outcome.
Because once Enterprise AI moves into production, “the answer looks good” is no longer enough.
You need to be able to explain why.
Then the subtle product attribution:
These are some of the lessons we’ve learned while building Knowgentiq — Enterprise Knowledge & Agents Platform at NXTGEN Innovate Technologies.
Turn enterprise knowledge into governed AI action.
This version gives us another authentic engineering story: we didn’t set out to “implement DeepEval.” We encountered a production-quality problem, separated the concerns, made architectural decisions, and DeepEval became one part of the solution. That’s the tone I’d carry into the full article.
