Agent Accuracy
Your agent returned an answer. It sounded confident. It sounded correct. Your user moved on.
But was it actually right? Was it grounded in anything real, or did it just pull something plausible-sounding from nowhere?
Here is the uncomfortable truth. Most teams never ask that question at the metric level. They check whether the agent responds. They do not check whether the response is trustworthy.
That gap is where production incidents live.
Everyone checks if the agent answers. Nobody checks if it answers correctly. These six metrics are the difference between the two.
Two ways an agent can fail you
Before the metrics, you need one mental model.
An agent that uses retrieval, meaning it fetches context before generating an answer, can fail in two completely different places.
- It can fail at retrieval. Wrong documents fetched. Irrelevant chunks ranked first. Good information buried at the bottom where the model never uses it.
2. Or it can fail at generation. Right documents fetched. Model ignores them. Makes something up instead. Sounds confident. Gets it completely wrong.
These are different problems with different fixes. And the only way to know which one you have is to measure both.
Groundedness and Faithfulness
These two are essentially the same thing. Most tools use them interchangeably. The question they answer: is every claim in the agent’s response supported by the retrieved context?
Not “does it sound right.” Not “is it probably true.” Can you trace each statement in the answer back to something the retriever actually fetched?
A grounded answer draws entirely from source documents. An ungrounded answer introduces facts the model generated from its own training data, not from what was retrieved. That is hallucination.
How tools measure it: they decompose the response into individual claims, check each one against the retrieved context, & score the proportion that are supported. A score of 1.0 means every claim is traceable. A score of 0.6 means four out of ten claims came from nowhere.
Fixes: lower the temperature, tighten your system prompt to explicitly tell the model to use only the provided context, or switch to a model with better instruction-following.
Answer Relevancy
Faithfulness tells you whether the answer is grounded. Relevancy tells you whether it actually addresses the question.
These are two different things. An answer can be perfectly grounded in the retrieved documents and still be completely beside the point. The retriever pulled topically related but non-matching content. The model answered that content faithfully. But the user’s actual question went unanswered.
Ragas measures this in a clever way. It generates several artificial questions from the agent’s answer, then checks how similar those generated questions are to the original user question. If the answer truly addressed the question, you should be able to reconstruct the original question from the answer. If you cannot, the answer drifted.
Target: answer relevancy above 0.75.
A low relevancy score paired with high faithfulness points at retrieval, not generation. The model did its job. The wrong documents were fetched.
That is the diagnostic power. High faithfulness plus low relevancy tells you to fix your retriever. Low faithfulness plus high relevancy tells you to fix your generator.
Context Precision
Context precision is a retrieval metric. It measures whether the relevant documents your retriever fetched are ranked at the top.
This matters because models pay more attention to context that appears earlier. A document that answers the question perfectly but appears at position eight in the retrieval list is much less useful than the same document at position one.
Context precision rewards putting the right information first. If your retriever finds the right document but buries it behind seven irrelevant ones, your context precision is low even though the answer was technically available.
Get Aadhirai’s stories in your inbox
Join Medium for free to get updates from this writer.
Target: context precision above 0.7.
A low score here means your retriever found the right content but ranked it poorly.
Fixes: add a reranker, rewrite the query, improve your chunking strategy.
Hallucination Rate
Hallucination rate is the percentage of responses where the agent generated information that is not in the source context or contradicts it.
It is essentially the inverse of faithfulness, measured at scale.
Here is what the numbers actually look like in production. For grounded summarization tasks, where the answer is sitting right there in the provided text, top models today hallucinate under ~2% of the time. For open-domain factual questions, where no source is provided, the same models hallucinate on ~15–35% of responses. The task matters enormously.
There is no universal acceptable hallucination rate. Define it against your specific task.
Do not set a threshold and forget it. Hallucination rates shift when you change models, change prompts, or change your retrieval strategy. Measure continuously.
Temperature
Temperature is the one metric here that you set, not measure.
It controls how random the model’s output is. Low temperature means the model almost always picks the most probable next token. The output is more consistent and predictable. High temperature means less probable tokens get a real chance. The output is more varied and sometimes more creative.
The common advice is: “set temperature to zero for factual work and you eliminate hallucination” - That advice is wrong.
Temperature zero makes the output deterministic. It does not give the model more knowledge or better reasoning. A model that does not know the correct answer will still generate a plausible-sounding wrong answer at temperature zero. Consistently.
Research across many models has also found that for some models, temperature zero actually increases certain types of failure. Longer reasoning chains at low temperature can generate more claims, and each claim is an opportunity to be wrong.
Practical guidance: start at 0.0 to 0.2 for factual or RAG work. Tune it empirically. Measure what actually happens to your accuracy metrics when you change it.
Temperature is a starting point, not a solution.
The diagnostic combinations worth knowing
Read these metrics together, not individually.
High faithfulness plus low answer relevancy: your generator is doing its job. Your retriever is fetching the wrong content. Fix the retrieval layer.
Low faithfulness plus high answer relevancy: your retriever found the right content. Your generator ignored it and made something up instead. Fix the generation layer, not the retrieval.
Low context precision plus reasonable faithfulness: you are getting lucky. The relevant content is there but buried. Add a reranker before this gets worse.
Everything low: start with retrieval. It is almost always the retrieval layer.
The tools that measure these
A practical starting stack: Ragas to explore and establish baselines, DeepEval to gate your CI/CD, LangSmith to monitor in production.
They interoperate. You can log Ragas scores into LangSmith traces. You do not have to pick just one.
The question worth sitting with
Think about the last response your agent gave. Could you trace every claim in that response back to something it actually retrieved? Or did some of it come from the model’s training data, dressed up to sound like it came from your documents?
If you do not have metrics running, you genuinely do not know. And “I think it is probably fine” is not a monitoring strategy.
Start with faithfulness and answer relevancy. Two metrics. A few hours of setup. You will see things you did not know were happening.
#AIAgent #LLMOps #MachineLearning #AgentDevelopment #RAG
