What RAGAS adds to RAG evaluation

What RAGAS adds to RAG evaluation, learned from building a vehicle safety assistant

GitHub - shashankag14/auto-safety-assist: Ask your car what's wrong. It answers with citations…

A user asked my vehicle safety assistant: “my bmw x5 lost power on the highway even after the fuel pump recall was supposed to be fixed.”

My hand-written retriever eval gave that query a precision of 0 and a recall of 0. Neither of the two documents I had labelled as relevant came back in the top 5.

RAGAS scored the same retrieval at 0.92 context precision.

I assumed one of them was broken. Neither was. They were answering different questions, and working out why taught me more about evaluating retrieval-augmented generation (RAG) than any metric definition did.

This post covers:

• the RAG app and the evals I wrote by hand first
• where label-based evals quietly undercount and overcount
• what each of the five RAGAS metrics I use actually computes
• how I wired RAGAS in so reruns are cheap and reproducible
• what it caught, what it missed, and the traps in trusting an LLM judge

The app, and the evals I wrote first

Auto Safety Assist answers questions about vehicle safety using public NHTSA recall and complaint data. It covers three vehicles: the 2018 BMW X5, the 2022 Toyota Camry and the 2022 Honda CR-V.

Each question goes through three services:

1. Intent classifier: decides whether the question needs a recall search, a complaint search, or no retrieval at all.
2. Retriever: embeds the question and pulls the top 5 chunks from Postgres with pgvector. Each recall is stored as three chunks (summary, consequence, remedy) that share one campaign number.
3. Response generator: gpt-4o-mini writes an answer from those chunks and cites recall or complaint IDs.

I built a golden dataset of 18 retrieval questions. Each one lists the relevant documents by their natural NHTSA key, with a grade: 2 for the recall that answers it, 1 for a supporting complaint.

Then I wrote two evals by hand.

The retriever eval maps each returned chunk back to its NHTSA key and compares it with the golden list. It reports Precision@5, Recall@5, MRR and NDCG@5. It also de-duplicates chunks, because three chunks of the same recall would otherwise inflate both precision and DCG.

The generator eval runs the real pipeline and checks two things:

• Faithfulness: an LLM judge returns faithful: true/false and lists any claim not supported by the context.
• Keyword coverage: a plain substring check that the answer contains expected strings such as 21V586 and fuel pump.

These evals are cheap, deterministic (except the judge) and easy to reason about. For a while they were enough.

Where label-based evals go wrong

ID-based retrieval metrics answer one question: did the documents I labelled come back? That is not the question you care about. You care whether the generator got the information it needed. The two drift apart in four ways.

1. Labels are never complete. Complaint databases are full of near-duplicates. Ten owners describe the same stalling problem in ten complaints. I labelled one or two per query, so any other complaint that says the same thing counts as a miss.

2. The right document can carry the wrong chunk. If the retriever returns the remedy chunk of the right recall, the ID matches and precision looks perfect. But the user asked why the recall happened, and that chunk doesn’t say.

3. A pass/fail judge hides degrees. An answer with one unsupported claim out of eight fails the same way as an answer that is entirely made up.

4. Keyword checks reward string matching, not understanding. An answer can contain 21V586 and fuel pump and still say the wrong thing about them.

You can patch each of these by hand: label more documents, grade chunks, rewrite the judge. Or you can measure against meaning instead of IDs, which is what RAGAS does.

The five RAGAS metrics, and what they really compute

RAGAS is a library of LLM-judged metrics for RAG. Every metric takes some mix of four fields per sample:

• user_input: the question
• retrieved_contexts: the chunks the generator saw
• response: the generated answer
• reference: a hand-written ideal answer

The key move is that reference is text, not a list of IDs. Each metric breaks text into claims and asks a judge LLM about each claim.

Retriever metrics

context_recall splits the reference answer into claims. For each claim, the judge asks whether the retrieved context supports it. The score is the share of claims that are supported. It measures information coverage: did the retriever fetch what a correct answer needs, from any document?

context_precision (the WithReference variant) asks, for each retrieved chunk, whether it was useful for producing the reference answer. It then computes a rank-weighted average, like mean average precision. Useful chunks at rank 1 count for more than useful chunks at rank 5.

Generator metrics

faithfulness splits the answer into statements and checks each against the retrieved context. The score is the share of statements that are supported. It catches hallucination, and it is graded, not pass/fail.

answer_relevancy asks the judge to generate several questions that the answer would be a good answer to. It embeds those questions and takes their cosine similarity to the real question. It penalises hedging, padding and drift. It does not check whether the facts are right.

Get Shashank Agarwal’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

factual_correctness compares the answer's claims with the reference's claims. I run it in recall mode: the share of reference claims that the answer covers. It is the only one of the five that checks the answer against the truth.

How they map to classic metrics

Notice that faithfulness and answer_relevancy never look at the reference. That gap matters, as the results below show.

Wiring it in: generate once, score many times

The most useful design decision was splitting the eval into two scripts.

Step 1: generate samples. Run the real pipeline once per golden question and save exactly what the generator saw and said. Use RAGAS field names so each sample can be passed straight to a metric.

def build_sample(ex: dict) -> dict:
    candidates = retrieve(ex["query"])
    # one context string per candidate, formatted exactly as the generator sees it,
    # so the judge can check citation IDs in the answer against the context
    retrieved_contexts = [build_context([Candidates(**c)]) for c in candidates]
    response = generate(ex["query"], candidates) if candidates else None
    return {
        "id": ex["id"],
        "user_input": ex["query"],
        "retrieved_contexts": retrieved_contexts,
        "retrieved_context_ids": [str(c["external_id"]) for c in candidates],
        "response": response,
        "reference": ex["reference_answer"],
    }

The samples file also records the git commit, generator model, temperature and top_k. Every score can then be traced back to the exact answers it judged.

Step 2: score the saved samples. Scoring never calls the pipeline, so you can change a judge or a metric and rerun on identical answers.

def build_metrics(use_cache: bool = True) -> dict:
    client = AsyncOpenAI(api_key=...)
    cache = DiskCacheBackend(cache_dir=".cache/ragas") if use_cache else None
    judge = llm_factory("gpt-4.1-mini", client=client, cache=cache, temperature=0)
    embeddings = embedding_factory("openai", "text-embedding-3-small", client=client, cache=cache)
    return {
        "faithfulness": Faithfulness(llm=judge),
        "answer_relevancy": AnswerRelevancy(llm=judge, embeddings=embeddings),
        "factual_correctness": FactualCorrectness(llm=judge, mode="recall"),
        "context_precision": ContextPrecisionWithReference(llm=judge),
        "context_recall": ContextRecall(llm=judge),
    }

Each metric runs as its own async job behind a semaphore of 8. A failed judge call is recorded on the sample rather than raised, so one bad response can’t sink a whole run.

async def score_one(metric, name, sample, semaphore):
    kwargs = {field: sample[field] for field in METRIC_INPUTS[name]}
    async with semaphore:
        try:
            result = await metric.ascore(**kwargs)
            return float(result.value), None
        except Exception as e:
            return None, f"{type(e).__name__}: {e}"

A note on the disk cache. Caching judge calls makes rescoring free, which is great while you iterate. But it also hides the judge’s own randomness: identical inputs always get the first verdict back. I added a --no-cache flag for any run where I'm measuring variation.

What RAGAS found

The headline: my generator almost never makes things up, but it leaves out about half of what a complete answer should say.

Three individual questions show what the averages hide.

The “wrong” retrieval that was right

Back to the opening query about the X5 losing power after the fuel pump recall. My golden labels were complaint 11450849 and recall 21V586000. The retriever returned neither, so my ID eval scored it 0.

RAGAS scored context precision at 0.92, so I read the five chunks. Two complaints, 11675079 and 11756344, describe exactly the user's situation. In both, a 2018 X5's high-pressure fuel pump was replaced under recall 21V-586, failed again, and the car lost power on the freeway. They were better answers than the documents I had labelled.

My labels were wrong, not the retriever. I added both complaints to the golden set and rewrote the reference answer. The disagreement between the two evals was the bug report.

Faithful, relevant and wrong

The question: “did bmw recall the coolant pump on the x5”. The answer:

“No, BMW did not recall the coolant pump on the X5… recall 24V608000 addresses an improperly sealed electrical connector on the water pump, but it does not indicate a recall for the coolant pump itself.”

The correct answer is yes. Recall 24V608000 is the coolant pump recall: the water pump is the coolant pump.

Every statement is grounded in the context, and the answer addresses the question directly, so the first two metrics are near perfect. Only factual_correctness, which compares against the reference, catches that the conclusion is backwards. Faithfulness is not correctness. If you only run reference-free metrics, this kind of failure is invisible.

Graded faithfulness

For “is there a fuel pump recall for the 2022 honda cr-v”, faithfulness came back at 0.80: roughly one statement in five not supported by the context. A pass/fail judge would have flagged it the same way as a fully invented answer. The graded score tells you it’s a small slip worth reading, not a fire.

Traps in trusting an LLM judge

RAGAS swaps label noise for judge noise. Four things bit me.

1. Don’t let the generator grade itself. My first runs used gpt-4o-mini as both generator and judge, and faithfulness came out at 0.975. A model judging its own style of output tends to be lenient. I moved the judge to gpt-4.1-mini, a different model from the generator.

2. Your reference answers are now the ground truth. context_recall, context_precision and factual_correctness are only as good as reference. A vague reference is easy to satisfy and inflates all three. Writing good references took longer than writing the eval code.

3. answer_relevancy measures style, not quality. Conversational questions ("anyone else's cr-v randomly slam the brakes in traffic") score lower because the reverse-generated questions are phrased differently. Hedged answers score lower too. I report it but never gate on it.

4. Scores move with no code change. Across three uncached runs of the same pipeline, the mean factual_correctness ranged from 0.43 to 0.58. A 0.05 "regression" can be pure judge noise.

To handle point 4, I built a small regression gate. It builds a baseline from at least two uncached runs and sets each metric’s tolerance to twice the run-to-run standard deviation, with a floor of 0.03. A later run fails if any gated metric drops by more than its tolerance, if any judge call failed, or if the judge, embedding model or RAGAS version differs from the baseline’s. Scores from different judges aren’t comparable, so the gate refuses to compare them.

The gate earned its keep straight away. The latest run’s context_precision came in at 0.83 against a 0.90 baseline, more than double its 0.03 tolerance. I had also edited the golden set since the baseline was built, so I have to rule that out before calling it a retriever regression. That's exactly what a gate is for.

Takeaways

RAGAS didn’t replace my hand-written evals. It gave me a second opinion, and the places where the two disagree are the most useful output of either one.

• ID eval low, RAGAS high: your labels are probably incomplete. Read the chunks and add the substitutes to the golden set, as I did for GD-010.
• ID eval high, RAGAS low: the right document came back but the wrong chunk did, or your reference asks for something the document doesn’t say.
• Faithfulness high, factual correctness low: the generator is reading its context faithfully and still reaching the wrong conclusion, as in GD-005. Reference-free metrics will never catch this.

What I’d keep from each:

• Keep the classic metrics for fast, free, deterministic regression checks when you change chunking, embeddings or top_k. Keep NDCG for graded relevance.
• Keep exact checks for the facts that can’t be wrong. In a safety app, a wrong recall number is a hard failure. A substring check catches it with no judge noise.
• Use RAGAS to measure meaning: information coverage, graded faithfulness and correctness against a reference.
• Treat the judge as a dependency. Pin its model and the RAGAS version, keep it different from the generator, measure its noise, and gate on that noise rather than on a guess.

The opening query scored 0 in one eval and 0.92 in the other. Both were right. Running both evals told me what was actually wrong: my labels.