RAG Evaluation: A Hands-On Workshop
Retrieval-Augmented Generation (RAG) is one of the ways to make large language models (LLMs) “accurate” by grounding their responses in relevant external documents. How do we retrieve the right information? How do we know whether retrieval is working well? And how can we evaluate the quality of the final answers?
In this hands-on workshop, we will walk through the RAG pipeline, from collecting and preparing data to building a hybrid retrieval system, evaluating its performance, and generating responses grounded in retrieved documents.
We will use PubMed abstracts as our data source, which gives us a practical biomedical use case to work with throughout the workshop. The complete scripts and supporting artifacts are available here: GitHub link
Part 1. Data Ingestion: From PubMed to JSON
Our first step is to collect data from PubMed. We will retrieve 100 abstracts along with their associated metadata, including publication dates, author information, and PubMed IDs (PMIDs), etc. The data extracted from the Pubmed should include a vision-language or multimodal model (block A in script), in a medical imaging context (block B), with some form of evaluation (block C). It also had to have entered PubMed between July 9 and October 9, 2026, have an abstract, and be in English. The last was the filter criteria.
The ingestion script covers the main steps: Constructing search queries, making API requests, extracting the returned records, and organizing the results into a structured JSON format that we can use in the next stages of the RAG pipeline.
To make the workshop easier to reproduce, I have included the collected abstracts in the GitHub repository. You can use the saved dataset or run the ingestion script yourself to explore the data collection process. If you choose to run the script yourself, you should be aware that the query may not output the same result; thus, the evaluation part will need to be edited. We will retrieve only the abstracts and keep each one as a single document, without chunking, since they are relatively short. This keeps data preparation simple and allows us to focus on retrieval and evaluation in the next steps. To save you from reading through the entire Python script, I’ll highlight only the key parts below. You can find the complete script and all related artifacts in the GitHub repository mentioned above.
# Assemble the query for PubMed
filters = '("2026/07/09"[edat] : "2026/10/09"[edat]) AND hasabstract AND english[lang]'
query = f'{blockA} AND {blockB} AND {blockC} AND {filters}'
# API call
handle = Entrez.esearch(db='pubmed', term=query, retmax=100)
results = Entrez.read(handle)
handle.close()
.......
# JSON construction
articleInfo = []
for article in articleList:
articleInfo.append({u'pubmed_id':article['pmid'],
u'title':article['title'],
u'keywords':article['keywords'],
u'journal':article['journal'],
u'abstract':article['abstract'],
u'doi':article['doi'],
u'accepted': article['history'].get('accepted').date().isoformat() if article['history'].get('accepted') else None,
u'published': article['history'].get('entrez').date().isoformat() if article['history'].get('entrez') else None,
u'authors':article['authors']})
Part 2. Hybrid RAG
Now that we have our abstracts, we need a way to find the right ones for a given question. This is the “retrieval” part of RAG. The LLM can enrich the answer from the documents we give it, so if retrieval brings back the wrong abstracts, even the best LLM will give a wrong answer.
There are two main ways to search documents, and each has its own strengths.
Keyword search (BM25) looks for the exact words from the question in the documents. It is fast and works well with exact terms, for example, the exact name of the company or process. But it fails when the question uses different words than the abstract, for example, “AI inventing findings” versus “hallucination.” The best use case to utilize BM25 search is if you need an exact word to be present in the corpus.
Semantic search (embeddings) turns the question and each abstract into vectors and finds the closest matches in meaning. The closest match is measured by cosine similarity, which is technically the angle between two vectors. It handles paraphrases well, but it can drift to papers on a similar topic and miss an exact term.
Both semantic search and Keyword search output the top list of documents. Hybrid RAG uses both retrieval approaches and combines their results utilizing the reciprocal rank fusion algorithm.
Step 1. Keyword index
For keyword search, we use the bm25s library. We index the title and the abstract together:
import bm25s
texts = [f"{r['title']} {r['abstract']}" for r in records]
retriever = bm25s.BM25()
retriever.index(bm25s.tokenize(texts, stopwords='en'))
retriever.save('index')Step 2. Semantic index
For semantic search, we compare three embedding models because we simply want to evaluate our pipeline, and this approach allows us to do this. These abstracts are half medical and half machine learning, so it is not an easy one-topic semantics. I picked the following embedding models:
- MedCPT, a biomedical model from NCBI trained on PubMed search logs. It uses one encoder for articles and a separate one for queries.
- BGE-base, a popular general-purpose model and a common baseline.
- Qwen3-Embedding-0.6B, a newer general-purpose model.
# This is the initialization of Qwen embedding model
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('Qwen/Qwen3-Embedding-0.6B')
emb = model.encode(texts, normalize_embeddings=True)
np.save('embeddings/emb_qwen3.npy', emb)One thing to keep in mind: each model expects queries to be encoded in its own way. MedCPT has a separate query encoder, BGE needs an instruction prefix before the query, and Qwen3 uses a built-in query prompt. If the query side does not match, retrieval quietly gets worse, with no error message.
Step 3. Combining the results: reciprocal rank fusion
Now we have two ranked lists for every query (question), one from BM25 and one from the embedding model. We use the reciprocal rank fusion algorithm. For every question, each paper gets a score of 1 / (60 + rank) from each list, and the scores are added up. A paper ranked high by both methods rises to the top. We will implement the reciprocal rank fusion algorithm with the ranx library:
# we give two ranked list from BM25 and emb model -> output one list with new raks
from ranx import fuse
hybrid = fuse(runs=[run_bm25, run_qwen3], method='rrf')In total, we end up with seven retrieval setups to compare: BM25 alone, the three embedding models alone, and three hybrids (BM25 + each embedding model). In the next part, we will see how to evaluate them and what performs well.
Part 3. Evaluating retrieval
As I mentioned, we have seven retrieval setups. But which one is the best? To answer that, we need two things: A set of test questions where we know the right answers, and metrics that tell us how good each ranking is.
The question set
I built 48 questions about the corpus, drafted with the help of an LLM and then revised them. Every question comes with the PMIDs (think of PMID as the ID of the abstract; it is uniquely identified in PubMed) of the abstracts that answer it. The questions are of five types, because each type tests something different:
- Specific questions (15): Answerable from one abstract, using similar words, for example “Which model was more accurate for diagnosing oral autoimmune blistering diseases: Claude Opus 4.7 or Gemini Pro 3?”
- Paraphrased questions (10): Also answerable from one abstract, but deliberately worded differently, for example “a chronic mouth condition with white patches on the tongue” instead of “oral lichen planus.” These test whether a retriever understands “meaning,” not just words.
- Multi-paper questions (15): Questions with several correct papers, for example “Which studies assessed hallucinations in vision-language model outputs?”
- Hard-negative questions (4): My PubMed query also pulled in papers where LLMs read only radiology report text, with no images. They look almost identical to VLM papers. A question like “Which studies tested GPT-4o’s ability to interpret medical images directly?” should retrieve the image-based studies and avoid the text-only ones.
- Unanswerable questions (4): Questions with no answer in the corpus, for example, VLMs for ECG tracings. The right behavior is to say “nothing found”. We will test these in the generation part.
I also got the false-positive papers from my imperfect PubMed query. Those papers do not mention VML or evaluation, or they are not medical papers. And this actually turned out to be useful. Real literature always has near-misses, and a good retriever should be able to tell them apart. Instead of throwing them away, I used them as a test. For 8 questions, I wrote down which of these look-alike papers would be tempting but wrong. I call them traps. For example, for the question “Which studies tested GPT-4o’s ability to interpret medical images directly?”, the traps are the five studies where GPT-4o worked only with text.
The metrics
Get Zulfiia Ditto’s stories in your inbox
Join Medium for free to get updates from this writer.
Again, we used the ranx library, which calculates standard retrieval metrics and also runs significance tests between setups. I focus on three:
- Recall@10: What fraction of the right papers made it into the top 10. For example, if only 6 out of 8 relevant papers had been returned, this metric would be 75% (6/8 * 100 = 75)
- MRR (mean reciprocal rank): How high the first relevant paper is ranked. Rank 1 gives 1, rank 2 gives 0.5, and so on. Remember, the LLM utilizes the retrieval information in order; thus, if key findings are located at rank 8 or 9 of the top ten documents, they probably will be lost in the noise.
- nDCG@10: Considers all the right papers and their positions, giving more credit to papers near the top. This is the best single number for our mix of single- and multi-paper questions.
from ranx import Qrels, compare
qrels = Qrels(gold) # {question_id: {pmid: 1}}
report = compare(qrels, runs=[run_bm25, run_qwen3, run_hybrid],
metrics=['recall@10', 'ndcg@10', 'mrr@10'], max_p=0.05)Results
To obtain results, I used generated questions as a user prompt query. I applied different retrieval approaches and measured the metrics outlined above. Below is the table with the metrics from the different retrieval approaches. Performance increases noticeably with the hybrid approach compared to the single approaches (BM25 or the embedded model).
Setup Recall@10 nDCG@10 MRR
BM25 0.801 0.743 0.804
MedCPT 0.750 0.625 0.659
BGE 0.786 0.776 0.868
Qwen3 0.843 0.820 0.914
BM25 + MedCPT 0.801 0.784 0.889
BM25 + BGE 0.829 0.806 0.893
BM25 + Qwen3 0.862 0.835 0.922The hybrid BM25 + Qwen3 is the best on every metric. But its gain over Qwen3 alone is small. So the honest summary is: Qwen3 is the strongest retriever, and adding BM25 gives a small extra boost.
Summary of the things:
- Adding BM25 improved nDCG@10 by 0.159 for MedCPT, 0.030 for BGE, and 0.015 for Qwen3. BM25 fills the gaps, especially for the weaker models, which have more gaps to fill.
- I honestly was expecting great results from MedCPT. I was hoping pre-training this model touched not only the clinical part but also the technology within the field, since it was trained for biomedical information retrieval. But MedCPT was the weakest of all setups, even behind plain keyword search. Bummer!
Results by question type
If you recall, I had different question types, and the table below displays nDCG@10 for each question type. As you can see, the average hides a lot:
Setup Specific Paraphrased Multi-paper Hard-negative
BM25 1.000 0.658 0.623 0.444
MedCPT 0.871 0.635 0.444 0.354
BGE 1.000 0.850 0.620 0.337
Qwen3 1.000 0.913 0.659 0.515
BM25 + MedCPT 1.000 0.863 0.600 0.465
BM25 + BGE 1.000 0.865 0.699 0.337
BM25 + Qwen3 1.000 0.856 0.744 0.501- Specific questions are too easy. Almost every setup scores a perfect 1.0, so they no longer tell the methods apart. No surprises here!
- Paraphrased questions break keyword search. BM25 drops from 1.0 to 0.658 (MedCPT) once the shared words are gone, while the embedding models stay high (except MedCPT). Here, Qwen3 alone is even better than the hybrid. Looks like BM25 just adds noise.
- Multi-paper questions are where the hybrid wins. Finding many right papers rewards breadth, and two different retrievers cover more ground than one.
- Hard-negative questions defeat everyone. No setup goes above 0.52. Unfortunately, this scores far below scores for other question types.
Do the retrievers fall into the traps?
If you recall, I told you that my PubMed query pulled some false-positive papers, which we utilized. We counted those papers if they had been retrieved in the top 5. For the 8 questions, I counted how often each setup put at least one trap PMID in its top 5:
Setup Questions with a trap in top 5
BM25 5/8
MedCPT 2/8
BGE 4/8
Qwen3 3/8
BM25 + MedCPT 5/8
BM25 + BGE 4/8
BM25 + Qwen3 4/8Keyword search falls into the traps most often because the traps share its keywords. The hybrid approach has some trade-offs. It finds more relevant papers on multi-paper questions, but it also brings back BM25’s traps, and those trap papers then go straight to the LLM in the generation step. The next logical question is how to fix this mess, and I think a cross-encoder reranker should help in this case, or we could put weights on the fusion process, with less weight on the keyword search (but this is a different story and a different post).
Part 4. Generation
Now we have a retriever that finds the right abstracts. The last step is the “generation” in RAG: Giving the question and the retrieved abstracts to an LLM and asking it to write an answer.
Choosing the model
I wanted an open-source model that anyone can run in Colab, so I used Qwen3–8B. It follows instructions well, and in 4-bit quantization it fits on a free Colab T4 GPU. It also completes a nice “Qwen3 end-to-end” story, since our best retriever was Qwen3-Embedding.
The prompt
The most important part of generation is the rules we give the model. For each question, the model gets the top 5 abstracts from the BM25 + Qwen3 hybrid and these instructions:
Answer the question using ONLY the abstracts provided.
- Cite every claim with the PMID in square brackets, e.g. [42853796].
- Cite only PMIDs from the provided abstracts.
- If an abstract is only loosely related, do not use it.
- If none of the abstracts answer the question, reply with exactly:
NO_RELEVANT_STUDIES
followed by one sentence explaining why.The last rule matters more than it looks. If we don’t explicitly allow the model to say “I don’t know”, it will almost always produce an answer from whatever we give it, even when the abstracts are irrelevant.
text = tokenizer.apply_chat_template(messages, tokenize=False,
add_generation_prompt=True,
enable_thinking=False)
inputs = tokenizer(text, return_tensors='pt').to(model.device)
out = model.generate(**inputs, max_new_tokens=400, do_sample=False)I turned off Qwen3’s “thinking” mode, so it answers directly, and I used greedy decoding (do_sample=False), so the answers are the same every time you run it.
Evaluating the answers
I checked the answers automatically, without a second LLM as a judge. All of these checks come down to looking at which PMIDs the model cited:
Check Result Ideal
Said "no relevant studies" on unanswerable questions 4/4 4/4
Wrongly said "no relevant studies" on answerable questions 1/44 0/44
Cited a PMID that was not in its context 0/48 0/48
Cited a trap paper (trap questions) 1/8 0/8
Cited the right paper when it was retrieved 76/78 78/78The model knows when to say “I don’t know.” On all four unanswerable questions, for example, “Which studies evaluated vision-language models for interpreting ECG tracings?”, it answered NO_RELEVANT_STUDIES instead of inventing an answer. And it was not over-cautious: It abstained wrongly only once out of 44 answerable questions.
It never invented a citation. Every PMID it cited was one of the five abstracts it was given.
The LLM is a second filter for traps. This was the most interesting result for me. In retrieval, the hybrid put a trap paper in the top 5 on 4 of the 8 trap questions. But the model cited a trap on only 1 of them. When it read the full abstracts, it noticed that most trap papers didn’t fit the question; for example, a study where GPT-4o read radiology reports when the question asked about images. So the thing that every retriever failed at, reading for a constraint, the LLM handled quite well.
The bottleneck is retrieval, not generation. When the right paper was in the top 5, the model used it 97% of the time. So when an answer misses the right paper, it is almost always because retrieval didn’t find it, not because the model ignored it.
What these checks can’t tell us. These checks look only at which papers were cited, not whether the answer describes them correctly. An answer can cite the right paper and still get its numbers wrong. A proper faithfulness evaluation, where an LLM judge checks every claim against its cited abstract, is the natural next step.
Limitations
This is a workshop, not a research paper, just an observation of results. So I want to be clear about what these results can and can’t tell you.
The corpus is tiny. We used only 100 abstracts. With so few documents, every question has few competitors, so all the scores are higher than they would be on a real corpus.
The labels are provisional. The right answer for each question is a list of “seed” PMIDs, drafted with the help of an LLM and reviewed by me, but not exhaustive. For multi-paper questions especially, there are probably relevant papers I didn’t list, and a retriever that finds them gets penalized for a correct answer.
The question set is small. 48 questions in total, and only 4 hard-negative and 4 unanswerable ones. With so few, a single question changes a result by 12–25 percentage points. Most differences between the top setups are not statistically significant, and the findings about traps and abstention should be read as observations, not conclusions.
