RAG: Giving a Frozen Model an Open Book Test
How We Got Here: Episode 16 of ~26. We are walking the AI timeline one contribution at a time.
At SciPy 2024, my colleagues and I at UW’s Scientific Software Engineering Center taught a tutorial on building a generative AI copilot for scientific software. We used OLMo and LangChain, with an astronomy example that let us compare answers with and without retrieved context.
One of the questions in the materials is simply, “What is Astropy?” The model can produce an answer without looking anything up. Once we give it documentation, we can see how that material changes what it says. The saved answers are still in the repository, which makes this a useful example to return to rather than asking you to take the demo’s success on faith.
For me, the question behind that work is whether I can ask about the material I already have and get an answer I can check. Scientific documentation, an internal wiki, a folder of papers. We have the information somewhere, but the model doesn’t automatically have access to it.
That becomes obvious when you ask about your own software. A model may explain authentication quite well and still have no idea where the authentication logic lives in your codebase. We need a way to put the relevant material in front of it when the question arrives.
Retrieval-Augmented Generation, or RAG, is a way to connect those pieces. The application searches a collection of documents, retrieves passages that might answer the question, and gives those passages to a model as material for its response. The search does some of the work before the model starts writing.
What the weights can’t give you
In Episode 15, we looked at what changed when people could download Llama’s weights and run the model themselves. Researchers could inspect and adapt the model themselves, instead of relying on access through someone else’s service.
Running a model locally doesn’t make it familiar with the files on your computer. Its pretrained weights reflect what it learned during training. Once that training stops, the weights are fixed unless someone trains the model further. Releasing it doesn’t start a process that keeps those weights up to date.
In that sense, the weights are a time capsule. The world keeps moving outside it: papers get published, APIs change, a team updates its internal documentation. None of that arrives in the weights automatically. Even the release date can be misleading, because the training data may stop well before it.
That doesn’t prevent the model from working with something new. Give it yesterday’s paper in the prompt and it can use that text to answer questions, without changing its weights. RAG gives us a way to find and supply that material when we need it.
The model can still produce an answer. In Episode 12 and Episode 13, we followed how predicting the next token gives language models their fluency. A plausible continuation can include a claim that is false or unsupported. The model may acknowledge that it lacks information, but we can’t rely on it doing so every time. An answer can read smoothly enough that the missing knowledge is difficult to notice.
One option is to train the model further on our documents. Fine-tuning can help adapt a model to a task, but it also means preparing training data, running training, and checking the result. If the goal is to make yesterday’s documentation changes available today, that is a lot of machinery between an edit and a usable answer.
There is also the source question. A fact learned during training doesn’t necessarily come with a reliable way to recover the document it came from. For my research work, knowing where to check an answer is part of what makes it useful.
Let it bring the book
Think about an open-book exam. You still need to understand the question and work out an answer, but you can consult the relevant page instead of relying entirely on memory.
A document-based assistant can work the same way. In the setup I’m describing here, the model stays as it is. When a question arrives, the application finds relevant passages and includes them in the prompt. Updating the documents changes what the model can read on its next request, without requiring another training run.
Suppose a new team member asks how to recover from an expired login. Our internal documentation includes a recovery procedure for invalidating stale session tokens. If we can find that section and pass it to the model, it has something specific to use when answering. We can also include the document title and a link so the person reading the answer can open the full procedure.
This idea predates ChatGPT. Patrick Lewis was the first author of the 2020 paper “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,” with a team that included Ethan Perez, Aleksandra Piktus, Fabio Petroni, Douwe Kiela, and other collaborators. They brought a pretrained retriever and generator together so an answer could draw on both the model’s parameters and retrieved Wikipedia passages.
One of the building blocks came from Vladimir Karpukhin and colleagues’ Dense Passage Retrieval, also published in 2020. That work trained separate question and passage encoders to put useful question–passage pairs near each other in vector space. RAG used that retrieval approach with BART, a text generation model. The contribution was in how those pieces worked and learned together; searching for evidence before answering already had a history in question-answering systems.
Their implementation trained a query encoder and a BART generator together while keeping the document encoder and index fixed. That differs from the prompt-based application we’re walking through, which can use an existing model without changing its weights. Both give the generator access to retrieved information. RAG describes that combination; it doesn’t require every implementation to use the same training process.
How words became vectors
A keyword search can miss a passage when the question and document use different words. “Car” and “automobile” are different strings, even though a person reading them sees a relationship immediately.
One way to represent that relationship is to give each word a list of numbers, called a vector, learned from the contexts where it appears. Words used in similar contexts can end up with similar vectors. We can then measure relationships between those lists of numbers instead of requiring the words to match exactly.
Researchers were learning numerical representations of words well before 2013. That year, Tomas Mikolov and colleagues at Google introduced efficient methods associated with word2vec, making it practical to learn word vectors from large collections of text.
The famous example is an analogy: take the vector for “king,” subtract “man,” add “woman,” and look for a nearby word vector. “Queen” can emerge as the answer. It is an approximation learned from patterns in text, and the result depends on the trained vectors. It isn’t a rule that works for every relationship we might try.
I still find that a little uncanny. A relationship we express in words can also appear as a direction in a space of numbers. We didn’t have to write a separate rule connecting every royal title to every other one. Training picked up enough regularity in how the words were used to make some of those relationships available to a calculation.
Modern text embeddings apply related ideas to sentences and passages. An embedding model turns a piece of text into a vector that can be compared with other vectors. The goal for retrieval is to make passages useful for similar questions easy to find, including when their wording differs.
Searching internal documentation
Before anyone asks a question, we split our documents into passages and compute an embedding for each one. We store the vectors alongside the text and enough information to identify its source.
Vector search can use a library such as FAISS, PostgreSQL with the pgvector extension, or a managed service such as Pinecone. Those are different kinds of tools. What we need from them here is the ability to find stored vectors similar to a query vector.
For a simple setup, we embed the question using the same model that embedded the passages. Other systems use separate question and document encoders trained to work together. Either way, the query vector has to be comparable with the document vectors we’re searching.
The search returns a small set of nearby passages. This is nearest-neighbor search: finding vectors close to the query according to the chosen similarity measure. The number requested is often called k, as in “retrieve the top five.” It is a setting we choose, not a guarantee that five useful passages exist.
For “how do we handle expired logins,” the recovery procedure for stale session tokens might rank highly even without an exact keyword match. We can inspect the results before we ask the language model to do anything with them.
Keyword search and combinations of keyword and vector search can also supply passages for RAG. For an assistant searching internal documentation, we could compare them on the same questions before choosing how to search.
From a passage to an answer
Once the search has returned its candidates, the application puts the selected text into the prompt alongside the question. It also supplies instructions about how to use that material: answer from the passages, identify the sources, and say when there isn’t enough information.
The generator still produces tokens in the usual way. Its context now includes the recovery procedure, so it has the team’s written procedure available while answering. If we attach source identifiers to the passages, we can ask it to use those identifiers in its citations.
The application follows this sequence:
documents → passages → embeddings and index → question → retrieval → prompt with sources → answer
The document work happens ahead of the question. Retrieval and generation happen when someone asks. When project documentation changes, we update the affected passages and their entries in the index.
A whole manual represented by one vector gives the search little ability to distinguish a paragraph about authentication from one about deployment. Smaller passages let us retrieve those sections separately. Cut too narrowly, though, and a sentence can lose the heading or exception that makes it understandable.
When testing this, I care about seeing the actual text retrieved for a question. Imagine the recovery instructions say, “Use this procedure for development environments only,” immediately before the steps. If the steps become one chunk and the warning ends up in another, the model may receive a perfectly accurate procedure with the restriction missing.
That is the tension in chunking: the unit we make easy to retrieve may be too small to interpret. A sentence beginning “this method” depends on something outside the sentence. A table needs its headers. Even an intact paragraph may describe an exception whose scope was established a page earlier. We haven’t changed the words, but we have changed what surrounds them.
I would start by preserving headings and document structure, then inspect whether the retrieved passage needs its neighboring text. Overlapping chunks help with boundaries; retrieving a small passage and expanding it to its parent section keeps more of the explanation together. Another approach, contextual retrieval, adds a short explanation of where a chunk belongs before indexing it. These choices address missing context more directly than simply asking for more unrelated search hits. They also increase the text we store or send, so the next question is how much time we can afford to spend finding the answer.
How long should finding the answer take?
My priority for a research assistant is getting the supporting evidence into the prompt. I can tolerate some waiting for a useful answer. An interactive tool still needs a response-time budget, though, and a batch literature review has a different budget from a chat someone is using during a meeting.
There are two meanings of “accuracy” to separate here. A vector index can accurately return the nearest vectors while the passages themselves fail to answer the question. Search engineers also measure how closely an approximate search matches an exact search. That is a property of the index, not proof that the answer is correct.
Get Cordero Core’s stories in your inbox
Join Medium for free to get updates from this writer.
At larger scales, approximate nearest-neighbor search can avoid comparing a query with every stored vector. It trades some ability to recover the exact nearest results for speed. FAISS’s index guide documents settings that let a search explore more candidates at the cost of more work. A slower search may recover evidence a faster setting missed, but it cannot make an unsuitable embedding represent the question better.
I would measure whether the needed evidence arrives, whether the resulting answer is supported, and how long the whole request takes. Keep the unusually slow requests in that measurement too. Improving a database lookup won’t help much if generation accounts for most of the wait. Decide what counts as an acceptable answer, then look for the fastest configuration that still meets that standard on the test questions.
A second look at the search results
A fast first search often leaves us with a mixed shortlist. Reranking scores those candidates again, with a method that can afford to examine each one more closely because the collection has already been narrowed down.
A common choice is a cross-encoder, which processes the question and a candidate passage together. That lets it compare their words directly, instead of relying only on independently computed vectors. The Sentence Transformers walkthrough uses this two-stage design. Running that model over an entire large collection would be expensive; running it over a shortlist is more manageable.
Say we retrieve 50 candidates, rerank them, and send five to the generator. Reranking adds work, and it only sees the candidates we give it. If the relevant passage never made the shortlist, it has nothing to promote. If the passage lost its qualifying context during chunking, assigning it a better score doesn’t restore that context either.
Rodrigo Nogueira and Kyunghyun Cho published “Passage Re-ranking with BERT” in 2019, before the paper that named RAG. RAG applications adopted an existing information-retrieval technique to improve which evidence reaches the generator.
When the question spans the collection
“Which Astropy function transforms these coordinates?” points toward a fairly specific piece of documentation. “What themes connect the research in this collection?” asks for something spread across many documents. A few individually similar passages may give us a narrow sample of that larger answer.
Microsoft’s Darren Edge, Ha Trinh, and colleagues addressed this problem in the 2024 paper “From Local to Global: A Graph RAG Approach to Query-Focused Summarization.” Their GraphRAG builds a graph of entities and relationships extracted from source text, groups related entities, and prepares summaries of those groups. Its global answering method combines responses drawn from those summaries.
GraphRAG still starts from chunks. The graph and summaries add connections across them; they don’t reconstruct every detail that a boundary might separate. Building those representations requires additional model calls, and extraction or summarization can introduce errors. I would consider it for questions about relationships or themes across a collection. A lookup of one documented function doesn’t, by itself, justify building that machinery.
The example we used at SciPy
Our SciPy tutorial used a simpler pipeline that we can inspect step by step.
Our SciPy tutorial materials include a pair of saved answers to a simple question: “What is Astropy?”
Without retrieval, OLMo gives a broad description of an astronomy software project. With retrieval, it describes Astropy as a Python package for astronomical data analysis and discusses material from astropy.utils, including utilities for downloading data and compatibility between versions.
The second answer also spends quite a bit of time on that utility package, even though the question asked about Astropy as a whole. You can see the retrieved context influencing what the model chooses to explain. Whether that makes it a better answer depends on what the person asking needed to know.
We organized the tutorial so participants could work through domain-specific questions, add retrieval, build a chat application, and then try their own data. If you want to follow the implementation, the retrieval notebook is in the archived 2024 materials. The README links the setup and the rest of the notebooks.
The collection behind that notebook combines astrophysics abstracts from arXiv with Astropy documentation. That gives us two kinds of material to search: descriptions of scientific work and instructions for using scientific software. A question about dark matter and a question about coordinate transformations can draw on the same collection, but they need very different passages.
There are also two models doing different jobs. The embedding model, sentence-transformers/all-MiniLM-L12-v2, turns text into vectors with 384 numbers. OLMo writes the answer. The embedding model never has to explain dark matter, and OLMo doesn't search the vector database itself. LangChain connects those operations in the application.
That separation is useful when changing the system. Replacing the generator doesn’t automatically require rebuilding the document embeddings. Replacing the embedding model does: the stored passages and incoming questions need compatible representations. Two models can both produce lists of 384 numbers without assigning those numbers the same meaning.
We used Qdrant to hold the vectors and their documents. The database preparation notebook shows how that collection was assembled; the retrieval lesson loads a prepared copy so participants can get to asking questions.
The retriever requests two passages using maximum marginal relevance, abbreviated MMR. This adds a preference for variety to the search: choose passages that relate to the question while avoiding results that repeat what an already selected passage says.
Imagine the collection contains several nearly identical introductions to Astropy. A similarity search could fill the available space with those introductions. MMR gives a different passage a chance to contribute something else. Whether that helps depends on the question. For a narrow question about a function parameter, two closely related passages might be exactly what we need. In the notebook, k=2 is a choice we can change and inspect.
Before OLMo gets the question
One cell in the tutorial is easy to hurry past: it prints the completed prompt.
By then, the application has retrieved documents, pulled out their text, and inserted that text into a message with the question. Printing the message lets us read what OLMo is about to receive. I like having that available because “we gave it the docs” leaves a lot unspecified. This cell shows which words actually made it through.
The formatting function joins each document’s page_content with blank lines. It doesn't include the document metadata in that text. So even though a retrieved document can carry information about its source, this particular prompt doesn't automatically give the generator a title or URL to cite. Those fields need to be carried through deliberately if we want them in the answer.
Later, the retrieval chain returns a result with input, context, and answer. That is a useful object to inspect: the original question, the documents retrieved for it, and the generated response together. When an answer looks odd, we can read the associated context without trying to reconstruct the search afterward.
The notebook also calls OLMo with and without retrieved context. That makes the effect visible in a lesson. For an evaluation, I would keep the question wording, instructions, and generation settings consistent and vary the supplied context. A single pair of answers can suggest something to investigate; repeated questions under comparable conditions let us measure it.
These are archived 2024 teaching materials, with their own setup instructions and dependencies. I use them here because the individual steps are exposed. You can follow the documents all the way into the prompt instead of stopping at a chat window.
A citation gives me somewhere to check
The presence of a source link doesn’t establish that the answer is supported. The model might cite the right document for the wrong claim, omit an important qualification, or combine passages in a way that none of them supports. Research on evaluating generated citations treats citation quality as something to measure alongside answer quality.
In the login example, I can open the cited passage and check whether the answer describes the documented procedure.
When the answer is wrong
I start with what the search returned. Did it find the section containing the answer? Did it include an older version of the recovery procedure? Was the warning separated from the steps?
Those are retrieval and document-preparation problems. A new prompt may change how the answer sounds without correcting any of them. Keeping the retrieved passages with the response makes them easier to investigate.
The generator can also mishandle a passage that contains the answer. It can miss a detail that was present or give too much weight to another passage. “Lost in the Middle” found that language models’ use of relevant information could vary substantially depending on where it appeared in a long context.
Adding more text is therefore a choice to test. Context windows have limits, and extra passages add processing work. Standard full attention has a quadratic cost in sequence length, a concern we met in Episode 10. Implementations differ, but even a model that accepts a long prompt has to find and use the relevant material within it.
We also have to separate similarity from relevance. A retrieved passage might discuss session tokens in detail and still say nothing about recovery after expiry. It shares the subject of the question without answering it. Changing the number of retrieved passages can help or hurt; the useful question is which additional text changes the answer and why.
For a first test on your own documents, collect questions that people actually ask and mark the passages needed to answer each one. Include a few where the collection has no answer. Otherwise, a system that always produces a confident response can look better than it deserves.
One test question might ask how to recover an expired session in development. Another asks for the production procedure, which the collection might not contain. A third uses an exact error message. That last case gives keyword search a fair chance: sometimes the string someone copied from a terminal is the best search query we have.
Run those questions through retrieval before generating answers. Record whether the needed passage appeared among the selected results and whether its essential qualifications survived the chunk boundaries. If a question needs two sections, finding only one is incomplete even when it ranks first. Keep the missed cases beside the successful ones as you change the chunk size or search method.
Then give the generator the passages you already checked. Read its answer against them, including whether it preserved the environment restriction and acknowledged the missing production instructions. This separates a model that cannot use the evidence from a search that never supplied it. You can also supply the correct passage manually to investigate a failed case without rebuilding the index first.
Version questions deserve their own examples. Two versions of the documentation can disagree because one describes a retired system. Preserve version and status information with the passages, then decide which documents are eligible for a particular question. A highly similar paragraph from the wrong release can otherwise rank above the procedure the team uses now.
For an initial comparison, I would keep a small worksheet: question, expected source, retrieved passages, answer, and what needed correcting. Record response time alongside answer quality, using the budget you chose earlier. Compare a basic keyword search with the vector approach on those same questions. That gives us something concrete to examine before introducing another model or a more elaborate retrieval pipeline.
The model and the material around it
RAG gives us another place to work when an answer is wrong. We can change the documents the system consults, how it searches them, and what reaches the prompt.
For a “chat with your docs” tool, that work can be more immediately useful than choosing a larger model. If the recovery procedure never reached the prompt, a more fluent answer won’t recover it. If the passage was there and the model misread it, we have a different problem to investigate.
The open-book comparison helps me keep those responsibilities in view. Supplying the book is a start. I still need to see which page the system opened and what it made of it.
Next Tuesday, Episode 17 turns to tool use: giving a model a way to request a function call, run code, or interact with an API. Reading internal documentation can help someone understand a procedure. Acting on that procedure brings another set of decisions about what the system should be allowed to do.
If you’re building an assistant over your own documents, where has it been hardest to get a useful answer: finding the passage, keeping the documents current, or getting the model to use what you gave it? I’m interested in the cases where the source was there and the answer still went wrong.
You can follow on Medium or subscribe on Substack as we work through the parts of the systems we’re building with now.
Cordero is a senior research software engineer at the Scientific Software Engineering Center at the University of Washington’s eScience Institute. If this was useful, send it to the teammate whose docs bot keeps recommending an outdated recovery procedure.
