Sitemap
AIbizX

Applied AI and data: findings you can trust, methods you can use.

GraphRAG

When Simple RAG Isn’t Enough

7 min readNov 12, 2026

--

Press enter or click to view image in full size
Photo by majed swan on Unsplash

Some questions depend on how things connect, not just which passage looks similar. Here’s what GraphRAG adds, what it costs, and a sensible way to decide whether you need it.

A standard RAG system is very good at questions like “What’s our refund policy for annual plans?” The answer lives in one passage, the search finds it and the model reads it out.

Now try a different kind of question: “Which of our suppliers could delay the launch of Product X?” The answer isn’t in a single paragraph. Product X uses Component Y, which comes from Supplier Z, whose factory is affected by a delay mentioned in a separate report. To answer, you have to follow a chain of connections across several documents.

Simple RAG struggles with that. This article explains why, introduces GraphRAG, and gives you an honest view of when it’s worth the extra effort.

Where simple RAG runs out

Standard RAG retrieves the passages most similar to the question. That works when the answer sits in one place. It starts to fail when:

  • The answer needs several hops. You must connect fact A to fact B to fact C, and no single passage mentions all three.
  • The question is about relationships. “Who reports to whom?” “Which projects depend on this system?”
  • The question is about the whole collection. “What are the main themes across all these customer interviews?” There’s no single passage to retrieve, because the answer is a summary of everything.
  • The names change. The same person, company or product appears under different names in different documents, and a similarity search doesn’t connect them.

The common thread is that similarity search finds passages that look like the question. These questions need passages connected by relationships.

What a knowledge graph is

A knowledge graph stores information as things and the links between them:

  • Entities are the things: people, companies, products, places, concepts.
  • Relationships are how they connect: “works for,” “is part of,” “supplies,” “depends on.”

The basic unit is a triple: subject, relationship, object. For example: (Product X, uses, Component Y), (Component Y, supplied by, Supplier Z), (Supplier Z, located in, Region Q).

Once your information is stored this way, following a chain is easy. Start at Product X, walk to Component Y, then to Supplier Z, then to Region Q. Each step is a lookup, and there’s no need to hope that a search engine finds one passage mentioning all of them.

Here’s a tiny illustration in Python, with made-up data:

triples = [
("Product X", "uses", "Component Y"),
("Component Y", "supplied_by", "Supplier Z"),
("Supplier Z", "located_in", "Region Q"),
("Product W", "uses", "Component V"),
]

def neighbours(entity):
return [(rel, obj) for subj, rel, obj in triples if subj == entity]

def follow(entity, hops=3):
path, current = [entity], entity
for _ in range(hops):
steps = neighbours(current)
if not steps:
break
rel, current = steps[0] # simplest case: follow the first link
path.append(f"-{rel}->")
path.append(current)
return " ".join(path)

print(follow("Product X"))

That prints the chain from Product X through its component and supplier to the region. Real graph databases do this at scale, with proper query languages, but the principle is the same.

What GraphRAG is

GraphRAG combines the two ideas. It builds a knowledge graph from your documents, and uses the graph, as well as text search, to find what the model needs.

There’s more than one design, but a typical one looks like this:

Building the graph (done ahead of time):

  1. A language model reads each chunk of your documents and extracts the entities and relationships it mentions.
  2. These are assembled into a graph, merging different mentions of the same thing where possible.
  3. Some designs also group the graph into clusters, called communities, of closely connected entities, and write a summary of each.

Answering a question:

  • For specific questions about particular things, find the relevant entities, follow their connections to gather the related facts and passages, and give those to the model.
  • For broad questions about the whole collection, use the community summaries, so the model can answer “what are the main themes?” from summaries of the whole, not from a handful of passages.

A well-known 2024 paper from Microsoft Research, “From Local to Global: A Graph RAG Approach to Query-Focused Summarization,” described this approach and showed that it improved answers to global questions about a whole text collection, compared with plain RAG. It’s worth knowing that the paper’s tests concentrated on that kind of whole-collection question. It doesn’t show that graph methods beat plain RAG on simple lookups.

Cloud providers have started to offer GraphRAG as a feature. For example, Amazon Bedrock Knowledge Bases has had a GraphRAG option that uses Amazon Neptune. Check the current documentation for what’s available, because AWS’s knowledge base offerings have been changing this year.

When it helps, and when it doesn’t

Press enter or click to view image in full size

The costs

GraphRAG isn’t free, and the costs come in several forms.

Indexing cost. Building the graph typically means a language model reads every chunk to extract entities and relationships. For a large collection, that’s a lot of model calls, and it’s repeated when documents change.

Get Saman Siadati’s stories in your inbox

Join Medium for free to get updates from this writer.

Quality risk. Extraction can be wrong. A missed or invented relationship becomes a wrong fact in the graph. Merging entities, deciding that “IBM” and “International Business Machines” are one thing, is a hard problem.

Complexity. There’s a graph database or graph store to run, a new pipeline to maintain, and more places for things to break.

Freshness. Updating a graph when documents change is harder than re-embedding a few chunks.

Debugging. When an answer is wrong, there are more layers to check: the documents, the extraction, the graph, the traversal and the generation.

Cheaper things to try first

Before you build a graph, work through simpler fixes. Many multi-hop failures come from retrieval problems that are easier to solve.

  1. Improve chunking and context, as in the earlier articles, so each chunk carries enough information.
  2. Use hybrid search and a reranker, to find the right passages more reliably.
  3. Break the question down. Have a model split “Which suppliers could delay Product X?” into smaller questions, answer each with ordinary RAG and combine the results. It’s a form of prompt chaining, and it handles a surprising share of multi-hop questions.
  4. Use structured data directly. If the relationships already live in a database, such as products, parts and suppliers, let the model write a query, and skip the extraction step entirely.
  5. Add metadata, such as the product, supplier and region that each chunk mentions, and filter on it.
  6. Try an agent that can search several times, using what it finds to decide what to look for next.

Only if these fall short on your own test questions is it worth building a graph.

How to decide: a small experiment

  1. Write 20 or more multi-hop and whole-collection questions that matter to you, with known answers.
  2. Run them through your best plain RAG setup, with hybrid search, reranking and question decomposition. Record how many it gets right.
  3. Look at the failures. Are they really about connections between facts, or about retrieval you could still improve?
  4. If connections are the problem, try a graph approach on a small slice of your documents, perhaps a few hundred, and test the same questions.
  5. Compare quality against cost and effort, including indexing and upkeep.

If the graph version clearly wins on questions that matter, scale it up. If the improvement is small, you’ve saved yourself a lot of work.

Pitfalls

  1. Reaching for a graph too early. It’s the heaviest option, not the first.
  2. Trusting the extraction. Check a sample of the entities and relationships by hand.
  3. Ignoring duplicates. Several names for one entity fragment your graph.
  4. Forgetting permissions. A graph can connect facts from documents that different people are allowed to see. Make sure access control carries through.
  5. Letting it go stale. Plan for how updates reach the graph.
  6. No evaluation. Test with the same retrieval and answer measures as before, and add multi-hop questions.

A checklist

  1. Do your important questions really depend on connections between facts?
  2. Have you tried better chunking, hybrid search, reranking and question decomposition first?
  3. Could the relationships come from an existing database instead of extraction?
  4. Have you tested a small slice before committing the whole collection?
  5. Do you know the cost of building and updating the graph?
  6. Have you checked the extracted entities and relationships by hand?
  7. Do permissions carry through to the graph?
  8. Do your test questions include multi-hop and whole-collection cases?

The takeaway

Simple RAG finds passages that look like the question. GraphRAG adds a map of how things connect, which helps with multi-hop and whole-collection questions. It costs more to build and maintain, so try the cheaper fixes first, and let a test on your own questions tell you whether you need it.

Where we go next

That completes our series on RAG. Next we’ll move on to agents without the hype, starting with a plain-language guide to what an AI agent is.

This series is part of the AIBizX publication, where we share research and practical guides on AI and data.

--

--