<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Utgarg on Medium]]></title>
        <description><![CDATA[Stories by Utgarg on Medium]]></description>
        <link>https://medium.com/@utgarg23?source=rss-fdba1c20553d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*nueiJ3laFloDg3DC</url>
            <title>Stories by Utgarg on Medium</title>
            <link>https://medium.com/@utgarg23?source=rss-fdba1c20553d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 20:38:49 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@utgarg23/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Agentic GraphRAG on TigerGraph: From 19% RAG to 95% Accuracy]]></title>
            <link>https://medium.com/@utgarg23/agentic-graphrag-on-tigergraph-from-19-rag-to-95-accuracy-c6e1638b2b9f?source=rss-fdba1c20553d------2</link>
            <guid isPermaLink="false">https://medium.com/p/c6e1638b2b9f</guid>
            <category><![CDATA[tigergraph]]></category>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[graphrag]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <dc:creator><![CDATA[Utgarg]]></dc:creator>
            <pubDate>Sun, 04 Oct 2026 05:11:03 GMT</pubDate>
            <atom:updated>2026-10-04T05:11:03.449Z</atom:updated>
            <content:encoded><![CDATA[<h3>How graph retrieval, deterministic tools, agentic investigation, evidence evaluation, and reliability engineering transformed a conventional RAG pipeline.</h3><p>Some questions can be answered with a single retrieval step.</p><p>Others cannot.</p><p>A question may require identifying an entity, finding related events, comparing multiple candidates, resolving a date, performing an aggregation, and then deciding whether the retrieved evidence is actually sufficient.</p><p>That distinction became the central engineering problem behind our <strong>Agentic GraphRAG system for the TigerGraph Hackathon 2026</strong>.</p><p>We did not want to simply build another RAG demo. We wanted to measure something more practical:</p><blockquote><strong><em>When does conventional RAG stop being enough, and how much can graph-based and agentic investigation improve the result?</em></strong></blockquote><p>We built and benchmarked three approaches on the same 100-question public evaluation set:</p><ul><li>Classical RAG</li><li>GraphRAG</li><li>Agentic GraphRAG</li></ul><p>The final results were:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/890/1*amlONbvAIhWvQ8brTKMaig.png" /></figure><p>The Agentic GraphRAG system improved accuracy by <strong>76 percentage points over classical RAG</strong> and <strong>75 percentage points over GraphRAG</strong>. Pasted markdown(20261002–111901) Pasted markdown(20261002–111901)</p><h3>1. The Problem: Retrieval Alone Is Not Investigation</h3><p>A conventional RAG pipeline is conceptually simple:</p><pre>Question<br>   ↓<br>Embedding<br>   ↓<br>Vector Search<br>   ↓<br>Top-k Documents<br>   ↓<br>LLM<br>   ↓<br>Answer</pre><p>This works well when the answer is directly represented in one or a few semantically similar chunks.</p><p>But complex questions create a different problem.</p><p>Consider a question that implicitly requires:</p><ol><li>identifying a particular Olympic event,</li><li>finding the athletes associated with it,</li><li>determining which event occurred first,</li><li>comparing multiple candidates,</li><li>and producing a final answer from the combined evidence.</li></ol><p>A vector search system may retrieve a relevant document while still missing the relationship required to answer the question.</p><p>That led us through three increasingly capable architectures.</p><h3>2. Baseline 1 — Classical RAG</h3><p>The first system was a conventional vector RAG pipeline.</p><p>The query was embedded and matched against document chunks using vector similarity search.</p><p>The baseline used:</p><ul><li>Qwen3 Embedding 0.6B</li><li>1024-dimensional embeddings</li><li>TigerGraph’s vector storage/indexing</li><li>top-k vector retrieval</li><li>one LLM call per question</li></ul><p>The result was surprisingly weak on the full benchmark:</p><p><strong>19/100 correct — 19% accuracy.</strong></p><p>Evidence recall was only <strong>3%</strong>. Pasted markdown(20261002–111901)</p><p>The important lesson was not that vector search is useless.</p><p>It was that <strong>semantic similarity is not equivalent to evidence completeness</strong>.</p><p>The system could find text that looked relevant without necessarily finding the exact evidence required to answer the question.</p><h3>3. Baseline 2 — GraphRAG</h3><p>The next step was to introduce graph structure.</p><p>Instead of treating documents as isolated chunks, we represented relationships between the underlying entities and events.</p><p>The architecture became:</p><pre>Question<br>   ↓<br>Vector Search<br>   ↓<br>Seed Document / Entity<br>   ↓<br>Graph Expansion<br>   ↓<br>Related Evidence<br>   ↓<br>LLM<br>   ↓<br>Answer</pre><p>This allowed the system to exploit relationships that pure vector retrieval could not see.</p><p>However, the improvement was much smaller than expected.</p><p>GraphRAG reached:</p><p><strong>20/100–20% accuracy.</strong></p><p>Evidence recall increased from 3% to 16%, but accuracy barely moved.</p><p>The reason was an important engineering discovery.</p><p>Graph traversal is only useful when you reach the <strong>right graph neighborhood</strong>.</p><p>Our GraphRAG implementation depended on a seed retrieved from vector search. If the initial seed was wrong or did not expose the required graph relationship, the graph could not rescue the query.</p><p>The GraphRAG benchmark recorded substantial zero-context failures caused by missing outgoing DOCUMENT_HAS_EVENT traversal paths from retrieved seed chunks. Pasted markdown(20261002-111901)</p><p>This produced a critical insight:</p><blockquote><strong><em>Adding a graph to RAG does not automatically make the system intelligent. The system still needs to decide how to investigate.</em></strong></blockquote><p>That was the point where we moved from GraphRAG to Agentic GraphRAG.</p><h3>4. Agentic GraphRAG</h3><p>The final architecture introduced an investigation loop.</p><p>Instead of:</p><pre>Question → Retrieve → Answer</pre><p>we built:</p><pre>Question<br>   ↓<br>Triage<br>   ↓<br>Plan<br>   ↓<br>Select Tool<br>   ↓<br>Retrieve Evidence<br>   ↓<br>Evaluate Evidence<br>   ↓<br>Enough Evidence?<br>   ├── Yes → Synthesize Answer<br>   └── No  → Replan / Investigate Further</pre><p>The agent could determine that different questions require different retrieval strategies.</p><p>For example:</p><ul><li>a lookup question may need entity retrieval;</li><li>an aggregation question may require a deterministic GSQL operation;</li><li>a temporal question may require date/event reasoning;</li><li>a superlative question may require comparing candidates;</li><li>a multi-hop question may require several retrieval operations.</li></ul><p>The final system therefore combined:</p><ul><li>deterministic GSQL tools,</li><li>hybrid vector search,</li><li>graph traversal,</li><li>similarity search,</li><li>entity and relationship retrieval,</li><li>evidence evaluation,</li><li>and bounded replanning.</li></ul><p>This was the core difference.</p><p><strong>GraphRAG gave the system a graph.</strong></p><p><strong>Agentic GraphRAG gave the system a mechanism for deciding how to use that graph.</strong></p><h3>5. The Technology Stack</h3><p>The production pipeline used:</p><h3>LLM</h3><p><strong>openai/gpt-oss-20b through AIRouter</strong></p><p>We used low reasoning effort and temperature 0 for more deterministic benchmark behavior.</p><h3>Embeddings</h3><p><strong>qwen3-embedding:0.6b</strong></p><p>The embeddings are <strong>1024-dimensional</strong> and stored in TigerGraph’s DocumentChunk.qwen_embedding vector attribute.</p><h3>Graph Backend</h3><p><strong>TigerGraph Cloud — Savanna</strong></p><p>The graph stores the relationships between document chunks, events, and Olympic entities.</p><h3>Retrieval</h3><p>The final system combines:</p><ul><li>hybrid vector retrieval,</li><li>deterministic GSQL queries,</li><li>graph traversal,</li><li>entity/relationship retrieval,</li><li>and specialized tools for structured questions.</li></ul><h3>Evaluation</h3><p>All three systems were evaluated against the same <strong>100-question public dataset</strong> using the unified evaluator.</p><p>This matters because comparing systems on different questions would make the results difficult to interpret.</p><p>The authoritative dataset contains exactly 100 questions and has a recorded SHA-256 hash for reproducibility. Pasted markdown(20261002–111901)</p><h3>6. Deterministic Tools Were a Major Part of the Improvement</h3><p>One of the biggest lessons from the project was that an agent does not need to solve everything through free-form LLM reasoning.</p><p>For structured questions, deterministic tools are often better.</p><p>We therefore added tools for operations such as:</p><ul><li>aggregation,</li><li>temporal queries,</li><li>superlative queries,</li><li>structured entity lookup,</li><li>graph relationships,</li><li>and other Olympic-specific retrieval operations.</li></ul><p>This allowed the LLM to decide <strong>what operation was needed</strong>, while the actual computation could be performed deterministically.</p><p>For example:</p><pre>User Question<br>      ↓<br>Agent identifies &quot;aggregation&quot;<br>      ↓<br>Aggregation GSQL Tool<br>      ↓<br>Exact structured result<br>      ↓<br>Agent evaluates evidence<br>      ↓<br>Final answer</pre><p>This is substantially safer than asking the LLM to estimate or reconstruct a numerical result from retrieved prose.</p><h3>7. The Reliability Engineering Was Just as Important as Retrieval</h3><p>Building an agentic system introduced another problem:</p><p><strong>agents can loop.</strong></p><p>An unrestricted agent can repeatedly:</p><ul><li>call an LLM,</li><li>retry a tool,</li><li>replan,</li><li>retrieve more documents,</li><li>and continue indefinitely.</li></ul><p>That became a real engineering issue during development.</p><p>We encountered provider rate limits, runaway retries, slow local-model execution, tool-call errors, and thermal problems while testing local models.</p><p>The solution was to introduce hard execution budgets.</p><p>The final architecture included controls such as:</p><pre>Maximum LLM calls<br>Maximum provider attempts<br>Maximum replans<br>Bounded agent execution<br>Bounded retry handling</pre><p>We also configured the OpenAI-compatible client with no automatic unbounded retries and implemented explicit provider-attempt limits.</p><p>This changed the agent from an experimental loop into a bounded production pipeline.</p><p>That distinction matters.</p><blockquote><strong><em>An agentic system is not production-ready simply because it can reason. It needs predictable failure boundaries.</em></strong></blockquote><h3>8. AIRouter Integration</h3><p>Another important engineering issue appeared when integrating AIRouter.</p><p>The provider was OpenAI-compatible, but the internal capability detection initially did not correctly recognize the openai/gpt-oss-20b model for tool-calling behavior.</p><p>We fixed the capability layer so that provider prefixes were stripped before checking model capabilities and OpenAI-compatible providers such as AIRouter were correctly recognized.</p><p>The resulting configuration was:</p><pre>Provider: AIRouter<br>Model: openai/gpt-oss-20b<br>Temperature: 0<br>Reasoning effort: low</pre><p>This enabled the Agentic modes to operate correctly through the provider.</p><p>The API key remained backend-side rather than being exposed to the frontend.</p><h3>9. Benchmarking the Three Approaches</h3><p>The final comparison was performed on the same 100-question public benchmark.</p><p>The results were:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/753/1*bkURxcGSCjjuAdN5_rTIqQ.png" /></figure><p>The final Agentic GraphRAG benchmark took <strong>3,943 seconds</strong>, approximately <strong>65.7 minutes</strong>, for all 100 questions.</p><p>The important tradeoff is clear.</p><p>Agentic GraphRAG is not the fastest architecture.</p><p>It is the architecture that spends additional computation where the question actually requires investigation.</p><h3>10. The Most Important Result: Multi-Hop Questions</h3><p>The final benchmark achieved:</p><ul><li>Aggregation: <strong>21/21–100%</strong></li><li>Lookup: <strong>19/19–100%</strong></li><li>Temporal: <strong>22/22–100%</strong></li><li>Superlative: <strong>10/10–100%</strong></li><li>Multi-Hop: <strong>23/28–82.1%</strong></li></ul><p>Overall:</p><p><strong>95/100–95% accuracy.</strong></p><p>The multi-hop category remained the hardest.</p><p>That makes sense.</p><p>A multi-hop question does not simply ask:</p><blockquote><em>“Find this entity.”</em></blockquote><p>It asks the system to construct a chain of evidence.</p><p>The final system solved 23 of the 28 multi-hop questions, while achieving perfect accuracy across the other four question categories. Pasted markdown(20261002–111901)</p><p>This also showed us where future improvements should be concentrated.</p><h3>11. The Final Five Failures</h3><p>We did not treat the remaining five failures as random mistakes.</p><p>They revealed recurring failure mechanisms.</p><p>The final failures involved combinations of:</p><ul><li>structural tool-routing failures for natural-language venue/date queries,</li><li>hybrid vector top-k retrieval misses,</li><li>underspecified multi-candidate venue/date questions,</li><li>an evaluator/benchmark labeling artifact,</li><li>and a case where the replanning budget was exhausted.</li></ul><p>These mechanisms overlap, so they should not be interpreted as five completely independent failure categories.</p><p>This was useful because the failures told us what to improve next.</p><p>The goal of benchmarking was not simply to produce a large percentage.</p><p>It was to identify <strong>why the system still fails</strong>.</p><h3>12. Phase-by-Phase Improvement</h3><p>The system did not jump directly from 19% to 95%.</p><p>It evolved through repeated engineering and evaluation cycles.</p><p>The agentic pipeline itself progressed through several benchmark stages:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/378/1*30PjZ2mqN-h3I2gKZOujjg.png" /></figure><p>Phase 15 added targeted repairs around multi-hop reasoning, output handling, synthesis, and multi-event disambiguation.</p><p>The final improvement from <strong>88% to 95%</strong> was therefore not achieved by simply changing the model.</p><p>It came from engineering the complete retrieval-and-reasoning system.</p><h3>13. What We Learned</h3><h3>1. Better retrieval is not always enough</h3><p>A stronger vector search can improve recall, but some questions require structured operations and relationships.</p><h3>2. GraphRAG is not automatically agentic</h3><p>Having a graph does not mean the system knows which part of the graph to investigate.</p><h3>3. Deterministic tools are extremely valuable</h3><p>If a question asks for a count, maximum, temporal comparison, or structured relationship, a deterministic query can be much more reliable than unconstrained generation.</p><h3>4. Evidence should be evaluated separately from the answer</h3><p>A model can produce a plausible answer without actually retrieving the evidence needed to support it.</p><p>Our final system explicitly treated evidence as a first-class part of the investigation.</p><h3>5. Agents need budgets</h3><p>Without limits, retries and replanning can turn a useful agent into an expensive or unbounded process.</p><h3>6. Benchmark failures are engineering signals</h3><p>The remaining failures were more useful than simply knowing that the score was 95%.</p><p>They showed us where retrieval, routing, disambiguation, and planning still need improvement.</p><h3>14. Building Explainability Into the System</h3><p>Another part of the project was the benchmark dashboard.</p><p>Instead of only reporting:</p><pre>95 / 100</pre><p>we wanted to be able to inspect:</p><ul><li>individual questions,</li><li>PASS/FAIL status,</li><li>evidence,</li><li>retrieval behavior,</li><li>agent traces,</li><li>benchmark comparisons,</li><li>failure categories,</li><li>provenance,</li><li>and system evolution.</li></ul><p>The dashboard therefore provides sections for:</p><ul><li><strong>Overview</strong></li><li><strong>System Evolution</strong></li><li><strong>Benchmark</strong></li><li><strong>Evidence / Logs</strong></li><li><strong>Technical / Provenance</strong></li></ul><p>This makes the benchmark more than a leaderboard number.</p><p>It becomes an engineering diagnostic tool.</p><h3>15. Why the 95% Matters</h3><p>The most interesting number in the project is obviously <strong>95%</strong>.</p><p>But the more important result is the progression:</p><pre>Classical RAG<br>19%<br>   ↓<br>GraphRAG<br>20%<br>   ↓<br>Agentic GraphRAG<br>95%</pre><p>The jump from RAG to GraphRAG was only one percentage point in normalized accuracy.</p><p>The jump from GraphRAG to Agentic GraphRAG was <strong>75 percentage points</strong>.</p><p>That suggests something important:</p><blockquote><strong><em>For complex knowledge tasks, the critical improvement may not come from adding more retrieved context. It can come from giving the system the ability to decide what to retrieve next.</em></strong></blockquote><p>The agent effectively turns retrieval into an investigation process.</p><h3>16. The Architecture in One View</h3><p>The final architecture can be summarized as:</p><pre>                         ┌──────────────────┐<br>                         │      User        │<br>                         └────────┬─────────┘<br>                                  │<br>                                  ▼<br>                         ┌──────────────────┐<br>                         │   Agent Triage   │<br>                         └────────┬─────────┘<br>                                  │<br>                                  ▼<br>                         ┌──────────────────┐<br>                         │ Planner / Router │<br>                         └────────┬─────────┘<br>                                  │<br>              ┌───────────────────┼───────────────────┐<br>              │                   │                   │<br>              ▼                   ▼                   ▼<br>       Vector Search        GSQL Tools         Graph Retrieval<br>              │                   │                   │<br>              └───────────────────┼───────────────────┘<br>                                  │<br>                                  ▼<br>                         ┌──────────────────┐<br>                         │ Evidence Eval.   │<br>                         └────────┬─────────┘<br>                                  │<br>                         ┌────────┴────────┐<br>                         │                 │<br>                      Sufficient?        Missing<br>                         │                 │<br>                         ▼                 ▼<br>                    Synthesis          Replan<br>                         │                 │<br>                         └───────┬─────────┘<br>                                 ▼<br>                         ┌──────────────────┐<br>                         │ Final Answer     │<br>                         └──────────────────┘</pre><p>The central design principle is simple:</p><p><strong>Retrieve → inspect → evaluate → investigate further if necessary → answer.</strong></p><h3>17. What We Would Improve Next</h3><p>A 95% benchmark is strong, but it is not the end of the engineering process.</p><p>The remaining 5% points toward several future improvements:</p><ul><li>stronger multi-hop planning,</li><li>improved natural-language-to-tool argument extraction,</li><li>better handling of ambiguous venue/date constraints,</li><li>broader retrieval beyond fixed top-k candidates,</li><li>smarter candidate disambiguation,</li><li>more efficient agent planning,</li><li>and additional evaluation beyond exact answer accuracy.</li></ul><p>The biggest opportunity is reducing the gap between <strong>82.1% multi-hop accuracy</strong> and the 100% achieved on the other question types.</p><h3>18. Conclusion</h3><p>This project started with a straightforward question:</p><blockquote><strong><em>Can a graph improve RAG?</em></strong></blockquote><p>The benchmark gave us a more interesting answer.</p><p>A graph by itself helped expose relationships, but it did not solve the core problem.</p><p>The larger improvement came when the system could <strong>reason about what it needed to investigate</strong>, select an appropriate retrieval mechanism, evaluate the evidence, and continue when the evidence was insufficient.</p><p>That is what changed the architecture from GraphRAG into Agentic GraphRAG.</p><p>The final benchmark demonstrates the progression:</p><p><strong>19% RAG → 20% GraphRAG → 95% Agentic GraphRAG.</strong></p><p>And perhaps the biggest lesson from the project is this:</p><blockquote><strong><em>The future of RAG is not simply retrieving more context. It is building systems that know when the context is insufficient, know what to investigate next, and know when they finally have enough evidence to answer.</em></strong></blockquote><h3>Project &amp; Reproducibility</h3><p>The complete project, benchmark artifacts, dashboard, implementation, and engineering history are available on GitHub:</p><p><a href="https://github.com/utgarg07-spec/TigerGraphHackathon?utm_source=chatgpt.com">TigerGraphHackathon — GitHub Repository</a></p><p>The authoritative benchmark artifacts include the final 100-question Agentic GraphRAG results, per-question traces, three-way comparison data, and the unified evaluator. Pasted markdown(20261002–111901)</p><p><strong>Technology:</strong> TigerGraph · GraphRAG · Agentic AI · RAG · GSQL · Vector Search · Qwen Embeddings · gpt-oss-20b · AIRouter</p><p><strong>Final benchmark:</strong> 95/100 accuracy · 95/100 evidence recall · 100/100 HTTP success</p><p><strong>Cost:</strong> $0 for the benchmark environment using the reported local/open-weight and free-tier setup. Pasted markdown(20261002–111901)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c6e1638b2b9f" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>