<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Rhitam Deb on Medium]]></title>
        <description><![CDATA[Stories by Rhitam Deb on Medium]]></description>
        <link>https://medium.com/@neevdeb26?source=rss-e44d9391c5fe------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*nYT0x-u7bS703YDkmw_TZA.jpeg</url>
            <title>Stories by Rhitam Deb on Medium</title>
            <link>https://medium.com/@neevdeb26?source=rss-e44d9391c5fe------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 14:40:00 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@neevdeb26/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[When Communities Refuse to Sit Still: A Hitchhiker’s Guide to Dynamic Knowledge Graphs]]></title>
            <link>https://medium.com/@neevdeb26/when-communities-refuse-to-sit-still-a-hitchhikers-guide-to-dynamic-leiden-2dce44df10c8?source=rss-e44d9391c5fe------2</link>
            <guid isPermaLink="false">https://medium.com/p/2dce44df10c8</guid>
            <category><![CDATA[graphrag]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[knowledge-graph]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Rhitam Deb]]></dc:creator>
            <pubDate>Fri, 04 Sep 2026 05:57:01 GMT</pubDate>
            <atom:updated>2026-09-04T07:22:41.784Z</atom:updated>
            <content:encoded><![CDATA[<p>How to update graph communities as networks evolve with Python, modularity and a little help from yesterday’s partition.</p><p>Hello there, fellow graph enthusiast! 👋</p><p>A graph looks static only when we take a screenshot of it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rURshWbTbiBMXpF5BTBrcQ.png" /><figcaption>A graph looks static only when we take a screenshot of it.</figcaption></figure><p>Social connections change. Web pages appear and disappear. Users join communities, leave them, or suddenly connect two previously unrelated groups. Protein-interaction networks evolve as new evidence arrives. Even a Stack Overflow graph changes every time someone asks a question or answers one.</p><p>Yet many community-detection workflows still behave as if the graph is frozen in time.They take the entire graph, run Leiden or Louvain from scratch, and repeat the same expensive process whenever a few edges change.</p><p>That raises an obvious question:</p><blockquote>If only a small part of the network changed, why should we forget everything we already learned?</blockquote><p>Recently, I came across the technical report <a href="https://arxiv.org/html/2405.11658v1">“A Starting Point for Dynamic Community Detection with Leiden Algorithm”</a>. The paper explores how Leiden can be adapted for evolving graphs by reusing previous community assignments and focusing computation around the parts of the graph affected by new updates.</p><p>Let us understand the idea step by step.</p><h3><strong>First things first: what exactly is a community?</strong></h3><p>A graph consists of vertices and edges.</p><p>Vertices may represent users, documents, proteins, cities, or web pages. Edges represent relationships between them: follows, cites, interacts with, connects to or resembles.</p><p>A community is a group of vertices that are more strongly connected to one another than to the rest of the graph.</p><p>Imagine a social network:</p><ul><li>One group frequently discusses machine learning.</li><li>Another group talks mostly about football.</li><li>A third group shares research papers.</li><li>A few users connect these groups together.</li></ul><p>The goal of community detection is to discover these hidden structures without being told how many groups exist beforehand.</p><p>This is important because real networks rarely come with neat labels. We do not usually know the correct number of communities, their sizes or where their boundaries should be.</p><h3><strong>How do we measure whether a community structure is good?</strong></h3><p>One popular measure is modularity.</p><p>At a high level, modularity asks:</p><blockquote>Are there more edges inside the detected communities than we would expect if the same graph were randomly connected?</blockquote><p>A simplified intuition is:<em>[Q = \text{observed internal connectivity}\text{expected internal connectivity}]</em></p><p>A higher modularity score generally means that the communities contain more internal connections than expected by chance.</p><p>The Leiden algorithm tries to improve this score by moving vertices between communities.</p><p>But modularity is not the same thing as truth.</p><p>A partition can have high modularity and still be difficult to interpret. Modularity also has a resolution limit, meaning that small communities may be merged into larger ones depending on the structure of the graph and the chosen objective.</p><p>So modularity is useful ,but it should be treated as a fitness signal, not as an absolute definition of meaning.</p><h3><strong>Why Leiden instead of Louvain?</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Sd_3t69H5Bsz89JN7njvTQ.png" /><figcaption>The three stages of Leiden: movement, refinement, and aggregation.</figcaption></figure><p>Louvain is one of the most popular algorithms for community detection. It works in two broad phases:</p><ol><li>Move vertices between neighboring communities to improve modularity.</li><li>Collapse each community into a super vertex and repeat.</li></ol><p>This is fast and effective, but it has an important weakness.</p><p>Louvain can produce communities that are internally disconnected. In other words, two groups of vertices may be assigned to the same community even though there is no path connecting them within that community.</p><p>Leiden introduces an additional refinement phase:</p><ol><li>Local movement</li><li>Refinement</li><li>Aggregation</li></ol><p>During refinement, Leiden examines the communities produced by the local-moving phase and separates poorly connected subgroups.</p><p>This gives Leiden a useful property: it is designed to produce communities that are better connected internally and better optimized.</p><p>Think of Louvain as quickly drawing rough borders on a map.</p><p>Leiden takes a second pass to check whether every town inside a region is actually connected to the rest of that region.</p><h3><strong>So what makes a graph dynamic?</strong></h3><p>A dynamic graph is not a single graph. It is a sequence of graph snapshots:<em>[G⁰, G¹, G², \ldots, G^t]</em></p><p>At every time step, some edges may be inserted or deleted.For example:</p><p>Time t-1: A — B — C D — E — F</p><p>Time t: A — B — C — D D — E — F</p><p>The new edge between C and D may connect two previously separate communities.</p><p>A dynamic community-detection algorithm receives:</p><ul><li>The previous graph</li><li>The current graph or batch of updates</li><li>The previous community assignment</li><li>The inserted edges</li><li>The deleted edges</li></ul><p>Its task is to produce an updated community assignment without doing unnecessary work.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ad1NfQpX_VqMrfGIeoHJGw.png" /><figcaption>Dynamic community detection follows the ripple created by changed edges.</figcaption></figure><p>The simplest solution is to ignore the previous result and run static Leiden again.</p><p>That is correct, but often wasteful.</p><p>If one edge changes in a graph containing a billion edges, recomputing every community from the beginning feels a little like rereading an entire library because one sentence was edited.</p><h3><strong>Three ways to update Leiden communities</strong></h3><p>The paper extends three dynamic approaches to a multicore implementation of Leiden.</p><h3>1. Naive-dynamic Leiden</h3><p>The Naive-dynamic approach is the easiest to understand.</p><p>We take the community assignment from the previous snapshot and use it as the starting point for the new graph. However, every vertex is still allowed to participate in the update.</p><p>The algorithm remembers yesterday’s answer, but it does not aggressively restrict today’s search.</p><p>This approach is simple and generally reliable.</p><p>It is also a useful baseline because it answers an important question:</p><blockquote>How much time can we save simply by warm-starting Leiden with the previous partition?</blockquote><h3>2. Delta-screening Leiden</h3><p>Delta-screening tries to identify vertices whose community assignments are likely to change.</p><p>For example:</p><ul><li>If an edge is deleted inside a community, its endpoints may become affected.</li><li>If an edge is inserted between two different communities, the endpoints may need to be reconsidered.</li><li>Neighboring vertices may also be affected because their local connectivity has changed.</li></ul><p>Instead of processing the entire graph equally, delta-screening creates an approximate region of interest around the update.</p><p>The word “delta” refers to the change in the graph and its effect on modularity.</p><h3>3. Dynamic Frontier Leiden</h3><p>Dynamic Frontier begins with the vertices directly affected by the update.</p><p>Then, whenever a vertex changes its community, its neighbors may be added to the frontier. The affected region expands incrementally until no additional changes need to propagate.</p><p>This resembles dropping a stone into a pond.</p><p>The update begins at a few vertices, and the algorithm follows the ripples only as far as necessary.</p><p>The benefit is that small, local changes can remain local.</p><h3>What does dynamic Leiden actually change?</h3><p>The key insight is not that Leiden itself becomes a completely different algorithm.</p><p>The local-moving phase becomes update-aware.</p><p>The dynamic version can:</p><ul><li>Reuse previous community memberships.</li><li>Reuse weighted degree information.</li><li>Reuse community-level edge weights.</li><li>Mark affected vertices.</li><li>Expand the affected region when memberships change.</li><li>Continue using Leiden’s refinement and aggregation phases.</li></ul><p>The refinement phase remains important.</p><p>A tempting optimization would be to process only affected vertices everywhere. However, Leiden’s refinement phase is what helps prevent internally disconnected communities. Skipping it too aggressively can damage the quality of the final partition.</p><p>This is one of the most important lessons from the paper:</p><blockquote>Faster community detection is not useful if the communities become structurally worse.</blockquote><h3>Can we see the idea in Python?</h3><p>Absolutely!</p><p>The production implementation described in the paper is a multicore implementation using low-level data structures and OpenMP. Python’s high-level leidenalg package does not expose the paper’s complete affected-vertex machinery.</p><p>But we can still demonstrate the central idea:</p><ol><li>Detect communities in an initial graph.</li><li>Apply a batch of edge updates.</li><li>Run Leiden again from scratch.</li><li>Run Leiden again using the previous membership as initialization.</li><li>Inspect which vertices are likely to be affected.</li></ol><p>Install the required libraries:</p><pre>pip install igraph leidenalg</pre><p>Now let us create a small graph with three dense regions.</p><pre>import igraph as ig<br>import leidenalg as la<br><br><br>N = 12<br><br>BASE_EDGES = [<br>    # Community 1<br>    (0, 1), (1, 2), (2, 3), (3, 0),<br><br>    # Community 2<br>    (4, 5), (5, 6), (6, 7), (7, 4),<br><br>    # Community 3<br>    (8, 9), (9, 10), (10, 11), (11, 8),<br><br>    # Weak bridges<br>    (3, 4),<br>    (7, 8),<br>]<br><br>INSERTED_EDGES = [<br>    (1, 4),<br>    (2, 5),<br>    (6, 9),<br>]<br><br>DELETED_EDGES = [<br>    (3, 4),<br>]</pre><p>The helper functions below construct the graph and run Leiden.</p><pre>def build_graph(edges, number_of_vertices):<br>    graph = ig.Graph(n=number_of_vertices)<br>    graph.add_edges(list(edges))<br>    return graph<br><br><br>def run_leiden(<br>    edges,<br>    number_of_vertices,<br>    initial_membership=None,<br>):<br>    graph = build_graph(edges, number_of_vertices)<br><br>    partition = la.find_partition(<br>        graph,<br>        la.ModularityVertexPartition,<br>        initial_membership=initial_membership,<br>        n_iterations=-1,<br>        seed=42,<br>    )<br><br>    return graph, partition<br><br><br>def normalize_edges(edges):<br>    return {<br>        tuple(sorted(edge))<br>        for edge in edges<br>        if edge[0] != edge[1]<br>    }<br><br><br>def apply_batch(edges, inserted, deleted):<br>    updated_edges = normalize_edges(edges)<br><br>    for edge in deleted:<br>        updated_edges.discard(tuple(sorted(edge)))<br><br>    updated_edges.update(normalize_edges(inserted))<br><br>    return sorted(updated_edges)<br><br><br>def communities(partition):<br>    return [<br>        sorted(list(group))<br>        for group in partition<br>    ]</pre><p>First, detect communities in the original graph.</p><pre>graph_before, partition_before = run_leiden(<br>    BASE_EDGES,<br>    N,<br>)<br><br>previous_membership = partition_before.membership<br><br>print(&quot;Communities before the update:&quot;)<br>print(communities(partition_before))</pre><p>Now apply the batch update.</p><pre>UPDATED_EDGES = apply_batch(<br>    BASE_EDGES,<br>    inserted=INSERTED_EDGES,<br>    deleted=DELETED_EDGES,<br>)</pre><p>We can now compare a cold start with a warm start.</p><pre># Cold start: no knowledge of the previous partition<br>_, cold_partition = run_leiden(<br>    UPDATED_EDGES,<br>    N,<br>)<br><br># Warm start: reuse the previous membership<br>_, warm_partition = run_leiden(<br>    UPDATED_EDGES,<br>    N,<br>    initial_membership=previous_membership,<br>)<br><br>print(&quot;Communities after a cold start:&quot;)<br>print(communities(cold_partition))<br><br>print(&quot;Communities after a warm start:&quot;)<br>print(communities(warm_partition))                                                              </pre><p>The exact community labels may differ between runs because community IDs are arbitrary.</p><p>Community in one result does not necessarily correspond to community in another result. What matters is the set of vertices assigned together.</p><p>The warm start does not guarantee a speedup on every graph. Its benefit depends on how large and disruptive the update is.</p><p>If half the graph changes, there may be little advantage in trying to preserve the old partition.</p><h3>A small delta-screening approximation</h3><p>We can also build a simplified screening function.</p><p>This is not the full implementation from the paper. It is an educational approximation that marks update endpoints and their immediate neighbors.</p><pre>def approximate_affected_vertices( <br>graph, old_membership, inserted, deleted, )<br>: affected = set()<br><br># Internal deletions may break an existing community <br>for u, v in deleted: if old_membership[u] == old_membership[v]: affected.update([u, v]) # Cross-community insertions may join existing communities <br>for u, v in inserted: if old_membership[u] != old_membership[v]: affected.update([u, v]) # Include one-hop neighbors <br>for vertex in list(affected): affected.update(graph.neighbors(vertex)) return sorted(affected)<br><br>graph_after = build_graph(UPDATED_EDGES, N) affected = approximate_affected_vertices( graph_after, previous_membership, INSERTED_EDGES, DELETED_EDGES)<br>print(&quot;Approximate affected vertices:&quot;)<br>print(affected)<br></pre><p>n a production implementation, this affected set would influence which vertices are considered during the local-moving phase.</p><p>The paper’s implementation goes further by tracking weighted degrees, community weights, and changes in membership. It also allows the affected region to expand dynamically when a vertex moves.</p><h3>What did the paper find?</h3><p>The authors evaluated Static Leiden, Naive-dynamic Leiden, Delta-screening Leiden, and Dynamic Frontier Leiden.</p><p>For large graphs with randomly generated batch updates, the dynamic approaches achieved average speedups over Static Leiden of approximately:</p><ul><li>1.25× for Naive-dynamic Leiden</li><li>1.24× for Delta-screening Leiden</li><li>1.37× for Dynamic Frontier Leiden</li></ul><p>The communities produced had approximately the same modularity as Static Leiden.</p><p>On real-world dynamic graphs, the speedups were more modest:</p><ul><li>1.14× for Naive-dynamic Leiden</li><li>1.11× for Delta-screening Leiden</li><li>1.09× for Dynamic Frontier Leiden</li></ul><p>This difference is understandable.</p><p>Dynamic approaches mainly reduce the work performed during the first local-moving phase. Leiden still needs to perform refinement and aggregation to obtain high-quality, well-connected communities.</p><p>In the paper’s experiments, only a fraction of the total runtime belonged to the part that dynamic processing could reduce.</p><p>That leads to a practical engineering lesson:</p><blockquote>Optimizing one phase of an algorithm does not automatically produce an equally large end-to-end speedup.</blockquote><h3>Where does this leave us?</h3><p>Static community detection asks:</p><blockquote>What communities exist in this graph?</blockquote><p>Dynamic community detection asks a more realistic question:</p><blockquote>Given what we already knew, what changed — and how far did that change travel?</blockquote><p>Leiden remains attractive because it combines modularity optimization with a refinement stage that produces better-connected communities.</p><p>Dynamic Leiden adds temporal awareness:</p><ul><li>Previous partitions become useful initialization.</li><li>Edge updates become signals about where to look.</li><li>Local changes can propagate through a dynamic frontier.</li><li>Expensive work can be reduced without discarding the entire previous solution.</li></ul><p>The Python example here is intentionally small. It demonstrates warm-starting and affected-region reasoning, but it is not a replacement for the paper’s optimized parallel implementation.</p><p>For real systems, we would also need to think about:</p><ul><li>Weighted edges</li><li>New and deleted vertices</li><li>Community identity tracking across time</li><li>Modularity resolution limits</li><li>Large batch updates</li><li>Periodic full recomputation</li><li>Changes that spread across the entire graph</li></ul><p><strong><em>A graph is not a photograph…It is a living system.</em></strong></p><p>And if our algorithms are going to understand living systems, they must learn to update their beliefs instead of starting from zero every time something changes.</p><h3>References</h3><ul><li>Sahu, S. <a href="https://arxiv.org/html/2405.11658v1">A Starting Point for Dynamic Community Detection with Leiden Algorithm</a></li><li>Traag, V. A., Waltman, L., and van Eck, N. J. <a href="https://arxiv.org/abs/1810.08473">From Louvain to Leiden: Guaranteeing Well-Connected Communities</a></li><li><a href="https://leidenalg.readthedocs.io/en/stable/intro.html">Leidenalg Python Documentation</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2dce44df10c8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Discrete Steps to Continuous Flows: The Dynamic Future of Artificial Intelligence]]></title>
            <link>https://medium.com/@neevdeb26/from-discrete-steps-to-continuous-flows-the-dynamic-future-of-artificial-intelligence-e4a655f9a919?source=rss-e44d9391c5fe------2</link>
            <guid isPermaLink="false">https://medium.com/p/e4a655f9a919</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[google-deepmind]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Rhitam Deb]]></dc:creator>
            <pubDate>Sun, 14 Jun 2026 16:07:02 GMT</pubDate>
            <atom:updated>2026-06-26T13:35:24.801Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A Google DeepMind paper argues that the AI architecture powering every frontier model has a structural flaw — and Chain of Thought is just a workaround, not a fix.</em></p><p>Let me start with a confession: <a href="https://arxiv.org/html/2604.17121v1">this paper</a> has one of the most accurate titles in recent AI research. The trouble with Transformers <em>is</em> topological. It’s a shape problem, and once you see it, you can’t unsee it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_3gWkTpwjKavhlsdyHjLeQ.png" /></figure><p>A weekend evening, while exploring content online, I came across a paper by DeepMind. Intrigued, I started reading it. Within minutes, I was completely absorbed and ended up glued to my screen for hours. It doesn not announce a new model, neither does this break a benchmark. Instead, it does something rarer and arguably more valuable. It carefully diagnoses <em>why</em> the architecture that powers essentially every frontier AI model has a hard ceiling — and maps out the space of ways to climb past it.</p><h3>First ,what is “state tracking” and why does it matter?</h3><p>Here is the core idea, the world changes, conversations evolve, a story unfolds, a game progresses an agent exploring an environment picks up new information with every step. Any system that reasons about the world needs to maintain an internal representation of <em>what is currently true</em> — and update it as new evidence arrives.</p><p>The paper calls this a <strong>belief state</strong>, a compact running summary of what the AI knows about its environment at any given moment. Formally, the authors define it as</p><blockquote>s_t = f(s_{t-1}, x_t)</blockquote><p>The current state is a function of the previous state and the new input. Simple equation, profound implication.</p><p>Let me use a Cricket analogy.</p><p>Think of Jasprit Bumrah coming back for a second spell on Day 3 of a Test match. He doesn’t start from scratch, nor does he stop to rewatch the whole match tape to plan his next ball. Instead, he carries a running compressed mental model of the game on how the pitch has dried, the shifts in cloud cover, how the ball is reversing compared to first innings, and which batsman is fishing outside off. Every single delivery smoothly updates that internal state in real-time.</p><p>A standard Transformer, on contrast acts like a bowler who has to re-read the entire match scorecard from ball one before every single delivery just to remember his line. That brute-force approach works fine in a T20 sprint, but it completely breaks down in a grinding, five-day Test match.</p><p>Having said that, Transformers <em>can</em> do some forms of state tracking through clever tricks — lookbacks, associative scans, parallel parity computation. Researchers have trained models that discover surprisingly elegant workarounds. But these all have limits.</p><h3>Okay, so what is the actual “topological trouble”?</h3><p>This is the architectural insight at the heart of the paper, and it is genuinely illuminating once you see it drawn out.</p><p>A Transformer has two axes: layers (vertical, shallow to deep) and input steps (horizontal, left to right in the sequence). When the model processes a new token, its activation flows from shallow layers at the bottom, upward through all the layers, to the output at the top.</p><p>Now here’s the problem. When the model tries to update its belief state integrating the previous state s_{t-1} with a new input x_t — the new state s_t must sit <em>deeper in the layer stack</em> than the old state. Because the architecture is feedforward, there&#39;s no way for information to travel horizontally at the same layer depth. State must move up.</p><p>With every new input token, state gets pushed one level deeper. Eventually, after enough tokens, it runs out of room. The model has no more layers. This is what the authors call <strong>depth exhaustion</strong>. The state tracking capability of the model is fundamentally bounded by how many layers it has.</p><p><strong>Analogy: The High Press Gone Wrong</strong></p><p>Imagine a football team that can only ever move the ball upfield, never back. Every pass must go forward. After 10 passes, the ball is at the byline and there is nowhere left to go. That is a Transformer running state tracking through its layer stack. With a long enough sequence, you hit the edge of the pitch.</p><p>What makes this topological rather than just “a depth problem”? It is about the <em>structure</em> of the computation graph. A Transformer is, at its core, a directed acyclic graph since information flows in one direction. There are no cycles, no loops, no way for a deeper layer to send a signal back to a shallower one. The shape of the graph prevents arbitrary state dynamics. That’s the topology.</p><h3>Can you show me where this actually fails in practice?</h3><p>Absolutely , and the paper has a wonderful live example of this. Two of them, actually.</p><p>The first is a number guessing game. The model is asked to think of a number between 1 and 100 and respond “higher” or “lower” to guesses. Maintaining consistency requires tracking a valid range a classic belief state update. Here’s what Gemini 3 (Fast) did:</p><p>Even Gemini 3 Thinking ,which explicitly generates an internal target failed to maintain that state consistently across turns:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/715/1*y3Q8XwW8-rJiS5vOfwuu7g.png" /></figure><p>The second example is the famous “bank” problem and this one reveals something deeper about <em>how</em> the failure happens internally:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/715/1*KIG45yuMkNi2iFUXsYgg1g.png" /></figure><p>The model flipped its interpretation of “bank” from river to financial institution without any acknowledgment that it was reversing itself. A human would say “Oh wait, I think you mean a financial bank now?” The model just silently contradicts itself.</p><p>Researchers studied this using a technique called <em>Patchscopes</em> and found exactly the topological problem the paper describes: the model correctly resolved “bank” to mean “river bank” at a deep layer. But that deep-layer resolution was <em>inaccessible to the shallower layers</em> processing the next tokens. The disambiguation happened too late, too deep and got stranded there.</p><p><em>It’s not that the model doesn’t know the right answer. It’s that the right answer is trapped deep in the layer stack, unavailable to the parts of the network that generate the response. Information doesn’t flow back up.</em></p><h3>So if Transformers are so broken at this, why are they so good at everything else?</h3><p>Transformers have become extraordinary precisely because they found a clever alternative to state tracking: working memory via the context window.</p><p>Instead of maintaining a compressed belief state, a Transformer can always look back through its full input history and retrieve relevant information on demand. The attention mechanism is extremely good at this and it’s essentially a powerful and learnable lookup table over its entire context.</p><h3>What about Chain of Thought — doesn’t that solve the problem?</h3><p>Partially, temporarily. Not architecturally.</p><p>Chain of Thought (CoT) works by making the model <em>write its intermediate reasoning steps as output tokens</em>. Those tokens then re-enter the context window as inputs for the next generation step. In doing so, the model sends signals from deep layers (where the reasoning lives) back to shallow layers (where token generation starts) , effectively creating an external loop to compensate for the missing internal one.</p><p>Chain of Thought does genuinely enhance model expressivity , there is formal proof that it helps with inherently serial problems. But it comes with real costs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/705/1*VBR5AQzlTN6eCkuHPYAoPg.png" /></figure><h3>So what architectures can actually fix this? The paper builds a whole taxonomy ,walk me through it.</h3><p>The authors argue that “recurrent” is not a single thing there are meaningfully different flavors, and they behave very differently when it comes to state tracking.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CJCCXkzApbMn2fAB6Eu3QA.png" /></figure><p>They categorise architectures along two dimensions: the recurrence axis (does the recurrence happen over depth, over input steps, or both?) and the ratio of input tokens to recurrence steps (how many tokens are processed per loop?).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/705/1*iFzf0RUJ_LjYr-fsFDiiKg.png" /></figure><h3>What does the paper actually recommend? What are the “promising directions”?</h3><p>The paper lays out five directions, each honest about what it can and can’t do:</p><h4>1. Enhanced State-Space Models</h4><p>Specifically RWKV-7 and PaTH Attention, which integrate recurrent updates directly into the attention mechanism. These are genuinely more powerful for stateful tasks while remaining trainable in parallel,which is a meaningful technical achievement. The paper also points to gated linear attention as another frontier.</p><h4>2. Approximating state tracking in feedforward Transformers</h4><p>Rather than switching architecture, train feedforward models with specialised objectives that steer them toward better state-tracking behaviour. The “Belief State Transformer” paper and similar work fall here. This is the least disruptive path ,but the paper notes it has limits because you’re steering a fundamentally shaped vehicle, not redesigning it.</p><h4>3. Coarse recurrence</h4><p>Instead of tracking state token-by-token, compress sequences into chunks sentences, paragraphs, thoughts and apply recurrence at that coarser level. This dramatically reduces the computational burden. One cited approach actually models language as a “sequence of thoughts,” chunking at the sentence level. This feels intuitive: humans don’t update their world model with every syllable they hear.</p><h3>4. Leveraging representational alignment</h3><p>An interesting empirical observation: variable-depth models (looped transformers) often work surprisingly well even when you just fine-tune a pretrained model to loop, sometimes even with no training at all. The authors speculate that residual connections create natural alignment between layer representations, predisposing models to cross layer communication. There may be further ways to exploit this alignment without full recurrent pretraining.</p><h3>5. Efficient training of recurrence</h3><p>The hardest problem. Full recurrent pretraining is expensive because you can’t parallelise across sequence length. The paper suggests a pragmatic multi-stage approach: first pretrain on a standard feedforward Transformer (fast, parallelisable), then introduce recurrent mechanisms at a later stage using truncated gradient methods or recurrent backpropagation (an older technique from the 1980s, now being revived).</p><p>The authors are <strong>not saying Transformers are finished</strong>. They are saying the current architecture is hitting a <strong>ceiling</strong> on a specific, important class of problems, the kind that require a model to maintain a fluid, evolving internal model of the world. The workarounds (bigger context windows, Chain of Thought, thinking tokens) work, but they are compensations for a structural gap, not solutions to it.</p><p>The paper is real contribution is the taxonomy and the clarity of diagnosis. Before you can fix something, you need to understand exactly what shape the problem is. That’s what Mozer, Siddiqui, and Liu have done here and the answer is, fittingly, topological.</p><p><strong>Key Takeaways : The Full Scorecard</strong></p><ul><li>The “topological trouble” is real and precise: Transformers are directed acyclic graphs. State must move up through layers with each new token. Eventually the model runs out of depth this is “depth exhaustion.”</li><li>The “bank” example from the paper shows this concretely: correct disambiguation happens deep in the network, but is stranded there, inaccessible to shallower layers processing the next tokens.</li><li>Transformers compensate via context window lookback and Chain of Thought .Both genuine and powerful, both fundamentally compensations rather than architectural solutions.</li><li>Depth recurrence (looped transformers) increases expressivity but does NOT solve indefinite state tracking ,state still drifts upward.</li><li>The most promising directions: enhanced SSMs (especially RWKV-7), latent-thought models with implicit activation dynamics, and coarse recurrence at the sentence or thought level.</li><li>Full recurrence requires sequential training ,the hard engineering challenge no one has cleanly solved yet at frontier scale.</li><li>The next generation of models, the paper argues, must maintain a <em>fluid, evolving representation of reality</em> ,not just re-scan a longer and longer history.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e4a655f9a919" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Valley’s Best Kept Secret: How to Build Production-Grade RAG on a Shoestring Budget]]></title>
            <link>https://medium.com/@neevdeb26/valleys-best-kept-secret-how-to-build-production-grade-rag-on-a-shoestring-budget-3fb5e01e1278?source=rss-e44d9391c5fe------2</link>
            <guid isPermaLink="false">https://medium.com/p/3fb5e01e1278</guid>
            <category><![CDATA[vector-database]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Rhitam Deb]]></dc:creator>
            <pubDate>Sat, 04 Apr 2026 17:21:31 GMT</pubDate>
            <atom:updated>2026-04-05T04:51:17.115Z</atom:updated>
            <content:encoded><![CDATA[<p>Scaling is easy, but scaling <em>affordably</em> is an art form. Vector databases makes this harder.</p><p>Let us be real: Building a RAG pipeline feels like magic until the monthly cloud bill hits your inbox like a …</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/0*C2dLp9-m3_cCyyP_.gif" /><figcaption>snap..</figcaption></figure><p>..suddenly, half your budget is gone. 🫰</p><p>We have all been there, staring our Cloud and SaaS Costs and wondering if we are accidentally funding a small space program just to store some embeddings. Today, we are peeling back the curtain on the “Acronym Soup” of vector indexing to help you stop overpaying for RAM.</p><p>If you have been building anything from recommendation systems “You like cats? Here are 1,000 more cats!” to question-answering bots “Tell me a joke about a quantum physicist…”, you’ve likely encountered Vector Databases. Think of them as the personal minions for your model, running around a colossal warehouse of information to find the perfect match.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/260/0*UAssDjXQyO4pW1yh.gif" /></figure><p>In the core of every great vector DB is an algorithm. I intend to look at the big ones: the cool kids, the workhorses, and the future proof legends. Let us decode the tech speak and make sense of the soup of acronyms you see on comparison charts.</p><h3><strong>De-Code the Acronym Soup</strong></h3><p>Let us be honest, terms like IVF, HNSW, and DiskANN sound like a team of scientists. While they <em>are</em> pretty smart, their core ideas are surprisingly simple. Let us break down the main contenders.</p><h3>1. IVF (Inverted File Index)</h3><p><strong>Think of it as…</strong> Monika’s Kitchen<em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/220/0*pgot4iKB4kxyMNLI.gif" /></figure><p>Imagine Monika <em>(A chef with OCD)</em> from “<em>Friends” is </em>to prepare a 4 course meal for twenty guests. If she opens every single unmarked container in the pantry just to find the one ingredient well, disaster is the name of the dish she would serve. And her stress level would be through the roof. That stress is what our servers will go through.</p><p><strong>IVF</strong> or Inverted File Index is the ultimate organization hack, precisely mapping ingredients to the specific dishes they are used for.</p><p>Why is it called “inverted”? Because it inverts the natural “dish to intgredient” relationship to a “ingreadent to dish relationship”.In general concept it works like:</p><p><strong>Traditional Index (Forward Index)</strong>: You have a list of documents, and for each document, you list all the words it contains.</p><ul><li><em>Document 1:</em> Contains words A, B, C</li><li><em>Document 2:</em> Contains words B, D, E</li></ul><p><strong>Inverted Index:</strong> You flip that around. You have a list of words, and for each word, you list all the documents it appears in.</p><ul><li><em>Word A:</em> Appears in Document 1</li><li><em>Word B:</em> Appears in Document 1, Document 2</li><li><em>Word C:</em> Appears in Document 1</li><li><em>Word D:</em> Appears in Document 2</li><li><em>Word E:</em> Appears in Document 2</li></ul><h3>2. PQ (Product Quantization)</h3><p><strong>Think of it as…</strong> The Foldable Pocket Map</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7KijowC1eSxt4Bem.gif" /></figure><p>This is an add on technique, often used with IVF, hence <strong>IVF(PQ)</strong>.</p><p>No one would carry a 10 foot wide architectural blueprint of the city map in a backpack , it would not fit and you would look ridiculous trying to unfold it. Instead, you use a tiny, simplified version that hits the highlights (the landmarks, the main roads). PQ takes your high resolution, memory hungry vectors and squashes them into these “pocket sized” summaries. You lose some of the tiny details, but you can carry thousands of them in your pocket without breaking a sweat/budget.</p><p>When we search with IVF(PQ), we do it <em>very</em> quickly on the small, compressed representations. It gives us a good enough list of potential matches, which we then refine. It trades a tiny bit of precision for incredible speed and memory savings. It is like searching for a movie by its compressed trailer first.</p><h3>3. HNSW (Hierarchical Navigable Small World)</h3><p><strong>Think of it as…</strong> The Ultimate Neighborhood Guide</p><p>This is the current champion and the most widely used algorithm. The name is a mouthful, but the principle is beautiful.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JVODkIVVqZVvhxhp.jpg" /></figure><p>“Small World” refers to the idea that any two points are just a few connections away (like any someone in Vice City is just five people away from Colonel Juan Cortez). “Navigable” means there’s a structure to these connections, not just a random mess. “Hierarchical” is the magic ingredient: HNSW builds several layers of these connections.</p><p>Imagine a city.</p><ul><li><strong>The top layer</strong> has a few main highways (big, long-range connections).</li><li><strong>The middle layer</strong> has main arterial roads.</li><li><strong>The bottom layer</strong> has every small street and alleyway.</li></ul><p>When you search for a specific address, you start on the highway, zoom toward the general area, then jump to an arterial road to get closer, and finally drop to the bottom layer to find the exact door. This multi layered “zooming in” is extremely fast.</p><h3>4. RHNSW (Reduced Hierarchical Navigable Small World)</h3><p><strong>Think of it as…</strong> HNSW on a Diet</p><p>This is a variation of HNSW designed to use less memory, similar to what PQ does for IVF. It is still HNSW, but with some clever tricks to trim the size of the graph index without sacrificing too much search accuracy. It is like having a streamlined map of the city that is easier to handle.</p><h3>5. DiskANN (Disk-based Approximate Nearest Neighbor)</h3><p><strong>Think of it as…</strong> The Infinite Storage Solution</p><p>What if you have so much vector data it won nott fit in your expensive RAM? This is where DiskANN steps in with a giant hard drive. Personally speaking I am yet to come across a scenario where I encounter this. Its Costly, no saving us here. Happy that you have reached here !</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/0*eX4aK8S1fH1RRkMF.gif" /></figure><p>From what I have read, most of the other algorithms <em>must</em> have everything in RAM. DiskANN is designed to work with data stored on super-fast modern SSD disks. It keeps a small index in memory and does a smart “IO dance” to fetch only the necessary data blocks from the disk. This lets you index billions of vectors for a fraction of the cost. It’s like having a massive archive on another planet that you can still query in milliseconds.</p><h3>6. NGT (Neighborhood Graph and Tree)</h3><p><strong>Think of it as…</strong> A Smart Tree-Graph Hybrid</p><p>This algorithm, popular in some specialized circles (and notably used by the Vald vector DB), combines the best of both worlds. It uses a graph like connection structure (like HNSW) but organizes it using a tree like hierarchy to prune search paths early. It is designed to build highly optimized graphs that are very precise for search.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*4uR1_q03DAWE1wSv" /></figure><p>Its a lot to unpack, I know..I appreciate your <strong><em>attention</em></strong> and your support to Human Written content.. Please do <strong>Follow</strong> and drop <strong>Claps</strong> multiple clicks below to show support.</p><p>Now that we know the power ups, let us look at the actual databases and see which algorithm they choose to back their power.</p><p>This diagram is a perfect cheat sheet!</p><p><strong>As of 04/04/2026.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8MKgBIcoSAqigW0KvDxhiw.png" /></figure><p>What I do? I intent to not just pick the one that “everyone says is best.”</p><p>Choosing the right vector database isn’t about finding the ultimate champion. It’s about being an Engineer and understanding one’s specific task. Hope you had a pleasant read.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3fb5e01e1278" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Shrinking Giants: Hickhiker’s Guide to Make a 3-Billion Parameter LLM Run Anywhere]]></title>
            <link>https://medium.com/@neevdeb26/shrinking-giants-hickhikers-guide-to-make-a-3-billion-parameter-llm-run-anywhere-2346e3417e32?source=rss-e44d9391c5fe------2</link>
            <guid isPermaLink="false">https://medium.com/p/2346e3417e32</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[fine-tuning]]></category>
            <dc:creator><![CDATA[Rhitam Deb]]></dc:creator>
            <pubDate>Fri, 09 Jan 2026 17:47:51 GMT</pubDate>
            <atom:updated>2026-01-12T02:14:13.502Z</atom:updated>
            <content:encoded><![CDATA[<h3>Shrinking Giants: Hitchhiker’s Guide to Make a 3-Billion Parameter LLM Run Anywhere</h3><p>Hello there, fellow AI Enthusiast HapPy New Year! 👋</p><p><em>This is my first post for the year 2026, if you are new to my articles or have been read my old publications, thank you for investing time in reading a human written blog in days of “Plastic Contents”.</em></p><p>Last year, I started a project : <a href="https://medium.com/@neevdeb26/project-saarthi-a-chat-with-krishna-3a01d2810fb8">Saarthi</a> , where I fine tuned Qwen 2.5 on teachings of Bhagavat Geeta. This blog is a continuation to that.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JLByJIWw6cxXYIpZMQ0kkQ.png" /></figure><p>This journey forced me to make some clever engineering choices. Let us ask ourselves some hard questions about quantization, LoRA, and fine-tuning strategies to understand exactly how we pulled this off.</p><p>First things first: How do we even load a 3 billion parameter model on a consumer GPU?</p><p>What exactly is happening when we “quantize” a model?</p><p>Imagine an LLM as a library where every book is written in a large and precise font (32-bit or 16-bit numbers), that takes up huge shelf space (VRAM). Quantization is like rewriting those books in a tiny, efficient shorthand (4-bit or 2-bit).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Bx6Xi69j7a9gBYAXFUMeig.png" /></figure><p>We are compressing the model’s weights — the numbers that make up its “brain” — into smaller integers. This reduces memory usage to about <strong>25%</strong> of the original size</p><p>Below is the code snippet that performs the same.</p><pre>from unsloth import FastLanguageModel<br><br>max_seq_length = 2048<br>dtype = None # Auto detection<br>load_in_4bit = True # 4bit quantization to reduce memory usage<br><br>model, tokenizer = FastLanguageModel.from_pretrained(<br>    model_name = &quot;unsloth/Qwen2.5-3B-Instruct&quot;,<br>    max_seq_length = max_seq_length,<br>    dtype = dtype,<br>    load_in_4bit = load_in_4bit, # We load it quantized from the start!<br>)</pre><p>Now that we have our compressed model, we need to teach it the <em>Geeta, b</em>ut we can’t retrain the whole thing that would be too expensive.</p><p>Enter <strong>LoRA</strong>. Instead of rewriting the model’s entire encyclopedic memory, we just hand it a small notebook .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/356/0*3hLfK5JRGhLmR733.gif" /></figure><p>LoRA “freezes” the massive pre trained weights and injects tiny, trainable “adapter” matrices alongside them.We only train the adapters. This means we are updating less than <strong>1%</strong> of the parameters, which saves huge amounts of memory.</p><p>Here is the code where we attach these adapters:</p><pre>model = FastLanguageModel.get_peft_model(<br>    model,<br>    r = 16, # The &quot;rank&quot; of our adapters<br>    target_modules = [&quot;q_proj&quot;, &quot;k_proj&quot;, &quot;v_proj&quot;, &quot;o_proj&quot;,<br>                      &quot;gate_proj&quot;, &quot;up_proj&quot;, &quot;down_proj&quot;],<br>    lora_alpha = 16,<br>    lora_dropout = 0,<br>    bias = &quot;none&quot;,<br>    use_gradient_checkpointing = &quot;unsloth&quot;,<br>    random_state = 3407,<br>)</pre><p>You might have noticed that list of target_modules: q_proj, k_proj, gate_proj... What exactly are we targeting here?</p><p>These are the specific linear layers inside the Transformer architecture where the “thinking” happens.</p><ul><li><strong>The Attention Team (</strong><strong>q_proj, </strong><strong>k_proj, </strong><strong>v_proj, </strong><strong>o_proj):</strong> These layers decide <em>what</em> the model pays attention to. By fine-tuning them, we help the model learn new relationships between words—like understanding that &quot;stress&quot; in our context relates to &quot;Arjuna&#39;s dilemma&quot;.</li><li><strong>The Thinking Team (</strong><strong>gate_proj, </strong><strong>up_proj, </strong><strong>down_proj):</strong> These are part of the Feed-Forward Network (specifically SwiGLU in modern models). They process the information and apply logic. Fine-tuning here helps the model learn the <em>style</em> and <em>reasoning</em> of a counselor.</li></ul><p>By targeting <strong>all</strong> of them, we give LoRA maximum flexibility to reshape how the model processes information.</p><p>This is the most critical decision in our project. How do we keep the model smart while compressing it?</p><p><strong>Q: Should we train first and then compress (PTQ), or compress first and then train?</strong></p><p>There are a few schools of thought here:</p><ol><li><strong>Post-Training Quantization (PTQ):</strong> We train a big model and then shrink it. <em>Risk:</em> The model gets “shocked” by the sudden loss of precision and becomes dumber.</li><li><strong>Quantization-Aware Training (QAT):</strong> We simulate quantization during the entire massive training process. <em>Risk:</em> extremely expensive.</li><li><strong>The Winner: QAFT (Quantization-Aware Fine-Tuning) / QLoRA:</strong> This is what we used.</li></ol><p>I chose this because it is the “best of both worlds.”</p><ol><li><strong>Quantize First</strong>: I loaded the model in 4-bit immediately to save memory.</li><li><strong>Fine-Tune Later</strong>: I trained the LoRA adapters <em>on top of</em> this quantized model.</li></ol><p>This means our adapters learned to work <em>with</em> the low-precision weights. They learned to compensate for any errors caused by the compression. It’s like practicing for a race wearing the heavy shoes you’ll actually run in, rather than training in sneakers and switching to boots on race day!</p><p>Now, we actually run the training. We use the SFTTrainer (Supervised Fine-Tuning Trainer ). We also use a special optimizer called adamw_8bit.<strong>Why </strong><strong>adamw_8bit?</strong> Just like we quantized the model, we can quantize the optimizer&#39;s memory. This saves us about <strong>4x the memory</strong> compared to a standard optimizer, ensuring we don&#39;t run out of VRAM.</p><pre>from trl import SFTTrainer<br>from transformers import TrainingArguments<br><br>trainer = SFTTrainer(<br>    model = model,<br>    tokenizer = tokenizer,<br>    train_dataset = dataset,<br>    dataset_text_field = &quot;text&quot;,<br>    max_seq_length = max_seq_length,<br>    dataset_num_proc = 2,<br>    packing = False, # Can make training faster for short sequences<br>    args = TrainingArguments(<br>        per_device_train_batch_size = 2,<br>        gradient_accumulation_steps = 4,<br>        warmup_steps = 5,<br>        max_steps = 60, # Adjust based on your dataset size<br>        learning_rate = 2e-4,<br>        fp16 = not is_bfloat16_supported(),<br>        bf16 = is_bfloat16_supported(),<br>        logging_steps = 1,<br>        optim = &quot;adamw_8bit&quot;, # The frugal scribe optimizer!<br>        weight_decay = 0.01,<br>        lr_scheduler_type = &quot;linear&quot;,<br>        seed = 3407,<br>        output_dir = &quot;outputs&quot;,<br>    ),<br>)<br><br>trainer.train()</pre><p>Finally, we have a quantized model with trained adapters. But for deployment, we often want a single, clean file.</p><p><strong>How do we make this portable?</strong></p><pre># Merge the LoRA adapters with the base model and save to 16-bit<br>model.save_pretrained_merged(&quot;Krishna_Counselor_Model&quot;, tokenizer, save_method = &quot;merged_16bit&quot;)</pre><p>We merge the LoRA adapters back into the base model and save it in 16-bit floating point (float16). This creates &quot;Saarthi&quot;—a unified impression where the general knowledge and our specific training are fused together.</p><p>And there you have it! We started with a giant, compressed it down, taught it a specific philosophy using lightweight adapters, and merged it all back into a ready for the real world.</p><p>What next in the project?</p><p>Well..I am planning to put it on Your Mobile.</p><p>Stay Tuned and Stay Hungry…</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2346e3417e32" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[LLMs With a Past: Inside the Generative Semantic Workspace Revolution]]></title>
            <link>https://medium.com/@neevdeb26/llms-with-a-past-inside-the-generative-semantic-workspace-revolution-1336f67ddef9?source=rss-e44d9391c5fe------2</link>
            <guid isPermaLink="false">https://medium.com/p/1336f67ddef9</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[research]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Rhitam Deb]]></dc:creator>
            <pubDate>Tue, 16 Dec 2025 00:23:28 GMT</pubDate>
            <atom:updated>2025-12-16T00:41:43.626Z</atom:updated>
            <content:encoded><![CDATA[<p>Hello, fellow AI enthusiast! As the use case of Large Language Models become more complex and our models tries to make sense of the “Cosmic Chaos”, longterm memory and world building are perhaps the biggest problem statements in making AI agents truly capable and able to cater to current expectations. Beyond simple chatbots to systems that need to reason across vast, evolving narratives and that is precisely what the <a href="https://arxiv.org/pdf/2511.07587"><strong>Generative Semantic Workspace (GSW)</strong></a> framework delivers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*0ncrkMlU83byTXel4bYI9w.gif" /><figcaption>Your model trying to make sense of the World around</figcaption></figure><h3><strong>Why is GSW such a game-changer?</strong></h3><p>It is important because existing solutions, even sophisticated RAGs are hitting a wall when dealing with complex, real world texts. Most LLM inputs are not lists of static facts like Wikipedia entries. They are narratives: crime reports, corporate filings, or multi-day news coverage where actors change roles, transition through states, and interact across space and time.</p><p>If we think of a traditional RAG system as a librarian who can only find books by keyword (semantic similarity), the GSW system is more like an expert analyst who reads the entire archive and builds a structured, evolving character profile for every entity. This “neuro inspired” approach, which consciously models the interplay between the neocortex (for semantic abstraction) and the hippocampus (for spatiotemporal binding), allows the LLM to track an entity’s changing situation rather than just pulling disconnected facts. Explained in the daigram below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vfJDEQ6twhAn_jPUS90ZZQ.png" /><figcaption><strong>Unifying Brain-Inspired and Generative Semantics for Episodic Memory Modeling </strong>The hippocampal complex. ((DG, CA3, CA1) and neocortical regions (NC) inspire the <em>Reconciler </em>(retrieval, workspace, update) and <em>Operator </em>(LLM-driven semantic role extraction), respectively. The neocortical complex, responsible for context-rich consolidation and predictive modeling, aligns with the Operator module’s functions. The hippocampal complex, which performs indexing, pattern separation,and sequence modeling, corresponds to the Reconciler. Together, the GSW framework offers a biologically inspired, interpretable model for simulating world knowledge from text inputs. Source: <a href="https://arxiv.org/pdf/2511.07587">https://arxiv.org/pdf/2511.07587</a></figcaption></figure><h3>How have they achieved this human-like episodic memory?</h3><p>The secret lies in moving away from indexing raw text chunks and toward building an internal, structured “world model” that enforces chronological and spatial coherence. GSW accomplishes this through two tightly coupled, brain-inspired components: the <strong>Operator</strong> and the <strong>Reconciler</strong>.</p><h3><strong>How does GSW turn sprawling, unstructured narrative into structured, machine-readable memories?</strong></h3><p>This is the domain of the <strong>Operator</strong>. The Operator performs a deep semantic analysis on small, local chunks of text, transforming unstructured language input into an intermediate, structured snapshot : a local semantic graph or index card.</p><p>You can think of the Operator as the LLM’s personal script supervisor on the set of a long running Cricket Match Commentary. It captures the critical semantics of every session(or text chunk), including all <strong>players</strong>(people, places, organizations), their contributions (like over bowled, economy ), their momentary <strong>States</strong> (like current over ), the <strong>Verbs</strong> describing their actions, and critical <strong>Spatiotemporal Continuity</strong> constraints.</p><h3><strong>But how does GSW stop these localized snapshots from becoming a fragmented mess over a million tokens?</strong></h3><p>Let us take example of Cricket World Cup Final, 2011. Indian captain MS-Dhoni enters the filed a a tensed situation.</p><p>Ravi Shastri at the commentators’ box introduces : <strong>“</strong><em>Now we have the Captain Cool down on the crease.. in this tense situation</em><strong>”.</strong></p><p>..</p><p>..</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/0*P8scDZSNUc4YG7G5.jpeg" /><figcaption>The Winning Runs</figcaption></figure><p>Dhoni hits the winning runs: <em>“Dhoni finishes off in style. A magnificent strike into the crowd! India lift the World Cup after 28 years!”</em></p><p>In the above example, the Operator takes in sementic information and connections like,</p><p>Dhoni → Batsman</p><p>Dhoni → Team India Captain</p><p>Captain → Captain Cool</p><p>Indian Team → Currently batting</p><p>Situation → tensed</p><p>Dhoni → Hits a “six” and India wins the World Cup</p><p>Word Cup Duration → 28 Years</p><p>The <strong>Reconciler</strong> takes the outputs from the Operator and iteratively integrates them into a single, persistent, global memory, ensuring logical and chronological coherence.</p><p>The Reconciler’s function is essential for continuous reasoning. It performs <strong>entity resolution</strong>, connecting a pronoun (“Captain Cool” in chunk 2) back to the named entity (“Dhoni” in chunk 1). Crucially, it tracks <strong>state progression</strong> over time, changing an entity’s status from “nervous” to “confident” and placing this transition on a specific timeline. This recursive updating process models the likelihood of generating the next workspace instance based on the previous state and the current context.</p><h3>Key Findings and Performance Compared to Other Architectures</h3><p>The GSW framework was rigorously evaluated on the Episodic Memory Benchmark (EpBench) and demonstrated state-of-the-art results across several metrics, particularly in handling long narratives and complex episodic queries.</p><h4>1. Superior Accuracy and Recall in Episodic Reasoning</h4><p>GSW achieved the highest overall performance on the EpBench-200 corpus (a 100k-token dataset), demonstrating superior accuracy in episodic memory recall and reasoning.</p><p><strong>· Overall Metrics:</strong> GSW achieved the highest overall F1-score (<strong>0.850</strong>), Precision (<strong>0.865</strong>), and Recall (<strong>0.894</strong>) on the EpBench-200 dataset, improving these metrics by more than 10% over the next-best method.</p><p><strong>· Complex Queries:</strong> The advantage was particularly pronounced in the most demanding queries, such as those requiring synthesis across as many as 17 distinct documents (the ‘6+ Cues’ category). In this challenging scenario, GSW achieved an F1-score of 0.834 (P: 0.891, R: 0.822) and improved recall by up to <strong>20%</strong> compared to the next most performant baseline, HippoRAG2. Recall is a critical measure of a framework’s ability to correctly map queries to the necessary context, and GSW maintained consistently strong recall even as query complexity increased, unlike competing frameworks.</p><p><strong>· Scalability:</strong> GSW maintained its performance lead on the 10x larger EpBench-2000 corpus (approximately 1 million tokens), achieving an overall F1-score of <strong>0.773</strong>, outperforming the strongest baseline (Embedding RAG) by more than 15% in F1-score and 14% in recall.</p><h4>2. Comparison with Existing RAG Architectures</h4><p>The GSW framework distinguishes itself from both standard and structured RAG methods by its specific focus on tracking dynamic, spatiotemporal narratives rather than just static facts.</p><p><strong>Standard Embedding-based RAG</strong></p><p>Fragments the narrative into independent chunks based on semantic similarity, making it difficult to connect dispersed information and track evolving storylines.</p><p>GSW builds a persistent, structured memory of evolving situations that enforces temporal and spatial coherence, successfully tracking state changes.</p><p><strong>Structured RAG (e.g., GraphRAG, HippoRAG2, LightRAG)</strong></p><p>Primarily designed for <strong>fact based retrieval</strong> and multi-hop reasoning, not for tracking <em>sequential, evolving</em> states and roles over time and space (episodic memory).</p><p>GSW explicitly models the <strong>chronological progression</strong> and the dynamic changes in entity roles and states, much like human episodic memory.</p><p><strong>Qualitative Example:</strong></p><p><strong>HippoRAG2</strong> often retrieves the full text of relevant chapters (up to 17 documents), which can overwhelm the LLM and lead to hallucinations. <strong>GraphRAG</strong> struggles with information loss in summarization and often includes excessive, irrelevant context.</p><p>GSW provides highly <strong>targeted, entity-specific summaries</strong> generated directly from its structured memory, minimizing irrelevant context and significantly reducing the rate of LLM hallucination.</p><h4>3. Token Efficiency and Cost Reduction</h4><p>A significant finding is GSW’s token efficiency, which translates directly into lower inference costs.</p><p>· GSW achieved a <strong>51% reduction in token usage</strong> compared to the next most token-efficient baseline (GraphRAG) and nearly a 59% reduction compared to Embedding RAG and HippoRAG2.</p><p>· This efficiency is realized because GSW generates concise, <strong>entity-specific summaries</strong> from its workspace structure at query time, instead of relying on passing large, raw document chunks or entire chapters to the LLM. This focused contextual information also helps to reduce hallucinations.</p><h3>Shortcomings in the Approach (Limitations)</h3><p>The researchers acknowledged several limitations in the current iteration of the GSW framework and its evaluation:</p><p><strong>1. Limited Scope of Current Benchmarks:</strong> While EpBench is valuable for assessing spatiotemporal reasoning, the current scope of episodic memory benchmarks is limited in thoroughly probing the complex evolution of actor roles and states within extended narratives. The researchers are actively developing a more comprehensive benchmark to specifically address this gap.</p><p><strong>2. Reliance on a Closed-Source LLM:</strong> The present GSW implementation relies on a powerful, closed-source LLM (GPT-4o) for its core Operator and Reconciler functions (such as semantic extraction and reconciliation). Empirical validation is needed to confirm if promising open-source LLMs could be effectively integrated into the Operator-Reconciler architecture.</p><p><strong>3. Monetary Cost of Indexing:</strong> The indexing process for GSW, while a one-time cost, involves significant interaction with the LLM API. Indexing the EpBench-200 dataset cost approximately <strong>$15</strong> using GPT-4o, though this is amortized over many queries.</p><p><strong>4. Modalities Constraint:</strong> Currently, GSW is designed to process text inputs. Expanding GSW to integrate information from diverse data modalities (e.g., video or dialogue) beyond text is an important direction for future development to broaden its real-world applicability.</p><p>In essence, <strong><em>GSW acts like a personalized, structured journal for an LLM, instead of searching through pages of loose notes (RAG chunks). T</em></strong>he LLM queries a meticulously organized timeline of events, guaranteeing chronological and spatial context, which is why it performs so accurately and efficiently on complex narratives.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1336f67ddef9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Planning Your Way to Better Answers: Introducing Plan-on-Graph (PoG) for LLM Reasoning]]></title>
            <link>https://medium.com/@neevdeb26/planning-your-way-to-better-answers-introducing-plan-on-graph-pog-for-llm-reasoning-e4105d27f0b7?source=rss-e44d9391c5fe------2</link>
            <guid isPermaLink="false">https://medium.com/p/e4105d27f0b7</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[research]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[knowledge-graph]]></category>
            <category><![CDATA[reasoning]]></category>
            <dc:creator><![CDATA[Rhitam Deb]]></dc:creator>
            <pubDate>Tue, 23 Sep 2025 15:31:16 GMT</pubDate>
            <atom:updated>2025-09-24T06:45:20.569Z</atom:updated>
            <content:encoded><![CDATA[<p>Hello there, fellow AI Enthusiast! While strolling across my LinkedIn feed, I came across a new methodology that might change how Large Language Models (LLMs) interact with Knowledge Graphs (KGs). We are exploring <a href="https://arxiv.org/pdf/2410.23875"><strong>Plan-on-Graph (PoG)</strong></a>, a self-correcting adaptive planning paradigm published recently by colleagues from the University of Science and Technology of China and The Hong Kong University of Science and Technology.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eOFZF18t0dyIMsgw" /></figure><p>PoG is fascinating because it allows an LLM not just to search a graph, but to dynamically plan and correct its reasoning journey, much like an expert solving a complicated puzzle. That solves my problem of how many “hops” do I need to take while performing a G-RAG and might be a wise step towards “<a href="https://www.ibm.com/think/topics/explainable-ai">Explainable AI</a> ” in LLMs.</p><p>Let’s break down what PoG is, why it was needed, and how this “intelligent” approach works.</p><h3>The Basics: Why do LLMs Needed Plan?</h3><p>We all love LLMs for their incredible fluency and ability to handle various tasks, from question answering to text generation. But despite their success, they have some persistent limitations: they can suffer from <strong>out-of-date knowledge</strong>, <strong>hallucinations</strong>, and <strong>opaque decision-making</strong>.</p><p>Enter Knowledge Graphs (KGs). KGs are structural knowledge bases that provide explicit and editable depictions of massive real-world facts. They are the perfect complement to shore up the weaknesses of LLMs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b2JetwdDA-apoYf2IJc-4A.png" /><figcaption>The illustration of the comparison between traditional RAG and GraphRAG. Source : <a href="https://arxiv.org/pdf/2501.13958">https://arxiv.org/pdf/2501.13958</a></figcaption></figure><p>Currently, KG-augmented LLM paradigms recognized this value, treating the LLM as an agent to explore entities and relations on the KG.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o6rlM5EKGOd-ZqHfa6Cy8g.png" /><figcaption>The development trends in the field of GraphRAG with representative works. Source : <a href="https://arxiv.org/pdf/2501.13958">https://arxiv.org/pdf/2501.13958</a></figcaption></figure><p>However, these approaches hit a bottleneck:</p><ol><li><strong>Fixed Exploration:</strong> They typically require manually defining the breadth of the exploration space, which limits efficiency and effectiveness.</li><li><strong>No Backtracking:</strong> The path exploration was often <strong>unidirectional</strong>. If the LLM started down the wrong path, it kept going, leading to flawed reasoning and failure.</li><li><strong>Forgetting Conditions:</strong> During complex, multi-step reasoning, the LLM could sometimes forget crucial conditions specified in the original question.</li></ol><p>The core problem PoG solves, therefore, is providing an LLM with the ability to <strong>adaptively explore reasoning paths based on question semantics</strong> and <strong>self-correct erroneous paths</strong>. PoG introduces a novel paradigm incorporating self-correction and adaptive planning to address these limitations.</p><h3>What Makes PoG Different from Other Approaches?</h3><p>The key distinction of PoG lies in its introduction of <strong>self-correcting adaptive planning</strong>. To the best of the authors’ knowledge, PoG is the first to integrate a <strong>reflection mechanism</strong> for self-correction and adaptive KG exploration into KG-augmented LLMs.</p><p>If we compare PoG to similar methodologies, like <a href="https://arxiv.org/abs/2410.18415"><em>Decoding on Graph</em></a> or <a href="https://arxiv.org/abs/2307.07697"><em>Think-on-Graph</em></a><em> </em>, the differences become clear:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/627/1*UTtwJb0UM6EkHo4KKLE0Ig.png" /><figcaption>A typical case to compare different methods to answer the complex question. For the convenience of display, we only provide the sub-objective status and partial reasoning paths stored in memory. Topic entities, wrong answers, and correct answers are highlighted in blue, red, and green. The revised path is highlighted with a yellow background. Source: <a href="https://arxiv.org/pdf/2410.23875">https://arxiv.org/pdf/2410.23875</a></figcaption></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/61bafda97bcc46c8515c4d8b40a110b2/href">https://medium.com/media/61bafda97bcc46c8515c4d8b40a110b2/href</a></iframe><p>In essence, previous methods often risked wasting resources exploring unproductive paths due to fixed breadth searches. PoG, however, acts as a smarter agent. If a path looks unproductive or insufficient, the LLM can decide to <strong>go back a step or two</strong> and explore alternative pathways, a capability crucial for complex, multi-hop questions.</p><h3>A Step-by-Step Look at Plan-on-Graph</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/0*9THf5E0yQvJeJ22X.gif" /></figure><p>PoG achieves its adaptive planning through an iterative process consisting of four main components: <strong>Task Decomposition, Path Exploration, Memory Updating, and Evaluation/Reflection</strong>. These steps are repeated until all sub-objectives are fulfilled or an answer is found.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/528/1*Lz4RUdctHtmjwK8l5Uq0oQ.png" /><figcaption>The framework overview of PoG, which includes four key components: Task Decomposition, Path Exploration, Memory Updating, and Evaluation. Source : <a href="https://arxiv.org/pdf/2410.23875">https://arxiv.org/pdf/2410.23875</a></figcaption></figure><p>Here is how the three key mechanisms — Guidance, Memory, and Reflection — work together:</p><h4>Step 1: Guidance through Task Decomposition</h4><p>The process begins by tackling the complexity of the query.</p><ul><li>The user question is broken down by the LLM into several <strong>sub-objectives</strong> or sub-queries. These sub-objectives specify the conditions that must be met to answer the question.</li><li>The sub-objectives serve as the <strong>Guidance</strong> mechanism. They focus the LLM’s attention only on relevant parts of the graph, constructing necessary subgraphs.</li></ul><p><strong>Example:</strong> If the question is, <em>“What element essential for human life is a key component of hemoglobin and is it also found in the Earth’s core?”</em>. The LLM decomposes this into steps like: 1. Find elements essential for human life. 2. Find which of those are key components of hemoglobin. 3. Check if that element is found in the Earth’s core.</p><h4>Step 2: Path Exploration</h4><p>Guided by the sub-objectives, the LLM iteratively accesses relevant information from the KG.</p><ul><li>Exploration starts with the <strong>topic entities</strong> mentioned in the question.</li><li>PoG uses an <strong>adaptive exploration strategy</strong> that doesn’t rely on a fixed exploration breadth. This strategy is split into two parts:</li></ul><ol><li><strong>Relation Exploration:</strong> The LLM retrieves candidate relations linked to the current entities and selects a flexible number of relations that are most relevant to the question and sub-objectives.</li><li><strong>Entity Exploration:</strong> Based on the selected relations, the LLM retrieves neighboring entities and chooses a flexible number of relevant paths ending with those entities.</li></ol><h4>Step 3: Memory Updating</h4><p>To support intelligent reflection, the LLM uses <strong>Memory</strong>. After each exploration step, the memory is dynamically updated, providing historical retrieval and reasoning context. The memory stores three types of information:</p><ol><li><strong>Complete Subgraph:</strong> The part of the KG explored so far, which can be used later to initialize new exploration or decide where to backtrack.</li><li><strong>Reasoning Paths (P):</strong> The sequences of entities and relations traversed. This preserves semantic structure and allows for path correction.</li><li><strong>Sub-Objective Status (S):</strong> Tracks which sub-objectives have been fulfilled and what is still needed. This is key to preventing the LLM from forgetting complex question parts.</li></ol><h4>Step 4: Evaluation and Reflection</h4><p>This is the <strong>core of the self-correction process</strong>.</p><ul><li>The LLM <strong>evaluates</strong> the information recorded in memory (S and P).</li><li><strong>If sufficient knowledge is acquired:</strong> The LLM integrates the reasoning paths, sub-objective status, and its own knowledge to generate the final answer.</li><li><strong>If insufficient knowledge is acquired:</strong> The LLM initiates the <strong>Reflection</strong> mechanism. It reflects on critical questions:</li><li>Is the current path likely to lead to the correct answer?</li><li>Should the current path be continued, or should a different path be explored (i.e., should I backtrack)?</li><li>If the LLM determines that additional exploration or correction is needed (self-correction), it decides which previous entity to <strong>backtrack to</strong> and adds new exploration steps from that point, ensuring that wrong, unproductive paths are avoided.</li></ul><p>This iterative loop of exploration, memory updating, and reflection repeats until all sub-objectives are met and the final answer is generated.</p><h3>Findings, Advantages, and Limitations</h3><p>PoG was tested against state-of-the-art baselines on three complex multi-hop KGQA datasets (CWQ, WebQSP, and GrailQA). The results confirmed both its effectiveness and its efficiency.</p><h4>Key Findings and Advantages</h4><ul><li><strong>Superior Performance:</strong> PoG consistently achieved the best performance across all three datasets, substantially outperforming the SOTA baseline, ToG. When using GPT-4, PoG exceeded all fine-tuned KG-augmented LLM baselines.</li><li><strong>Crucial for Complexity:</strong> The sub-objective decomposition makes PoG better suited for handling complex, multi-hop questions. The self-correction mechanism proved essential, particularly on the zero-shot subset of GrailQA, where PoG significantly outperformed others. In the CWQ dataset, 24% of cases involved reversing the path to achieve self-correction, showing how necessary this mechanism is.</li><li><strong>Remarkable Efficiency:</strong> PoG demonstrated superior time efficiency compared to ToG.</li><li>It requires <strong>at least 40.8% fewer LLM calls</strong>.</li><li>It significantly reduces token consumption (e.g., roughly <strong>76.2% decrease in output tokens</strong> on CWQ compared to ToG).</li><li>It showed a speedup of <strong>over 4 times</strong> on CWQ and GrailQA compared to ToG. This efficiency stems from adaptive breadth reducing irrelevant explorations and self-correction avoiding the extension of wrong paths.</li><li><strong>Transparency:</strong> The memory component explicitly records the reasoning steps, providing transparency to the process.</li></ul><h4>Limitations and Disadvantages</h4><p>While PoG is a significant advancement, it still faces several limitations based on the complexity of its implementation and reliance on LLM capabilities:</p><ol><li><strong>Heavy Reliance on LLM Power:</strong> The reflection and self-correction steps require a <strong>really powerful, intelligent LLM</strong> that is well pre-trained and fine-tuned to handle these inherent decisions. If the LLM starts to “hallucinate” during the complex reflection process, the results can be inaccurate.</li><li><strong>LLM Confidence Issues:</strong> LLMs are not always certain about when to stop, how many steps are needed, or if the current information is sufficient. The inherent <strong>“low self-confidence”</strong> of the LLM in evaluating its own retrieval needs further work.</li><li><strong>Computational Cost:</strong> Although PoG is more efficient than unidirectional fixed-breadth search methods, answering complex questions still requires multiple steps. Simpler approaches (like Decode on Graph) might still be preferable if computational resources are restricted.</li><li><strong>Query Standardization:</strong> For less standardized queries, the LLM’s semantic understanding might be insufficient, leading to decreased effectiveness.</li><li><strong>Replication Challenge:</strong> Unfortunately, at the time of publication, the authors noted that the code and data were not yet publicly available pending approval, making it difficult for the broader community to immediately replicate and evaluate the impressive findings.</li></ol><p>In conclusion, Plan-on-Graph represents a huge step forward in LLM reasoning over Knowledge Graphs, giving the model crucial meta-cognitive tools — Guidance, Memory, and Reflection — to plan its search, learn from its mistakes, and ultimately deliver more accurate and efficient answers to complex questions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e4105d27f0b7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Project Saarthi : A chat with Krishna]]></title>
            <link>https://medium.com/@neevdeb26/project-saarthi-a-chat-with-krishna-3a01d2810fb8?source=rss-e44d9391c5fe------2</link>
            <guid isPermaLink="false">https://medium.com/p/3a01d2810fb8</guid>
            <category><![CDATA[fine-tuning]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[india]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[spirituality]]></category>
            <dc:creator><![CDATA[Rhitam Deb]]></dc:creator>
            <pubDate>Mon, 18 Aug 2025 12:53:16 GMT</pubDate>
            <atom:updated>2025-10-03T03:20:28.600Z</atom:updated>
            <content:encoded><![CDATA[<h3>Project Saarthi : A chat with Krishna</h3><p><em>In a departure from my usual technical writings, this article shares a heartfelt personal experience that led to the creation of an AI project inspired by the teachings of Lord Krishna.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y0lA8Z4VKwYb0YJ8XNIacw.png" /></figure><p>Life often has a way of catching us by surprise. Just the other day, I was on my way to work, lost in thought like anyone in the daily grind. Suddenly, my phone buzzed to remind me that Janmashtami was only three days away. At that moment, a wave of nostalgia washed over me. Krishna was about to be born again. It struck me deeply and made me reflect on the profound teachings of the Bhagavad Gita.</p><p>The Bhagavad Gita is more than just a religious scripture; it is a manual for life, full of wisdom that resonates even today. You might say it offers us a dialogue on the human experience itself. I have always found solace in its pages and this was a perfect moment to revisit its essence.</p><p>Looking back, I can clearly see how my understanding of the divine transformed. I no longer viewed Krishna as an aloof deity watching over me from the heavens. Instead, I began to see Him as a friend, a guide, and a constant source of encouragement. At a time when life felt particularly heavy, I found hope in the idea of “<a href="https://huggingface.co/debadtman26/krishna-saarthi-counselor"><strong><em>Saarthi</em></strong></a>.” In this spiritual context, Saarthi means “charioteer,” and for me, it was about having a confidant guiding me through turbulent times, steering my life back toward the light.</p><p>Fast forward to the present day, and I find myself in the familiar comfort of my workplace, ready to embark on my tasks. My clients were looking for QnA bots, but as I sat there typing away, a thought popped into my mind. The Bhagavad Gita is essentially structured as a QnA dialogue, a conversation between Arjuna and Krishna that addresses the profound questions of existence. What if I could take that golden wisdom and translate it into something modern? What if I could give life to Saarthi through an AI that resonates with the spiritual queries of everyday people seeking guidance?</p><p>That thought ignited a spark within me. I realised I wanted to channel my skills into something meaningful, a project that could potentially illuminate paths for others as the Gita had once illuminated mine. And just like that, <em>Project Saarthi was born</em>.</p><p>In simple terms, Project Saarthi is an AI model designed to tackle psychological, emotional, spiritual, and personal conflicts by providing guidance from the teachings of the Bhagavad Gita. It’s my way of ensuring that when someone is grappling with life’s challenges, they can draw on the wisdom of these timeless teachings. I want this tool to be there for individuals until they can seek professional help.</p><p><strong><em>It is essential to emphasise that it should never be a substitute for institutional therapy.</em></strong></p><p>With my vision clear, I dove headfirst into the technical aspects of the project. The process of fine-tuning the model was an exciting challenge.</p><p>Bellow is the hardware specs I used:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bc9vJhUskVfZqXK4s19otg.png" /></figure><p>I have used Qwen2.5–7B-Instruct. Relevant code snippets are below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/998/1*5vtgPkf0J5GbOgveFLQGyw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/998/1*QAIir0fvf0_3kE8N8vtuEw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hV6l5o1eUZFs-b6wRHWxAQ.png" /></figure><p>Voalaa! This is how my response looks :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*coUXNhi754TXiwCHxdgVDQ.png" /></figure><p>Well, seems like the model is getting the tonality and the verses correct.</p><p>As I wrap up this reflection, I want to put it simply this is just the beginning. <a href="https://huggingface.co/debadtman26/krishna-saarthi-counselor-merged"><strong><em>Project Saarthi</em></strong></a> represents not only a personal journey but also a larger vision of creating impactful tools that can support many just as the <strong>Bhagavat</strong> <strong>Gita</strong> supports all.</p><p><em>If you have made it this far, I would love to hear your thoughts. Please share your ideas in the comments. Together, we can collaborate on something that lets the light of wisdom shine in the lives of those who need it.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3a01d2810fb8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hyper Tree Planning: Teaching LLMs to Master Multi-Step Tasks]]></title>
            <link>https://medium.com/@neevdeb26/hyper-tree-planning-teaching-llms-to-master-multi-step-tasks-babd30560b00?source=rss-e44d9391c5fe------2</link>
            <guid isPermaLink="false">https://medium.com/p/babd30560b00</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[knowledge-graph]]></category>
            <category><![CDATA[graphrag]]></category>
            <category><![CDATA[large-language-models]]></category>
            <dc:creator><![CDATA[Rhitam Deb]]></dc:creator>
            <pubDate>Mon, 21 Jul 2025 13:42:41 GMT</pubDate>
            <atom:updated>2025-07-21T13:42:41.036Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p6SccOnzg3DaTRxnGdk_lg.png" /></figure><p>Hello there, fellow AI Enthusiast! Last weekend while working on my Agents, I went into rabbit-hole to improve “Planning in LLMs” and came across this <a href="https://arxiv.org/pdf/2505.02322">paper</a>, which gave novel approach to solve the problem at hand, hence wanted to share in the blog. Let’s dive into a stunning research paper on Hyper-Tree Planning where we teach our LLMs to swap their linear thinking caps for a let’s say.. a layered approach. Our dear Large Language Models (LLMs) like GPT, Claude, and Mistral are absolute superstars when it comes to short, snappy tasks: whipping up a sentence, translating a phrase, or answering a quick question. But ask them to plan a week-long trip or solve a 10-step puzzle, and things can get a bit… messy. They often <strong>lose context, skip crucial steps, or even contradict themselves entirely</strong> — that’s the real challenge of <strong>Long Horizon Reasoning</strong>!.</p><p><strong>The Problem They are Solving</strong></p><p>Traditionally, LLMs struggle with complex planning tasks because they involve:</p><ul><li><strong>Long reasoning steps</strong>.</li><li><strong>Multiple constraints</strong> (think budget limits, user preferences, pet-friendly hotels!).</li><li><strong>Several interconnected sub-tasks</strong> (finding hotels, booking flights, picking activities).</li></ul><p>Even advanced techniques like Chain-of-Thought (CoT) guide models linearly, which is too simplistic for real-world planning. Tree-of-Thought (ToT) improves on this by allowing multi-path reasoning, but it still treats each step as an independent node, limiting its ability to manage interconnected tasks with shared constraints. Many existing methods also heavily rely on human-crafted examples or predefined agent setups, which makes them less generalizable.</p><p>Enter <strong>HyperTree Planning (HTP)</strong>, a novel reasoning paradigm that teaches LLMs to think in levels, not just lines. It’s like teaching an LLM to become a <strong>project manager</strong>, capable of breaking down colossal problems into manageable subgoals and conquering each one systematically.</p><p><strong>What is HyperTree Planning?</strong></p><p>Instead of merely generating text linearly, HTP helps LLMs build a <strong>tree (or, more precisely, a hypertree)</strong>. This isn’t just any tree; it’s a flexible structure where:</p><ul><li>Each branch can contain <strong>multiple sub-steps</strong>.</li><li>Tasks are <strong>grouped logically</strong>.</li><li>The model thinks <strong>top-down</strong>, rather than left-to-right.</li></ul><p>Think of it as giving the LLM a <strong>whiteboard</strong> where it can map out the entire problem before diving into solutions. A hypertree is uniquely suited for planning because <strong>each parent node can simultaneously point to multiple child nodes</strong>, which is perfect for scenarios where subtasks need to be solved in conjunction, like selecting transportation and accommodations for a trip.</p><p><strong>How Does This Super Cool System Work?</strong></p><p>HTP proceeds in three elegant phases:</p><ol><li><strong>HyperTree Construction — “Breaking it down first”</strong> Before anything else, the LLM asks itself, “What are the <em>main</em> parts of this task?”. It identifies the core subgoals and constructs the entire hierarchical structure upfront.</li></ol><ul><li><strong>Example</strong>: For a “Trip to Rishikesh,” the LLM might first outline:</li><li>Flights</li><li>Departing Flight</li><li>Returning Flight</li><li>Bus/Train</li><li>Accommodations</li><li>Budget Limits</li><li>Preferable locations given by user (if any)</li><li>Activities (Day 1 : Shopping, Day 2 : River Rafting, Day 3 -&gt; Chopta Trek)</li></ul><p>This initial outline is crucial because it gives the LLM a <strong>structured scaffold</strong> to guide its reasoning, preventing it from getting lost in a “big block of text”. The process is fully autonomous and doesn’t require human intervention to generate this outline.</p><p><strong>2. Self-Guided Reasoning — “Solve Each Branch with Deep Thought”</strong> Once the HyperTree is beautifully laid out, the model tackles each branch independently, one sub-problem at a time. This is where the magic happens!</p><ul><li>For <strong>flights</strong>, it’ll diligently search for the cheapest round trip.</li><li>For <strong>accommodations</strong>, it will pick hotels within the budget and near specified landmarks.</li><li>For <strong>activities</strong>, it will find the closest, most highly-rated options that fit the budget. This independent handling of branches is a game-changer, significantly <strong>reducing confusion, repetition, or hallucinations</strong> because the model only has to focus on one subgoal at a time, rather than trying to remember everything at once. It can also integrate external knowledge, like tourist attraction lists or flight databases, to enrich its decisions.</li></ul><p><strong>3. Final Plan Generation — “Combine all the Extracted Information”</strong> Voila! With all branches meticulously solved, the model simply compiles everything into one <strong>clean, well-structured, and constraint-aligned output</strong>.</p><ul><li><strong>Example</strong>:</li><li>Flights -&gt; Round trip from Bengaluru to New-Delhi</li><li>Train -&gt; Delhi to Haridwar</li><li>Bus — Haridwar to Rishikesh</li><li>Hotel -&gt; The Hostellers(dorm)</li></ul><p><strong>Why HTP is a Big Deal!</strong></p><p>HTP isn’t just another incremental improvement; it’s a <strong>significant advancement</strong> for LLMs handling complex planning. Here’s why:</p><ul><li><strong>Superior Performance</strong>: Researchers put HTP through its paces on complex reasoning tasks like Multi-Day Traveling Plans (TravelPlanner), Blocksworld Logic, and Hierarchical Task Decomposition (Trip Planning). HTP <strong>outperformed all popular reasoning methods</strong> like Chain of Thought or Tree of Thought in accuracy and consistency, especially on <strong>longer and more detailed tasks</strong>. It was even <strong>up to 4x more effective</strong> in planning quality!</li><li><strong>Efficiency on a Budget</strong>: Even smaller models using HTP <strong>beat larger models</strong> relying on basic prompting. HTP also proved more cost-effective, achieving better results at a lower API token cost. For instance, Gemini-1.5-Pro + HTP achieved a 36.1% success rate on TravelPlanner, a 4.06x improvement over previous methods.</li><li><strong>Autonomy and Generalization</strong>: HTP doesn’t rely on fine-tuning, human intervention, human-crafted examples, or predefined agent roles. It’s a <strong>modular framework</strong> that can be applied to <em>any</em> LLM system needing to handle long-horizon tasks. This is super powerful!</li><li><strong>Mimicking Human Thought</strong>: The core takeaway? <strong>Structure matters just as much as intelligence</strong>. HTP allows LLMs to reason in levels, mirroring how humans naturally think and plan projects, rather than just acting as autocomplete engines.</li></ul><p><strong>A Glimpse at the Future (and Current Hurdles)</strong></p><p>While HTP is a giant leap, our LLMs still have some growing to do. Current limitations include their struggle with complex single-step reasoning, a lack of human-like prior knowledge (like budgeting strategies), and vulnerability to errors in very long reasoning chains.</p><p>But don’t despair! HTP opens exciting avenues for future research. It naturally integrates with <strong>self-reflection and backtracking</strong>, making it ideal for real-world tasks like meeting planning or calendar scheduling, with more accurate error correction. It’s also a strong candidate for equipping <strong>autonomous agents</strong> with robust hierarchical reasoning abilities for domains like logistics or robotics.</p><p>In conclusion, HyperTree Planning is revolutionizing how LLMs tackle complex tasks. By instilling a structured, hierarchical thinking process akin to human project managers, HTP empowers LLMs to break down vast problems, solve them systematically, and deliver impressive results. It’s a testament to the idea that sometimes, the <em>how</em> you think is just as important as the <em>what</em> you know.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=babd30560b00" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Hitchhiker’s Guide to Fine Tune LLMs on a Mac]]></title>
            <link>https://medium.com/@neevdeb26/the-hitchhikers-guide-to-fine-tune-llms-on-a-mac-85174455457a?source=rss-e44d9391c5fe------2</link>
            <guid isPermaLink="false">https://medium.com/p/85174455457a</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[fine-tuning]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Rhitam Deb]]></dc:creator>
            <pubDate>Tue, 24 Jun 2025 13:50:27 GMT</pubDate>
            <atom:updated>2025-10-02T17:16:34.119Z</atom:updated>
            <content:encoded><![CDATA[<p>Hello there, fellow AI enthusiasts! Rhitam here, hoping to save you from the head-scratching, late-night debugging sessions I recently endured, let me tell you, for a while there, it felt like chasing a unicorn through a Python environment that kept crumbling around me!</p><p>I had this grand vision of tweaking a Mistral-7B-Instruct-2410–4Bit to do my bidding, to learn my quirky style, or perhaps just to stop hallucinating that CSK won the IPL of 2025 (no offence to the Yellow Army, of course, a true ‘Thala Fan’!). I dived headfirst into various frameworks, battling memory limits, struggling with compatibility issues, and spending what felt like eons just trying to get models to run efficiently. It was like trying to fit a square peg into a round hole, only the peg was a multi-billion parameter model, and the hole was my beautifully sleek, yet stubbornly uncooperative, MacBook Air M3. I would run a script, hear the fans whir up like a jet engine, and then…a crash. Or worse, a silent failure that left me wondering if the LLM Gods were playing a cruel joke.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V9W5zmmBe0sXcNj-v31c6Q.png" /></figure><p>But then, a beacon of hope appeared on my horizon: <strong>Apple’s MLX framework</strong>. It was like discovering the secret handshake for my Mac to finally play nice with LLMs! This isn’t just another library; it’s Apple’s native machine learning framework, specifically designed for Apple silicon. And suddenly, fine-tuning became not just possible, but surprisingly straightforward. So, if you’ve been in my shoes, wrestling with your Mac to bend an LLM to your will, pull up a chair, because this guide is for you. We’re going to do it all on your own hardware, and it won’t cost you a penny.</p><p><strong>Why Bother Fine-Tuning? Aren’t LLMs Smart Enough Already?</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EqHDh-a0zYzpqmo_4epH2Q.png" /></figure><p>That’s a fair question! Pre-trained LLMs, like the ones I download from Hugging Face or the Ollama library, are indeed miraculous. They’re equipped with a broad understanding of language, having been trained on vast datasets. Think of them as brilliant generalists, capable of answering almost any question you throw at them.</p><p>However, sometimes “general” isn’t “perfect.” This is where <strong>fine-tuning</strong> swoops in, representing a strategic approach to model development. Instead of training an entire large language model from scratch (which, let’s be honest, is incredibly resource-intensive and time-consuming), fine-tuning allows us to take that pre-trained generalist and further train it on a smaller, specialized dataset. The model retains its foundational knowledge but develops expertise in your target domain.</p><p><strong>Fine-tuning is crucial for a few reasons:</strong></p><ul><li><strong>The Power of Customization<br> </strong>It adjusts how the model will phrase what it says, its syntax, and the format of its output. Imagine wanting an LLM that sounds exactly like you when it drafts mails, or one that adheres to very specific reporting protocols for medical summaries.</li><li><strong>Built for Niche Tasks<br> </strong>For certain custom tasks, fine-tuning can yield better results than relying on a general LLM. It allows for achieving high performance on specific tasks with less data and computational requirements.</li><li><strong>The Adaptability Edge<br> </strong>It’s about tweaking the model weights to change how it responds. While you <em>can</em> add new information directly into the prompt for the model to use, fine-tuning fundamentally changes the model’s inherent behaviour and style. This is different from approaches like Retrieval Augmented Generation (RAG), which involve providing external knowledge to the model at inference time to enhance its responses. The sources provided here focus on modifying the model itself through fine-tuning, not on external knowledge retrieval systems.</li></ul><p><strong>Pre-training vs. Fine-tuning vs. The Other Guys: A Quick Dive</strong></p><p>To truly appreciate fine-tuning, let’s quickly clarify where it sits in the LLM ecosystem:</p><ul><li><strong>Pre-training<br> </strong>This is the initial, colossal phase where LLMs learn broad language understanding from massive datasets. It’s the heavy lifting that gives the model its foundational intelligence.</li><li><strong>Fine-tuning<br> </strong>This is the focused, subsequent training on a smaller, specialized dataset. It’s about taking that broadly intelligent model and refining its skills for specific tasks, styles, or domains. This involves adjusting the model’s internal “weights”.</li><li><strong>RAG (Retrieval Augmented Generation)<br> </strong>This concept, while not explicitly detailed in these sources, is often contrasted with fine-tuning. Unlike fine-tuning, RAG doesn’t alter the model’s weights. Instead, it involves retrieving relevant external information <em>at inference time</em> and providing it to the LLM alongside the user’s prompt, allowing the model to generate responses based on up-to-date or specific knowledge it wasn’t pre-trained on.</li></ul><p><strong>The Art of Fine-Tuning: Methods and Approaches</strong></p><p>Fine-tuning isn’t a one-size-fits-all endeavour. There are various techniques, each with its own trade-offs:</p><ul><li><strong>Instruction Fine-tuning<br> </strong>This method focuses on training the model using examples that demonstrate the desired response to a query. You build datasets of prompt-completion pairs (e.g., “Summarize this text:” followed by the text and its summary). This teaches the model to “think” in a new, niche way, which is often how foundation models are trained in their post-training phase.</li><li><strong>Full Fine-tuning<br> </strong>As the name suggests, this involves updating <em>all</em> of the model’s weights. While it creates a new version of the model with comprehensively updated weights, it demands significant memory and computational resources, much like pre-training itself. Even for a 7-billion parameter model, this can chew up 45GB of memory and run relatively slowly on a Mac.</li><li><strong>Parameter-Efficient Fine-tuning (PEFT)<br></strong>This is where things get really exciting for us Mac users! Training a full LLM is computationally intensive, and storing all the necessary parameters for training (gradients, optimizers, etc.) requires a lot of memory. PEFT methods come to the rescue by updating only a small subset of parameters, effectively “freezing” the rest. This approach not only reduces memory footprint significantly but also speeds up training and helps combat “catastrophic forgetting,” where the model might lose its original abilities after specialized training.</li><li><strong>LoRA (Low-Rank Adaptation)</strong><br> This is a prominent PEFT method and a personal favourite for fine-tuning on Apple silicon. Instead of modifying the entire weight matrix of the model, LoRA introduces small, trainable “rank decomposition matrices”. These matrices, when multiplied, approximate the full weight update, dramatically reducing memory requirements while maintaining performance. For instance, LoRA can reduce memory usage by 65% and enable much faster training. With LoRA, a 7-billion parameter model that previously used 45GB memory and ran at 95 tokens/second could run at 328 tokens/second and only use 17GB of memory. It’s a game-changer!</li><li><strong>QLoRA (Quantized LoRA)<br> </strong>This takes LoRA’s efficiency a step further by using 4-bit quantization of the base model. It employs clever memory management techniques, allowing training on consumer GPUs with minimal performance loss. MLX supports various fine-tuning types, including LoRA, Dora, Cura, and full model fine-tuning.</li></ul><p><strong>Step-by-Step Guide to LoRA Fine-Tuning with MLX on Your Mac</strong></p><p>Ready to dive in? Here’s how you can fine-tune an LLM using MLX on your Apple silicon Mac. This process typically involves setting up your environment, preparing your data, running the fine-tuning, and then testing your newly minted model.</p><p><strong>Step 1: Setting Up Your Environment (It’s Easier Than You Think!)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/220/1*riVMe7RXKsyCTeA1ZizjJg.gif" /></figure><p>First things first, let’s get your Mac ready. This part is surprisingly simple thanks to MLX.</p><p><strong>Install MLX LM:</strong> Open your terminal and run:</p><p><em>pip install -U mlx-lm</em></p><p>That’s it! No complex dependencies or hour-long compilation processes.</p><p><strong>Hugging Face CLI (Optional but Recommended):</strong> Many pre-trained models reside on Hugging Face. To easily download them, you’ll want the Hugging Face CLI:</p><p><em>pip install “huggingface_hub[cli]”</em></p><p><em>huggingface-cli login</em></p><p>You’ll need to create an access token on Hugging Face’s settings page to log in.</p><p><strong>Step 2: Building Your Training Data (The Art, Not the Dark Arts)</strong></p><p>This is often considered the trickiest part, but it’s more about thoughtful preparation than black magic. The quality and diversity of your dataset directly impact your fine-tuned model’s performance.</p><p><strong>Format Requirements<br> </strong>MLX expects your data in specific JSONL files. You’ll need three files in a designated “data” directory:</p><ul><li>train.jsonl (for training the model)</li><li>valid.jsonl (for validating its learning progress)</li><li>test.jsonl (for a final evaluation).</li><li>Each line in these files must be a self-contained JSON object.</li></ul><p><strong>Content Structure<br> </strong>The simplest format for MLX fine-tuning (especially for completion tasks) is a JSON object with exactly two fields: prompt and completion. For chat-like models, you might use role and content fields to define user and assistant turns. You can also use a ‘text’ format which merges all information into a single text field. For example, in a medical diagnosis use case, a record might look like: {‘text’: “You are a medical diagnosis expert. You will give patient symptoms: ‘I have been experiencing increased urination, abdominal pain, nausea.’. Question: ‘What is the diagnosis I have?’. Response: You may be diagnosed with Pneumonia.”}.</p><p><strong>Data Volume &amp; Diversity<br> </strong>Aim for at least 500 to 1000 examples, though more is often better. Crucially, <strong>diversity is key!</strong> If all your “completions” are short, your model might learn to give short responses for <em>everything</em>, even unrelated questions. The same goes if you repeatedly use the same phrase (like “The result is…”); the model might integrate that phrase into unrelated answers. You can use a “mixer” dataset, which includes reminders of general tasks the model should still be able to do, to prevent “catastrophic forgetting” of previous capabilities.</p><ol><li><strong>Splitting Your Data:</strong> A common split is 60% for training, 20% for testing, and 20% for validation.</li><li><strong>Tools for Data Preparation:</strong> For large JSONL files, tools like JQ on the command line can be invaluable for reformatting or stripping unwanted fields.</li></ol><p><strong>Step 3: Selecting Your Base LLM (The Goldilocks Principle)</strong></p><p>Choosing the right pre-trained model is important. Consider your MacBook’s specifications, your dataset, and the specific task you’re aiming for.</p><p><strong>Size Matters (But Bigger is Not Always Better)</strong></p><p><strong>Small Models (e.g., 500 million parameters): </strong>These are lightning-fast (e.g., 1300 tokens/second for a full fine-tune, 2300 tokens/second with LoRA) and memory-efficient (e.g., 3.7GB peak memory). They can learn patterns well but might struggle with complex reasoning or be more susceptible to being over-influenced by your training data’s biases (e.g., always trying to use a “calculator” if that’s what you trained it on heavily).</p><p><strong>Large Models (e.g., 7 billion parameters): </strong>These offer better quality, make fewer mistakes, and handle data diversity better. However, they are slower (e.g., 95 tokens/second for full fine-tune, 328 tokens/second with LoRA) and very memory-intensive (e.g., 45GB peak memory for full fine-tune).</p><p><strong>Sweet Spot (e.g., 3 billion parameters): </strong>Often, a medium-sized model can strike a good balance between performance, speed, and memory usage.</p><p>Quantized models from the mlx-community on Hugging Face often work best on Apple silicon GPUs. For this guide, we’ll imagine using a Mistral-8B-Instruct-4bit model as an example, as the provided sources fine-tuned a Ministral-8B-Instruct-2410–4bit model.</p><p><strong>Download your model:</strong> You can use the huggingface-cli command you set up earlier:</p><p><em>huggingface-cli download mlx-community/Ministral-8B-Instruct-2410–4bit</em></p><p>This will download the model to your Hugging Face cache.</p><p><strong>Step 4: Fine-tuning Your LLM Using the MLX Framework (The Magic Moment!)</strong></p><p>With your data ready and model selected, it’s time to unleash MLX! We’ll use the mlx_lm.lora command.</p><p><strong>The Command:</strong></p><p>python -m mlx_lm.lora \<br> — model mistralai/Ministral-8B-Instruct-2410 \<br> — data ./data \<br> — train \<br> — fine-tune-type lora \<br> — adapter-path ./adapters_ministral8b \<br> — batch-size 1 \<br> — iters 1000 \<br> — load-in-4bit</p><p><strong>Step 5: Monitoring Progress<br> </strong><br> During training, you’ll see valuable metrics-</p><ul><li><strong>Trainable parameters:</strong> Shows the small percentage of parameters being updated with LoRA (e.g., 0.71%).</li><li><strong>Validation loss (Val loss):</strong> This is like a pop quiz for your model on unseen data. You want this number to gradually decrease. If it starts going up while training loss goes down, your model might be memorizing instead of truly learning.</li><li><strong>Training loss (Train loss):</strong> Indicates how well the model is learning from your training data; this should also decrease over time.</li><li><strong>Speed metrics:</strong> iterations per second, tokens per second give you an idea of training speed.</li><li><strong>Peak memory:</strong> Helps you monitor your Mac’s memory usage.</li></ul><p><strong>Memory Management<br> <br> </strong>Remember, LoRA itself is a memory-saver. If you still face issues, consider reducing batch-size, num-layers, or using a smaller base model. MLX also supports grad-checkpoint to trade some speed for lower memory usage.</p><p><strong>Step 6: Testing and Interacting with Your Fine-tuned LLM</strong></p><p>Once training is complete, MLX saves your LoRA adapters. Now, let’s see what your model has learned!</p><p><strong>Testing with Adapters:</strong> You can generate responses using your base model combined with the new adapters:</p><p>python -m mlx_lm.generate \</p><p>— model mlx-community/Ministral-8B-Instruct-2410–4bit \</p><p>— adapter-path ./adapters_ministral8b \</p><p>— prompt “Symptoms: I have been experiencing memory loss, stiffness and difficulty walking. Question: What could be the diagnosis I have?”</p><p>You should observe a “clear difference” in the output, aligning with your training data’s style and content. For instance, if you trained it to act as a medical diagnosis expert, it should give precise, focused responses as per your examples.</p><p><strong>Using Chat Mode (if applicable):</strong> If you fine-tuned for chat, you’d use mlx_lm.chat:</p><p><em>mlx_lm.chat \</em></p><p><em>— model mlx-community/Ministral-8B-Instruct-2410–4bit \</em></p><p><em>— adapter-path ./adapters_ministral8b \</em></p><p>You can then converse with your fine-tuned model directly.</p><p><strong>Step 7: Fusing the Adapters (Creating a Standalone Masterpiece)</strong></p><p>The final flourish! After successful fine-tuning with LoRA, you can “fuse” your adapter weights back into the original base model. This creates a standalone fine-tuned model that no longer requires separate adapter files.</p><p><strong>Fusing Command</strong></p><p><em>python -m mlx_lm.fuse \</em></p><p><em>— model mlx-community/Ministral-8B-Instruct-2410–4bit \</em></p><p><em>— adapter-path ./adapters_ministral8b \</em></p><p><em>— save-path models/fine-tuned_Mistral-8B</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tvkS8-DC6t47PLjjaDZ37Q.png" /></figure><p>This command combines your base model with the learned adaptations.</p><p><strong>Using the Fused Model<br> <br> </strong>Now you can interact with your fused model directly, just by pointing to its new saved path:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/1*10evJS7jSdu9IvXtWTto3Q.png" /></figure><p><strong>Parting Thoughts: The Power is Now Yours!</strong></p><p>Fine-tuning LLMs can be a complex process, a delicate balance of model size, data diversity, training iterations, and method choice. You’ve seen how easy it is for an LLM to “forget” tasks if your data isn’t diverse, or to pick up unintended habits like shortening all responses. It’s a push and pull, a constant dance of influencing and reminding.</p><p>But hopefully, you’ve also seen how Apple MLX, especially with techniques like LoRA, makes this incredibly powerful capability accessible on your personal Mac. Whether it’s teaching a model to call a calculator for math problems (training it on <em>process</em> rather than <em>outcome</em>) or making it sound just like you, the possibilities are immense. And the fact that even smaller models (like the half-billion or three-billion parameter ones) can perform remarkably capable tasks is nothing short of incredible.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/451/1*nlTuV9FyR0crBt8nt4AyKw.gif" /></figure><p>So, go forth and fine-tune! Your Mac, armed with MLX, is ready for the challenge. And trust me, the satisfaction of seeing your LLM respond exactly as you trained it is a pretty sweet reward. Happy fine-tuning!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=85174455457a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Grounding LLMs with Knowledge Graphs for Zero-Shot QA]]></title>
            <link>https://medium.com/@neevdeb26/grounding-llms-with-knowledge-graphs-for-zero-shot-qa-8fa50de07d46?source=rss-e44d9391c5fe------2</link>
            <guid isPermaLink="false">https://medium.com/p/8fa50de07d46</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[chatbots]]></category>
            <category><![CDATA[knowledge-graph]]></category>
            <dc:creator><![CDATA[Rhitam Deb]]></dc:creator>
            <pubDate>Sun, 08 Jun 2025 04:09:33 GMT</pubDate>
            <atom:updated>2025-06-08T09:21:53.696Z</atom:updated>
            <content:encoded><![CDATA[<p>We all love how powerful Large Language Models (LLMs) are, especially their ability to answer questions right out of the box — what researchers call “zero-shot closed-book question answering”. This is possible because of the vast amount of knowledge they’ve internalized during pre-training.</p><p>Pretty cool, right?</p><p>But here’s the catch: this internalized knowledge can sometimes be insufficient, incorrect, or even outdated. This can lead to a common problem in LLMs: hallucination, where the model generates factually wrong answers. Updating this knowledge by fine-tuning the LLMs is also quite expensive, especially when information is constantly changing.</p><p>This is where <a href="https://arxiv.org/abs/2306.04136">KAPING</a> comes in! KAPING, which stands for Knowledge-Augmented Language Model Prompting, is a framework designed to tackle these issues head-on. Its core idea is to retrieve and inject relevant external knowledge directly into the LLM’s input, known as a prompt. The best part? KAPING requires no model training, making it entirely zero-shot. It can work with any off-the-shelf LLM.</p><p>The knowledge source KAPING uses is a Knowledge Graph (KG). A KG stores symbolic knowledge as triples in the form of (head entity, relation, tail entity). For example, the fact “Upanishads written in India” can be represented as the triple (“Upanishads”, “written in”, “India”). KGs can provide accurate and contextual facts that are helpful for answering questions, also it finds patters which might be lost while having a vector index.</p><p>This article is my approach to implement the same onto a pet project of mine.</p><p>Happy reading!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/1*t1wMlcbuOFtDLse3EKp5eA.png" /></figure><p><strong>The KAPING Methodology: A Step-by-Step Look</strong></p><p>How does KAPING actually work? It’s a clever pipeline that processes the input question and integrates KG knowledge before handing it off to the LLM.</p><p>For the project, I am using <a href="https://networkx.org/">networkx</a>, shout out to them!</p><p>Below is how I have structured my Knowledge Graph Class and its constructors :</p><pre>class KnowledgeGraph:<br>    def __init__(self):<br>        self.graph = nx.DiGraph()<br>        self.encoder = SentenceTransformer(&#39;sentence-transformers/all-mpnet-base-v2&#39;)<br><br>    def add_triple(self, subject: str, relation: str, object: str):<br>        &quot;&quot;&quot;Add a triple to the knowledge graph.&quot;&quot;&quot;<br>        self.graph.add_edge(subject, object, relation=relation)<br><br>    def get_all_entities(self) -&gt; Set[str]:<br>        &quot;&quot;&quot;Get all entities in the knowledge graph.&quot;&quot;&quot;<br>        return set(self.graph.nodes())<br><br>    def get_triples_for_entity(self, entity: str, hop: int = 1) -&gt; List[Tuple[str, str, str]]:<br>        &quot;&quot;&quot;Get all triples associated with an entity within specified hop distance.&quot;&quot;&quot;<br>        triples = []<br>        visited = set()<br><br>        def collect_triples(node, current_hop):<br>            if current_hop &gt; hop or node in visited:<br>                return<br>            visited.add(node)<br><br>            # Collect outgoing edges<br>            for successor in self.graph.successors(node):<br>                triples.append((node, <br>                              self.graph[node][successor][&#39;relation&#39;],<br>                              successor))<br>                if current_hop &lt; hop:<br>                    collect_triples(successor, current_hop + 1)<br><br>            # Collect incoming edges<br>            for predecessor in self.graph.predecessors(node):<br>                triples.append((predecessor,<br>                              self.graph[predecessor][node][&#39;relation&#39;],<br>                              node))<br>                if current_hop &lt; hop:<br>                    collect_triples(predecessor, current_hop + 1)<br><br>        collect_triples(entity, 1)<br>        return triples</pre><p>Here’s the breakdown of the approach:</p><ol><li>Knowledge Access: First, the system needs to figure out which entities in the input question are relevant to the KG. For a question like “Who is the author of Lady Susan?”, it would identify “Lady Susan” as a key entity. Existing entity linking techniques can be used to match entities in the question to those in the KG. Once the KG entity is found, all its associated triples (facts) are considered potential candidates.</li></ol><p>In the below code I have used SpaCy and DBpedia Spotlight to identofy named entities in text.</p><p>The extract_entities function combines both SpaCy&#39;s entity recognition and DBpedia Spotlight&#39;s entity linking capabilities, hence providing complementary entity detection capabilities for better overall results.</p><p>In this function (link_to_kg) we match the extracted entities from text with entities in a knowledge graph (KG), I call it “finding the tribe”, using SpaCy’s word vector similarity calculations(we can use other models as well). For each extracted entity, it finds the most similar KG entity by computing similarity scores (using SpaCy’s word vectors), but only accepts matches above a threshold of 0.7 to ensure quality matches.</p><pre>class EntityLinker:<br>    def __init__(self):<br>        self.nlp = spacy.load(&quot;en_core_web_lg&quot;) # change it with respect to language<br>        self.entity_linker = pipeline(&quot;entity_extraction&quot;, model=&quot;DBpedia/spotlight-model&quot;)<br><br>    def extract_entities(self, text: str) -&gt; List[str]:<br>        &quot;&quot;&quot;Extract entities from text using SpaCy and DBpedia Spotlight.&quot;&quot;&quot;<br>        # SpaCy entity extraction<br>        doc = self.nlp(text)<br>        spacy_entities = {ent.text for ent in doc.ents}<br>        <br>        # DBpedia Spotlight entity linking<br>        try:<br>            spotlight_entities = set(self.entity_linker(text))<br>        except:<br>            spotlight_entities = set()<br>            <br>        # Combined results<br>        return list(spacy_entities.union(spotlight_entities))<br><br>    def link_to_kg(self, entities: List[str], kg_entities: Set[str]) -&gt; List[str]:<br>        &quot;&quot;&quot;Match extracted entities to KG entities.&quot;&quot;&quot;<br>        linked_entities = []<br>        for entity in entities:<br>            # Find closest matching KG entity<br>            max_similarity = 0<br>            best_match = None<br>            <br>            for kg_entity in kg_entities:<br>                similarity = self.nlp(entity).similarity(self.nlp(kg_entity))<br>                if similarity &gt; max_similarity and similarity &gt; 0.7:  # threshold<br>                    max_similarity = similarity<br>                    best_match = kg_entity<br>                    <br>            if best_match:<br>                linked_entities.append(best_match)<br>                <br>        return linked_entities</pre><p>2. Knowledge Verbalization: LLMs understand text, but KG triples are symbolic (like (entity, relation, entity)). So, the candidate triples are transformed into a textual string — this is called verbalization.</p><p>What I found that a simple linear verbalization works well: just concatenating the subject, relation, and object texts in the triple, example: (“Upanishads”, “written in”, “India”) becomes the text string “(Upanishads written in India)”.</p><pre>class TripleVerbalizer:<br>    def __init__(self):<br>        pass<br><br>    def verbalize_triple(self, triple: Tuple[str, str, str]) -&gt; List[str]:<br>        &quot;&quot;&quot;Convert a triple into a natural language string.&quot;&quot;&quot;<br>        subject, relation, object_ = triple<br>        return [self.verbalize_triple(triple) for triple in triples]</pre><p>3 . Question-Relevant Knowledge Retrieval: This is a crucial step! Instead of overwhelming the LLM with all associated triples, KAPING retrieves only the top-K most relevant ones to the specific question.</p><p>How?</p><p>It represents both the verbalized triples and the input question in an embedding space using off-the-shelf sentence embedding models. Then, it calculates the semantic similarity between the question embedding and each triple embedding. Only the triples with the highest similarity scores (the top-K) are kept. This filtering step is essential because many triples associated with an entity might be irrelevant to the question and could actually mislead the LLM. We experint with K=5 retrieved facts and found that retrieving from 1-hop neighbors of question entities generally worked better than 2-hops. In the below code I have used 1.</p><pre><br>class KnowledgeRetriever:<br>    def __init__(self, kg: KnowledgeGraph, k: int = 5):<br>        self.kg = kg<br>        self.k = k<br>        self.encoder = kg.encoder<br>        self.entity_linker = EntityLinker()<br>        self.verbalizer = TripleVerbalizer()<br><br>    def retrieve(self, question: str) -&gt; List[str]:<br>        &quot;&quot;&quot;Retrieve and verbalize the most relevant knowledge for a question.&quot;&quot;&quot;<br>        # 1. Entity Linking<br>        extracted_entities = self.entity_linker.extract_entities(question)<br>        kg_entities = self.kg.get_all_entities()<br>        linked_entities = self.entity_linker.link_to_kg(extracted_entities, kg_entities)<br><br>        # 2. Collect candidate triples (1-hop)<br>        all_triples = []<br>        for entity in linked_entities:<br>            all_triples.extend(self.kg.get_triples_for_entity(entity, hop=1))<br><br>        # 3. Rank triples by relevance<br>        question_embedding = self.encoder.encode(question)<br>        triple_scores = []<br><br>        for triple in all_triples:<br>            verbalized_triple = self.verbalizer.verbalize_triple(triple)<br>            triple_embedding = self.encoder.encode(verbalized_triple)<br>            <br>            score = torch.cosine_similarity(<br>                torch.tensor(question_embedding).unsqueeze(0),<br>                torch.tensor(triple_embedding).unsqueeze(0)<br>            ).item()<br>            <br>            triple_scores.append((triple, score))<br><br>        # 4. Select top-K most relevant triples<br>        triple_scores.sort(key=lambda x: x[1], reverse=True)<br>        selected_triples = [t[0] for t in triple_scores[:self.k]]<br><br>        # 5. Verbalize selected triples<br>        return self.verbalizer.verbalize_triples(selected_triples)</pre><p>If you’ve made it this far, here’s a little reward: a “wizard” function, if you want to play with hops :</p><pre>def get_n_hop_neighbors(self, start_node: str, n: int) -&gt; List[tuple]:<br>        &quot;&quot;&quot;Get all n-hop neighbors from a starting node.&quot;&quot;&quot;<br>        if n == 0:<br>            return []<br><br>        all_triples = []<br>        visited = set()<br>        <br>        def dfs(node, current_hop, path=[]):<br>            if current_hop &gt; n or node in visited:<br>                return<br>            <br>            visited.add(node)<br>            <br>            # Get outgoing edges<br>            for successor in self.kg.graph.successors(node):<br>                edge_data = self.kg.graph[node][successor]<br>                triple = (node, edge_data[&#39;relation&#39;], successor)<br>                all_triples.append((triple, current_hop))<br>                if current_hop &lt; n:<br>                    dfs(successor, current_hop + 1, path + [triple])<br>            <br>            # Get incoming edges<br>            for predecessor in self.kg.graph.predecessors(node):<br>                edge_data = self.kg.graph[predecessor][node]<br>                triple = (predecessor, edge_data[&#39;relation&#39;], node)<br>                all_triples.append((triple, current_hop))<br>                if current_hop &lt; n:<br>                    dfs(predecessor, current_hop + 1, path + [triple])<br>        <br>        dfs(start_node, 1)<br>        return all_triples</pre><p>4. Prompting : The selected top-K verbalized triples are now formatted and injected into the prompt for the LLM. This step requires comprehension of the data pattern received and the task you want to get done through the LLM, since it’s zero-shot.</p><p>Let’s Pray to the LLM Gods!</p><p>5. Answer Generation: Finally, the LLM receives the augmented prompt containing both the question and the relevant KG facts. Conditioned on this factual knowledge, the LLM generates the answer. Because the LLM is provided with accurate, external facts, it is more likely to generate factually correct answers, alleviating the hallucination problem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/1*XGsYxlM87k1jWwDDGopjlQ.png" /></figure><p>This pipeline allows LLMs to use factual knowledge from KGs without needing to modify their internal parameters.</p><p><strong>Why This Approach Matters?</strong></p><p>This has potential in significant performance improvements on Knowledge Graph Question Answering (KGQA) tasks, compared to baselines that don’t use knowledge or use less sophisticated augmentation methods. The retrieval of relevant facts is key, as opposed to just any facts associated with entities.</p><p>Furthermore, using cosine KG triples can be more computationally efficient for LLM prompting than augmenting longer documents, especially when considering out-of-memory or out-of-length errors with larger models or many documents. It also shows potential for adapting to updated knowledge — if the facts in the KG change, updating the triples in the prompt allows the LLM to generate answers based on the new information without expensive model updates.</p><p><strong>Looking Ahead</strong></p><p>While KAPING shows <strong><em>great promise</em></strong>, I can point out some limitations like performance is highly dependent on the quality of the retrieved facts. Handling complex questions requiring multi-hop reasoning might require improvements in the retrieval component. Also, evaluating the generative output of LLMs for KGQA can be tricky, as they tend to produce sentence-level answers while KG answers are typically entities. Finally, in real-world scenarios, the initial step of entity linking (identifying entities in the question and mapping them to the KG) is crucial, and its accuracy impacts KAPING’s overall performance.</p><p>Despite these challenges, KAPING offers a <strong><em>compelling and practical zero-shot mechanism</em></strong> for grounding LLMs with external knowledge from KGs, paving the way for more reliable and factual question answering.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8fa50de07d46" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>