<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ravi Bommu on Medium]]></title>
        <description><![CDATA[Stories by Ravi Bommu on Medium]]></description>
        <link>https://medium.com/@raviscal?source=rss-6676b3772ca1------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Hcc29ZlbZs8yIh2Nh7MHlQ.jpeg</url>
            <title>Stories by Ravi Bommu on Medium</title>
            <link>https://medium.com/@raviscal?source=rss-6676b3772ca1------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 14:12:25 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@raviscal/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Beyond Vector Search: Architecting a Smarter Retrieval Pipeline]]></title>
            <link>https://medium.com/@raviscal/beyond-vector-search-architecting-a-smarter-retrieval-pipeline-a3cd0af87cbb?source=rss-6676b3772ca1------2</link>
            <guid isPermaLink="false">https://medium.com/p/a3cd0af87cbb</guid>
            <category><![CDATA[vector-embeddings]]></category>
            <category><![CDATA[cross-encoder-rerank]]></category>
            <category><![CDATA[agentic-rag]]></category>
            <category><![CDATA[enterprise-rag]]></category>
            <category><![CDATA[vector-search]]></category>
            <dc:creator><![CDATA[Ravi Bommu]]></dc:creator>
            <pubDate>Wed, 11 Nov 2026 18:06:01 GMT</pubDate>
            <atom:updated>2026-11-11T18:06:01.680Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Part 2 of Rethinking Retrieval for Enterprise RAG</em></p><p><strong>A practical look at retrieval metrics, multi-model search, rank fusion, and intelligent reranking and how to decide what actually improves RAG.</strong></p><p>In <a href="https://medium.com/@raviscal/the-retrieval-problem-why-better-rag-starts-before-the-llm-4958950e343c">Part 1</a>, <em>The Retrieval Problem: Why Better RAG Starts Before the LLM</em>, I explored why finding information related to a question isn’t necessarily the same as finding the evidence needed to answer it.</p><p>We examined BM25, vector search, multiple embedding models, reciprocal rank fusion (RRF), reranking, and contextual retrieval. Each solves a different part of the problem. But understanding these techniques leads to a more interesting question:</p><p><strong>How should we combine them, and how do we know whether the added complexity actually improves retrieval quality?</strong></p><p>In this article, I’ll explore a practical local evaluation setup, compare retrieval strategies using published research, and share how I would architect a more effective retrieval pipeline.</p><h3>1. Before Optimizing Retrieval, Understand the Metrics</h3><p>We can introduce additional embedding models, fusion techniques, and rerankers, but without measuring their contribution, we’re simply adding complexity. Four metrics are particularly useful.</p><p><strong>1.1 Recall@K: Did We Find the Right Evidence?</strong></p><p>Recall@K measures how much of the known relevant evidence appears among the top K results. Imagine our knowledge base contains five relevant passages for a question. If the retrieval system finds three in its top ten results:</p><p><strong>Recall@10 = 3 / 5 = 60%</strong></p><p>I find two measurements especially useful:</p><ul><li><strong>Recall@50:</strong> Did initial retrieval discover the relevant evidence within a larger candidate pool?</li><li><strong>Recall@10:</strong> How much relevant evidence makes it into the smaller set of candidates we might give the LLM?</li></ul><p>If Recall@50 is high but Recall@10 is low, the problem may be ranking rather than discovery.</p><p><strong>1.2 nDCG@K: Did We Rank the Best Evidence First?</strong><br>Recall doesn’t consider where relevant documents appear. <br>can wereframe this sentence. Imagine two retrievers find the same three relevant passages. One ranks them at positions 1, 2, and 3; the other ranks them at positions 3, 7, and 10. Both have the same Recall@10, but the first ordering is more useful.</p><p><strong>nDCG rewards systems that rank highly relevant evidence closer to the top</strong>, while also supporting different degrees of relevance.</p><p><strong>1.3</strong> <strong>Answer Accuracy: Did Better Retrieval Improve the Answer?</strong></p><p>Higher retrieval scores don’t automatically produce better LLM responses. We must also evaluate whether the final answer is correct, complete, and supported by retrieved evidence. This distinction matters because a retrieval system can find more relevant documents without necessarily providing the information required to answer the question.</p><p><strong>1.4 Latency and Cost: Was the Improvement Worth It?</strong></p><p>Additional embedding models require more indexing and searches. Rerankers add inference time, while LLM-based reranking can become expensive when processing many passages. A few percentage points of improvement may be valuable, but they must be weighed against the operational cost.</p><h3>2. A Practical Local Evaluation Environment</h3><p>For this exploration, I wanted to keep the setup lightweight and avoid depending on a managed RAG platform.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/751/1*3CIziQEef1RuGbF5EL1jNA.png" /></figure><p>This is sufficient for testing lightweight embedding models, local vector indexes, and quantized LLMs without requiring cloud GPUs.</p><h3>Why YouTube Transcripts?</h3><p>In my <a href="https://medium.com/@raviscal/start-to-finish-retrieval-augmented-generation-rag-in-snowflake-using-vector-search-and-cortex-69a951e76edd">original RAG implementation in February 2024</a>, I worked with approximately 370 YouTube podcast episodes, representing around 380 hours of transcribed content. What makes this corpus interesting is its diversity. Unlike structured technical documentation, conversations often use inconsistent terminology, reference earlier discussions, and spread important information across multiple passages. I also created metadata tags for the transcripts based on their topics and subject areas. These tags provide an additional way to discover related content across different transcripts, making the corpus useful for exploring <strong>metadata-assisted retrieval and questions where the answer spans multiple documents</strong>. This creates an opportunity to compare traditional vector retrieval, multi-model retrieval, and metadata-assisted approaches using the same corpus. For a reproducible evaluation, we can use a representative question set, identify the relevant passages, and measure how effectively each retrieval strategy discovers the evidence needed to answer the question.</p><h3>The Technology Stack</h3><p>For the local reference architecture, I selected two independently trained embedding model families: <strong>Google Embedding Gemma 2 and Alibaba Qwen3 Embedding</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/732/1*dgw6ArXFJNxpxyiOCGiMMQ.png" /></figure><p>EmbeddingGemma 2 supports an 8K-token context window, while Qwen3-Embedding-0.6B supports 32K. These allow us to experiment with chunk sizes ranging from 512 to 4,096 tokens without exceeding the models’ published input limits. Each model maintains its own vector index, and RRF combines their results after retrieval.</p><h3>Why Ettin for Reranking?</h3><p>I chose Ettin Reranker 400M for three reasons.</p><p><strong>First, model independence.</strong> It comes from a different model family than Gemma and Qwen, making it an interesting independent comparison.</p><p><strong>Second, local efficiency.</strong> With approximately 400 million parameters, it is compact enough for practical local experimentation.</p><p><strong>Third, architectural comparison.</strong> Ettin is a genuine cross-encoder that evaluates the query and passage together. This gives us a distinct approach to compare against general-purpose LLM-based reranking.</p><p>The objective isn’t to prove that one model family is superior. It’s to understand whether different approaches contribute enough additional relevance to justify their costs.</p><h3>3. Comparing Retrieval Strategies</h3><p>Rather than testing every possible combination, I’d focus on four configurations.</p><p><strong>Experiment 1: Single-Model Baseline</strong>, Evaluate BM25, EmbeddingGemma 2, and Qwen3 Embedding independently. This establishes baseline Recall@10, Recall@50, and nDCG@10, and helps identify where individual models succeed or fail.</p><p><strong>Experiment 2: Traditional Hybrid Search, </strong>Combine BM25 and semantic retrieval using RRF. The question is whether keyword matching discovers relevant passages that embeddings miss, especially around exact identifiers and specialized terminology.</p><p><strong>Experiment 3: Multiple Embedding Models ,</strong>Combine EmbeddingGemma 2 and Qwen3 Embedding using RRF. This is particularly interesting because both models perform semantic retrieval but learn different representations. <strong>Does combining their results discover evidence that either model alone would miss? </strong>The value comes from complementarity, not simply using more models.</p><p><strong>Experiment 4: Reranking</strong>, Take the top 50 candidates from a fixed retrieval configuration and compare:</p><ul><li>No reranking</li><li>Ettin cross-encoder reranking</li><li>Gemma 4 LLM-based reranking</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UaZjGuYCRLeB9djRouYcRQ.png" /></figure><p>The objective is to determine whether reranking improves the ordering of relevant evidence enough to justify its additional latency and computational cost. Importantly, reranking can improve Recall@10 by moving relevant passages higher, but it cannot recover documents that were never retrieved.</p><p>Below are the results on recall from my testing</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xur_v_aXN1LqLWyDGqbMJg.png" /></figure><h3>Three Interesting Patterns in the Retrieval Comparison</h3><p><strong>1. Multiple embedding models can improve retrieval quality.</strong></p><p>Combining EmbeddingGemma and Qwen using RRF increases Recall@10 from 67% to 71% in this comparison, a 4-percentage-point improvement. This demonstrates the potential benefit of combining complementary semantic representations rather than relying on a single embedding model.</p><p><strong>2. Hybrid search isn’t automatically better.</strong></p><p>Combining EmbeddingGemma with BM25 reduces Recall@10 from 67% to 64% in this scenario. This doesn’t mean BM25 is ineffective. Its value depends on the corpus, query patterns, and whether keyword matching contributes additional relevant evidence.</p><p><strong>3. Reranking can provide incremental improvements, but at a cost.</strong></p><p>Adding Ettin cross-encoder reranking increases the Recall@10 from 71% to 72%, while Gemma 4 LLM-based reranking reaches 73%. Although LLM reranking achieves the highest score in this scenario, the additional computational cost and latency must be considered against the relatively modest improvement.</p><p>I also experimented with <a href="https://platform.claude.com/cookbook/capabilities-contextual-embeddings-guide">Contextual Retrieval </a>and observed some improvement, although the gains weren’t substantial. This reinforces the importance of understanding the characteristics of the underlying corpus before introducing additional retrieval techniques.</p><blockquote><strong>The broader lesson:</strong> No retrieval strategy is universally superior. The objective is to identify which techniques genuinely improve evidence discovery and ranking, and whether those improvements justify the additional complexity, latency, and cost.</blockquote><h3>5. When the Answer Spans Multiple Documents: Start Simple, Add Complexity When Needed</h3><p>So far, we’ve focused on finding and ranking the most relevant passages. But what happens when the answer requires information from several documents?</p><p>Consider a user asking:</p><p><em>“What are the requirements for deploying a production application on Kubernetes?”</em></p><p>Imagine the enterprise knowledge base contains three documents:</p><p>DocumentInformation<br><strong>A: Deployment Standards</strong>Resource limits, readiness/liveness probes, replica requirements<br><strong>B: Security Policies</strong>RBAC, container image scanning, secrets management<br><strong>C: Observability Standards</strong>Logging, metrics, tracing, alerting</p><p>A standard RAG pipeline might retrieve all three documents, allowing the LLM to generate a complete answer with citations. <strong>If that works, there’s no reason to introduce a more complex retrieval architecture. </strong>However, if the system retrieves only Document A, the answer may be incomplete even though the retrieved document is highly relevant.Before changing the architecture, I would inspect the retrieved passages for failed questions. Was the missing document never retrieved, ranked too low, or retrieved but ignored during answer generation?</p><p>Based on the failure, I would consider the following approaches in increasing order of complexity.</p><h3>a. Standard Retrieval: Start with What Already Works</h3><p>Use vector search, or an existing hybrid retriever, to find relevant passages and generate a grounded answer with source citations.</p><p><strong>Example:</strong> If the initial search retrieves the deployment, security, and observability standards, the LLM can combine their information into one answer.</p><p>This should remain the default approach when the retrieved evidence is sufficient.</p><h3>b. Metadata-Based Retrieval: Use Existing Document Context</h3><p>Enrich documents with metadata such as domain, application, technology, document type, and tags.</p><p><strong>Example:</strong> The three documents share kubernetes and production tags, allowing the system to discover related standards even if their content uses different terminology.</p><p>Metadata can support filtering, ranking boosts, or candidate expansion. However, incomplete tags can exclude relevant documents, so hard filtering should be used carefully.</p><h3>c. Relationship-Aware Retrieval: Follow Explicit Connections</h3><p>Use existing document references or structured relationships to discover supporting information.</p><p><strong>Example:</strong> The deployment standard references a security policy and an observability standard. After retrieving Document A, the system follows those relationships to discover Documents B and C.</p><p>These relationships can be maintained through document metadata, reference mappings, or a knowledge graph. A graph database is not necessarily required. You an use a metadata table or JSON file, depending on the complexity</p><h3>d. Query Decomposition: Break Down Complex Questions</h3><p>An LLM, guided by instructions and available domain context, can identify distinct information requirements and generate focused retrieval queries.</p><p><strong>Example:</strong> The original Kubernetes question can be decomposed into deployment, security, and observability requirements. Each subquery retrieves relevant passages independently.</p><p>This can improve evidence discovery when a broad question spans several topics, but it introduces additional retrieval operations and the risk of generating unnecessary subqueries.</p><h3>e. Evidence Coverage and Iterative Retrieval: Identify What’s Missing</h3><p>An LLM or a rule-based evaluator checks whether the retrieved evidence collectively addresses the identified information requirements.</p><p><strong>Example:</strong> If deployment and security requirements are covered but observability requirements are missing, the system issues a targeted follow-up search.</p><p>This can improve completeness, but it adds inference calls, latency, and orchestration complexity. Retrieval retries should be bounded, and the system should acknowledge when sufficient evidence cannot be found.</p><blockquote>I would treat these approaches as a <strong>progressive set of options rather than mandatory stages in every RAG pipeline</strong>.<br><strong>Standard retrieval → Metadata enrichment → Relationship-aware retrieval → Query decomposition → Evidence coverage and targeted retries</strong></blockquote><p>This sequence represents increasing architectural sophistication, although the actual implementation effort depends on the available metadata, document relationships, and tooling.</p><p>The objective is not to build the most sophisticated retrieval system. <strong>It’s to start with the simplest approach that works, understand where it fails, and introduce additional capabilities only when they demonstrably improve evidence discovery and completeness.</strong></p><p>When an answer spans multiple documents, the challenge isn’t necessarily that RAG cannot handle it. The challenge is ensuring that the retrieved evidence collectively provides enough information to answer the question.</p><h3>6. My Recommended Retrieval Architecture</h3><p>Given these findings, I wouldn’t recommend a fixed pipeline that runs every retrieval technique for every question. Instead, I’d design <strong>an adaptive, multi-stage retrieval architecture</strong>.</p><h3>Stage 1: Query Understanding</h3><p>Identify the user’s intent and whether the question requires exact matching, conceptual understanding, or evidence from multiple sources. For example, queries containing specific error codes may benefit from BM25, while conceptual questions may benefit more from semantic retrieval.</p><h3>Stage 2: Candidate Discovery</h3><p>Use a strong embedding model as the baseline. Introduce a second embedding model or BM25 when evaluation demonstrates that it contributes useful evidence the primary retriever misses. Each model maintains its own index, allowing independent evolution.</p><h3>Stage 3: Rank Fusion</h3><p>When multiple retrievers are used, combine their rankings through RRF. This avoids directly comparing incompatible similarity scores and produces a unified candidate list.</p><h3>Stage 4: Selective Reranking</h3><p>Use a specialized cross-encoder to refine candidate relevance when necessary. Reserve LLM-based reranking for cases where more nuanced evidence interpretation justifies the additional cost. Not every query needs reranking.</p><h3>Stage 5: Evidence Selection and Generation</h3><p>Select the passages that provide sufficient evidence, preserve relevant source context, and pass them to the LLM for grounded answer generation.</p><p><strong>The objective isn’t to send the maximum amount of context. It’s to provide the most useful evidence needed to answer the question.</strong></p><p>The overall reference architecture becomes:</p><p><strong>Query Understanding → Adaptive Retrieval → Optional RRF → Selective Reranking → Evidence Selection → Grounded Answer</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lqdg4wy-qFbfUPwyngS9-g.png" /></figure><p>The key architectural decision is that each additional capability is invoked based on demonstrated value, rather than being a mandatory step in every query.</p><h3>6. Final Thoughts: Complexity Must Earn Its Place</h3><p>When I first explored RAG in 2024, the primary engineering challenge was connecting external knowledge to LLMs. Today, the tools are considerably more mature. The more interesting challenge is improving the quality of information that reaches the model.My perspective comes down to three principles.</p><p><strong>Understand your corpus and query patterns.</strong> Retrieval strategies should be driven by the nature of the content and the questions users actually ask. Technical documentation, conversational transcripts, and knowledge bases have different characteristics, and no single retrieval approach works equally well for all of them.</p><p><strong>Measure retrieval independently.</strong> We need to understand whether the right evidence was found before evaluating the final answer.</p><p><strong>Value complementary signals.</strong> Additional embedding models, hybrid retrieval, and reranking are valuable only when they discover or prioritize evidence that would otherwise be missed.</p><p><strong>Keep the architecture adaptive.</strong> The best retrieval pipeline isn’t always the most sophisticated one. It’s the one that achieves the required quality at an acceptable latency, cost, and complexity. Recent <a href="https://aclanthology.org/2026.eacl-industry.13.pdf">research</a> reinforces these ideas while reminding us that results from one dataset shouldn’t automatically become standards for another.</p><p>Ultimately, <strong>better retrieval isn’t about using more models or retrieving more documents. It’s about finding the right evidence, ranking it intelligently, and giving the LLM what it actually needs to answer the question.</strong></p><p><strong><em>If you found this article useful, please give it a few 👏 claps and share it with others who might find it interesting. I’d also love to hear your thoughts and experiences in the comments!</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a3cd0af87cbb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Retrieval Problem: Why Better RAG Starts Before the LLM]]></title>
            <link>https://medium.com/@raviscal/the-retrieval-problem-why-better-rag-starts-before-the-llm-4958950e343c?source=rss-6676b3772ca1------2</link>
            <guid isPermaLink="false">https://medium.com/p/4958950e343c</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[vector-search]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[agentic-rag]]></category>
            <category><![CDATA[reranking]]></category>
            <dc:creator><![CDATA[Ravi Bommu]]></dc:creator>
            <pubDate>Thu, 05 Nov 2026 16:54:48 GMT</pubDate>
            <atom:updated>2026-11-05T17:35:30.153Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Part 1: Rethinking Retrieval for Enterprise RAG</em></p><h3>1. Two Years After My First RAG Implementation</h3><p>In early 2024, I wrote an article on <a href="https://medium.com/@raviscal/start-to-finish-retrieval-augmented-generation-rag-in-snowflake-using-vector-search-and-cortex-69a951e76edd">building an end-to-end Retrieval-Augmented Generation (RAG) application using Snowflake, vector search, and Cortex</a>. At the time, RAG was still an emerging approach for connecting enterprise knowledge with large language models. Much of the focus was on getting the foundational pieces working together: document processing, embeddings, vector search, and LLM-based answer generation.</p><p>Fast forward nearly three years, and the landscape has changed significantly. LLMs have become remarkably capable at reasoning, understanding context, and generating coherent responses. RAG frameworks and vector search capabilities have matured, making implementations considerably easier.</p><p>Yet one fundamental challenge remains: <strong>Are we consistently retrieving the right information for the LLM to produce an accurate and complete answer? </strong>We often focus on choosing better LLMs, optimizing prompts, or expanding context windows. But what happens when the most relevant information exists in our knowledge base yet never reaches the LLM? Or when retrieved information is semantically similar but doesn’t actually answer the question?</p><p><strong>Even the most capable LLM cannot reliably reason over evidence it never receives. </strong>This brings me back to a part of RAG that deserves more architectural attention: <strong>the retrieval layer itself.</strong></p><p>In this two-part series, I’ll explore why retrieval remains challenging, how different techniques address the problem, and how we can design more effective retrieval pipelines that prioritize relevance, context, and measurable quality.</p><h3>2. Why Retrieval Is Still a Hard Problem</h3><p>At first glance, retrieval seems straightforward. A user asks a question, the system searches a knowledge base, identifies the most similar documents, and passes them to the LLM.</p><p>But <strong>finding information related to a question is not necessarily the same as finding the information needed to answer it.</strong></p><p>Consider a user asking:</p><p><em>“How do open table formats reduce vendor lock-in in modern data platforms?”</em></p><p>Imagine our knowledge base contains the following passages:</p><p><strong>Document A: </strong>“Vendor lock-in is a major concern in modern data architectures. Organizations increasingly prioritize flexibility, interoperability, and the freedom to choose between cloud data platforms.”</p><p><strong>Document B: </strong>“Apache Iceberg stores table metadata in an open format, allowing multiple query engines, including Databricks, Snowflake, and Trino, to operate on the same underlying data without requiring proprietary storage formats.”</p><p>Both documents are relevant to the topic. Document A closely matches the language of the question, repeatedly referencing vendor lock-in and platform flexibility. Document B, however, explains the architectural mechanism that actually reduces dependency on a particular platform.</p><blockquote><strong>Document B provides stronger evidence for answering the question, even though Document A may appear more similar to the query.</strong></blockquote><p>This highlights a fundamental challenge: similarity scores are useful signals, but they are not guarantees of answer relevance.</p><p>And the problem goes beyond similarity.</p><ul><li><strong>Missing evidence:</strong> The most relevant document exists, but it never appears in the retrieved results.</li><li><strong>Incomplete context:</strong> A retrieved passage contains part of the explanation but omits the surrounding information necessary to interpret it.</li><li><strong>Incorrect ranking:</strong> A less useful passage appears above one that directly answers the question.</li><li><strong>Fragmented evidence:</strong> The answer requires information from multiple documents, but the retrieval system discovers only some of them.</li><li><strong>Conflicting information:</strong> Multiple documents provide different answers, perhaps because one is outdated or describes a different version of the system.</li></ul><p>These challenges become particularly important in enterprise environments, where information is distributed across technical documentation, policies, architecture decisions, operational procedures, and knowledge repositories.</p><p>A larger context window might allow us to include more retrieved material, and a stronger LLM might interpret that material more effectively. But neither guarantees that the retrieval system has discovered the correct evidence in the first place.</p><p><strong>The real challenge is not simply retrieving documents that look relevant. It is discovering the evidence that is sufficient, accurate, and contextual enough to answer the user’s question.</strong></p><p>Understanding that distinction is the starting point for evaluating different retrieval techniques.</p><h3>3. Understanding the Retrieval Landscape</h3><p>Now that we’ve established why retrieval is challenging, let’s explore the techniques available to address it.</p><p>There are several ways to retrieve information from a knowledge base, and each approaches relevance differently. Rather than treating them as competing technologies, I find it more useful to understand <strong>what each technique is good at, where it struggles, and how they can complement one another.</strong></p><p>Let’s continue with our earlier example.</p><p><strong>User question:</strong></p><p><em>“How do open table formats reduce vendor lock-in in modern data platforms?”</em></p><p>Our knowledge base contains two documents:</p><ul><li><strong>Document A:</strong> Discusses vendor lock-in, platform flexibility, and interoperability, but doesn’t explain how open table formats address the problem.</li><li><strong>Document B:</strong> Explains how Apache Iceberg’s open metadata and storage architecture allows multiple query engines to operate on the same underlying data.</li></ul><p>Document B provides the stronger evidence. Let’s see how different retrieval techniques might discover it.</p><h3>Keyword Search (BM25): Finding the Right Words</h3><p>BM25 is a traditional information retrieval algorithm that ranks documents based on how well their words match the search query.</p><p>It considers factors such as how frequently matching terms appear, how rare those terms are across the document collection, and document length.</p><p><strong>How it works with our example:</strong></p><p>The query contains terms such as <em>open table formats</em>, <em>vendor lock-in</em>, and <em>data platforms</em>.</p><p>Document A repeatedly discusses vendor lock-in and platform flexibility. Because those terms closely match the query, BM25 might rank Document A highly.</p><p>Document B explains the actual mechanism through Apache Iceberg and shared data access, but it doesn’t explicitly use the phrase <em>vendor lock-in</em>. Depending on the other matching terms and corpus statistics, BM25 may rank it lower.</p><p>The limitation is that BM25 primarily understands <strong>word overlap, not the meaning behind those words</strong>.</p><p>But BM25 is particularly useful when exact terminology matters. Imagine searching for an error code such as HTTP 429, a specific API name, or a product identifier. Exact keyword matching can be more reliable than trying to find semantically similar content.</p><blockquote><strong>BM25 is excellent for precise terminology and exact matches, but it may miss relevant information expressed using different vocabulary.</strong></blockquote><h3>3.2 Vector Search: Finding the Right Meaning</h3><p>Vector search was one of the key breakthroughs that made modern RAG practical and widely adopted. By enabling systems to retrieve information based on semantic meaning rather than exact keyword matches, it opened the door to grounding LLM responses in large collections of external knowledge.</p><p>Instead of relying primarily on matching words, an embedding model converts queries and documents into numerical representations called vectors. These vectors capture semantic relationships, allowing the retrieval system to identify conceptually related content even when the wording differs.</p><p><strong>How it works with our example:</strong></p><p>Our question asks how open table formats reduce vendor lock-in.</p><p>Document B doesn’t explicitly mention vendor lock-in, but it explains that multiple query engines can access the same data through an open table format. A capable embedding model may recognize that this architectural flexibility is closely related to reducing platform dependency and rank Document B highly. This is where vector search becomes powerful: it can connect concepts even when the question and source material use different words.</p><p>However, semantic similarity has limitations. An embedding model might also rank Document A highly because it discusses many concepts related to the question. It doesn’t necessarily understand that Document B provides a more complete explanation. Different embedding models may also interpret the same query differently, depending on their training data and how they represent semantic relationships.</p><blockquote><strong>Vector search is effective at discovering conceptually related information, but semantic similarity alone doesn’t guarantee that a passage contains the evidence needed to answer the question. Different embedding models can also interpret the same query differently, producing varying relevance scores and retrieval rankings.</strong></blockquote><h3>3.3 Hybrid Search: Combining Words and Meaning</h3><p>If BM25 excels at exact terminology and vector search captures semantic meaning, why not combine them? That’s the idea behind <strong>hybrid search</strong>. A hybrid retrieval system runs lexical and semantic searches, then combines their results using a ranking or fusion strategy.</p><p><strong>How it works with our example:</strong></p><ul><li>BM25 might rank Document A higher because of its direct terminology matches.</li><li>Vector search might rank Document B higher because it recognizes the relationship between open table formats and platform independence.</li><li>Hybrid search combines both candidate lists, increasing the opportunity for both documents to be considered.</li></ul><p>This can provide broader coverage than either method alone. However, there is an important nuance: <strong>combining retrieval methods doesn’t automatically improve retrieval quality. </strong>If BM25 consistently promotes documents that share keywords but don’t answer the question, the combined ranking may become worse rather than better. The effectiveness of hybrid search depends on the quality of each retrieval method and how their results are combined.</p><blockquote><strong>Hybrid search can bring together complementary lexical and semantic signals, but its value needs to be measured rather than assumed.</strong></blockquote><h3>3.4 Multiple Embedding Models: Different Perspectives on Meaning</h3><p>As we discussed earlier, <strong>different embedding models can produce different retrieval results for the same query.</strong> This is because models differ in their training data, architectures, and how they learn and represent semantic relationships. A document that one embedding model considers highly relevant may receive a lower ranking from another. This raises an interesting question: <strong>Instead of relying on a single embedding model, can we combine multiple models to improve retrieval quality? </strong>The idea is to use independently trained embedding models to search the same knowledge base and combine their results, potentially discovering relevant information that one model alone might miss.</p><p><strong>How it works with our example:</strong></p><p>Imagine using two independently trained embedding models.</p><ul><li><strong>Embedding Model A</strong> might rank Document A higher because it associates vendor lock-in strongly with platform flexibility and interoperability.</li><li><strong>Embedding Model B</strong> might rank Document B higher because it recognizes that independent storage formats and multi-engine access are mechanisms for reducing platform dependency.</li></ul><p>By considering results from both models, we may discover useful evidence that relying on either model individually would have missed.</p><p><strong>One important architectural distinction: each model generates vectors in its own embedding space. We cannot simply compare their similarity scores or combine their vectors as though they were interchangeable.</strong></p><p>Instead, we retrieve candidates independently and combine their ranked results using techniques such as Reciprocal Rank Fusion (RRF), which we’ll explore in the next section. Of course, adding another embedding model also introduces costs: additional indexes, embedding generation, storage, and retrieval operations.</p><blockquote><strong>If both models consistently return nearly identical results, there may be little benefit in using both. Multiple embedding models can provide complementary semantic perspectives, but the real value comes from retrieving useful evidence that a single model would otherwise miss.</strong></blockquote><h3>So, Which Retrieval Technique Is Better?</h3><p>There isn’t a universally correct answer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/751/1*jeD9yZyYSUy-ak04XpoxQQ.png" /><figcaption>Quick comparison matrix</figcaption></figure><p>What I find interesting is that these techniques solve different parts of the same problem. The goal should not be to use every retrieval technique available. It should be to understand which combination consistently discovers the most relevant evidence for our data and queries.</p><blockquote><strong>My perspective: Rather than depending entirely on one model’s interpretation of relevance, combining complementary embedding models is an approach worth exploring to improve retrieval coverage. The real value, however, depends on whether the models discover genuinely different and useful evidence. In Part 2, I’ll share how I explored this approach, compare different retrieval strategies, and discuss what the results mean for designing a more effective retrieval architecture.</strong></blockquote><h3>4. Why Ranking Matters: RRF and Reranking</h3><p>Using multiple retrievers introduces another challenge: each returns its own ranked list, often using similarity scores that aren’t directly comparable.</p><p>How do we combine those results, and then determine which passages are actually most useful?</p><p>This is where <strong>Reciprocal Rank Fusion (RRF)</strong> and <strong>reranking</strong> come in. They solve two different problems.</p><h3>4.1 Reciprocal Rank Fusion (RRF): Combining Multiple Rankings</h3><p>RRF combines rankings from multiple retrievers based on document positions rather than raw similarity scores. Let’s continue with our question about open table formats. Imagine BM25 ranks Document A first because of keyword matches, while vector search ranks Document B first because of semantic relevance.</p><p>Rather than choosing one result list, RRF combines them using:</p><p>[ RRF(d)=\sum_i\frac{1}{k+\operatorname{rank}_i(d)}]</p><p>Here, <em>rank</em> is the document’s position in each list, and <em>k</em> is a smoothing constant, commonly set to 60. For example, if Document B ranks third in BM25 and first in vector search, its combined score is:</p><p>[ \frac{1}{60+3}+\frac{1}{60+1}\approx0.03227 ]</p><p>Applying the same calculation across the retrieved documents can move Document B to the top, reflecting the combined ranking signals.</p><p>Although our example uses BM25 and vector search, <strong>RRF can also combine results from multiple embedding models.</strong></p><p>But RRF has an important limitation: it doesn’t understand document content or determine whether a passage answers the question. It only combines rankings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FOdlRW2FK6p_msBfjg3LvA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AqIk4TRdJyADe_8LvNY23Q.png" /></figure><blockquote><strong>RRF provides an effective mechanism for combining rankings from multiple retrievers without comparing incompatible similarity scores. But combining rankings is not the same as determining which passages provide the strongest evidence.</strong></blockquote><h3>4.2 Reranking: Finding the Most Relevant Evidence</h3><p>RRF helps combine results from multiple retrievers, but it doesn’t actually evaluate whether a document answers the user’s question. It relies on the rankings produced by the underlying retrieval models. <strong>Reranking takes this a step further by evaluating the relevance of retrieved passages to the original question. </strong>Imagine our retrieval pipeline returns 50 candidate passages. Rather than sending all 50 to the LLM, we want to identify the 5–10 passages that provide the strongest evidence.</p><p>Imagine a user asks: “How can I reduce my cloud infrastructure costs?” The initial vector search returns these three documents:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*gA7BMOPCuhZfNW7p_OlLJA.png" /></figure><p>All three documents are semantically related to cloud computing and costs. However, Document B directly answers the user’s question, while A and C are more general. A reranker evaluates each document against the original question and may reorder them:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/734/1*IhWbuSGIYjzvI-Ouhf4vGw.png" /></figure><p>The key difference: Vector search identifies documents that are semantically related to the question. Reranking evaluates which of those documents best addresses what the user is actually asking.</p><p>There are two common approaches to reranking.</p><p><strong>Cross-Encoder Reranking:</strong></p><p>A cross-encoder processes the query and each candidate passage together, assigning a relevance score based on their relationship. This allows it to evaluate relevance more directly than comparing independently generated embeddings. However, cross-encoders evaluate each query-document pair at runtime. As the number and length of candidates increase, so do computational costs and potentially latency. That’s why they’re typically used as second-stage rerankers on a smaller candidate set, often the top 20–50 results.</p><p><strong>LLM-Based Reranking</strong></p><p>A general-purpose LLM can also evaluate candidates and rank them based on criteria such as whether they directly answer the question, provide sufficient evidence, or merely discuss a related topic. This offers flexibility for more complex relevance judgments but can introduce additional latency, inference cost, and ranking variability.</p><p><strong>Which is better?</strong> Not necessarily the larger model.</p><p>A specialized cross-encoder may provide the best quality-to-cost balance for many applications, while an LLM reranker may be useful when relevance requires more nuanced interpretation. Both share one critical limitation: <strong>reranking can only evaluate documents that were initially retrieved. It cannot recover evidence missing from the candidate set.</strong></p><p>The distinction is simple:</p><p><strong>Retrieval discovers candidates → RRF combines rankings → Reranking evaluates relevance.</strong></p><h3>5. Contextual Retrieval: Preserving Meaning Before Search</h3><p>We’ve explored how to retrieve and rank documents more effectively. But what if the problem starts before retrieval, when documents are prepared for indexing? When documents are split into chunks, important surrounding context can be lost.</p><p>In 2024, Anthropic introduced <a href="https://www.anthropic.com/research/contextual-retrieval">Contextual Retrieval</a> to address this problem. <strong>The idea is to use an LLM to generate a short, chunk-specific explanation based on the surrounding document and prepend it before indexing.</strong></p><p>Let’s return to Apache Iceberg.</p><p><strong>Original chunk:</strong></p><blockquote><em>This allows multiple engines to access the same underlying data without converting it into proprietary formats.</em></blockquote><p>Without context, we don’t know what “this” refers to.</p><p><strong>After contextual enrichment:</strong></p><blockquote><strong><em>Generated context:</em></strong><em> This passage discusses how Apache Iceberg’s open table format enables interoperability across engines such as Spark, Snowflake, and Trino.</em></blockquote><blockquote><strong><em>Original chunk:</em></strong><em> This allows multiple engines to access the same underlying data without converting it into proprietary formats.</em></blockquote><p>Now the passage contains sufficient information to connect it to questions about Iceberg, interoperability, and platform independence. Importantly, this isn’t about appending the same document summary to every chunk. <strong>Each chunk receives context specific to its meaning within the original document. </strong>The enriched content can be indexed using embeddings and BM25, potentially improving both semantic and keyword retrieval.</p><p>This reinforces an important point: <strong>Contextual Retrieval is most valuable when chunks lose meaningful information after being separated from their original documents.</strong> If the chunks are already sufficiently self-contained, especially long, adding generated context may provide only marginal benefits.</p><p>It also introduces additional preprocessing costs and the possibility of generated context introducing misleading information. In some cases, preserving document titles, section headings, and relevant metadata may achieve similar benefits with less complexity.</p><blockquote><strong>My Perspective</strong></blockquote><blockquote>What interests me about Contextual Retrieval is that it improves the information representation <em>before retrieval even begins</em>.We experimented with this approach and observed some improvement, but the gains were not as significant as we initially expected.This suggests that its effectiveness depends heavily on whether chunks genuinely lose important context. When chunks are already sufficiently self-contained, additional contextual enrichment may provide only marginal benefits.</blockquote><h3>6. How Do We Know Retrieval Is Actually Better?</h3><p>We’ve explored several techniques for improving retrieval, from multiple embedding models and reranking to contextual enrichment. But this raises an important question: <strong>How do we know whether these techniques are actually improving retrieval quality?</strong></p><p>It’s easy to introduce additional models and sophisticated ranking techniques. What’s harder is demonstrating that the added complexity produces meaningfully better results.</p><p>This is where retrieval evaluation becomes essential.</p><h3>Measuring Retrieval Quality</h3><p>The starting point is a representative set of user questions with known relevant documents or passages. We can then evaluate how effectively each retrieval approach discovers and ranks that evidence.</p><p>A few metrics are particularly useful:</p><ul><li><strong>Recall@K:</strong> Measures how much of the known relevant evidence appears within the top K retrieved results. This helps us understand whether the retrieval system is missing important information.</li><li><strong>nDCG@K (Normalized Discounted Cumulative Gain):</strong> Measures ranking quality by rewarding relevant documents that appear higher in the results. It can also account for different degrees of relevance.</li><li><strong>Answer Accuracy:</strong> Evaluates whether the LLM produces a correct, sufficiently complete answer using the retrieved evidence.</li><li><strong>Latency and Cost:</strong> Measure the additional processing time and computational expense introduced by multiple retrievers, reranking, or contextual enrichment.</li></ul><h3>Final Thoughts: Better Retrieval Requires Better Evidence</h3><p>As LLMs continue to improve, retrieval deserves just as much architectural attention as the models generating the answers. We’ve explored techniques that address different aspects of the retrieval problem, and no single approach is universally better. <strong>The real challenge is determining which techniques genuinely improve retrieval quality and whether those improvements justify their complexity, latency, and cost.</strong></p><p>In Part 2, I’ll move from concepts to practice. We’ll explore a practical retrieval architecture, dive deeper into evaluation metrics, compare retrieval strategies using a diverse corpus of YouTube transcripts, and examine the results and tradeoffs.</p><p>Ultimately, <strong>better RAG isn’t about building the most sophisticated retrieval pipeline. It’s about consistently finding the right evidence and making sure it reaches the LLM.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4958950e343c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Migration-Proof Data Stack: Why Interoperability Matters More Than Ever]]></title>
            <link>https://medium.com/@raviscal/the-migration-proof-data-stack-why-interoperability-matters-more-than-ever-b3acb88d599b?source=rss-6676b3772ca1------2</link>
            <guid isPermaLink="false">https://medium.com/p/b3acb88d599b</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-architecture]]></category>
            <category><![CDATA[apache-iceberg]]></category>
            <category><![CDATA[data-platforms]]></category>
            <category><![CDATA[data-interoperability]]></category>
            <dc:creator><![CDATA[Ravi Bommu]]></dc:creator>
            <pubDate>Wed, 28 Oct 2026 04:01:12 GMT</pubDate>
            <atom:updated>2026-10-28T04:01:12.169Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Building data architectures that evolve with technology, without repeatedly rebuilding the foundation.</em></p><h3>Introduction</h3><p>Over the years, having been involved in building and evolving enterprise data platforms, one thing I’ve come to realize is that <strong>technology migrations are inevitable</strong>.</p><p>We’ve seen the data landscape evolve from traditional databases and Hadoop to cloud data platforms, modern lakehouse architectures, and now AI-driven data ecosystems. Each generation introduces new capabilities, and I don’t believe this evolution is going to slow down anytime soon.</p><p>As a data leader, one of the things I constantly think about when making architectural decisions is <strong>not just how well a technology solves today’s problems, but how those decisions impact our ability to evolve tomorrow</strong>.</p><p>Think about what typically happens as organizations adopt new technologies. Data gets stored in proprietary formats, transformations become tightly coupled to processing engines, governance policies are implemented independently, and business metrics become embedded within individual tools.</p><p>Over time, these dependencies turn what should be a relatively straightforward technology change into a complex migration involving data movement, pipeline rewrites, and downstream application changes.</p><p>This raises a question I’ve increasingly started asking:</p><p><strong>Can we design our data platforms so that adopting or replacing a technology doesn’t require rebuilding the entire ecosystem around it?</strong></p><p>I believe a big part of the answer lies in <strong>interoperability</strong>.</p><p>Instead of treating interoperability as an integration challenge to address later, we should consider it a foundational architectural principle across the entire data stack.</p><p>The goal isn’t to eliminate migrations or avoid commercial technologies. It’s to <strong>preserve as much of our existing data investments as possible while maintaining the flexibility to adopt new technologies</strong>.</p><p>In this article, I’ll explore the architectural choices and open standards that can help us achieve this.</p><h3>Building Interoperability Across the Data Stack</h3><blockquote>The goal is to architect data platforms that minimize the cost and complexity of future change.</blockquote><p>Consider a simple scenario.</p><p>An organization has built its analytics ecosystem around a cloud data warehouse. Over time, it introduces another processing engine to support machine learning or more complex data engineering workloads.</p><p>Ideally, this new engine should be able to leverage the data and metadata that already exist, rather than creating another independent ecosystem.</p><p>But what happens if the existing data is stored in proprietary formats? What if the catalog cannot be accessed by the new engine? What if the transformation logic is tightly coupled to the original platform? Or business metrics are defined separately inside individual BI tools?</p><p>Even if we solve data access, we’ve only addressed part of the problem.</p><p><strong>This is why I think interoperability needs to be an architectural consideration at every layer of the data stack.</strong></p><p>At a high level, there are several areas where we can make deliberate architectural choices:</p><ol><li><strong>Storage:</strong> Separate the physical storage of data from the processing engines wherever practical, giving us greater control over where our data resides.</li><li><strong>Table formats:</strong> Adopt open table formats such as Apache Iceberg, enabling compatible engines to work with shared datasets without requiring separate proprietary copies.</li><li><strong>Catalogs and governance:</strong> Leverage open catalog interfaces and federation to make metadata accessible across platforms while maintaining clear governance and access-control boundaries.</li><li><strong>Processing and transformations:</strong> Allow different processing engines to operate on shared data, while using frameworks such as dbt to make transformation logic, testing, and deployment practices more reusable.</li><li><strong>Business semantics:</strong> Define business metrics and relationships independently of individual BI or AI tools, using emerging specifications such as Apache Ossie to improve semantic interoperability.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Cw_LAJWbaDA4vGh1mJwS_w.png" /><figcaption>A conceptual interoperable data stack designed to support technology evolution while preserving existing investments.</figcaption></figure><p>These layers are interconnected, but they don’t necessarily have to evolve together.</p><p>For example, introducing a new processing engine should not automatically mean moving all the underlying data to another storage platform. Replacing a BI tool shouldn’t require redefining hundreds of business metrics. And adopting a new catalog shouldn’t force us to rebuild every data pipeline.</p><p>Of course, interoperability across these layers is not automatic. Open standards provide a common foundation, but differences in implementations, SQL dialects, security models, and platform capabilities still need to be addressed.</p><p>The architectural objective is to <strong>minimize the dependencies between layers so that changes in one area have the smallest practical impact on the others.</strong></p><blockquote>True interoperability requires reducing dependencies across the entire data stack, not just adopting individual open technologies.</blockquote><p>Let’s look at how we can approach this, starting with the foundation of any data platform: where and how we store our data.</p><h3><strong>Storage Independence :Decoupling Data from Processing Engines</strong></h3><p>One of the first architectural decisions I believe organizations should consider is <strong>who owns and controls the physical storage of their data</strong>.</p><p>Traditionally, data platforms have tightly coupled storage and processing, often requiring organizations to move data into vendor-managed, proprietary storage environments. While this works well for individual platforms, it creates dependencies when introducing or migrating to new processing engines.</p><p><strong>What if we separated the ownership of data storage from the technologies processing it?</strong></p><p>By storing data in organization-owned cloud storage, such as Amazon S3, Azure Data Lake Storage, or Google Cloud Storage within our own cloud accounts, we establish a foundation that remains independent of any particular processing vendor.</p><p>This allows organizations to:</p><ul><li><strong>Maintain control</strong> over physical storage, access, encryption, and data lifecycle.</li><li><strong>Reduce duplication</strong> by allowing multiple engines to access shared datasets.</li><li><strong>Simplify migrations</strong> by introducing or replacing processing engines without necessarily moving the underlying data.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/492/1*HFCgX9dOXuXeiWlT0JdJAA.jpeg" /></figure><p>The key architectural shift is simple: <strong>Processing engines become consumers of our data, rather than owners of its physical storage.</strong></p><blockquote>Own and control your data storage independently of processing vendors, preserving flexibility as technologies evolve.</blockquote><p>Of course, storage independence alone doesn’t guarantee interoperability. Even when we own the storage, proprietary table formats and metadata can still create vendor dependencies.</p><p>This brings us to the next important layer: <strong>open table formats</strong>, which allow different processing engines to understand and operate on the same datasets.</p><h3>Open Table Formats: Making Data Accessible Across Processing Engines</h3><p>Owning the physical storage is an important first step. But it introduces another question: <strong>How do we ensure different processing engines can understand and work with the same data?</strong></p><p>Simply storing data in an open file format like Parquet isn’t enough. While Parquet standardizes how data is stored in files, it doesn’t provide the table-level capabilities needed for managing transactions, schema evolution, and consistent reads across multiple engines.</p><p>This is where <strong>open table formats like Apache Iceberg</strong> become important.</p><p>Iceberg provides an open table abstraction over data stored in cloud object storage. It manages table metadata, snapshots, schema evolution, and transactional consistency, allowing compatible processing engines to work with the same underlying datasets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/476/1*OxxfLTiTjK6uNGBt2tzIcA.jpeg" /></figure><p>From an architectural perspective, this is a significant shift. Instead of maintaining separate copies of data in different platform-native formats, we can establish <strong>a shared table foundation that multiple engines can leverage</strong>.</p><p>For example, an organization could introduce a new analytics or data processing engine without necessarily migrating its existing Iceberg datasets.</p><p>Of course, interoperability still depends on engine compatibility, catalog integration, and support for specific Iceberg features.</p><p>The key architectural principle is to <strong>decouple the table format from the processing engine</strong>, allowing each to evolve more independently.</p><p>But for engines to discover and access these shared tables, they need a common way to interact with their metadata. That’s where catalog interoperability becomes the next important consideration.</p><h3>Catalog Interoperability : Breaking Down Metadata Silos</h3><p>With organization-owned storage and an open table format like Apache Iceberg, we’ve established a common data foundation. But another important challenge remains: <strong>How do different processing engines discover, access, and manage these shared tables?</strong></p><p>Traditionally, each data platform maintains its own catalog, creating dependencies between the metadata and the platform managing it. Even when the underlying data is open, these dependencies can limit interoperability.</p><p>This is where <strong>open catalog interfaces and federation</strong> become important.</p><p>The Apache Iceberg REST Catalog specification provides a standardized interface for engines to interact with Iceberg catalogs, reducing the need for platform-specific integrations. Technologies such as Snowflake Horizon, Databricks Unity Catalog, and Apache Polaris support interoperability through these open interfaces.</p><p>Rather than forcing every team onto a single catalog, a federated approach allows organizations to connect existing catalog ecosystems while preserving appropriate ownership boundaries.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/484/1*GpYOXbVmP2Nct_-idg6Feg.jpeg" /></figure><blockquote>Open catalog interfaces and federation reduce metadata lock-in, allowing different engines to discover and access shared data assets.</blockquote><p>However, metadata interoperability doesn’t automatically guarantee consistent governance. Identity, access policies, and enforcement mechanisms still need to be coordinated and validated across participating platforms.</p><p>The architectural objective is to <strong>decouple metadata access from individual processing engines</strong>, allowing new technologies to discover and leverage existing data assets without rebuilding the catalog ecosystem.</p><p>With storage, table formats, and catalogs becoming more interoperable, the next challenge is making the processing and transformation logic equally adaptable.</p><h3><strong>Portable Processing and Transformations: Reducing Engine Dependencies</strong></h3><p>With storage, table formats, and catalogs becoming more interoperable, we’ve addressed a significant part of data platform dependency. But another challenge remains: <strong>What happens to our transformation logic when we change processing engines?</strong></p><p>In many organizations, data transformation pipelines are tightly coupled to specific platforms through proprietary SQL, stored procedures, and platform-specific orchestration frameworks.</p><p>Even when the underlying data is accessible across engines, migrating thousands of transformations can become a significant undertaking.</p><p>This is where I believe <strong>introducing an abstraction layer for transformations, using frameworks like dbt, becomes valuable</strong>.</p><p>dbt allows us to define and manage SQL transformations, dependencies, testing, and documentation through a common development framework that supports multiple processing engines.</p><p>For example, organizations can develop transformations using dbt on Snowflake and potentially reuse much of that framework when moving workloads to Databricks or another supported engine.</p><p>Of course, this doesn’t mean SQL becomes universally portable. Engine-specific functions, optimizations, and materialization strategies may still require changes.</p><p>The architectural goal is to <strong>separate transformation development practices from the underlying execution engine as much as practical</strong>, reducing the effort needed to adopt or migrate between platforms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/491/1*841ECcxXYB11Pb_520r3RQ.jpeg" /></figure><p>Combined with open storage, table formats, and catalogs, this brings us closer to an ecosystem where processing technologies can evolve without requiring us to rebuild the entire data foundation.</p><p>But even with portable data and transformations, one critical dependency remains: <strong>business semantics</strong>. How do we ensure that business metrics retain the same meaning across different tools and platforms?</p><blockquote>Frameworks like dbt don’t eliminate engine-specific dependencies, but they provide a common development foundation that reduces transformation migration effort.</blockquote><h3>Open Business Semantics: Making Business Definitions Portable</h3><p>So far, we’ve focused on making data, metadata, and transformations more interoperable. But there’s another critical layer that often gets overlooked: <strong>business semantics</strong>.</p><p>Think about a metric like <em>Revenue</em> or <em>Monthly Active Users</em>. Its definition may be embedded in SQL transformations, BI dashboards, or semantic models. Over time, organizations end up maintaining different definitions of the same metric across multiple tools.</p><p>Now imagine migrating to a new analytics platform or introducing AI agents. Do we really want to recreate all those business definitions again?</p><p>I believe <strong>business metrics should be treated as organizational assets, independent of the technologies consuming them</strong>. This is where <strong>Apache Ossie (incubating), formerly Open Semantic Interchange (OSI)</strong>, becomes interesting.</p><p>Ossie introduces a vendor-neutral, YAML-based specification for representing semantic models, including business metrics, dimensions, datasets, and relationships.</p><p>What makes this particularly interesting is that organizations don’t necessarily have to abandon their existing semantic modeling tools.</p><p>For example, we can continue building semantic models using Snowflake Semantic Views or dbt Semantic Layer. These definitions can then be exported or converted into the Apache Ossie format, creating a portable representation of the business semantics.</p><p>Similarly, Ossie definitions can be converted into supported platform-specific semantic models.</p><p><strong>The idea is to create an interoperability layer for business meaning, rather than requiring every platform to use the same semantic modeling technology.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/487/1*3Hec_TjW_cbgsY-ZBdajmQ.jpeg" /></figure><p>This also opens up an interesting architectural opportunity: maintain business definitions in an open, version-controlled format and generate the representations required by different analytics, BI, and AI platforms.</p><p>The ecosystem is already gaining traction. Snowflake supports native import and export of Apache Ossie definitions, while open-source converters are emerging for dbt MetricFlow, Databricks Metric Views, Power BI, Cube, etc.,and several other semantic platforms. Although interoperability is still evolving and not every feature translates seamlessly, these integrations demonstrate how organizations can increasingly preserve and reuse business definitions across technologies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/797/1*jsfXzHr7Ijfz8RoCIOG_Fg.jpeg" /><figcaption>Apache OSSIE Interoperability</figcaption></figure><blockquote>Business definitions should outlive the technologies used to calculate, visualize, or consume them. Apache Ossie provides an emerging interoperability foundation to make that possible.</blockquote><h3>Putting It All Together: What Happens When We Replace a Processing Engine?</h3><p>Imagine an organization running its analytics workloads on Snowflake that decides to introduce Databricks for additional data processing workloads or migrate some existing workloads.</p><p>In a tightly coupled architecture, this might require copying large volumes of data, recreating tables, rewriting transformation pipelines, rebuilding governance controls, and potentially redefining business metrics.</p><p>But with the interoperable architecture we’ve discussed, the migration looks different:</p><ul><li><strong>Storage:</strong> Data remains in organization-owned cloud storage without requiring wholesale relocation.</li><li><strong>Table formats:</strong> Compatible engines can access the same Apache Iceberg tables, reducing unnecessary duplication.</li><li><strong>Catalogs:</strong> Open interfaces and federation enable engines to discover existing tables without rebuilding the entire metadata ecosystem.</li><li><strong>Transformations:</strong> dbt allows us to reuse development frameworks, tests, and portions of transformation logic, with adjustments for engine-specific SQL.</li><li><strong>Business semantics:</strong> Apache Ossie provides a way to exchange existing semantic definitions across supported platforms rather than recreating them from scratch.</li></ul><p>This doesn’t eliminate migration effort. Security policies, performance optimizations, SQL compatibility, and operational processes still need attention.</p><p>But instead of migrating the entire data ecosystem, we can focus much of the effort on the technology we’re actually changing.</p><p><strong>More importantly, interoperability gives organizations the freedom to choose the right technology for each workload without unnecessarily recreating their existing data ecosystem.</strong></p><p>And that’s the real architectural value of interoperability.</p><h3>Final Thoughts</h3><p>As data leaders, we cannot predict which technologies will define the next generation of data platforms. What we can control is how we architect our platforms today to accommodate those changes tomorrow.</p><p>I believe interoperability should be a fundamental consideration in our technology decisions, not something we think about only when faced with another migration.</p><p>This doesn’t mean avoiding proprietary technologies or compromising on the capabilities we need today. It means being intentional about where we introduce dependencies and where open standards can preserve flexibility.</p><p><strong>The real value of interoperability isn’t simply the ability to use multiple technologies. It’s the freedom to evolve our technology stack without repeatedly rebuilding the investments we’ve already made.</strong></p><h3>Acknowledgments</h3><p>A special thanks to <strong>Amanpreet Kaur</strong> and <strong>Parthiban Jaganathan</strong> for the thoughtful discussions and brainstorming that helped shape and refine some of the architectural ideas shared in this article.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b3acb88d599b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Transformers: The Engine Behind LLMs, Simplified]]></title>
            <link>https://medium.com/@raviscal/understanding-transformers-the-engine-behind-llms-simplified-34053e83bbcc?source=rss-6676b3772ca1------2</link>
            <guid isPermaLink="false">https://medium.com/p/34053e83bbcc</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[llm-transformer]]></category>
            <category><![CDATA[transformer-architecture]]></category>
            <category><![CDATA[generative-ai]]></category>
            <dc:creator><![CDATA[Ravi Bommu]]></dc:creator>
            <pubDate>Tue, 22 Sep 2026 04:14:47 GMT</pubDate>
            <atom:updated>2026-10-28T17:57:33.244Z</atom:updated>
            <content:encoded><![CDATA[<p>Large Language Models have quickly become part of our everyday lives. Most of us use them in some form now, whether for writing, coding, research, learning, or simply answering questions.</p><p>But how many of us actually understand what happens underneath when we type a question and an LLM responds?</p><p>And to be fair, <strong>most people probably do not need to know.</strong> Transformers, attention, embeddings, vectors, Query, Key, Value. It can get complicated very quickly. If an LLM helps you get your work done, there is really no reason you need to understand the mathematics behind it.</p><p>But in conversations with friends and colleagues, I have come across another group. <strong>People who are genuinely curious about how LLMs work, but find most explanations difficult to digest.</strong></p><blockquote><strong>I was one of them.</strong></blockquote><p>I have gone through the original <strong>“Attention Is All You Need”</strong> paper multiple times, along with many articles, videos, and explanations, trying to build a mental model of what a Transformer is actually doing. What I found is that the individual concepts are not necessarily that difficult. The harder part is understanding <strong>how they all fit together. </strong>So that is what I am going to attempt in this article.</p><p>This is not intended to be a mathematical deep dive or another explanation of the Transformer research paper. Instead, I want to build a simple mental model of what happens inside an LLM, and particularly inside the <strong>Transformer</strong>, which is at the heart of how modern LLMs process context and relationships between tokens. In chat GPT (T stands for Transformer)</p><p>My own learning followed that same path, starting with embeddings and positional information, then moving into self-attention, Query, Key, Value, and multi-head attention.</p><h3>Before We Dive In: The Process in 30 Seconds</h3><p>At a very high level, this is what happens when you give an LLM a prompt:</p><ol><li><strong>Tokenize the text</strong><br> The prompt is broken into smaller pieces called tokens.</li><li><strong>Convert tokens into embeddings</strong><br> Each token is converted into a vector of numbers that gives the model a numerical representation it can work with.</li><li><strong>Add positional information</strong><br> The model also needs to know where each token appears in the sequence.</li><li><strong>Pass them through Transformer blocks</strong><br> Self-attention allows each token to determine which other tokens matter to it. Multiple attention heads do this in parallel, and this process is repeated across many Transformer blocks. This is the heart of LLMs.</li><li><strong>Build context-aware representations</strong><br> The token vectors are progressively updated so they now represent not just the token itself, but also its context.</li><li><strong>Predict the next token</strong><br> The final representation is converted into probabilities across the model’s vocabulary. A token is selected, added to the sequence, and the process repeats.</li></ol><p>Let’s start with the whole picture and then peel it apart, one layer at a time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t1Zk5kNKOU4m2-YntCpxqw.png" /><figcaption>Mental model of how Transformer works</figcaption></figure><p>This picture could be daunting at first, but be assured, we will be peeling one layer at a time with the goal of keeping it simple to understand as much as possible.</p><h3>Step 1: It Starts With the Prompt</h3><p>Everything starts with the text we give the LLM, commonly called the <strong>prompt</strong>.</p><p>Let us use this simple example throughout the article: <strong><em>The player could not buy the game because his card was</em></strong></p><p>As humans, we already understand quite a bit from this sentence. Someone is trying to buy a game, a card is involved in the purchase, and many of us would probably guess that the next word could be <strong>“declined.” </strong>But the LLM does not receive that understanding directly. At this point, it simply receives a sequence of text. Its eventual job is straightforward: <strong><em>Given everything I have seen so far, what should come next?</em></strong></p><p>To get there, the first thing it needs to do is convert our text into something a neural network can work with. That starts with <strong>tokenization</strong>.</p><h3>Step 2: Breaking Language Into Tokens</h3><p>An LLM does not process a sentence exactly as we see it. It first breaks the text into smaller pieces called <strong>tokens</strong>. You may have already come across the word <em>token</em> when looking at LLM usage or cost. That is because LLM providers typically measure how much text is processed in <strong>tokens</strong>, not simply in words or characters. The more tokens you send in and receive back, the more computation the model has to perform, which is why token counts are often used for usage limits and pricing.</p><p>For our example:</p><p>The | player | could | not | buy | the | game | because | his | card | was</p><p>For simplicity, we can think of each token as a word here. In reality, a token can also be part of a word, depending on the tokenizer used by the model.So instead of storing every possible word in the vocabulary, the tokenizer can reuse common pieces. A word like:player might be represented as: play | er</p><blockquote>If you are curious about how tokenizers decide these splits, look up Byte-Pair Encoding (BPE). It is one of the foundational techniques used to build token vocabularies</blockquote><p>Once the text has been tokenized, each token is mapped to a <strong>token ID</strong>. Think of this as an identifier for that token in the model’s vocabulary.</p><p>For example:</p><p>The → 482<br> player → 7214<br> buy → 1843<br> game → 963<br> card → 5210</p><p>For a given model and tokenizer, that mapping is fixed. So if card is token 5210, the same token will continue to map to 5210 whenever that tokenizer encounters it. The important thing to understand is that 5210 does not somehow contain the meaning of <strong>card</strong>. It is simply the ID assigned to that token, similar to an index. Different models can use different tokenizers and vocabularies, so the same word may have a different token ID in another model. It may even be split into multiple tokens.</p><p>So now we have converted our sentence from text into a sequence of token IDs.But these IDs still do not tell the model what the tokens <strong>mean</strong>.They are mainly used to look up something much more useful: the token’s <strong>embedding</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lxmiyMuxgcIqAOMEeTXRpA.jpeg" /><figcaption>Words to tokens and mapped with tokenID</figcaption></figure><h3>Understand Embedding:</h3><p>Before we move to positional information, let us first understand what an <strong>embedding</strong> is.</p><p>At the end of tokenization, every token has an ID. For example: card → 5210.But 5210 is only an identifier. It does not contain the meaning of <strong>card</strong>.The model uses that ID to look up a learned numerical representation called an <strong>embedding</strong>.</p><p>For example:</p><p>card → 5210 → [0.21, -0.73, 0.44, 0.18, ...]</p><p>An embedding is simply a <strong>vector</strong>, which is a list of numbers. A simple 3-dimensional vector might look like: [0.2, 0.7, -0.4]</p><p>Each number represents one dimension:</p><p>Dimension 1 → 0.2<br> Dimension 2 → 0.7<br> Dimension 3 → -0.4</p><p>Real models use much larger vectors. For illustration, imagine that our model uses <strong>768 dimensions</strong>: card → [0.21, -0.73, 0.44, 0.18, ........ 768 values] We cannot visualize 768 dimensions, so throughout this article we will occasionally reduce them to two or three dimensions to make the concepts easier to see.</p><h3>But what do those numbers actually represent?</h3><p>This is where embeddings become interesting. Think of an embedding as giving every token an <strong>address in a very large map of meaning</strong>.</p><p>Let us make that map unrealistically simple and give it only two dimensions. Imagine the model has learned these made-up vectors:</p><p>banana → [6,5]<br> cherry → [6,4]<br> apple → [5,5]</p><p>We can treat those two numbers as coordinates on a graph, just like x and y. Notice what happens. <strong>Apple, banana, and cherry end up close to one another.</strong></p><p>Why would that happen?</p><p>During training, the model sees enormous amounts of text. It repeatedly encounters patterns such as:</p><p><em>“I ate a banana”<br> “I ate an apple”<br> “fresh apples and cherries”<br> “banana is a fruit”</em></p><p>Over time, it learns that <strong>apple, banana, and cherry are used in many similar contexts</strong>. Their numerical representations therefore become related as the model learns.</p><p>Now imagine adding:</p><p>car → [1,1]</p><p>A car is used in very different contexts, so its representation may end up much farther away from the fruits.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/569/1*yaUeOmHcqa0odlgBES65uQ.jpeg" /></figure><p>The important idea is not the specific numbers. Those are just made-up numbers to help us visualize the concept.</p><blockquote><strong><em>An embedding converts a token into numbers in such a way that relationships learned from language are encoded in those numbers.</em></strong></blockquote><p>And this is happening in far more than two dimensions. If our embedding has <strong>768 dimensions</strong>, every token gets 768 values:</p><p>apple → [0.12, -0.41, 0.83, ..., 768 values]</p><p>banana → [0.10, -0.38, 0.79, ..., 768 values]</p><p>We cannot draw a 768-dimensional space, but the same basic idea applies. Tokens with related learned usage patterns can occupy related areas of this enormous numerical space. This is the intuition behind the simplified fruit example in my learning notes as well.</p><p>And there is still something missing.</p><p>The embedding gives <strong>apple</strong> a starting representation of <em>apple</em>. It does not yet know whether we mean:</p><blockquote><strong><em>I ate an apple </em></strong>or <strong><em>I bought an Apple phone.</em></strong></blockquote><p>That meaning comes from <strong>context</strong>, which we will get to when we reach <strong>attention. (Hold on to this until we get to Attention)</strong></p><p>Before that, the Transformer first needs to know something much simpler: <strong>where each token appears in the sentence.</strong> That takes us to positional information.</p><h3>Step 3: Giving Tokens Meaning and Position</h3><p>We now have an embedding for every token. Each token has been converted into a vector of numbers that the model can work with.</p><p>But there is still something missing: <strong>the model needs to know where each token appears in the sentence.</strong></p><p>Consider these two sentences:</p><blockquote><strong><em>Dog bites man.</em></strong><em><br> </em><strong><em>Man bites dog.</em></strong></blockquote><p>They contain exactly the same words, but the order completely changes the meaning. In our example:</p><p>The | player | could | not | buy | the | game | because | his | card | was</p><p>we can think of the tokens as occupying positions:</p><p>The → 1<br> player → 2<br> could → 3<br> ...<br> card → 10<br> was → 11</p><p>Why do we need to explicitly provide this?</p><p>Older architectures such as Recurrent Neural Networks processed words one after another, so sequence order was naturally part of the processing. Transformers changed that approach. They can process the tokens in parallel using attention, which is one of the reasons they are so powerful and efficient. But that also means we need another way to tell the model <strong>which token came first, second, third, and so on.</strong></p><p>That is the purpose of <strong>positional encoding</strong>.</p><h3>Adding Position to the Embedding</h3><p>Let us continue with our token <strong>card</strong>. Assume its embedding is a simplified four-dimensional vector:</p><p>card embedding = [0.21, -0.73, 0.44, 0.18]</p><p>The model creates another vector representing its position in the sequence:</p><p>position 10 = [0.54, -0.84, 0.10, 0.99]</p><p>The two vectors are added together:<br> Token Embedding [0.21, -0.73, 0.44, 0.18] + Position [0.54, -0.84, 0.10, 0.99] = [0.75, -1.57, 0.54, 1.17]</p><p>So the representation entering the Transformer now contains information about both:</p><p><strong><em>What is this token?</em></strong><em><br> </em><strong><em>Where does it appear in the sequence?</em></strong></p><p>The same process happens for every token. Your learning notes illustrate this by creating a positional vector with the same number of dimensions as the word embedding and then adding the two together.</p><h3>Optional: How Is the Position Vector Calculated?</h3><p>(<strong>Feel free to skip this portion and move to step 4)<br></strong>In the original Transformer architecture, positional information was created using <strong>sine and cosine waves with different frequencies</strong>.</p><p>The easiest way to think about this is to imagine several waves drawn across the token positions.</p><p>For our simple example:</p><p>the | player | was<br> 1 | 2 | 3</p><p>Now look at <strong>Position 1</strong>. We take the value of each wave at that point.</p><p>For example:</p><ul><li>Wave 1 gives 0.84</li><li>Wave 2 gives 0.54</li><li>Wave 3 gives 0.01</li><li>Wave 4 gives 0.99</li></ul><p>Put those values together:</p><p>Position 1 → [0.84, 0.54, 0.01, 0.99]</p><p>Then we move to <strong>Position 2</strong> and read the values from those same waves again:</p><p>Position 2 → [0.91, -0.42, 0.02, 0.99]</p><p>And again for Position 3:</p><p>Position 3 → [0.14, -0.99, 0.03, 0.99]</p><p>So each position ends up with a different <strong>numerical fingerprint</strong>.</p><p>Why use waves with different frequencies? Because if all the waves moved in exactly the same way, they would not give us enough variation. Some waves change slowly, while others change faster. Together, they create a much richer pattern that helps distinguish one position from another. This is the same idea illustrated, where different sine and cosine waves contribute values to different dimensions of the positional vector. Then we simply add that positional fingerprint to the token embedding:</p><p>Token embedding + Position embedding = Final input representation</p><p>So for <strong>player</strong> at Position 2:</p><p>player embedding = [0.40, -0.10, 0.20]</p><p>position 2 = [0.54, 0.87, -0.76]</p><p>Adding them gives:</p><p>[0.94, 0.77, -0.56]</p><p>Now the vector carries information about both: <strong>what the token is</strong> and <strong>where it appears in the sentence</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mZjeLwxf04xtVJUVJbBIiQ.png" /></figure><h3>Step 4: The Heart of the Transformer: Self-Attention</h3><p>We now have two things for every token:</p><p><strong>1. Its embedding</strong>, which gives the model a numerical representation of the token.<br> <strong>2. Its position</strong>, which tells the model where that token appears in the sequence.</p><p>But something important is still missing: <strong>context</strong>.</p><p>Take our sentence again: <strong><em>The player could not buy the game because his card was</em></strong></p><p>By the time we reach the word <strong>“was”</strong>, not every previous word is equally important. For example:</p><p>card → very relevant<br> buy → relevant<br> game → relevant<br> player → somewhat relevant<br> the → probably much less relevant</p><p>As humans, we do this almost automatically. We connect <strong>card</strong> with <strong>buy</strong>, <strong>game</strong>, and <strong>was</strong>, and that context makes a word like <strong>declined</strong> feel like a very reasonable next token. The Transformer needs a numerical way to do something similar. That is the purpose of <strong>self-attention</strong>.</p><blockquote><strong><em>Self-attention allows each token to look at the other relevant tokens in its available context and determine how much attention to give each one.</em></strong></blockquote><p>In simple terms, it is asking: <strong><em>For this token, which other tokens matter, and how much?</em></strong></p><p>We can describe self-attention in essentially this way: producing weights that represent how relevant tokens are to one another within the context</p><h3>A Simple Example</h3><p>Let us focus on the last token:</p><p><strong>was</strong></p><p>The model might calculate attention weights that look conceptually like this:</p><p>The → 0.02<br> player → 0.05<br> could → 0.03<br> not → 0.06<br> buy → 0.14<br> the → 0.02<br> game → 0.10<br> because → 0.08<br> his → 0.10<br> card → 0.40</p><p>These numbers are just an illustration, but notice what happened.</p><p><strong>“card</strong> “receives much more attention than “<strong>the”</strong>.</p><p>The model is effectively saying: <strong>When I build the representation for “was,” information from “card” is more useful than information from “the.”</strong></p><p>The attention weights are then used to combine information from those tokens and produce a <strong>new representation of “was” that contains context from the sentence</strong>.</p><p>This is the key shift.</p><p>Before attention: was → its starting embedding + position</p><p>After attention: was → representation influenced by card, buy, game, his, ...</p><p>And the same process happens across the tokens in the sequence.</p><h3>This Is Why the Same Word Can Mean Different Things</h3><p>Remember our earlier example:<br> <strong><em>I like an apple </em></strong>vs <strong><em>I like an Apple phone.</em></strong></p><p>The token <strong>apple</strong> may start from a similar initial representation, but attention allows surrounding tokens to influence it. In the first sentence: apple ↔ like and other nearby context can support the fruit meaning. In the second: Apple ↔ phone creates a very different context.</p><p>So one useful way to think about the journey is:</p><p><strong>Embedding gives us a starting meaning.</strong><br> <strong>Position tells us where the token is.</strong><br> <strong>Attention starts making that representation context-aware.</strong></p><h3>One Important Detail for LLMs</h3><p>When an LLM is generating text, it should not be allowed to look into the future.</p><p>So when it is processing: “<strong><em>The player could not buy the game because his card was” </em></strong>the token “<strong>was”</strong> can pay attention to the tokens before it, but it cannot see “<strong>declined”</strong>, because that token has not been generated yet.</p><p>Using everything available so far, the model builds a context-rich representation that will eventually help it predict what comes next. And now we get to the interesting question:</p><blockquote><strong><em>How does the model actually calculate that “card” should receive more attention than “the”?</em></strong></blockquote><p>That is where <strong>Query, Key and Value</strong>, or <strong>Q, K and V</strong>, come in.</p><h3>Step 4a: Inside One Attention Head: Query, Key, and Value</h3><p>We now understand the purpose of self-attention: <strong><em>For each token, determine which other tokens matter and how much.</em></strong></p><p>But how does the model calculate that? This is where <strong>Query, Key, and Value</strong>, usually written as <strong>Q, K, and V</strong>, come in.</p><p>Before we go further, one important clarification:</p><blockquote><strong><em>What we are about to open up is one attention head, not the entire Transformer block.</em></strong></blockquote><p>Several attention heads run in parallel to form <strong>multi-head attention</strong>. Multi-head attention, along with a feed-forward network, residual connections, and normalization, makes up a Transformer block.</p><p>We will first understand what happens inside <strong>one head</strong>.</p><h3>Three Different Views of Every Token</h3><p>At this point, every token already has an input vector containing its embedding and position.</p><p>The attention head transforms each token vector into three new vectors:</p><ul><li><strong>Query</strong></li><li><strong>Key</strong></li><li><strong>Value</strong></li></ul><p>A simple way to think about them is:</p><p><strong>Query : </strong>What information am I looking for?<br><strong>Key: </strong>What kind of information do I contain?<br><strong>Value: </strong>What information should I contribute if I am relevant?</p><blockquote><strong>These are not manually defined descriptions. The model learns how to create these three views during training.</strong></blockquote><p>For every input token, the model creates all three:</p><pre>Token representation<br>        │<br>        ├── Query<br>        ├── Key<br>        └── Value</pre><p>So <strong>card</strong> gets a Query, Key, and Value.<br> <strong>buy</strong> gets a Query, Key, and Value.<br> <strong>was</strong> gets a Query, Key, and Value.</p><p>The Query, Key, and Value vectors are created by multiplying the input representation by three different sets of learned weights.</p><h3>Let Us Focus on “was”</h3><p>Take our sentence: “<strong><em>The player could not buy the game because his card was”</em></strong></p><p>We want to create a context-aware representation for the final token, <strong>was</strong>.</p><p>The model starts with the <strong>Query for “was.” </strong>You can think of it as the token asking: <strong><em>Which information from the sentence will help me understand my role here? </em></strong>The model then compares the Query of <strong>was</strong> with the Key of every token it is allowed to see:</p><pre>Query(was) compared with Key(The)<br>Query(was) compared with Key(player)<br>Query(was) compared with Key(buy)<br>Query(was) compared with Key(game)<br>Query(was) compared with Key(card)<br>Query(was) compared with Key(was)</pre><p>Each comparison produces a <strong>relevance score</strong>. Conceptually, the scores might eventually become attention weights like these:</p><pre>The       → 0.01<br>player    → 0.04<br>could     → 0.03<br>not       → 0.07<br>buy       → 0.14<br>the       → 0.01<br>game      → 0.09<br>because   → 0.08<br>his       → 0.10<br>card      → 0.38<br>was       → 0.05</pre><p>These are made-up values for illustration, but they must add up to 1.</p><blockquote><strong>Add up to 1</strong>, because the model uses a function called <strong>Softmax</strong>, which converts the raw relevance scores into normalized weights that can be thought of as how the model distributes its attention across the tokens. For example, a weight of 0.38 means that, in this simplified example, about <strong>38% of the attention for “was”</strong> is being assigned to <strong>“card.”</strong></blockquote><p>Notice that <strong>card</strong> has received the largest weight. Words such as <strong>buy</strong>, <strong>his</strong>, <strong>game</strong>, and <strong>because</strong> also contribute, while <strong>the</strong> contributes very little.</p><p>The Query and Key vectors therefore answer:</p><blockquote><strong><em>How relevant is each token to the token we are currently processing?</em></strong></blockquote><h3>What Does the Value Do?</h3><p>Query and Key determine <strong>which tokens matter</strong>. But they do not yet carry the information that will be brought forward. That is the job of the <strong>Value</strong> vectors. The model multiplies each token’s Value by its attention weight:</p><pre>0.01 × Value(The)<br>0.04 × Value(player)<br>0.14 × Value(buy)<br>0.09 × Value(game)<br>0.38 × Value(card)<br>...</pre><p>It then adds all of those weighted Values together.</p><p>Conceptually:</p><pre>Output for “was”<br>=<br>0.01 × V(The)<br>+ 0.04 × V(player)<br>+ 0.03 × V(could)<br>+ ...<br>+ 0.38 × V(card)<br>+ 0.05 × V(was)</pre><p>The result is a <strong>new vector for “was.” </strong>Before attention, the vector represented mainly: <strong>”was” + its position</strong></p><p>After attention, it contains information influenced by: <br><strong>“card” + “buy” + “game” + “his” + “because” + the rest of the context</strong></p><p>That is how the representation becomes <strong>context-aware</strong>.</p><h3>Putting One Attention Head Together</h3><p>What happened inside this head can now be summarized in four steps:</p><ol><li>Every token creates a <strong>Query, Key, and Value</strong> vector.</li><li>The Query for <strong>“was”</strong> is compared with the Keys of all visible tokens.</li><li>Those comparisons become attention weights that show how much each token matters.</li><li>The weights are applied to the Value vectors and combined into a new, context-aware representation for <strong>“was.”</strong></li><li>In one line: <strong><em>Queries and Keys decide where to look. Values provide the information that is brought forward.</em></strong></li></ol><h3>Optional: How Are the Attention Weights Calculated? (skip to step 4b)</h3><p>At a high level, the model:</p><ol><li>Calculates a similarity score between the Query and each Key using a dot product.</li><li>Scales those scores so they do not become too large.</li><li>Uses <strong>Softmax</strong> to convert them into positive weights that add up to 1.</li></ol><p>Conceptually:</p><pre>Query × Keys<br>      ↓<br>Similarity scores<br>      ↓<br>Scale<br>      ↓<br>Softmax<br>      ↓<br>Attention weights</pre><p>The full equation is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/402/1*zxck00eTM7MqlE_qEa-DxQ.png" /></figure><p>But the equation is simply expressing what we already explained:</p><blockquote><em>Compare the Query with the Keys, turn the results into weights, and use those weights to combine the Values.</em></blockquote><h3>Step 4b: Why Do We Need Multiple Attention Heads?</h3><p><em>One attention head gives the model one learned way of looking at the sentence. But language contains many relationships at the same time. That is why Transformers run several attention heads in parallel.</em></p><p>Take our example again:<strong><em>The player could not buy the game because his card was</em></strong></p><p>To understand what might come after <strong>“was,”</strong> the model may need to consider several things:</p><ul><li><strong>card</strong> is closely related to <strong>was</strong></li><li><strong>buy</strong> and <strong>game</strong> establish that this is a purchase</li><li><strong>his</strong> connects the card to the player</li><li><strong>not</strong> tells us that the purchase was unsuccessful</li><li><strong>because</strong> signals that an explanation is coming</li></ul><p>Trying to capture all of these relationships through only one attention pattern can be limiting.</p><p>So instead of running one attention head, the Transformer runs <strong>multiple attention heads in parallel</strong>. Each head receives the same input token representations, but each has its own learned Query, Key, and Value transformations. This allows every head to calculate a different attention pattern over the same sentence. Your learning notes introduce multi-head attention in the same way, as multiple self-attention calculations running in parallel and later being combined.</p><h3>Different Heads Can Notice Different Relationships</h3><p>For the token <strong>“was,”</strong> one head might produce an attention pattern like:</p><pre>card → high attention<br>buy  → medium attention<br>game → medium attention<br>the  → low attention</pre><p>Another head might focus more on:</p><pre>not     → high attention<br>because → high attention<br>was     → medium attention</pre><p>And another might emphasize relationships such as:</p><pre>his    → card<br>player → his<br>buy    → game</pre><p>These examples are only meant to build intuition. The model is not explicitly told:</p><blockquote><em>“Head 1 should find purchase relationships.”<br> “Head 2 should find grammatical relationships.”<br> “Head 3 should connect pronouns.”</em></blockquote><p>The heads learn their own useful patterns during training, and those patterns do not always map neatly to human-readable roles.</p><p>A simple way to think about it is:</p><blockquote><strong><em>Each head looks at the same sentence through a different learned lens.</em></strong></blockquote><h3>The Same Token Gets Different Q, K, and V Vectors in Each Head</h3><p>Remember that every attention head creates Query, Key, and Value vectors.</p><p>Because each head has its own learned weights, the same input token produces different Q, K, and V vectors in different heads.</p><p>Conceptually:</p><pre>Input representation for “was”<br>            │<br>            ├── Head 1 → Q₁, K₁, V₁<br>            ├── Head 2 → Q₂, K₂, V₂<br>            └── Head 3 → Q₃, K₃, V₃</pre><p>Each head then independently performs the process we already discussed:</p><pre>Compare Queries with Keys<br>            ↓<br>Calculate attention weights<br>            ↓<br>Combine the Values<br>            ↓<br>Produce one context-aware output</pre><p>So we might get:</p><pre>Head 1 → Output 1<br>Head 2 → Output 2<br>Head 3 → Output 3</pre><h3>How Are the Head Outputs Combined?</h3><p>The outputs from all the heads are placed together, or <strong>concatenated</strong>, and then passed through another learned linear transformation.</p><pre>Head 1 output ─┐<br>Head 2 output ─┼── Concatenate ── Linear projection ── Combined output<br>Head 3 output ─┘</pre><p>This final step mixes the information discovered by the different heads into one representation that can move to the next part of the Transformer block. Your notes show this same progression from multiple heads to concatenation and a final projection.</p><p>The important point is not that one head is chosen over the others. The model combines information from all of them.</p><h3>Putting It Together</h3><p>For one token such as <strong>“was,”</strong> the journey now looks like this:</p><pre>Input representation for “was”<br>            ↓<br>Multiple attention heads run in parallel<br>            ↓<br>Each head creates its own Q, K, and V<br>            ↓<br>Each head produces a different attention pattern<br>            ↓<br>Each head creates a context-aware output<br>            ↓<br>All head outputs are combined<br>            ↓<br>Richer representation for “was”</pre><p>So the difference between self-attention and multi-head attention is:</p><blockquote><strong><em>Self-attention calculates which tokens matter. Multi-head attention allows the model to calculate several different views of which tokens matter at the same time.</em></strong></blockquote><p>This entire multi-head attention process is one important component inside a <strong>Transformer block</strong>. After it, the representation moves through residual connections, normalization, and a feed-forward network before being passed to the next Transformer block.</p><p>At this point, we have covered the most important part of the Transformer block: <strong>multi-head attention</strong>.</p><p>After the different attention heads finish their work, their outputs are combined and passed through a few additional steps before moving to the next block.</p><p>At a very high level:</p><pre>Multi-Head Attention<br>        ↓<br>Combine the head outputs<br>        ↓<br>Residual Connection + Normalization<br>        ↓<br>Feed-Forward Network<br>        ↓<br>Residual Connection + Normalization<br>        ↓<br>Output of the Transformer Block</pre><p>The <strong>feed-forward network</strong> further processes the representation of each token after attention has brought in the relevant context. The <strong>residual connections</strong> help preserve useful information as the representation moves through the block, while <strong>normalization</strong> helps keep the values stable during processing. We do not need to go much deeper into these pieces for this article. The important thing to remember is:</p><blockquote><strong><em>Attention helps each token gather context from other tokens, and the rest of the block further refines that representation</em></strong></blockquote><p>And this does not happen just once. The output of one Transformer block becomes the input to the next:</p><pre>Transformer Block 1<br>        ↓<br>Transformer Block 2<br>        ↓<br>Transformer Block 3<br>        ↓<br>       ...<br>        ↓<br>Transformer Block N</pre><p>With every block, the token representations can become progressively more <strong>context-aware and refined</strong>. That is what the <strong>“Transformer Blocks × N”</strong> box in our opening picture represents.</p><h3>Step 5: Contextualized Representation?</h3><p>What changed in the Transfomer block?</p><p>We started with a simple embedding for each token. For example, <strong>“was”</strong> had a vector that represented the token itself, plus its position in the sentence.</p><p>But after passing through all those Transformer blocks, that vector is no longer just about the word <strong>“was.” </strong>It has now absorbed information from the context around it. In our sentence: “<strong><em>The player could not buy the game because his card was”</em></strong></p><p>the representation for <strong>“was”</strong> is now influenced by words such as:</p><p>card<br> buy<br> game<br> because<br> not</p><p>So conceptually:</p><pre>Before Transformer:<br>was → [basic meaning + position]</pre><pre>After Transformer:<br>was → [basic meaning + position + context from the sentence]</pre><p>That is what we mean by a <strong>contextualized representation</strong>.</p><p>And this matters because the model is no longer trying to predict the next token based on the word <strong>“was”</strong> alone.</p><p>It is predicting based on something closer to: <strong><em>“was” in the context of a failed game purchase involving a card. </em></strong>That richer representation is what gets used in the next step to calculate the probabilities for the next token.</p><p>So the key takeaway is:</p><blockquote><strong><em>The Transformer takes simple token representations and turns them into context-aware representations. Those context-aware representations are what make useful next-token prediction possible.</em></strong></blockquote><p>This is the bridge from <strong>understanding context</strong> to <strong>predicting “declined.”</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tGuGlNnDL7nrnk68BOOJzQ.png" /></figure><h3>Step 6: Turning Context Into Next-Token Probabilities</h3><p>Now the model has a rich representation of everything it has seen so far.</p><p>The next question is: <strong><em>What token should come next?</em></strong></p><p>The model converts the final representation into a score for every token in its vocabulary.</p><p>Conceptually, it might produce something like:</p><p>declined → 0.42<br> charged → 0.14<br> invalid → 0.09<br> blocked → 0.06<br> ...</p><p>These probabilities are calculated across the model’s <strong>entire vocabulary</strong>, not just the few words shown here.</p><p>And just like the attention weights we saw earlier, a <strong>Softmax</strong> function is used to turn the raw scores into probabilities that add up to 1.</p><p>So the model is effectively saying: <em>Given everything I have seen so far, </em><strong><em>“declined” appears to be the most likely next token.</em></strong></p><h3>Step 7: Choose the Token and Repeat</h3><p>In our example, <strong>“declined”</strong> has the highest probability, so let us assume the model chooses it.</p><p>Our sequence now becomes: <strong><em>The player could not buy the game because his card was declined. </em></strong>But generation does not stop there. The newly generated token is added back to the sequence:</p><pre>The player could not buy the game because his card was declined</pre><p>And the whole process happens again.</p><pre>Tokenize<br>   ↓<br>Embeddings + Position<br>   ↓<br>Transformer Blocks<br>   ↓<br>Contextualized Representations<br>   ↓<br>Next-Token Probabilities<br>   ↓<br>Choose Next Token<br>   ↓<br>Add It to the Sequence<br>   ↓<br>Repeat</pre><p>One token at a time, the model keeps extending the text.</p><p>And that brings us back to the picture we started with.</p><blockquote><strong><em>An LLM may look like it is generating an entire answer at once, but underneath, it is repeatedly predicting one token, adding it to the context, and doing it all over again.</em></strong></blockquote><h3>Conclusion: Putting It All Together</h3><p>We started with a simple prompt: “<strong><em>The player could not buy the game because his card was”</em></strong></p><p>What looks like a simple sentence goes through quite a journey inside the model.</p><p>The text is first broken into <strong>tokens</strong>. Those tokens are converted into <strong>embeddings</strong>, giving the model numerical representations it can work with. Positional information is then added so the model knows where each token appears in the sequence.</p><p>From there, the Transformer does the interesting part.</p><p>Through <strong>self-attention</strong>, each token can determine which other tokens are relevant to it. Query, Key, and Value provide the mechanism for calculating those relationships. Multiple attention heads do this in parallel, allowing the model to capture different relationships in the same sentence.</p><p>And this happens not just once, but across <strong>many Transformer blocks</strong>.</p><p>By the time the tokens reach the other side, their vectors are no longer just representations of individual words. They have become <strong>context-aware representations</strong>, influenced by the other relevant tokens in the sequence.</p><p>So when the model reaches: <strong><em>…his card was</em></strong></p><p>it is not predicting the next token based only on the word <strong>“was.” </strong>It is using a representation that now carries information about <strong>card, buy, game, not, because</strong>, and the rest of the context. That representation is then converted into probabilities across the model’s vocabulary:</p><p>declined → 0.42<br> charged → 0.14<br> blocked → 0.06<br> ...</p><p>A token is selected, added to the sequence, and the process repeats. That is how an LLM generates text, <strong>one token at a time</strong>.</p><h3>Closing Note</h3><p>There is obviously much more happening inside modern LLMs than what we covered here. We simplified several concepts, used very small vectors, and focused primarily on the Transformer and attention mechanisms. Modern architectures also vary in how they implement positional information, attention, normalization, and other parts of the model.</p><p>But my goal was not to explain every mathematical detail. It was to build a mental model that makes the Transformer a little less mysterious. If there is one idea I would take away from this article, it is this:</p><blockquote><strong><em>A Transformer takes simple numerical representations of tokens and progressively turns them into representations that carry context. Those contextual representations are what allow the model to make a much better prediction about what should come next.</em></strong></blockquote><p>Once that idea clicks, concepts like embeddings, attention, Query-Key-Value, multi-head attention, and even the original <strong>“Attention Is All You Need”</strong> architecture start becoming much easier to connect.</p><p>And perhaps the picture we started with does not look quite as daunting anymore.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=34053e83bbcc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Enterprise Context: A Layered Approach for Agentic AI]]></title>
            <link>https://medium.com/@raviscal/enterprise-context-a-layered-approach-for-agentic-ai-56b18465da90?source=rss-6676b3772ca1------2</link>
            <guid isPermaLink="false">https://medium.com/p/56b18465da90</guid>
            <category><![CDATA[ai-agent-context]]></category>
            <category><![CDATA[agentic-workflow]]></category>
            <category><![CDATA[enterprise-context]]></category>
            <category><![CDATA[context-layer]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <dc:creator><![CDATA[Ravi Bommu]]></dc:creator>
            <pubDate>Tue, 18 Aug 2026 04:13:40 GMT</pubDate>
            <atom:updated>2026-08-18T05:49:20.719Z</atom:updated>
            <content:encoded><![CDATA[<p>Large language models are remarkably good at answering general questions because they have been trained on a vast amount of public information. Ask an LLM what supply chain optimization means, and it can give you a reasonable answer. But ask why your company delivered fewer orders than expected yesterday, and the problem changes completely. The model does not inherently know how your business defines a successful delivery, which fulfillment centers handled those orders, whether inventory was available, whether a carrier was delayed, or whether severe weather disrupted a route. <strong>That missing business-specific information is context.</strong> Context is what turns a model’s general reasoning capability into reasoning that is relevant to your business.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HqD14v_pfx3x96YeUSc0JA.jpeg" /><figcaption>LLM cannot answer without your business context</figcaption></figure><h4>Today, Users provide the Context</h4><p>This problem is not new. We are already solving it today by supplying context to an LLM through documents, spreadsheets, messages, and other information in the conversation.</p><p>When someone asks an AI assistant a business-specific question, they usually do more than type the question. They attach a document, upload a spreadsheet, paste information from another system, add a policy, or explain some business logic that the model would otherwise not know. If the response is not quite right, they add more information and try again. The important part is what is happening behind the scenes.</p><p>The user already knows where the relevant information lives. They know which document matters, which data source to trust, what a particular business term means, and what additional information the model needs to answer the question.</p><blockquote>In other words, <strong>the user is doing the context discovery and assembly for the model. In this case, the human is effectively the context layer.</strong></blockquote><p>For a human-driven interaction, this works surprisingly well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EBIwLvZPWMq9MVM2TdfrvA.jpeg" /></figure><p>The challenge appears when we move from a user interacting with one model to a swarm of agents expected to reason, collaborate, make decisions, and execute workflows on their own.</p><h3>Agentic Workflows Change the Context Problem</h3><p>Today, the user knows which files to upload, which report to reference, and what business rules to explain. In an agentic workflow, there may be multiple agents reasoning and handing work to each other. We cannot expect a human to continuously find and reassemble the right context at every step.</p><p>What works when a human is sitting in front of an AI assistant does not work when agents are expected to operate on their own.</p><blockquote><em>The context has to become available to agents themselves consistently, securely, and on demand. This is where </em><strong><em>Enterprise Context </em></strong><em>becomes important.</em></blockquote><h3>So, What Is Enterprise Context?</h3><p>Enterprise Context is the business knowledge an agent can discover and use to understand and reason about enterprise-specific tasks. That knowledge already exists across the organization, in tables, data models, Confluence pages and SharePoint sites, data catalogs and lineage tools, policies and SOPs, business rules, service inventories, incident-management systems, observability platforms, and other operational systems.</p><blockquote>In most enterprises, no single system contains the complete context. The knowledge an agent needs is distributed across many of them.</blockquote><p>The goal is <strong>not to copy all of that into one giant knowledge base</strong>. Instead, Enterprise Context should make this distributed knowledge <strong>connected, governed, and consumable by agents</strong>, while allowing the authoritative information to remain where it belongs. When an agent receives a task, it should be able to determine what it needs, retrieve the relevant context from these sources, and compose that context as part of its reasoning.</p><blockquote>Enterprise Context is the broader set of enterprise knowledge made discoverable and usable by agents.</blockquote><p>When an agent receives a task, it should be able to determine what information it needs, retrieve the relevant context from these sources, and assemble it as part of its reasoning. For a specific task, the agent selects only the portion it needs. That becomes the <strong>task-relevant context</strong> for that question, decision, or action.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tzXa8Vq4DCbdXaC57zG1lw.jpeg" /><figcaption>Fragmented systems to connected context</figcaption></figure><h3>Enterprise Context Is Not One Giant Repository</h3><p>Once we understand that enterprise context is distributed, the next mistake is to treat it as one homogeneous knowledge layer. It is not. Different kinds of context answer different kinds of questions, come from different systems, and need different governance and serving patterns.</p><blockquote>A metric definition is very different from a policy document. A data lineage tells us something different from a service incident. A product hierarchy serves a different purpose from the current health of a fulfillment system.</blockquote><p>A more practical way to think about Enterprise Context is as a set of <strong>peer context layers</strong>. These layers are not sequential, and an agent does not need all of them for every task. Depending on what it is trying to understand, decide, or do, it may use one layer, several layers, or sometimes all of them.</p><p>I currently think about Enterprise Context in five layers:</p><p><strong>1. Structured Semantic Context</strong><br> <em>What does the business data mean, and how should it be calculated?</em></p><p><strong>2. Asset, Provenance &amp; Trust Context</strong><br> <em>Where did this information come from, and can I trust it?</em></p><p><strong>3. Business Knowledge &amp; Process Context</strong><br> <em>What business rules, policies, and processes apply?</em></p><p><strong>4. Ontology, Taxonomy &amp; Relationships</strong><br> <em>How are business and service concepts organized and connected?</em></p><pre>Model only the relationships that add reasoning value; <br>don&#39;t graph what semantic context layer already solves well.</pre><p><strong>5. Runtime &amp; Service signals Context</strong><br> <em>What is happening in the operating environment right now?</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l1UWZcqTKGvqU-AWmfPu3Q.png" /></figure><h3>A Common Trap: More Context Is Not Always Better</h3><p>Having five context layers does not mean loading all five into the model for every task. Context windows are finite, and flooding an agent with every available metric, document, relationship, policy, and operational signal can make reasoning worse, not better.</p><p>The agent should retrieve context progressively based on the task, starting with what it needs, and pulling additional context only when the reasoning requires it.</p><blockquote><strong><em>Enterprise Context should make knowledge available. The agent should decide what belongs in the context window.</em></strong></blockquote><h3>Seeing the Layers in Action</h3><p>To make this concrete, let’s use one simple retail example throughout:<br><strong><em>Why were customer deliveries lower than expected yesterday?</em></strong></p><h3>1. Structured Semantic Context</h3><p>This is the context around structured business data: metrics, dimensions, business definitions, relationships, filters, and the logic required to calculate something consistently. It helps the agent answer:</p><blockquote><strong><em>What happened?</em></strong></blockquote><p>For our delivery question, the semantic context may allow the agent to determine:</p><blockquote><em>Deliveries were 12% below plan yesterday, with most of the shortfall concentrated around two fulfillment centers.</em></blockquote><p>The important part is not only just generating SQL. The agent understands what a “delivery” means to the business, how the metric is calculated, which dimensions are valid, and which tables and columns (data) should be used.</p><h3>2. Asset, Provenance &amp; Trust Context</h3><p>Knowing the number is not enough. The agent also needs to know whether that number is trustworthy. This layer provides context such as lineage, authoritative sources, ownership, certification, expected freshness, and quality. It helps answer:</p><blockquote><strong><em>Can I trust this, and where did it come from?</em></strong></blockquote><p>For example:</p><blockquote><em>The delivery metric is certified and traces back to order, fulfillment-center, and carrier data. Its freshness SLA is 30 minutes.</em></blockquote><p>Notice the distinction here: this layer tells us <strong>what the freshness expectation is and where the data comes from</strong>. Whether the supporting pipeline is actually healthy right now belongs somewhere else.</p><h3>3. Business Knowledge &amp; Process Context</h3><p>Not everything about how a company operates is represented in a table.</p><p>A lot of important business knowledge lives in Confluence pages, SharePoint sites, policy documents, SOPs, operating procedures, decision guides, and sometimes simply in documentation created by business teams. This layer helps answer:</p><blockquote><strong><em>What business rules, policies, or processes apply?</em></strong></blockquote><p>In our example, the agent might discover:</p><blockquote><em>Severe-weather procedures allow shipments to be rerouted through alternate carriers, and customer delivery commitments may be adjusted when specific disruption thresholds are met.</em></blockquote><p>This gives the agent context about <strong>how the business is expected to respond</strong>, not just what the data says.</p><h3>4. Ontology, Taxonomy &amp; Relationships</h3><p>Some questions require the agent to understand how business concepts, organizational structures, and services relate to one another. This may include business taxonomies and hierarchies, but also relationships across entities and services. It helps answer:</p><blockquote><strong><em>How is this connected?</em></strong></blockquote><p>For the delivery example, the relationship model might tell the agent:</p><p>Business entity relationship:<br>Order → Fulfilled by → Fulfillment Center → Handed to → Carrier → Serves → Delivery Region</p><p>Service relationship:<br>Fulfillment Process → Depends on → Carrier Integration Service<br>Fulfillment Center → Depends on → Warehouse Management Service</p><blockquote>The ontology may define relationships such as above, allowing the agent to understand how the affected orders, facilities, carriers, and regions are connected.</blockquote><pre>One thing to be careful about here is duplication. <br>Many relationships already exist in structured data and can be resolved <br>through joins in the semantic layer. Do not recreate relational data in a graph just because the ontology can <br>represent it; that adds duplication, operational cost, and potentially more <br>context for the agent to process. <br><br>The ontology layer should focus on relationships that add reusable business<br>meaning taxonomies, hierarchies, cross-domain mappings, service dependencies,<br>or relationships that are difficult to infer repeatedly from the underlying <br>data.</pre><h3>5. Runtime &amp; Service Context</h3><p>The first four layers can tell us what the business means, what we can trust, how things should work, and how they are connected. But the agent may still need to know:</p><blockquote><strong><em>What is happening right now?</em></strong></blockquote><p>Runtime &amp; Service Context provides the current operating state: service health, active incidents, telemetry, recent deployments or configuration changes, affected services or regions, and recovery state. External live signals, such as severe weather, may also be retrieved when they are relevant to the task.</p><p>For our example, the agent might find:</p><blockquote><em>Inventory and warehouse systems are healthy. One carrier integration is experiencing elevated errors, and severe weather is disrupting routes around one of the affected fulfillment centers.</em></blockquote><h3>Putting the Context Together</h3><p>No single layer was enough to answer the original question.</p><p>The semantic layer showed <strong>where the delivery shortfall occurred</strong>. Provenance established that the data was <strong>authoritative and current</strong>. Business knowledge explained <strong>how disruptions should be handled</strong>. Relationships showed <strong>which fulfillment centers, carriers, routes, and services were connected to the affected orders</strong>. Runtime context identified <strong>what was actually happening in the operating environment</strong>.</p><p>An agent can now produce a much more useful answer:</p><blockquote><strong><em>Deliveries were 12% below plan, with most of the shortfall concentrated around two fulfillment centers. Inventory and warehouse systems were operating normally, but one carrier integration experienced elevated errors while severe weather disrupted routes around one of the affected centers. The evidence points to transportation and carrier disruption, rather than inventory availability, as the primary driver.</em></strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PMGD1dSa9rGn0V3dOFeBSA.png" /></figure><p>This is the value of thinking about Enterprise Context as multiple layers.</p><p><strong>Each layer contributes a different kind of evidence. The agent decides which context it needs and composes those pieces into an answer for the task in front of it.</strong></p><h3>How Could These Context Layers Be Implemented?</h3><p>If you are wondering how these five layers can be implemented technically, there is no single stack. Each can use different platforms and patterns depending on what already exists in the enterprise.</p><ul><li><strong>Structured Semantic Context:</strong> A semantic layer combined with Text-to-SQL or natural-language analytics that exposes governed metrics, dimensions, relationships, and business definitions to agents. Modern data platforms increasingly provide these capabilities natively, while standards such as Open Semantic Interchange aim to improve portability across tools.</li><li><strong>Asset, Provenance &amp; Trust Context: </strong>Data catalogs and governance platforms that expose ownership, lineage, certification, freshness, quality, and other trust metadata.</li><li><strong>Business Knowledge &amp; Process Context: </strong>Retrieval Augmented Generation (<a href="https://medium.com/@raviscal/start-to-finish-retrieval-augmented-generation-rag-in-snowflake-using-vector-search-and-cortex-69a951e76edd">RAG</a> ) over enterprise knowledge sources such as Confluence, SharePoint, policies, SOPs, and process documentation, typically using search/vector retrieval.</li><li><strong>Ontology, Taxonomy &amp; Relationships: </strong>A graph database can be one implementation option where graph traversal adds value, but relationships that already work well in relational or semantic models do not need to be duplicated.</li><li><strong>Runtime &amp; Service Context: </strong>Live integrations with observability, incident-management, telemetry, and event systems. Agents can retrieve current service health, incidents, deployments, alerts, and recovery state through APIs, tools, or event streams rather than replicating everything into another database.</li></ul><blockquote><strong>Enterprise Context does not require one technology stack. Each layer should use the system best suited to that type of context, while agents discover and retrieve what they need at runtime.</strong></blockquote><h3>Enterprise Context Becomes Shared Infrastructure</h3><p>Today, users often provide the missing context themselves ,the right document, data, or business rule. That works for human-driven AI, but it does not scale to agentic workflows.</p><p>Agents need to discover and use enterprise knowledge on their own. This does not require one giant knowledge base or graph. It requires making existing knowledge <strong>connected, governed, discoverable, and consumable by agents</strong>. The layered approach provides a practical way to do this, allowing agents to retrieve and compose only the context needed for each task.</p><blockquote><strong><em>The model provides the reasoning. Enterprise Context provides the business understanding that makes that reasoning useful.</em></strong></blockquote><p>As agentic systems grow, Enterprise Context can become shared infrastructure<strong>, continuously improved and reused across agents, applications, and workflows.</strong></p><h3>A Few Practical Principles Before We Close</h3><p>Before we close, there are a few practical principles worth keeping in mind. Enterprise Context is still an evolving space, and there will be different ways to implement it. But regardless of the technology choices, I think these principles can help keep the architecture practical and avoid creating another complex enterprise platform.</p><ul><li><strong>Avoid duplicating context across layers.</strong> Keep governed analytical relationships and calculations in the semantic layer. Use ontology and taxonomy only when they add business hierarchy or relationship meaning that is not already represented well elsewhere.</li><li><strong>More context is not always better.</strong> Start with the minimum needed for the task and retrieve more only when required. Avoid loading entire semantic models, document sets, or graphs into the context window upfront.</li><li><strong>Use the right source of truth.</strong> When multiple sources look relevant, the agent needs signals such as certification, freshness, version, and source to determine which one is current and trusted.</li><li><strong>Access must follow the user and the task.</strong> Agents should only retrieve context that the user, service, or workflow they represent is authorized to access, and those controls must persist across multi-agent handoffs. I covered this in more detail in <a href="https://medium.com/@raviscal/id-jag-the-missing-piece-in-mcp-enterprise-authorization-4a4f835336a2"><strong>ID-JAG: The Missing Piece in MCP Enterprise Authorization</strong></a>.</li><li><strong>Learning should improve the source, not create another silo.</strong> Validated feedback should update the context asset that owns the knowledge — a metric correction should improve the semantic model; a process correction should improve the process knowledge.</li></ul><p><strong>If this helped you think differently about Enterprise Context, give it a clap and share it with someone working on agentic AI.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=56b18465da90" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ID-JAG: The Missing Piece in MCP Enterprise Authorization]]></title>
            <link>https://medium.com/@raviscal/id-jag-the-missing-piece-in-mcp-enterprise-authorization-4a4f835336a2?source=rss-6676b3772ca1------2</link>
            <guid isPermaLink="false">https://medium.com/p/4a4f835336a2</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[oauth]]></category>
            <category><![CDATA[enterprise-ai]]></category>
            <category><![CDATA[agent-identity]]></category>
            <category><![CDATA[mcps]]></category>
            <dc:creator><![CDATA[Ravi Bommu]]></dc:creator>
            <pubDate>Sat, 27 Jun 2026 19:47:03 GMT</pubDate>
            <atom:updated>2026-06-29T22:14:03.795Z</atom:updated>
            <content:encoded><![CDATA[<p>What is this article about?<br>MCP gives AI clients a standard way to call tools. ID-JAG gives enterprises a standard way to authorize those calls on behalf of real users.</p><p>Everyone is racing to adopt MCP because it gives AI clients a standard way to call enterprise tools like Jira, GitHub, Confluence, ServiceNow, and internal APIs. That is powerful. But once these tools need to act on behalf of real users, a harder problem appears :</p><blockquote><strong>How does the enterprise decide whether this AI client is allowed to act for this user, against this MCP server, under the right identity and access policy?</strong></blockquote><p>In this article, we’ll build up from authentication and authorization basics, explain the normal OAuth flow (in case you are not familair with this), show why MCP introduces new complexity and MCP gateway solves some of it, and then walk through how <strong>ID-JAG</strong> and Cross App Access help solve the missing enterprise authorization handoff.</p><h3>AuthN vs AuthZ</h3><p>Before we get into MCP, OAuth, or ID-JAG, it helps to separate two ideas that often get bundled together as “auth”: <strong>authentication</strong> and <strong>authorization</strong>.</p><p><strong>Authentication</strong>, often shortened to <strong>AuthN</strong>, answers:</p><blockquote><em>Who is this person? </em>In simple terms:<br><em>This is Ravi.</em></blockquote><p><strong>Authorization</strong>, often shortened to <strong>AuthZ</strong>, answers:</p><blockquote><em>What is this person allowed to do? </em>In simple terms:<br><em>Ravi can create a Jira ticket.</em></blockquote><p>Enterprise systems need both.</p><p>Knowing that the user is Ravi is only the first step. The system also needs to know what Ravi is allowed to do. Can Ravi read this Confluence page? Can Ravi create a Jira ticket? Can Ravi review a GitHub pull request? Can Ravi call this internal API?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xODgnaOZXPtqJnddJHHMcw.png" /></figure><p>That distinction becomes even more important with MCP. When an AI client calls an MCP server, the enterprise still has to answer the same two questions:</p><pre>Authentication:<br>Is this really Ravi?<br><br>Authorization:<br>Is Ravi allowed to perform this action through this MCP server?</pre><blockquote>But enterprise AI workflows add one more question</blockquote><pre>Delegation:<br>Is this AI client allowed to act on Ravi’s behalf?</pre><p>That third question is where things get interesting. MCP is not just another login flow. It introduces a <strong>delegation problem</strong>: How does an AI client or IDE is trying to use an enterprise tool like Github through an MCP server, while still respecting the real user’s identity, permissions, and company policy?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Wboqi9CopJxeXtQ9EuAZGA.png" /></figure><p><strong>How Normal OAuth Works</strong></p><p>It is worth quickly reviewing how a normal OAuth flow works.</p><p>If you already know OAuth, think of this as a quick refresher. If you are newer to OAuth, this section gives you the basic mental model we need for the rest of the article.</p><p>Imagine a user is using an application that wants to access Jira on their behalf. For example, the app may want to create Jira issues, read project data, or update tickets.</p><p>The flow usually looks something like this:</p><pre>The user opens an app that wants Jira access<br>  → The app redirects the user to Atlassian’s authorization server<br>  → Atlassian sends the user to Okta for enterprise SSO if needed<br>  → Okta authenticates the user<br>  → Atlassian now knows who the user is<br>  → Atlassian redirects back to the app’s callback URL with an authorization code<br>  → The app exchanges that code with Atlassian’s authorization server<br>  → Atlassian returns an access token for Jira APIs<br>  → The app uses that access token to call Jira APIs on behalf of the user</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5pUgLGomCgLk-2RiVp0_Hg.png" /></figure><p>Key distinctive roles:</p><ol><li><strong>Okta is acting as the enterprise identity provider. </strong>Its job is to authenticate the user.</li><li><strong>Atlassian’s authorization server is responsible for Jira access</strong>. Once the user is authenticated, Atlassian decides what access token to issue based on the user’s Jira permissions, the scopes requested by the app, and any consent or admin policy that applies.</li><li><strong>The authorization code is not the final credential</strong>. It is a short-lived code sent back to the app’s pre-registered callback URL. The app then exchanges that code with Atlassian’s authorization server, specifically its token endpoint, for an <strong>access token</strong>.</li></ol><blockquote>That access token is what the app uses to call Jira APIs. It represents the permission granted to the app to act within the user’s Jira access.</blockquote><p>When the app later calls Jira APIs with that token, Jira still enforces what the user is actually allowed to do.</p><p>This model works well when one application is integrating with one resource system. But AI agents accessing enterprise tools through MCP changes the shape of the problem.</p><h3><strong>Why Not Service Accounts?</strong></h3><p>Before, we see how MCP changes shape of the problem, lets answer the service account question. In traditional enterprise integrations, teams use service accounts in production. So, you might be thinking why this complexity, why not use a service account?</p><p>AI agents are different because they often operate in user-driven workflows. The same AI client may serve many users. One user may ask the agent to take an action, another user may approve it through a human-in-the-loop step, and the action may touch systems with different permissions. Hence, we provide agents access to different tools via MCP to be able to do multiple tasks based on the requirements dynamically (probabilistic). So, a generic service account is not enough.</p><p>If everything runs through a broad service account, those answers become blurry. The system may know that “the agent” did something, but not clearly which user’s authority was used, whether the right approval happened, or whether the action respected the user’s actual permissions.</p><blockquote><strong>That is why enterprise AI needs user-aware delegation.</strong></blockquote><p>The goal is not just to let an AI agent call tools. The goal is to let it call tools on behalf of the right user, with the right approval, under the right enterprise policy.</p><h3>Why MCP Changes the Shape of Authorization</h3><p>In the normal OAuth example, the relationship is fairly easy to understand.</p><p>One application wants access to one resource system. The user signs in. The resource system’s authorization server issues an access token. The app uses that token to call the resource system’s APIs.</p><pre>App<br>  → Authorization server<br>  → Resource APIs</pre><p>At first glance, an AI client may look like just another application. And in some ways, it is. An AI client or IDE can also ask for access on behalf of a user.</p><p>In older integrations, you might solve this with a service account, but that breaks down for user-driven agent workflows.</p><p>So you may wonder: how is AI agent or AI client access any different?</p><p>The difference is not the AI agent itself. The difference is the access path introduced by MCP. Without MCP, an application may integrate directly with a resource API:</p><pre>User<br>  → App<br>  → Jira APIs</pre><p>With MCP, the AI client usually does not integrate directly with every enterprise API. Instead, the AI client acts as an MCP client and calls an MCP server that exposes tools for systems like Jira, GitHub, Confluence, ServiceNow, or internal APIs.</p><pre>User<br>  → AI client / IDE<br>  → Jira MCP server<br>  → Jira APIs</pre><p>So the difference is not simply “normal app vs AI client.” The difference is this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W5My907JFdH43Bdl7TyJbQ.png" /></figure><p>That MCP layer becomes a new authorization surface. The important point is that MCP introduces another layer between the AI client and the resource system.</p><p>The enterprise now just doesn’t answer: Can this app access Jira on behalf of the user?</p><p>It also has to answer:</p><blockquote>Is this AI client approved?<br>Is this MCP server approved?<br>Is this user allowed to use this MCP server?<br>Is this AI client allowed to act on behalf of this user?<br>Is the requested MCP tool or action allowed?<br>Will the downstream system still enforce the user’s permissions?</blockquote><p>This is where authorization starts to spread across more places.</p><p>The enterprise identity provider anchors who the user is. The AI client needs to be approved to act for that user. The MCP server needs to know whether the user and client are allowed to call a specific tool. And the downstream system, such as Jira or GitHub, still needs to enforce its own permissions.</p><p>So MCP is not only changing how tools are exposed to AI agents. It is also changing where authorization decisions need to happen.</p><p>Authorization is no longer a simple handoff between one app and one resource system. With MCP, authorization has to be coordinated across the AI client, the MCP layer, the enterprise identity provider, and the actual system being accessed.</p><p>This is where enterprise MCP deployments start getting complicated.</p><h3>Many MCP Servers Become Many Secured App Surfaces</h3><p>The previous example showed one MCP server:</p><pre>User<br>  → AI client / MCP client<br>  → Jira MCP server<br>  → Jira APIs</pre><p>But enterprises rarely stop at one tool.</p><p>Once teams start adopting MCP, they quickly want access to many systems: Jira, GitHub, Confluence, Slack, ServiceNow, internal APIs, databases, dashboards, deployment systems, and more. A natural deployment pattern is to expose each MCP server as its own remote application endpoint.</p><p>For example:</p><pre>mcp-github.company.example<br>mcp-jira.company.example<br>mcp-confluence.company.example<br>mcp-slack.company.example<br>mcp-servicenow.company.example</pre><p>At first, this looks clean. Each MCP server has its own hostname, routing, configuration, and ownership.</p><p>But in an enterprise, each of those endpoints can become another secured application surface. In other words, every MCP server is another place where the enterprise must handle or validate authentication, authorize access, manage secrets and tokens, monitor activity, and revoke access when needed.</p><blockquote>Adding a new MCP server is not just, Add another tool.</blockquote><p><strong>It can also mean:</strong></p><blockquote>Host another endpoint.<br>Configure another route.<br>Register another OAuth client.<br>Set up another callback URL.<br>Manage another set of secrets.<br>Define another access policy.<br>Map enterprise users and groups to this MCP server.<br>Decide which tools and scopes are allowed.<br>Capture another stream of audit logs.</blockquote><p>That is where MCP authorization becomes operationally heavy.</p><p>The hard part is not only exposing tools over MCP. The harder part is creating a repeatable authorization model across many MCP servers, each connected to a different resource system with its own permissions and policies.</p><h3>MCP Gateways Help, But They Are Not the Whole Answer</h3><p>Technically, each MCP server can be exposed directly to AI clients.</p><pre>User<br>  → AI client / MCP client<br>  → Jira MCP server<br>  → Jira APIs</pre><pre>User<br>  → AI client / MCP client<br>  → GitHub MCP server<br>  → GitHub APIs</pre><pre>User<br>  → AI client / MCP client<br>  → Internal MCP server<br>  → Internal APIs</pre><p>But as the number of MCP servers grows, this direct model becomes harder to secure, monitor, and operate. A more enterprise-friendly pattern is to put an MCP gateway or proxy in front of the MCP servers.</p><pre>User<br>  → AI client / MCP client<br>  → Enterprise MCP gateway / proxy<br>  → Many MCP servers<br>      → Jira MCP server → Jira APIs<br>      → GitHub MCP server → GitHub APIs<br>      → Internal MCP server → Internal APIs</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UYoc0qVGaGGBJRdeaLtjUg.jpeg" /><figcaption>MCP Gateway</figcaption></figure><p>The gateway becomes the common front door for MCP access. This helps in two big ways.</p><p>First, security and policy become centralized. The enterprise can control which AI clients are allowed, which users or groups can access which MCP servers, which tools should be exposed, and which requests should be blocked, rate-limited, or logged.</p><p>Second, visibility and operations become centralized. Instead of chasing logs across every MCP server, platform and security teams get one place to observe MCP traffic, audit tool usage, apply controls, and revoke access.</p><blockquote><strong>But the gateway does not replace downstream authorization. Authorization still depends on the enterprise IdP, MCP/resource authorization server, and downstream resource permissions.</strong></blockquote><p>Jira still decides whether the user can create a Jira issue. GitHub still decides whether the user can read a repository or merge a pull request.</p><p>The gateway controls the front door into the enterprise MCP layer. The downstream MCP server and resource system still need to enforce what the user is actually allowed to do. So the gateway helps with control, visibility, and operational scale.</p><p>But the <strong>core cross-app authorization problem , the focus of this article, still remains</strong>. The enterprise still needs a standard way to answer:</p><blockquote>Who is the user?<br>Did the user authenticate through the enterprise identity provider?<br>Is this AI client allowed to act on behalf of this user?<br>Is this user allowed to access this MCP server?<br>Which scopes, tools, or actions are allowed?<br>How does the MCP server trust the request?<br>How is access revoked centrally?<br>How is this audited?</blockquote><p>Without a standard handoff, teams often end up building custom MCP authorization infrastructure: token brokers, OAuth callback handling, delegated token storage, refresh logic, session mapping, policy enforcement, and audit. What enterprises need is not another custom token broker. They need a standard way for the enterprise identity provider to hand off trusted user, client, and policy context to the MCP/resource authorization server.</p><h3>Enter ID-JAG: The Standard Handoff</h3><p>This is where ID-JAG comes in.</p><p>ID-JAG stands for <strong>Identity Assertion JWT Authorization Grant</strong>. It is defined in the IETF <a href="https://datatracker.ietf.org/doc/draft-ietf-oauth-identity-assertion-authz-grant/">Internet-Draft</a>.</p><p>The IETF document describes ID-JAG as a mechanism for using an identity assertion to obtain an access token for a third-party API through a trusted identity provider, while the MCP Enterprise-Managed Authorization extension describes the MCP client requesting an ID-JAG from the enterprise IdP as part of delegated authorization</p><p>It sounds complex, but the idea is simple. ID-JAG gives the enterprise identity provider a standard way to say:</p><blockquote>This user has authenticated through the enterprise identity system.<br>This AI client is allowed to act on behalf of this user.<br>This request is intended for this specific MCP server or resource application.<br>These are the scopes, tools, or actions being requested.</blockquote><p>In other words, ID-JAG creates a standard handoff between the enterprise identity provider, the AI client, and the MCP/resource application. This is important because the MCP server or resource application should not blindly trust the AI client. It needs a trusted signal from the enterprise identity provider.</p><blockquote><strong>ID-JAG provides that signal. One important point: ID-JAG is not the final API access token.</strong></blockquote><p>Think of it as a signed authorization grant. The AI client presents this grant to the MCP server’s authorization server. That authorization server validates the grant, checks policy, and then issues an access token for the MCP server.</p><p>The access token is what the AI client uses to call the MCP server. A simplified flow looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X_ylyhccoj6-UxAnlXnrgA.png" /></figure><p>So instead of every MCP server, MCP gateway, or platform inventing its own delegation model, the enterprise identity provider becomes the control point.</p><blockquote><strong>That is the missing enterprise handshake.</strong></blockquote><p>MCP gives AI clients a standard way to call tools. ID-JAG gives enterprises a standard way to authorize those calls on behalf of real users.</p><p>However, it is importnat to understand, ID-JAG is not a complete MCP security model. It does not replace gateway policy, downstream application permissions, audit logging, or tool-level safety controls. Its value is more specific: it standardizes the authorization handoff so every MCP server or platform does not have to invent its own delegation model.</p><p>A quick terminology note before we conclude:</p><p><strong>ID-JAG</strong> is the emerging OAuth mechanism: Identity Assertion JWT Authorization Grant.</p><p><strong>Cross App Access</strong> is Okta’s productized implementation of this pattern.</p><p><strong>MCP Enterprise-Managed Authorization</strong> is how this pattern is being applied to MCP so enterprise identity providers can help authorize AI clients and MCP servers on behalf of real users.</p><h3>Conclusion</h3><p>MCP gives AI clients a standard way to connect to enterprise tools. That is a big step forward.</p><p>But connection is only the first part of the problem.</p><p>In the enterprise, AI clients need to act on behalf of real users. MCP servers need to know whether those users and clients are allowed to call specific tools. Downstream systems like Jira, GitHub and internal APIs still need to enforce their own permissions. And security teams need central policy, audit, and revocation.</p><p>MCP gateways help by centralizing access, routing, observability, and operational control. But they do not fully solve the cross-app authorization problem by themselves.</p><p>That is where ID-JAG matters.</p><p>ID-JAG provides a standard handoff between the enterprise identity provider, the AI client, and the MCP/resource authorization server. Instead of every MCP platform inventing its own token broker or delegation layer, the enterprise identity provider can become the trusted control point for deciding which AI clients can act for which users, against which MCP servers, under which policies.</p><p>That is why ID-JAG feels like the missing piece in MCP enterprise authorization.</p><p>MCP standardizes how AI clients call tools. ID-JAG helps standardize how enterprises authorize those calls on behalf of real users.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4a4f835336a2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MCP Is Not the New API: A Practical Guide to Using Both]]></title>
            <link>https://medium.com/@raviscal/mcp-is-not-the-new-api-a-practical-guide-to-using-both-3feb39060774?source=rss-6676b3772ca1------2</link>
            <guid isPermaLink="false">https://medium.com/p/3feb39060774</guid>
            <category><![CDATA[mcp-vs-api]]></category>
            <category><![CDATA[mcp-is-dead]]></category>
            <category><![CDATA[context-overlaod]]></category>
            <category><![CDATA[agentic-workflow]]></category>
            <category><![CDATA[mcps]]></category>
            <dc:creator><![CDATA[Ravi Bommu]]></dc:creator>
            <pubDate>Wed, 27 May 2026 03:34:29 GMT</pubDate>
            <atom:updated>2026-05-27T03:41:43.354Z</atom:updated>
            <content:encoded><![CDATA[<p>Every new technology wave creates a familiar pattern: once a tool becomes popular, teams start applying it everywhere. That is happening with MCP (Model context protocol). Today, many teams are wrapping every API, every internal tool, every database action, and every business workflow behind an MCP server. On the surface, this feels modern. It makes systems “agent-ready.” It gives AI assistants access to more tools. It creates the impression that the architecture is ready for the next generation of AI agents.</p><p>But there is a problem.</p><p>Not every workflow needs an agent. Not every API needs to become an MCP tool. Not every business action should be placed behind a probabilistic decision-making layer. There are some caveats of wrapping every API behind MCP which we will discuss in later sections.</p><p>Before asking <strong>“Should we use MCP or API?”</strong>, we should ask a more important question:</p><blockquote><strong><em>Is this workflow deterministic or probabilistic? </em></strong>That one question changes the entire conversation</blockquote><h3>1. APIs and MCP are not the same thing</h3><p>I know, your first thought is, “ofcourse, it is Obvious” but lets spend a minute to understand how are they different</p><p>An <strong>API</strong> is a contract between software systems. It defines how one system calls another system, what request format is expected, what response format comes back, and how the two sides communicate. On other side, <strong>a MCP server</strong> is different. MCP, or Model Context Protocol, was introduced by Anthropic as an open standard for connecting AI assistants to systems where data and tools live, including business tools, content repositories, and development environments.</p><p>So the simple distinction is:</p><blockquote><strong><em>APIs are built for software-to-software communication. MCP is built fo</em>r <em>model-to-tool communication.</em></strong></blockquote><p>Until this point, majority of the tech world is clear. However, as mentioned before, many teams are wrapping every API, every internal tool, every database action, and every business workflow behind an MCP server assuming they are making all systems “agent-ready”.</p><p><strong>The deeper distinction : Deterministic or Probabilistic?</strong></p><p>APIs are best for deterministic execution. MCP is best for probabilistic orchestration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M2Gpox6l3390Xx0HRKytqg.png" /><figcaption>Probabilistic vs deterministic</figcaption></figure><p>A <strong>deterministic workflow</strong> is one where the path is known in advance.The input and steps are known. The output is expected. The rules are defined and system should behave the same way every time.</p><p>Examples include:</p><pre>Create an invoice<br>Fetch an order status<br>Reset a password<br>Update a customer address<br>Process a refund<br>Place a trade after approval<br>Sync CRM records<br>Generate a shipping label<br>Run a payroll calculation</pre><p>That is API territory.</p><p>A <strong>probabilistic workflow</strong> is different. The user’s intent may be ambiguous. The system may need to gather context. The correct path may not be obvious at the beginning. The output may require interpretation, synthesis, comparison, or recommendation.</p><p>Examples include:</p><pre>Investigate why this customer is unhappy<br>Find the root cause of this production issue<br>Analyze this codebase and suggest a fix<br>Review these documents and identify risks<br>Explain why revenue dropped last quarter<br>Help decide the next best action for this support case</pre><p>These workflows may require reasoning. The model may need to decide which information matters, which tools to use, and how to interpret the results. This is where MCP can be valuable.</p><p><strong>A Simple decision tree:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3IVx2Hu3HZjxzm1dd1s9aA.png" /><figcaption>Decision tree</figcaption></figure><p>MCP should not be the default. It should be chosen when the model needs agency, not when the system already knows what to do.</p><h3><strong>2. Why wrapping every API as MCP is risky</strong></h3><p>The current hype cycle is pushing teams to expose everything as an MCP tool. That sounds flexible, but it creates several issues. As mentioned at the beginning, here are some caveats of wrapping every API in MCP</p><h4><strong>2.1 Tool overload and context window burden is real.</strong></h4><p>To use tools properly, an LLM needs to understand what each tool does, how to call it, what inputs it needs, and what outputs it returns. With a small number of tools, this works fine. But when there are hundreds or thousands of tools, the model ends up spending a lot of its memory and attention just reading tool instructions instead of solving the actual problem. Research has also shown that simply adding more tools does not automatically make agents smarter or better at tasks . This increases cost and slows things down, but it also hurts accuracy.</p><p>Imagine giving someone a giant toolbox with 2,000 tools and asking them to quickly pick the right one every single time. The more tools available, the easier it becomes to get distracted, choose the wrong tool, or waste time searching. The same thing happens with LLM agents. Too many tools can clutter the context window, overload reasoning, and reduce overall performance. Even the results returned by tools add more text into the context, making the problem worse over time.</p><p>That is why scalable agent systems are not just about adding more tools. They also need smart ways to expose only the relevant tools at the right time instead of loading everything upfront.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6Ui0gjAMwydUEmNd7RxUmQ.png" /></figure><p>A practical way to reduce this problem is <strong>progressive disclosure</strong>. Instead of loading every MCP tool, schema, and description into the model’s context upfront, the system should expose only a small, relevant set of tools first. When the model needs more capability, it can search or request additional tools, and only then load the detailed schema for the selected tool. This keeps the context window cleaner, reduces input and output tokens, lowers latency, and helps the model focus on the task instead of choosing from a large, noisy tool catalog. In other words, MCP should not mean “show the model everything”; it should mean “show the model the right thing at the right time.”</p><h4>2.2 Local MCP is more mature than remote MCP at scale</h4><p>MCP works especially well in local developer environments. A coding assistant can use local MCP servers to inspect files, search code, query docs, or run tests. That local workflow is one of MCP’s strongest use cases.</p><p>Remote MCP is a harder problem.</p><p>Once MCP servers become production infrastructure, they need the same qualities expected from API platforms: load balancing, horizontal scaling, stateless operation, authentication, observability, session handling, and enterprise governance. The <a href="https://modelcontextprotocol.io/development/roadmap">official</a> MCP roadmap acknowledges this directly. It says Streamable HTTP gave MCP a production-ready transport, but running it at scale has revealed gaps around horizontal scaling, stateless operation, and middleware patterns.</p><h3>2.3 MCP can blur execution boundaries</h3><p>APIs usually have clear contracts and ownership. They are built around explicit inputs, outputs, permissions, and error handling. MCP tools are different because they are exposed to a model that may choose when and how to use them. That makes governance more important.</p><p>The MCP <a href="https://modelcontextprotocol.io/specification/2025-11-25">specification</a> itself warns that the protocol enables powerful capabilities through arbitrary data access and code execution paths, and says implementors must carefully address security and trust considerations.</p><p>That matters especially when MCP tools wrap sensitive actions.</p><p>A tool that reads documentation is low risk.<br> A tool that searches a codebase is moderate risk.<br> A tool that deletes production resources is high risk.<br> A tool that executes trades, issues refunds, changes payroll, or modifies customer records is <strong>very high risk</strong>.</p><h4>The higher the risk, the more deterministic control you need</h4><h3>3.How to avoid MCP misuse</h3><p>A production MCP implementation should not expose every internal API directly to the model. Instead, it should be curated.</p><p>Before wrapping a workflow with MCP, teams should ask a simple question: <strong>does this really need to be probabilistic?</strong> If the workflow already has a known sequence of steps, a fixed business rule, and a predictable output, then MCP may not add much value. In fact, it may add unnecessary cost, token usage, latency, and risk. MCP should earn its place by providing a clear advantage: interpreting ambiguous intent, choosing between tools, gathering context across systems, summarizing results, or recommending the next best action.</p><p>A better MCP pattern is to expose <strong>task-level functions</strong> instead of wrapping every low-level API endpoint as its own tool. These task-level functions should also include human-in-the-loop controls where the action is sensitive or irreversible.</p><p>For example, instead of giving the model separate tools for get_customer, list_invoices, get_payments, and check_refunds, expose one higher-level function like investigate_billing_issue(customer_id). Internally, that function can call multiple deterministic APIs, combine the results, and return only the useful summary back to the model. This reduces the number of tool definitions the model has to consider, lowers intermediate input/output tokens, and makes tool selection more reliable.</p><p>The goal is not to expose your backend architecture to the model. The goal is to expose the <strong>task interface</strong> the model actually needs.</p><p>A simple rule of thumb:</p><blockquote><strong><em>If the system already knows exactly what to do, use an API. If the model needs to figure out what matters before deciding what to do, MCP may be useful.</em></strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Jyudcoec7J4oA03p41c9TA.png" /></figure><h4>Where MCP shines</h4><p>The point of this article is not that MCP is bad. MCP is useful when the workflow actually needs model-driven reasoning. In these cases, the user may not know the exact endpoint, query, or workflow upfront. The model needs to interpret intent, gather context, and decide what information matters. That is where MCP adds value. The mistake is using the same agentic pattern for workflows that are already deterministic.</p><h3>Conclusion</h3><p>A simple analogy, think of APIs as the reliable roads that systems already use to communicate with each other. MCP is more like a smart GPS for AI agents, helping them discover which roads to take, what tools are available, and how to use them dynamically.</p><p>The key is knowing when to use each.</p><ul><li>Use <strong>APIs</strong> when you need <strong>deterministic behavior.</strong></li><li>Use <strong>MCP</strong> when you need <strong>probabilistic and adaptive </strong>AI behavior</li></ul><p>The mistake is assuming the GPS should replace the roads. The problem today is that many teams are trying to convert every API into an MCP tool, assuming MCP should replace traditional APIs entirely. That is where things start to break down.</p><p><strong>Not every API interaction needs an AI agent in the middle.</strong></p><p>If MCP is overused:</p><ul><li>Accuracy can drop because LLMs may choose the wrong tool, generate incorrect parameters, or misinterpret context.</li><li>Costs can increase significantly due to repeated reasoning cycles, larger prompts, tool orchestration overhead, and unnecessary token usage.</li><li>Latency becomes higher because agent workflows are slower than direct API calls.</li><li>Governance becomes harder since deterministic API contracts get replaced with probabilistic AI behavior.</li><li>Debugging becomes more difficult because failures are no longer simple API errors — they become reasoning-chain problems.</li><li>Security risks can grow if agents dynamically access tools without strong boundaries and validation.</li></ul><p><strong>The real question is not whether MCP is powerful. It is.</strong></p><p><strong>The real question is whether the workflow needs a probabilistic agent in the first place.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3feb39060774" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Start to Finish Retrieval Augmented Generation (RAG ) in Snowflake using vector search and Cortex]]></title>
            <link>https://medium.com/@raviscal/start-to-finish-retrieval-augmented-generation-rag-in-snowflake-using-vector-search-and-cortex-69a951e76edd?source=rss-6676b3772ca1------2</link>
            <guid isPermaLink="false">https://medium.com/p/69a951e76edd</guid>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[retrieval-augmented]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[vector-search]]></category>
            <dc:creator><![CDATA[Ravi Bommu]]></dc:creator>
            <pubDate>Tue, 13 Feb 2024 17:51:08 GMT</pubDate>
            <atom:updated>2024-02-14T21:32:28.576Z</atom:updated>
            <content:encoded><![CDATA[<p>I have recently completed a coursework on RAG - Retrieval Augmented Generation. This has increased my curiosity and I started to do more research around it and want to create an end to end RAG application. As part of the completed course work, I created a RAG application by putting together multiple open source tech stack(Open AI API, weaviate database, streamlit, llama2 ). This needed considerable amount of effort to setup, understand vector db client API and also write some programming (modules) for re-usability. For data engineers who are not very familiar with infrastructure and programming (Python/ Scala), this might be a hindrance to get their hands on RAG. Lately, I stumbled upon the latest features released by snowflake (Cortex) and found it has all the required pieces to create a RAG app and decided to give it a try to see how accurate it is. This would help data engineers to move forward easily and create RAG apps. Note, these features are in private preview at the time of writing this article and might not be available on your account yet. I will walk you through the components I used to create my RAG application using only the Snowflake platform, which has a chat user interface ( via Snowflake native <a href="https://urldefense.com/v3/__https://streamlit.io/__;!!JmoZiZGBv3RvKRSx!5pi2bC63zlppngTSgyy9bu7-wq51-7kpDg7UwBjeuAnhNmHc4Gj38RrhwY6U2zTSKDTVtoNtQn_m4sodx8yMaChMFR4$">Streamlit</a>) and can respond back to user queries on my test data set.</p><p>Before, I jump in, let me provide some context on what RAG is. I am sure, lately, most of you have heard of the term LLM (Large Language Model). If you ever tried using it, from your experience, you should have found that when you ask LLM a question, it provides you an answer as if you are talking to a human. Sometimes these models magically get answers very accurately, however, sometimes they are terribly wrong. LLM answers your question based on trained data (raw content ). The reason for wrong answers could be mostly either LLM doesn’t have the source to provide a good answer for you question or it has stale data from which it provided a wrong answer. Now, if you are trying to build an application using LLM with a chat-GPT kind of experience on your private dataset/ own content, this is an issue you would exactly face as the LLM was never trained on your data. There are two ways to fix this. One is to fine tune you model by training on your dataset and adjusting the model’s weight. This process needs lot of effort and still has concerns of black-box ML and not being able to validate/trust your application answers completely. The other option is RAG. You can think of RAG as a search engine that can pull relevant data for your question from your dataset and provide it to LLM. This helps your application LLM have contextual data for your question and unlike fine tuning we can validate the answer as we control and see the input data to LLM .</p><p>At a high level, a RAG application needs :<br>1) preprocessed data(chunking, conversion of chunks to vectors) <br>2) Vector database to store your data vectors and can provide functions for vector search<br>3) UI that can take the user queries and responds back with the answer <br>4) A retriever that takes user queries, convert to vectors using same model as your data encoding and use Hybrid search (combine vector and word similarity search) to retrieve the right contextual data<br>5) Reranker — cross encoders that can reorder/rerank our retrieval results by relevance to our query<br>6) LLM that can synthesis your contextual data and provide an answer</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MwlvSALE6nxYXRUcHbDGsA.png" /><figcaption>RAG application high level components</figcaption></figure><p>Now we understand what RAG is and it’s high level components, I want to take a minute on how all these components were available at my disposal in snowflake and allowed me to quickly create an app in few hours. I was using podcast dataset from YouTube that has 370 episodes, each ~ 1hr duration, discussing wide variety of topics ranging from social media, currency, health habits etc., I used Snowpark in Python/Scala to preprocess all the podcast data and store in Snowflake as embedded vectors. If you are not familiar with Snowpark, you can use Stored procedures to achieve the same, however, I highly recommend Snowpark as it provides you the power of Programming language with SQL and makes it amazingly easy to process complex data. Snowflake now supports vector data type(2) natively and provides llama2–7b-chat(6), BERT based models for extracting answers and E5-base-v2 model(1 and 4) for embedding vectors.At the time of this article, they do not have a re-ranker(5) yet but I m confident they would come with Hybrid search for re-ranking very soon. Snowflake supports streamlit(3) as a native snowflake object and makes it very easy to develop and publish a UI app without any infrastructure needs and knowledge of Java script, react, node etc., You will need basic Python programming skills.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pwjMKNKVv2PJPMtphE2KLw.png" /><figcaption>RAG in snowflake at a glance (courtesy of Snowflake from summit)</figcaption></figure><p>Once I have the data in Snowflake, I built a UI chatbot using Streamlit chatbot functionality and Snowflake Cortex ( ML ) functions that can take a user query , convert it to embed text, perform a vector_l2_distance similarity search, pull relevant results and send them to LLM. The LLM then wordsmith the answer based on the input data and provides an answer. Below you can see it in action where I am asking the chatbot questions on topics from the 370 podcasts (380 hours transcribed data) and it is able to search for relevant chunks really quick and provide an answer. FYI, I used XL warehouse for this application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*ngro_pe91K5pgXzZHqZDSQ.gif" /><figcaption>End to End RAG Application built on Snowflake based on podcast data</figcaption></figure><p>The search results from snowflake are as good as the other stack I tested from Opensource, even for its private preview version. Hope this read creates an interest for you in RAG as well and if so, try getting your hands dirty and see if you can build an application from you own data that can serve your business. Before I end this article, for fun, I input this article as source to my RAG app and asked it to summarize for me. Here is the output.</p><p>“<em>The author of the article has recently completed a coursework on RAG (Retrieval Augmented Generation) and is interested in creating an end-to-end RAG application using Snowflake, a cloud-based data warehousing platform. The author notes that Snowflake provides all the necessary components for building a RAG application, including pre-trained models, vector database, chat UI, and ML capabilities.</em></p><p><em>To create the application, the author used Snowpark in Python/Scala to preprocess a podcast dataset and store it in Snowflake as embedded vectors. The author then leveraged Snowflake’s native support for vector data types and used LLama2 to extract answers from the data. The author also used E5-base-v2 model for embedding vectors. While Snowflake does not currently have a re-ranker (step 5 in the RAG process), the author is confident that it will be available soon.</em></p><p><em>The author built a UI chatbot using Streamlit chatbot functionality and Snowflake Cortex (ML) functions to take user queries, convert them to embed text, perform a vector_l2_distance similarity search, and pull relevant results. The LLM then wordsmiths the answer based on the input data. The author tested the application by asking the chatbot questions on topics from the 370 podcasts (380 hours transcribed data) and was able to search for relevant chunks quickly and provide accurate answers.</em></p><p><em>The author concludes that Snowflake provides a powerful platform for building a RAG application and encourages readers to try getting their hands dirty and building an application from their own data to serve their business needs</em>”.</p><p>Happy learning!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=69a951e76edd" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>