<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Vivek Rathi on Medium]]></title>
        <description><![CDATA[Stories by Vivek Rathi on Medium]]></description>
        <link>https://medium.com/@vivekrathi8993?source=rss-9213a50669a3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*F8qGX3deyYsMHrJnNOxNfw.jpeg</url>
            <title>Stories by Vivek Rathi on Medium</title>
            <link>https://medium.com/@vivekrathi8993?source=rss-9213a50669a3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 02:47:43 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@vivekrathi8993/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Foundations of Effective Data Governance]]></title>
            <link>https://medium.com/@vivekrathi8993/foundations-of-effective-data-governance-ae4a88023a65?source=rss-9213a50669a3------2</link>
            <guid isPermaLink="false">https://medium.com/p/ae4a88023a65</guid>
            <dc:creator><![CDATA[Vivek Rathi]]></dc:creator>
            <pubDate>Mon, 21 Jul 2025 21:29:04 GMT</pubDate>
            <atom:updated>2025-08-09T17:25:52.402Z</atom:updated>
            <content:encoded><![CDATA[<p>With Gen AI use-cases evolving, organizations continue to invest heavily in collecting, storing, and analyzing data. However, the true value of these investments is only realized when the data is trustworthy, secure, and understandable. This is where <strong>Data Governance</strong> comes in. It’s a strategic framework of rules, processes, and technologies that ensures your data is managed as a valuable asset. Working on several data governance projects across various domains, I distilled the following four key pillars that work together to build an efficient data governance strategy..</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KU-pL4fyYFsR6MyfdKosfg.png" /></figure><h3>Data Quality</h3><p>Data quality is at the core of data initiatives. Put simply, if you can’t trust your data, its even more difficult to trust the insights, reports, and AI models built upon it. Poor data quality directly undermines the <strong>ROI</strong> of your data investments. It can range from simple issues, like incorrect data formats in a column, to more complex business rule violations that are harder to detect.</p><p><strong>Addressing Data Quality</strong></p><p><strong>Custom SQL Queries:</strong> s writing custom SQL queries to check for inconsistencies is a common practice and starting point. While this works for specific, isolated checks, it’s difficult to manage, standardize, and scale across an enterprise. It lacks features like shared definitions, centralized reporting, and automated alerting.</p><p><strong>In-Pipeline Validation:</strong> A more modern approach is to embed data quality checks directly into your data transformation pipelines. For instance, <strong>Databricks Delta Live Tables (DLT)</strong> allow you to declaratively define data quality constraints. This is powerful because you can validate data as it flows through the pipeline, report on quality metrics, send alerts, and even quarantine bad records for later investigation without halting the entire process.</p><p><strong>Specialized Governance Tools:</strong> For enterprise-grade data quality, specialized tools like <strong>IBM Knowledge Catalog</strong> provide a comprehensive solution. They allow you to catalog and share data quality definitions so different teams can reuse common checks. These platforms can auto-generate quality rules based on data profiles, produce interactive dashboards, and manage remediation workflows by automatically creating and assigning tasks to the correct data owners.</p><h3>Data Protection</h3><p>Data protection is all about controlling and monitoring access to ensure that sensitive information is shielded from <strong>unauthorized users</strong>. For many industries, it’s a strict compliance requirement. The goal is to enforce access rules that grant the right level of access to the right people for the right data and be able to manage this at scale.</p><p><strong>Addressing Data Protection</strong></p><p><strong>Platform-Level Access Controls:</strong> Nearly all modern data platforms — databases, data lakes, and lakehouses — have built-in access management. For example, <strong>Databricks Unity Catalog</strong> provides a centralized way to define access rules on catalogs, schemas and tables that can span multiple workspaces.</p><p><strong>Catalog-Driven Policy Enforcement:</strong> Metadata catalogs manage access differently since they don’t typically store the actual data. They can apply masking rules to data previews within the catalog itself. However, more advanced solutions take this a step further. Tools like <strong>IBM Knowledge Catalog</strong> can deeply integrate with underlying data platforms (like IBM Data Virtualization or watsonx.data), allowing protection policies defined in the catalog to be automatically enforced on the data at its source.</p><p>A critical element here is to ensure that right <strong>business metadata </strong>(explained below)<strong> </strong>is attached to technical data assets. This context is what allows automated policies to correctly identify and protect sensitive information across the organization.</p><h3>Data Lineage</h3><p>Data lineage provides a complete, traceable map of a data object’s journey, showing all upstream sources and downstream dependencies. This context is invaluable for understanding how a data point was calculated and what other reports or applications rely on it. For critical functions like financial reporting, end-to-end lineage is essential for <strong>root cause analysis</strong> and debugging. It’s also incredibly useful for understanding complex, cross-database dependencies between different applications.</p><p><strong>Addressing Data Lineage</strong></p><p><strong>Open Standards:</strong> The <strong>OpenLineage</strong> framework has become a popular open-source standard for collecting lineage metadata from various sources. This standardized metadata can then be consumed by visualization tools, both open-source (like <strong>Marquez</strong>) and proprietary, to build and display lineage graphs.</p><p><strong>Automated Lineage Platforms:</strong> Proprietary technologies like <strong>IBM Manta</strong> significantly simplify lineage collection through automation. Manta can scan a wide array of sources — including databases, reporting tools, ETL scripts, and programming languages — to automatically extract lineage information and stitch it together into a single, end-to-end view. Users can not only see the data flow but also drill down into the transformation logic at each step, such as viewing the source code of a stored procedure. These platforms integrate tightly with data catalogs (like <strong>Collibra</strong> or <strong>IBM Knowledge Catalog</strong>) to enrich data assets with this powerful lineage context.</p><h3>Business Context</h3><p>Technical metadata like table and column names (CUST_ID, TXN_AMT) is often cryptic and meaningless to business users. Providing <strong>business context</strong> is crucial for increasing the adoption of data assets, establishing a common enterprise vocabulary, and ensuring governance policies are applied correctly.</p><p><strong>Addressing Business Context</strong></p><p><strong>The Business Glossary:</strong> A simple column description is a start, but it doesn’t scale. Leading data governance platforms (<strong>IBM Knowledge Catalog</strong>, <strong>Collibra</strong>, <strong>Informatica Data Catalog</strong>, <strong>Apache Atlas</strong>) allow you to create a formal <strong>Business Glossary</strong>. This glossary is a curated repository of <strong>business terms</strong> — well-defined concepts like “Active Customer” or “Gross Revenue” — complete with their definitions, associated policies, classifications (e.g., PII), and relationships to other terms.</p><p><strong>Linking and Automation:</strong> These business terms are then linked to the technical metadata. For example, the term “Customer Identifier” could be associated with columns named CUST_ID, CUSTOMER_ID, and CUSTOMER_NUM across different databases. This creates a unified business view of your data. To accelerate this process at scale, leading platforms use <strong>AI and rule-based algorithms</strong> to automatically discover and suggest these links. This automation can also trigger the inheritance of classifications and policies, saving countless hours of manual effort. Generative AI features, such as those appearing in <strong>IBM Knowledge Catalog</strong>, are now being used to help data stewards generate clear, consistent definitions for business glossary terms, further accelerating the process of building a well-governed data ecosystem.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ae4a88023a65" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Agentic RAG: Agent driven adaptive workflows for RAG]]></title>
            <link>https://medium.com/@vivekrathi8993/agentic-rag-agent-driven-adaptive-workflows-for-rag-ccdd28bc6096?source=rss-9213a50669a3------2</link>
            <guid isPermaLink="false">https://medium.com/p/ccdd28bc6096</guid>
            <category><![CDATA[watsonx-ai]]></category>
            <category><![CDATA[langgraph]]></category>
            <category><![CDATA[agentic-rag]]></category>
            <category><![CDATA[watsonx]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Vivek Rathi]]></dc:creator>
            <pubDate>Thu, 27 Feb 2025 23:45:15 GMT</pubDate>
            <atom:updated>2025-03-02T23:37:35.430Z</atom:updated>
            <content:encoded><![CDATA[<p>The purpose of this article is to compare and contrast single-step RAG workflows to Agentic RAG approach and provide an example of using LLM agent to implement Agentic RAG using LangGraph and IBM watsonx.ai.</p><p>In single-step RAG workflows, the process is linear and static. The system retrieves documents, generates a response, and delivers it without validating its accuracy or relevance. This works in case of simple queries where direct information is available in the knowledge base but the accuracy takes a toll as soon as queries start to get complex. If the retrieved context is inadequate or the LLM misinterprets it, the output suffers. The lack of iterative reasoning and adaptability limits the solution and can leave the user with incomplete or misleading answers. Using an ai agent to orchestrate your RAG workflow (Agentic AI) allows you to have additional steps like routing the query to appropriate source, validating the context, executing additional searches, checking the response for hallucinations etc. These steps are triggered dynamically by the agent based on the current state of the workflow.</p><h3>Agentic RAG</h3><p>This approach introduces an LLM-driven agent that dynamically orchestrates the retrieval and generation workflow. Below are high level details of how this approach is different from single-step RAG:</p><ol><li>Reflection and Adaptation: The agent evaluates its state at every step, revisiting or modifying steps as needed. For example, if retrieved documents are irrelevant, the agent can trigger a web search to refine results.</li><li>Dynamic Workflow Execution: Unlike static workflows, the agent decides which steps to execute based on its current state. The workflow evolves as the agent updates its understanding of the task.</li><li>Validation: Each response is validated for grounding in retrieved documents. If the response is inaccurate or incomplete, the agent revisits earlier steps until a satisfactory answer is generated.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wPfeZz3P2H22TmR8be9iYw.png" /><figcaption>Agentic RAG flow example implemented in LangGraph and IBM watsonx.ai</figcaption></figure><h3>Example using LangGraph and IBM watsonx.ai</h3><p>Check out this <a href="https://github.com/vivekrathi8993/agentic_rag/tree/main">github repo</a> for full code to implement an AI agent using <a href="https://langchain-ai.github.io/langgraph/">LangGraph</a> and <a href="https://www.ibm.com/products/watsonx-ai">IBM watsonx.ai</a>. The code provided in this article is an adaptation of examples provided by LangGraph (<a href="https://langchain-ai.github.io/langgraph/tutorials/rag/langgraph_adaptive_rag/#define-graph-flow">here</a>) and Eden Marco (<a href="https://github.com/emarco177/langgaph-course">here</a>).</p><p>Here are the steps in the agents workflow. As each of the step runs, a state object is updated, and this information is used to conditionally determine which step to take next.</p><ol><li>The user submits a question.</li><li>The LLM routes the query to either a vector database or web search based on its topic.</li><li>Grade documents:<br>- If all retrieved documents are relevant, proceed to response generation.<br>- If any document is irrelevant, perform additional searches and recheck relevance.</li><li>Response Generation: The LLM generates an answer based on retrieved documents.</li><li>Validation:<br>- If the response is grounded in documents and addresses the question, the workflow ends. (happy path)<br>- If the response is grounded in documents but does not completely address the original question then revisit search (Step 3) and run the process from there until a valid response is found. (adapt and iterate as needed)<br>- If the response is not grounded in the documents then re-generate and validate again</li></ol><h3>Considerations and closing thoughts</h3><p>In the sections above we’ve seen how a more dynamic RAG workflow can help in improving overall accuracy of a RAG solution, however there are a few things to keep in mind. Since the agentic workflow is more complex and iterative, this means you are making more LLM calls as compared to a single-step approach. This could have both performance (latency) and cost implications. One way to strike a balance is to choose the right model for each LLM step, instead of choosing the biggest model for every step. This is where using IBMs watsonx.ai platform fits in as it allows you to choose from a wide range of LLMs of different sizes and capabilities built by IBM and others leading companies like Meta and Mistral.</p><p><em>Disclaimer: All views in this article are my own and do not represent my employer or their views in any manner.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ccdd28bc6096" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>