<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Loys Belleguie on Medium]]></title>
        <description><![CDATA[Stories by Loys Belleguie on Medium]]></description>
        <link>https://medium.com/@loysbelleguie?source=rss-dcbd37cb0d2e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*spIQbk3Ps3JecAWnHRZIow.jpeg</url>
            <title>Stories by Loys Belleguie on Medium</title>
            <link>https://medium.com/@loysbelleguie?source=rss-dcbd37cb0d2e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 15:08:30 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@loysbelleguie/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Semantic Twin: The Decision Layer for the Agentic Enterprise (4/n)]]></title>
            <link>https://loysbelleguie.medium.com/the-semantic-twin-the-decision-layer-for-the-agentic-enterprise-4-n-a7d787d2cf16?source=rss-dcbd37cb0d2e------2</link>
            <guid isPermaLink="false">https://medium.com/p/a7d787d2cf16</guid>
            <category><![CDATA[metadata]]></category>
            <category><![CDATA[semantic-layer]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Loys Belleguie]]></dc:creator>
            <pubDate>Fri, 23 Jan 2026 05:03:56 GMT</pubDate>
            <atom:updated>2026-01-23T05:03:56.769Z</atom:updated>
            <content:encoded><![CDATA[<p><em>This is Part 4 of the Enterprise Data Intelligence series. In </em><a href="https://medium.com/@loysbelleguie/enterprise-data-intelligence-use-cases-3-n-3a97a63c899d"><em>Part 3</em></a><em>, we explored the use cases. Now, we dive into the architecture.</em></p><p>If you have been following the AI investment landscape recently, you might have noticed a sudden obsession with a “new” concept: the <em>Context Graph</em>.<br>Foundation Capital recently declared Context Graphs as <a href="https://foundationcapital.com/context-graphs-ais-trillion-dollar-opportunity/">AI’s Trillion-Dollar Opportunity</a> [1]. The thesis is simple but profound: traditional Systems of Record (like SAP or Salesforce) store the <em>result</em> of a decision (e.g., “Discount Approved”), but they might lose the <em>context</em> of that decision (e.g., “Approved because the customer is at risk of churn, per VP email”).<br>For an AI agent to function safely, it cannot just read the database state; it needs the <em>decision trace, </em>and this is not just a lineage extracted from SQL logs. It needs the context, and if it is presented as mind map — visually like a graph — then it is even better, hence the hype around the “context graph”!<br>As Sergey Vasiliev, Jessica Talisman, and other graph experts have noted, this concept is not actually new [2, 3]. It is a rebranding of what can be named, a little bit pedantically I admit, a <em>Neuro-Symbolic Knowledge Graph</em>. The “Trillion-Dollar” insight isn’t the graph itself — it’s the realization that AI agents cannot run on SQL tables alone, whatever rich comments the data owner may have added to its code.<strong><br></strong>In this article, I will move beyond the VC hype and try to show what this architecture actually looks like practically, with some code. I call it the <strong>Semantic Twin </strong>[4]. This is not a <em>Semantic Digital Twin </em>as found here and there in the literature, which relates to an extension of a digital twin per se, but more the semantic sibling of the <em>enterprise digital thread</em>.</p><h3>1. The Problem: “Meaning Silos”</h3><p>With the development of AI agent frameworks in the enterprise, one problem has become central: how data domains can share semantic information. While raw data has been made accessible across silos, thanks to a decade of Digital Transformation (DX) consolidation with the data warehouses, the Modern Data Stack (MDS) and the like, yet the<em> meaning, </em>or business information, is still fragmented. <br>In a typical enterprise, the definition of a “Critical Supplier Risk” is scattered:<br>● <strong>The logic</strong> lives in SQL scripts or dbt models.<br>● <strong>The state</strong> lives in the Data Warehouse (e.g., Snowflake).<br>● <strong>The business context</strong> lives in the metadata store (structured data) and all other unstructured information, PDFs, emails, or the heads of domain experts.<br>When an AI agent tries to act, it may encounter <em>Meaning Silos</em>. It can query the data, but it lacks the system understanding to know if that data is actionable. Relying on LLMs to infer this structure from schema names alone is dangerous; they need a reliable “World Model” for agents, providing real-time context on data lineage, quality, and policy compliance. The old-fashion data catalog era must evolve into an Active Data Intelligence Platform— areas where solutions like <a href="https://quollio.com/">Quollio Technologies</a> are positioned to become the operational backbone of the AI workforce.</p><h3>2. Defining the Semantic Twin: A Projection, Not a Replica</h3><p>A common misconception would be that a Semantic Twin is just a Knowledge Graph where we dump all our data, but this is not at all that. The petabytes of transaction data stay where they should, in databases and DWH. We only extract the metadata of those big data, which is the core information stored in the data catalog (ex. the <a href="https://quollio.com/jp/solutions/datacatalog">Quollio QDIC</a>); but we also add many other enterprise information such as internal or open sources. Then, we may indeed load some raw data, such as the enterprise value chain entities information (ex. suppliers), and extract (business) relationships within the graph. <br>This builds a solid “semantic Single Source of Truth” (SSoT) layer, like a semantic projection of the enterprise data landscape. It separates the <em>Physical Data Layer</em> from the <em>Reasoning Layer</em>:</p><ol><li><strong>The Physical Data Layer (ex. Snowflake + dbt):</strong> Handles heavy compute, storage, and metric aggregation. It stores the “Truth.”</li><li><strong>The Semantic Twin (Graph + Agents):</strong> Stores the <em>structure</em>, the <em>dependencies</em>, and the <em>state</em> of business objects, in other words, it stores the “Meaning.”</li></ol><p>From the data layer, we can then choose only what is needed to analyze a particular business case, extract the <em>data signal</em>s, project them within the semantic layer, connect all the relevant entities. As a matter of fact, AI “agents in high-volume enterprise environments must never sit in the data path” [5]. One does not pass billions of rows through an LLM’s context window. So, the semantic layer only loads the <em>quintessence</em> of the data landscape.</p><p>We can then apply any methods at our disposal, from direct graph traversal (queries) to more elaborate graph science algorithms (community detection, betweenness centrality, PageRank etc.) can be used to analyze the whole system and feed the consumer layer (Users and Agents) with the a governed context.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vdEEwqs3EOJ34mL_PmU8mA.png" /><figcaption>Logical view of the Semantic Twin</figcaption></figure><h3>3. The Architecture in Code</h3><p>Let’s look at how we build this Semantic Twin in practice, with simple blue print code excerpts. We follow a three-step pattern: <strong>Define, Project, Reason.</strong></p><h4>Step A: Define the Data Signals</h4><p>First, we define a “Risk Signal” deterministically in the data warehouse using dbt. This is not vague AI “intuition”; it is governed code, based on the usual dbt semantics. This is what usually goes to data marts to finally land directly into BI tools dashboards.</p><pre>-- models/marts/signals/signal_shipment_delay_risk.sql<br>-- We calculate the raw signal in the warehouse where compute is cheap.<br>WITH shipments AS (<br>    SELECT * FROM {{ ref(&#39;fact_shipments&#39;) }}<br>)<br>SELECT<br>    shipment_id,<br>    order_id,<br>    -- The logic that defines &quot;Risk&quot; is governed here, not guessed by AI<br>    CASE<br>        WHEN delay_days &gt; 14 THEN &#39;CRITICAL&#39;<br>        WHEN delay_days &gt; 7  THEN &#39;HIGH&#39;<br>        ELSE &#39;MODERATE&#39;<br>    END as severity_band,<br>    delay_days as risk_score<br>FROM shipments<br>WHERE delay_days &gt; 0</pre><h4>Step B: Project into the Graph (The Context)</h4><p>Next, we project this signal into the Semantic Layer, materialized as a graph (I use <a href="https://neo4j.com/">Neo4j</a> as the graph engine and triple store, but other solution can be employed as well). <br>This is where the ST is born. We don’t just store the row; we link the Signal to the Supplier and the Data Product that generated it.</p><pre>// ./cypher/upsert_signal_supplier.cypher<br>// We map the deterministic Signal (from dbt) to the Semantic Entity (Supplier)<br>UNWIND $batch AS row<br><br>// 1. Create the Signal Node<br>MERGE (s:Signal {id: row.signal_id})<br>SET s.type = &#39;ShipmentDelay&#39;,<br>    s.severity = row.severity_band,<br>    s.score = row.risk_score<br><br>// 2. Link it to the Business Object (The Context)<br>MERGE (sup:Supplier {id: row.supplier_id})<br>MERGE (s)-[:AFFECTS]-&gt;(sup)<br><br>// 3. Link it to the Source of Truth (Provenance)<br>MERGE (s)-[:DERIVED_FROM]-&gt;(:DataProduct {name: &#39;signal_shipment_delay_risk&#39;})</pre><p>Now, the Agent doesn’t just see a table row. It sees that a Signal AFFECTS a Supplier, and it knows exactly which dbt model (DERIVED_FROM) created that signal. This is the <em>Epistemic Boundary</em> — the agent will know <em>why</em> it knows this.</p><h4>Step C: Reasoning within the Semantic Twin</h4><p>Finally, the Agent uses this graph to “think”, with full knowledge of the facts, instead of hallucinating dependencies. It can choose which tool (MCP) to use, how to query the graph to find downstream impacts, or any other root causes, hidden collateral effects that may be extracted from the graph analysis.</p><pre>// ./cypher/gds_supplier_bottleneck.cypher<br>// The Agent triggers this to understand structural risk<br>CALL gds.pageRank.stream(&#39;supply_chain_graph&#39;, {<br>  relationshipWeightProperty: &#39;risk_propagation_factor&#39;<br>})<br>YIELD nodeId, score<br>RETURN gds.util.asNode(nodeId).name AS Supplier, score<br>ORDER BY score DESC</pre><p>This relates to the <em>Neuro-Symbolic AI</em> in [2]:<br>● <strong>Symbolic:</strong> The graph is a complicated network of nodes (symbols) and relationships which corresponds to business entities. <br>● <strong>Neural:</strong> The LLM agent can interpret this result (“Supplier X is a bottleneck”) to make a business recommendation, and importantly, explain the why.</p><h3>4. The “Epistemic Boundary”</h3><p>The innovative idea here, at least from my point of view, is what I call the <em>Epistemic Boundary</em>. Epistemology is the branch of philosophy concerned with knowledge, justification, and the distinction between knowledge, justified belief, and mere opinion — you know, the distinction that protects us from mistaking a personal desire to buy a sovereign island for a geopolitical fact 🧐.<br>In the context of AI agents, the Epistemic Boundary is the hard line between <em>Raw Information</em> and <em>Validated</em>, <em>Governed Knowledge</em>. <br>The Semantic Twin provides this boundary. By forcing agents to interact with the Semantic Twin — rather than querying raw tables directly — we ensure they only reason using validated signals with explicit meaning, lineage, and trust.</p><h3>Conclusion</h3><p>The transition from the Modern Data Stack to the Agentic Data Stack is not about buying better LLMs. It is about restructuring our data to be machine (I should say agent)-understandable.<br>Whether you call it a Semantic Twin, a Context Graph, or a Neuro-Symbolic Architecture, the imperative is the same: we must move from storing <em>records</em> to storing <em>reasoning</em>.<br>In an agentic world, the question is no longer <em>“Can we query the data?”<br></em>It is: <em>“Does the system understand itself ?”</em></p><h3>References</h3><p>● <strong>[1]</strong> Gupta, J., Garg, A. <em>Context Graphs: AI’s Trillion-Dollar Opportunity</em>. Foundation Capital. Available at: <a href="https://foundationcapital.com/context-graphs-ais-trillion-dollar-opportunity/">foundationcapital.com</a></p><p>● <strong>[2]</strong> Vasiliev, S. <em>Knowledge Graphs, LPGs, and Neuro-Symbolic AI</em>. Available at: <a href="https://substack.com/home/post/p-184057323">Substack</a></p><p>● <strong>[3]</strong> Talisman, J. <em>Context Graphs are Process Knowledge</em>. Available at: <a href="https://www.linkedin.com/pulse/context-graphs-process-knowledge-jessica-talisman-fbyic/">LinkedIn</a></p><p>● <strong>[4]</strong> Belleguie, L., Hayashi, T., &amp; Sanada, T. (2025). <em>Metadata-Driven Semantic Integration of Multi-Source Data with a Graph-Augmented LLM Agent Framework for Supply Chain Risk Management</em>. SIG-DC. Available at: <a href="https://www.jstage.jst.go.jp/article/jsaisigtwo/2025/DC-001/2025_03/_pdf/-char/ja">jstage.jst.go.jp</a></p><p>● <strong>[5]</strong> <a href="https://medium.com/@sdsankolli_44118?source=post_page---byline--5c9ae2acf5b4---------------------------------------">Sankolli</a>, S. <em>The Agentic Data Stack &amp; Playbook</em> (Dec. 2025). Available at: <a href="https://medium.com/@sdsankolli_44118/the-agentic-data-stack-playbook-5c9ae2acf5b4">Medium</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a7d787d2cf16" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Enterprise data intelligence use cases (3/n)]]></title>
            <link>https://loysbelleguie.medium.com/enterprise-data-intelligence-use-cases-3-n-3a97a63c899d?source=rss-dcbd37cb0d2e------2</link>
            <guid isPermaLink="false">https://medium.com/p/3a97a63c899d</guid>
            <category><![CDATA[data-intelligence]]></category>
            <category><![CDATA[graphrag]]></category>
            <category><![CDATA[mcps]]></category>
            <category><![CDATA[metadata]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Loys Belleguie]]></dc:creator>
            <pubDate>Fri, 05 Sep 2025 00:08:09 GMT</pubDate>
            <atom:updated>2025-09-05T00:08:09.685Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Using MCP to connect AI agents (LLM) to multiple data sources</blockquote><h3>Introduction</h3><p>In the previous articles, I explored several tools that are becoming mandatory in the realm of <strong>Enterprise Data Intelligence</strong> use cases: Knowledge Graph (GraphRAG), AI agents Framework, etc. Here, pushed by one of the latest hypes (!) of the industry, I will discuss (and test) how the <strong>Model Context Protocol</strong> (MCP) can help bridging the gap between data and AI agents (LLM), be it to explore the <strong>Semantic Layer </strong>(metadata) or creating <strong>Data Products</strong> by accessing directly the data sources in DWH for example.</p><h4>Briefly, what is MCP?</h4><p>This article does not aim to provide a full technical description of MCP; instead, I will only highlight the aspects most relevant for the following discussion (further details are available in the references). The Model Context Protocol (MCP) is an open standard that enables large language models to interact with external tools, APIs, and data sources in a structured and governed way. According to <a href="https://en.wikipedia.org/wiki/Model_Context_Protocol">Wikipedia</a> and <a href="https://docs.anthropic.com/en/docs/mcp">Anthropic’s documentation</a>, MCP defines a lightweight server–client model where services (e.g., databases, knowledge graphs, or SaaS APIs) expose capabilities that LLM agents can call as tools. This avoids embedding every connector directly into the model, while ensuring observability and access control. For our purposes, MCP serves as the control plane that bridges LLM-based agent frameworks (such as Claude Desktop) with enterprise catalogs and <a href="https://neo4j.com/developer/genai-ecosystem/model-context-protocol-mcp/">Neo4j-powered knowledge graphs</a>, thereby grounding AI outputs in validated enterprise knowledge.<br>Note that here I am focusing on the MCP connection to the graph database but we could also connect to the Quollio metadata catalog directly via APIs.</p><h3>Why using MCP in LLM agent frameworks?</h3><p>In the previous post, I described an <a href="https://medium.com/@loysbelleguie/an-implement-of-an-agentic-ai-framework-powered-by-a-knowledge-graph-2-n-24319275291f">LLM Agents framework powered by a GraphRAG</a>. The agents were designed as either call an LLM or directly access the Knowledge Graph, through the tools defined in each agent.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/699/1*i3SrF860-R7jhqsCEpbnnw.png" /><figcaption>User interaction with the KG powered AI agent framework</figcaption></figure><p>As a matter of fact, this is conceptually identical to using MCP to connect with data and another LLM agent framework such as <a href="https://claude.ai/download">Claude desktop</a>. <br>Using Claude Desktop (CD) as the AI agent framework, equipped with an MCP server connecting to the KG, we get the following diagram.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5eLZNmtH9bLD-txQ4BCeWg.png" /><figcaption>Using MCP to connect Claude Desktop (Sonnet) with metadata Knowledge Graph</figcaption></figure><h4>Connection to the Graph database</h4><p>Since I am using Neo4j as the main graph database in this implementation, I used the MCP servers published by Neo4j labs or the community: <a href="https://neo4j.com/developer/genai-ecosystem/model-context-protocol-mcp/">https://neo4j.com/developer/genai-ecosystem/model-context-protocol-mcp/</a>. <br> I also tweak the code a bit, adding some specific functions like for example an <strong>end-to-end lineage exploration function</strong> that I already included in my own agent tools. This avoids the Claude LLM part to spend unnecessary time, trying to reinvent the wheel for specific questions such as “Give me the upstream lineage down to table X, and grab all tags, as well as the schema each table belongs along the path”.<br>The implementation is straightforward: just download the code and add the configuration file to CD as follows.</p><ul><li><strong>Shell: install first mcp-neo4j-cypher</strong><br> git clone <a href="https://github.com/neo4j-contrib/mcp-neo4j.git">https://github.com/neo4j-contrib/mcp-neo4j.git</a> <br> cd mcp-neo4j/servers/mcp-neo4j-cypher <br> rm -rf .venv ; uv venv ; source .venv/bin/activate ; uv pip install -e .</li><li>Claude Desktop config file (json):</li></ul><pre>{<br>    &quot;mcpServers&quot;: {<br>        &quot;neo4j&quot;: {<br>            &quot;command&quot;: &quot;/opt/homebrew/bin/uv&quot;,<br>            &quot;args&quot;: [<br>                &quot;--directory&quot;,<br>                &quot;/Users/LoysBelleguie/dev/MCP_servers/mcp-neo4j/servers/mcp-neo4j-cypher/src&quot;,<br>                &quot;run&quot;,<br>                &quot;mcp-neo4j-cypher&quot;,<br>                &quot;--db-url&quot;,<br>                &quot;neo4j://localhost:7692&quot;,<br>                &quot;--username&quot;,<br>                &quot;neo4j&quot;,<br>                &quot;--password&quot;,<br>                &quot;Poipoi99!&quot;<br>            ]<br>        },<br> (other servers follows)<br>}</pre><p>Obviously, the command “ — db-url”, “neo4j://localhost:7692” implies that the KG server be ready before starting the CD app. Restart then Claude Desktop and we can verify that the MCP server is correctly connected.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/654/1*kzpmjz6tzpTJ5-wN4XzlXw.png" /><figcaption>MCP server connection setup on Claude Desktop</figcaption></figure><h4>Testing Neo4j Knowledge Graph — MCP — Claude LLM scenario</h4><p>We start with the same “metadata lake” scenario as in the previous article: the data in the KG contains (1) a <em>carbon copy</em> of the whole database of the Quollio data catalog (QDIC) — directly linked to the enterprise business entities, to which we add (2) unstructured data, such as pdf documents or html pages. The latter are parsed, specific terms are extracted (entities) and stored in the KG as business domain glossaries.</p><h4>How Claude (Sonnet model) talks to how KG through MCP?</h4><p>Let’s check the technical functionality of our MCP setup with the original version (no code added). We make a blind test, namely a one-shot prompt without any additional context: Ask Sonnet to explore an enterprise business by only scanning the KG with the MCP server.<br>(real test prompt)</p><p><em>Using your MCP, explore the KG and give a summary of the context of this customer. What kind of business they are dealing with? Based on the documentation loaded under the “knowledge_base” sub-graph, what are their main concerns currently?</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7F9MVIuBET2sQ4tcPaPkBA.png" /><figcaption>One-shot prompt test to query the Knowledge Graph through MCP from Claude LLM</figcaption></figure><p>The important point here is that the first actions taken by the LLM are:<br>1. List up the tools made available by the MCP server</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/1*yxADzuCNL4nClXF7DJCgXw.png" /></figure><p>2. Check the schema</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/756/1*372PQg16hVA5JHvrAobarA.png" /></figure><p>The KG schema query was not instructed by the prompt or hard coded somewhere in the workflow, but it is the result of the MCP server function design and LLM inference capability. The former acts as a tool of an AI agent which nudges the LLM to check the schema through the instruction below. The latter have likely been trained on many database interaction examples where schema exploration precedes querying, making this a learned behavior pattern rather than explicit code instruction.</p><pre>@mcp.tool(name=namespace_prefix+&quot;get_neo4j_schema&quot;, <br>             ...<br>    async def get_neo4j_schema() -&gt; list[ToolResult]:<br>        &quot;&quot;&quot;<br>        List all nodes, their attributes and their relationships to other nodes in the neo4j database.<br>        This requires that the APOC plugin is installed and enabled.<br>        &quot;&quot;&quot;<br><br>        get_schema_query = &quot;&quot;&quot;<br>        CALL apoc.meta.schema();<br>        &quot;&quot;&quot;</pre><p>The analysis continues with several <strong>LLM generated queries</strong> to explore the graph in a seemingly rational way.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IlrwH7B38sEGzL4uQYqRaQ.png" /><figcaption>Excerpt of the Chain of Thoughts of Claude LLM using MCP-KG connection</figcaption></figure><p>The Chain of Thoughts (CoT) is rather convincing. It creates relevant Cypher queries and differentiates between the knowledge base built out of the parsed documents (unstructured data) and the metadata of the source data.<br>The aim of this experiment is only to test and validate the MCP setup; the final report is a little bit off target (it infered that the database was from a company within the EU!), but few more context lines in the prompt would easily correct this.</p><h3>Closing the loop with the connection to the data source</h3><p>The previous scenario focused on a “generic” user exploring the enterprise data landscape from the metadata or semantic layer, questioning an AI chatbot (Anthropic Claude) that was connected under the hood to a knowledge base through an MCP server.</p><p>We then turn to the next MCP use case: the <em>user</em> is now a <em>business analyst</em>. He has found the relevant data sets from the previous metadata search and is going to switch to the source data in Snowflake to build a dashboard.<br>In this case, the LLM does not create queries to a graph db, i.e. “Cypher” queries for Neo4j. It will create plain old SQL queries or, in other words, produce an<em> </em>SQL command out of a CoT initiated by a question in natural language, something usually called “<em>text2SQL”</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QpTTVaYXr5xCm4eZMGf21A.png" /><figcaption>MCP connection to Snowflake database from Claude LLM</figcaption></figure><p>I tested the concept with a simple Snowflake MCP connector (<a href="https://github.com/davidamom/snowflake-mcp.git">https://github.com/davidamom/snowflake-mcp.git</a>). The process is, as expected, very similar to the previous case using the Neo4j Knowledge Graph, the SF MCP server is now sending SQL queries directly to Snowflake and return the data to Claude Sonnet LLM model.</p><p>Using the well-established benchmark data set TCP-H, we can compare the answer given by Claude to one of the business cases listed in the TCP-H documentation (<a href="https://www.tpc.org/tpc_documents_current_versions/pdf/tpc-h_v2.17.1.pdf">https://www.tpc.org/tpc_documents_current_versions/pdf/tpc-h_v2.17.1.pdf</a>, question Q2, page 30):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/1*gtiyVoLc1ZL2wvHBDcrxwg.png" /><figcaption>One-shot prompt to query Snowflake through MCP from Claude LLM</figcaption></figure><p>As before, the LLM starts by looking at the schemas, then scans each table to explore the available data. It returns a report with SQL commands to build the data products. I asked to just create one single SQL file with some explanation so that I could run it directly in my Snowflake account. The SQL commands ran without glitches, but the opposite would be strange as the LLM has already validated the result for us!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FtABCSNXt_KTovr45Wz4WQ.png" /><figcaption>Left: save the SQL created by Claude LLM | Right: run the SQL on Snowflake</figcaption></figure><h4>Catching a data quality issue with MCP/LLM</h4><p>Interestingly, during the last session with the TPC-H data set on Snowflake, as the report was being displayed on the screen, I just noticed some data that seemed wrong: the data assigns country code “+33” to the UK, although it is the code for France.<br>I wondered if it was another hallucination from the LLM and asked:<br> “Check the SQL where you found that Supplier#000005359 (UK) has a phone number with +33 which is actually France… Did you invent those results? Show me the SQL query and the result as a readable table.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tNlm5xwux5p-O8f7xBQ9EQ.png" /><figcaption>Catching a data issue from the SQL generated by Claude LLM/MCP</figcaption></figure><p>The answer from Sonnet speaks for itself, we just got a sort of <em>Data Quality</em> check powered by LLM/MCP!</p><h3>Final words</h3><p>This article showed how a GraphRAG powered AI agent framework can be <strong>modified to use MCP</strong> servers in place of a self-built AI Agent framework and tools. With the ease of use of the available MCP servers it becomes conceptually simpler to visualize the whole setup:</p><ol><li>The users have access to the metadata and semantic layer for data discovery, exploration, semantic analysis etc.</li></ol><p>2. On the other hand, once they have found the <strong>data entry points</strong>, they can switch, in the same LLM session, to the data source, build a<strong> data product</strong> and finalize a business analysis (a dashboard), an AI model, or just store the new data set!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1Zt3c7RX40_OhXmd8jZmjg.jpeg" /><figcaption>Using multiple types of MCP servers to query both the semantic layer (metadata) and the data source (ex. a Snowflake database)</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3a97a63c899d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An implementation of an Agentic AI framework powered by a Knowledge Graph(2/n)]]></title>
            <link>https://loysbelleguie.medium.com/an-implement-of-an-agentic-ai-framework-powered-by-a-knowledge-graph-2-n-24319275291f?source=rss-dcbd37cb0d2e------2</link>
            <guid isPermaLink="false">https://medium.com/p/24319275291f</guid>
            <category><![CDATA[corporate-data]]></category>
            <category><![CDATA[knowledge-graph]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[metadata-management]]></category>
            <dc:creator><![CDATA[Loys Belleguie]]></dc:creator>
            <pubDate>Tue, 25 Mar 2025 06:53:52 GMT</pubDate>
            <atom:updated>2025-03-25T06:56:03.954Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>As we are all experiencing recently, “<em>deepsearch</em>”-like features are ubiquitous in recent Large Language Model (LLM); “<strong><em>AI agent</em></strong>” or “<strong><em>agentic</em></strong>” have become buzz words and seem to be the cornerstone of every innovations in AI. Indeed, these agents provide an additional “reasoning” layer to what was only a question/answer process up to now (cf. the article on <a href="https://note.com/rytmq/n/n39cd819126a9"><em>note</em></a> from our CEO Ryota Matsumoto about this).</p><p>In my previous articles (on <a href="https://medium.com/@loysbelleguie/reflections-on-using-knowledge-graphs-and-llm-in-metadata-management-context-1-n-e1015eb85e2a">Medium</a> and in <a href="https://quollio.com/jp/journal/kr1e6nnjp0xi">Quollio blog</a>), I already discussed about AI agents, and how to use an “<em>agentic framework</em>” with Knowledge Graphs (KG) to extend metadata management systems (a.k.a. data catalogs). Here, I will extend the discussion and focus on how to make AI agents interact with the KG.</p><p>AI agents are built with the capacity to search information outside their pre-trained scope. The concept is comparable with, for example, the <a href="https://modelcontextprotocol.io/introduction">Model Context Protocol </a>(MCP) recently proposed by Anthropic (<a href="https://www.anthropic.com/news/model-context-protocol">https://www.anthropic.com/news/model-context-protocol</a>), where we can add many different information sources to feed the LLM. While searching additional information directly on the Internet is fine for general topics, this should be carefully monitored when working with corporate data, which means with data domains where confidentiality may be mandatory or at least controllable.</p><p>Our KG is used as a customized and fully controlled knowledge base that can be <em>interrogated</em>, i.e. queried in the traditional SQL sense, but also searched with keywords or natural language questions or explored like a network. Therefore, contrary to a generic <em>one-size-fits-all</em> protocol like MCP, we design a customized solution to fulfill the corporate governance requirements.</p><h3>The Agentic-Graph RAG framework</h3><p>Our framework can be decomposed in 3 steps:</p><ul><li><strong>Step 1</strong>: Create a knowledge base from 3 main sources of data:<br>1. The core metadata lake from the <a href="http://quollio.com/">Quollio metadata management engine</a> (QDIC),<br>2. The company internal documents (a priori confidential), with specific term glossaries, and<br>3. Open source documents and business glossaries, etc.</li></ul><p>The important point of this process is that all data, be it pure metadata, glossaries in csv, html pages or pdf files, are parsed and loaded in a common graph database space, to create the main knowledge graph. The documents are analyzed and entities extracted to build additional semantic glossaries (Figure 1).</p><figure><img alt="Figure 1 Knowledge BaFigure 1 Knowledge Base Building (Step 1)se Building (Step 1)" src="https://cdn-images-1.medium.com/max/1024/1*SFE0VYsgEV822dowIwd04A.png" /><figcaption><em>Figure 1 Knowledge Base Building (Step 1)</em></figcaption></figure><ul><li><strong>Step 2</strong>: Once the KG is built, we run several procedures and algorithms to enrich the knowledge base with new relationships based on several similarity metrics (described in the following), attach glossary terms automatically or create domains (Figure 2-left). This process is semi-automatic in the sense that some monitoring (a.k.a <em>human-in-the-loop</em>) remains necessary to avoid unavoidable inaccuracies even with the current state-of-the-art LLM models (this will obviously improve in the near future!).</li><li><strong>Step 3</strong>: This it the core of the system: the agentic framework that will act as the interface between our knowledge base and the end-user (Figure 2-right). The details of the implementation is developed in the next section, but in summary, we create a series of agents equipped with specific tools in 2 broad categories: (1) the KG queries and search and (2) the LLM contextual analyses.</li></ul><figure><img alt="Figure 2 Step (2) build new relationships within KG and (3) end-user interaction with KG using agentic LLM" src="https://cdn-images-1.medium.com/max/1024/1*96-O1bP-vfYbx1DqyqnG0Q.png" /><figcaption><em>Figure 2 Step (2) build new relationships within KG and (3) end-user interaction with KG using agentic LLM</em></figcaption></figure><h3>The implementation</h3><h4>The knowledge base building</h4><blockquote>Load data into the graph database (figure 1)</blockquote><p>The core metadata is exported from <a href="https://guide.quollio.com/hc/ja/articles/41563381871001-%E3%83%87%E3%83%BC%E3%82%BF%E3%82%92%E7%90%86%E8%A7%A3%E3%81%99%E3%82%8B">QDIC</a>. The technical aspects were already developed in the previous <a href="https://loysbelleguie.medium.com/reflections-on-using-knowledge-graphs-and-llm-in-metadata-management-context-1-n-e1015eb85e2a">article</a>, so we won’t repeat it again here, we just reemphasize that the process is only a direct mapping of the QDIC “metadata database” into a graph database. Then, we add any available glossaries, we also include <strong>semi-structured data</strong>, such as web pages (HTML) or documents in PDF, which are parsed and analyzed: this brings invaluable contextual information within the knowledge base.</p><p>Let’s detail the document parsing process and load into the KG. We only used two types of formats, HTML pages and PDF, for documents that we want to be part of our local KG, in contrast with the information that LLM agent would be accessing <em>on the fly</em> on the Internet. The former would be for example the company confidential information, or, open source data that is deemed particularly important for the business use case, and from which we may want to extract specific entities and their definitions.</p><p>The parsing process decomposes the documents in sections and chunks, following their original structure, then loads them as nodes in the graph. In addition, we extract entities and store them in a business glossary. We use <a href="https://www.llamaindex.ai/llamaparse">LLamaParse</a> for this PoC, but any other solutions could be used. Our choice was motivated by the quality, the limited free access to the API (about 3000 pages per month depending on the options) and ease of use. The entity extraction is done with an LLM that we provide; we tested several solutions: AzureOpenAI, Anthropic Sonnet 3.5 (on AWS), Gemini etc, and chose Azure, for the sake of simplicity (the focus is more on the knowledge base concept rather than on the specific tools effectively employed in this PoC).</p><pre># reduced code for better lisibility<br>from llama_parse import LlamaParse<br>from llama_index.core import SimpleDirectoryReader<br>parser = LlamaParse(<br>    api_key=os.environ.get(&#39;LLAMA_API_KEY&#39;),<br>    result_type=&quot;markdown&quot;,  <br>    split_by_page=False,<br>    num_workers=4, <br>    verbose=True,<br>    language=&quot;en&quot;,  <br>    parsing_mode=&quot;full&quot;<br>)<br>documents = SimpleDirectoryReader(input_files=input_files, <br>              exclude_empty=True,filename_as_id=True,<br>              file_extractor=file_extractor).load_data(show_progress=True)<br><br># transform the markdown to node objects<br>from llama_index.core.node_parser import MarkdownElementNodeParser<br>node_parser = MarkdownElementNodeParser(llm=llm, num_workers=8)<br>nodes = node_parser.get_nodes_from_documents(documents)<br>base_nodes, objects = node_parser.get_nodes_and_objects(nodes)</pre><blockquote>Deploying graph data science algorithms and NLP techniques to extract hidden information from the graph.</blockquote><p>This part is, with the <em>agentic framework</em> described in the next section, the main contribution of this article.</p><p>The initial state of our metadata lake, after uploading to the graph database (Figure 1), consists of yet unrelated domains of assets: the company data metadata per se (including existing tags), the business glossaries, and the parsed documentations. The next step will be to explore the semantic relationships between all these assets (the nodes in the KG), and create a new network of associations (edges). Each edge will be calculated with a metric specific to a topic: for example, evaluate if two assets have similar descriptions and therefore may be grouped in the same business objects category. In the same way, we will look for entities (those extracted from the aforementioned documents) that could be semantically linked to metadata, and create new classifications.</p><blockquote>Lexical and vector similarities</blockquote><p>There are two main approaches to calculate semantic similarities between assets: (1) the approach based on statistical commonalities at word level, with for example, <a href="https://en.wikipedia.org/wiki/Jaccard_index"><em>Jaccard</em></a> or <a href="https://en.wikipedia.org/wiki/Tf%E2%80%93idf"><em>TF-IDF</em></a> metrics and (2) methods based on vector <em>embeddings</em>. The former are purely lexical and do not account for synonyms, paraphrases, or context. The latter captures a more holistic meaning of the texts, table descriptions, or entity names, and will be the preferred choice for general similarity search. However, the lexical metrics can still be beneficial in token-based comparison, for instance comparing the set of attached tags or common column names.</p><blockquote>KG and/or vector database ?</blockquote><p>As a side note, as the project is still very limited in scale (both the metadata lake and the number of unstructured data are relatively small), we won’t use any vector database. All embeddings are stored as attributes in the KG nodes and we use Neo4j plugins for all the vector related calculations. In a product implementation, we may opt for a hybrid solution with a dedicated vector database matching the KG nodes. Therefore, in our implementation, based on Neo4j and AzureOpenAI API, the asset embeddings are calculated for example as follows:</p><pre>from langchain_neo4j import Neo4jVector<br>similarityType = ‘Table’<br>attributes = [&#39;physical_name&#39;, &#39;logical_name&#39;, &#39;description&#39;,&#39;ddl_comment&#39;]<br>vector_index = Neo4jVector.from_existing_graph(<br>        embedding=current_app.embeddings_model,<br>        url=current_app.config[&#39;NEO4J_DB_URI&#39;],<br>        username=current_app.config[&#39;NEO4J_DB_USERNAME&#39;],<br>        password=current_app.config[&#39;NEO4J_DB_PWD&#39;],    <br>        index_name=f&#39;{similarityType}VectorIndex&#39;,<br>        node_label=similarityType,<br>        text_node_properties=attributes,<br>        embedding_node_property=&#39;embedding&#39;<br>        )</pre><p>In this case, we choose a set of attributes to extract the semantic information from the asset. In other cases, we may add other technical attributes to extend the vector scope.</p><blockquote>Similarity calculations</blockquote><p>For the graph algorithm we test <em>Jaccard </em>similarity for lexical matching of tags and schema attributes. We could also refine the results with TF-IDF but it doesn’t fundamentally change the concept. In addition, <em>cosine similarity</em> based on LLM generated embedding (vector) is used to align semantic meanings of asset descriptions. The implementation is straightforward with a KG, as shown in the code excerpt below.</p><pre>MATCH (t1:Table), (t2:Table)<br>WHERE t1.id &lt; t2.id<br><br>....<br><br>WITH t1, t2, <br>     size(apoc.coll.intersection(tags1, tags2)) AS sharedTags,<br>     size(apoc.coll.intersection(columns1, columns2)) AS sharedColumns,<br>     size(apoc.coll.intersection(schemas1, schemas2)) AS sharedSchemas,<br>     size(apoc.coll.union(tags1, tags2)) AS allTags, <br>     size(apoc.coll.union(columns1, columns2)) AS allColumns,<br>     size(apoc.coll.union(schemas1, schemas2)) AS allSchemas<br>with t1, t2, allTags, sharedTags, sharedColumns, sharedSchemas, allColumns, allSchemas,<br>     CASE WHEN allTags = 0 THEN 0 ELSE sharedTags / toFloat(allTags) END AS tagSimilarity,<br>     CASE WHEN allColumns = 0 THEN 0 ELSE sharedColumns / toFloat(allColumns) END AS columnSimilarity,<br>     CASE WHEN allSchemas = 0 THEN 0 ELSE sharedSchemas / toFloat(allSchemas) END AS schemaSimilarity<br><br>WITH t1, t2, tagSimilarity, columnSimilarity, schemaSimilarity, sharedTags, sharedColumns, sharedSchemas,<br>     ($alpha * tagSimilarity + $beta * columnSimilarity <br>          + $gamma * schemaSimilarity) AS similarityScore<br><br>OPTIONAL MATCH (t1)-[r1:SIMILAR_TO]-&gt;(t2)<br><br>MERGE (t1)-[r:SIMILAR_TO {similarity: similarityScore}]-&gt;(t2)<br>SET r = coalesce(r, r1)<br>SET r.sharedTags = sharedTags,<br>    r.sharedColumns = sharedColumns,<br>    r.sharedSchemas = sharedSchemas,<br>    r.tagSimilarity = tagSimilarity,<br>    r.columnSimilarity = columnSimilarity, <br>    r.schemaSimilarity = schemaSimilarity<br><br>RETURN t1.physical_name, t2.physical_name, r.similarity as score, <br>r.tagSimilarity, r.columnSimilarity, r.schemaSimilarity order by score desc</pre><p>A typical calculation of <em>Jaccard</em> similarities will look like the following table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Cp04yPbJ167nhtDAWEQzvA.png" /><figcaption>Lexical “Jaccard” similarity evaluation using a combination of items: common tags, columns and schema.</figcaption></figure><p>Next, the vector embedding similarities are calculated as follows. It is worth noting, that this code is not optimized as the number of pairs is growing as ~O(n²), though this is not really an issue for metadata lakes (remember that metadata are only a fraction of the data their represent!).</p><pre>from neo4j import GraphDatabase<br>query = f&quot;&quot;&quot;MATCH p=(t1)-[:SIMILARITY]-&gt;(t2)<br>            WHERE t1.id &lt; t2.id <br>            OPTIONAL MATCH (d1:Database)-[]-(s1:Schema)-[]-(t1)<br>            OPTIONAL MATCH (d2:Database)-[]-(s2:Schema)-[]-(t2)<br>            WHERE …<br>            call(t1,t2) {{<br>                return<br>                gds.similarity.cosine(t1.embedding,t2.embedding) AS similarity  <br>            }}<br>            MERGE (t1)-[r:SIMILARITY]-(t2)  <br>            SET r.similarity_level =  toFloat(apoc.text.format(&quot;%.2f&quot;, [similarity]))<br>        &quot;&quot;&quot;<br>with GraphDatabase.driver(NEO4J_DB_URI, auth=AUTH) as neo4jDriver:<br>     records, summary, keys = neo4jDriver.execute_query(query, <br>                    parameters_=parameters,<br>                    database_=self.neo4jDatabase,</pre><p>A typical vector similarity calculation is shown on the following figure. The left is done directly in the KG browser interface (Neo4j), while the right part is the python call to the KG.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LwXXrvBb4x368xGm3LLqHA.png" /><figcaption>Figure 3 Metadata linked by cosine similarity.</figcaption></figure><blockquote>Entity extraction example and automatic link with metadata</blockquote><p>The following example shows how metadata can be automatically classified using business terms (entities), imported from documents, so that, a data set can be linked with the a specific documentation, e.g. regulation, report etc, based on semantic relationships.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/1*oRf8PQpXOvi3RMIyYptIRQ.png" /><figcaption>Figure 4: Entity extraction from documents, linked to Tables based on vector similarity score</figcaption></figure><h3>The agentic framework</h3><p>Let’s now discuss in more details how the AI agents can be built to interact in a controllable manner with the KG.</p><blockquote>ReAct Agent concept</blockquote><p>As we chose LangGraph as the base framework for this PoC, we use all the tools available, knowing that many other solutions are available, e.g. CrewAI, and focus on the concepts more than on product level implementation. For example, our agent are based on the pre-built <em>ReAct</em> type (<a href="https://python.langchain.com/docs/tutorials/agents/">https://python.langchain.com/docs/tutorials/agents/</a>), as this fulfills our requirements and saves a lot of unnecessary coding. Actually, this package is supposed to be based on the paper “<em>ReAct: Synergizing Reasoning and Acting in Language Models</em>” (<a href="https://arxiv.org/abs/2210.03629">https://arxiv.org/abs/2210.03629</a>) from Princeton and Google Research.</p><blockquote>Agent types and tools</blockquote><p>Each agent acts as an independent entity, using predefined tools that will each take the output from the previous agent, perform an action, using the <em>chosen</em> tool, and feed the result of this tool to the next agent. It is worth noting that the all the reasoning happens that this stage: the agent will decide which tool or LLM to call. In our experiment, we defined 3 main categories:<br>1. A guardrail to select only the request that are in scope, <br>2. a search agent with a KG query tool to fetch certain categories of assets or paths (ex. lineage) in the graph, or, make <strong>natural language search</strong> within the KG,<br>3. then, an analyzer focusing on LLM calls.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/851/1*Pix8ytAAREtICz2Ve3OClA.png" /><figcaption>Figure 5 : Example of AI Agent types used in this experiment</figcaption></figure><blockquote>How can this concept be implemented? Let’s look at some code based on LangChain.</blockquote><p>The agents use the LangGraph pre-built ReAct function, where the tools and some additional prompt can be assigned. Note the definition of the tool <em>semantic_search_tool </em>(in the search agent node): the first comment <em>“This tool performs semantic vector search using natural language queries</em>.<em>…”</em> is like a prompt that instruct the tool what to do, in this case, launch a search in the KG and format the output so that the next agent can process it appropriately.</p><pre>from langgraph.prebuilt import create_react_agent<br>def _init_agents(self):<br>    &quot;&quot;&quot;Initialize all agents for the workflow.&quot;&quot;&quot;<br>    # Create analyze_prompt_agent<br>    self.log(&quot;Creating ReAct Agent: analyze_prompt_agent&quot;)<br>    self.analyze_prompt_agent = create_react_agent(<br>        self.chatOpenAImodel,<br>        tools=[self.analyze_prompt_tool],<br>        state_modifier=self._make_system_prompt(&#39;analyze_prompt_agent_prompt&#39;),<br>    )<br>    <br>    # Create search_assets_agent<br>    self.log(&quot;Creating ReAct Agent: search_assets_agent&quot;)<br>    self.search_assets_agent = create_react_agent(<br>        self.chatAIModel,<br>        tools=[self.search_assets_tool, self.semantic_search_tool, self.similarity_search_tool],<br>        state_modifier=self._make_system_prompt(&#39;search_assets_agent_prompt&#39;),<br>    )<br>...<br><br>@tool<br>def semantic_search_tool(query: str, min_score = 0.5, limit = 10) -&gt; str:<br>    &quot;&quot;&quot;<br>    This tool performs semantic vector search using natural language queries.<br>    Specify the minimum level of similarity with min_score.<br>    Specify the minimum number of results with limit.<br>    <br>    Converts your query into vector embeddings to find conceptually similar assets in the <br>    Knowledge Graph, even when exact keywords aren&#39;t present.<br>    <br>    Ideal for:<br>    - Complex domain-specific queries about regulations or standards<br>    - Finding assets related to business concepts rather than exact terms<br>    - Questions involving specialized terminology<br>    <br>    Returns JSON with semantically matched assets and relevance scores.<br>    &quot;&quot;&quot;<br>    self.debug_print(f&quot;\n ======= in semantic_search_tool ======== \n Query: {query}&quot;)<br>    self.log(f&quot;\n ======= in semantic_search_tool ======== \n Query: {query}&quot;)<br><br>    try:<br>        # Default parameters for semantic search<br>        asset_types = [&#39;Table&#39;, &#39;Column&#39;, &#39;Schema&#39;, &#39;Entity&#39;, &#39;Tag&#39;]<br>        <br>        # Execute semantic search using the KG&#39;s get_semantic_neighbors function<br>        # Note: No embedding model is passed, so it will use the built-in fallback<br>        semantic_results = self.qdcNeo4jAssetDiscovery.get_semantic_neighbors(<br>            text_query=query,<br>            asset_types=asset_types,<br>            min_score=min_score,<br>            limit=limit,<br>            embedding_model=self.embeddingModel<br>        )<br>        # Process results into the expected format<br>        assets_found = []<br>        for result in semantic_results:<br>            # Extract and clean up fields<br>            asset = {<br>                &quot;id&quot;: result.get(&quot;id&quot;, &quot;unknown&quot;),<br>                &quot;name&quot;: result.get(&quot;name&quot;, &quot;unnamed&quot;),<br>                &quot;asset_type&quot;: result.get(&quot;asset_type&quot;, &quot;unknown&quot;),<br>                &quot;relevance_score&quot;: result.get(&quot;score&quot;, 0.5),<br>                &quot;description&quot;: result.get(&quot;description&quot;, &quot;&quot;)<br>            }<br>            <br>            # Add tags if available<br>            if &quot;tags&quot; in result and result[&quot;tags&quot;]:<br>                asset[&quot;tags&quot;] = result[&quot;tags&quot;]<br>                <br>            assets_found.append(asset)<br>        <br>        # Create structured result<br>        result = {<br>            &quot;query&quot;: query,<br>            &quot;assets_found&quot;: assets_found,<br>            &quot;total_count&quot;: len(assets_found),<br>            &quot;search_strategy&quot;: &quot;VECTOR_SIMILARITY&quot;,<br>            &quot;reason&quot;: f&quot;Found {len(assets_found)} semantically related assets&quot;<br>        }<br>           ...</pre><p>Then the agent framework per se, is created as follows. Note the definition of the “nodes” to which agents are attached, and the “edges” where messages originating from one node are processed and directed to the appropriate node (or END node).</p><pre>def _create_workflow_graph(self):<br><br>try:<br>  # Define the graph<br>  workflow = StateGraph(MessagesState)<br>  <br>  # Add nodes<br>  workflow.add_node(&quot;analyze_prompt&quot;, self._analyze_prompt_node)<br>  workflow.add_node(&quot;search_assets&quot;, self._search_assets_node)<br>  workflow.add_node(&quot;discover_linked_assets&quot;, self._discover_linked_assets_node)<br>  workflow.add_node(&quot;analyzer&quot;, self._analyzer_node)<br><br>  # Add conditional edges<br>  workflow.add_conditional_edges(<br>      &quot;analyze_prompt&quot;,<br>      self.router,<br>      {&quot;continue&quot;: &quot;search_assets&quot;, END: END},<br>  )<br>  workflow.add_conditional_edges(<br>      &quot;search_assets&quot;,<br>      self.router,<br>      {&quot;continue&quot;: &quot;discover_linked_assets&quot;, END: END},<br>  )<br>  workflow.add_conditional_edges(<br>      &quot;discover_linked_assets&quot;,<br>      self.router,<br>      {&quot;continue&quot;: &quot;analyzer&quot;, END: END},<br>  )<br>  workflow.add_conditional_edges(<br>      &quot;analyzer&quot;,<br>      self.router,<br>      {&quot;new_assets&quot;: &quot;search_assets&quot;, <br>      &quot;continue&quot;: &quot;analyzer&quot;,<br>      END: END},<br>  )<br><br>  # Add starting edge<br>  workflow.add_edge(START, &quot;analyze_prompt&quot;)<br>  <br>  # Compile the graph<br>  self.WF_graph = workflow.compile()<br>  ...<br><br>def _analyze_prompt_node(self, state: MessagesState) -&gt; MessagesState:<br>  &quot;&quot;&quot;Process the analyze_prompt node in the workflow.&quot;&quot;&quot;<br>  try:<br>      result = self.analyze_prompt_agent.invoke(state)<br>      <br>      # Try to extract structured data from the response<br>      last_message = result[&quot;messages&quot;][-1]<br>      content = last_message.content if hasattr(last_message, &#39;content&#39;) else str(last_message)<br>      <br>      # Try to extract JSON data if present<br>      try:<br>          # Look for JSON structure in the content<br>          json_match = re.search(r&#39;```json\s*(.*?)\s*```&#39;, content, re.DOTALL)<br>          if json_match:<br>              json_str = json_match.group(1)<br>              structured_data = json.loads(json_str)<br>              self.log(f&quot;Extracted structured data from analyze_prompt: {json.dumps(structured_data, indent=2)}&quot;)<br>          else:<br>              ....<br>      except Exception as e:<br>          ....      <br>      # Wrap in a human message for the next agent<br>      result[&quot;messages&quot;][-1] = HumanMessage(<br>          content=result[&quot;messages&quot;][-1].content, name=&quot;analyze_prompt&quot;<br>      )<br>      self.log(f&quot;====== analyze_prompt_node ===== {result[&#39;messages&#39;][-1].content}&quot;)<br>      return {<br>          &quot;messages&quot;: result[&quot;messages&quot;],<br>      }<br>  except Exception as e:<br>    ...</pre><p>The agent workflow is shown in the following diagram; this is just a pictorial representation of the nodes (the <em>display_WF_graph</em> function in the code below) and is not related with the KG introduced earlier, and doesn’t show the tools used by each node, which is actually the most important information!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/422/1*DVULvbI0OSA9pfA2I-ZNvw.png" /><figcaption>Figure 6 : AI Agent framework diagram</figcaption></figure><p>Anyway, let’s make some simple tests:</p><h4>Search about assets that could be linked with compliance issues:</h4><pre>def main():<br>    neo4j_connector, neo4j_queries, asset_discovery = setup_neo4j()<br>    chat_model, embeddings_model = setup_azure_openai()<br><br>    # Create the AgenticFramework<br>    domain_scope = &quot;&quot;&quot;The agentic framework is designed to work with a knowledge graph that contains information about data assets,<br>    data sources, and relationships between them, based on metadata and other available information, <br>    such as semi-structured and unstructured data from internal and external sources.<br>    The business domain is a global manufacturing and supply chain management, covering any industries with global value chains.<br>    In any case, the focus is on exploring data assets, their relationships, in a global value chain, <br>    and the impact of regulations, disruptions risks, as well as identifying opportunities for improvement and innovation.<br>    &quot;&quot;&quot;<br><br>    agentic_framework = AgenticFramework({<br>        &quot;neo4jConnector&quot;: neo4j_connector,<br>        &quot;qdcNeo4jQueries&quot;: neo4j_queries,<br>        &quot;qdcNeo4jAssetDiscovery&quot;: asset_discovery,<br>        &quot;chatAIModel&quot;: chat_model,<br>        &quot;embeddingModel&quot;: embeddings_model,<br>        &quot;domain_scope&quot;: domain_scope,<br>        &quot;debug&quot;: True<br>    })<br><br>    # Display the workflow graph<br>    agentic_framework.display_WF_graph()...<br>...<br>    combined_query = &quot;&quot;&quot;Find tables related to product compliance and show <br>    their relationships with regulatory data.<br>    Focus on tables that might contain information relevant for EU regulations.&quot;&quot;&quot;<br>    combined_result = run_test_query(agentic_framework, combined_query)</pre><blockquote>Results:</blockquote><p>First, the prompt parsing node correctly identifies the task to be performed</p><pre>==== STARTING WORKFLOW (ID: 20250325_135332) ====<br><br>&gt;&gt;  INFO: Query: Find tables related to product compliance and show their <br>relationships with regulatory data. Focus on tables that might contain <br>information relevant for EU regulations.<br>&gt;&gt;  INFO: Extracted embedded JSON from parse_prompt: {<br>  &quot;primary_search_targets&quot;: [<br>    &quot;Tables related to product compliance&quot;<br>  ],<br>  &quot;secondary_search_terms&quot;: [<br>    &quot;EU regulations&quot;,<br>    &quot;regulatory data&quot;<br>  ],<br>  &quot;question_intent&quot;: &quot;Identify tables that contain information about product <br>compliance and their relationships with regulatory data, specifically focusing <br>on EU regulations.&quot;,<br>  &quot;search_strategy&quot;: &quot;VECTOR_SIMILARITY&quot;,<br>  &quot;reasoning&quot;: &quot;The query involves complex concepts like &#39;product compliance&#39; <br>and &#39;EU regulations&#39;, which require understanding the semantic meaning rather <br>than exact terms. VECTOR_SIMILARITY is suitable for finding tables related to <br>these concepts and understanding their relationships.&quot;,<br>  &quot;next_action&quot;: &quot;CONTINUE&quot;<br>}</pre><p>Then, the vector similarity search method is selected as the query is a natural language question. Two assets are identified; more could be searched provided we added some more feedback or tuned the search function parameters. The next node is called with the “CONTINUE” keyword sent to the router.</p><pre>&gt;&gt;  INFO: Processing workflow event 2<br>&gt;&gt;  INFO: Output from node &#39;search_assets&#39;:<br>&gt;&gt;  INFO: [search_assets] HumanMessage from search_assets: ```json<br>{<br>    &quot;search_summary&quot;: &quot;Found tables related to product compliance and EU regulations.&quot;,<br>    &quot;search_strategy_used&quot;: &quot;VECTOR_SIMILARITY&quot;,<br>    &quot;assets_to_explore&quot;: [<br>        {<br>            &quot;id&quot;: &quot;tbl-cac58b386a58dcc29d28d02642fac392&quot;,<br>            &quot;name&quot;: &quot;COMPLIANCE_RISK_FACT&quot;,<br>            &quot;asset_type&quot;: &quot;Table&quot;,<br>            &quot;reason_for_selection&quot;: &quot;This table is related to compliance risk,<br>               which is relevant for understanding product compliance.&quot;,<br>            &quot;semantic_relevance&quot;: true<br>        },<br>        {<br>            &quot;id&quot;: &quot;tbl-8eb532dd50261af64fc0ba42e8f745d2&quot;,<br>            &quot;name&quot;: &quot;REGULATION_DETAILS&quot;,<br>            &quot;asset_type&quot;: &quot;Table&quot;,<br>            &quot;reason_for_selection&quot;: &quot;This table contains details about <br>                  regulations, which is pertinent to EU regulatory data.&quot;,<br>            &quot;semantic_relevance&quot;: true<br>        }<br>    ],<br>    &quot;needs_discovery&quot;: true,<br>    &quot;next_action&quot;: &quot;CONTINUE&quot;<br>}<br>```...</pre><p>The next node <strong>queries the KG</strong> directly for any assets linked to the tables “COMPLIANCE_RISK_FACT” or “REGULATION_DETAILS”, such as assets in the lineage path and any attached assets such as schema, tags etc, but nothing is found that meets the requirements.</p><pre>&gt;&gt;  INFO: Processing workflow event 3<br>&gt;&gt;  INFO: Output from node &#39;discover_linked_assets&#39;:<br>&gt;&gt;  INFO: [discover_linked_assets] HumanMessage from discover_linked_assets: ```json<br>{<br>  &quot;exploration_summary&quot;: &quot;Explored connections for tables related to product<br>   compliance and EU regulations. No additional linked assets were discovered.&quot;,<br>  &quot;discovered_assets&quot;: [],<br>  &quot;next_action&quot;: &quot;CONTINUE&quot;<br>}<br>```...</pre><p>And finally, the last agent <em>analyzer</em> is called to perform the final analysis:</p><pre>&gt;&gt;  INFO: FINAL ANSWER detected in router<br>&gt;&gt;  INFO: Processing workflow event 4<br>&gt;&gt;  INFO: Output from node &#39;analyzer&#39;:<br>&gt;&gt;  INFO: [analyzer] HumanMessage from analyzer: ```json<br>{<br>    &quot;analysis_summary&quot;: &quot;The analysis focused on two tables: <br>&#39;COMPLIANCE_RISK_FACT&#39; and &#39;REGULATION_DETAILS&#39;. <br>The &#39;COMPLIANCE_RISK_FACT&#39; table is relevant as it pertains to compliance risk,<br> which is a critical aspect of product compliance. The &#39;REGULATION_DETAILS&#39; <br>table is pertinent as it contains details about regulations, which are <br>essential for understanding EU regulatory requirements.&quot;,<br>    &quot;relevant_assets&quot;: [<br>        {<br>            &quot;id&quot;: &quot;tbl-cac58b386a58dcc29d28d02642fac392&quot;,<br>            &quot;name&quot;: &quot;COMPLIANCE_RISK_FACT&quot;,<br>            &quot;relevance&quot;: &quot;This table is related to compliance risk, which is <br>            relevant for understanding product compliance.&quot;<br>        },<br>        {<br>            &quot;id&quot;: &quot;tbl-8eb532dd50261af64fc0ba42e8f745d2&quot;,<br>            &quot;name&quot;: &quot;REGULATION_DETAILS&quot;,<br>            &quot;relevance&quot;: &quot;This table contains details about regulations, <br>            which is pertinent to EU regulatory data.&quot;<br>        }<br>    ],<br>    &quot;conclusion&quot;: &quot;The identified tables provide a foundational understanding <br>    of product compliance and regulatory details relevant to EU regulations. <br>    However, no additional linked assets were discovered that could further <br>    elaborate on the relationships between these tables and other regulatory <br>    data.&quot;,<br>    &quot;next_action&quot;: &quot;FINAL ANSWER&quot;<br>}<br>```...<br>&gt;&gt;  INFO: FINAL ANSWER: Workflow execution complete (Processing time: 13.39s)</pre><p>As a sanity check, we asked a question that is out of context, the first <em>Guardrail</em> agent (the parse_prompt) is acting appropriately and cancel the request, as it should:</p><pre>random_query = &quot;&quot;&quot;What do you think about the sakura season in Japan? <br>                  Can you provide some insights?&quot;&quot;&quot;<br>keyword_result = run_test_query(agentic_framework, random_query)...<br><br># Results:<br><br>&gt;&gt;  INFO: <br>==== STARTING WORKFLOW (ID: 20250325_142031) ====<br><br>&gt;&gt;  INFO: Query: What do you think about the sakura season in Japan? Can you <br>provide some insights?<br>&gt;&gt;  INFO: Out of scope query detected - terminating workflow<br>&gt;&gt;  INFO: Processing workflow event 1<br>&gt;&gt;  INFO: Output from node &#39;parse_prompt&#39;:<br>&gt;&gt;  INFO: [parse_prompt] AIMessage from parse_prompt: {&quot;next_action&quot;: &quot;END&quot;, <br>                      &quot;reasoning&quot;: &quot;Query determined to be out of scope&quot;}<br>&gt;&gt;  INFO: FINAL ANSWER: Workflow execution complete (Processing time: 2.39s)</pre><h3>Conclusion</h3><p>In this article, I explored further on AI agent framework including knowledge graph support. Agentic AI is one of the main topic in this year and every day brings new solutions; I therefore continue with the tool kits I started to experiment with last year, and added some more realistic use case.</p><blockquote>So, what are the lessons learned here?</blockquote><p>Using a KG is a very powerful way to organize any available knowledge base, from internal (i.e. confidential) or open sources, and provide the agents access to it. Like the MCP protocol mentioned at the beginning, querying directly the KG with <em>cypher</em> language or searching nodes from vector or lexical similarities can be implemented without real hurdles, and, this is important in enterprise context, can be made fully controllable.</p><blockquote>Use cases</blockquote><p>The agentic framework can, semi-autonomously, explore the knowledge base (KG) to answer a business question within the scope of a precise context, and return a list of relevant data sets. I guess this is one of the basic brick needed to build a real Data as a Product (see this <a href="https://note.com/rytmq/n/n4ebf2fea4fe3">other article</a> from our CEO about DaaP). Indeed, the initial necessary task to make sure a data product gets all the necessary data sets is to be able to find them! This sounds obvious, but if it was so, we wouldn’t spend time trying to build <em>intelligent</em> data catalogs…</p><p>But many other use cases can be envisioned.</p><p>For instance, supply chain disruption issues or compliance along global value chains, are among the most cited use cases, not only in pure manufacturing. Any industry may be at some point or another be confronted with these problems: agents may use graph traversals to reveal how suppliers are connected and how material sourcing indirectly influence product delivery schedule or cost, or help finding the relevant data for Product Carbon Footprint (PCF) calculations through revealed relationships with ERP objects. The algorithms that can be used in the KG can expose non-obvious dependencies, such as shared compliance risks between geographically dispersed suppliers or correlations between logistics data and carbon emissions. Unlike traditional vector searches that retrieve isolated text chunks, the graph enriches LLM inputs with interconnected metadata (e.g., regulatory documents linked to supplier audit tables) and business glossary terms (e.g., “EU Digital Product Passport”). For example, when auditing a supplier’s carbon emissions, the RAG pipeline first queries the KG to retrieve all nodes connected to the supplier’s material flows, compliance history, and sustainability reports.</p><h3>Disclaimer</h3><p>The text is written by the author (not an LLM 😀), but the PoC code heavily relies on LLMs (at the time of this writing, Anthropic Sonnet 3.7 was my best ally), as the focus is more on the concepts than on code quality. As a result, the code may contain some <em>automatically generated inaccuracies </em>(!), but I could run and test it. The code excerpt may not be the latest version I use, as it is an on-going experiment.</p><h3>References</h3><blockquote>Technical tool kit used in this experiment</blockquote><p><strong>LangGraph</strong>: <br><a href="https://langchain-ai.github.io/langgraph/">https://langchain-ai.github.io/langgraph/</a><br><a href="https://python.langchain.com/docs/tutorials/graph/#advanced-implementation-with-langgraph">https://python.langchain.com/docs/tutorials/graph/#advanced-implementation-with-langgraph</a></p><p><strong>LlamaParse</strong>:<br><a href="https://www.llamaindex.ai/llamaparse">https://www.llamaindex.ai/llamaparse</a><br><a href="https://docs.llamaindex.ai/en/stable/llama_cloud/llama_parse/">https://docs.llamaindex.ai/en/stable/llama_cloud/llama_parse/</a></p><p><strong>Neo4j</strong>: <br><a href="https://neo4j.com/">https://neo4j.com</a><br><a href="https://neo4j.com/product/graph-data-science/">https://neo4j.com/product/graph-data-science/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=24319275291f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Reflections on using Knowledge Graphs and LLM in metadata management context (1/n)]]></title>
            <link>https://loysbelleguie.medium.com/reflections-on-using-knowledge-graphs-and-llm-in-metadata-management-context-1-n-e1015eb85e2a?source=rss-dcbd37cb0d2e------2</link>
            <guid isPermaLink="false">https://medium.com/p/e1015eb85e2a</guid>
            <dc:creator><![CDATA[Loys Belleguie]]></dc:creator>
            <pubDate>Wed, 20 Nov 2024 11:58:03 GMT</pubDate>
            <atom:updated>2024-11-20T11:58:03.298Z</atom:updated>
            <content:encoded><![CDATA[<h4>Leveraging KG and LLM towards Enterprise Knowledge Management</h4><h4>Introduction</h4><p>As a member of a <a href="http://quollio.com/">startup</a> whose main product is a <em>data catalog</em>, I sometimes feel an unsettling sentiment arising from the impact of the inexorable commoditization of <strong>metadata management</strong> tools. As a matter of fact, beside our direct competitors, almost all data warehouse platforms, Snowflake, Databricks, AWS, Microsoft Azure or Google, to name a few, are providing some kind of metadata cataloging capabilities.</p><p>But this is just an impression. A data catalog is not just a <em>metadata repository</em> where technical metadata are collected for the sake of building a nice ordered list of names and descriptions, like in a library… Indeed, collecting information from an enterprise data landscape, namely as metadata, is one of the most valuable assets for a business: it goes beyond mere structure or definition of the underlying raw data and opens the door to real <em>Enterprise Knowledge Management </em>(EKM).</p><p>But what are the tools that can be used to develop corporate-wide EKM based on metadata? Actually, as often the case in the IT industry, concepts or tools have been around for a while, sometimes decades, and new combinations of usage bring innovation not to say disruption. I think this is the case with the <em>semantic layer</em>, i.e. the level above the technical metadata layer where we deal with the meaning of the underlying data. This is where Large Language Model (LLM) and Knowledge Graph (KG) excel, especially the synergy of both.</p><h4>A graph view of the Metadata lake?</h4><p>First, let’s start with a different approach than the conventional “tabular” organization of the meta-database by reshaping it as a graph, namely a collection of <em>node-relation-node</em> triplets. This is often easier to visualize a data structure as graph rather than as an excel sheet table! But the benefit of a graph representation doesn’t only lie in its graphical appeal; the relations (= edges) between the nodes add another dimension to the information space. To be accurate, there are actually two different levels of information contained in a graph: <br>(1) the semantic or human message: <em>(A) — [:isAFriendOf] →(B)</em> and, <br>(2) the topological aspect: how the nodes are connected to each other to form a network. <br>In this first article, I will concentrate on the first topic and investigate how this can be leveraged to build <em>intelligibility</em> in the metadata lake. If we think of it, metadata are, intrinsically, semantic objects: they are meant to describe raw data in a meaningful and human-readable way. We are not acting on metadata like data Engineers do on tables with SQL code and JOIN’s; we only perform very<em> human-like</em> actions such as search data from natural language, query vast amount of data, try to find similarities between unrelated assets, build classifications in disordered data sets etc.. Therefore, it makes perfect sense to represent metadata assets as interconnected nodes in a graph. The next question we may ask is:</p><blockquote><strong>Which business benefit would a Knowledge Graph bring to data users?</strong></blockquote><p>The fundamental reason for organizing the metadata assets as Knowledge Graphs is because, as mentioned above, we are dealing with objects not just bits and bytes, or as explained in a seminal article from Google Engineering: “<a href="https://blog.google/products/search/introducing-knowledge-graph-things-not/"><strong>things, not strings</strong></a>”, and this ideas also transpired from the last year’s Gartner Impact Radar…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s6G4nJgydRLTO2QFoh9EUg.png" /><figcaption><a href="https://www.gartner.com/en/articles/30-emerging-technologies-that-will-guide-your-business-decisions">https://www.gartner.com/en/articles/30-emerging-technologies-that-will-guide-your-business-decisions</a></figcaption></figure><p>Here, I will investigate how KG enhance the Enterprise Knowledge Management, from the metadata point of view and how LLM can leverage KG queries.</p><h4>The metadata lake as a knowledge graph</h4><p>Technically, data catalogs are basically a database of metadata (= assets) with management tools such as search engines, assets editing workflows (including access control) to enrich descriptions (business metadata), lineage or data quality to investigate specific aspect of the underlying data etc..</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h41AjSSsVQwEvjkbRvP-wA.png" /><figcaption><a href="https://quollio.com/"><em>Quollio Data Intelligence Platform</em></a><em> (QDIC) UI</em></figcaption></figure><p>But such systems, like the <a href="https://quollio.com/"><em>Quollio Data Intelligence Platform</em></a>, are not isolated nor closed applications: the metadata assets can be exchanged via several ways such as API, csv files or shared database. I’ll use a csv download here. The demo environment I created mimics (in a simplified way) real use cases: ingest metadata from many DWH, csv files as well as business Apps sources, which represent several business domains or entities. It is worth noting that those entities may be sitting out of the corporate domain and metadata be exchanged via specific data contracts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Jl2qnozbqWbiYO4wye7tRg.png" /><figcaption>QDIC data sources hub (=service names) (see <a href="https://docs.quollio.com/">https://docs.quollio.com</a>)</figcaption></figure><blockquote>A metadata Knowledge Graph is in fact a data model.</blockquote><p>So, let’s translate this metadata repository into a knowledge graph and perform some actions on it. To show how flexible the platform is, we use a 3rd party application: here <a href="https://neo4j.com/">Neo4j</a>, but it could be any other similar application (AWS Neptune etc..). The point is more to emphasize the importance of working with metadata from the semantic layer with convenient tools rather than focusing on a particular solution. The data pipeline is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xgoh8cLgAAp9XAUi3Evt_g.png" /><figcaption>Metadata usage in an open environment and KG</figcaption></figure><p>Once the metadata is ingested into the graph database, we can explore the data landscape from a different angle compared to the QDIC interface. At this point we only get a different view of the metadata lake as it already exists in the data catalog. It shows both the hierarchical structure, (:Table)-[:isInSchema]-&gt;(:Schema), but also any other relationships such as <strong>lineage</strong> or <strong>foreign keys</strong>, as well as any <strong>tags </strong>or <strong>glossary terms</strong> that may be already attached to assets in the catalog. Catalog administrators have the flexibility to export their metadata lake to a KG or other business applications without any impact on their production environment. Both options have their specific uses and target personas. But as the KG approach focuses on the semantic layer it may be more suitable for business analysis use cases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sGdhOBxvOAZrOylpDtyl_w.png" /><figcaption>QDIC metadata lake shown as a KG (using Neo4j console)</figcaption></figure><h3>Benefits of a KG approach in a metadata lake</h3><h4>Search and discoverability performance</h4><p>It is well documented that graph engines are superior than traditional relational databases when the number of SQL JOIN’s layers is increasing (see for example <a href="https://neo4j.com/news/how-much-faster-is-a-graph-database-really/">https://neo4j.com/news/how-much-faster-is-a-graph-database-really/</a>). In our case, we are not performing JOIN’s between metadata, but we can still benefit from the query engine to explore the assets in multiple dimensions. As an illustrative example, a user could be asking:</p><blockquote>I want to get all tables along the end-to-end lineage path from table with id = ‘tbl-xyz’, up to 8 hops;<br>Return all the tags attached to the tables along the path, as well as the schema they belong to.</blockquote><p>This kind of search could be quite complex to formalize with a standard search engine but it is very easy to write as a graph database query, even for a non-specialist! (The following code is a simplified version to illustrate the concept).</p><pre>MATCH p=(d:Database)-[]-(s:Schema)-[]-(t:Table)-[:tableLineage*8]-&gt;(t2)<br>where lower(s.name) contains &#39;value chain&#39;<br>    and t.id = &#39;tbl-xyz&#39;<br>optional match (t2)-[]-(tg:Tag)<br>optional match (t2)-[]-(st:Schema)<br>RETURN p,tg,st</pre><p>And the result looks like the figure below. The query only takes few ms, as the request is straightforward, and all metadata, such as attributes (table descriptions, properties etc..) are also available in the returned payload for further analyses (we’ll see how in a minute). Interestingly, being able to extract tags or which schema (in the original database) the nodes are attached to along the path is extremely important: this provides precious insights to the business analyst, for example to identify which data domains the lineage is traversing. The same kind of exploration can also be used for other kinds of relationships, for example node clusterings from similarities algorithms etc..</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7ANgQGkZKZbGDG9laUkrVw.png" /><figcaption>End-to-end lineage query from one table up to 8 hops, with tags and schema along the path.</figcaption></figure><h4>Ontology based asset classification</h4><p>Another useful tool is to use <em>ontologies</em> to add some formalized terminologies within the knowledge graph. Since a KG is inherently <em>schema agnostic </em>(i.e. horizontal rather than strictly hierarchical), <strong>domain specific ontologies</strong> can be added and linked to metadata assets. This helps for the understanding of the business entities assets belong to as well as their mutual relationships (see for example Sequeda &amp; Lassila, 2021).</p><p>As a simple example, let’s consider a company gathering data from several departments and business domains: the procurement via an ERP, the supplier information through an SCM, the product lifecycle and manufacturing via the PLM application and then the logistics between those entities. Although data owners of each domain will know perfectly the data they manage, the data stewards (DS) on the catalog side will certainly have difficulties to fully understand how each metadata asset is related to which business entity or data sources. Having an ontology to organize these assets may greatly improve the organization of the data landscape. The following figure shows how the addition of a standardized ontology translates as classes in the KG. Metadata assets can then be classified using this domain specific taxonomy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/913/1*_VTu9QUFAj5AaR2yZIxsEQ.png" /><figcaption><em>Metadata KG with an additional domain specific ontology</em></figcaption></figure><p>The nice thing with this approach is that we can, incrementally, add domain (company) or application specific ontologies to supplement the metadata semantic layer. For example, say we ingest metadata from the SAP ERP system, our data steward (DS) is not an SAP expert, but we are asked by the sales department to prepare a data product to calculate cost forecast linked with the change of a supplier. They ask us to locate a list of tables to investigate about a compliance issue found in critical parts from this supplier. Sounds familiar? Well, practically our DS would need to find tables such as EKKO, EKPO, LFA1, and MARA etc… which is, without being an SAP consultant, not a straightforward task. Incidentally, MARA stands for <a href="https://learning.sap.com/learning-journeys/exploring-basic-data-for-manufacturing-and-product-management-in-sap-s-4hana/introducing-material"><em>Material Master</em></a> and contains essential information about materials, such as product data, cost information. The DS would probably be advised to start gathering assets around this table, but this would be quite an endeavor without a well-documented KG. This is where adding an SAP-specific ontology to the KG would greatly help, as discussed in this <a href="https://community.sap.com/t5/technology-blogs-by-sap/knowledge-graphs-for-llm-grounding-and-avoiding-hallucination/ba-p/13779734">SAP Technology blog article</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/0*M7wEqTGUqtseGLj7" /><figcaption>From SAP Technology blog: <a href="https://community.sap.com/t5/technology-blogs-by-sap/knowledge-graphs-for-llm-grounding-and-avoiding-hallucination/ba-p/13779734">Knowledge graphs for LLM grounding and avoiding hallucination</a>.</figcaption></figure><p>The important point in this SAP example, is the explicit definition of entities: “<strong>abaptable:</strong>”, “<strong>bo:</strong>” for business objects, “<strong>cdsview:</strong>” for Core Data Services etc.. Of course, most of these definitions already exist in the LLM models but having the metadata prepared in advance with those definitions and their relationships to SAP internal entities will make a big difference when querying the KG.</p><h3>Knowledge Graph powered LLM tasks</h3><p>So, how can we leverage a knowledge graph to improve LLM tasks, and vice versa? There are several specific tasks, roughly two main categories, where metadata management applications can greatly benefit from AI/LLM:</p><p>(1) Metadata enrichment: to help data stewards in the <strong>metadata maturation process</strong>. With possibly millions of assets in real production environments with incomplete metadata, creating “human readable” versions of database entity names such as tables, columns, longer descriptions etc.. is of paramount importance, but can lead to an enormous amount of work if not at least partially automated. Also, automatic matching tagging or glossary terms to help classification can saves precious IT resource time.</p><p>(2) Asset discovery (i.e. query) from natural language questions or business analyses. Most of business users are not SQL savvy and would like to be able to gather the data they need from business-like questions.</p><p>In (1), the accuracy of the generated description will depend on the understanding that the LLM engine has about the overall context. For example, given the following table (from a real IoT device). Let’s suppose that the logical name and description have not been provided by the data owner, as may be the case with sensor data that are automatically ingested in data lakes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/651/1*m8mHLf3iv1uQMx7dimcpyQ.png" /><figcaption>Technical and Business Metadata in Quollio application</figcaption></figure><p>Let’s ask Gemini what it thinks about “KYAKU_DENRYOKU_SYOHI”:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xgFg-F7J9a5ZPQm-v_inHQ.png" /></figure><p>Obviously, without any context, the proposed solutions can only be “guessed” from the physical name. In this case, Gemini is able to infer the Japanese word behind the romaji Hepburn transcription, but this is trivial, no need of an LLM to achieve this. Providing the column names will reinforce the result, but nothing specific about the physical context, where is this IoT sensor implemented, is it in a plant, is it in logistics, in an inventory, or a cargo container? Reading the raw data may provide location from GPS data but from the metadata application, we wouldn’t have access to those information. <br>Obviously, we should pass the schema and database information, as well as the attached tags (here “工場BOP” = “Plant BOP”) that may already provide context, but we know that names in production database can be cryptic, like abbreviated group of words or just acronyms. <br>We would expect that the LLM engine be able to generate the logical name “IOTセンサーデータ” and most importantly the description: “製造ライン１の消費電力・CFP検査装置”, which provides the crucial information “CFP”. Again without more context, it would be difficult to guess that CFP, in this particular case, means “Carbon Footprint of Product”, related with a new compliance requirement for export…</p><p>In the second category, a question like the following could be asked:</p><pre>&quot;The business team for EMEA, and the procurement want to analyze product X <br>carbon footprint and evaluate suppliers cost vs CFP, taking into account the <br>Carbon tax, provide the relevant data in ERP, PLM and SCM&quot;.<br><br>(*)In this example, we are a Japanese company, in Japan, exporting to EMEA, <br>dealing with worlwide suppliers and new export regulations...</pre><p>How can we query a standard data catalog with such business inquiry without being able to understand, at the semantic level, how business applications are mutually connected and provide the relevant information about product “X”?</p><p>In both cases, it seems clear that the only way to get the necessary contextual information is to query the knowledge graph (cf. the lineage example above). It is worth noting at this point, that querying the KG can be done semi-autonomously either from predefined templates or directly from the LLM engine (i.e. with dedicated agents).</p><h4>Generative AI Architecture with contextual RAG</h4><p>LLM based on Retrieval Augmented Generation (RAG) architecture have become extensively studied since the introduction of chatGPT, especially the benefit of using a KG in different <a href="https://neo4j.com/developer-blog/graphrag-field-guide-rag-patterns/">RAG architectures</a>. The main reason for using a RAG in an LLM task is to provide contextual information to the AI agent to avoid, or at least reduce, hallucinations and inaccuracies in generated responses. <br>However, the experience shows that the naive approach of just vectorizing pdf files to feed the RAG falls short in accuracy, as the semantic relationships between assets are blurred in the process.<br>To keep the semantic information in the LLM context, we need to retrieve data from the KG and feed it to the LLM. A simplified logical data architecture looks like the following diagram.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rOHBGTtg3ygg397ppaPUvA.png" /><figcaption>Example of KG powered LLM-RAG logical architecture</figcaption></figure><p>The left part is the metadata lake with the additional ontologies and unstructured documents that are made available as a graph database. The KG engine serves the required contextual information to the LLM engine and at the same time can be queried directly by users to retrieve assets (like in the lineage example above).<br>The implementation of the LLM part is more complicated than the previous picture seems to imply: it is actually necessary to work with dedicated agents to drive the prompts creation based on KG queries. Here I will only show a simple example based on the <a href="https://www.langchain.com/langgraph"><strong>langGraph</strong></a> framework, using <strong>Azure OpenAI</strong> <strong>LLM</strong> and <strong>Neo4j KG</strong>. <br>I defined few agents to perform specific tasks, like querying the KG then use the LLM, fed by the return of the KG query, to provide the expected answer. The workflow looks like the following diagram. <br>The merit of this agent-based approach lies in the possibility to have <em>nodes</em> acting (semi-)autonomously and decide when to query the KG to get additional contextual information.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/594/1*RoBvMoKjnRCzOxFNuhikHg.png" /><figcaption>Simple KG/LLM workflow using langGraph</figcaption></figure><p>Practically, we ask a question like the following: <em>Provide the business context for this asset: “BEV_URIAGE_POS”</em>. The agents will first decide which query to send to the KG engine and create a prompt based on the information returned.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/709/1*txNApTG4o48dyepVa4TLTQ.png" /><figcaption>Assets linked with the queried table as returned by a query to the KG</figcaption></figure><p>The answer will look like the following:</p><pre>### Business Context of the Table: BEV_URIAGE_POS<br><br>1. **Overall Business Context:**<br>   The table `BEV_URIAGE_POS` is part of a comprehensive data mart designed <br>for performance analysis within the automotive industry, specifically focusing <br>on Battery Electric Vehicles (BEVs). The data mart, named `総合データマート` <br>(General Data Mart), is used for analyzing various aspects of the company&#39;s <br>value chain, including sales, inventory, and promotional activities. <br>The data mart supports detailed analysis of sales performance, <br>inventory management, and competitive benchmarking, which are crucial for <br>strategic decision-making and operational efficiency in the automotive sector.<br><br>2. **Purpose of the Table:**<br>   The `BEV_URIAGE_POS` table contains detailed sales data for BEVs, including <br>information about sales transactions at the point of sale (POS) and associated <br>dealership information. This table is the second version and includes <br>additional details about the dealerships. The primary purpose of this table <br>is to provide granular sales data that can be used to analyze sales <br>performance at the SKU level, track inventory status, and evaluate the <br>effectiveness of promotional activities. It serves as a critical data source <br>for understanding sales trends, managing inventory, and optimizing sales <br>strategies.<br><br>3. **Business Analysis with the Tables:**<br>   Several business analyses can be carried out using the `BEV_URIAGE_POS` <br>table and its linked tables. Here are some key analyses and the relevant <br>table sets:<br><br>   - **SKU Analysis:**<br>     - **Tables:** `BEV_URIAGE_POS`, `BEV_URIAGE_TOTAL`, `MART_BEV_URIAGE`<br>     - **Description:** Analyze sales and inventory status for individual <br>SKUs to support efficient sales planning and order management. <br>This analysis helps in identifying high-performing products and managing <br>stock levels effectively.<br><br>   - **Promotional Effectiveness Analysis:**<br>     - **Tables:** `BEV_URIAGE_POS`, `BEV_URIAGE_TOTAL`, `SALES_DEMO`<br>     - **Description:** Evaluate the impact of various promotional activities <br>(e.g., discounts, campaigns) on sales performance. This analysis helps in <br>determining the return on investment (ROI) of promotional efforts and <br>identifying the most effective promotional strategies.<br><br>&lt;... *answer truncated&gt;</pre><p>With this simplistic example, the answer provided by the LLM engine is still very basic, but it surely leverages the contextual information extracted from the KG. In real production environments, several workflows will need to be created for specific goals, but the concepts discussed here will remain valid.</p><h4>Some references about KG</h4><ul><li>Sequeda, J. &amp; Lassila, O., 2021. <em>Designing and Building Enterprise Knowledge Graphs. </em>s.l.:Springer International Publishing.</li><li>Olesen-Bagneux, O. V. G., 2023. <em>The enterprise data catalog. </em>First edition ed. Beijing: O’Reilly.</li><li>Kejriwal, M., 2021. <em>Knowledge Graphs. </em>Cambridge: MIT Press.</li><li><a href="https://neo4j.com/knowledge-graphs-data-in-context-for-responsive-businesses-developers/">https://neo4j.com/knowledge-graphs-data-in-context-for-responsive-businesses-developers/</a></li><li><a href="https://neo4j.com/knowledge-graphs-practitioners-guide/">https://neo4j.com/knowledge-graphs-practitioners-guide/</a></li><li><a href="https://www.deloitte.com/content/dam/assets-zone2/nl/en/docs/products/2024/deloitte-nl-risk-responsible-enterprise-decisions-with-knowledge-enriched-generative-ai-whitepaper-download.pdf">https://www.deloitte.com/content/dam/assets-zone2/nl/en/docs/products/2024/deloitte-nl-risk-responsible-enterprise-decisions-with-knowledge-enriched-generative-ai-whitepaper-download.pdf</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e1015eb85e2a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Governance implementation leveraging Snowflake and Quollio Data Intelligence Cloud…]]></title>
            <link>https://loysbelleguie.medium.com/data-governance-use-case-implementation-leveraging-snowflake-and-quollio-data-intelligence-cloud-46ddaae38be9?source=rss-dcbd37cb0d2e------2</link>
            <guid isPermaLink="false">https://medium.com/p/46ddaae38be9</guid>
            <dc:creator><![CDATA[Loys Belleguie]]></dc:creator>
            <pubDate>Tue, 15 Oct 2024 05:32:55 GMT</pubDate>
            <atom:updated>2025-02-04T00:58:53.344Z</atom:updated>
            <content:encoded><![CDATA[<h3>Data Governance implementation leveraging Snowflake and Quollio Data Intelligence Cloud synchronization</h3><p><a href="https://quollio.com/">https://quollio.com</a></p><h3>Foreword</h3><p>We recently participated to the <a href="https://www.snowflake.com/events/snowflake-world-tour-tokyo/">Snowflake World Tour</a>, at the Tokyo stage on September 12, 2024, and I will take this opportunity to extend a little bit on some of the interesting features of the Quollio Data Intelligence Cloud that helps implementing robust data governance in large enterprises. Indeed, leveraging the extensive capabilities in Snowflake such as tag-based data masking, Role Based Access Control (RBAC) to name a few, greatly simplifies a great deal of advanced governance tasks for Data Stewards (DS).</p><h3>Quollio and Snowflake</h3><p>Our core business is, in a nutshell, helping large enterprises, I mean, the largest (!), to make sense of their data landscape, especially from a digital transformation (DX) strategy perspective, and to achieve this, we automatically create a metadata lake and provide all the management tools they need to connect the dots between data silos!</p><p>The key word here is metadata management. Metadata is what describes the underlying data, for example, names of tables in databases, descriptions, technical characteristics, statistics and many other information. But our goal is also to provide intelligibility, a holistic view of the whole data landscape to our customers, in other words, we deal with knowledge of the corporate data, which is often referred as the semantic layer. And for big companies, knowing precisely what data they have, where it is stored, how business analysts can access and use it (i.e. self-BI), without delay or security issues, with the expected quality level, is a critical factor for business success.</p><p>Providing access to data for all potential users is crucial to create business value and the <em>raison d’être</em> of the DX strategies. But obviously data must be well managed, we say “governed”, to avoid security issues, privacy leaks etc.; unattended data can lead to opportunity cost or worse, compliance issues leading to financial penalties.</p><h3>Quollio Solution</h3><p>Quollio Data Intelligence Cloud working closely with data management platforms like Snowflake, can make a huge difference for the DX teams when dealing with data governance. Practically, we need to ensure that, for example, security and privacy policies implemented on the Snowflake side, are also reflected in QDIC. Indeed, the data stewards in charge of the metadata management need to have a crystal-clear view of the sensitivity level of the underlying data before opening access to business users or data analysts. This applies also to data quality, data lineage or usage statistics, ownership etc…</p><h3>Data governance in practice</h3><p>Data governance is a vast subject, and I will refer to the extensive literature on the subject (see Bibliography). There are also a lot of debates about what it is or should be and many practitioners may have somewhat different interpretations or priorities. Anyway, one common reference is the DMBOK [1]:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/582/1*C8FMdSMfOEn6_BohtLwDqg.png" /><figcaption>The DMBOK wheel</figcaption></figure><p>From the data catalog point of view, data governance is like a control tower of the underlying data landscape. The image can be stretched even further: the air controllers do not need to know all the details of what’s going on in each airplane. They manage the whole traffic with certain critical data, flight details, quality or compliance information (is the plane authorized etc.), but one crucial aspect is that the data transmitted to the control tower be as accurate as possible.</p><p>Similarly, the data governance team will use specific metadata from well-identified areas to ensure that the underlying data is properly managed and flows in the pipeline without incident, security or compliance issues. There are no one-fits-all list of data governance tasks as it may greatly depend on the context and industry, however there are some items that are almost always present, such as:</p><p>The technical level</p><ul><li><strong>Access control</strong>: ensure that only the authorized person has access to the right content,</li><li><strong>Privacy</strong>: business sensitive data or personally identifiable information (PII) is masked with the appropriate granularity,</li><li><strong>Data Quality</strong>: monitor any change or sudden anomaly of data,</li><li><strong>Active metadata</strong>: make sure that information is up to date, constantly enriched from several sources.</li></ul><p>The business level:</p><ul><li><strong>Compliance</strong>: data is managed (stored and consumed) according to regulatory requirements,</li><li><strong>Transparency and explainability</strong>: especially in the context of AI governance.</li></ul><h3>Snowflake Governance features</h3><h3>The Horizon program</h3><p>During the last months, Snowflake have announced a series of initiatives such as AI features with Cortex but also a lot of emphasis has been put on data governance, which is recognized as a major, not to say a core pillar of any enterprise DX strategies. For details see for example [3].</p><h3>Advanced Snowflake features for Data Governors and Stewards</h3><p>The Data Steward (DS) task in a large enterprise may be daunting! I have seen many examples of data landscapes built from hundreds of different sources and business domains, converging into a single integration platform like Snowflake. In practice, a typical data integrated platform of a global company may contain hundreds of databases and schemas, thousands of tables and up to millions of columns… In such situations, it is virtually impossible to manage, manually, all data without specific tools and some automatization. The Snowflake platform provides very powerful tools to implement data governance, yet rather easy to use as everything can be done in SQL, which is something that is expected to be a ubiquitous expertise in IT teams.</p><p>The data Governance capabilities, currently available in the Snowflake platform are as follows <a href="#_ftn2">[4]</a>:</p><p>· Column-level Security: for controlling the access of a complete column</p><p>· Row-level Security: to filter out groups of rows based on certain criteria such as user role</p><p>· Object Tagging: used to classify data from high level such as business categories, database, schema down to the lowest granularity as table or columns</p><p>· Data Classification: to identify sensitive data, works with tags</p><p>· Tag-based masking policies: hide raw data to non-authorized roles</p><p>· Access History: for daily data steward activity or for auditing purpose</p><p>In addition, there are plenty of Data Quality and data metric functions that can be used to monitor the state and integrity of raw data. New features have been announced in private or public review, such as <em>sensitive data auto-classification</em> where schemas are monitored and tagging, or policy are assigned automatically.</p><p>Data classification is one of the most important requirements in data governance. But as I mentioned above, classifying thousands of tables and millions of columns can’t be done manually. Ideally, data owners from the upstream (data source) side would be expected to properly tag the data they share, but this is not always the case, especially when SQL join’s or any ETL processes mix different provenances of data. The solution is to tag those sensitive tables or columns.</p><p>Obviously, we would like to get that critical information propagated and synchronized with the metadata management in the Quollio platform. The goal is to ensure that the governance policies implemented on Snowflake are reflected on the metadata lake side. Indeed, it wouldn’t make sense to catalog data that is identified as <em>sensitive</em> on the source side without propagating the <em>governance metadata</em>. For example, one of the easy-to-understand examples may be the object tagging, data classification or tag-based masking policies.</p><p>Another important feature is the capability to read the lineage data from Snowflake and display the end-to-end path of data, from the source up to the dashboard, for instance Tableau.</p><h3>Technical enforcement of data governance</h3><h4>Leveraging Snowflake data governance features in Quollio platform</h4><p>Let’s consider a simple data pipeline from a raw data producer, a DWH storage and processing stage in Snowflake and a consumer, for example a BI tool, such as Tableau.</p><p>So practically speaking, how do we implement this in Quollio?</p><p>First, access to metadata assets is fully controlled by a system very similar to Snowflake, namely, through user groups (like roles) with specific privileges to asset groups (at the table level granularity).</p><p>When metadata is ingested automatically from SF, it enters a Workflow where Data stewards can monitor the data curation, approve access requests or any other edits or AI recommendation on metadata descriptions for example.</p><p><strong><em>What specific features of Snowflake can we leverage to achieve AI Governance?</em></strong></p><h3>Access control and Privacy</h3><p>The basic tool: <em>policy</em> definitions and role-based access control (RBAC) in Snowflake. We don’t want to have on DWH side strong policies and leave all the metadata unattended on the QDIC side. <br>To address this issue, any data catalog needs to implement a robust asset access control (AAC) and precisely define who can see which asset, and in fine get information or even full access to the source data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/911/1*NPKJM2Lt-1h90bpJLjXXcA.png" /><figcaption>Figure 1 Asset control in QDIC</figcaption></figure><p>Now let’s look at some specific examples.</p><h4>Governance tags</h4><p>Tags are easily created in Snowflake with SQL:</p><pre>-- Create risk management tags<br>CREATE OR REPLACE TAG RISK_MGT COMMENT = &#39;リスク分析用&#39; ;<br>CREATE OR REPLACE TAG PARTS_MGT COMMENT = &#39;部品調達リスク分析用&#39; ;<br>-- assign tag to table for risk management<br>ALTER TABLE PARTSUPP<br>  SET TAG RISK_MGT = &#39;調達リスク・注意&#39; ;<br>-- assign tag to columns<br>ALTER TABLE PARTSUPP<br>  MODIFY COLUMN PS_AVAILQTY<br>  SET TAG PARTS_MGT = &#39;在庫レベル注意点&#39; ;<br>ALTER TABLE PARTSUPP<br>  MODIFY COLUMN PS_SUPPLYCOST<br>  SET TAG PARTS_MGT = &#39;部品価格情報&#39; ;</pre><p>Result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OaBjA988X4_HsUvEcMY_PQ.png" /></figure><p>These governance tags will be propagated to Quollio metadata management so that any user, data steward, data scientist or business analysts will be able to get the same classification granularity as on the data source side.</p><h4>Masking policies</h4><p>Masking sensitive data on DWH is a common practice in governance teams and Snowflake is offering standard procedures to apply robust and clear policies on certain columns or rows. The following code is an example of a column being masked to certain roles.</p><p><strong>On Snowflake</strong>:</p><ol><li>We start by creating a <em>masking policy</em> using SQL command:</li></ol><pre>USE ROLE SHOWCASE;<br>CREATE OR REPLACE MASKING POLICY email_mask AS (val string) returns string -&gt;<br>  CASE<br>    WHEN current_role() IN (&#39;SHOWCASE&#39;) THEN VAL<br>    ELSE &#39;*********&#39;<br>  END;</pre><p>2. Then we apply this policy to the CUSTOMER table:</p><pre>ALTER TABLE DEMO_TPCH.STAGE.CUSTOMER MODIFY COLUMN C_PHONE <br>          SET MASKING POLICY email_mask;</pre><p>3. We can then confirm that the column in indeed masked when user role, say “TB_DOC_AI”, is not “SHOWCASE”:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j1s_gH7WcQL9DkhoCiSvyg.png" /><figcaption>Figure 2: Masking policy on Snowflake</figcaption></figure><p><strong>On Quollio Data Intelligence Platform</strong>:</p><p>1. The data ingestion agent automatically fetches the metadata and lineage data</p><p>2. Governance tags from Snowflake are also ingested, so that the level of policy is made visible on QDIC side: see the diagram below. This simple mechanism is indeed very powerful, as it allows the exact same classification from Snowflake to be reflected in the metadata catalog.</p><p>The overall process is summarized on the following diagram:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DXgpYsqPdgCOVGMoxfQ0CA.png" /><figcaption>Figure 3: Governance tag and lineage synchronization from Snowflake to Quollio QDIC</figcaption></figure><blockquote>Leveraging Snowflake automatic system level tagging</blockquote><p>Snowflake also offers some intelligent tagging feature (before the full AI powered feature) which can also be synchronized with Quollio metadata.</p><p><strong>On Snowflake</strong>:</p><p>Run the Sensitive data detection function (stored procedure):</p><pre>CALL SYSTEM$CLASSIFY(&#39;DEMO_SEIZO.CUSTOMERS_JP.CUSTOMERS&#39;, {&#39;auto_tag&#39;: true})</pre><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nmE62aAvmNoTzKoEeCc5vA.png" /></figure><p>and the system tags will be assigned to the relevant columns:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2zjCvwhwRnQePBhyhlFJrw.png" /></figure><p>This automatic tagging feature can be used effectively on large data sets (with hundred thousands of columns!) before synchronization with Quollio metadata lake.</p><h3>Conclusion</h3><p>Quollio Data Intelligence Platform works hand-in-hand with Snowflake especially when implementing robust data governance practices. In this article, I only scratched the surface of what is possible, and I will continue to explore the subject as long as new features are being released. For example, new features such as <em>automatic tag propagation</em> currently in private preview at Snowflake will be a great tool to ensure governance information is not lost along the data path, and properly synchronized in the metadata lineage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/762/1*6n2PLlRXbMhhT0xRFgvVYw.png" /><figcaption>Figure 4 from <a href="https://info.snowflake.com/01-Whats-New-Snowflake-Horizon-Series-Master-Processor_2-in-1-thank-you-page.html?utm_cta=website-solution-horizon-timely-content-horizon-series&amp;wvideo=2x1tyvt644">DATA CLOUD SUMMIT 2024</a></figcaption></figure><h4>References</h4><p>[1] D. Henderson and S. Earley, Eds., DAMA-DMBOK, Second edition ed., Basking Ridge, New Jersey: Technics Publications, 2017.</p><p>[2] E. Eryurek, Data Governance: The Definitive Guide, U. Gilad, V. Lakshmanan, A. Kibunguchy-Grant and J. Ashdown, Eds., Sebastopol: O’Reilly Media, Incorporated, 2021.</p><p><a href="#_ftnref1">[3]</a> See for example: <br> — <a href="https://www.snowflake.com/en/why-snowflake/partners/horizon-partners/">https://www.snowflake.com/en/why-snowflake/partners/horizon-partners/</a> <br> — <a href="https://info.snowflake.com/01-Whats-New-Snowflake-Horizon-Series-Master-Processor_2-in-1-thank-you-page.html?utm_cta=website-solution-horizon-timely-content-horizon-series">Data Cloud SUMMIT 24</a></p><p><a href="#_ftnref2">[4]</a> <a href="https://docs.snowflake.com/en/guides-overview-govern">https://docs.snowflake.com/en/guides-overview-govern</a></p><p>[5] <a href="https://docs.snowflake.com/en/sql-reference/stored-procedures/associate_semantic_category_tags">https://docs.snowflake.com/en/sql-reference/stored-procedures/associate_semantic_category_tags</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=46ddaae38be9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Intelligence applied: how to use lineage in real life use cases?]]></title>
            <link>https://loysbelleguie.medium.com/data-intelligence-applied-how-to-use-lineage-in-real-life-use-cases-08e5d17b646c?source=rss-dcbd37cb0d2e------2</link>
            <guid isPermaLink="false">https://medium.com/p/08e5d17b646c</guid>
            <category><![CDATA[bill-of-materials]]></category>
            <category><![CDATA[data-intelligence]]></category>
            <category><![CDATA[data-lineage]]></category>
            <dc:creator><![CDATA[Loys Belleguie]]></dc:creator>
            <pubDate>Thu, 09 May 2024 04:49:43 GMT</pubDate>
            <atom:updated>2024-05-09T07:50:36.245Z</atom:updated>
            <content:encoded><![CDATA[<p>This article is the first of a series on how to realize Data Intelligence with <a href="https://quollio.com/">Quollio Technologies</a><a href="#5910">(*)</a>. Here, we introduce the <strong>lineage</strong> feature and show how it can be used to enhance the Enterprise data intelligibility.<br>Although all executives agree to the fact that data is a critical business asset, it is still underutilized in many companies. Often, data created in one Line of Business (LoB) may generate value in other domains with different usage. Unfortunately, the knowledge about data (i.e. its semantic metadata layer) is not always available or understandable. As a result, data may never be used beyond its original context, which means many lost opportunities.</p><p>We will discuss here how company can start building meaningful business insights using lineage and which developments we expect in the corporate data management area to fully turn data into actionable knowledge.</p><h4>Data lineage: an essential part of Quollio Data Intelligence Cloud</h4><p>With the fast development of the Digital Transformation (DX) in all industries, also referred as Industry 4.0, Society 5.0 (in Japan), one of the major issues is the necessity to get a holistic and intelligible view of the whole corporate data ecosystem. However, as many data sources or business applications remain spread all over different company domains, an enterprise-wide data management system is mandatory. This includes an intelligent metadata cataloging capability bridging the gap between silos, asset management tools such as access control or workflow for governance tasks, data quality monitoring, and a cross-domain data lineage capability.</p><p><em>So, what is a data lineage exactly, why is it important?</em></p><p>In simple words, lineage in data management helps visualizing the whereabouts of data as it flows between the sources, a database, a data warehouse or datalake and the last stage, for example a business analyst dashboard, an ML model or just another storage area. Any kind of transformations may happen in between and should be recorded in the lineage.<br>Before going further, we need to make some definitions clear. The usual meaning of <em>data lineage</em> corresponds to the real (physical) connections between raw data sources. It is also referred as technical lineage. Data lineage is built from the operational level of data sources and metadata are used to represent those data. The other important concept is <em>business lineage</em> which focuses on the business context and processes in which the data flows between entities. While the former shows complex data transformations, the latter can explain the business context or reasons for those processes. While the business lineage may focus on critical data elements somewhere in the corporate value chain, for example a supply issue at a manufacturing stage, the data lineage will detect which data entities are impacted within the whole pipeline. Conversely, if a sudden change is detected in a BI dashboard, the technical lineage will help locate its origin and provide the link with the business process which is impacted. Moreover, it makes any evolution over time more conspicuous, as it is often difficult to guess when a simple change in a table far away in the upstream has any impact on the downstream part. A typical data platform and the associated data lineage may look like the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lS6C7s635SSfZjfjt5iJcw.png" /><figcaption>Typical data platform and the associated data lineage</figcaption></figure><p>Here I will emphasize the business use case from a slightly different angle, showing how it can also be used for more than just data pipeline issues.</p><h4>Which questions a lineage representation helps answering?</h4><p>There are numerous cases where lineage can help identifying issues in a data pipeline.<br>According to the DMBOK, data lineage is a primordial tool to achieve high-quality data management in the enterprise:</p><p><em>If an organization does not know where data came from and how it has changed as it has moved between systems, then the organization cannot prove that the data represents what they claim it represents.</em></p><p>The corporate governance team may want to know: “What data in our worldwide enterprise needs to be investigated for compliance with local and industry regulations?”. Imagine a Japanese company with multiple subsidiaries or sales offices operating in Europe wondering about which data may be subject to regulations such as <a href="https://gdpr-info.eu/">GDPR</a> in Europe or <a href="https://oag.ca.gov/privacy/ccpa">CCPA</a> in California. Governance teams must ensure that personal identifiable indicators (PII), names, addresses etc., can be easily found and erasable if requested (GDPR Article 15–17 “right to be forgotten”).<br>The need of data lineage is not new. It has been around in the finance industry since the 2008 crisis. It then appeared that several major banks were unable to accurately track their data, and this led to the publication of the “Basel Committee Banking Supervision” regulations (BCBS 239). Banks have the obligation to show, during audits, the data flow used to create risk reports.</p><p><em>Making sure that data flows and processes associated with the risk measurement system are transparent and accessible. In particular, it is necessary that auditors and supervisory authorities are in a position to have easy access, whenever they judge it necessary and under appropriate procedures, to the models’ specifications and parameters.</em><a href="#3f02">[1]</a></p><p>This is exactly the <em>raison d’être</em> of a data lineage tool to enable this kind of risk analyses. With the progress of Digital Transformation (DX) and data integration, it is made possible in virtually any industries, with very specific needs in each one, as we will see in the following.</p><h4>Benefits of building a data lineage</h4><p><em>Impact or root-cause analysis</em></p><p>Identify downstream impact related with any change upstream:<br> A very simple example is inconsistencies or discrepancies between views that should represent the same business reality. If a sales team has different results compared to its finance counterpart, there must be a problem with the data creation or transformation somewhere, and chances are that it may not be obvious to track down the paths through the source data and find out where the error has been introduced.</p><p><em>Data validity check</em></p><p>Business analysts or scientists can check within a limited time whether the data they are going to use is what they expect, in other words, to verify the truthfulness of the data:<br>Search with keywords or tags along the data path and locate the relevant assets or, display additional metadata on each node and edge.<br>A data scientist wants to improve confidence and trust in the data used in ML models.</p><p><em>Verify provenance of data</em></p><p>Another common case is to check whether a table, say from a data mart that is used to build a dashboard, is getting its data from the correct sources. This can be particularly important if there is a suspicion that data sources have been duplicated and that the table used may have stale or worse, corrupted data.<br>A business analyst may try to locate data to run a study on improving customer experience based on a mix of marketing and personal information data distributed across the organization.</p><p><em>Data integrity</em></p><p>For data architects or stewards, to verify there are no accidents in their data pipeline design.<br>Data corruption can be very difficult to trace and hard to protect against, especially when not detected by data quality procedures. Data corruption can occur very far, at data ingestion phases, when joining clean data with corrupt data or even data being transformed due to misinterpretation by auto-formatting tools. <br>The lineage tool can flag a critical node upstream and propagate the information along the data path.<br>A data steward wants to investigate how data quality changes across multiple lineage hops? A deep analysis with a lineage graph helps locating the potential issues before the risk materializes as a disastrous business decision based on incorrect data.</p><p><em>Track sensitive data</em></p><p>For data governance teams, to ensure that they would be able to track any data, especially sensitive data, all along the data lifecycle. This is of paramount importance during audits.</p><p><em>Data migration</em></p><p>Investigate how data sets are structured on legacy on-premises databases before carrying out the data transformation and move to the cloud.<br>A data owner is planning to migrate some data sets to a low-cost long-term archive repository and needs to know how this will affect users.</p><h4>Leverage the graphical views</h4><p>In some respects, a lineage graph shares similar goals as a mind mapping or a knowledge graph representation: it shows on one figure, very diverse but somehow connected information, that can be easily grasped in the blink of an eye by a human. It lets users freely rearrange or focus on objects that have a specific importance for the on-going analysis. When there are few dozens, hundreds or more objects, it is virtually impossible, without a modern graphical representation, to really use the lineage information to its full potential. Being able to synthesize an entire data lifecycle in only one figure is very valuable from a business perspective as well as for data practitioners. The possibility to traverse end-to-end a lineage path to trace specific data, add tags and propagate them throughout this path, identify data owners in each steps, flag some nodes for further investigation by a Steward are some examples of those critical features a technical graph enables.<br>A lineage graph will look like the following image. Reading from left, all tables used in the upstream of the data pipeline to create the data used in the last stage (downstream) are represented as nodes and their interconnections by line or edges.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/726/1*jBE1H0xe38_Ifc46iM82qA.png" /><figcaption>Data lineage as seen on <a href="http://quollio.com/">Quollio Data Intelligence Cloud</a></figcaption></figure><h4>Going beyond technical lineage</h4><p>When describing lineage in a data management system, it is often approached from a technical or database centric perspective, for example, how a change in a table upstream, impacts those downstream. However, much <strong>deeper business insights can be extracted from</strong> <strong>lineage if it is used as a representation of the whole value chain</strong>. As a matter of fact, the data pipeline itself can be seen as a digital thread of the business process, for example the product life cycle, from the extraction of raw materials, supply chain, manufacturing, sales, up to customer usage and finally recycling.</p><p>There are numerous examples, just to mention a very illustrative case from a WBCSD publication<a href="#c79d">[2]</a> about data sharing in industries showing the data mapping between producers, stakeholders, and consumers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/1*hLE_iWcCUWVo7I5dnXDP7A.png" /><figcaption><em>WBCSD (August 2023)</em></figcaption></figure><p>As a matter of fact, traceability of data and the visibility of their inter-dependencies have become a central industrial topic extending beyond value chains optimization issues to sustainability and carbon footprint management. As the physical reality of material flow is extremely intricate so is the data pipeline. However, the technical lineage only connects explicitly the data domains when for example a new table or report is created downstream through some SQL or ETL operations upstream. But all processes may not be fully captured in the automatic lineage creation. It could be necessary to add some missing parts manually or through other mechanisms such as AI recommendations.</p><h3>Using lineage in a manufacturing context</h3><p>Any business transaction or operation in a value chain is linked to a physical process and a data event: basically, every aspect of the physical flow can be represented by a data pipeline. The concept of <em>digital thread</em> comes in handy here: <em>collecting data from each stage of production, storing that data in an integrated database, and then providing access for tracking changes over time</em><a href="#9d19">[3]</a>. A data lineage is one digital representation of the physical flow of matter from raw materials to a finished product sold in a shop.</p><p>Let’s consider the whole value chain of a hypothetical car manufacturer as an illustrative case. The end product, the car in the field, has been produced after a succession of steps from raw materials extraction, supply chain (with potentially hundreds of suppliers), many factories, assemblies, inventories, logistics hubs in between not to mention customs inspections before reaching the points of sales. We will see how the material flow from “cradle to gate” and associated business operations could be captured in a business lineage, supported by the underlying technical lineage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/828/1*oPQjS0SdVNR8oyTjdppu0g.png" /><figcaption>Typical material and data flow in manufacturing industry.</figcaption></figure><h3>The business insights extracted from the digital thread</h3><p>As shown on the above diagram, there are many different types of data in the digital thread that can be captured and shared between business domains from product design to sales and recycle. Besides addressing many distinct technical needs, it carries deep business information that can only be made available to users if the metadata are properly managed throughout the digital thread. This is a critical element to bridge the gap between the data flow and the business analysis.</p><p>In manufacturing, especially in a <strong>smart factory</strong> context, one of the central pieces of data is the Bill of Materials<a href="#0815">[4]</a> (BOM) which is basically the list of all the elements that constitute a product. Each part of a BOM is linked with different business entities (and their corresponding data in the digital thread) such as suppliers, product manufacturing lines (MES), technological or geopolitical dependencies (raw material) etc… The engineering BOM (E-BOM), produced from the CAD data (design phase), is usually managed by the PLM<a href="#2faa">[5]</a>. It goes through many updates and version synchronization is critical. It is the source for the manufacturing BOM (M-BOM) which is reflected in the ERP<a href="#d5c7">[6]</a>.</p><p>This is where the business analysis is connected to the manufacturing value chain. The total number of items in a BOM can range from few units (for example a bicycle) up to millions for an Airbus!<a href="#03ec">[7]</a>. It is rather easy to imagine the hurdle it might be to understand what impact some supply disruption may induce on the production, especially in a just-in-time production context.<br>As a matter of fact, each interface between domains, either inside the company or in the supply chain is a potential disruption risk, and there are still a lot of cases where there are not enough connections between data domains. This adds to the overall risk level: when a problem arises at the physical level, it may be virtually impossible to perform a root-cause or impact analysis if the data repositories are not properly connected to each other’s.<br>A typical data thread architecture would look like the following diagram. The SoI (System of Intelligence — or Insights) contains the data extracted from the corporate value chain and the Metadata Intelligence System (Quollio) which generates a data product out of business questions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1I1nB-tHTpLNPB3JaulVCA.png" /><figcaption>A typical data thread architecture</figcaption></figure><p>The data lineage capability of the <a href="https://quollio.com/">Quollio Data Intelligence Cloud</a> associated with its rich business metadata engine, offers a very powerful yet affordable solution to implement an enterprise-wide knowledge-base.<br>In essence, the BOM is a highly sensitive and confidential piece of information. From the pure business perspective, it contains a part of the “Cost Of Goods Manufactured or Sold” (COGM/S) which is directly linked with pricing, margin forecast, and the overall corporate revenues and profitability! But it is also implicitly linked with many other business data points that may greatly impact production, such as suppliers, compliance issues etc…<br>From a business perspective, each item in a BOM is linked to a market reality: the product specifications depend on part characteristics, quantities (supply) and pricing strategy, generic information about compliance or sustainability etc. All these information exist somewhere in the data platform and should be made available to business users.<br>Think about the supply chain disruptions. The recent pandemic has showed us that a simple disruption somewhere in the value chain may greatly impact delivery<a href="#86fb">[8]</a>. For example, automotive industry is directly linked to semiconductor supply which, according to Intel, will represent up to 20% of the total premium vehicle BOM in the near future<a href="#7076">[9]</a>.. Having a clear picture of the whole supply chain and how disruptions may impact the downstream production and sales forecast is crucial for a corporation.</p><h4>Solving a business problem with the Quollio Data Intelligence Cloud: Methodology</h4><p><em>The business question<br></em>Let’s see a particular example in the previous manufacturing context.<br>For example, a business analyst would like to prepare a cost review of a new product line and make a predictive analysis of the potential supply chain disruption risks that would directly impact the product go-to-market timing. This would require locating the relevant data that may be spread over the corporate data repositories, ERP’s, maybe in remote factory on-premise legacy systems etc…, then potentially also require communicating with data owners, get approvals from governance teams, most probably have the data be prepared (modeling, masking sensitive data etc.) before being released as a data product to the business consumers. This whole process may take too much time just to gather and validate the necessary data.<br>So, practically speaking, how do I track, search, and fetch the list of all potential parts in a BOM that may be unavailable in the future? Where is the list of those suppliers? Can I find a replacement for the missing parts? Do we have this data in-house or do we need to investigate further? Can I cross these findings with other geopolitical data, where can I access it?</p><p>As discussed above, the technical lineage alone is not the only tool to address a real business issue from the metadata layer. We have several features available in the Quollio application which will complement the data lineage view.<br>The search capability with business metadata, tagging etc., enhanced with AI, is one of the core elements of the Intelligent Engine.<br>Then, the workflow capability allows to manage all the communication between users and data stewards as well as the approval process at each step.</p><h4>Methodology workflow</h4><p>A possible workflow for a business analysis could look like the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/874/1*jLdnA62j23_VcK6FjxSMPw.png" /><figcaption>A possible workflow for a business analysis</figcaption></figure><p><strong>(1)</strong> <strong>Enrich semantic layer metadata:</strong></p><p>The Intelligence Engine will provide the Enterprise Knowledge Base that will allow translating business questions to a mapping to the relevant data sets.<br>The initial step is to enrich the semantic layer (or business metadata) of the data catalog. For example, data steward will work with the Subject-Matter Experts (SME’s) or data owners of each entity to make sure that logical names and meaningful descriptions are properly recorded. They will also use a contextual business glossary and tag nomenclature that reflects as close as possible the corporate business domain.<br>This process is particularly important for building a powerful Enterprise knowledge base linking with the data assets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1fjBZOQn5UUmYSbcxytbyw.png" /></figure><p><strong>(2)</strong> <strong>Search the metadata catalog</strong></p><p>Once the metadata has been enriched with semantic (i.e. business) data, we can locate the relevant data sets within the whole corporate assets repository, using intelligent search, or get recommendation from an AI agent. The latter will recommend several data sets which can be used for a specific analysis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/786/1*G3eTdQp6lIQq15WuBKF2Ow.png" /></figure><p><strong>(3)</strong> <strong>View the data lineage</strong></p><p>Once the data sets are found, it is time to start creating data marts or investigate existing ones with the lineage graphic view. A very simplified (yet realistic) view for this scenario may look like the following (using Quollio lineage feature):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h8E66zuAOIeStcKYepdCZw.png" /></figure><p><strong>(4)</strong> <strong>Business analysis</strong></p><p>Using this graph, a business analyst building a report in a BI tool (Tableau, PowerBI) from a materialized view (BA_MART) will be able to identify the source of the PRODUCT or SKU data from the EV factory database (BEV_LINE) as well as all the relevant information about parts that is spread across the data warehouse.<br>Without the lineage information it might be very difficult to “guess” that the SKU data is linked with PARTNUMBER which is an element of the product BOM located in a specific table coming from a remote ERP.<br>The linkage between the dashboard analysis and the origin of the data is made visible in the lineage graph. From that point on, the business analyst can check the metadata, extract the information about the ERP, and the entry point for further investigations, for example, finding the related factory or supplier’s data.</p><p>It is worth noting that the work of the data steward will be limited to the data lineage scope (metadata quality, data ownership etc.), but the business analyst will, in addition, use the lineage as a first step to build knowledge about a specific domain.</p><p>The business analysis can be further enhanced with LLM. Let’s take the supply chain context again. Global enterprise procurement teams deal with thousands of suppliers, whose contracts are stored as unstructured, purely language, documents. LLM engine will help building a knowledge center to support the search of all the relevant data sets from their metadata in the semantic layer. In the same way, macro-economic data such as geopolitical trends, semiconductors shortage or shipping issues can be connected simply from a semantic approach.<br>The same kind of approach can be used in manufacturing plants, connecting debug reports, machine logs, engineer’s analyses stored in dozens of factories around the globe, to the IoT from MES or BOM repository databases.</p><p>The Data Intelligence platform then enables managing enterprise knowledge as well as provide co-pilot assistance fine-tuned with the relevant business context.</p><h4>Conclusion and future developments</h4><p>At <a href="https://quollio.com/"><em>Quollio</em></a>, our main goal is to make Data Intelligence serve the business, create value, be it pure economics (creating new revenue streams, efficiency improvement, cost savings…), governance and compliance or sustainability. <br>As shown in this note, customers can manage the corporate data estate covering all domains of their value chain and make it fully <em>intelligible</em> from a technical up to a semantic point of view.<br>One crucial strategy is to extract semantically meaningful connections between corporate datasets as it enables business decision makers gain context on the existing data landscape of an organization, and therefore understand how knowledge is structured in the enterprise. This goes beyond technical lineage between data sets and adds a new dimension to data management to anticipate business changes that might serve either as threats or opportunities.<br>We are now extending the semantic layer capability using new technologies such as AI/LLM (RAG) that will make possible to build Enterprise Knowledge Base at a much higher level. This will enable business issues to be addressed using enhanced metadata search to automatically create the necessary data sets as data products for business consumers.</p><p><a href="#_ftnref1">[1]</a> <a href="https://www.bis.org/publ/bcbs265.pdf">Fundamental review of the trading book: A revised market risk framework — consultative document (bis.org)</a></p><p><a href="#_ftnref1">[2]</a> <a href="https://www.wbcsd.org/contentwbc/download/16875/238918/1">https://www.wbcsd.org/contentwbc/download/16875/238918/1</a> (World Business Council for Sustainable Development)</p><p>[3] <a href="https://www.plataine.com/blog/benefits-of-digital-thread-in-manufacturing/">https://www.plataine.com/blog/benefits-of-digital-thread-in-manufacturing/</a></p><p><a href="#_ftnref1">[4]</a> See for example the series of articles on the importance of BOM/PLM data management for the success of DX strategies: <a href="https://monoist.itmedia.co.jp/mn/series/25843/">https://monoist.itmedia.co.jp/mn/series/25843/</a></p><p><a href="#_ftnref2">[5]</a> Product Lifecycle Management</p><p><a href="#_ftnref3">[6]</a> Enterprise Resource Planning</p><p><a href="#_ftnref4">[7]</a> “<em>Each A380 consists of around 4 million individual components with 2.5 million part numbers produced by 1500 companies from 30 countries around the world.”</em> (<a href="https://www.airbus.com/sites/g/files/jlcbta136/files/2021-12/EN-Airbus-A380-Facts-and-Figures-December-2021_0.pdf">Airbus Facts &amp; Figures</a>)</p><p><a href="#_ftnref5">[8]</a> <a href="https://xtech.nikkei.com/atcl/nxt/news/18/10384/">https://xtech.nikkei.com/atcl/nxt/news/18/10384/</a></p><p><a href="#_ftnref6">[9]</a> <a href="https://www.intel.com/content/www/us/en/newsroom/news/intel-mobileye-iaa-mobility.html">https://www.intel.com/content/www/us/en/newsroom/news/intel-mobileye-iaa-mobility.html</a></p><p>(*) Although I am referencing to my current employer product, the content of this blog only reflects my own ideas.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=08e5d17b646c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>