<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Atul Upadhyay on Medium]]></title>
        <description><![CDATA[Stories by Atul Upadhyay on Medium]]></description>
        <link>https://medium.com/@atulatul_69582?source=rss-af1598684dd7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*ugbhlXppKRFLL5zdT0N-ig.jpeg</url>
            <title>Stories by Atul Upadhyay on Medium</title>
            <link>https://medium.com/@atulatul_69582?source=rss-af1598684dd7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 18:22:34 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@atulatul_69582/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[From Data Chaos to AI-Ready Analytics: Building a Semantic Layer at Multi-Billion-Row Scale]]></title>
            <link>https://medium.com/@atulatul_69582/from-data-chaos-to-ai-ready-analytics-building-a-semantic-layer-at-multi-billion-row-scale-e678f0fc8601?source=rss-af1598684dd7------2</link>
            <guid isPermaLink="false">https://medium.com/p/e678f0fc8601</guid>
            <category><![CDATA[self-service-analytics]]></category>
            <category><![CDATA[data-strategy]]></category>
            <category><![CDATA[microsoft-fabric]]></category>
            <category><![CDATA[semantic-layer]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Atul Upadhyay]]></dc:creator>
            <pubDate>Tue, 29 Sep 2026 14:11:22 GMT</pubDate>
            <atom:updated>2026-09-29T14:11:22.900Z</atom:updated>
            <content:encoded><![CDATA[<p><em>How we replaced conflicting KPIs and scattered reporting with one governed source of truth — stress-tested against 36 billion rows before building anything in production.</em></p><p>Every data leader wants their org to be “data-driven.” Almost none can answer one question cleanly: <strong>“What is the correct version of this KPI?”</strong></p><p>That was exactly the situation we encountered. The organization already had a modern data platform in place. The issue wasn’t a lack of data. The problem was that business logic had been recreated repeatedly across different tools and teams. The same metric existed in cubes, Power BI datasets, Excel macros, SQL views, and other reporting assets. Over time, each version evolved independently, resulting in conflicting definitions and inconsistent reporting.</p><p>As data volumes grew into the tens of billions of rows, the organization found itself rich in data but lacking a consistent source of truth.</p><p>Our objective was clear: establish a governed semantic layer that would provide a single, trusted definition of business metrics, enable true self-service analytics, and create a foundation suitable for AI-powered decision-making.</p><h3>Evaluating Multiple Architectural Options</h3><p>Rather than selecting technology based on familiarity, we approached the challenge through a structured proof of concept. We evaluated five architectural patterns: <em>Databricks → </em>Power BI, Microsoft Fabric → Power BI, Snowflake → Power BI, Databricks &amp; Fabric → Power BI, Snowflake &amp; Fabric → Power BI</p><p>Each option was tested using datasets ranging from <strong>6 billion to 36 billion rows</strong> and assessed across three critical dimensions: performance, governance capabilities, operational complexity.</p><p>The goal was not simply to identify the fastest platform, but to determine the most sustainable architecture for enterprise-scale analytics.</p><p>One key finding emerged early. While Microsoft Fabric Direct Lake delivers impressive performance, it is guardrail-bound (model complexity, capacity sizing). Recognizing this, we designed DirectQuery as a planned fallback, not a failure state.</p><p>The final architecture combined <strong>Databricks for data engineering</strong>, <strong>Microsoft Fabric and Power BI Semantic Models for governed analytics</strong>, and <strong>OneLake as the underlying storage</strong>.</p><h3>The Solution We Implemented</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FiFeZGXzZWxyYYYKygIqKg.png" /><figcaption><strong>Implemented Platform Architecture</strong></figcaption></figure><p>The implemented platform runs end to end as one governed pipeline:</p><ul><li><strong>Ingestion:</strong> Source systems (including Cloudera and Amazon S3) feed in through Apache NiFi for flow-based ingestion and Databricks Auto Loader for native, schema-evolving ingestion.</li><li><strong>Medallion architecture in Databricks:</strong> Bronze holds raw data as-ingested, schema-on-read, with full history. Silver cleanses, conforms, and validates it. Gold curates business-level aggregates ready for consumption — all as Delta tables, orchestrated through Databricks Jobs and Delta Live Tables.</li><li><strong>Governance backbone:</strong> Unity Catalog sits underneath the whole Lakehouse, providing unified lineage, access control, and governance across every Delta table — plus the supporting layer of Workflows, MLflow, Secrets, CI/CD, and monitoring that keeps it operable.</li><li><strong>Semantic layer:</strong> Gold tables are mirrored into OneLake with no data duplication, and exposed through Power BI Semantic Models running on Direct Lake — the single governed point where metrics and reuse actually live.</li><li><strong>Consumption:</strong> From there, the same governed metrics are exposed to business users through self-service Power BI, Microsoft Copilot, APIs and applications, and Databricks Genie for natural-language queries straight against the Lakehouse.</li></ul><p>The most important design principle is not any individual technology component. It is the existence of <strong>one centralized semantic layer</strong> that serves every consumption channel. Dashboards, AI assistants, applications, and APIs all reference the same metric definitions rather than maintaining their own versions of key business measures such as revenue or active customers.</p><h3>Governance Beyond Documentation</h3><p>A semantic layer is only as trustworthy as the governance process behind it. Every certified KPI went through: propose → document → feasibility review → business owner sign-off → data steward validation → build → test → certify → publish → monitor → version-controlled change. Every metric has a <strong>business owner and a data steward. </strong>This separation of responsibilities proved critical. The person defining a metric is not necessarily the person responsible for maintaining its quality and accuracy over time. Establishing clear ownership reduces the risk of metric drift and helps preserve trust long after initial implementation.</p><h3>A Real Performance Number</h3><p>Performance claims mean little without measurable targets. For this initiative, our benchmark was straightforward: Key analytical queries needed to complete within ~50 seconds against datasets containing 36 billion rows. This requirement influenced several architectural decisions, including:</p><ul><li>Partitioning strategies</li><li>Data model design</li><li>Relationship configurations</li><li>Direct Lake and DirectQuery usage patterns</li></ul><p>The semantic models contained between 100 and 200 KPIs each and were tested under realistic concurrent access patterns rather than isolated demonstration scenarios.</p><h3>We Tested the AI-Readiness Claim</h3><p>Instead of asserting “AI-ready,” we ran <strong>Databricks Genie, Microsoft Copilot, and Fabric Data Agent</strong> against more than 50 business prompts checked against ground truth:</p><ul><li><strong>Fabric Data Agent:</strong> fastest response time (averaging ~39s), easiest setup, text-only</li><li><strong>Databricks Genie:</strong> average response time ~44s, best UI/UX and visualization, strongest feedback loop</li><li><strong>Copilot:</strong> ~68s on Databricks data / ~45s on Fabric data, but best enterprise integration and exploration</li></ul><p>All three generated queries closely matched actual SQL/DAX logic — proof the model’s definitions were clear enough for an LLM to reason over. Hallucination still appeared on edge cases in all three: a good semantic layer reduces that risk, it doesn’t erase it.</p><h3>Business Outcomes</h3><p>The initiative delivered several tangible benefits:</p><ul><li>A single trusted definition for every business metric</li><li>Elimination of conflicting KPI discussions</li><li>Faster access to insights at production scale</li><li>Self-service analytics with reduced dependency on support teams</li><li>Simplified governance through centralized lineage and access management</li><li>A validated AI foundation accessible through dashboards, Copilot, Genie, and applications</li></ul><h3>Key Lessons Learned</h3><ul><li>Run the PoC across platforms you’re <em>not</em> already biased toward</li><li>Design the grain on purpose — it buys more performance than query tuning ever will</li><li>Two owners per metric, or accountability evaporates</li><li>Test AI-readiness with real assistants and real prompts — don’t just claim it</li><li>One semantic layer, many consumption paths — never the reverse</li></ul><h3>Looking Forward</h3><p>The future of analytics isn’t another dashboard — it’s a shared business language an executive, an analyst, and an AI assistant can all query and trust equally. A governed semantic layer makes this possible by ensuring every insight and decision is based on a single, consistent source of truth. As organizations adopt AI at scale, this foundation will become far more valuable than any individual reporting tool.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e678f0fc8601" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>