<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Nadakkannu Kuthalaraja on Medium]]></title>
        <description><![CDATA[Stories by Nadakkannu Kuthalaraja on Medium]]></description>
        <link>https://medium.com/@nadakkannu.smart?source=rss-d35209138c2e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*jvxerbOyCiYZXtrxlUp0PQ.jpeg</url>
            <title>Stories by Nadakkannu Kuthalaraja on Medium</title>
            <link>https://medium.com/@nadakkannu.smart?source=rss-d35209138c2e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 11:23:22 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nadakkannu.smart/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The “One Platform” Strategy: A Solution Architect’s Perspective on Databricks]]></title>
            <link>https://medium.com/@nadakkannu.smart/the-one-platform-strategy-a-solution-architects-perspective-on-databricks-7cd39e0a2645?source=rss-d35209138c2e------2</link>
            <guid isPermaLink="false">https://medium.com/p/7cd39e0a2645</guid>
            <category><![CDATA[databricks-solutions]]></category>
            <category><![CDATA[one-platform]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[all-in-one-platform]]></category>
            <category><![CDATA[business]]></category>
            <dc:creator><![CDATA[Nadakkannu Kuthalaraja]]></dc:creator>
            <pubDate>Tue, 07 Jul 2026 15:47:42 GMT</pubDate>
            <atom:updated>2026-07-07T15:47:42.186Z</atom:updated>
            <content:encoded><![CDATA[<p>One of the conversations I have more frequently with enterprise customers isn’t about technology, it’s about license optimization.</p><p>Many organizations have built their analytics ecosystem over several years:</p><ul><li>One platform for data ingestion and ETL</li><li>Another for the data warehouse</li><li>A separate reporting platform</li><li>Multiple AI/ML tools</li><li>Additional orchestration and automation products</li><li>Separate environments for custom applications</li></ul><p>While each tool solves a specific problem, the overall architecture often becomes expensive, complex to govern, and difficult to scale.</p><p>As Solution Architects, our responsibility is not only to design scalable data platforms but also to simplify the technology landscape and maximize business value.</p><p>This is where Databricks. stands out as a unified Data Intelligence Platform.</p><p>Instead of managing multiple disconnected platforms, organizations can consolidate many workloads into a single ecosystem:</p><ol><li>Data Engineering with scalable ETL/ELT pipelines</li></ol><p>2. Lakehouse architecture powered by Delta Lake</p><p>3. SQL Analytics and interactive dashboards</p><p>4. AI/BI capabilities for conversational analytics and intelligent insights</p><p>5. Machine Learning and Generative AI development</p><p>6. Data Governance through Unity Catalog</p><p>7. Workflow orchestration and automation</p><p>8. Real time streaming and batch processing</p><p>9. Databricks Apps for building business-friendly internal applications</p><p>10. Python, SQL, Spark, ML, and AI development in one collaborative workspace</p><p>From an architectural perspective, this consolidation delivers several advantages:</p><ul><li>Reduced platform complexity</li><li>Simplified governance and security</li><li>Lower operational overhead</li><li>Faster project delivery</li><li>Better collaboration between Data Engineers, BI Developers, Data Scientists, and Business Users</li><li>Improved scalability for enterprise workloads</li><li>Potential opportunities to optimize software licensing and infrastructure costs, depending on the organization’s current technology stack</li></ul><p>The goal isn’t simply to replace tools.</p><p>The goal is to build a modern, unified data platform where data engineering, analytics, AI, governance, and application development work together seamlessly.</p><p>As architects, we should always ask one question:</p><p>“Are we adding another tool to solve a problem, or are we fully utilizing the capabilities of the platform we already have?”</p><p>Sometimes the biggest transformation isn’t adopting new technology, it’s simplifying the technology ecosystem.</p><p>#Databricks #Lakehouse #DataArchitecture #SolutionArchitecture #DataEngineering #ArtificialIntelligence #GenerativeAI #BusinessIntelligence #Analytics #DataPlatform #CloudArchitecture #UnityCatalog #DataGovernance #EnterpriseArchitecture #DigitalTransformation</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7cd39e0a2645" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Conversational Analytics on Top of a Centralized Power BI Semantic Model Using Databricks…]]></title>
            <link>https://medium.com/@nadakkannu.smart/building-conversational-analytics-on-top-of-a-centralized-power-bi-semantic-model-using-databricks-bc988e8c83f5?source=rss-d35209138c2e------2</link>
            <guid isPermaLink="false">https://medium.com/p/bc988e8c83f5</guid>
            <category><![CDATA[conversational-analytics]]></category>
            <category><![CDATA[microsoft-power-bi]]></category>
            <category><![CDATA[power-bi-semantic-model]]></category>
            <category><![CDATA[databricks-ai]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Nadakkannu Kuthalaraja]]></dc:creator>
            <pubDate>Mon, 06 Jul 2026 03:30:20 GMT</pubDate>
            <atom:updated>2026-07-06T03:30:20.918Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building Conversational Analytics on Top of a Centralized Power BI Semantic Model Using Databricks AI</h3><h3>Introduction</h3><p>Enterprise Business Intelligence has evolved significantly over the past decade. Organizations have invested heavily in building centralized semantic models that provide governed, reusable, and trusted business metrics. These semantic models have become the foundation for executive dashboards, operational reports, and self service analytics.</p><p>However, business users still spend considerable time navigating dashboards, applying filters, and searching for insights.</p><p>The next evolution of Business Intelligence is <strong>Conversational Analytics </strong>allowing users to interact with enterprise data using natural language.</p><p>Imagine asking questions like:</p><ul><li>“Which region contributed the most revenue this quarter?”</li><li>“Why did sales decrease compared to last month?”</li><li>“Which product categories are driving the highest profit?”</li><li>“Summarize the financial performance of the West region.”</li></ul><p>Instead of manually opening reports, the system understands the question, retrieves the relevant business data, performs the analysis, and presents both visualizations and narrative insights.</p><p>This article explains how a centralized <strong>Power BI Semantic Model</strong> can be combined with <strong>Databricks AI</strong> to build an enterprise-grade conversational analytics solution.</p><h3>Enterprise Architecture</h3><p>The architecture consists of four major layers.</p><h3>1. Enterprise Data Layer</h3><p>The foundation begins with an enterprise data platform where data is ingested from multiple operational systems.</p><p>Typical sources include:</p><ul><li>ERP</li><li>CRM</li><li>Finance Systems</li><li>HR Applications</li><li>Manufacturing Systems</li><li>Marketing Platforms</li></ul><p>Data is processed through a modern Medallion Architecture:</p><ul><li>Bronze Layer</li><li>Silver Layer</li><li>Gold Layer</li></ul><p>This ensures high quality, standardized, and governed business data.</p><h3>2. Centralized Power BI Semantic Layer</h3><p>Instead of creating separate datasets for every report, organizations can build one centralized semantic model.</p><p>The semantic model includes:</p><ul><li>Large Fact Tables</li><li>Shared Dimension Tables</li><li>Star Schema</li><li>Business KPIs</li><li>DAX Measures</li><li>Calculation Groups</li><li>Row Level Security</li><li>Business friendly naming conventions</li></ul><p>This becomes the organization’s single source of truth.</p><p>Benefits include:</p><ul><li>Consistent KPI calculations</li><li>Improved governance</li><li>Reduced duplication</li><li>Better report performance</li><li>Reusable business logic</li></ul><h3>3. Databricks AI Layer</h3><p>This is where conversational analytics begins.</p><p>Using a Databricks App, organizations can build an intelligent interface powered by Large Language Models (LLMs).</p><p>The application can include:</p><ul><li>Python</li><li>Streamlit</li><li>Databricks Apps</li><li>Databricks SQL</li><li>AI Functions</li><li>Model Serving</li><li>Vector Search</li><li>Retrieval Augmented Generation (RAG)</li></ul><p>Rather than querying raw tables, the AI understands the semantic model and business metadata.</p><h3>4. Conversational User Experience</h3><p>Business users simply ask questions using natural language.</p><p>Examples include:</p><blockquote><em>Show total revenue by country.</em></blockquote><blockquote><em>Compare last quarter with the previous quarter.</em></blockquote><blockquote><em>Why did customer churn increase?</em></blockquote><blockquote><em>Which business unit exceeded its sales target?</em></blockquote><p>The AI translates these requests into analytical queries, retrieves the required data, and returns:</p><ul><li>Charts</li><li>Tables</li><li>Executive summaries</li><li>Trend analysis</li><li>Key insights</li><li>Suggested follow up questions</li></ul><p>The experience feels more like having a conversation with a business analyst than navigating a traditional dashboard.</p><h3>High-Level Implementation Flow</h3><p>A practical implementation can follow these steps:</p><p><strong>Step 1:</strong> Build an enterprise scale Power BI Semantic Model using a well designed Star Schema.</p><p><strong>Step 2:</strong> Publish and certify the semantic model as the trusted analytical layer.</p><p><strong>Step 3:</strong> Develop a Databricks App that serves as the conversational interface.</p><p><strong>Step 4:</strong> Enable the application to understand semantic model metadata, including measures, dimensions, hierarchies, KPIs, and business definitions.</p><p><strong>Step 5:</strong> Use an LLM with Retrieval Augmented Generation (RAG) to interpret user intent and map questions to business concepts.</p><p><strong>Step 6:</strong> Generate analytical queries against the semantic layer while honoring existing security rules.</p><p><strong>Step 7:</strong> Return results as interactive visualizations, narrative explanations, and actionable recommendations.</p><h3>Key Design Considerations</h3><p>Building conversational analytics requires more than connecting an AI model to data.</p><p>Important considerations include:</p><ul><li>Enterprise data governance</li><li>Semantic consistency</li><li>Performance optimization</li><li>Security and Row Level Security (RLS)</li><li>Metadata management</li><li>Prompt engineering</li><li>Business glossary integration</li><li>Audit logging</li><li>Responsible AI practices</li></ul><p>A well-governed semantic model significantly improves the quality and reliability of AI generated insights.</p><h3>Business Benefits</h3><p>Organizations can realize several advantages:</p><ul><li>Faster access to business insights</li><li>Reduced dependence on BI developers</li><li>Higher adoption of analytics</li><li>Consistent KPI definitions</li><li>Improved executive decision making</li><li>Better user experience through natural language</li><li>Scalable AI built on trusted enterprise data</li></ul><p>Instead of searching through dozens of dashboards, business users can simply ask questions and receive meaningful answers in seconds.</p><h3>Looking Ahead</h3><p>The future of Business Intelligence lies in combining governed semantic models with conversational AI.</p><p>Power BI provides a trusted semantic layer, while Databricks brings powerful AI capabilities that can transform how users interact with enterprise data.</p><p>As organizations continue investing in modern data platforms, conversational analytics will become a natural extension of enterprise BI moving from static dashboards to intelligent, interactive business conversations.</p><p>The goal is no longer just to visualize data, but to make enterprise knowledge accessible through natural language, empowering every business user to make faster, smarter, and more confident decisions.</p><p><strong>What are your thoughts?</strong></p><p>Have you explored combining Power BI Semantic Models with Databricks AI for conversational analytics? I’d love to hear how your organization is approaching the next generation of enterprise BI.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bc988e8c83f5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Avoiding the Medallion Mess: Most Common Mistakes in Data Lakehouse Design]]></title>
            <link>https://medium.com/@nadakkannu.smart/avoiding-the-medallion-mess-most-common-mistakes-in-data-lakehouse-design-3016adbe882b?source=rss-d35209138c2e------2</link>
            <guid isPermaLink="false">https://medium.com/p/3016adbe882b</guid>
            <category><![CDATA[medallion-architecture]]></category>
            <category><![CDATA[common-mistakes]]></category>
            <dc:creator><![CDATA[Nadakkannu Kuthalaraja]]></dc:creator>
            <pubDate>Mon, 12 Jan 2026 04:16:34 GMT</pubDate>
            <atom:updated>2026-01-12T04:16:34.350Z</atom:updated>
            <content:encoded><![CDATA[<p>The Medallion Architecture (Bronze, Silver, and Gold) has become the de facto standard for organizing data in modern Lakehouses. It promises a clear path from raw data to actionable insights. However, many teams find themselves six months into a project with a “Bronze” layer that is an unmanageable swamp, a “Silver” layer that provides no value, and a “Gold” layer that is consistently late.</p><p>Here is a breakdown of the most common mistakes teams make when applying the Medallion approach, why they happen, and how to fix them.</p><h3>1. The “Silver” Identity Crisis: Over-Processing or Under-Cleaning</h3><h3>The Mistake</h3><p>Treating the Silver layer as either a carbon copy of Bronze or a premature version of Gold. Teams often fall into two traps:</p><ul><li><strong>The Pass-Through:</strong> Moving data from Bronze to Silver with zero cleaning or schema enforcement.</li><li><strong>The Pre-Aggregation:</strong> Applying business logic (like summing sales by region) in Silver before the data is actually “cleared” for general use.</li></ul><h3>Why it Happens</h3><p>A lack of clear definitions. Teams often view Silver as “just the middle part” rather than the <strong>Source of Truth</strong>. Without a strict definition of what “Clean” means, engineers either do too little or too much.</p><h3>The Fix</h3><p><strong>Define Silver as the “Integration Layer.”</strong> * <strong>Fix:</strong> Ensure Silver data is always filtered, cleaned, and augmented. It should be “Join-ready.”</p><ul><li><strong>Rule of Thumb:</strong> If a Data Scientist wants to build a model, they should start at Silver. If an Executive wants a dashboard, they should look at Gold.</li></ul><h3>2. Neglecting the “Raw” in Bronze</h3><h3>The Mistake</h3><p>Applying schema-on-write or heavy transformations before the data even hits the Bronze layer. When the source schema changes at the API or Database level, the ingestion pipeline breaks immediately.</p><h3>Why it Happens</h3><p>Engineers often want to “fix” data as early as possible to avoid downstream errors. This “proactive” approach actually creates a brittle architecture where data is lost if it doesn’t fit the expected mold.</p><h3>The Fix</h3><p><strong>Treat Bronze as Immutable and Schemaless.</strong></p><ul><li><strong>Fix:</strong> Use cloudFiles (Auto Loader) or similar tools to ingest data in its native format (JSON, Parquet, CSV).</li><li><strong>Pro-Tip:</strong> Include metadata columns in Bronze, such as _input_file_name and _ingest_timestamp. This allows you to &quot;re-play&quot; your entire architecture from Bronze if a logic bug is discovered in Silver or Gold.</li></ul><h3>3. The “Gold” Explosion: Logic Duplication</h3><h3>The Mistake</h3><p>Creating hundreds of Gold tables that share 90% of the same logic but vary slightly (e.g., gold_sales_eu, gold_sales_us, gold_sales_marketing). This leads to &quot;Logic Drift,&quot; where two dashboards showing &quot;Total Revenue&quot; provide different numbers.</p><h3>Why it Happens</h3><p>Pressure from different business units to have “their” data. Instead of building a robust, reusable semantic layer, data engineers take the path of least resistance: Copy-Paste-Tweak.</p><h3>The Fix</h3><p><strong>Adopt a Semantic Hub Model.</strong></p><ul><li><strong>Fix:</strong> Build core “Feature” or “Dimension” tables in Silver/Gold that serve multiple use cases.</li><li><strong>Tooling:</strong> Use a transformation tool like <strong>dbt</strong> (data build tool) or <strong>SQL Mesh</strong> to manage dependencies and ensure that shared logic lives in a single macro or model.</li></ul><h3>4. Treating Medallion as a One-Way Street (Ignoring Data Quality)</h3><h3>The Mistake</h3><p>Assuming that data only moves forward and never checking if the data in Gold actually makes sense compared to the source. Teams often build “Fire and Forget” pipelines.</p><h3>Why it Happens</h3><p>The Medallion approach looks like a waterfall in diagrams. This leads to the misconception that once data passes a stage, it’s “someone else’s problem.”</p><h3>The Fix</h3><p><strong>Implement “Circuit Breakers.”</strong></p><ul><li><strong>Fix:</strong> Use Data Quality frameworks (like Great Expectations or Delta Live Tables Expectations).</li><li><strong>Implementation:</strong> If a Silver table fails a “null check” on a primary key, stop the pipeline before it reaches Gold. It is better to have “stale” data in a dashboard than “wrong” data.</li></ul><h3>5. Over-Engineering the “Gold” Layer for Small Teams</h3><h3>The Mistake</h3><p>Small startups with three engineers trying to build a complex, multi-layered Medallion architecture with complex partitioning and Z-Ordering for data that is only a few gigabytes.</p><h3>Why it Happens</h3><p>“Big Data Envy.” Teams follow blogs from Netflix or Uber and apply those same complexities to small-scale problems, leading to massive maintenance overhead.</p><h3>The Fix</h3><p><strong>Start with “Bronze-to-Gold” and Refactor.</strong></p><ul><li><strong>Fix:</strong> For very small datasets, it is okay to skip a formal Silver layer if the transformation is simple.</li><li><strong>Principle:</strong> Focus on <strong>Value-to-Market</strong>. Build the Gold table the business needs today. As complexity grows and you find yourself repeating cleaning logic, <em>then</em> extract that logic into a Silver layer.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/802/1*B7XbY3yDoPWoy5_9hYPOwg.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3016adbe882b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Engineering Excellence: Architecting Medallion Layers Across Different Clouds]]></title>
            <link>https://medium.com/@nadakkannu.smart/engineering-excellence-architecting-medallion-layers-across-different-clouds-c27ac70c4855?source=rss-d35209138c2e------2</link>
            <guid isPermaLink="false">https://medium.com/p/c27ac70c4855</guid>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[medallion-architecture]]></category>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[microsoft-fabric]]></category>
            <dc:creator><![CDATA[Nadakkannu Kuthalaraja]]></dc:creator>
            <pubDate>Mon, 29 Dec 2025 03:12:21 GMT</pubDate>
            <atom:updated>2025-12-29T03:12:21.152Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Medallion Architecture</strong> is not just a technical configuration — it is a <strong>Data Maturity Model</strong>.</p><p>It enables organizations to evolve from being merely <strong>data heavy</strong> to truly <strong>data driven</strong> by ensuring that every byte of information is <strong>refined, validated, and analytics ready</strong> to drive measurable business outcomes.</p><p>At its core, Medallion Architecture is a <strong>data design pattern for the Lakehouse</strong>, where data is logically organized into progressive layers. As data moves through each layer, its <strong>structure, quality, and usability improve incrementally</strong>, transforming raw data into trusted, business-ready insights.<br>The Business Value of the Three Layers</p><p>To understand why this matters to the business, we can look at the architecture as a journey of increasing value:</p><ul><li><strong>The Bronze Layer (The Vault):</strong> This is where we capture everything. It ensures that the business never loses a single detail from its original sources (Sales, Marketing, IoT, etc.). It provides a “permanent memory” for the company.</li><li><strong>The Silver Layer (The Clean Room):</strong> This is where the chaos is managed. We remove duplicates, fix errors, and align different departments. From a business perspective, this is the <strong>“Single Version of the Truth”</strong> — where Marketing and Finance finally agree on the same numbers.</li><li><strong>The Gold Layer (The Storefront):</strong> This is the high-value, “ready-to-consume” data. It is optimized for speed and clarity, designed specifically for executive dashboards, quarterly reports, and predictive AI models.</li></ul><p><strong>Technical Deep-Dive: Medallion Implementations</strong></p><blockquote><strong>1.</strong> <strong>Databricks</strong></blockquote><blockquote><strong>2.</strong> <strong>Snowflake</strong></blockquote><blockquote><strong>3.</strong> <strong>Microsoft Fabric</strong></blockquote><h4><strong>1. Databricks: The Unified Lakehouse</strong></h4><p>Databricks uses a <strong>Code-First</strong> approach, leveraging the power of Apache Spark and Delta Lake for maximum flexibility and performance.</p><p><strong>Bronze (Ingestion):</strong></p><ul><li><strong>Technology:</strong> <strong>Auto Loader</strong> (cloudFiles). It incrementally and efficiently processes new data files as they arrive in cloud storage (S3/ADLS).</li><li><strong>Technique:</strong> Schema Inference and <strong>Schema Evolution</strong> to handle changing raw data structures without breaking pipelines.</li></ul><p><strong>Silver (Transformation):</strong></p><ul><li><strong>Technology:</strong> <strong>Delta Live Tables (DLT)</strong>. This is a declarative framework for building reliable data pipelines.</li><li><strong>Technique:</strong> <strong>Expectations</strong> (Data Quality Gates) to quarantine bad records. Transformations use <strong>PySpark</strong> or <strong>Spark SQL</strong> for joining and deduplication.</li></ul><p><strong>Gold (Curated):</strong></p><ul><li><strong>Technology:</strong> <strong>Databricks SQL</strong> and <strong>Unity Catalog</strong>.</li><li><strong>Technique:</strong> <strong>Z-Ordering</strong> and <strong>Liquid Clustering</strong> to optimize tables for high-speed BI queries. Data is often modeled into <strong>Star Schemas</strong> (Facts/Dimensions).</li></ul><p><strong>2. Snowflake: The SQL-First Data Cloud</strong></p><p>Snowflake focuses on <strong>Automation and Performance</strong>, treating the Medallion architecture as a high-performance relational stream.</p><p><strong>Bronze (Landing):</strong></p><ul><li><strong>Technology:</strong> <strong>Snowpipe</strong> for automated micro-batch loading from external stages.</li><li><strong>Technique:</strong> <strong>VARIANT</strong> data types to store raw JSON/XML without pre-defining a schema (Schema-on-read).</li></ul><p><strong>Silver (Processing):</strong></p><ul><li><strong>Technology:</strong> <strong>Streams &amp; Tasks</strong> or <strong>Dynamic Tables</strong>.</li><li><strong>Technique:</strong> <strong>Change Data Capture (CDC)</strong> using Streams to track offsets. Use <strong>Snowpark (Python/Java)</strong> if complex logic is needed beyond standard SQL.</li></ul><p><strong>Gold (Analytics):</strong></p><ul><li><strong>Technology:</strong> <strong>Materialized Views</strong> and <strong>Search Optimization Service</strong>.</li><li><strong>Technique:</strong> Heavy use of <strong>Secure Data Sharing</strong> to expose Gold datasets to partners or clients without moving the actual data.</li></ul><p><strong>3. Microsoft Fabric: The SaaS-Integrated Lakehouse</strong></p><p>Fabric is <strong>Integration-First</strong>, providing a unified environment where storage (OneLake) is shared across all tools.</p><p><strong>Bronze (Raw):</strong></p><ul><li><strong>Technology:</strong> <strong>Data Factory Pipelines</strong> and <strong>OneLake Shortcuts</strong>.</li><li><strong>Technique:</strong> <strong>Shortcutting</strong> allows you to “virtualize” data from AWS S3 or ADLS Gen2 directly into the Bronze layer without physically copying it.</li></ul><p><strong>Silver (Refinement):</strong></p><ul><li><strong>Technology:</strong> <strong>Fabric Notebooks (Spark)</strong> or <strong>Dataflows Gen2</strong>.</li><li><strong>Technique:</strong> <strong>V-Order</strong> optimization on Delta files for lightning-fast reads. Low-code users use Dataflows, while engineers use Spark for complex cleansing.</li></ul><p><strong>Gold (Business):</strong></p><ul><li><strong>Technology:</strong> <strong>SQL Analytics Endpoint</strong> and <strong>Power BI (Direct Lake mode)</strong>.</li><li><strong>Technique:</strong> <strong>Direct Lake</strong> technology allows Power BI to read Gold Delta tables directly from OneLake without importing data, providing real-time speed on massive datasets.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/907/1*tyTkIQsIIYGbTn0hwf5hmQ.png" /></figure><h4><strong>4. Security Comparison Summary</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/901/1*46JWiXTPiQSiXpiWR9r-eg.png" /></figure><p><strong>Final Thoughts</strong></p><p>There is no “one-size-fits-all” in modern data engineering.</p><ul><li><strong>Go Databricks</strong> if you need high-performance Spark and ML.</li><li><strong>Go Snowflake</strong> if you want a powerful, automated SQL data cloud.</li><li><strong>Go Microsoft Fabric</strong> if you want a unified, SaaS-like experience for your entire organization.</li></ul><p>The Medallion Architecture is the bridge between raw data and real-world impact. Start building your Bronze layer today, but always keep your eyes on the Gold.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c27ac70c4855" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The End of ETL? Mirroring vs. Shortcuts in Microsoft Fabric]]></title>
            <link>https://medium.com/@nadakkannu.smart/the-end-of-etl-mirroring-vs-shortcuts-in-microsoft-fabric-e5f070aec293?source=rss-d35209138c2e------2</link>
            <guid isPermaLink="false">https://medium.com/p/e5f070aec293</guid>
            <category><![CDATA[fabric-shortcut]]></category>
            <category><![CDATA[fabric-mirroring]]></category>
            <category><![CDATA[fabric]]></category>
            <category><![CDATA[microsoft-fabric]]></category>
            <dc:creator><![CDATA[Nadakkannu Kuthalaraja]]></dc:creator>
            <pubDate>Fri, 26 Dec 2025 01:10:50 GMT</pubDate>
            <atom:updated>2025-12-26T01:10:50.968Z</atom:updated>
            <content:encoded><![CDATA[<p>The fundamental goal of both Mirroring and Shortcuts is to eliminate the traditional “Extract, Transform, Load” (ETL) pain points. They both provide a way to access data without manually building and managing complex pipelines, but they do so using two very different architectural approaches.</p><ol><li>Detailed Architecture &amp; Comparison</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/660/1*w2mSVzKkKHNcOrzmRIq4ZQ@2x.jpeg" /></figure><p>2. Deep Dive: Mirroring (The “Live Replica”)</p><p>Mirroring is a replication technology. It doesn’t just link to data; it brings a copy over in a very specific way.</p><ul><li>The Architecture: It uses Change Data Capture (CDC) technology. When a row changes in your source SQL database, that change is streamed in near real-time to Fabric’s OneLake.</li><li>• The Transformation: It automatically converts proprietary data (like SQL Server rows) into Delta Parquet format. This is huge because it makes “closed” data “open.”</li><li>• Key Use Case: You have an operational database (Azure SQL, Cosmos DB, or Snowflake) and you want to run heavy analytics without slowing down the production app.</li><li>• The “Secret Sauce”: Once mirrored, the data is automatically indexed and available for “Direct Lake” mode in Power BI — meaning your reports update almost the second the database does.</li></ul><p>3. Deep Dive: Shortcuts (The “Virtual Portal”)</p><ul><li>Shortcuts are the ultimate “Zero-Copy” solution.</li><li>• The Architecture: Think of a Shortcut as a symbolic link. The data stays exactly where it is (AWS S3, Google Cloud Storage, or ADLS Gen2). Fabric simply “points” to it.</li><li>• The Mechanism: When you write a SQL query in Fabric against a Shortcut, Fabric’s compute engine reaches out to the external cloud, pulls the necessary bits, and processes them in memory.</li><li>• Key Use Case: You have a massive 100TB Data Lake in AWS S3. You don’t want to pay for 100TB of storage in Fabric, and you don’t want to wait 10 hours for a transfer. You create a Shortcut and start querying in seconds.</li><li>• The Limitation: Because the data is “remote,” network latency can impact performance. It is best used when data is already in an open format (Parquet/Delta).</li></ul><p>4. Strategic Advice: Which one to choose?</p><p>This is the “expert opinion” section of your Medium post:</p><ul><li>Choose Mirroring when your data is “locked” in a database and needs to be converted into a warehouse-ready format for high-performance reporting.</li><li>• Choose Shortcuts when you have a “Multi-Cloud” strategy. If your data is already sitting in S3 or GCS, Shortcuts allow you to use Fabric as your unified analytics pane without moving a single byte.</li></ul><p>5. The “Unified” Vision</p><ul><li>End your post by explaining that you don’t have to choose just one. A modern Fabric architecture uses Shortcuts to connect to existing Data Lakes and Mirroring to bring in operational SQL data. Together, they create a “Single Source of Truth” without the traditional ETL nightmare.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e5f070aec293" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introducing Databricks LakeFlow — Simplifying Data Integration and Orchestration]]></title>
            <link>https://medium.com/@nadakkannu.smart/introducing-databricks-lakeflow-simplifying-data-integration-and-orchestration-8ed9e33f2484?source=rss-d35209138c2e------2</link>
            <guid isPermaLink="false">https://medium.com/p/8ed9e33f2484</guid>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[databricks-lakeflow]]></category>
            <dc:creator><![CDATA[Nadakkannu Kuthalaraja]]></dc:creator>
            <pubDate>Sun, 15 Jun 2025 17:55:05 GMT</pubDate>
            <atom:updated>2025-06-15T17:55:05.217Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introducing Databricks LakeFlow — Simplifying Data Integration and Orchestration</h3><p><strong>Databricks LakeFlow</strong> is a new feature introduced in 2025 by Databricks that simplifies data integration and orchestration across the lakehouse platform. It provides a visual, unified, and fully managed solution for building ETL pipelines, data workflows, and event-driven processing — all natively integrated with the Databricks Lakehouse.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/985/0*t1mgTHyQn0VQLajr" /></figure><p><strong>What is Databricks LakeFlow?</strong></p><p><strong>LakeFlow</strong> is a <strong>serverless, low-code</strong> framework for:</p><ul><li>Building real-time and batch data pipelines</li><li>Performing data ingestion, transformation, and orchestration</li><li>Managing pipeline dependencies, retries, error handling, and scheduling</li></ul><p>It combines the best of:</p><ul><li><strong>Data engineering tools</strong> (like Apache Airflow, DBT)</li><li><strong>Event-driven architecture</strong> (Kafka-like triggers)</li><li><strong>Delta Live Tables (DLT)</strong> enhancements</li><li><strong>Databricks Workflows</strong></li></ul><p><strong>Key Features of Databricks LakeFlow</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-xwJmJFw8ZYRyIUi3bacUQ.png" /></figure><p><strong>What Can You Do With LakeFlow?</strong></p><p><strong>1. Data Ingestion</strong></p><ul><li>Load data from files (CSV, Parquet, JSON), cloud storage, databases, Kafka, etc.</li><li>Automatically detect schema and partitioning</li></ul><p><strong>2. Data Transformation</strong></p><ul><li>Use <strong>PySpark, SQL, Delta Live Tables (DLT)</strong> for clean and validated datasets</li><li>Add <strong>data quality checks</strong>, enrichments, and joins</li></ul><p><strong>3. Workflow Orchestration</strong></p><ul><li>Chain jobs like <strong>extract → transform → load → ML model scoring</strong></li><li>Add <strong>if/else branching</strong>, <strong>retries</strong>, <strong>error handling</strong></li></ul><p><strong>4. Real-Time Use Cases</strong></p><ul><li>Build <strong>streaming pipelines</strong> for fraud detection, alerts, sensor processing</li><li>Ingest and process <strong>IoT data, logs, CDC events</strong></li></ul><p><strong>5. Data Delivery</strong></p><ul><li>Write results to <strong>Delta Lake, data warehouse, ML models</strong>, or external systems</li><li><strong>Notify users or downstream systems</strong> via webhook or email</li></ul><p><strong>Why Use Databricks LakeFlow?</strong></p><ul><li>Removes need for 3rd-party tools like <strong>Airflow</strong>, <strong>DBT Cloud</strong>, or <strong>Informatica</strong></li><li><strong>Unified experience</strong> for data engineers, scientists, and analysts</li><li><strong>Governed by Unity Catalog</strong>, supports <strong>role-based access</strong> and <strong>lineage</strong></li><li>Built for <strong>modern lakehouse architectures</strong> with <strong>low-code agility</strong> and <strong>AI readiness</strong></li></ul><p><strong>Real-world Use Case Example:</strong></p><blockquote>Ingest daily sales data from S3 → Clean it → Validate records → Aggregate → Publish to Power BI.</blockquote><p>Here’s a <strong>demo workflow example using Databricks LakeFlow</strong> that simulates a <strong>real-world pipeline</strong>: loading sales data, transforming it, validating it, and publishing results to a dashboard.</p><p><strong>LakeFlow Demo Workflow: Sales Data Pipeline</strong></p><p><strong>Use Case:</strong></p><p>Build a pipeline that:</p><ol><li>Ingests raw sales data from S3 (daily files)</li><li>Cleans and transforms the data</li><li>Applies quality checks</li><li>Publishes a summary to a Power BI or Tableau dashboard</li></ol><p><strong>Workflow Overview</strong></p><pre>     [Trigger] (S3 File Arrival: sales_YYYYMMDD.csv)<br>           ↓<br>  ┌────────────────────┐<br>  │ Ingest Sales Data  │  (Step 1)<br>  └────────────────────┘<br>           ↓<br>  ┌────────────────────┐<br>  │ Clean + Transform  │  (Step 2)<br>  └────────────────────┘<br>           ↓<br>  ┌────────────────────┐<br>  │ Data Quality Check │  (Step 3)<br>  └────────────────────┘<br>           ↓<br>  ┌────────────────────┐<br>  │ Publish to Delta   │  (Step 4)<br>  └────────────────────┘<br>           ↓<br>  ┌───────────────────────┐<br>  │ Notify BI Dashboard   │  (Step 5)<br>  └───────────────────────┘</pre><p><strong>LakeFlow Steps (with Details)</strong></p><p><strong>Step 1: Ingest Sales Data</strong></p><ul><li><strong>Source</strong>: AWS S3 bucket → /raw/sales/*.csv</li><li><strong>Trigger</strong>: File arrival event (event-driven or daily schedule)</li><li><strong>Tool</strong>: Built-in file connector or Auto Loader</li><li><strong>Output</strong>: Bronze Delta Table: bronze_sales</li></ul><p><strong>Step 2: Clean &amp; Transform</strong></p><ul><li><strong>Logic</strong>: Remove nulls, standardize date, join with dim_store and dim_product</li><li><strong>Tool</strong>: SQL or PySpark script</li><li><strong>Output</strong>: Silver Table: clean_sales</li></ul><pre>SELECT <br>  store_id,<br>  product_id,<br>  sale_date,<br>  amount,<br>  quantity<br>FROM tbl_Sales<br>WHERE amount IS NOT NULL AND quantity &gt; 0</pre><p><strong>Step 3: Data Quality Check</strong></p><ul><li><strong>Tool</strong>: LakeFlow validation step with DLT or SQL</li><li><strong>Rules</strong>:</li><li>No NULL store_id</li><li>Quantity &gt; 0</li><li>Amount &gt; 0</li></ul><p>Failing rows are logged to invalid_sales.</p><p><strong>Step 4: Publish to Delta</strong></p><ul><li><strong>Write to</strong>: gold_sales_summary</li><li><strong>Aggregation</strong>: Total revenue by product and region</li></ul><pre>SELECT<br>  product_id,<br>  region,<br>  SUM(amount) as total_revenue<br>FROM clean_sales<br>GROUP BY product_id, region</pre><p><strong>Step 5: Notify Dashboard</strong></p><ul><li><strong>Tool</strong>: Webhook or trigger that refreshes Power BI/Tableau extract</li><li><strong>Optional</strong>: Send Slack/email alert that data is updated</li></ul><p>What are your thoughts about this approach and let us know in the comments ?</p><p>Happy Learning !!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8ed9e33f2484" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Snowflake New Generation 2 Warehouse]]></title>
            <link>https://medium.com/@nadakkannu.smart/snowflake-new-generation-2-warehouse-bc8af8405c5d?source=rss-d35209138c2e------2</link>
            <guid isPermaLink="false">https://medium.com/p/bc8af8405c5d</guid>
            <category><![CDATA[snowflake-gen2]]></category>
            <category><![CDATA[snowflake-warehouses]]></category>
            <category><![CDATA[snowflake]]></category>
            <dc:creator><![CDATA[Nadakkannu Kuthalaraja]]></dc:creator>
            <pubDate>Fri, 30 May 2025 03:48:27 GMT</pubDate>
            <atom:updated>2025-05-30T03:48:27.952Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Snowflake Gen 2 isn’t a new warehouse type and it’s a smarter, more powerful evolution of the existing STANDARD engine.</strong><br> The fundamental upgrade lies in the underlying hardware and enhanced engine-level optimizations designed to address real-world performance bottlenecks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*Ur8KICsgqjJhm46FrFGuOQ.png" /></figure><blockquote><strong>What makes Gen 2 different?</strong></blockquote><p>It runs on faster, next-generation hardware (e.g., AWS Graviton3) and introduces deep architectural improvements that deliver:</p><ul><li><strong>Significantly better DML performance</strong> (MERGE, UPDATE, DELETE)</li><li><strong>Faster table scans and joins</strong> for large-scale analytics</li><li><strong>Improved concurrency management</strong> for high-volume workloads</li><li><strong>Reduced spin-up latency</strong> — warehouses start and scale faster</li><li><strong>Smarter parallelism</strong> out of the box for optimized query execution</li><li><strong>No SQL or pipeline changes required</strong> — just upgrade and go!</li></ul><blockquote><strong>Available Regions</strong></blockquote><p>This feature is Generally Available (GA) from its release, but it is currently only available in the following regions:</p><ul><li>AWS us-west-2 (Oregon)</li><li>AWS eu-central-1 (Frankfurt)</li><li>Azure East US 2 (Virginia)</li><li>Azure West Europe (Netherlands)</li></ul><blockquote><strong>Snowflake Service Consumption Table</strong></blockquote><p>Based on the Snowflake Service Consumption Table, the cost differences are as follows:<br> Gen 2 warehouses incur a higher credit usage compared to standard (Gen 1) warehouses:</p><ul><li>💰 <strong>1.35× higher on AWS</strong></li><li>💰 <strong>1.25× higher on Azure</strong></li></ul><p>These cost increases reflect the enhanced performance, improved concurrency, and faster execution delivered by the Gen 2 engine.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*an1n7vZ2GsvJLQAsDZ2JUA.png" /></figure><blockquote><strong>Warehouse — SQL Script</strong></blockquote><p>To use a “Generation 2 standard warehouse”, simply set resource_constraint = standard_gen_2. Execute the following query to define the warehouses for verification.</p><pre>-- The traditional warehouse for comparison<br><br>-- Create the Gen1 warehouse<br><br>CREATE OR REPLACE WAREHOUSE Snow_wh_gen1<br>WAREHOUSE_SIZE = MEDIUM<br>AUTO_SUSPEND = 60<br>INITIALLY_SUSPENDED = True<br>AUTO_RESUME = TRUE;<br><br>-- Create the Gen2 warehouse<br><br>CREATE OR REPLACE WAREHOUSE Snow_wh_gen2<br>WAREHOUSE_SIZE = MEDIUM<br>RESOURCE_CONSTRAINT = STANDARD_GEN_2<br>AUTO_SUSPEND = 60<br>INITIALLY_SUSPENDED = True<br>AUTO_RESUME = TRUE;<br><br>-- Migrate an Existing Gen1 Warehouse to Gen2<br><br>ALTER WAREHOUSE Snow_wh_gen1 SUSPEND;<br>ALTER WAREHOUSE Snow_wh_gen1 SET RESOURCE_CONSTRAINT = STANDARD_GEN_2;<br>ALTER WAREHOUSE Snow_wh_gen1 RESUME;<br><br></pre><p><strong>Important </strong>— Gen2 is <strong>not available</strong> for X5LARGE or X6LARGE. Use XLARGE or downsize to LARGE if needed.</p><blockquote><strong>Should You Migrate to Snowflake Gen 2? — Evaluation Checklist</strong></blockquote><p><strong>1. Workload Type</strong></p><p>✔️ Yes, if your workload is compute- or DML-intensive</p><p><strong>Detailed Note: </strong>Gen 2 significantly improves performance for MERGE, DELETE, UPDATE, and other transformation-heavy operations. Ideal for ETL/ELT pipelines, large joins, and data engineering tasks.</p><p><strong>2. Query Performance Bottlenecks</strong></p><p>✔️ Yes, if queries or jobs are slow in Gen 1</p><p><strong>Detailed Note: </strong>If you observe long-running queries, frequent queueing, or performance issues in dashboards or pipelines, Gen 2’s improved CPU utilization and execution engine may drastically reduce job time.</p><p><strong>3. Region Compatibility</strong></p><p>✔️ Yes, if your warehouse is in a supported region</p><p><strong>Detailed Note: </strong>As of now, Gen 2 is only available in selected AWS and Azure regions like us-west-2, eu-central-1, and East US 2. You cannot use Gen 2 if your Snowflake account is in an unsupported region or on GCP.</p><p><strong>4. Cost-Benefit Tradeoff</strong></p><p>✔️ Yes, if faster execution offsets increased per-second credit cost</p><p><strong>Detailed Note: </strong>Gen 2 consumes ~1.25× (Azure) to ~1.35× (AWS) more credits per second. However, jobs often complete significantly faster, reducing total cost. Conduct benchmarking to evaluate net credit savings.</p><p><strong>5. Concurrency Requirements</strong></p><p>✔️ Yes, if your workload involves many concurrent queries or users</p><p><strong>Detailed Note: </strong>Gen 2 handles higher concurrency better due to improved query scheduling and resource management. Helpful in BI dashboard environments and multi-user scenarios.</p><p><strong>6. ETL/ELT Pipelines</strong></p><p>✔️ Yes, if you run frequent, heavy transformations or scheduled tasks</p><p><strong>Detailed Note: </strong>Pipelines running MERGE, CTAS, or complex staging steps benefit from Gen 2’s optimized parallelism and I/O improvements.</p><p><strong>7. Large-Scale Data Processing</strong></p><p>✔️ Yes, if you work with large tables or data models</p><p><strong>Detailed Note: </strong>Gen 2 improves table scan speed and memory handling, making it more suitable for analytical workloads involving billions of rows or large partitioned data.</p><p><strong>8. Ease of Migration</strong></p><p>✔️ Yes, if you’re looking for a quick upgrade with minimal change</p><p>Migration involves only a change to the RESOURCE_CONSTRAINT property. No changes needed in SQL, data models, or pipelines. Downtime is minimal (warehouse must be suspended briefly).</p><p>What are your thoughts about this approach and let us know in the comments ?</p><p>Happy Learning !!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bc8af8405c5d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Create custom roles in Snowflake and assigning permissions]]></title>
            <link>https://medium.com/@nadakkannu.smart/how-to-create-custom-roles-in-snowflake-and-assigning-permissions-b750cd6c57b5?source=rss-d35209138c2e------2</link>
            <guid isPermaLink="false">https://medium.com/p/b750cd6c57b5</guid>
            <category><![CDATA[warehouse]]></category>
            <category><![CDATA[custom-roles]]></category>
            <category><![CDATA[snowflake]]></category>
            <dc:creator><![CDATA[Nadakkannu Kuthalaraja]]></dc:creator>
            <pubDate>Sun, 18 May 2025 01:46:44 GMT</pubDate>
            <atom:updated>2025-05-18T03:34:20.699Z</atom:updated>
            <content:encoded><![CDATA[<p>To create custom roles and grant appropriate permissions in Snowflake, you define the role using CREATE ROLE, grant database/schema/warehouse privileges using GRANT, and assign the role to a user with GRANT ROLE TO USER.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lHVUqJvxmx1M2fvv.jpg" /></figure><p>Creating custom roles in <strong>Snowflake</strong> and assigning permissions to them involves several key steps:</p><ol><li><strong>Create the custom roles</strong></li><li><strong>Grant privileges to those roles</strong></li><li><strong>Assign roles to users</strong></li></ol><p>Here’s how you can do this for:</p><ul><li>Tester</li><li>Developer</li><li>Senior Developer</li><li>Data Architect</li><li>Project Manager</li></ul><p><strong>Assumptions</strong></p><p>We’ll assume:</p><ul><li>Your database is called PROJ_DB</li><li>Your schema is called PUBLIC</li><li>You have a warehouse called PROJ_WH</li><li>Users already exist in the system</li></ul><ol><li><strong>Create Custom Roles</strong></li></ol><pre>-- Create custom roles<br>CREATE ROLE TESTER;<br>CREATE ROLE DEVELOPER;<br>CREATE ROLE SENIOR_DEVELOPER;<br>CREATE ROLE DATA_ARCHITECT;<br>CREATE ROLE PROJECT_MANAGER;</pre><p><strong>2. Grant Privileges to Roles</strong></p><blockquote>Role : Tester</blockquote><p>Testers usually need read-only access.</p><pre>GRANT USAGE ON WAREHOUSE PROJ_WH TO ROLE TESTER;<br>GRANT USAGE ON DATABASE PROJ_DB TO ROLE TESTER;<br>GRANT USAGE ON SCHEMA PROJ_DB.PUBLIC TO ROLE TESTER;<br>GRANT SELECT ON ALL TABLES IN SCHEMA PROJ_DB.PUBLIC TO ROLE TESTER;<br><br><br>-- Optional: To allow future tables to be readable:<br><br>GRANT SELECT ON FUTURE TABLES IN SCHEMA PROJ_DB.PUBLIC TO ROLE TESTER;</pre><blockquote>Role : Developer</blockquote><p>Developers usually need access to create and modify objects.</p><pre>GRANT USAGE ON WAREHOUSE PROJ_WH TO ROLE DEVELOPER;<br>GRANT USAGE, CREATE SCHEMA ON DATABASE PROJ_DB TO ROLE DEVELOPER;<br>GRANT USAGE ON SCHEMA PROJ_DB.PUBLIC TO ROLE DEVELOPER;<br><br>GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA PROJ_DB.PUBLIC TO ROLE DEVELOPER;<br>GRANT CREATE TABLE, CREATE VIEW, CREATE FUNCTION ON SCHEMA PROJ_DB.PUBLIC TO ROLE DEVELOPER;<br><br>-- Future objects<br><br>GRANT SELECT, INSERT, UPDATE, DELETE ON FUTURE TABLES IN SCHEMA PROJ_DB.PUBLIC TO ROLE DEVELOPER;</pre><blockquote>Role : Senior Developer</blockquote><p>Senior Developers can have elevated permissions, like modifying objects created by others.</p><pre>GRANT USAGE ON WAREHOUSE PROJ_WH TO ROLE SENIOR_DEVELOPER;<br>GRANT USAGE, CREATE SCHEMA ON DATABASE PROJ_DB TO ROLE SENIOR_DEVELOPER;<br>GRANT USAGE ON SCHEMA PROJ_DB.PUBLIC TO ROLE SENIOR_DEVELOPER;<br><br>GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA PROJ_DB.PUBLIC TO ROLE SENIOR_DEVELOPER;<br>GRANT ALL PRIVILEGES ON SCHEMA PROJ_DB.PUBLIC TO ROLE SENIOR_DEVELOPER;<br>GRANT CREATE TABLE, CREATE VIEW, CREATE FUNCTION, CREATE PROCEDURE TO ROLE SENIOR_DEVELOPER;<br><br>-- Future objects<br><br>GRANT ALL PRIVILEGES ON FUTURE TABLES IN SCHEMA PROJ_DB.PUBLIC TO ROLE SENIOR_DEVELOPER;</pre><blockquote>Role : Data Architect</blockquote><p>They design and manage structures across multiple schemas.</p><pre>GRANT USAGE ON WAREHOUSE PROJ_WH TO ROLE DATA_ARCHITECT;<br>GRANT ALL PRIVILEGES ON DATABASE PROJ_DB TO ROLE DATA_ARCHITECT;<br>GRANT ALL PRIVILEGES ON SCHEMA PROJ_DB.PUBLIC TO ROLE DATA_ARCHITECT;<br>GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA PROJ_DB.PUBLIC TO ROLE DATA_ARCHITECT;<br>GRANT ALL PRIVILEGES ON ALL VIEWS IN SCHEMA PROJ_DB.PUBLIC TO ROLE DATA_ARCHITECT;<br>GRANT MONITOR, OPERATE ON WAREHOUSE PROJ_WH TO ROLE DATA_ARCHITECT;<br><br>-- Future objects<br><br>GRANT ALL PRIVILEGES ON FUTURE TABLES IN SCHEMA PROJ_DB.PUBLIC TO ROLE DATA_ARCHITECT;<br>GRANT ALL PRIVILEGES ON FUTURE VIEWS IN SCHEMA PROJ_DB.PUBLIC TO ROLE DATA_ARCHITECT;</pre><blockquote>Role : Project Manager</blockquote><p>Usually needs visibility, not object modification.</p><pre>GRANT USAGE ON WAREHOUSE PROJ_WH TO ROLE PROJECT_MANAGER;<br>GRANT USAGE ON DATABASE PROJ_DB TO ROLE PROJECT_MANAGER;<br>GRANT USAGE ON SCHEMA PROJ_DB.PUBLIC TO ROLE PROJECT_MANAGER;<br>GRANT SELECT ON ALL TABLES IN SCHEMA PROJ_DB.PUBLIC TO ROLE PROJECT_MANAGER;<br>GRANT USAGE ON ALL VIEWS IN SCHEMA PROJ_DB.PUBLIC TO ROLE PROJECT_MANAGER;<br><br>-- Future objects<br><br>GRANT SELECT ON FUTURE TABLES IN SCHEMA PROJ_DB.PUBLIC TO ROLE PROJECT_MANAGER;</pre><p><strong>3. Assign Roles to Users</strong></p><pre>-- Assign roles to users (assuming users already exist)<br><br>GRANT ROLE TESTER TO USER alice;<br>GRANT ROLE DEVELOPER TO USER bob;<br>GRANT ROLE SENIOR_DEVELOPER TO USER carol;<br>GRANT ROLE DATA_ARCHITECT TO USER dave;<br>GRANT ROLE PROJECT_MANAGER TO USER emma;</pre><p><strong>4. Optional: Hierarchical Role Assignment</strong></p><p>If you want to simplify permissions:</p><pre>-- Grant roles to higher-level roles<br><br>GRANT ROLE DEVELOPER TO ROLE SENIOR_DEVELOPER;<br>GRANT ROLE SENIOR_DEVELOPER TO ROLE DATA_ARCHITECT;</pre><p>This means the Data Architect automatically inherits Senior Developer and Developer permissions.</p><p>What are your thoughts about this approach and let us know in the comments ?</p><p>Happy Learning !!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b750cd6c57b5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI-based Automated Report Generation Application]]></title>
            <link>https://medium.com/@nadakkannu.smart/ai-based-automated-report-generation-application-506793d27e41?source=rss-d35209138c2e------2</link>
            <guid isPermaLink="false">https://medium.com/p/506793d27e41</guid>
            <category><![CDATA[bi-reporting-tools]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[automated-reporting-tools]]></category>
            <dc:creator><![CDATA[Nadakkannu Kuthalaraja]]></dc:creator>
            <pubDate>Mon, 21 Apr 2025 18:59:28 GMT</pubDate>
            <atom:updated>2025-04-21T18:59:28.054Z</atom:updated>
            <content:encoded><![CDATA[<p>Here’s a technical solution for building My AI-based Automated Report Generation Application — broken down into architecture, technology stack, and component-wise explanation:</p><p><strong>High-Level Architecture Overview</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5AbazvsQVGzND-Fv" /></figure><h3><strong>Suggested Tech Stack:</strong></h3><p><strong>Frontend</strong></p><ul><li><strong>Framework:</strong> React.js / Blazor (for .NET ecosystem)</li><li><strong>Styling:</strong> Tailwind CSS or Bootstrap</li><li><strong>State Management:</strong> Redux / Context API</li><li><strong>Charting Libraries:</strong> Chart.js / Recharts / D3.js</li></ul><p><strong>Backend</strong></p><ul><li><strong>Framework:</strong> .NET Core (C#) or Python (FastAPI/Flask)</li><li><strong>AI/NLP:</strong> OpenAI GPT, Hugging Face Transformers</li><li><strong>Data Processing:</strong> Pandas (Python) / LINQ + DataTable (C#)</li><li><strong>Database:</strong> SQL Server / PostgreSQL / SQLite</li><li><strong>Authentication:</strong> IdentityServer4 (for .NET) / OAuth2 + JWT</li></ul><p><strong>Storage &amp; Sharing</strong></p><ul><li><strong>File Storage:</strong> Azure Blob Storage / Azure Data Lake Gan2</li><li><strong>Link Sharing:</strong> Generate UUID + short URLs</li><li><strong>Email Sending:</strong> SendGrid / SMTP / Amazon SES</li></ul><p><strong>Reporting &amp; Visualization</strong></p><ul><li><strong>Report Engine:</strong> Custom-built HTML-to-PDF (e.g. Puppeteer, wkhtmltopdf)</li><li><strong>Narrative Generation:</strong> GPT-4 or similar LLM</li></ul><h3><strong>Page-Wise Technical Breakdown:</strong></h3><p><strong>1. Home Page</strong></p><p><strong>Purpose:</strong> Starting point for the user<br> <strong>Features:</strong></p><ul><li>Buttons: “Create New Report”, “View Existing Reports”</li><li>User greeting / quick summary</li></ul><p><strong>Implementation:</strong></p><ul><li>React component with routing to respective pages</li><li>Pulls recent reports from database (via API)</li></ul><p><strong>2. Data Source Connection Page</strong></p><p><strong>Purpose:</strong> Upload or connect to data source<br> <strong>Features:</strong></p><ul><li>Upload CSV, Excel</li><li>Connect to database (MySQL, SQL Server, PostgreSQL)</li></ul><p><strong>Implementation:</strong></p><ul><li>File upload: HTML5 + drag &amp; drop UI → backend stores to Blob/S3</li><li>DB Connect: Form to enter DB credentials → backend tests connection and fetches schema</li><li>Use background tasks for large file ingestion</li></ul><p><strong>3. Data Cleaning Page</strong></p><p><strong>Purpose:</strong> Transform, clean, filter data<br> <strong>Features:</strong></p><ul><li>Column selection</li><li>Null handling, type conversion</li><li>Basic transformations (filters, sort, group-by)</li></ul><p><strong>Implementation:</strong></p><ul><li>React data grid (e.g. AG Grid) for data preview</li><li>Transform commands sent to backend → processed using Pandas/.NET LINQ</li><li>Option to export cleaned version to JSON/table</li></ul><p><strong>4. Report Type Selection Page</strong></p><p><strong>Purpose:</strong> Choose layout of the report<br> <strong>Features:</strong></p><ul><li>Choose single vs multi-page</li><li>Choose focus: KPI summary, trend analysis, comparison, etc.</li></ul><p><strong>Implementation:</strong></p><ul><li>React form with dropdowns and checkboxes</li><li>Pass preferences to backend → triggers AI prompt configuration</li></ul><p><strong>5. Report Viewer Page</strong></p><p><strong>Purpose:</strong> View, share, and edit AI-generated report<br> <strong>Features:</strong></p><ul><li>Interactive dashboard: charts, tables, narratives</li><li>Download as PDF</li><li>Share via link or email</li><li>Edit visuals or narrative content</li></ul><p><strong>Implementation:</strong></p><ul><li>Report rendered using charts and HTML narrative</li><li>Share: create UUID + short link stored in DB</li><li>Email: call backend email API with report attached</li><li>Edit: WYSIWYG editor for text and dynamic chart manipulation</li></ul><p>Please share your thoughts and ideas about this Application.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=506793d27e41" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DAX Cheat Sheet]]></title>
            <link>https://medium.com/@nadakkannu.smart/dax-cheat-sheet-4adb9d45a616?source=rss-d35209138c2e------2</link>
            <guid isPermaLink="false">https://medium.com/p/4adb9d45a616</guid>
            <category><![CDATA[power-bi]]></category>
            <category><![CDATA[dax]]></category>
            <dc:creator><![CDATA[Nadakkannu Kuthalaraja]]></dc:creator>
            <pubDate>Mon, 14 Apr 2025 17:10:40 GMT</pubDate>
            <atom:updated>2025-04-14T17:10:40.770Z</atom:updated>
            <content:encoded><![CDATA[<p>Power BI stands out as a top-tier business intelligence tool, renowned for its robust capabilities. Among its arsenal, DAX (Data Analysis Expressions) shines as a key feature, offering advanced formula expressions tailored for in-depth data analysis. DAX formulas encompass a range of functions, operators, and statements, making it a go-to resource for comprehensive data insights. This cheat sheet serves as a valuable aid for navigating the intricacies of DAX within the Power BI environment.</p><p><strong>Maths &amp; Statistical Functions</strong></p><ul><li>SUM(&lt;column&gt;) Adds all the numbers in a column.</li><li>SUMX(&lt;table&gt;, &lt;expression&gt;) Returns the sum of an expression evaluated for each row in a table.</li><li>AVERAGE(&lt;column&gt;) Returns the average (arithmetic mean) of all the numbers in a column.</li><li>AVERAGEX(&lt;table&gt;, &lt;expression&gt;) Calculates the average (arithmetic mean) of a set of expressions evaluated over a table.</li><li>MEDIAN(&lt;column&gt;) Returns the median of a column.</li><li>MEDIANX(&lt;table&gt;, &lt;expression&gt;) Calculates the median of a set of expressions evaluated over a table.</li><li>GEOMEAN(&lt;column&gt;) Calculates the geometric mean of a column.</li><li>GEOMEANX(&lt;table&gt;, &lt;expression&gt;) Calculates the geometric mean of a set of expressions evaluated over a table.</li><li>COUNT(&lt;column&gt;) Returns the number of cells in a column that contains non-blank values.</li><li>COUNTX(&lt;table&gt;, &lt;expression&gt;) Counts the number of rows from an expression that evaluates to a non-blank value.</li><li>DIVIDE(&lt;numerator&gt;, &lt;denominator&gt; [,&lt;alternateresult&gt;]) Performs division and returns alternate result or BLANK() on division by 0.</li><li>MIN(&lt;column&gt;) Returns a minimum value of a column.</li><li>MAX(&lt;column&gt;) Returns a maximum value of a column.</li><li>COUNTROWS([&lt;table&gt;]) Counts the number of rows in a table.</li><li>DISTINCTCOUNT(&lt;column&gt;) Counts the number of distinct values in a column.</li><li>RANKX(&lt;table&gt;, &lt;expression&gt;[, &lt;value&gt;[, &lt;order&gt;[, &lt;ties&gt;]]]) Returns the ranking of a number in a list of numbers for each row in the table argument.</li></ul><p><strong>Filter Functions</strong></p><ul><li>FILTER(&lt;table&gt;, &lt;filter&gt;) Returns a table that is a subset of another table or expression.</li><li>CALCULATE(&lt;expression&gt;[, &lt;filter1&gt; [, &lt;filter2&gt; [, …]]]) Evaluates an expression in a filter context.</li><li>HASONEVALUE(&lt;columnName&gt;) Returns TRUE when the context for columnName has been filtered down to one distinct value only. Otherwise, it is FALSE.</li><li>ALLNOBLANKROW(&lt;table&gt; | &lt;column&gt;[, &lt;column&gt;[, &lt;column&gt;[,…]]]) Returns a table that is a subset of another table or expression.</li><li>ALL([&lt;table&gt; | &lt;column&gt;[, &lt;column&gt;[, &lt;column&gt;[,…]]]]) Returns all the rows in a table, or all the values in a column, ignoring any filters that might have been applied.</li><li>ALLEXCEPT(&lt;table&gt;, &lt;column&gt;[, &lt;column&gt;[,..]]) Returns all the rows in a table except for those rows that are affected by the specified column filters.</li><li>REMOVEFILTERS([&lt;table&gt; | &lt;column&gt;][, &lt;column&gt;[, &lt;column&gt;[,…]]]]) Clear all filters from designated tables or columns.</li></ul><p><strong>Logical Functions</strong></p><ul><li>IF(&lt;logical_test&gt;, &lt;value_if_true&gt;[, &lt;value_if_false&gt;]) Checks a condition, and returns a certain value depending on whether it is true or false.</li><li>AND(&lt;logical 1&gt;, &lt;logical 2&gt;) Checks whether both arguments are TRUE, and returns TRUE if both arguments are TRUE. Otherwise, it returns FALSE.</li><li>OR(&lt;logical 1&gt;, &lt;logical 2&gt;) Checks whether one of the arguments is TRUE to return TRUE. The function returns FALSE if both arguments are FALSE.</li><li>NOT(&lt;logical&gt;) Changes TRUE to FALSE and vice versa.</li><li>SWITCH(&lt;expression&gt;, &lt;value&gt;, &lt;result&gt;[, &lt;value&gt;, &lt;result&gt;]…[, &lt;else&gt;]) Evaluates an expression against a list of values and returns one of possible results</li><li>IFERROR(&lt;value&gt;, &lt;value_if_error&gt;) Returns value_if_error if the first expression is an error and the value of the expression itself otherwise.</li></ul><p><strong>Date &amp; Time Functions</strong></p><ul><li>CALENDAR(&lt;start_date&gt;, &lt;end_date&gt;) Returns a table with a single column named “Date” that contains a contiguous set of dates.</li><li>DATE(&lt;year&gt;, &lt;month&gt;, &lt;day&gt;) Returns the specified date in datetime format.</li><li>DATEDIFF(&lt;date_1&gt;, &lt;date_2&gt;, &lt;interval&gt;) Returns the number of units between two dates as defined in &lt;interval&gt;.</li><li>DATEVALUE(&lt;date_text&gt;) Converts a date in text to a date in datetime format.</li><li>DAY(&lt;date&gt;) Returns a number from 1 to 31 representing the day of the month.</li><li>WEEKNUM(&lt;date&gt;) Returns weeknumber in the year.</li><li>MONTH(&lt;date&gt;) Returns a number from 1 to 12 representing a month.</li><li>QUARTER(&lt;date&gt;) Returns a number from 1 to 4 representing a quarter.</li></ul><p><strong>Time Intelligence Functions</strong></p><ul><li>DATEADD(&lt;dates&gt;, &lt;number_of_intervals&gt;, &lt;interval&gt;) Moves a date by a specific interval.</li><li>DATESBETWEEN(&lt;dates&gt;, &lt;date_1&gt;, &lt;date_2&gt;) Returns the dates between specified dates.</li><li>TOTALYTD(&lt;expression&gt;, &lt;dates&gt;[, &lt;filter&gt;][, &lt;year_end_date&gt;]) Evaluates the year-to-date value of the expression in the current context.</li><li>SAMEPERIODLASTYEAR(&lt;dates&gt;) Returns a table that contains a column of dates shifted one year back in time.</li><li>STARTOFMONTH(&lt;dates&gt;) // ENDOFMONTH(&lt;dates&gt;) Returns the start // end of the month.</li><li>STARTOFQUARTER(&lt;dates&gt;) // ENDOFQUARTER(&lt;dates&gt;) Returns the start // end of the quarter.</li><li>STARTOFYEAR(&lt;dates&gt;) // ENDOFYEAR(&lt;dates&gt;) Returns the start // end of the quarter.</li></ul><p><strong>Relationship Functions</strong></p><ul><li>CROSSFILTER(&lt;left_column&gt;, &lt;right_column&gt;, &lt;crossfiltertype&gt;) Specifies the cross-filtering direction to be used in a calculation.</li><li>RELATED(&lt;column&gt;) Returns a related value from another table.</li></ul><p><strong>Table Manipulation Functions</strong></p><ul><li>SUMMARIZE(&lt;table&gt;, &lt;groupBy_columnName&gt;[, &lt;groupBy_columnName&gt;]…[, &lt;name&gt;, &lt;expression&gt;]…) Returns a summary table for the requested totals over a set of groups.</li><li>DISTINCT(&lt;table&gt;) Returns a table by removing duplicate rows from another table or expression.</li><li>ADDCOLUMNS(&lt;table&gt;, &lt;name&gt;, &lt;expression&gt;[, &lt;name&gt;, &lt;expression&gt;]…) Adds calculated columns to the given table or table expression.</li><li>SELECTCOLUMNS(&lt;table&gt;, &lt;name&gt;, &lt;expression&gt;[, &lt;name&gt;, &lt;expression&gt;]…) Selects calculated columns from the given table or table expression.</li><li>GROUPBY(&lt;table&gt; [, &lt;groupBy_columnName&gt;[, [&lt;column_name&gt;] [&lt;expression&gt;]]…) Create a summary of the input table grouped by specific columns.</li><li>INTERSECT(&lt;left_table&gt;, &lt;right_table&gt;) Returns the rows of the left-side table that appear in the right-side table.</li><li>NATURALINNERJOIN(&lt;left_table&gt;, &lt;right_table&gt;) Joins two tables using an inner join.</li><li>NATURALLEFTOUTERJOIN(&lt;left_table&gt;, &lt;right_table&gt;) Joins two tables using a left outer join.</li><li>UNION(&lt;table&gt;, &lt;table&gt;[, &lt;table&gt; [,…]]) Returns the union of tables with matching columns.</li></ul><p><strong>Text Functions</strong></p><ul><li>EXACT(&lt;text_1&gt;, &lt;text_2&gt;) Checks if two strings are identical (EXACT() is case sensitive).</li><li>FIND(&lt;text_tofind&gt;, &lt;in_text&gt;) Returns the starting position a text within another text (FIND() is case sensitive).</li><li>FORMAT(&lt;value&gt;, &lt;format&gt;) Converts a value to a text in the specified number format.</li><li>LEFT(&lt;text&gt;, &lt;num_chars&gt;) Returns the number of characters from the start of a string.</li><li>RIGHT(&lt;text&gt;, &lt;num_chars&gt;) Returns the number of characters from the end of a string.</li><li>LEN(&lt;text&gt;) Returns the number of characters in a string of text.</li><li>LOWER(&lt;text&gt;) Converts all letters in a string to lowercase.</li><li>UPPER(&lt;text&gt;) Converts all letters in a string to uppercase.</li><li>TRIM(&lt;text&gt;) Remove all spaces from a text string.</li><li>CONCATENATE(&lt;text_1&gt;, &lt;text_2&gt;) Joins two strings together into one string.</li><li>SUBSTITUTE(&lt;text&gt;, &lt;old_text&gt;, &lt;new_text&gt;, &lt;instance_num&gt;) Replaces existing text with new text in a string.</li><li>REPLACE(&lt;old_text&gt;, &lt;start_posotion&gt;, &lt;num_chars&gt;, &lt;new_text&gt;) Replaces part of a string with a new string.</li></ul><p><strong>Information Functions</strong></p><ul><li>COLUMNSTATISTICS() Returns statistics regarding every column in every table. This function has no arguments.</li><li>NAMEOF(&lt;value&gt;) Returns the column or measure name of a value.</li><li>ISBLANK(&lt;value&gt;) // ISERROR(&lt;value&gt;) Returns whether the value is blank // an error.</li><li>ISLOGICAL(&lt;value&gt;) Checks whether a value is logical or not.</li><li>ISNUMBER(&lt;value&gt;) Checks whether a value is a number or not.</li><li>ISFILTERED(&lt;table&gt; | &lt;column&gt;) Returns true when there are direct filters on a column.</li><li>ISCROSSFILTERED(&lt;table&gt; | &lt;column&gt;) Returns true when there are crossfilters on a column.</li><li>USERPRINCIPALNAME() Returns the user principal name or email address. This function has no arguments.</li></ul><p><strong>DAX Statements</strong></p><ul><li>VAR(&lt;name&gt; = &lt;expression&gt;) Stores the result of an expression as a named variable. To return the variable, use RETURN after the variable is defined.</li><li>COLUMN(&lt;table&gt;[&lt;column&gt;] = &lt;expression&gt;) Stores the result of an expression as a column in a table.</li><li>ORDER BY(&lt;table&gt;[&lt;column&gt;]) Defines the sort order of a column. Every column can be sorted in ascending (ASC) or descending (DESC) way.</li></ul><p>What are your thoughts about this approach and let us know in the comments ?</p><p>Happy Learning !!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4adb9d45a616" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>