<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Durga Prasad Patsa on Medium]]></title>
        <description><![CDATA[Stories by Durga Prasad Patsa on Medium]]></description>
        <link>https://medium.com/@prasad.patsa?source=rss-b740d4e2a5b7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*iBAealYCMapZ_rUqpRhZww.jpeg</url>
            <title>Stories by Durga Prasad Patsa on Medium</title>
            <link>https://medium.com/@prasad.patsa?source=rss-b740d4e2a5b7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 17:13:33 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@prasad.patsa/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How an AI Architect Builds a Self-Aware Excavator: NVIDIA’s Physical AI Stack, Explained Simply]]></title>
            <link>https://medium.com/@prasad.patsa/how-an-ai-architect-builds-a-self-aware-excavator-nvidias-physical-ai-stack-explained-simply-f71eff761b22?source=rss-b740d4e2a5b7------2</link>
            <guid isPermaLink="false">https://medium.com/p/f71eff761b22</guid>
            <dc:creator><![CDATA[Durga Prasad Patsa]]></dc:creator>
            <pubDate>Fri, 02 Oct 2026 17:07:36 GMT</pubDate>
            <atom:updated>2026-10-02T17:07:36.164Z</atom:updated>
            <content:encoded><![CDATA[<p>From a dusty mine site to a Jetson module in the cab: how perception, edge computing, and model deployment fit together.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Hd0dY3y86gYCsCz7DL5ZnQ.png" /></figure><h3>Why physical AI is different</h3><p>ChatGPT lives in a data center. If it’s slow, you wait an extra second. If it’s wrong, you get a bad paragraph.</p><p><strong>Physical AI</strong> lives in the real world. It has a body, sensors, and actuators. If an excavator’s AI is slow or wrong, a 40-ton machine swings a bucket into a person.</p><p>That one difference drives every architecture decision:</p><ul><li><strong>Latency matters.</strong> Decisions must happen in milliseconds, not seconds.</li><li><strong>Connectivity can’t be assumed.</strong> Mines are remote. The internet is spotty or absent.</li><li><strong>Compute is limited.</strong> The brain has to fit in a box that handles vibration, dust, heat, and a 24V power supply.</li><li><strong>Safety is non-negotiable.</strong> “Usually works” isn’t good enough.</li></ul><p>So the AI can’t live in the cloud. It has to live <strong>on the machine</strong>. That’s <strong>edge computing</strong>, and it’s where NVIDIA’s Jetson platform comes in.</p><p>Let’s build an autonomous excavator step by step, the way an AI architect would.</p><h3>The mission</h3><p>Our excavator needs to:</p><ol><li><strong>See</strong> its surroundings: terrain, dirt piles, haul trucks, people, other machines.</li><li><strong>Understand</strong> what it sees: “that’s a truck, that’s a person 12 meters away, that’s the dig face.”</li><li><strong>Decide</strong> what to do: dig here, swing there, stop now.</li><li><strong>Act</strong> safely through hydraulic controls.</li></ol><p>Think of it as <strong>Sense → Think → Act</strong>, running in a loop many times per second. Everything below serves that loop.</p><h3>Step 1: Perception, giving the machine eyes</h3><p>Humans use eyes, ears, and balance. An excavator uses a <strong>sensor suite</strong>:</p><ul><li><strong>Cameras</strong> capture rich color and texture, and are good for recognizing <em>what</em> something is.</li><li><strong>LiDAR</strong> shoots laser pulses to build a 3D point cloud, which is great for <em>where</em> things are and how far away.</li><li><strong>Radar</strong> sees through dust, rain, and fog, which matters a lot at a mine.</li><li><strong>IMU/GNSS</strong> tell the machine how it’s moving and where it is on Earth.</li></ul><p>No single sensor is perfect. Cameras struggle in dust and glare. LiDAR gets confused by heavy particulates. Radar is low-resolution. So architects use <strong>sensor fusion</strong>, combining them so each one covers the others’ weaknesses.</p><h4>The perception techniques</h4><p><strong>Object detection</strong> answers “what’s in the image, and where?” A neural network draws boxes around people, trucks, and rocks. This is the safety backbone: <em>detect a person near the swing radius, stop immediately.</em></p><p><strong>Semantic segmentation</strong> labels every pixel: “this is loose soil, this is rock, this is a haul road, this is sky.” For an excavator, this tells it where it’s safe to dig and where it’s safe to drive.</p><p><strong>Depth estimation</strong> works out how far away things are. Stereo cameras do this like human eyes: two views, and the difference between them gives depth. NVIDIA’s Isaac ROS includes GPU-accelerated stereo depth models for this.</p><p><strong>3D perception</strong> takes LiDAR point clouds and builds a live map of the terrain, including the shape of the dirt pile, the depth of the trench, and the position of the truck bed you’re about to load.</p><p><strong>SLAM (Simultaneous Localization and Mapping)</strong> answers “where am I, and what does the world around me look like?” when GPS is unreliable, such as in a pit with high walls. NVIDIA provides GPU-accelerated visual SLAM (cuVSLAM) and 3D reconstruction (nvblox) through Isaac ROS.</p><p><strong>Tracking</strong> follows objects over time. A person isn’t just “detected” in every frame; the system knows it’s the <em>same</em> person walking toward the machine and can predict where they’ll be in two seconds.</p><p>Together, this creates what robotics people call a <strong>world model</strong>: a live, continuously updated understanding of the machine’s surroundings.</p><h3>Step 2: Edge computing, putting the brain in the cab</h3><p>All of this perception work is computationally heavy: multiple camera streams, LiDAR, several neural networks, all running at 30 or more frames per second.</p><p>You can’t stream it to the cloud. Sending video up, running inference, and getting an answer back adds latency you can’t afford, and the mine may have no connectivity at all.</p><p>So we use an <strong>edge computer</strong>: a small, rugged, GPU-powered computer mounted on the excavator.</p><h4>Enter NVIDIA Jetson</h4><p>Jetson is NVIDIA’s family of compact “AI computers on a module.” Think of it as a data center GPU shrunk to fit in your hand and sip power.</p><ul><li><strong>Jetson Orin</strong> (AGX, NX, Nano variants) is the workhorse of today’s robotics. It balances performance and power, so you pick the size that matches your workload.</li><li><strong>Jetson Thor</strong> is the newer generation, built for humanoids and heavy robots, with enough horsepower to run large transformer and generative models on the machine itself.</li></ul><p>On an excavator, a Jetson module would typically sit in a ruggedized enclosure inside or near the cab, connected to cameras, LiDAR, and the machine’s control network.</p><h4>What’s inside that “AI computer”?</h4><p>Jetson isn’t just a GPU. It has specialized blocks working together:</p><ul><li><strong>GPU</strong> runs neural networks (the heavy lifting).</li><li><strong>CPU</strong> runs the control logic, ROS 2, and general software.</li><li><strong>Deep Learning Accelerators (DLA)</strong> are low-power engines that offload some neural networks, freeing the GPU.</li><li><strong>Video encoders/decoders and vision accelerators</strong> handle camera streams efficiently.</li></ul><p>An architect’s job is to decide <em>which workload runs where</em>. For example, the person-detection model might run on the DLA while the GPU handles terrain segmentation. The goal is to get the most out of a fixed power and thermal budget.</p><h3>Step 3: Teaching the models, where the data comes from</h3><p>Here’s the part people underestimate: <strong>a model is only as good as its training data.</strong></p><p>A generic object detector trained on city photos has probably never seen a mine at dawn with dust in the air, a half-buried tire, or a worker in a high-vis vest standing behind an articulated truck.</p><p>You need <em>mining</em> data. But collecting it has problems:</p><ul><li>Dangerous situations (a person too close to a machine) are rare, and you don’t want to stage them.</li><li>Labeling thousands of images by hand is slow and expensive.</li><li>You can’t easily capture every weather, light, and dust condition.</li></ul><h4>Synthetic data to the rescue: Isaac Sim and Omniverse</h4><p>NVIDIA’s answer is to <strong>build the mine in a simulator first.</strong></p><p><strong>Omniverse</strong> is NVIDIA’s platform for building physically accurate 3D worlds, and <strong>Isaac Sim</strong> is the robotics simulator built on it. An architect can:</p><ol><li>Create a digital twin of the mine site, with terrain, trucks, equipment, and workers.</li><li>Add a virtual excavator with virtual cameras and LiDAR that behave like the real ones.</li><li>Use <strong>Replicator</strong> to generate thousands of labeled images automatically, varying lighting, dust, weather, camera angles, and object positions. This is called <strong>domain randomization</strong>.</li></ol><p>The simulator already knows where every object is, so labels are <em>free and perfect</em>. You can also simulate dangerous scenarios as many times as you want, with zero risk.</p><p>NVIDIA’s <strong>Cosmos</strong> world foundation models take this further by generating realistic video of varied scenarios, helping fill gaps that simulation alone doesn’t cover.</p><p>The real-world data you do collect (from cameras on actual machines) gets mixed in with the synthetic data. Typically you pretrain on lots of synthetic data, then fine-tune on a smaller amount of real data.</p><h3>Step 4: Tuning the models</h3><p>You rarely train from scratch. Architects start with a <strong>pretrained model</strong> and adapt it, which is called <strong>fine-tuning</strong> or <strong>transfer learning</strong>.</p><p>NVIDIA’s <strong>TAO Toolkit</strong> streamlines this. The workflow:</p><ol><li><strong>Start from a pretrained model</strong> (for example, a detector that already understands people and vehicles).</li><li><strong>Fine-tune on mining data</strong> (real plus synthetic) so it learns mining-specific things like excavators, dump trucks, and PPE.</li><li><strong>Evaluate</strong>, looking especially at failure cases like dusty conditions, partial occlusion, and night operation.</li><li><strong>Iterate</strong>, adding more data where the model is weak.</li></ol><p>The best part is that fine-tuning needs far less data and compute than training from scratch.</p><h4>Making the model fit the edge</h4><p>A model that works great on a data center GPU might be too big or slow for a Jetson. Before deployment, you <strong>optimize</strong> it:</p><ul><li><strong>Quantization</strong> reduces numerical precision (for example, from 32-bit to 8-bit numbers). The model gets dramatically smaller and faster, usually with little accuracy loss. Think of it as rounding 3.14159265 to 3.14: you keep what matters and save a lot of space.</li><li><strong>Pruning</strong> removes parts of the network that contribute little, like trimming a hedge.</li><li><strong>Distillation</strong> trains a small “student” model to mimic a large “teacher” model.</li></ul><p>The key tool here is <strong>TensorRT</strong>, NVIDIA’s inference optimizer. You hand it a trained model, and it:</p><ul><li>Fuses operations together to cut overhead</li><li>Picks the fastest GPU kernels for the specific hardware</li><li>Applies quantization (FP16, INT8)</li><li>Produces a highly optimized “engine” for that exact Jetson device</li></ul><p>The result is often several times faster than running the raw model, which is the difference between 8 frames per second (too slow to be safe) and 40 (comfortable).</p><h3>Step 5: Deployment, running it on the machine</h3><p>Now we have optimized models. How do they run as a real-time system?</p><h4>The software stack on the Jetson</h4><p>Think of the edge software as layers:</p><ul><li><strong>JetPack SDK</strong> is the foundation: operating system, drivers, CUDA, TensorRT, and the libraries everything else builds on.</li><li><strong>ROS 2 (Robot Operating System)</strong> is the standard “nervous system” for robots. It lets separate software modules (camera driver, detector, planner, controller) talk to each other by passing messages.</li><li><strong>Isaac ROS</strong> is a set of GPU-accelerated ROS 2 packages from NVIDIA for detection, depth, SLAM, mapping, and more. With a feature called NITROS, data moves between modules without being copied back and forth, which saves a lot of time.</li><li><strong>DeepStream</strong> is a toolkit for efficiently processing multiple video streams, including decoding, running inference, and tracking, all inside one optimized pipeline. It’s well suited to multi-camera setups.</li><li><strong>Docker containers</strong> package each module with its dependencies, so what you tested is exactly what runs on the machine.</li></ul><h4>What the real-time loop looks like</h4><p>Every fraction of a second on our excavator:</p><pre>Cameras + LiDAR + Radar<br>        ↓<br>  Sensor drivers (ROS 2)<br>        ↓<br>  Perception: detect, segment, depth, track   ← TensorRT-optimized models<br>        ↓<br>  Fusion + World model (map of surroundings)<br>        ↓<br>  Planning: where to dig, how to swing, what&#39;s safe<br>        ↓<br>  Safety layer: person in zone? → STOP<br>        ↓<br>  Controller → Hydraulic actuators</pre><p>A very important architectural point: <strong>the safety layer is separate and deterministic.</strong> Machine learning models are probabilistic, which means they can be wrong. So the final “stop if a person is in the danger zone” logic is typically built with certified, rule-based safety systems, and the AI feeds into it rather than replacing it. AI proposes; the safety layer disposes.</p><h3>Step 6: The loop that never ends</h3><p>Deployment isn’t the finish line, it’s the starting line. Mines change daily. New equipment arrives. Seasons change. Models drift.</p><p>So architects design a <strong>continuous improvement loop</strong>:</p><ol><li><strong>Deploy</strong> the optimized model to the fleet (using containers and over-the-air updates).</li><li><strong>Monitor</strong> performance on real machines.</li><li><strong>Collect hard cases</strong>, such as moments of low confidence, operator overrides, and near-misses.</li><li><strong>Send the data back</strong> (when connectivity allows, often when the machine returns to a depot).</li><li><strong>Retrain and fine-tune</strong> in the cloud or data center on powerful GPUs.</li><li><strong>Validate in simulation</strong> against thousands of scenarios before touching a real machine.</li><li><strong>Redeploy</strong>, and repeat.</li></ol><p>This split is the heart of modern physical AI architecture: <strong>train in the cloud, validate in simulation, run at the edge.</strong> Heavy learning happens where compute is abundant; fast, safe decisions happen where the machine actually operates.</p><h3>The whole picture in one view</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/705/1*LKDX1hpg0xjPCm_etXIaoA.png" /></figure><h3>Takeaways for first-time readers</h3><p>If you remember only four things about physical AI and edge computing:</p><ol><li><strong>The AI must live on the machine.</strong> Latency, connectivity, and safety make the cloud a poor place for real-time decisions.</li><li><strong>Perception is the foundation.</strong> Cameras, LiDAR, and radar are fused into a live model of the world.</li><li><strong>Simulation is a superpower.</strong> You can generate perfect, labeled, risk-free training data for situations that are rare or dangerous in real life.</li><li><strong>Training and running are different jobs.</strong> You train big in the cloud, then compress and optimize (TensorRT) to run small and fast at the edge (Jetson).</li></ol><p>The excavator is just one example. Swap the bucket for a gripper and you have a warehouse robot. Swap the hydraulics for legs and you have a humanoid. The same Sense → Think → Act loop and the same simulate → train → optimize → deploy pipeline apply across all of physical AI.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f71eff761b22" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Architecting a Metadata-Driven Migration Framework: Moving 3,000 Oracle Tables to Azure Delta Lake]]></title>
            <link>https://medium.com/@prasad.patsa/architecting-a-metadata-driven-migration-framework-moving-3-000-oracle-tables-to-azure-delta-lake-54e6e0f00587?source=rss-b740d4e2a5b7------2</link>
            <guid isPermaLink="false">https://medium.com/p/54e6e0f00587</guid>
            <dc:creator><![CDATA[Durga Prasad Patsa]]></dc:creator>
            <pubDate>Mon, 17 Aug 2026 17:55:51 GMT</pubDate>
            <atom:updated>2026-08-17T17:55:51.865Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UKj56mkLlNi73oca9yo0og.png" /></figure><p>Migrating a handful of database tables to the cloud is a straightforward task. Migrating 3,000 operational Oracle tables, however, transforms a routine data-engineering exercise into an enterprise platform-architecture challenge.</p><p>When facing a modernization program of this scale, the naive approach is to build individual Azure Data Factory (ADF) pipelines or Copy activities for every table. This strategy quickly breaks down — maintaining thousands of distinct pipelines introduces insurmountable operational overhead whenever a schema updates, a retry policy shifts, or logging requirements evolve.</p><p>To modernize an enterprise data platform reliably, you need a <strong>metadata-driven data migration framework</strong>. By decoupling <em>how</em> data is processed from <em>what</em> data is processed, you can manage thousands of source tables using a small footprint of fully parameterized, reusable pipelines.</p><h3>End-to-End Migration Architecture</h3><p>The framework extracts operational data from on-premises Oracle databases, lands raw files in Azure Data Lake Storage Gen2 (ADLS Gen2), and transforms them into Delta Lake tables within Azure Databricks using a medallion architecture design.</p><pre>ORACLE SOURCE<br>                          │<br>            ┌─────────────┴─────────────┐<br>            │                           │<br>   Data Dictionary               Operational Tables<br>            │                           │<br>            ▼                           │<br>    Discovery Engine                    │<br>            │                           │<br>            ▼                           │<br>   Metadata/Control DB                  │<br>            │                           │<br>            └─────────────┬─────────────┘<br>                          │<br>                          ▼<br>                 ADF Master Pipeline<br>                          │<br>             ┌────────────┴────────────┐<br>             │  (Wave &amp; Group Logic)   │<br>             ▼                         ▼<br>      Child Pipeline             Child Pipeline<br>     (Full Ingestion)         (Incremental/CDC)<br>             │                         │<br>             └────────────┬────────────┘<br>                          │<br>                          ▼<br>                   ADLS Gen2 Landing<br>                    (Raw / Bronze)<br>                          │<br>                          ▼<br>                  Azure Databricks<br>                          │<br>                          ▼<br>                  Delta Lake Target<br>                   (Silver / Gold)<br>                          │<br>            ┌─────────────┴─────────────┐<br>            ▼                           ▼<br>     Audit Framework             Reconciliation</pre><h3>Core Architectural Components</h3><h4>1. Source Discovery and Metadata Control Layer</h4><p>Manually configuring metadata for 3,000 tables is inefficient and error-prone. Instead, an automated discovery script queries Oracle’s data dictionary (ALL_TABLES, ALL_TAB_COLUMNS, ALL_CONSTRAINTS) to extract structural details—such as column data types, primary keys, and table sizes—and populates a centralized <strong>Control Database</strong> (e.g., Azure SQL Database).</p><p>The Control Database acts as the orchestrator’s brain, governing execution behavior without requiring pipeline code changes:</p><p><strong>Field NameDescriptionExample</strong>source_schemaSource schema identifierSALESsource_tableSource table identifierORDERSload_typeIngestion strategy (FULL, INCREMENTAL, CDC)INCREMENTALwatermark_columnHigh-watermark tracking field for incremental readsMODIFIED_DATElast_watermarkStores the timestamp/ID of the last successful extraction2026-08-15 10:00:00migration_waveLogical grouping for phased cutover3batch_groupParallelism bucket based on table sizeLARGE_TABLES_GROUPenabled_flagOperational toggle (Y/N)Y</p><h4>2. Reusable Parameterized Orchestration</h4><p>Instead of creating thousands of discrete pipelines, the ADF orchestration model relies on a two-tier structure:</p><ul><li><strong>Master Pipeline (</strong><strong>PL_ORACLE_MIGRATION_MASTER):</strong> Queries the Control Database for active tables assigned to a specific migration wave, handles batching and concurrency limits, and iterates through records using a ForEach activity.</li><li><strong>Generic Child Pipeline (</strong><strong>PL_ORACLE_TABLE_LOAD):</strong> Contains no hardcoded values. It accepts dynamic parameters (p_SourceSchema, p_SourceTable, p_LoadType, p_LastWatermark) and constructs source extraction SQL queries dynamically at runtime.</li></ul><p>SQL</p><pre>-- Dynamic SQL query generated inside the Child Pipeline for incremental loads<br>SELECT * <br>FROM @{pipeline().parameters.p_SourceSchema}.@{pipeline().parameters.p_SourceTable}<br>WHERE @{pipeline().parameters.p_WatermarkColumn} &gt; &#39;@{pipeline().parameters.p_LastWatermark}&#39;</pre><h4>3. Landing Layer &amp; Delta Lake Conversion</h4><p>To keep data ingestion decoupled from heavy analytical processing, data is first extracted from Oracle and landed directly into ADLS Gen2 in raw Parquet or CSV format using a structured directory hierarchy:</p><p>abfss://bronze/oracle/sales/orders/year=2026/month=08/day=17/</p><p>After landing in the Raw/Bronze layer, Azure Databricks orchestrates the conversion into Delta Lake. For incremental loads, Databricks reads the landed staging files and executes a Delta MERGE operation to update target records in place while maintaining ACID compliance:</p><p>SQL</p><pre>MERGE INTO silver.orders AS target<br>USING bronze.orders_updates AS source<br>ON target.order_id = source.order_id<br>WHEN MATCHED THEN UPDATE SET *<br>WHEN NOT MATCHED THEN INSERT *;</pre><h3>Enterprise Operational Guardrails</h3><p>Extracting 3,000 tables simultaneously will exhaust network bandwidth, saturate database connections, and overwhelm Integration Runtimes. Enterprise migration frameworks require robust governance controls embedded directly into the pipeline logic:</p><ul><li><strong>Controlled Parallelism &amp; Classification:</strong> Tables are categorized by volume (SMALL, MEDIUM, LARGE, VERY_LARGE). The framework sets strict concurrency limits per category to protect source systems.</li><li><strong>Dependency Management:</strong> Parent-child foreign key mappings are stored in a dependency metadata table to ensure dimension tables populate before facts (e.g., CUSTOMER finishes before ORDERS begins).</li><li><strong>Automated Audit Logging:</strong> Every pipeline run writes start times, end times, extracted row counts, written row counts, and status codes to an Audit_Log table for centralized operational dashboards.</li><li><strong>Idempotency &amp; Restartability:</strong> If a execution run fails midway, the framework evaluates the audit log on retry and re-executes <strong>only</strong> tables marked as FAILED or NOT_STARTED, picking up precisely where it stopped.</li></ul><h3>Automated Data Reconciliation Framework</h3><p>A cloud migration is only complete when data completeness and accuracy are proven. The framework incorporates an automated validation engine that runs immediately after the landing step:</p><pre>Reconciliation Engine<br>                         │<br>        ┌────────────────┴────────────────┐<br>        ▼                                 ▼<br>Oracle Source Metrics            Delta Lake Target Metrics<br> (Row Count, Sum)                 (Row Count, Sum)<br>        │                                 │<br>        └────────────────┬────────────────┘<br>                         │<br>                         ▼<br>             Compare &amp; Log Results<br>                         │<br>           ┌─────────────┴─────────────┐<br>           ▼                           ▼<br>     MATCH: PASS                 MISMATCH: FAIL<br> (Proceed to Cutover)        (Block Cutover &amp; Alert)</pre><ul><li><strong>Row-Count Comparison:</strong> Validates that total records extracted from Oracle match the row count landed in Delta Lake.</li><li><strong>Aggregates &amp; Hashes:</strong> Performs checksum checks on key columns (e.g., SUM(order_amount), MAX(modified_date)) to catch subtle data truncation or field mapping issues.</li><li><strong>Status Gatekeeping:</strong> Discrepancies automatically flag the table status as FAILED_VALIDATION in the Control DB, stopping downstream silver/gold processing and alerting operations teams.</li></ul><h3>Architectural Takeaway</h3><p>The core principle of enterprise data engineering is <strong>separating execution logic from metadata parameters</strong>.</p><p>Creating thousands of hardcoded ETL jobs creates severe operational bottlenecks. By building a metadata-driven foundation using Oracle, Azure Data Factory, ADLS Gen2, and Azure Databricks, onboarding new database schemas becomes a simple metadata configuration step — delivering a scalable, auditable, and repeatable cloud modernization framework.</p><p>By <em>Durga Prasad Patsa</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=54e6e0f00587" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Running 650 Kafka Connectors on Kubernetes: Architecting a Self-Healing Streaming Platform with…]]></title>
            <link>https://medium.com/@prasad.patsa/running-650-kafka-connectors-on-kubernetes-architecting-a-self-healing-streaming-platform-with-36f5a738b7c0?source=rss-b740d4e2a5b7------2</link>
            <guid isPermaLink="false">https://medium.com/p/36f5a738b7c0</guid>
            <dc:creator><![CDATA[Durga Prasad Patsa]]></dc:creator>
            <pubDate>Thu, 13 Aug 2026 19:04:06 GMT</pubDate>
            <atom:updated>2026-08-13T19:06:30.930Z</atom:updated>
            <content:encoded><![CDATA[<h3>Running 650 Kafka Connectors on Kubernetes: Architecting a Self-Healing Streaming Platform with Rancher and Strimzi</h3><p><em>How we moved from a fragile, hand-managed Kafka cluster to a Kubernetes-native, self-healing streaming platform that ingests real-time data from hundreds of field endpoints — and the API layer that made it usable for the rest of the business.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*jYeLGBaqfza3k8i_bj8E5A.png" /></figure><h3>The problem: streaming infrastructure that couldn’t keep up</h3><p>When you’re ingesting real-time operational data — dispatch events, logistics telemetry, sensor readings — from hundreds of distributed sources, the Kafka cluster itself stops being the hard part. The hard part is everything around it: connector sprawl, broker upgrades that require a maintenance window, and no consistent way to add a new data source without a ticket to the platform team.</p><p>That was the situation I inherited on a recent North America logistics and dispatch project. Field and operational systems needed to stream continuously into a lakehouse for downstream AI-driven insight, but the Kafka layer was managed by hand — brokers on VMs, connectors configured manually, and no self-healing when a node dropped. It worked, until it didn’t, and every “didn’t” was a 2 a.m. page.</p><p>The mandate was clear: rebuild the streaming backbone as Kubernetes-native infrastructure that could scale to hundreds of source connectors, recover from failure without intervention, and give application teams a clean way to publish events without touching Kafka internals directly.</p><h3>Why Kubernetes, Rancher, and Strimzi</h3><p>Kafka on Kubernetes is a well-trodden path, but the tooling choices matter a lot in practice.</p><p><strong>Rancher</strong> became our cluster management layer because it gave us a single control plane across environments — dev, staging, and production clusters — with consistent RBAC, monitoring, and node-pool management. For a platform team supporting multiple business units, that consistency matters more than any single feature: onboarding a new environment stopped being a bespoke exercise.</p><p><strong>Strimzi</strong>, the Kafka operator, was the more consequential decision. Instead of managing Kafka as a set of StatefulSets and configuration files by hand, Strimzi lets you declare the entire Kafka topology — brokers, topics, users, and connectors — as Kubernetes custom resources:</p><p>yaml</p><pre>apiVersion: kafka.strimzi.io/v1beta2<br>kind: Kafka<br>metadata:<br>  name: dispatch-streaming-cluster<br>spec:<br>  kafka:<br>    replicas: 3<br>    listeners:<br>      - name: tls<br>        port: 9093<br>        type: internal<br>        tls: true<br>    storage:<br>      type: jbod<br>      volumes:<br>        - id: 0<br>          type: persistent-claim<br>          size: 500Gi<br>          deleteClaim: false<br>    config:<br>      default.replication.factor: 3<br>      min.insync.replicas: 2<br>      auto.create.topics.enable: false<br>  zookeeper:<br>    replicas: 3<br>    storage:<br>      type: persistent-claim<br>      size: 100Gi</pre><p>That declarative model is what unlocked the self-healing behavior we needed. If a broker pod dies, Kubernetes reschedules it and Strimzi reconciles cluster state automatically. Certificate rotation, rolling upgrades, and rack-awareness for availability-zone distribution all become configuration rather than runbooks.</p><h3>Scaling to 650 HTTP source connectors</h3><p>The real complexity wasn’t the Kafka cluster — it was the connector layer. We had hundreds of upstream systems that could only push data via HTTP, not native Kafka producers: legacy dispatch systems, third-party logistics APIs, field devices reporting through gateway services.</p><p>Strimzi’s KafkaConnect and KafkaConnector custom resources let us treat each of those 650 HTTP source connectors as its own declarative unit, deployed and scaled independently of the core brokers:</p><p>yaml</p><pre>apiVersion: kafka.strimzi.io/v1beta2<br>kind: KafkaConnector<br>metadata:<br>  name: dispatch-http-source-connector-042<br>  labels:<br>    strimzi.io/cluster: connect-cluster<br>spec:<br>  class: org.apache.kafka.connect.http.HttpSourceConnector<br>  tasksMax: 2<br>  config:<br>    topic: dispatch.events.raw<br>    http.api.url: https://gateway-042.internal/events<br>    http.request.method: GET<br>    poll.interval.ms: 5000<br>    http.response.data.format: JSON</pre><p>A few decisions made this manageable at scale rather than becoming its own operational burden:</p><ul><li><strong>Templated connector generation.</strong> With 650 near-identical connectors differing mainly by endpoint and topic, we generated KafkaConnector manifests from a single template plus a config table, rather than hand-authoring YAML. New source onboarding went from a day of engineering work to a config-table entry and a GitOps commit.</li><li><strong>Dedicated Connect worker pools per domain.</strong> Grouping connectors by business domain (cement operations, aggregates, logistics) onto separate KafkaConnect worker pools meant a noisy or failing connector in one domain couldn&#39;t starve resources from another.</li><li><strong>Per-connector health surfaced, not buried.</strong> Connector status, task failures, and throughput needed to be queryable independently — which later became the foundation for an agentic health-monitoring chatbot covering all 650 connectors (more on that in a future post).</li></ul><h3>The API layer: getting data from the front end into Kafka</h3><p>Not every producer was a backend system with direct Kafka access, and we didn’t want front-end or partner applications holding Kafka client credentials. So we built a thin REST API layer in front of Kafka specifically for front-end and lightweight service producers:</p><ul><li>Front-end and gateway services call a versioned REST endpoint (POST /v1/events/dispatch).</li><li>The API validates payload schema, attaches metadata (source, timestamp, trace ID), and produces to the appropriate Kafka topic on the caller’s behalf.</li><li>Kafka credentials and topic routing logic never leave the API layer — front-end teams integrate against a stable HTTP contract instead of Kafka internals, and we can change partitioning or topic strategy without breaking any client.</li></ul><p>This turned Kafka from “infrastructure only the platform team touches” into a shared capability that application teams could build against confidently.</p><h3>What this bought us</h3><ul><li><strong>650+ source connectors</strong> running as independently deployable, independently scalable units instead of one brittle monolith of configuration.</li><li><strong>Self-healing infrastructure</strong> — broker and connector failures recover through Kubernetes scheduling and Strimzi reconciliation, not pages.</li><li><strong>A stable API contract</strong> for front-end and partner teams to publish events without Kafka expertise.</li><li><strong>Consistent multi-environment operations</strong> through Rancher, cutting new-environment setup from days to hours.</li></ul><h3>Lessons learned</h3><p>A few things I’d tell anyone starting down this path:</p><ol><li><strong>Template your connectors early.</strong> If you’re heading toward dozens or hundreds of source connectors, hand-written YAML per connector becomes the bottleneck long before Kafka itself does.</li><li><strong>Separate Connect worker pools by blast radius</strong>, not just by convenience. A single noisy connector shouldn’t be able to degrade unrelated data domains.</li><li><strong>Don’t expose Kafka directly to front-end teams</strong>, even when it’s technically easy. An API contract is cheaper to maintain long-term than a fleet of clients holding broker credentials.</li><li><strong>Design for observability from day one.</strong> With hundreds of connectors, “is everything healthy?” needs to be a queryable answer, not a manual check — which is exactly what led us to build an agentic monitoring layer on top of this platform.</li></ol><h3>Tool reference: what does what</h3><p>For anyone piecing together their own stack, here’s how each layer of this architecture maps to a job:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/747/1*K_VSWfywPq3qfUcOvXe3cw.png" /></figure><p>Next in this series: securing this same streaming platform’s Databricks layer inside an Azure private network — Entra ID, Private Link, and Unity Catalog permissioning for a lakehouse that never touches the public internet.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=36f5a738b7c0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Architecting a Near-Real-Time Retail Analytics Platform on Azure Databricks]]></title>
            <link>https://medium.com/@prasad.patsa/architecting-a-near-real-time-retail-analytics-platform-on-azure-databricks-afa470f46741?source=rss-b740d4e2a5b7------2</link>
            <guid isPermaLink="false">https://medium.com/p/afa470f46741</guid>
            <dc:creator><![CDATA[Durga Prasad Patsa]]></dc:creator>
            <pubDate>Tue, 28 Jul 2026 06:38:05 GMT</pubDate>
            <atom:updated>2026-07-28T06:38:05.774Z</atom:updated>
            <content:encoded><![CDATA[<h3>A whiteboard-to-production walkthrough for a high-volume streaming ingestion pipeline</h3><p>Most “big data architecture” articles start with the technology. This one doesn’t. It starts with a business problem, works through the math, and only <em>then</em> lands on Databricks, Delta Lake, and Azure Event Hubs — because that’s the order a good architect actually thinks in.</p><p><strong>The scenario:</strong> a retail chain’s point-of-sale and e-commerce systems emit a continuous stream of order, product, store, and customer events. Store managers need to see what’s happening on the floor within minutes, not next-day. Data scientists need clean, point-in-time-correct features to train demand and fraud models. And the whole thing has to hold up at real retail scale — roughly <strong>100 million events per hour</strong> at peak.</p><p>Here’s the full picture before we zoom in:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rTbBD1M0WONTn4ndhlVr-g.png" /></figure><h3>1. Start With the Business Outcome, Not the Stack</h3><p>Before touching a single Azure resource, the first question is: <em>what decision does this data enable, and how fast does it need to arrive to be useful?</em></p><p>For this platform, two personas define the requirement:</p><ul><li><strong>Store managers</strong> need near-real-time store reports — inventory levels, sales velocity, anomalies — to act on <em>during</em> a shift, not review the next morning.</li><li><strong>Data scientists</strong> need a governed feature store to train ML models (demand forecasting, fraud detection) without hand-rolling their own pipelines.</li></ul><p>Both converge on a single goal: <strong>near-real-time data visibility</strong>, anchored to a <strong>15-minute SLA</strong>. Scoping out executives and external partners for now also keeps the governance surface smaller and the v1 build tractable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SWnq2TPVD7GQT_3HP3uhaw.png" /></figure><p>This is the step architects skip at their peril. A 15-minute SLA and a sub-second SLA are <em>completely different architectures</em> — different costs, different failure modes, different teams to staff. Get this wrong and everything downstream is either overbuilt or underbuilt.</p><h3>2. Translate the Business Goal Into Requirements</h3><p>With the outcome defined, it’s time to turn it into concrete functional and non-functional requirements.</p><p><strong>Functional:</strong> the source is a continuous stream of events from retail POS and e-commerce APIs — explicitly <em>not</em> a batch export or a change-data-capture (CDC) feed off a transactional database. That distinction matters enormously: it rules out nightly ETL jobs and file-drop patterns from the outset and commits the team to a streaming-first ingestion layer.</p><p><strong>Non-functional:</strong></p><ul><li><strong>Latency:</strong> 15 minutes end-to-end, event to dashboard.</li><li><strong>Volume:</strong> 100 million records/hour, sized for 3x peak load.</li><li><strong>Availability:</strong> 99.9% — technically just short of “always on,” but acceptable for an operations dashboard rather than a payments system.</li><li><strong>Retention:</strong> Bronze (raw) data kept 30–90 days; ML feature history retained longer to support backtesting.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D1slNQfKad2CwhGahwLs5A.png" /></figure><p>Notice the retention split. Raw operational data doesn’t need to live forever, but a data scientist backtesting a model six months from now absolutely needs historical features. Baking that distinction in early avoids an awkward retroactive redesign of the Silver/Gold layers later.</p><h3>3. Do the Back-of-the-Envelope Math Before the Budget Conversation</h3><p>This is the step that separates a credible platform proposal from a guess. Before anyone talks about cluster sizing or reserved capacity, size the data.</p><p>At 100M events/hour and an average event size of 0.75 KB:</p><ul><li><strong>~75 GB/hour</strong></li><li><strong>~1.8 TB/day</strong></li><li><strong>~54 TB/month</strong></li><li><strong>~650 TB/year of raw data</strong></li></ul><p>Apply a “medallion multiplier” to account for Bronze + Silver + backup copies, and the platform’s total footprint over its useful life lands around <strong>2.5 PB</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QtHdTtO43BwBHfyUOTaXkQ.png" /></figure><p>Why does this matter so much? Because this single number — done on a napkin, before any vendor conversation — is what lets you walk into a budget meeting with a defensible number instead of a hand-wave. It also immediately tells you this is a job for cloud object storage with lifecycle policies, not a fixed-size on-prem SAN.</p><h3>4. Design the Ingestion Layer for Streaming, Not Batch</h3><p>Given the “continuous events, not batch/CDC” requirement, the ingestion layer has to be built around a real message bus and a streaming compute engine from day one.</p><p>The pipeline:</p><ul><li><strong>Azure Event Hubs</strong> (using the Kafka protocol) as the ingestion front door for retail POS streaming events — a natural fit since most retail systems already speak Kafka-compatible protocols.</li><li><strong>Databricks Structured Streaming / Auto Loader</strong> continuously consuming from Event Hubs.</li><li>A <strong>Schema Registry</strong> to enforce contract discipline on incoming events.</li><li>A <strong>Dead Letter Queue</strong> to catch and quarantine malformed records instead of failing the whole pipeline.</li><li><strong>Checkpointing</strong> for exactly-once processing guarantees.</li><li><strong>Watermarking</strong> to gracefully handle late-arriving events, which are inevitable at retail scale.</li><li><strong>Idempotent merge-on-write</strong> so replays and retries never produce duplicate records.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SE0VIyEc30lPGAlupnNw3A.png" /></figure><p>This layer is deliberately unglamorous. Nobody notices a well-built ingestion pipeline — they only notice a broken one, at 2 a.m., during a Black Friday spike. Checkpointing, watermarking, and DLQs aren’t nice-to-haves here; they’re the difference between a platform that self-heals and one that pages an on-call engineer every time a store’s network hiccups.</p><h3>5. Structure the Data With a Medallion Architecture</h3><p>Once events are landing reliably, the next question is how to shape raw events into something usable. The medallion pattern (Bronze → Silver → Gold) is the natural fit for a streaming-first Databricks platform.</p><ul><li><strong>Bronze — Raw Events:</strong> Untouched, append-only, partitioned by date and store for efficient pruning later.</li><li><strong>Silver — Cleaned &amp; Canonical:</strong> Deduplication, normalization, and CDC-style merges resolve the raw stream into clean, canonical entities — Order, Product, Store, Customer.</li><li><strong>Gold — Analytics Ready:</strong> Silver data is reshaped for two very different consumers:</li><li>A <strong>reporting shape</strong> — aggregated KPIs in a star schema, built for Power BI dashboards.</li><li>A <strong>feature shape</strong> — point-in-time features with rolling windows, built for ML training and inference.</li></ul><p>The whole pipeline refreshes on a micro-batch cadence (e.g., every 5 minutes), which is what actually delivers the 15-minute SLA end to end.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ILEFW_o34azuiZqi0NEnTw.png" /></figure><p>The key design insight here is that <strong>Gold isn’t one table — it’s two shapes serving two different consumers.</strong> Trying to force a single Gold table to serve both dashboards and ML feature retrieval usually ends in a table that does neither job well. Splitting them early keeps both consumers fast and keeps the semantics honest (a rolling 30-day feature window is a very different query pattern than a daily KPI rollup).</p><h3>6. Build the Serving and Sharing Layer Around the Actual Consumers</h3><p>With Gold data in place, the serving layer just needs to route each shape to where it’s consumed.</p><ul><li>The <strong>reporting shape</strong> flows through <strong>Databricks SQL Serverless</strong> into <strong>Power BI</strong>, refreshed every 15 minutes via DirectQuery or scheduled refresh — directly hitting the SLA.</li><li>The <strong>feature shape</strong> flows into the <strong>Databricks Feature Store</strong>, feeding <strong>Azure ML</strong> for training and inference, which data scientists consume directly.</li><li><strong>External partner sharing via Delta Sharing is explicitly out of scope for now</strong> — franchise and third-party access is a “not yet” rather than a “never,” which keeps v1 governance simpler without closing the door later.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OPnagYPRSlSkGlsv1DeUxA.png" /></figure><p>That crossed-out Delta Sharing box is doing real architectural work. It’s a conscious decision to <em>not</em> build for a requirement that doesn’t exist yet, while leaving Unity Catalog and Delta Sharing available as a switch to flip later rather than a rebuild.</p><h3>7. Make Governance and Security a Day-1 Concern, Not a Retrofit</h3><p>Everything in this platform is registered in <strong>Unity Catalog</strong>, organized into retail_bronze, retail_silver, and retail_gold catalogs, with automatic lineage tracked across all three.</p><p>Access is role-based and layered:</p><ul><li><strong>Store managers</strong> get <strong>row-level security</strong> — a filter like store_id = X — so a manager only ever sees their own store&#39;s data, enforced at the query layer, not in application code.</li><li><strong>Data scientists</strong> get RBAC access to Silver and Gold for model development.</li><li>Underlying security policies cover <strong>column masking</strong> for PII and payment fields, <strong>encryption</strong> at rest and in transit, and <strong>Entra ID SSO</strong> for authentication.</li><li>Compliance checks are baked in structurally: <strong>PCI</strong> controls wherever payment data appears, <strong>GDPR/CCPA</strong> controls wherever customer IDs appear.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZoyWyop1bcJT6UTfa4PeHQ.png" /></figure><p>Row-level security for store managers is called out specifically as a <strong>Day 1</strong> requirement, and that’s deliberate. Retrofitting row-level security after a hundred dashboards already exist is one of the more painful migrations in enterprise data platforms — every downstream report has to be re-validated. Building it in from the first Gold table sidesteps that entirely.</p><h3>8. Instrument the Platform for Operations, Not Just Analytics</h3><p>A streaming pipeline that nobody monitors is a pipeline that fails silently. This platform treats operability as a first-class layer.</p><ul><li><strong>Databricks Workflows</strong> orchestrate the streaming jobs, with automatic restart on failure.</li><li>An <strong>observability dashboard</strong> tracks micro-batch duration against the trigger interval, with alerts firing when duration approaches the SLA boundary — a 5-minute warning zone before an actual SLA breach.</li><li><strong>Event Hub consumer lag</strong> and <strong>Azure Monitor + Log Analytics</strong> provide the underlying telemetry.</li><li>A proper <strong>Dev → Test → Prod</strong> promotion path, managed through <strong>Databricks Asset Bundles</strong> or <strong>Terraform</strong>, keeps changes from landing directly in production.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wL2485mSitFgOsMu4pclzA.png" /></figure><p>The subtlety worth calling out: the alert threshold is set <em>before</em> the SLA is actually breached. That’s the difference between a dashboard that tells you something already went wrong and one that gives an engineer a few minutes to intervene before a store manager notices stale data.</p><p>These three concerns are usually treated as separate workstreams. On this platform, they’re designed together, because a decision in one almost always trades against the other two.</p><h3>9. Tune for Performance, Cost, and Reliability Together</h3><p>These three concerns are usually treated as separate workstreams. On this platform, they’re designed together, because a decision in one almost always trades against the other two.</p><p><strong>Performance:</strong> the Photon engine accelerates query execution; liquid clustering on store_id/date combined with partition pruning keeps micro-batch writes fast and controlled; scheduled OPTIMIZE/VACUUM jobs keep file sprawl in check.</p><p><strong>Cost:</strong> streaming clusters auto scale to a steady-state size rather than sitting over-provisioned; the Power BI-facing SQL warehouse runs serverless with zero idle compute cost; spot instances handle non-critical backfill jobs where interruption is tolerable.</p><p><strong>Reliability:</strong> the Delta transaction log means no heroic custom code is needed for consistency; geo-redundant storage delivers near-zero RPO; checkpointing plus the dead-letter queue from the ingestion layer deliver the exactly-once guarantee end to end.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0iLcnv7iVc4c5LaV4zaMaA.png" /></figure><h3>10. Be Explicit About Trade-offs</h3><p>Every real architecture is a series of decisions where something was consciously given up. Documenting those trade-offs — not just the final design — is what makes a proposal defensible in review.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/910/1*dew7nTKx9rva9Pgp-p_F5w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9SST3bqziQmQbLrArMFMqQ.png" /></figure><p>None of these choices are universally “correct” — a fraud-detection system might reasonably choose sub-minute latency despite the cost, and a smaller organization might accept a multi-engine stack for portability. What matters is that each choice was made on purpose, against the actual requirements gathered in step 2, not by default.</p><h3>11. Tie It Back to Principles</h3><p>Step back from the diagrams and four architecture principles emerge, each mapping directly back to the requirements this platform was built to satisfy:</p><ul><li><strong>Business-driven:</strong> one SLA, one pipeline — store managers and data scientists both draw from the same governed Gold layer rather than parallel, drifting pipelines.</li><li><strong>Data-quality-first:</strong> nothing reaches reporting or features without passing through the Silver merge and its data-quality checks.</li><li><strong>Secure by design:</strong> row-level security from Day 1 prevents unauthorized access and preempts a costly retrofit.</li><li><strong>Cost-efficient and AI-ready:</strong> idle compute stays at zero, serverless SQL and the Feature Store keep spend matched to actual usage, and the platform is already positioned for ML training rather than bolted on later.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PBCfw5S4HAGzR9kB2LfUFg.png" /></figure><p>Put another way: <strong>right data, right time, right place, right cost.</strong> Governed and deduplicated data, delivered inside a 15-minute SLA, routed to the tool each consumer actually uses (Power BI vs. the Feature Store), at a cost structure that scales with usage instead of sitting idle.</p><h3>Closing Thought</h3><p>It’s tempting to start an architecture conversation with “we’re using Databricks and Event Hubs.” This walkthrough deliberately didn’t — because the technology choices only make sense in light of what came before them: a business SLA, a volume calculation, and a set of explicit trade-offs. Swap the numbers, the personas, or the compliance requirements, and the diagrams above still hold as a <em>method</em>, even if the specific Azure services underneath end up different.</p><p>That’s really the test of a good architecture write-up: could someone else use this same sequence of steps to design a completely different platform? If the answer is yes, the diagrams did their job.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=afa470f46741" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kafka Health Monitoring Agentic AI custom Web App using Databricks-Real Troubleshooting]]></title>
            <link>https://medium.com/@prasad.patsa/kafka-health-monitoring-agentic-ai-custom-web-app-using-databricks-real-troubleshooting-d24afed33ef2?source=rss-b740d4e2a5b7------2</link>
            <guid isPermaLink="false">https://medium.com/p/d24afed33ef2</guid>
            <dc:creator><![CDATA[Durga Prasad Patsa]]></dc:creator>
            <pubDate>Sun, 14 Jun 2026 15:49:25 GMT</pubDate>
            <atom:updated>2026-06-14T15:49:25.819Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k7skhO5YNlBw9OUCRtjKlQ.png" /></figure><h3>Building an End-to-End Multi-Agent AI for Kafka Observability on Databricks</h3><p><em>How I turned “why did the order topic fail?” into a question you can just ask — across 650+ connectors, multiple tenants, and a Rancher/Kubernetes streaming platform — using Llama 3, RAG, and a team of cooperating AI agents.</em></p><h3>The 2 a.m. problem</h3><p>Picture a streaming data platform that moves the lifeblood of a business — invoices, orders, quotes, dispatch and shipment events — across <strong>650+ Kafka connectors</strong>, spread over multiple tenants, all running on Rancher/Kubernetes.</p><p>Now picture one of those connectors quietly failing.</p><p>An order topic stops committing. A sink connector’s task dies. A pod hits an out-of-memory condition and crash-loops. Downstream, invoices go missing and orders stall — but the platform doesn’t shout. We had Grafana dashboards, and they were good at telling us <em>that</em> something was wrong. They were bad at telling us <em>why</em>, fast. And there was a cruel twist: failed pods <strong>auto-restarted</strong>, so by the time an engineer opened a terminal to investigate, the evidence had already rotated away.</p><p>Root-cause analysis meant a human pivoting across three systems under pressure: the Kafka Connect REST API to check connector and task state, Rancher/Kubernetes to inspect pod health and throughput, and Elasticsearch to mine logs. Three tools, three mental models, one stressed engineer, at 2 a.m.</p><p>I wanted to collapse all of that into a single sentence: <em>“Why did the order topic fail yesterday?”</em> — and get a grounded, root-caused answer back.</p><p>This is how I built that, end to end.</p><h3>Why agentic AI, and not just another dashboard</h3><p>The instinct is to build one more dashboard. But dashboards answer questions you anticipated. Incidents are, by definition, the ones you didn’t.</p><p>What I actually needed was something that could <strong>reason across multiple systems on demand</strong> — decide <em>which</em> data sources a given question requires, go fetch from each, correlate the findings, and explain the result in plain language. That’s not a dashboard. That’s an <strong>agent</strong> — or, more precisely, a <em>team</em> of specialist agents coordinated by an orchestrator.</p><p>The design principle I started from: <strong>one agent per data source, each an expert in exactly one thing</strong>, and a coordinator that composes them per question. This keeps each agent simple and testable and lets the system answer questions that span systems (“the pod is crash-looping <em>and</em> the logs show a sink timeout”) without any single agent having to know everything.</p><h3><strong>The architecture</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0JxivM81PTRLvOiL6ozF8w.png" /></figure><p>Six layers, each with a clear job. Let me walk through them the way data actually flows.</p><h3>Layer 1 — The data sources (what the agents can see)</h3><p>Three sources cover the observability surface of a Kafka streaming platform:</p><p><strong>Kafka Connect REST API.</strong> The source of truth for connector and task state. A connector can report RUNNING while one of its tasks has silently FAILED with a stack trace — a distinction that matters enormously, because the connector looks alive while doing no work.</p><p><strong>Rancher / Kubernetes.</strong> The connectors run as pods. This is where you see restart counts, CPU/memory pressure, throughput (messages/min), and the tell-tale CrashLoopBackOff with an OOMKilled event — the pattern that used to vanish on auto-restart.</p><p><strong>Elasticsearch.</strong> The centralized log store. This is where root cause actually lives: the SQLException: connection timed out to sink target, the retry-buffer-full warnings, the latency spikes that precede a failure.</p><p>Each source is wrapped in a thin client with a single responsibility, so the agents never speak raw HTTP — they ask for <em>state</em>, <em>diagnostics</em>, or <em>logs</em> and get back structured data.</p><h3>Layer 2 — RAG grounding (the anti-hallucination layer)</h3><p>This is the layer that makes the whole thing trustworthy, and it’s the one I’d build first if I did it again.</p><p>With 650+ connectors, an LLM asked about “the invoice connector” will happily <em>invent</em> a connector id, a topic, or a cluster that sounds right and is completely wrong. In observability tooling, a confident wrong answer is worse than silence — it sends engineers chasing a connector that doesn’t exist during a live incident.</p><p>So I exported the metadata for all 650+ connectors — id, name, aliases, topic, tenant, cluster, owner — into a structured JSON store and put a <strong>Retrieval-Augmented Generation</strong> layer in front of every query. On each question, the system retrieves the matching connector record <em>first</em> and injects it as grounding context. Crucially, the agents then act on the <strong>retrieved id</strong>, never on free text the model produced:</p><blockquote>def retrieve(query, k=4):<br> “””Return the real connector records most relevant to the query.”””<br> # lexical (or vector) match over the 650+ metadata records<br> …</blockquote><blockquote>record = retrieve(user_question)[0] # a verified connector<br>status = kafka_connect.get_status(record[“id”]) # grounded lookup</blockquote><p>The model is free to <em>phrase</em> the answer; it is not free to <em>invent the entity</em> it’s answering about. Hallucinated connector names went to zero.</p><h3>Layer 3 — The specialist agents</h3><p>Three agents, each an expert in one source, each returning <strong>structured findings</strong> rather than prose (so the system can correlate them and so they stay unit-testable):</p><ul><li><strong>Connector Health Agent</strong> — queries Kafka Connect for connector and per-task state, surfacing the silent “connector RUNNING, task FAILED” case and the task’s error trace.</li><li><strong>Pod Diagnostics Agent</strong> — queries Rancher/Kubernetes for the pod backing the connector: phase, restarts in the last hour, CPU/memory, throughput, and the latest event (e.g., the OOM crash-loop).</li><li><strong>Log Mining Agent</strong> — searches Elasticsearch for the connector’s recent errors and anomalies, returning the actual log lines and an error count.</li></ul><p>Because each agent emits data, the orchestrator can combine them: a “no throughput” reading from the pod agent plus a “sink timeout” from the log agent compose into a single, defensible root cause.</p><h3>Layer 4 — The orchestrator and router</h3><p>The orchestrator is the brain that turns a question into a plan. Using Llama 3, it reads the question (plus the RAG-grounded connector context) and decides <strong>which agents to run</strong>:</p><ul><li><em>“Is the invoice connector healthy?”</em> → Connector Health only.</li><li><em>“Why did the order topic fail yesterday?”</em> → Log Mining (and Connector Health).</li><li><em>“How many messages were processed during the dispatch outage?”</em> → Pod Diagnostics + Log Mining.</li></ul><p>It then executes the chosen agents against the grounded connector id, collects their findings, and hands everything to the synthesis step. Routing through the model — rather than hard-coded if/else — is what lets the system handle questions I never explicitly programmed for, while the RAG grounding keeps it honest.</p><h3>Layer 5 — The Llama 3 layer (and making it affordable)</h3><p>I used <strong>Llama 3</strong> as the reasoning engine, served through Databricks model serving. The LLM does two jobs: routing (which agents) and synthesis (the final answer). Both are wrapped so the backend is swappable — Databricks serving in production, with the same interface working against other Llama 3 hosts for development.</p><p>Two pieces of tuning made it production-viable:</p><p><strong>Token control.</strong> Left unbounded, the model wrote sprawling reasoning on every call — multiplied across multiple agent steps, that’s a real cost and latency problem. I made max_tokens an explicit, tuned parameter (I settled on <strong>2034</strong> after testing — enough for a complete answer, tight enough to stop the rambling), and summarized tool outputs before feeding them back into context.</p><blockquote><strong>response = llm.chat(messages=messages, max_tokens=2034, temperature=0.2)</strong></blockquote><p><strong>Prompt engineering for shape.</strong> The synthesis prompt enforces a strict output: connector id, state, likely root cause, recommended next action — and nothing else. During an incident, nobody reads six paragraphs.</p><blockquote>SYNTH_SYSTEM = (<br> “You are a Kafka observability assistant. Using ONLY the agent findings, “<br> “answer in at most 6 short lines: connector id, state, root cause (if any), “<br> “and the recommended next action. No preamble.”<br>)</blockquote><h3>Layer 6 — The conversational interface and the analytics loop</h3><p>The whole thing is exposed through a <strong>Databricks App</strong> as a chat interface — an engineer types a question and gets a grounded answer, no dashboard-hopping. Behind the scenes, the orchestrator is one call: handle(question) → answer.</p><p>And because every query and finding flows into <strong>Delta Lake</strong> (organized in a Medallion bronze → silver → gold structure), the platform also feeds <strong>Databricks Genie Spaces, Metric Views, and dashboards</strong> — so the same system that answers ad-hoc incident questions also powers self-service trend analytics: which connectors fail most, mean time to recovery, failure patterns by tenant.</p><h3>A question, end to end</h3><p>Here’s <em>“Why did the order topic fail yesterday?”</em> flowing through every layer:</p><ol><li><strong>RAG</strong> retrieves the real record: order-source-na-01, topic na.orders.v1, cluster connect-na-east-1.</li><li><strong>Router</strong> (Llama 3) decides this needs Log Mining (and Connector Health).</li><li><strong>Log Mining Agent</strong> queries Elasticsearch for that connector → finds 2 ERRORs, top line: <em>&quot;Task failed: SQLException: connection timed out to sink target.&quot;</em></li><li><strong>Connector Health Agent</strong> confirms connector RUNNING, task FAILED.</li><li><strong>Synthesis</strong> (Llama 3) composes: <em>“order-source-na-01: task FAILED while connector shows RUNNING. Root cause: sink target connectivity/latency timed out. Recommended action: verify the downstream target and the retry-tolerance config.”</em></li><li>The answer lands in the Databricks App, grounded and cited — in seconds, not a 2 a.m. scavenger hunt.</li></ol><h3>From demo to production: the five bugs that almost killed it</h3><p>A demo that works once is not a system. Here are the five problems that stood between “it answered” and “I trust it” — each in <strong>STAR</strong> format (Situation, Task, Action, Result), because the fixes are the real transferable knowledge.</p><h3>Bug #1 — The agent that ran forever</h3><p><strong>Situation.</strong> The first time I wired the orchestrator to live connectors, it never returned. It just kept thinking, calling, thinking, calling — an infinite loop.</p><p><strong>Task.</strong> Find out why an agent that should answer in 2–3 steps looped indefinitely, and guarantee it always terminates.</p><p><strong>Action.</strong> The root cause was subtle: the data the connector API returned <strong>wasn’t rich enough for the LLM to consider the question “answered.”</strong> So the agent kept deciding it needed one more tool call — forever. I added a hard maximum-iteration ceiling, and made the agent <strong>finalize gracefully</strong> on the last step instead of erroring out:</p><blockquote>MAX_AGENT_STEPS = 15</blockquote><blockquote>for step in range(MAX_AGENT_STEPS):<br> action = agent.plan_next_step(state)<br> if action.is_final or step == MAX_AGENT_STEPS — 1:<br> return agent.finalize(state) # force a best-effort answer<br> state = agent.execute(action)</blockquote><p><strong>Result.</strong> No more runaway loops. Every query terminates in a bounded number of steps, returning a partial-but-honest answer worst case. The deeper lesson: <strong>an agent must be able to answer with incomplete information, or it will never stop.</strong></p><h3>Bug #2 — The token bill that wouldn’t stop climbing</h3><p><strong>Situation.</strong> Even after the loop was bounded, each query consumed an enormous number of tokens — verbose reasoning on every call, multiplied across up to 15 steps and 650 connectors of context.</p><p><strong>Task.</strong> Bring token usage (and therefore cost and latency) under control without hurting quality.</p><p><strong>Action.</strong> I made max_tokens an explicit, tuned parameter rather than a default — <strong>2034 tokens per call</strong> after testing — and summarized tool outputs before feeding them back into context.</p><blockquote>response = llm.chat(messages=messages, max_tokens=2034, temperature=0.2)</blockquote><p><strong>Result.</strong> Token utilization dropped sharply and responses came back faster, at a fraction of the cost — with no loss in answer quality. In agentic systems, <strong>unbounded generation is a cost bug, and </strong><strong>max_tokens is a control knob, not an afterthought.</strong></p><h3>Bug #3 — Nobody reads a six-paragraph answer during an outage</h3><p><strong>Situation.</strong> The answers were correct but unusable — six paragraphs of prose when an on-call engineer needs the answer in five seconds.</p><p><strong>Task.</strong> Make every response crisp, scannable, and action-oriented.</p><p><strong>Action.</strong> I rewrote the synthesis prompt to enforce a strict shape — connector id, state, root cause, recommended action, nothing else:</p><blockquote>SYNTH_SYSTEM = (<br> “You are a Kafka observability assistant. Using ONLY the agent findings, “<br> “answer in at most 6 short lines: connector id, state, root cause (if any), “<br> “and the recommended next action. No preamble, no filler.”<br>)</blockquote><p><strong>Result.</strong> Responses became tight and operational. A small prompt change, but it’s the difference between a tool people <em>use</em> and one they close. <strong>In ops tooling, brevity is a feature.</strong></p><h3>Bug #4 — The hallucinated connector (the dangerous one)</h3><p><strong>Situation.</strong> The scariest bug: with 650+ connectors, the model would confidently reference a connector that didn’t exist, or answer about the wrong one — inventing plausible ids and topics. A confident wrong answer sends engineers chasing ghosts mid-incident.</p><p><strong>Task.</strong> Guarantee the system only ever talks about <em>real</em> connectors, with their <em>real</em> ids, topics, and clusters.</p><p><strong>Action.</strong> I implemented the RAG grounding layer described above — all 650+ connectors’ metadata in a JSON store, retrieved first, with the agents acting on the <strong>retrieved id</strong>, never free text the model produced.</p><p><strong>Result.</strong> Hallucinated connector names went to <strong>zero</strong>. Every answer is anchored to a verified record and even cites which connector ids grounded it. <strong>RAG here isn’t about knowledge — it’s about entity grounding, the single highest-leverage reliability fix in the project.</strong></p><h3>Bug #5 — One typo, five different answers</h3><p><strong>Situation.</strong> Users type fast and misspell. Someone asks about the <em>“invioce”</em> connector; the system finds no exact match and silently answers about a <em>different</em> connector — or a different one each time for the same typo.</p><p><strong>Task.</strong> Make the system resilient to spelling mistakes so a typo resolves to the <em>correct</em> connector, consistently.</p><p><strong>Action.</strong> I added a fuzzy-matching / spell-correction layer between the user’s text and the connector lookup:</p><blockquote>from difflib import get_close_matches</blockquote><blockquote>def resolve_connector(term, known_names, cutoff=0.6):<br> match = get_close_matches(term.lower(), known_names, n=1, cutoff=cutoff)<br> return match[0] if match else None # “invioce” -&gt; “invoice”</blockquote><p>If nothing clears the threshold, the system asks for clarification rather than guessing.</p><p><strong>Result.</strong> Typos resolve deterministically to the right connector — same question, same correct answer, every time. And when intent is genuinely ambiguous, the system asks instead of inventing. <strong>Robustness to messy human input is what separates a demo from a tool people rely on.</strong></p><h3>The impact</h3><p>The result was a single, conversational front door to a sprawling streaming platform. Instead of pivoting across Kafka Connect, Rancher, and Elasticsearch under pressure, an engineer asks one question and gets a grounded, root-caused answer with a recommended action — <em>[insert your metric: e.g., reduced mean-time-to-diagnosis from ~30 minutes to under 2; cut incident escalations by X%; gave non-Kafka-experts self-service visibility into 650+ connectors]</em>.</p><p>More importantly, it changed <em>who</em> could troubleshoot. Deep Kafka/Kubernetes expertise was no longer the price of entry to ask “what’s wrong with the invoice pipeline?”</p><h3>What’s next: evolving to MCP (Model Context Protocol)</h3><p>The system works — but its agents are wired to their data sources with bespoke client code. Adding a new source (say, a Prometheus metrics agent or a Schema Registry agent) means writing a new client and hand-wiring it into the orchestrator. That custom integration tax is exactly the problem the <strong>Model Context Protocol (MCP)</strong> was built to remove.</p><p>MCP is an open standard — created by Anthropic and now governed under the Linux Foundation’s Agentic AI Foundation — for connecting AI agents to tools, resources, and data through one common protocol. Think of it as a <em>USB-C port for AI</em>: any MCP-compatible agent can connect to any MCP server the same way, instead of every team rebuilding the same adapters.</p><p>Here’s how I’m planning to evolve the platform:</p><ol><li><strong>Wrap each data source as an MCP server.</strong> Instead of three bespoke clients, I expose Kafka Connect, Rancher/Kubernetes, and Elasticsearch as MCP servers, each declaring its tools through the standard tools/list / tools/call interface:</li></ol><blockquote>from mcp.server.fastmcp import FastMCP</blockquote><blockquote>mcp = FastMCP(“kafka-observability”)</blockquote><blockquote><a href="http://twitter.com/mcp">@mcp</a>.tool()<br>def connector_status(connector_id: str) -&gt; dict:<br> “””Return Kafka Connect connector and task state for a connector id.”””<br> return kafka_connect.get_status(connector_id)</blockquote><blockquote><a href="http://twitter.com/mcp">@mcp</a>.tool()<br>def pod_diagnostics(connector_id: str) -&gt; dict:<br> “””Return Rancher/Kubernetes pod metrics for the connector’s pod.”””<br> return k8s.diagnostics(connector_id)</blockquote><p><strong>2. Turn the orchestrator into an MCP client.</strong> Rather than importing agent classes, the orchestrator <em>discovers</em> available tools at runtime via the protocol and lets Llama 3 call them. New capability = stand up a new MCP server; no orchestrator rewrite.</p><p><strong>3. Govern it through Unity Catalog.</strong> This is the part that makes MCP enterprise-ready on Databricks: MCP servers are governed through the <strong>Unity AI Gateway</strong>, with Unity Catalog permissions controlling who can call each server and managed OAuth handling credentials so secrets never touch the agent. The same governance layer that protects the lakehouse now protects the observability tools.</p><p><strong>4. Keep RAG, add MCP — the hybrid.</strong> The recommended pattern is to combine both: RAG over the large, fairly static connector corpus for grounding, and MCP for the live, authoritative lookups against the running systems. Grounding stays fast; actions stay fresh.</p><p><strong>5. Stream long-running diagnostics and add human-in-the-loop.</strong> MCP’s stateful channels let a server stream partial results during a long investigation, so the agent can make intermediate decisions or pause to ask a human — exactly what you want for a complex root-cause trace.</p><p>The payoff: the Kafka observability tools stop being a one-off integration and become <strong>reusable, governed building blocks</strong> that any agent in the organization can compose — and that other teams could even discover and adopt through a shared catalog. That’s the difference between building a tool and building a platform.</p><h3>What I’d tell another architect</h3><ul><li><strong>Build the grounding layer first.</strong> For any LLM system touching many real entities, RAG-for-grounding is the highest-leverage reliability investment you’ll make.</li><li><strong>One agent per source.</strong> Small, single-responsibility agents that return <em>data</em> (not prose) compose into answers you can trust and tests you can write.</li><li><strong>Let the model route, but never let it invent.</strong> LLM routing buys flexibility; grounded ids keep it honest.</li><li><strong>Treat termination, tokens, and brevity as first-class design constraints</strong>, not afterthoughts.</li><li><strong>Standardize your integrations early.</strong> What starts as three custom clients becomes a maintenance burden fast — MCP turns tools into reusable, governed building blocks.</li><li><strong>Close the loop into your lakehouse.</strong> The same telemetry that answers today’s incident becomes tomorrow’s trend analytics through Delta, Genie, and metric views.</li></ul><h3><em>Agentic AI isn’t magic — it’s good distributed-systems design with an LLM as the coordinator. The engineering discipline that makes any platform reliable is exactly what makes an agentic one reliable too.</em></h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d24afed33ef2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Delta Sharing in Databricks]]></title>
            <link>https://medium.com/@prasad.patsa/delta-sharing-in-databricks-04bfe2de4fe0?source=rss-b740d4e2a5b7------2</link>
            <guid isPermaLink="false">https://medium.com/p/04bfe2de4fe0</guid>
            <category><![CDATA[data-architecture]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[data-governance]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Durga Prasad Patsa]]></dc:creator>
            <pubDate>Wed, 15 Apr 2026 22:36:00 GMT</pubDate>
            <atom:updated>2026-04-15T22:36:00.156Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*GuXSDb2YzMgwKBO0vAbGHA.png" /></figure><p><strong>Executive Summary</strong></p><p>Delta Sharing eliminates the need to copy, replicate, or pipeline data to external consumers. By granting live, read-only access directly to source Delta tables in cloud storage, organizations can cut data infrastructure costs by 40–70%, reduce time-to-data from days to minutes, and eliminate the security and compliance risks that come with proliferating data copies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/811/1*KJQACWtzsQC-ki6uHim8Sw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/804/1*ghEPBbaTasgt8FZ2WXCxGA.png" /></figure><p>Before Delta Sharing, organizations that needed to share data externally — with partners, subsidiaries, customers, or vendors — had only a handful of options. Every single one of them involved moving, copying, or replicating data. That pattern creates a cascade of problems.</p><h3>1.1 The Traditional Data Sharing Stack</h3><p>A typical enterprise data sharing pipeline involves many moving parts:</p><blockquote>An ETL/ELT job to extract data from the source system</blockquote><blockquote>• Transformation and cleansing pipelines to prepare data for the target schema</blockquote><blockquote>• A replication or transfer mechanism (SFTP, database replication, cloud sync tools)</blockquote><blockquote>• A destination store (a second S3 bucket, an Azure Blob container, a replicated database)</blockquote><blockquote>• Access management on the destination side, separately from the source</blockquote><blockquote>• A refresh schedule — hourly, daily, or weekly — meaning data is always stale</blockquote><h3>1.2 The Pain Points</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/811/1*-9FKEHZgXJODY9OvikC8jQ.png" /></figure><p><strong>Real-world example</strong></p><p>A mid-sized financial services firm sharing trade data with 8 partner institutions maintains 8 separate replication pipelines, 8 destination buckets, and a team of 3 engineers dedicated solely to keeping those pipelines healthy. That is an engineering team whose entire output is infrastructure maintenance — zero business value added.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/803/1*qzaaXitiY5hiF7hTLc6i5w.png" /></figure><p>Delta Sharing takes a fundamentally different approach: instead of moving data to the consumer, it gives the consumer a governed, time-limited, read-only window into the original data where it already lives.</p><h3>2.1 The Core Principle: Access, Not Copy</h3><p>When a provider creates a Delta Share, they are not duplicating data. They are registering a set of permissions in Unity Catalog that say: “Recipient X may read table Y, partition Z.” When the recipient queries, the Delta Sharing server issues a short-lived signed URL pointing to the specific Parquet files in the provider’s cloud storage. The data never moves.</p><p>This single architectural decision — access over copy — eliminates the root cause of most data sharing pain: duplicate storage, stale pipelines, sprawling credentials, and governance black holes.</p><h3>2.2 Problem-by-Problem Resolution</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/811/1*WJfukipSS1lvxgNtSjCGjQ.png" /></figure><h3>2.3 How It Makes Engineering Jobs Easier</h3><p>Delta Sharing does not just solve business problems — it fundamentally changes the daily experience of the data engineers who support data sharing workflows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/814/1*S61GZ44nD80t1KJT9kQQ0Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/798/1*f85_BTGctNnw7mkVI2LH9A.png" /></figure><p>The most structurally significant advantage of Delta Sharing is that data never leaves the provider’s cloud storage environment. The Parquet files that make up a Delta table remain in the provider’s S3, ADLS, or GCS bucket. The Delta Sharing server generates pre-signed, time-limited URLs (typically 1-hour TTL) that allow the recipient to read those specific files directly. Once the URL expires, access is automatically revoked without any manual action.</p><p>This means there is no secondary copy of the data to maintain, secure, back up, or pay for. For organizations sharing large datasets — hundreds of gigabytes to petabytes — this is an enormous structural cost reduction.</p><h3>3.2 Always-Current Data</h3><p>Because recipients read from the source Delta table directly, they are always reading the most recently committed version of the data. There is no replication lag, no batch window, and no “refresh” button. A partner running a query at 9:00 AM reads data that reflects transactions committed at 8:59 AM.</p><p>This real-time freshness has meaningful downstream business value: partner forecasting models are more accurate, compliance reports reflect actual current state, and joint operational processes are synchronized.</p><h3>3.3 Centralized Governance with Unity Catalog</h3><p>All sharing relationships are defined and managed in Unity Catalog — the same governance layer the provider uses internally. This means:</p><blockquote>• Access control is consistent: the same column-masking, row-filtering, and permission model applies to shared data as to internal data.</blockquote><blockquote>• Audit is automatic: every read by every recipient is logged with a timestamp, recipient identity, and table version accessed.</blockquote><blockquote>• Revocation is instant: revoking a GRANT in Unity Catalog immediately stops all future access — no pipeline to delete, no bucket to lock down.</blockquote><blockquote>• Lineage is preserved: Unity Catalog tracks data lineage end-to-end, so compliance teams can trace exactly what data a recipient was seeing at any point in time.</blockquote><h3>3.4 Cross-Cloud, Cross-Platform Reach</h3><p>Delta Sharing is an open protocol governed by the Linux Foundation. This means a provider on AWS can share with a recipient on Azure. A provider using Databricks can share with a recipient using only Pandas or Apache Spark. Recipients do not need a Databricks license — they need only the delta-sharing client library, which is free and open source.</p><p>This removes a major barrier in partner data exchange: the provider no longer needs to know or care what technology stack the partner uses. One share works for all of them.</p><h3>3.5 Partition-Level and Column-Level Precision</h3><p>Delta Sharing does not require sharing an entire table. Providers can expose:</p><blockquote>• Specific partitions only (e.g., only the rows for region = EMEA for a European partner)</blockquote><blockquote>• Column subsets via shared views (e.g., share revenue data but exclude cost columns)</blockquote><blockquote>• Row-filtered views via Unity Catalog dynamic views (e.g., share only rows where customer_id matches the recipient’s account range)</blockquote><blockquote>This precision means partners see exactly what they are entitled to see — no more, no less — without needing separate filtered exports.</blockquote><p><strong>04 Cost Savings: Where the Money Goes</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/807/1*TAo3mmB4irlH2ECtuUZQcg.png" /></figure><p>Delta Sharing generates cost savings across multiple dimensions. The following breakdown reflects typical enterprise scenarios; actual savings vary by data volume, number of sharing relationships, and existing infrastructure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/817/1*HJ2d-CS7VIXuPoLaC1FvuQ.png" /></figure><p><strong>Illustrative 3-Year TCO Example</strong></p><p>A large retailer sharing inventory data with 20 supplier partners: Traditional approach — 20 replication pipelines, 20 destination buckets (3 TB each = 60 TB additional storage), 4 engineers. Estimated 3-year cost: $2.8M in infrastructure + labor. With Delta Sharing — zero replication storage, zero replication pipelines, 0.5 FTE governance overhead. Estimated 3-year cost: $320K. Net saving: ~$2.5M over 3 years.</p><h3><strong>05 Real-World Scenarios</strong></h3><p><strong>Scenario A: Retail Chain Sharing Inventory with Supplier Network</strong></p><blockquote><strong>Problem: </strong>20 suppliers require weekly inventory snapshots. Current setup: 20 SFTP jobs, 20 CSV exports, frequent failures, suppliers always working with stale data.</blockquote><blockquote><strong>Delta Sharing Solution: </strong>Provider creates one Delta Share on the inventory table, partitioned by supplier_id. Each supplier gets a credential and reads their partition via Pandas or Power BI. Data refreshes in real time.</blockquote><blockquote><strong>Business Result: </strong>Supplier order accuracy improves 18% due to fresher data. Engineering saves 2 FTEs previously dedicated to SFTP maintenance. Storage cost drops by 60 TB.</blockquote><p><strong>Scenario B: Healthcare Network Sharing Clinical Data with Research Partners</strong></p><blockquote><strong>Problem: </strong>3 university research partners need de-identified patient outcome data. Sharing via replicated databases creates HIPAA compliance complexity and breach risk from duplicate PII stores.</blockquote><blockquote><strong>Delta Sharing Solution: </strong>Provider shares a Unity Catalog view with PHI columns masked or excluded. Row-level filters ensure each research partner sees only the cohorts they are authorized for. Every access is audit-logged with full lineage.</blockquote><blockquote><strong>Business Result: </strong>Compliance audit time reduced by 50%. Breach risk reduced: zero additional copies of sensitive data exist outside the source system. Partner researchers gain self-service access to current data.</blockquote><p><strong>Scenario C: Financial Services Firm Sharing Market Data with Clients</strong></p><blockquote><strong>Problem: </strong>Clients subscribe to daily market data feeds delivered as flat files via email or SFTP. Files are large, delivery is unreliable, and clients want intraday freshness.</blockquote><blockquote><strong>Delta Sharing Solution: </strong>Provider creates a Delta Share on the market data table. Clients connect their preferred tool (Python, Excel via Power Query, or their own Spark cluster) and query on demand. Intraday commits are immediately available.</blockquote><blockquote><strong>Business Result: </strong>Client satisfaction scores increase. Provider engineering team reclaims 1,200 hours/year previously spent managing file delivery. New intraday data product launched at zero marginal infrastructure cost.</blockquote><p><strong>06 Getting Started: What Organizations Need to Do</strong></p><p>Adopting Delta Sharing is straightforward for organizations already on Databricks. The primary prerequisite is enabling Unity Catalog, which also delivers significant standalone governance benefits.</p><h3>Prerequisites</h3><blockquote>• Databricks workspace with Unity Catalog enabled</blockquote><blockquote>• Delta tables registered in Unity Catalog (existing Hive tables can be migrated)</blockquote><blockquote>• CREATE SHARE and CREATE RECIPIENT privileges assigned to data owners</blockquote><blockquote>• Recipients identified and onboarded (Databricks metastore ID or email for open sharing)</blockquote><h3>Rollout Approach</h3><blockquote>1. Audit current sharing relationships: list all active replication pipelines, destination buckets, and partner credentials.</blockquote><blockquote>2. Prioritize high-value, high-cost relationships: focus first on the pipelines consuming the most engineering time or storage.</blockquote><blockquote>3. Create shares and recipients in Unity Catalog for the top 3–5 sharing relationships.</blockquote><blockquote>4. Run Delta Sharing in parallel with existing pipelines initially, validating data consistency with recipients.</blockquote><blockquote>5. Decommission replication pipelines and destination buckets once recipients confirm Delta Sharing meets their needs.</blockquote><blockquote>6. Expand to remaining sharing relationships and onboard new partners directly via Delta Sharing.</blockquote><p><strong>Key Takeaway</strong></p><p><em>Delta Sharing is not an incremental improvement to the existing data replication model. It is a structural replacement that eliminates the cost, risk, and complexity of copying data at the source. Organizations that fully adopt it for external data sharing typically reclaim millions of dollars in infrastructure spend and dozens of engineering person-years — and give their partners better, fresher data in the process.</em></p><p>For more information on implementing Delta Sharing in your organization, refer to the Databricks Unity Catalog documentation and the open-source Delta Sharing protocol specification</p><p><a href="https://www.databricks.com/product/delta-sharing">Databricks Delta Sharing — Data Sharing | Databricks</a></p><p>Document prepared April 2025 · Databricks Platform Engineering</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=04bfe2de4fe0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Architecture: The Unglamorous Truth About Backfilling Historical Data]]></title>
            <link>https://medium.com/@prasad.patsa/the-unglamorous-truth-about-backfilling-historical-data-7ffed4cb55b2?source=rss-b740d4e2a5b7------2</link>
            <guid isPermaLink="false">https://medium.com/p/7ffed4cb55b2</guid>
            <dc:creator><![CDATA[Durga Prasad Patsa]]></dc:creator>
            <pubDate>Fri, 20 Mar 2026 13:43:09 GMT</pubDate>
            <atom:updated>2026-04-15T13:42:31.813Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Nobody warns you how messy it gets across clouds. Here is what actually works</blockquote><p>It always starts with the same Slack message. Something like: “Hey, can we just reprocess the last 18 months of events? The business logic changed.” You nod. You open your laptop. And then you remember exactly how complicated your data platform actually is.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FiXo0Ekj6qxKNXHiAc5EuQ.png" /></figure><p>Backfilling historical data in Databricks sounds deceptively simple on paper. You have the source data. You have the pipeline code. You just… run it again. But the moment you factor in cloud storage semantics, partition pruning, Unity Catalog permissions, incremental checkpoints, and the fact that your AWS S3 bucket has a slightly different folder layout than your Azure Data Lake, you realize you are not reprocessing data. You are negotiating with infrastructure.</p><p>I have done this enough times now across Azure, AWS, and GCP to have a few hard-won opinions. This is not a documentation dump. This is what I would tell a colleague over coffee before they walked into a backfill project blind.</p><p><em>A backfill is not a re-run. It is a controlled rewrite of your data history, and it deserves its own playbook.</em></p><h3>First, Let’s Agree on What Backfilling Actually Means</h3><p>There are at least three different things people call a “backfill” and they all have different failure modes:</p><p>- <strong>Full historical reload: </strong>Scrapping everything in a range and regenerating it from source. The most thorough and the most dangerous if your source data is mutable.</p><p>- <strong>Incremental gap fill: </strong>You missed some dates, or a job failed silently, and you need to plug specific holes. More surgical, but requires you to know exactly where the holes are.</p><p>- <strong>Schema evolution backfill: </strong>Your pipeline added a new column or changed business logic, and you need old records to reflect the new semantics. This is the one that really hurts.</p><p>Most engineers conflate all three until they are halfway through and realize the approach they chose does not fit the problem they have. The first question to ask your stakeholder is always: do you want the data corrected, or do you want the history to look like the new logic always existed?</p><p>💡 <strong>The two-table pattern: </strong>Keep a _current and a _history table. Backfill _history with corrected data, but never overwrite _current until validation passes. You can roll back. Always give yourself an exit.</p><h3>The Delta Lake Foundation (This Part Applies Everywhere)</h3><p>Before we get cloud-specific, there is a Databricks-level foundation that applies whether your storage is ADLS, S3, or GCS. Delta Lake is doing a lot of heavy lifting here, and understanding it saves hours of confusion.</p><h3>MERGE is your friend, until it isn’t</h3><p>The standard approach for a schema-evolution backfill is a MERGE statement. You write your corrected data to a staging table, then merge it into the target on a key:</p><p><strong>Python / PySpark — Delta MERGE pattern</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2tOwtQfBXQomMTOrBJBrhg.png" /></figure><p>The danger is table size. MERGE on a 500-billion-row table without partition pruning in the ON clause is a full table scan in disguise. Always include a date or partition column in the join condition, even if it feels redundant. Delta will use it to skip files.</p><h3>The RESTORE command nobody uses enough</h3><p><strong>RESTORE</strong> is one of the most underrated features in Delta Lake. If you are doing a full-range reload, consider snapshotting the current state of the table before you touch anything:</p><p><strong>SQL — snapshot before you rewrite</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6bdwSiCa5CvuPHgXcgvtUA.png" /></figure><p>The RESTORE is instant for Delta tables backed by transaction logs. It is not a copy. The moment something goes sideways in a backfill at 2am, you will thank past-you for noting the version number.</p><h3>Partition your backfill batches, always</h3><p>Never try to backfill 18 months in a single job. Always loop by month or week, and checkpoint your progress explicitly. Databricks clusters die. Network timeouts happen. Cloud storage throttles. You want to pick up from where you left off, not from zero.</p><p><strong>Python — date-batched backfill loop with checkpointing</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RmApXkcXZDVB2COE00L6dQ.png" /></figure><p><strong>☁ Azure: ADLS Gen2 + Unity Catalog</strong></p><p>Azure is where most of my backfill scars come from, partly because Azure Databricks was where I spent the most time, and partly because the Unity Catalog permission model adds a layer of nuance that catches people off guard.</p><h3>External locations and the permissions trap</h3><p>If your target table sits on an external location in Unity Catalog, your backfill job identity needs explicit WRITE FILES permission on that location, not just table-level INSERT. This is different from the legacy mount-based world, and the error message when you get it wrong is unhelpfully vague:</p><p><strong>SQL — grant what the backfill job actually needs</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rXbk_wHqZtwoGZdQzpBgWQ.png" /></figure><p>I usually create a short-lived backfill service principal or group, grant it the minimum permissions, and revoke them when the job is done. The least-privilege instinct that feels like overkill in dev saves you an audit conversation in prod.</p><h3>ADLS Gen2 soft delete: the hidden performance killer</h3><p>Azure ADLS Gen2 has a soft-delete feature that sounds like a safety net. During a large backfill that is doing lots of small file rewrites, it becomes a performance anchor. Every overwrite of a Delta transaction log file creates a soft-deleted version. Over 18 months of daily partitions, you can accumulate tens of thousands of soft-deleted blobs.</p><p>If you are doing a heavy backfill in Azure, temporarily disable soft delete on the storage account for the duration of the job, and re-enable it after. Or at minimum, make sure your VACUUM commands run aggressively afterward:</p><p><strong>SQL — vacuum and optimize after backfill</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eLz-vUgmZ0v4uYt0hXmQMg.png" /></figure><p>⚠️ <strong>Production caution: </strong>RETAIN 0 HOURS removes all old files immediately. Only use this during a maintenance window when you have confirmed no concurrent readers and you have tested RESTORE to the prior version.</p><h3>Key Vault secrets and job rotation</h3><p>If your backfill job runs for days (and multi-month backfills often do), be aware that Azure service principal client secrets have a maximum lifetime. If your job authenticates via a short-lived secret and that secret expires mid-run, the job dies silently in a way that looks like a storage permission error, not an auth error.</p><p>Two ways to avoid this: use managed identity for the compute cluster (the preferred path), or set your secret rotation to be longer than your estimated backfill window and rotate after the job completes.</p><p><strong>🟠 AWS: S3 + Glue Catalog + IAM</strong></p><p>AWS has a different texture of problems. S3 is extremely reliable and fast, but its eventually consistent metadata model (before the strong consistency update) left a generation of engineers with superstitious behaviors around add-sleep-after-write that persist to this day. The bigger gotchas now are IAM, the Glue catalog, and S3 request rate limits under heavy writes.</p><h3>IAM instance profiles vs. credential passthrough</h3><p>On Databricks on AWS, there are two ways your cluster gets credentials: instance profile attached to the EC2 nodes, or Unity Catalog credential passthrough. These behave differently during a backfill at scale, and mixing them up creates permission errors that are hard to diagnose.</p><p>The cleanest pattern I have found is a dedicated IAM role for backfill jobs with an inline policy scoped to exactly the S3 paths you need to write:</p><p><strong>JSON — minimal IAM policy for a backfill job role</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/950/1*FyDI94R9Ib5qWjaCznPjxw.png" /></figure><p>Attach this role as the instance profile for your backfill cluster. When the job finishes, detach it. This keeps your blast radius small if something goes wrong.</p><h3>S3 request rate limits are real</h3><p>S3 can handle thousands of requests per second per prefix, but a Databricks cluster with 20 workers doing small-file writes across the same S3 prefix during a backfill can hit those limits faster than you expect. You will see SlowDown errors in the Spark UI disguised as intermittent task failures.</p><p>Two things help here. First, enable S3 prefix partitioning in your table layout if you have not already (year= / month= / day= prefixes distribute the request load across S3’s internal sharding). Second, set the following Spark configs on your backfill cluster to slow down the small-file problem at the source:</p><p><strong>Databricks cluster config — reduce S3 write pressure</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yCDHI37JBWxAPQ6Tfg7nrw.png" /></figure><h3>Glue catalog and the schema sync problem</h3><p>If your AWS environment uses the Glue Data Catalog as the metastore (common in older Databricks on AWS setups before Unity Catalog), schema changes introduced by your backfill do not propagate to Glue automatically. After a backfill that adds columns or changes partition structure, you need to explicitly sync:</p><p><strong>Python — force Glue schema refresh after backfill</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OA1E_RUmaO7ycq_jefq0UA.png" /></figure><p><em>S3 is not a filesystem. The sooner you stop treating it like one in your backfill code, the fewer mysterious failures you will debug at midnight.</em></p><p><strong>🔵 GCP: GCS + BigQuery + Hive Metastore</strong></p><p>GCP is the most pleasant cloud to do a backfill on, in my experience, largely because GCS is remarkably consistent and the IAM model is less arcane than AWS. The challenges tend to be organizational (GCP workspaces often exist alongside BigQuery-heavy stacks, so there is always a political question of where the canonical data lives) and metastore-related.</p><h3>Service accounts and Workload Identity</h3><p>On Databricks on GCP, your cluster nodes authenticate to GCS via a service account. For backfills, I recommend creating a dedicated backfill service account with a distinct set of GCS bucket permissions, separate from your production read/write service account:</p><p><strong>Bash — GCP service account setup for backfill</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7JysqPD2oQ5avWzjM_SUqQ.png" /></figure><p>The condition expression in the IAM binding is underused and extremely powerful. It gives you path-level access control without a separate bucket, which keeps your bucket structure clean.</p><h3>GCS composite objects and Delta log conflicts</h3><p>GCS has a concept called <strong>composite objects</strong> — objects assembled from multiple components using the compose API. Some older versions of the Delta-GCS connector created these, and they cause silent read errors when you try to time-travel to a version that was written during the composite-object era. If your historical data predates 2022 on GCP, check your Delta logs:</p><p><strong>Python — detect composite object issues in Delta history</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e-ijpEr5ZND22CJajibvFw.png" /></figure><h3>BigQuery interop during a backfill</h3><p>The awkward reality of GCP Databricks setups is that BigQuery often holds the source of truth for some data while Databricks holds it for others. During a backfill, you may need to pull from BigQuery to recompute something in Databricks, or push results back.</p><p>The Spark BigQuery connector is good but has its own consistency rules. Never read from a BigQuery table that is actively being loaded during your backfill window. The connector will silently return a stale snapshot. Use export-to-GCS-first, then read from GCS, rather than direct connector reads for large historical pulls:</p><p><strong>Python — safe BigQuery export before backfill read</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/1*W_wCqrZKj17eEUlmQL47Ew.png" /></figure><h3>Cross-Cloud Backfill Comparison Cheat Sheet</h3><p>For those who work across clouds or are choosing where to run a particularly painful backfill, here is an honest summary:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7Hj3_eC6B4Lmz90kOrP7WQ.png" /></figure><h3>The Operational Stuff Nobody Puts in the Docs</h3><p>Beyond the technical patterns, there are a handful of practices that make the difference between a backfill that completes cleanly and one that turns into a week-long incident.</p><h3>Tell the consumers before you start</h3><p>If your gold-layer tables are actively read by dashboards or downstream jobs, a multi-week backfill that overwrites data mid-flight will produce confusing results for those consumers. A lot of engineer-to-engineer trust is lost in these moments. Send a message to #data-consumers or whatever the equivalent is, before you start.</p><h3>Run a dry-run against a one-week slice first</h3><p>Before committing to 18 months, take a single representative week from the middle of the range and run the complete backfill pipeline against it. Validate row counts, check for duplicate keys, spot-check business metrics. If it fails or produces wrong results on one week, it will fail or produce wrong results at scale.</p><h3>Budget your cluster differently</h3><p>Backfill clusters have a different profile than production pipelines. Production jobs are usually I/O-bound. Backfills are often both I/O and CPU-bound because you are running complex transformations over large historical datasets all at once. Consider increasing the memory-to-core ratio on your backfill cluster. Spot or preemptible instances are fine for most backfill work as long as your checkpoint pattern handles restarts gracefully.</p><h3>Monitor file counts, not just row counts</h3><p>A successful backfill that produces millions of tiny Parquet files is not a successful backfill. It is a ticking time bomb for the next query that needs to scan that table. Always run OPTIMIZE after each batch completes:</p><p><strong>SQL — check file health after each batch</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/946/1*dAkEDd8SCWkJ68E-MmgK1A.png" /></figure><p>📝 <strong>Rule of thumb: </strong>If DESCRIBE DETAIL shows more than 10,000 files on a table you can query in under a minute, your small file problem is already affecting performance. Backfills make this worse before OPTIMIZE makes it better.</p><h3>Closing Thoughts</h3><p>Backfilling data is one of those tasks that reveals the true architecture of your data platform. It exposes assumptions you baked in two years ago when the tables were small. It surfaces permission models that only make sense when someone is writing, not just reading. It forces you to think about what “correct” data history actually means for your business.</p><p>The engineers I have seen do this well all share one trait: they treat the backfill as a first-class project, not a weekend task. They write a plan. They test a slice. They communicate with consumers. They checkpoint everything. They clean up after themselves.</p><p>The engineers I have seen struggle with it all share the opposite trait: they assumed it would be “just a re-run.”</p><p><em>There is no such thing as ‘just a re-run’ when the data is already in production.</em></p><p>Good luck out there. May your Delta logs be consistent and your RESTORE commands swift.</p><p><strong>#Databricks</strong> #<strong>DeltaLake</strong> #<strong>DataEngineering</strong> #DataArchitecture #<strong>Azure</strong> #<strong>AWS</strong> #<strong>GCP</strong> #History<strong>Backfill</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7ffed4cb55b2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Architecture: From Data Platforms to Intelligence Platforms: Reimagining Data Architecture for…]]></title>
            <link>https://medium.com/@prasad.patsa/from-data-platforms-to-intelligence-platforms-reimagining-data-architecture-for-the-ai-era-c4e57ad9f0f9?source=rss-b740d4e2a5b7------2</link>
            <guid isPermaLink="false">https://medium.com/p/c4e57ad9f0f9</guid>
            <dc:creator><![CDATA[Durga Prasad Patsa]]></dc:creator>
            <pubDate>Mon, 16 Mar 2026 16:20:41 GMT</pubDate>
            <atom:updated>2026-04-15T13:43:12.586Z</atom:updated>
            <content:encoded><![CDATA[<h3>Data Architecture: From Data Platforms to Intelligence Platforms: Reimagining Data Architecture for the AI Era</h3><p>For a long time, data architecture was treated like plumbing. We built pipes, moved records, cleaned tables, and delivered reports. It worked well for a world where most business questions were answered after the fact: what happened last quarter, which region underperformed, how many orders were delayed. The architecture was designed for hindsight.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3_-ypmZ5JeCO4ChNmLPdVg.png" /></figure><p>Artificial intelligence changes that expectation completely. Today, the business does not only want hindsight. It wants decisions, recommendations, predictions, and actions in near real time. That subtle shift in expectation creates a deep shift in architecture. We are no longer only building data platforms. We are building intelligence platforms.</p><p>In the old model, teams often celebrated successful pipeline completion. In the AI-first model, success is not “pipeline finished at 2 AM.” Success is “the right decision was made at the right time with the right confidence.” This sounds simple, but it changes almost every design choice.</p><p>The first change is how we think about data ownership. In traditional environments, data was often collected as a by-product of applications and then handed over to a central team to sort out later. In an AI world, that creates chaos quickly. Models and agents consume data continuously, and if definitions are unclear or quality is inconsistent, they fail quietly before anyone notices. So data has to be treated as a product. It needs an owner, a contract, quality standards, and accountability. Not because it is fashionable, but because AI systems are unforgiving to ambiguity.</p><p>The second change is context. Humans can guess meaning from messy labels because they carry domain knowledge in their heads. Models cannot do that reliably. If one table calls something “customer,” another calls it “account,” and a third uses a coded ID with no business glossary, your AI system will produce inconsistent reasoning. This is why semantic architecture becomes central: not a documentation exercise, but a functional layer that gives meaning, relationships, and common vocabulary. In practice, this is often the difference between an AI assistant that sounds fluent and one that is actually useful.</p><p>Then comes the speed challenge. Many enterprises still run on batch assumptions. Data lands nightly, transforms in windows, and appears in dashboards the next morning. For BI this can be acceptable. For AI-driven operations, it can be too late. If your fraud signal arrives six hours late, or your predictive maintenance signal comes after failure, the intelligence is technically correct but operationally useless. AI-first architecture does not eliminate batch, but it does stop pretending batch is enough. It embraces dual-speed design: real-time for action, batch for depth and reconciliation.</p><p>Quality and trust become non-negotiable in this transition. In classic architectures, quality checks were sometimes treated as a stage gate. In AI architectures, quality has to be continuous. Data drift, schema drift, freshness issues, and semantic drift all need active monitoring. A model can remain “healthy” from an infrastructure point of view while becoming functionally wrong due to subtle data changes upstream. That is why observability must evolve from “is the job green?” to “is the decision substrate still reliable?”</p><p>Security also changes shape. In the past, many controls were built around human users and static reporting roles. AI systems introduce machine identities, tool calls, autonomous retrieval paths, and composite contexts. That means your old perimeter assumptions are not enough. Governance must become policy-driven and automated. Access controls have to be enforced not just at database login, but at field level, purpose level, and workflow level. If an agent can reach a tool, that tool must enforce policy by design, not by convention.</p><p>Another major shift is retrieval architecture. Traditional thinking asked: “Which single platform should hold all enterprise data?” AI-first thinking asks a more useful question: “Which retrieval pattern best serves this decision?” Some answers need strict SQL accuracy. Some need semantic similarity from vectors. Some require relationship traversal like a graph. Some need operational traces from logs. The winning architecture is rarely one engine. It is an orchestrated retrieval fabric.</p><p>As this becomes operational, old organizational boundaries start to break down. DataOps, MLOps, and now AgentOps cannot remain separate silos for long. If data quality drifts, model behavior shifts. If retrieval changes, agent reasoning changes. If prompts evolve without data context controls, risk increases. The lifecycle has to be unified: data versions, model versions, prompt/tool versions, evaluations, and rollback paths all connected. Without this, teams spend their time arguing over ownership of incidents instead of improving system quality.</p><p>And then there is the human question. AI-first does not mean human-less. In reality, responsible organizations are designing calibrated autonomy: let low-risk decisions run automatically, route medium-risk decisions for approval, and keep high-risk decisions under human authority with AI support. Architecture must enable this, not fight it. That means preserving evidence, confidence signals, provenance, and feedback loops. The strongest systems are not the most automated. They are the ones that know when to ask for human judgment.</p><p>This is the true change of gear. We are moving from static repositories of historical truth to dynamic systems of contextual decision support. We are moving from “data delivered” to “outcomes improved.” We are moving from architecture that stores information to architecture that powers intelligence safely.</p><p>If you look at this through the lens of the eight pillars of modern data architecture, the message is clear. Productized ownership, semantic grounding, real-time readiness, trust observability, policy automation, hybrid retrieval, unified operations, and human-in-loop governance are no longer optional enhancements. Together, they define whether AI in the enterprise becomes a reliable capability or a fragile experiment.</p><p>The organizations that thrive in this era will not be the ones with the biggest platforms. They will be the ones that build architectures that are understandable, governable, adaptable, and trusted by both humans and machines. That is the future-facing architecture mindset: not data at rest, but intelligence in motion.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c4e57ad9f0f9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Architecture: How to configure databricks workspace in Azure using Key vaults and Service…]]></title>
            <link>https://medium.com/@prasad.patsa/how-to-configure-databricks-workspace-in-azure-using-key-vaults-and-service-principles-5b37b59b2246?source=rss-b740d4e2a5b7------2</link>
            <guid isPermaLink="false">https://medium.com/p/5b37b59b2246</guid>
            <category><![CDATA[keyvault]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[azure-databricks]]></category>
            <category><![CDATA[data-security]]></category>
            <category><![CDATA[secrets]]></category>
            <dc:creator><![CDATA[Durga Prasad Patsa]]></dc:creator>
            <pubDate>Tue, 24 Feb 2026 00:21:24 GMT</pubDate>
            <atom:updated>2026-04-15T13:43:32.070Z</atom:updated>
            <content:encoded><![CDATA[<h3>Data Architecture: How to configure databricks workspace in Azure using Key vaults and Service principles</h3><p><strong>Purpose: </strong>This guide provides step-by-step instructions for setting up a complete Databricks workspace on Azure with proper security, authentication, and configuration management. After following this guide, you’ll have a production-ready Databricks environment that can securely access Azure Storage and Azure Key Vault. Target Audience: Engineers with basic Azure knowledge but no prior Databricks experience.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gmJXAa6gcTKvp9VL9GHJMQ.png" /></figure><p>Table of Contents</p><p>1. Architecture Overview</p><p>2. Prerequisites</p><p>3. Phase 1: Azure Resource Setup</p><p>4. Phase 2: Service Principal Configuration</p><p>5. Phase 3: Key Vault Setup</p><p>6. Phase 4: Databricks Workspace Setup</p><p>7. Phase 5: Secret Scope Configuration</p><p>8. Phase 6: Storage Authentication Setup</p><p>9. Phase 7: Code Implementation</p><p>10. Phase 8: Testing and Validation</p><p>11. Troubleshooting</p><p>12. Quick Reference Table</p><h3>1. Architecture Overview</h3><p>Before touching the Azure Portal, it is important to understand how the components relate to each other. A secure Databricks deployment on Azure is not a single service — it is an orchestration of four distinct Azure services working together to provide authentication, secrets management, data access, and compute.</p><p>The core security principle is zero hardcoded credentials. Instead of embedding passwords or connection strings directly in notebooks or cluster configs, Databricks fetches secrets at runtime from Azure Key Vault through a mechanism called a Secret Scope. The identity that does the fetching is a Service Principal — a non-human Azure AD identity you control.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1015/1*UATWwrqNg-LjlWrkhMimoA.png" /></figure><p>The flow works as follows: your Databricks notebook calls <strong>dbutils.secrets.get() </strong>with a scope and key name. Databricks resolves the scope to a Key Vault, fetches the secret using its managed identity or the linked Service Principal credentials, and returns the value in memory — never persisted to disk or logs. That secret (typically a Client Secret) is then used to authenticate the Service Principal against Azure Storage or other services.</p><h3>2. Prerequisites</h3><p>Before beginning, ensure the following access levels are available. Missing any of these will block one or more phases of the setup.</p><p>• <strong>Azure Access: </strong>Azure Subscription with Owner or Contributor role at the subscription or resource group level</p><p>• <strong>Azure EntraID Permissions: </strong>Azure Active Directory permissions to create App Registrations and assign roles (Application Administrator or Global Administrator)</p><p>• <strong>Key Vault: </strong>Ability to create Azure Key Vault resources and manage Access Policies</p><p>• <strong>Databricks: </strong>Azure Databricks workspace creation rights, or an existing workspace to configure</p><p>• <strong>Storage: </strong>Azure Storage Account (ADLS Gen2) already created or permission to create one</p><h3>3. Phase 1: Azure Resource Setup</h3><p>The first phase establishes the foundational Azure infrastructure. All subsequent components live within or depend on these resources. If you already have a Resource Group and Databricks Workspace, you may skip ahead to Phase 2.</p><p><strong>Resource Group</strong></p><p>A Resource Group is a logical container for related Azure resources. Create a dedicated resource group for your Databricks environment to simplify access control, cost tracking, and cleanup. Use a consistent naming convention such as <strong>rg-databricks-prod</strong> or <strong>rg-data-platform-uat</strong>.</p><p><a href="https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/manage-resource-groups-portal">Use the Azure portal and Azure Resource Manager to Manage Resource Groups — Azure Resource Manager | Microsoft Learn</a></p><p><strong>Azure Databricks Workspace</strong></p><p>The Databricks Workspace is your primary working environment. When creating it, select the Premium tier — this is mandatory if you intend to use Azure Active Directory passthrough authentication or fine-grained access controls. The Standard tier does not support Secret Scopes backed by Azure Key Vault. Choose the same Azure region as your Key Vault and Storage Account to avoid cross-region latency and data egress charges.</p><p><a href="https://docs.databricks.com/aws/en/admin/workspace/">Create a workspace | Databricks on AWS</a></p><p><strong>Azure Storage Account (ADLS Gen2)</strong></p><p>Create a Storage Account with Hierarchical Namespace enabled — this is what makes it ADLS Gen2 rather than standard Blob Storage. Hierarchical namespace is required for directory-level permissions and is the expected format for most Databricks data lake patterns. Within the storage account, create a container (equivalent to a filesystem root) that your Databricks notebooks will read from and write to.</p><p><a href="https://learn.microsoft.com/en-us/azure/storage/">Azure Storage Documentation Hub | Microsoft Learn</a></p><h3>4. Phase 2: Service Principal Configuration</h3><p>A Service Principal is Azure’s equivalent of a service account. It is an identity that represents your application — in this case, the Databricks workspace — when it needs to authenticate to other Azure services. Using a Service Principal instead of a user account means access is not tied to any individual’s employment status, password changes, or MFA challenges.</p><p><strong>Creating the App Registration</strong></p><p>In Azure Active Directory, navigate to App Registrations and create a new registration. The name should clearly identify its purpose, such as sp-databricks-prod. After registration, you will immediately see two critical values on the overview page: the Application (Client) ID and the Directory (Tenant) ID. Copy both — they are used in every authentication call Databricks makes on behalf of this identity.</p><p><a href="https://learn.microsoft.com/en-us/entra/identity-platform/app-objects-and-service-principals?tabs=browser">Apps &amp; service principals in Microsoft Entra ID — Microsoft identity platform | Microsoft Learn</a></p><p><strong>Generating the Client Secret</strong></p><p>Under Certificates &amp; Secrets, create a new Client Secret. This is the password equivalent for the Service Principal. Azure will show the secret value exactly once — the moment it is created. If you navigate away without copying it, you must delete and regenerate it. Set an expiry period that aligns with your organization’s secret rotation policy, typically 12 or 24 months. Store this value immediately in Key Vault as described in Phase 3.</p><p><strong>Role Assignments</strong></p><p>The Service Principal needs explicit permission to access Azure resources. At minimum, assign it the Storage Blob Data Contributor role on your ADLS Gen2 storage account. This is done through the storage account’s IAM (Access Control) blade. Without this role, authentication will succeed but all storage operations will return 403 Forbidden errors — a common source of confusion during initial setup.</p><h3>5. Phase 3: Key Vault Setup</h3><p>Azure Key Vault is a managed secrets store. It handles encryption at rest, audit logging, access control, and secret versioning automatically. The goal of this phase is to store your Service Principal’s Client Secret in Key Vault so that Databricks can retrieve it at runtime without anyone needing to handle the raw credential.</p><p><a href="https://learn.microsoft.com/en-us/azure/key-vault/">Azure Key Vault documentation | Microsoft Learn</a></p><p><strong>Creating the Key Vault</strong></p><p>When creating the Key Vault, ensure it is in the same region as your other resources. Choose the Standard pricing tier for most use cases. Notably, enable Soft Delete and Purge Protection — these are now required by Microsoft and prevent accidental permanent deletion of secrets, which could break production pipelines.</p><p><strong>Storing the Service Principal Secret</strong></p><p>Navigate to Secrets within the Key Vault and create a new secret. Use a descriptive name such as sp-databricks-client-secret or databricks-adls-secret. Paste the Client Secret value you copied during Phase 2. The name you choose here becomes the key you reference in Databricks notebooks via dbutils.secrets.get().</p><p><strong>Access Policies</strong></p><p>By default, nothing can read from a Key Vault — not even the subscription owner. Access must be explicitly granted. Add an Access Policy for the Service Principal you created in Phase 2, granting it Get and List permissions on Secrets. This is what allows Databricks (acting as the Service Principal) to read the secrets stored in the vault.</p><p>Additionally, if you are configuring a Databricks-backed Secret Scope (described in Phase 5), you may also need to add the Azure Databricks Enterprise Application as a reader. This is a first-party Microsoft application that acts as the bridge between Databricks Secret Scopes and Key Vault.</p><h3>6. Phase 4: Databricks Workspace Setup</h3><p>With the Azure-side infrastructure in place, the next phase focuses on the Databricks workspace itself — creating a cluster with appropriate configuration and ensuring the environment is prepared for secure operations.</p><p><strong>Launching the Workspace</strong></p><p>From the Azure Portal, open your Databricks resource and click Launch Workspace. This opens the Databricks UI in a new tab. Bookmark this URL — it is your workspace’s permanent address. The URL format is typically adb-&lt;workspace-id&gt;.&lt;region&gt;.azuredatabricks.net.</p><p><strong>Creating a Cluster</strong></p><p>A Cluster is the compute resource that runs your notebooks and jobs. Create an All-Purpose Cluster for development and testing. Key configuration decisions include: the Databricks Runtime version (choose an LTS release for stability), the VM size (Standard_DS3_v2 is a cost-effective starting point), and Auto Termination (set this to 30–60 minutes to avoid idle cost accumulation).</p><p>For production workloads, you will eventually separate All-Purpose Clusters (used by interactive notebooks) from Job Clusters (ephemeral clusters spun up and torn down for each automated job run). Job Clusters are significantly cheaper for batch processing.</p><p><a href="https://docs.databricks.com/aws/en/admin/workspace/">Create a workspace | Databricks on AWS</a></p><h3>7. Phase 5: Secret Scope Configuration</h3><p>A Secret Scope is a named group of secrets that Databricks can access. When backed by Azure Key Vault, the scope acts as a proxy — Databricks never stores the actual secret values; it retrieves them from Key Vault each time they are requested. This means revoking access at the Key Vault level immediately blocks Databricks, and all secret rotation in Key Vault is automatically reflected in Databricks.</p><p><strong>Creating the Scope</strong></p><p>Secret Scope creation is done through a special URL — not through the standard Databricks UI menus. Append #secrets/createScope to your workspace URL. For example: <a href="https://adb-1234567890.12.azuredatabricks.net/#secrets/createScope">https://adb-1234567890.12.azuredatabricks.net/#secrets/createScope</a></p><p>You will be prompted for three values: a Scope Name (a logical label you choose, e.g., prod-keyvault), the DNS Name of your Key Vault (found in Key Vault → Overview, format: <a href="https://your-vault-name.vault.azure.net/),">https://your-vault-name.vault.azure.net/),</a> and the Resource ID of your Key Vault (found in Key Vault → Properties, a long ARM resource path).</p><p><strong>Accessing Secrets in Notebooks</strong></p><p>Once the scope is created, any notebook running on a cluster with appropriate permissions can retrieve secrets using a single line: dbutils.secrets.get(scope=’prod-keyvault’, key=’sp-databricks-client-secret’). The returned value is a string held in memory and will never appear in notebook output, logs, or Spark UI — Databricks redacts secret values automatically and displays [REDACTED] if they are accidentally printed.</p><p><a href="https://docs.databricks.com/aws/en/security/secrets/">Secret management | Databricks on AWS</a></p><h3>8. Phases 6–8: Storage Auth, Code &amp; Testing</h3><p><strong>Phase 6: Storage Authentication</strong></p><p>To connect Databricks to ADLS Gen2, configure OAuth 2.0 authentication using the Service Principal. This involves setting three Spark configuration properties on the cluster or at the start of a notebook: the OAuth endpoint (using your Tenant ID), the Client ID, and the Client Secret (retrieved via Secret Scope). Once configured, Databricks uses these credentials transparently for all storage operations against ADLS.</p><p><strong>Phase 7: Code Implementation</strong></p><p>With authentication configured, notebooks access storage using standard Spark APIs. The recommended pattern is to mount the ADLS container to a Databricks File System (DBFS) path using dbutils.fs.mount(). After mounting, all reads and writes use familiar path syntax (/mnt/your-mount-name/folder/file.parquet) rather than long ADLS URIs. Mounts persist across cluster restarts and are available to all users in the workspace with appropriate permissions.</p><p><strong>Phase 8: Testing and Validation</strong></p><p>Validation should confirm three things independently: that the Secret Scope can retrieve the Client Secret (test with dbutils.secrets.get()), that the Service Principal can authenticate to ADLS (test with a simple dbutils.fs.ls() on the mount point), and that read and write operations succeed end-to-end (write a small test file, read it back, delete it). Check Azure Key Vault diagnostic logs to confirm access events are being recorded — this is your audit trail.</p><h3>11. Troubleshooting</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oT1hLtJ8XQcDuL-r-coA0g.png" /></figure><h3>12. Quick Reference: Where to Do What</h3><p>Use this table as a checklist during your setup. Each phase maps to a specific Azure service or tool, and can be completed largely independently once its prerequisites are met.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/1*Af_pN3GjFl4N_pssBYBCvw.png" /></figure><p><strong><em>Key Principle:</em></strong><em> No credentials should ever appear in notebook code, cluster environment variables, or version control. Every secret flows through Key Vault → Secret Scope →</em><strong><em> dbutils.secrets.get() </em></strong><em>at runtime.</em></p><h3>Conclusion:</h3><p>With this setup, your Databricks environment follows Azure security best practices — no hardcoded credentials, centralized secret management, and a clear audit trail. This pattern scales cleanly across dev, QA, and prod by simply changing Key Vault references and Secret Scope names.</p><h3>Disclaimers:</h3><p><strong>Pricing</strong> — Azure services used in this guide (Databricks Premium, Key Vault, ADLS Gen2) incur costs. Readers should review Azure pricing before deploying to production.</p><p><strong>Version currency</strong> — Azure Portal UI and Databricks features change frequently. The steps reflect the state of the platform at time of writing and may differ slightly in future releases.</p><p><strong>Not official documentation</strong> — This is a community guide, not an official Microsoft or Databricks publication. For production-critical decisions, cross-reference the official docs.</p><p><strong>Security responsibility</strong> — Secret rotation, access policy reviews, and Key Vault audit log monitoring remain the reader’s responsibility post-setup.</p><p>Happy Learning!</p><blockquote>Durga Prasad Patsa, Data Architect</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5b37b59b2246" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>