<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Aditya Bhatraju on Medium]]></title>
        <description><![CDATA[Stories by Aditya Bhatraju on Medium]]></description>
        <link>https://medium.com/@adityabnsemp?source=rss-b86ed7f17786------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*d14JQMiBm5vrog8brB9N7Q.jpeg</url>
            <title>Stories by Aditya Bhatraju on Medium</title>
            <link>https://medium.com/@adityabnsemp?source=rss-b86ed7f17786------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 10:18:07 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@adityabnsemp/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Six Guiding Principles of the Databricks Well-Architected Framework]]></title>
            <link>https://medium.com/@adityabnsemp/six-guiding-principles-of-the-databricks-well-architected-framework-607b0f513f51?source=rss-b86ed7f17786------2</link>
            <guid isPermaLink="false">https://medium.com/p/607b0f513f51</guid>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[wellarchitectedframework]]></category>
            <category><![CDATA[principles]]></category>
            <dc:creator><![CDATA[Aditya Bhatraju]]></dc:creator>
            <pubDate>Fri, 26 Jun 2026 20:00:23 GMT</pubDate>
            <atom:updated>2026-06-26T20:24:42.214Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Every production lakehouse begins with constraints, not technology choices. These six principles define what your architecture must achieve regardless of scale, cloud provider, or organizational structure.</em></p><h3>Why Principles Before Pillars</h3><p>The Databricks Well-Architected Framework defines seven architectural pillars, spanning governance, interoperability, security, reliability, performance, operational excellence, and cost. But in practice, pillars are only as strong as the ground they stand on. Before your teams debate monitoring tools or compute policies, the organization needs shared agreement on what the platform <em>must</em> achieve at a foundational level.</p><p>That is what guiding principles provide. They are not best practices you adopt incrementally, they are architectural constraints that bound every downstream decision. They shape how you structure workspaces, how you grant access, how you share data across teams, and how you decide between building and buying.</p><p>Without this alignment, entropy (chaos) wins. One team clones a dataset into their own schema for convenience. Another chooses a proprietary connector for short-term speed. A third provisions always-on clusters for workloads that run twice a day. Individually, each decision seems rational. Collectively, you end up with a platform that is expensive to run, impossible to govern, and fragile under change.</p><p>The six principles below are an antidote.</p><h3>Principle 1: Curate Data and Offer Trusted Data-as-Products</h3><p>Most organizations do not have a data problem, they have a trust problem. Reports contradict each other because two teams compute revenue differently. Dashboards go stale because nobody owns the underlying pipeline. Data scientists spend 80% of their time wrangling instead of modelling, because they cannot find a clean, reliable version of the data they need.</p><p>The fix is not more data. It is treating data as a product with the same rigour you would apply to a customer-facing API.</p><h3>Progressive Quality Through Layered Architecture</h3><p>The medallion pattern is the structural backbone for this principle:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_74MaaS3QsC6xEILfXZyVw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AFGrkilhGkVz8BBwmwqzoQ.png" /><figcaption>Medallion Architecture Layers</figcaption></figure><p>The critical insight: quality is not a property you check at the end, it is a <em>gradient</em> that improves at each transition. Bronze data is messy but complete. Silver data is clean but general. Gold data is trusted and fit for a specific purpose.</p><h3>What Product Thinking Demands</h3><p>Shipping a table and calling it a “data product” is not enough. True product thinking requires:</p><ul><li><strong>Ownership:</strong> a named team or individual accountable for freshness, accuracy, and uptime. When something breaks at 2 AM, there is no ambiguity about who gets paged.</li><li><strong>Contracts:</strong> versioned schemas with explicit evolution policies. Consumers know what to expect and get advance notice of breaking changes.</li><li><strong>SLAs:</strong> measurable commitments on latency, completeness, and availability. Not aspirational but enforceable.</li><li><strong>Discoverability:</strong> published metadata, documentation, and lineage so consumers can find and evaluate data without filing a ticket.</li></ul><p>When this is done well, data products become the unit of trust in the organization. Teams stop building shadow pipelines because they trust the canonical version. New use cases launch faster because the building blocks already exist.</p><h3>Principle 2: Adopt an Organization-Wide Data and AI Governance Strategy</h3><p>Here is a pattern that plays out in nearly every enterprise: the data engineering team implements access controls in their ETL layer, the analytics team manages permissions through their BI tool, and the ML team governs model artifacts through a separate registry. Three teams, three governance models, zero consistency.</p><p>The result? Nobody can answer two deceptively simple questions: <em>Who can access what?</em> and <em>Who accessed what recently?</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UUuGmh4nuE6IhOsWFOZM7g.png" /></figure><h3>Four Dimensions That Must Work Together</h3><p>Governance is not a single concern, it is four interlocking capabilities:</p><ol><li><strong>Quality governance:</strong> Every pipeline step needs guardrails. Schema validation on ingest. Expectation checks on transformation. Freshness monitoring on published tables. Without these, you are governing access to data that might be wrong, which is arguably worse than no governance at all.</li><li><strong>Catalog governance: </strong>If “customer” means one thing in marketing and another in finance, your governance is already broken regardless of how tight your permissions are. A central catalog enforces semantic consistency: one name, one definition, one lineage trail. Metadata is not documentation, it is infrastructure.</li><li><strong>Access governance: </strong>Build with least-privilege from day one. The teams who say “we will tighten access later” never do, and when they try, they discover that revoking permissions from running workloads is an order of magnitude harder than granting them correctly upfront. Column masks, row filters, and attribute-based policies should be part of your initial design, not a remediation project.</li><li><strong>Audit governance: </strong>Every read, every query, every export must leave a trace. This is non-negotiable for regulated industries, but it is equally important for debugging data quality issues. When a downstream report shows unexpected numbers, lineage and audit logs are how you trace the problem to its source.</li></ol><h3>The Case for a Single Governance Layer</h3><p>Unity Catalog exists to solve the fragmentation problem. One permissions model across tables, views, volumes, models, and functions. One audit trail regardless of whether data was accessed from a notebook, a SQL warehouse, or a serving endpoint. One lineage graph that connects raw ingestion to gold-layer reporting. When governance is unified, the gap between “what should be true” and “what is actually enforced” collapses.</p><h3>Principle 3: Eliminate Data Silos and Minimize Data Movement</h3><p>There is an important distinction that gets lost in data architecture discussions: a copy is not inherently a silo. A developer cloning a table into a sandbox for experimentation? That is healthy. An analyst exporting a snapshot to test a hypothesis in a notebook? That is fine.</p><p>The copy becomes a silo at the moment it acquires downstream dependents, when other teams, dashboards, or models start relying on it as a source of truth. At that point, you have two versions of reality, and they <em>will</em> diverge.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YG6_kN9N6REfsHWGG3pn_Q.png" /></figure><h3>How Silos Compound</h3><p>Once a copy becomes operational, teams build synchronization pipelines to keep it current. These pipelines are where the real damage occurs:</p><ul><li>They introduce latency. The copy is always at least one pipeline execution behind the source.</li><li>They fail silently. A broken sync job means consumers are making decisions on stale data and often do not know it.</li><li>They accumulate logic. Transformations get baked into the sync pipeline, creating a divergent definition of the data that nobody documents.</li><li>They multiply costs. Every copy is storage you pay for, governance you must enforce, and lineage you must track.</li></ul><h3>Share, Do Not Copy</h3><p>The architectural response is straightforward: when data needs to reach another team, another business unit, or another organization, share it in place rather than replicating it using access controls or Delta Sharing. Recipients query live data through a governed, auditable protocol without any files being copied. The data stays in one place, governed by one policy set, with one version of the truth.</p><p><strong>The nuance worth preserving:</strong> not all copies are evil. Caches, materialized views, and read replicas serve legitimate performance purposes. The litmus test is dependency. If nothing <em>depends</em> on the copy as a source of truth, it is just a performance optimization. If downstream workloads treat the copy as authoritative, it becomes a silo in disguise.</p><h3>Principle 4: Build to Scale and Optimize for Performance and Cost</h3><p>A lakehouse designed for today’s data volume will be redesigned within 18 months. Data growth is not linear, it accelerates as organizations instrument more systems, onboard more teams, and adopt more ML use cases. The architecture must accommodate this growth without requiring structural rework.</p><h3>Storage and Compute Are Separate Concerns</h3><p>This seems obvious in 2026, but the implications are often under-applied. Consider a 50 TB dataset sitting in cloud object storage. On Monday, a 4-node cluster runs a routine dashboard refresh against it after scanning a few partitions, returning in minutes. On Friday, a 64-node cluster runs annual model retraining against the same data by performing a full table scan with heavy shuffle operations.</p><p>Same storage. Radically different compute requirements. The architecture must support both without over-provisioning for the peaks or under-serving the troughs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*stbAQ9Kgz_radhwq9HpP8Q.png" /></figure><h3>Elasticity as an Architectural Property</h3><p>Predictable workloads (nightly ETL, hourly dashboard refreshes) have stable resource profiles. But the workloads that deliver the most business value such as ad-hoc analysis, model experimentation, seasonal forecasting are inherently spiky. The platform must treat elasticity as a first-class property:</p><ul><li><strong>Horizontal scaling:</strong> more nodes when parallelism matters.</li><li><strong>Vertical scaling:</strong> larger nodes when single-task memory or CPU matters.</li><li><strong>Instant provisioning:</strong> seconds to first query, not minutes.</li><li><strong>Automatic release:</strong> idle resources should disappear without human intervention.</li></ul><h3>Reframing the Cost Conversation</h3><p>Performance and cost are not opposing forces to be traded off but they are two axes of the same optimization space. The question is never “fast or cheap?” but rather “what is the minimum spend required to meet this workload’s performance SLA?” Over-provisioning is waste disguised as reliability. Under-provisioning is cost savings that show up as SLA breaches and frustrated users.</p><h3>Principle 5: Encourage Open Interfaces and Open Formats</h3><p>Lock-in is not always obvious. It does not announce itself but accumulates quietly in proprietary file formats, vendor-specific APIs, and closed query engines. By the time you realize the constraint, migration costs have compounded to the point where switching is no longer economically viable.</p><p>Open formats and interfaces are the architectural immune system against this accumulation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BHw0VzscFaw21URW5EZLRQ.png" /></figure><h3>The Strategic Case for Openness</h3><ul><li><strong>Portability:</strong> When your data lives in an open format on cloud object storage, you can read it with any engine that speaks that format. Your data outlives any single vendor relationship. That is not a hypothetical concern, it is a board-level risk management decision.</li><li><strong>Ecosystem leverage:</strong> Proprietary formats start with one integration (the vendor’s own tool). Open formats start with dozens. Delta Lake already has native support across major BI tools, query engines, and streaming platforms. That ecosystem compounds over time.</li><li><strong>Talent accessibility:</strong> Requiring expertise in a proprietary system shrinks your hiring pool. Requiring Python, SQL, and Spark expertise gives you access to the largest talent pools in data engineering and data science.</li><li><strong>Direct economics:</strong> When data can be accessed directly on object storage, there are no egress fees, no per-query licensing, and no proprietary compute markup. You read your own files with whatever engine makes sense for the workload.</li></ul><h3>The Stack That Delivers This</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6nVZhmqANiPZjhiy03f5aA.png" /></figure><p>The point is not that open is always better in every dimension. It is that open <em>preserves optionality</em> and in enterprise architecture, optionality is one of the most valuable property you can maintain.</p><h3>Principle 6: Democratize Value Creation Through Self-Service</h3><p>The fastest way to destroy the value of a well-governed, high-quality data platform is to gate access behind a central team with a two-week request queue. Every day a business analyst waits for access is a day the organization is not leveraging its data investment.</p><p>Self-service is not a nice-to-have, it is the mechanism through which a data platform delivers ROI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*01n4UTGjt20jplyCaLG0Bw.png" /></figure><h3>Redefining Self-Service</h3><p>Self-service does not mean “everyone gets admin access.” It means removing unnecessary friction while maintaining governance:</p><ul><li>A marketing analyst should be able to discover customer segmentation data, understand its schema and lineage, and query it without filing a ticket.</li><li>A data scientist should be able to spin up a compute environment, connect to feature tables, and run experiments without waiting for infrastructure provisioning.</li><li>A business unit leader should be able to request a new data domain in the lakehouse and have it provisioned, automatically, with governance, compute policies, and monitoring.</li></ul><h3>What Makes This Possible at Scale</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7yyga6ICgC2qhVXVH8RMlg.png" /><figcaption>Few capabilities enabling self-serve at scale</figcaption></figure><h3>Guardrails, Not Gates</h3><p>The relationship between Principle 6 (self-service) and Principle 2 (governance) is not adversarial, it is complementary. Governance defines the boundaries. Self-service provides freedom <em>within</em> those boundaries. Users do not need to understand the governance layer to benefit from it, they simply experience a platform where they can discover, access, and query data without obstruction, while the platform silently enforces permissions, logs access, and monitors quality.</p><h3>How the Six Principles Interlock</h3><p>These principles are not a checklist you work through sequentially, they form a reinforcing system:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mSY9_OZ2FRMCmIBNpcOZUw.png" /></figure><ul><li><strong>Principles 1 and 2</strong> create <em>trust</em>: curated data products governed by a unified strategy. Without trust, self-service (Principle 6) becomes a liability rather than an enabler.</li><li><strong>Principles 3 and 4</strong> create <em>efficiency</em>: no wasted storage in silos, no wasted compute on idle clusters. Without efficiency, scaling (Principle 4) becomes prohibitively expensive.</li><li><strong>Principles 5 and 6</strong> create <em>velocity</em>: open formats mean no integration bottlenecks, self-service means no access bottlenecks. Without velocity, the platform is technically excellent but business-irrelevant.</li></ul><h3>A Practical Adoption Sequence</h3><ol><li><strong>Start with governance and openness</strong> (Principles 2 + 5): These are hardest to retrofit. Deploy Unity Catalog. Commit to Delta Lake. Establish audit logging. Do this before you have 50 teams on the platform.</li><li><strong>Build the product model</strong> (Principle 1): Define ownership, schema contracts, and quality gates for your first gold-layer assets. Prove the pattern works before mandating it organization-wide.</li><li><strong>Enforce sharing over copying</strong> (Principle 3): As teams onboard, make access controls and Delta Sharing the path of least resistance. Make creating a silo <em>harder</em> than sharing.</li><li><strong>Design for elasticity</strong> (Principle 4): Adopt server-less compute and autoscaling policies. Eliminate always-on clusters that serve intermittent workloads.</li><li><strong>Enable self-service progressively</strong> (Principle 6): Start with read access and catalog discovery. Expand to self-service compute, then self-service data publishing.</li></ol><p>A lakehouse built on these principles can evolve its tooling, scale to new clouds, and onboard new teams without structural rework. A lakehouse built without them will accumulate debt that no amount of pillar-level optimization can repay.</p><p>Thank-you very much for reading! If you find this helpful please clap, it helps the article’s reach.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=607b0f513f51" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Beyond Model Drift: Monitoring on Databricks]]></title>
            <link>https://medium.com/@adityabnsemp/beyond-model-drift-monitoring-on-databricks-df276aef7edb?source=rss-b86ed7f17786------2</link>
            <guid isPermaLink="false">https://medium.com/p/df276aef7edb</guid>
            <category><![CDATA[data-quality-monitoring]]></category>
            <category><![CDATA[lakehouse-monitor]]></category>
            <category><![CDATA[data-drift]]></category>
            <category><![CDATA[model-drift]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Aditya Bhatraju]]></dc:creator>
            <pubDate>Mon, 22 Jun 2026 13:40:01 GMT</pubDate>
            <atom:updated>2026-06-22T13:40:01.607Z</atom:updated>
            <content:encoded><![CDATA[<p>In production AI, models rarely fail catastrophically. They degrade. Quietly. A credit risk model trained in a low-rate environment begins misclassifying borrowers few months after rates double. An inference pipeline silently drops a third of its records due to an upstream schema change, and no one notices until a compliance review surfaces the anomaly. By then, decisions have already been made on corrupted signals.</p><p>In regulated industries, financial services in particular, where credit risk models must conform to model risk management guidelines, this is not a technical inconvenience but a regulatory exposure.</p><p>Databricks has natively solved this problem by evolving Lakehouse Monitoring into a fully integrated, Unity Catalog-governed suite: <strong>Data Quality Monitoring</strong>. This is not a monitoring add-on. It is an architectural component of the platform, designed so that observability is embedded, not bolted on afterward.</p><h3>The Architecture: Two Complementary Pillars</h3><p>The most common mistake in ML observability is treating data quality and the drift as the same problem. They are not. Before we can ask whether feature distributions have shifted, we must first verify that the data arrived on time and at the right volume.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zScvyt2MLgTnrfKwum_wrQ.png" /><figcaption>Monitoring Hirearchy</figcaption></figure><p>Databricks addresses this by decomposing observability into two distinct, native capabilities:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nRiISV-wrDsBb7_viznvjA.png" /></figure><ul><li><strong>Anomaly Detection: </strong>Monitoring for data reliability. Schema-level, zero-configuration monitoring that ensures data is arriving and arriving correctly before any statistical analysis begins.</li><li><strong>Data Profiling:</strong> Observability for data and ML. Table-level statistical monitoring that tracks distribution shifts, feature integrity, and model performance over time. Formerly known as Lakehouse Monitoring.</li></ul><p>Both capabilities live inside Unity Catalog. Confusing between them leads to monitoring architectures that catch the drift while missing the silent pipeline failure that invalidated the batch upstream.</p><h3>Pillar 1: Anomaly Detection</h3><p>Anomaly Detection is enabled at the schema level. Navigate to any schema in Catalog Explorer, open the <strong>Details</strong> tab, click enable the <strong>Data Quality Monitoring</strong> dialog, turn on <strong>Anomaly detection</strong>, then click Save. Databricks handles the rest.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vdu-7UfgaK8rqJ07eTVhMg.png" /><figcaption>Anomaly Detection Flow</figcaption></figure><p><strong>Freshness and Completeness</strong></p><p>Every table in the enabled schema is evaluated on two dimensions:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S7xoRNQvOukBvKQXjFP1sQ.png" /></figure><p><strong>Freshness</strong> measures how recently a table was updated. Databricks builds a per-table predictive model from the historical pattern of Delta commits. If a commit arrives significantly later than expected because an upstream job failed, or a dependency timed out then the table is flagged as stale. No threshold tuning is required. The model learns the expected delivery cadence from history, and deviation from that cadence becomes the signal.</p><p><strong>Completeness</strong> measures row volume. Databricks analyzes the historical distribution of rows written to the table within any 24-hour window and establishes a predictive lower bound. If overnight batch volume falls below this lower bound, the table is marked as incomplete. This is specifically about whether sufficient data arrived.</p><p>Percent null adds a column-level completeness check (beta as of June 2026). Databricks predicts an expected range for nulls over the last 24 hours, and if a column’s percent null exceeds the upper bound, that table is also marked incomplete.</p><p>A critical operational point: Anomaly Detection runs on serverless compute and adds <strong>zero overhead</strong> to the pipelines populating your tables. It observes passively.</p><p><strong>Intelligent Scanning</strong></p><p>Databricks does not scan every table at the same frequency. The intelligent scanning engine prioritizes tables by <strong>downstream usage and popularity</strong> for example, the number of queries, dashboards, and downstream tables that depend on each one. High-impact tables are evaluated most aggressively. Low-impact tables are scanned less frequently. Critical data assets receive the most monitoring attention automatically, without manual prioritization.</p><p>Tables can be explicitly excluded from scanning using the `excluded_table_full_names` parameter in the Create or Update Monitor API.</p><p>One planning consideration: smart scanning may delay the population of health indicators by <strong>up to two weeks</strong> for tables skipped during the initial scan. The indicator is populated at the next scheduled rescan. Account for this window when rolling out monitoring across a new schema.</p><p><strong>Health Indicators and the Catalog Explorer Surface</strong></p><p>Once a scan completes, <strong>health indicators</strong> appear on table and schema pages in Catalog Explorer. Any user with `SELECT` or `BROWSE` permission can see the health status of a table without navigating to the Data Quality Monitoring UI. This surfaces data reliability directly in the environment where data consumers already work.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1_AXdq6wIhE5WJgS9CZm5A.png" /><figcaption>Health Indicator Types</figcaption></figure><p><strong>Incident Management</strong></p><p>Unhealthy tables surface in the <strong>Unhealthy</strong> tab of the Data Quality Monitoring UI with the reason (freshness or completeness), first detection timestamp, root cause analysis, and a downstream impact classification (High, Medium, or Low) based on the number of dependent tables and queries.</p><p>From the incident details view, two resolution actions are available:</p><ul><li><strong>Assign to me</strong>: Claims ownership of the incident to indicate active investigation. The table remains Unhealthy. The assignment persists for seven days.</li><li><strong>Not an issue</strong>: Marks the incident as a false positive and dismisses it. The table reverts to Healthy, and the resolution is logged in the Recently Resolved Incidents section. The dismissal also persists for seven days.</li></ul><p>The <strong>auto-resolved incidents</strong> view deserves equal attention. These are self-healing issues such as upstream delays or staleness windows that resolved once fresh data arrived. Reviewing them over time separates flaky pipelines from persistent structural problems and provides evidence of transient issues for compliance documentation.</p><p><strong>Programmatic Alerting: The System Table</strong></p><p>All anomaly detection results are written to `system.data_quality_monitoring.table_results`, a system table accessible across the entire metastore. This table can be queried directly in Databricks SQL to build custom alerts, filtered by catalog, status, and scan timestamp. By default, only account admins have access to it. If your MLOps team needs to configure alerts or automated responses against it, explicit `GRANT` statements are required before any automation is built on top of it.</p><h3>Pillar 2: Data Profiling</h3><p>Where Anomaly Detection verifies that the pipeline is flowing, Data Profiling verifies that the data means what it is supposed to mean. For ML teams, this is where the operational signal lives.</p><p><strong>Profile Types</strong></p><p>Data Profiling supports three profile types. Choosing the right one depends on whether your table has a time dimension and whether it contains model prediction data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U8aswM_dVV4PYWyq0fnq4w.png" /><figcaption>Choosing Profile Type Flowchart</figcaption></figure><p><strong>1. Snapshot</strong> profile works against any Delta managed table, external table, view, materialized view, or streaming table, with no timestamp column required. Every refresh re-processes the entire table and CDF does not apply. This makes Snapshot appropriate for tables that are relatively static or updated infrequently: reference data, configuration tables, dimension tables, or any table where a current-state quality assessment is more valuable than trend analysis. The maximum supported table size is 4TB and for tables larger than this, a Time Series profile is the recommended alternative.</p><p><strong>2. Time Series</strong> profile is for tables that have a meaningful timestamp column and where understanding how data properties change over time windows is the primary concern. Profiling partitions data into configurable time windows with granularities from 5 minutes to 1 year and computes quality metrics over time, enabling trend detection and window-to-window drift comparison. The timestamp column must be of TIMESTAMP type, or a type convertible using the `to_timestamp` function. Use Time Series for event streams, feature tables with event timestamps, and any dataset where detecting distribution shifts over a rolling window matters more than a single point-in-time view. CDF is strongly recommended: with it enabled, each refresh processes only newly appended rows rather than re-scanning the full table. Note that profiles defined on materialized views do not support incremental processing regardless of CDF.</p><p><strong>3. Inference</strong> profile is similar to Time Series profile, additionally it also has model quality metrics. This profile is used to track the performance of machine learning models and model-serving endpoints by profiling inference tables. These profiles are specifically designed for tables that store machine learning model request logs, with one row per inference request and its prediction. The tables include columns for a timestamp, a model ID, model inputs, predictions, and optionally ground truth labels for both classification and regression models. The profile compares model performance and data quality metrics across time-based windows.</p><p>Tables for Inference profiles also support demographic metadata columns that are not used as inputs to the model but are useful for fairness and bias analysis. This capability is directly relevant in regulated domains where disparate impact must be documented and audited.</p><p>Change Data Feed (CDF) is strongly recommended for the same efficiency reasons as with Time Series profiles.</p><p>A practical decision rule: if the table has no timestamp, use Snapshot. If it has a timestamp and contains feature or event data, use Time Series. If it has a timestamp and contains model predictions, use Inference.</p><h3><strong>The 30-Day Lookback</strong></h3><p>This is the implementation nuance and most teams discover only after a painful production deployment.</p><p>When you first create a Time Series or Inference profile, <strong>Databricks processes only the data from the last 30 days prior to creation</strong>. Any older data in the table is ignored. After the monitor is created, all newly appended data is processed on each refresh. The consequence: if you attach a monitor to a table that already contains six months of inference history, only the most recent 30 days of that history will ever be evaluated. The good news is, <strong>Databricks account team helps in adjusting this 30 day limit.</strong></p><p>Since all the newly added data will be processed after creating these profiles, enabling CDF is strongly recommended for both Time Series and Inference profiles. When active, each refresh processes only newly appended rows rather than re-scanning the entire table. As inference logs grow to hundreds of millions of rows, this is the difference between a two-minute refresh and a two-hour one.</p><h3>Primary Table and Baseline Table</h3><p>A table that is to be profiled is called Primary Table. In addition to the primary table being profiled, you can optionally specify a baseline table as a reference for measuring drift that is, the change in values over time. The baseline table should contain data that reflects the expected statistical distributions, column distributions, missing value rates, and overall data quality characteristics of the input data. It must match the schema of the primary table, with the exception of the timestamp column for Time Series and Inference profiles.</p><p>The appropriate baseline varies by profile type. For Snapshot profiles, use a prior snapshot where distributions represent an acceptable quality standard. For Time Series profiles, use data from time windows where distributions were within expected norms. For Inference profiles, the ideal baseline is the training or validation dataset. Drift is then measured relative to what the model was actually trained on, and the baseline must include the same feature columns and the same `model_id_col` as the primary table to ensure consistent aggregation.</p><h3>Output Tables: Profile Metrics and Drift Metrics</h3><p>Every profile generation produces or updates two output Delta tables:</p><p><strong>1. Profile Metrics</strong> contains per-column, per-time-window statistics: row counts, means, null/zero percentages, distinct value counts and quantile distributions. For Inference profiles, it also includes model performance metrics (such as accuracy, precision, recall, F1, confusion matrix, RMSE, MAPE, R²) plus fairness and bias metrics.</p><p><strong>2. Drift Metrics</strong> contains statistics related to the data’s drift over time. The drift table is generated or updated when a baseline table is provided or when a consecutive time window exists after aggregation. The two types of drift computed are:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DoxwgJzt8k2fBASgiirTQA.png" /></figure><ul><li><strong>BASELINE:</strong> Compares each time window against the baseline table distribution. Detects gradual regime drift that accumulates over weeks or months and is invisible to consecutive-only monitoring.</li><li><strong>CONSECUTIVE:</strong> Compares each time window against the immediately preceding window. Detects abrupt changes such as feature pipeline regressions, data source migrations, or upstream schema changes.</li></ul><p>For each column, Databricks applies statistically appropriate tests based on data type. For <strong>numerical columns</strong>: the Kolmogorov-Smirnov (KS) test, Wasserstein distance, and Population Stability Index (PSI). And, for <strong>categorical columns</strong>: Chi-squared test, Total Variation (TV) distance, L-infinity distance, and Jensen–Shannon (JS) distance.</p><p>Additionally, the drift metrics table carries `count_delta`, `avg_delta`, `percent_null_delta`, `percent_zeros_delta`, and `percent_distinct_delta` giving a complete quantitative picture of the shift between windows.</p><h3>The Monitoring Dashboard</h3><p>When a profile runs, Databricks automatically generates a dashboard displaying key metrics computed by the profile. The visualizations included depend on the profile type with different metrics organized into sections. The dashboard is created in the user’s account and is customizable and shareable like any other dashboard. Teams can modify existing charts, add new ones, and adjust user-editable parameters for date range, data slices, and model filters across the entire dashboard or for individual charts.</p><h3>Use Case: Credit Risk Monitoring on the Lakehouse</h3><p><strong>Business Problem</strong></p><p>A financial institution wants to operationalize a credit risk model in production, but the real challenge is not just model training. It is proving that incoming data remains timely, complete, statistically stable, and operationally trustworthy as macroeconomic conditions change.</p><p>The business problem is therefore two-fold: protect the upstream data pipeline from hidden failures, and detect when the model is making decisions on data that no longer resembles the environment it was trained on. In regulated settings, that monitoring layer must also create an auditable record for investigation, escalation, retraining, and governance review.</p><p><strong>Data</strong></p><p>The use case is built on synthetic borrower application data designed to simulate realistic economic regime changes across stable, stressed, and recessionary periods. The dataset includes core underwriting features such as annual income, debt-to-income ratio, loan amount, interest rate, credit score, employment length, number of open accounts, recent delinquencies, home ownership, and loan purpose.</p><p>The target label is a credit risk grade, represented as ordered classes from A through E. Because the data is synthetic, the use case is safe for demonstration while still being rich enough to show how input drift, data quality degradation, and model performance decay emerge together in a production-like environment.</p><p><strong>Medallion Architecture</strong></p><p>A Medallion architecture provides the right operating model for this use case. The <strong>Bronze</strong> layer captures raw loan application events exactly as they arrive from source systems, preserving lineage and making it possible to apply freshness and completeness monitoring at the point of ingestion.</p><p>The <strong>Silver</strong> layer standardizes and validates the records, applies data quality checks, resolves schema issues, and prepares consistent business-ready fields for downstream consumption. The <strong>Gold</strong> layer produces model-ready features and the inference log used for quality monitoring, making it the right place to evaluate drift, feature integrity, and outcome quality over time.</p><p><strong>Model Development and Management with MLflow</strong></p><p>The model development flow follows a standard Databricks MLOps pattern. A classification model is trained on a stable historical window, feature engineering is versioned in the pipeline, and experiments are tracked in MLflow so that parameters, metrics, model artifacts, and validation outcomes are captured in a reproducible system of record.</p><p>Once validated, the model is registered for governed deployment and promoted through the model registry. This creates the operational baseline for monitoring: the production model, its training window, and its evaluation results become the reference point against which future production behaviour can be compared.</p><p><strong>Monitoring</strong></p><p>Once deployed, Data Quality Monitoring becomes the control plane for the end-to-end use case. At the data layer, anomaly detection protects the ingestion and transformation pipeline by flagging delayed arrivals and unexpected drops in volume. At the model layer, data profiling on the inference log tracks how feature distributions, prediction behavior, and model quality evolve across time-based windows.</p><p>The dashboard turns those metrics into an operational workspace. For this use case, the most valuable sections are <strong>Model Performance</strong>, <strong>Feature Profiling</strong>, <strong>Drift Analysis</strong>, <strong>Data Quality</strong>, <strong>Fairness and Bias</strong> and <strong>Inference Volume</strong>. Together, they show whether accuracy, precision, recall, and F1 are degrading; which numerical or categorical features are drifting; where null, zero, or invalid value issues are emerging; whether fairness metrics are changing across demographic slices; and whether the scoring volume itself has shifted enough to explain or intensify the observed behavior.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bIyudH7LG6hUVLMiet-PsA.png" /><figcaption>Inference Volume Change</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6LAElLeueiyPlU6xlqoAIQ.png" /><figcaption>Data Quality Drop</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NV0myA1lebOVmSLp-NdVNQ.png" /><figcaption>Model Performance Drop</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F_uTsCyJEPI4CMJvEt7sYA.png" /><figcaption>Feature Drift Heatmap</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9xEiV8cqyID--5evL9ycrQ.png" /><figcaption>Label Drift Detection</figcaption></figure><p>In the credit risk scenario, this monitoring pattern makes deterioration visible early. A model trained in a stable economy can still appear operational while silently drifting out of policy relevance as rates rise, borrower quality changes, and delinquency patterns worsen. By combining baseline drift, consecutive drift, performance metrics, and data quality indicators in one view, the monitoring system tells a coherent story about what changed, when it changed, and where to investigate first.</p><p><strong>Closing the Loop: From Observation to Automated Action</strong></p><p>Monitoring without action is expensive logging. The operational architecture must close the feedback loop: observation triggers retraining, retraining restores performance, and the cycle repeats.</p><p>A Databricks SQL Alert configured against the Drift Metrics output table serving as the trigger. The alert queries the baseline drift rows for features where KS test p-values fall below a significance threshold, with annual income, debt-to-income ratio, and interest rate acting as high-priority signals in a credit risk context. When the alert trips, it fires a webhook notification that triggers a Databricks Job. That job pulls the most recent feature data from the Gold layer, retrains the model via MLflow with a fresh training window, evaluates it against a holdout set, and registers a new champion version in the model registry. The production serving endpoint then picks up the new version, and the monitoring cycle restarts.</p><p>This architecture satisfies the core governance requirement behind modern model risk management. The Drift Metrics table becomes the evidence trail, capturing when drift was detected and what triggered the response. MLflow captures every retraining run, hyperparameter choice, and evaluation result. Unity Catalog lineage connects the model version back to the data and transformation pipeline that produced it. The result is not just automated retraining, but automated retraining with traceability, reproducibility, and auditability.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C9mUFj_kVG7pd24jOXonlA.png" /></figure><h3>Conclusion</h3><p>The failure mode of production ML is not a model that stops working overnight. It is a model that degrades slowly, on data that drifts silently, in a pipeline no one is watching. By the time the signal surfaces in business metrics, the damage is done.</p><p>Databricks Data Quality Monitoring closes this gap architecturally. Anomaly Detection protects the pipeline, ensuring data arrives on schedule and at the right volume, flagging failures before they reach inference. Data Profiling interrogates the data, tracking statistical distributions, feature integrity, model performance, and fairness across every time window. The monitoring dashboard surfaces all of this through a production-ready analytical interface. And SQL alerts against the drift metrics output tables convert passive observation into active automated remediation.</p><p>The patterns covered here Anomaly Detection, CDF enabled profiles, dual consecutive and baseline drift tracking, triaged data quality monitoring, fairness/bias metrics integration and SQL alert-triggered retraining form a reproducible blueprint for any team operating ML models in a regulated or high-stakes environment.</p><p>Build systems that know when they are failing. Build systems that know how to heal themselves. Databricks Data Quality Monitoring enables it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=df276aef7edb" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>