By Josh Rosenberg (Sr. Solutions Engineer @ Databricks) & Cassi Sullivan (Sr. Solutions Engineer @ Databricks)

We spend a lot of energy building dashboards. We agonize over the right chart type, argue about color palettes, and rewrite the same aggregation four different ways to shave 200 milliseconds off a load. Then we publish, share the link, and move on.

And then… we mostly stop paying attention.

Six months later, nobody can answer the simple questions. Which dashboards do people actually open? Who are the power users? Which one has quietly become the most expensive object in the workspace because it re-scans a billion rows every time someone breathes near it? Are we paying to keep three near-identical “exec overview” dashboards alive when only one gets traffic?

Those aren’t dashboard-building questions. They’re dashboard operations questions, and Databricks already collects everything you need to answer them. It’s sitting in the system tables. You just have to assemble it.

So we built an AI/BI dashboard that does exactly that, and we’re sharing it so you can import it into your own account in a couple of minutes. This post walks through what it shows, how it works under the hood, how to set it up, and — importantly — the caveats you should understand before you present any of these numbers to a VP.

What it actually shows

The dashboard is organized into two analytical pages plus a shared filter panel, all driven by a single date range and a set of cross-cutting filters (viewer, dashboard ID, warehouse).

The Overview page answers the adoption question. At a glance you get total dashboard views, unique viewers, the count of dashboards that saw any activity, and total queries fired. Below the headline counters, it breaks down daily query volume over time, ranks your most-used dashboards, surfaces your top viewers, and — the one that tends to start conversations — estimated cost by dashboard. There’s also a detailed query-level table for when you need to drill into specifics.

The Performance & Attributed Costs page answers the efficiency question. Average query duration, cache hit rate, and total DBUs consumed sit up top. From there it trends cost and query duration over time, plots cache hits versus misses (a fast, honest read on how much your dashboards benefit from result caching), and slices cost and DBU consumption by user, by SKU, and by warehouse. The by-warehouse breakouts are useful for spotting a dashboard that’s pointed at an oversized or poorly-sized warehouse.

The Global Filters pane is the control panel. Everything is parameterized, so you can zoom from “the whole account, last 90 days” down to “this one dashboard, this one warehouse, last week” without editing a single query.

The design goal was that someone should be able to open it cold and, within about thirty seconds, know whether their dashboard estate is healthy, loved, and reasonably priced.

How it works under the hood

The whole thing is built on Unity Catalog system tables — the read-only, account-wide operational logs Databricks maintains in the system catalog. No pipelines, no jobs, no copies of data. The dashboard queries the source directly. Four datasets do the work:

• Views come from the audit log. The system.access.audit table records dashboard interactions under service_name = ‘dashboards’. We filter to the getDashboard, getPublishedDashboard, and executeDashboardWidget actions and pull the viewer’s email and the dashboard ID out of the request parameters. A left join to system.access.workspaces_latest swaps opaque workspace IDs for human-readable names.
• Performance comes from query history. system.query.history carries a query_source struct, and for any statement triggered by an AI/BI dashboard, query_source.dashboard_id is populated. That single field is the key that lets you attribute duration, rows read, cache behavior, and execution status back to a specific dashboard.
• Cost is the interesting part. Query history tells you which statements a dashboard ran; system.billing.attributed_usage tells you the DBUs each of those statements consumed, joined on the statement ID (usage_metadata.dbsql_statement_id = statement_id). To turn DBUs into dollars, we join system.billing.account_prices for any negotiated contract pricing on the SKU — if no contract price exists (as is the case for most accounts), we fall back to system.billing.list_prices, which always carries the public list rate. The result is an estimated cost per query that rolls up cleanly to per-dashboard, per-user, and per-warehouse totals. Note that attributed_usage lags by approximately 2 days, so cost figures for the most recent days may be incomplete until billing attribution settles.
• Daily activity is a pre-aggregation. A fourth dataset groups query history by day, dashboard, and warehouse to produce the trend lines efficiently — query counts, unique users, average duration, bytes read, and a cache-hit/miss split.

That’s the entire architecture. Because it reads live from system tables, there’s nothing to schedule and nothing to maintain. When Databricks writes new audit and billing records, your dashboard reflects them on the next refresh.

Getting it running

The dashboard is importable directly from the repo (see below). Grab the .lvdash.json, import it as a new AI/BI dashboard in your workspace, point it at a SQL warehouse, and refresh.

There’s one prerequisite that trips people up, and it’s not the import — it’s permissions. System tables have to be enabled by an account admin, and the schemas this dashboard reads (access, query, and billing) each have to be turned on. Whoever runs the dashboard also needs SELECT on those system schemas. If you import it and every tile comes back empty or throws a permissions error, that’s almost always the cause. Enabling system tables is a one-time account-level action; your platform or admin team can do it in a few minutes.

Once it’s importing cleanly, set the date range on the Global Filters page to something reasonable (we default to the last 30–90 days), pick a warehouse, and let it populate. From there it’s just filtering.

Possible Upgrades

If you want to take the dashboard a step further, there are four upgrades worth calling out.

• How to find the dashboard ID: The fastest path is to grab it from the dashboard URL. For AI/BI dashboards, the dashboard ID is the value in the /sql/dashboardsv3/{id} URL, and if you are starting from the workspace file instead, workspace/get-status returns the dashboard resource_id, which is also the identifier used by the Dashboard API and the published dashboard URL.
• How to automatically include the dashboard name: The main dashboard access audit events log dashboard_id, but not the human-readable dashboard name. A practical improvement is to resolve that ID through the dashboard API, whose response includes display_name, so your reporting layer can show readable dashboard names without relying on a manual mapping file.
• Pull it via API into your own table: If you need longer retention or a cleaner reporting layer, snapshot the dashboard metadata into your own table on a schedule instead of depending only on the live retention window of system tables. The same dashboard_id can be used with the Dashboard API to pull dashboard details such as display_name, path, and warehouse_id, which gives you a lightweight dimension table you can reuse across joins and historical analysis.
• Add extra joins to show warehouse name explicitly: Query history records the warehouse used for a statement through compute.warehouse_id, while the warehouses system table exposes the readable warehouse_name. Joining on warehouse_id, and taking the latest warehouse snapshot for each ID, lets you replace opaque IDs with explicit warehouse names and makes cost and performance breakdowns much easier to interpret.

Where it earns its keep

A few concrete ways we’ve seen this pay off:

• Rationalizing the estate. Sort dashboards by activity and the long tail of zero-view dashboards becomes obvious. Those are candidates to archive — less clutter, fewer stale numbers floating around, and in some cases real savings.
• Cost attribution and chargeback. “Cost by dashboard” and “cost by user” give you a defensible, data-backed starting point for conversations about who’s driving warehouse spend, instead of guessing.
• Performance triage. The cache hit rate and average-duration trends make it easy to spot a dashboard that’s slow because it’s not benefiting from caching, or one whose performance has quietly degraded as its underlying tables grew.
• Adoption tracking. If you just shipped a dashboard to a new team, the viewer and query-volume trends tell you within days whether it landed or whether you need to go do some enablement.

Caveats — read these before you quote any numbers

This is the part I’d want a colleague to read most carefully, because system-table data is powerful but it has sharp edges. Treat the dashboard as a strong directional signal, not a billing-grade source of truth.

“Views” are an approximation. A single person opening one dashboard generates multiple audit events — an open event plus a widget-execution event for each tile on the page. Counting those actions gives you an excellent engagement signal, but it is not a literal count of unique human page loads, and it will run higher than a naive “one open = one view” intuition. If you need stricter semantics, narrow the action list (for example, count only getDashboard/getPublishedDashboard) and calibrate to your own environment.

Cost is a list-price estimate. The dollar figures are computed from account_prices, which reflect list pricing. They do not reflect your organization’s actual negotiated or effective rate, discounts, or commit-based pricing. Use them to compare dashboards against each other and to spot outliers — not as a substitute for your real invoice.

Cost attribution is cleanest on serverless SQL. Statement-level DBU attribution via attributed_usage lines up best with serverless SQL warehouses, where usage is tied to individual statements. On classic or pro SQL warehouses, billing is driven by cluster uptime rather than per-statement execution, so per-dashboard cost attribution can be partial or absent. Know which warehouse type your dashboards run on before you read too much into the cost tiles.

System tables aren’t real-time. There’s ingestion latency between an event happening and it landing in the audit and billing tables — typically on the order of hours, and billing records can lag longer. Expect “today” to look sparse; judge trends on data that’s had time to settle.

Retention is finite. System tables retain history for a bounded window, and the dashboard can only show you what’s still retained. If you need longer-horizon analysis, snapshot the data into your own table on a schedule.

Query history is still in Public Preview. The system.query.history table — the backbone of the performance and cost datasets — is a Public Preview feature at the time of writing. Its schema can evolve, so keep an eye on the release notes if a field stops behaving the way you expect.

None of these should stop you from using the dashboard. They’re the difference between someone confidently presenting garbage and someone presenting a well-understood estimate with the right footnotes. The second person is the one who gets trusted with the next question.

Grab it and make it yours

Everything here is importable and fully editable. If your organization tags warehouses by team, add a tag breakdown. If you care about published-link traffic specifically, tighten the audit filter. If you want an alert when a dashboard’s cost crosses a threshold, wire one up on top of the cost dataset.

The point isn’t this exact set of tiles — it’s that the data to run your dashboards like a product, rather than shipping and forgetting them, is already in your account. This just connects the wires.

Repo and import instructions: Repo Link

If you end up extending it, we’d love to hear what you added.

*If there are any system tables referenced here that are not currently accesible, you can reach out to your account team for support.

Who’s Actually Using Your Databricks Dashboards (and What Do They Cost)? was originally published in DBSQL SME Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

In my previous article, I discussed the cost of Genie Code. In this article, we'll explore the key differences between Assistant and Genie Code. If you want to use Genie Code, you'll also need to understand the basics of working with agents, as it can now not only assist you, but also edit and delete tables, scripts, and data with your permission.

Databricks is keeping up with the times and has decided to abandon the simple Assistant-style prompts in favor of Agentic AI. Let's break down the key differences:

1. Operating mode:

• Databricks Assistant: Works on a question-and-answer basis. You ask it to write an SQL query, explain a piece of PySpark code, or find an error in a specific notebook cell, and it generates text or code and awaits your next command. It uses local context, specifically what's in your current notebook and the current chat history.
• Genie Code: This is an agent-based system. You can assign it a high-level task, and it will automatically break it down into steps, find the necessary tables, write the code, execute it, and verify the result. It is deeply integrated into the Unity Catalog. It automatically understands all your company data: table semantics, columns, table lineages, popular assets, and corporate access policies. You don't need to explain to it where the data is. It finds it automatically.

2. Hints and support:

• Assistant: Helps only during the development phase, writing and debugging code in real time.
• Genie Code: Runs as a proactive agent in the background. It can monitor Lakeflow pipelines and ML models, automatically analyze logs during crashes, perform basic troubleshooting, and configure resource allocation:

3. Advanced Agent capabilities:

Genie Code is a fully configurable multi-agent. For each task, it dynamically routes requests, selecting the most appropriate model, be it a leading commercial LLM, open-source, or your custom model deployed directly in Databricks. Furthermore, via the Model Context Protocol (MCP), it can interact with external systems like Jira or GitHub. You can also configure general rules in the form of custom instructions or skills for specific tasks, immersing the agent in context:

Basics of working with agents

Genie Code takes over more complex engineering processes: from independent exploratory data analysis and model training to building and maintaining ETL pipelines. But now you need to understand the nuances and fundamentals of working with agents.

1. Context isolation. For each new task or fundamentally different data set, it is essential to start a new session (chat):

The agent builds a decision tree that remembers the action plan and local context. If, after creating an ETL pipeline for the finance department, you ask it to analyze web server logs in the same window, its context will be confused. It may start applying previous logic or attempt to connect unrelated data, leading to failures and hallucinations.

2. Confirming actions for the Agent. By default, the Agent will ask permission for every action, but you can grant "Always Allow" for all changes. Be careful in a production environment! It can permanently delete something, and finding such actions can be quite difficult. I recommend reviewing each action and confirming or rejecting it:

3. Session management and interruptions. The agent's lifecycle in interactive mode is tied to an active communication session. When a page is reloaded or the network is interrupted, the connection is lost. The system stops the agent to prevent uncontrolled resource consumption, as it doesn't know whether the user will return. Wait for the agent to complete generation or an intermediate response without refreshing the tab.

If the agent is running and you switch to another browser tab, it is paused, switch back and it continues running.

4. Task Setting: Goal + Limits. The agent should be given not step-by-step instructions (like the Assistant), but a final goal and strict limitations.

• Set limits during the development phase: "Build the display, but first limit the selection to 1000 rows to test the logic." This will prevent situations where the agent tries to sift through petabytes of data interactively.
• Specify security rules: Explicitly state if the agent is not allowed to modify data (for example: “Do not DROP/UPDATE existing tables, save results only to the sandbox schema” ).

5. In-process correction. Agents can get stuck in an infinite loop (e.g., write code with an error, get an error, try to fix it the same way, get another error).

• Follow the detailed logs of his actions.
• If you see that the agent has gone down the wrong path or is stuck in a loop, don't wait - forcefully stop the generation and direct it to the text: "This table is empty, look for data in another directory. "

Author: Maksim Pachkouski [Databricks MVP & Databricks Solutions Architect Champion]. Follow my posts on Medium and LinkedIn to stay up-to-date on the latest news in Data Engineering and learn about best practices.

Databricks Genie Code. Full practice guide and Examples was originally published in DBSQL SME Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

By Josh Rosenberg (Sr. Solutions Engineer @ Databricks) & Josh Bae (Delivery Solutions Architect @ Databricks)

Summary

• Embed the Genie Conversation API directly inside multi-tenant apps to deliver stateful analytics conversations backed by a curated Genie space.
• Authenticate requests through your application using OAuth M2M with service principals, not individual Databricks user accounts or external workspace provisioning.
• Securely personalize data with Unity Catalog ABAC policies and implement row-level security guardrails.

Introduction

Since the launch of AI/BI Genie spaces, we have seen enterprise-scale adoption of self-service analytics using natural language for many Databricks users. While this served customers well for internal reporting and operational visibility, the lack of access for external audiences to the UI-native experience on Databricks made it difficult to share this capability more broadly.

However, with the Public Preview release of the AI/BI Genie Conversation API across AWS, Azure, and GCP, users could now serve Genie-level insights from any application, even external or custom-built ones. The API provided a seamless way for business users to “talk to their data” from any surface, but it raised an immediate governance issue: how would the app ensure that only the data that is permitted for a particular viewer could be returned?

It is a problem we have seen many of our customers wrestle with, so this blog offers guidance on designing your multi-tenant Genie space apps with robust per-tenant guardrails via service principals (using OAuth M2M authentication) and Unity Catalog ABAC (attribute-based access control) policies.

Reference Architecture

This section outlines the high-level request flow and required infrastructure components for the integration.

At a high level, the runtime flow looks like this:

1. Your application receives a question from a user in a multi-tenant app.
2. The app determines which tenant the request belongs to.
3. The app retrieves that tenant’s Databricks service principal credentials from a secure vault.
4. The app uses the OAuth client credentials flow to get an access token.
5. The app calls the Genie Conversation API using that tenant-specific identity.
6. Genie generates SQL against curated Unity Catalog tables (in a Genie space).
7. ABAC policies evaluate current_user() during query execution.
8. Only the rows and columns that the identity is allowed to access are returned.

That basic runtime architecture is what makes the pattern powerful: every request runs as a tenant-specific principal, so identity becomes the control point for access.

Why this pattern works

The major design choice here is to use a dedicated service principal (SP) per tenant (which we will elaborate on in later sections). That way, your application no longer has to be the primary enforcement layer for data isolation. Instead, Databricks governance via Unity Catalog handles it at query time.

When a request runs, Unity Catalog evaluates policies using the current caller, and because each tenant has its own SP, you can write rules directly against that identity. This dramatically simplifies the mental model. Rather than sprinkling tenant filters throughout your app and hoping they are always applied correctly, you let the Databricks Platform evaluate access where it matters most: when the SQL actually runs. The tradeoff is operational: you now have many identities and secrets to manage, so automation is essential.

Multi-Tenancy Design

Tenant isolation hinges on how you structure service principals and how Unity Catalog ABAC evaluates row filters and column masks at query time. Let’s clarify these terms to better articulate how they will serve the implementation in the proposed solution later in this blog.

What’s a service principal (SP)?

Databricks recommends using SPs as specialized identities for accessing Databricks resources because they avoid relying on individual user credentials. It provides a programmatic way to grant/restrict access to the Genie space(s) without defining how each individual user will query the underlying data. Since all requests run under the tenant SP, filters/masks apply consistently without per-user identity.

What’s an attribute-based access control (ABAC) policy?

ABAC in Unity Catalog is a governance model that lets you centrally express access rules using attributes and have them enforced automatically at query time. In practice, ABAC policies typically take the form of row filters (to limit which rows are visible) and column masks (to redact sensitive fields), and those policies can be inherited down the object hierarchy (catalog → schema → table). Under our recommended design, applications can authenticate via SPs and rely on ABAC to enforce tenant boundaries at the data layer, rather than implementing security solely in application code.

We will cover authoring ABAC policies in a later section, but you can read more at Unity Catalog attribute-based access control (ABAC). (AWS | Azure | GCP)

Implementation Guide

Prerequisites

Before implementing this pattern, make sure you have the following in place:

• Be an account admin and/or workspace admin (SP creation, generating OAuth secrets, etc.)
• Have access to the following:
  - A Databricks workspace with the Databricks SQL entitlement
  - At least CAN EDIT permissions on the Genie space
  - CAN USE access on at least one pro or serverless SQL warehouse
  - SELECT privileges on the data used in the space
  - A control-plane process or service for onboarding tenants
  - A secure secret store or vault to manage SP credentials

Additionally, to create OAuth secrets for SPs, you should also account for several ABAC-related constraints when designing this architecture. ABAC in Unity Catalog requires Serverless compute (AWS | Azure | GCP) or Standard compute running on Databricks Runtime (DBR) 16.4+, and there are policy limits to consider, such as one row filter per table per user and one mask per column per user. Those constraints should help inform how you design your governance and policy layer from the outset, avoiding conflicts or unnecessary complexity later.

For best practices, limitations, and example user-defined functions (UDFs), see UDFs for ABAC policies best practices (AWS | Azure | GCP)

Step 1: Provision a service principal for each tenant

The first step is to treat tenant identity as a first-class component of your onboarding process.

When a new tenant is created, your provisioning system should create (or retrieve) a dedicated SP for that tenant, assign it to the appropriate workspace, and treat that identity as the credential your application will use whenever it needs to execute requests on the tenant’s behalf. Your app should not create identities dynamically at runtime. Instead, identity provisioning should live in a separate onboarding or control-plane process responsible for identity lifecycle management.

Maintaining this separation is important. Your application runtime should consume tenant credentials rather than manage tenant identities directly. Centralizing identity provisioning within a control plane ensures clearer lifecycle management, stronger security practices, and a more maintainable multi-tenant architecture.

You can leverage the Account SCIM REST API to do this.

To execute the HTTP requests, you will need your Databricks account ID, the account console URL (such as https://accounts.cloud.databricks.com for AWS), and an account-level access token. You will also need the exact SCIM ID of the group where you plan to assign the new service principal.

Note: The code can also be run with WorkspaceClient since workspace admins can also add SPs to the Databricks account.

import os
import requests

# Retrieve configuration from environment variables
ACCOUNT_ID = os.environ["DATABRICKS_ACCOUNT_ID"]
ACCOUNT_HOST = "https://accounts.cloud.databricks.com"
TOKEN = os.environ["DATABRICKS_ACCOUNT_TOKEN"]  # admin token

# Define a function that creates a new service principal
def create_service_principal(display_name: str):

    # Build the account-level API endpoint for creating a service principal
    url = (
        f"{ACCOUNT_HOST}/api/2.0/accounts/{ACCOUNT_ID}"
        f"/scim/v2/ServicePrincipals"
    )

    # Send a POST request to Databricks to create the service principal
    response = requests.post(
        url,
        headers={
            "Authorization": f"Bearer {TOKEN}",
            "Content-Type": "application/scim+json"
        },
        json={
            "schemas": ["urn:ietf:params:scim:schemas:core:2.0:ServicePrincipal"],
            "displayName": display_name
        },
        timeout=30,
    )

    # Raise an exception if Databricks returns an HTTP error status
    response.raise_for_status()
    return response.json()

Not included above, but Databricks recommends assigning each SP in Unity Catalog to groups. They simplify access to workspaces, data, and other securable objects.

Check out Manage service principals (AWS | Azure | GCP) for more explanations on how to manage SPs for your Databricks account and workspaces.

Step 2: Generate and store the OAuth secret

Once the tenant’s SP has been created and registered in Unity Catalog, the next step is to generate an OAuth secret that your application can use as the client_secret in the OAuth client-credentials flow.

You can create this secret programmatically using the Databricks API.

Note: A service principal can have up to five OAuth secrets, each with a configurable lifetime (up to two years).

import os
import requests

# Retrieve configuration from environment variables
ACCOUNT_ID = os.environ["DATABRICKS_ACCOUNT_ID"]
ACCOUNT_HOST = "https://accounts.cloud.databricks.com"
TOKEN = os.environ["DATABRICKS_ACCOUNT_TOKEN"]  # admin token

# Define a function that creates an OAuth secret for a given SP ID
def create_oauth_secret(service_principal_id: str):

    # Build the account-level API endpoint for creating a SP secret
    url = (
        f"{ACCOUNT_HOST}/api/2.0/accounts/{ACCOUNT_ID}"
        f"/servicePrincipals/{service_principal_id}/credentials/secrets"
    )

    # Send a POST request to Databricks to create the OAuth secret
    response = requests.post(
        url,
        headers={"Authorization": f"Bearer {TOKEN}"},
        json={"lifetime": "15552000s"},  # 180 days
        timeout=30,
    )

    # Raise an exception if Databricks returns an HTTP error status
    response.raise_for_status()

    return response.json()

One important detail is that the OAuth secret string value is displayed only once in the API response and cannot be retrieved again. As soon as it is shown, you should immediately store it in a secure secrets manager and never persist it in plaintext outside the vault. In practice, you will store the tenant’s client_id, client_secret, secret_id, and expires_at so you can authenticate at runtime and rotate secrets safely later.

Check out Authorize service principal access to Databricks with OAuth (AWS | Azure | GCP) for more information and instructions on how to do this via UI.

Step 3: Grant the minimum permissions during provisioning

After the SP and its associated OAuth secret are created, the next step is to grant only the permissions required for the tenant to use Genie and query the data it is authorized to access.

To use a Genie space, the SP must have two things:

• The consumer access or Databricks SQL entitlement
• At least SELECT privileges on all of the Unity Catalog data objects used in the space

Note: Queries run using the compute credentials embedded by the author who configured the warehouse. End users do not need direct warehouse permissions.

This should be part of your automated provisioning flow, not a manual admin step. A Python example using the Databricks SDK for Genie permissions looks like the following:

import requests

WORKSPACE_HOST = "https://<your-workspace-instance>.cloud.databricks.com"
WORKSPACE_TOKEN = os.environ["DATABRICKS_WORKSPACE_TOKEN"] 

def provision_genie_client_permissions(
    host: str,
    token: str,
    sp_app_id: str,      # client_id of the tenant SP
    genie_space_id: str,
):
    headers = {"Authorization": f"Bearer {TOKEN}"}

    # Allow the service principal to run the Genie space
    requests.patch(
        f"{WORKSPACE_HOST}/api/2.0/permissions/genie/{genie_space_id}",
        headers=headers,
        json={
            "access_control_list": [{
                "service_principal_name": sp_app_id,
                "permission_level": "CAN_RUN"
            }]
        },
        timeout=15
    ).raise_for_status()

To grant SELECT permissions on the data, you can use SQL:

GRANT USE CATALOG ON CATALOG main TO `<service_principal>`;
GRANT USE SCHEMA ON SCHEMA main.sales TO `<service_principal>`;
GRANT SELECT ON ALL TABLES IN SCHEMA main.sales TO `<service_principal>`

At this point, the SP has sufficient permissions to function (to query the underlying tables of the Genie space). However, we still have not defined the guardrails that control what data it can actually see. This is where ABAC policies come into play, enforcing which rows and columns are visible to the SP at query time.

Step 4: Enforce tenant isolation with ABAC

With identities and permissions in place, you can now enforce the actual tenant boundary. Because each tenant has its own SP, you can write ABAC policies that evaluate the current identity directly.

A common pattern is to maintain a mapping table between SPs and tenant IDs, then use a function that checks whether the current caller is authorized for a given row. For example:

CREATE OR REPLACE FUNCTION main.sec.filter_by_tenant(tenant_id STRING)
RETURNS BOOLEAN
RETURN EXISTS (
  SELECT 1
  FROM main.sec.tenant_acl t
  WHERE t.tenant_id = tenant_id
    AND t.principal = current_user()
);

You can then apply that function as a row filter:

ALTER TABLE main.sales.customer_orders
SET ROW FILTER main.sec.filter_by_tenant ON (tenant_id);

At that point, queries are automatically scoped at execution time. It does not matter whether the SQL was written by hand or generated by Genie — if the current SP is not allowed to see a row, that row will not be returned.

This is one of the strongest aspects of the pattern: enforcement stays close to the data rather than in the application layer.

Column masking follows the same idea. If you have fields such as SSNs, account numbers, or other sensitive attributes, you can apply a masking policy so those values are exposed only when the SP is explicitly authorized.

Step 5: Curate the Genie space for quality answers

Once the security model for your tenants is in place, you can focus on improving the quality of responses in your Genie space. A Genie space is only as good as the data model and business context it understands, so it is worth spending time curating it.

Focus on adding clear table and column descriptions, defining joins and key measures, and including business-friendly synonyms or metadata so Genie can interpret questions accurately. Provide a small set of high-quality example queries — typically five to ten — to demonstrate common questions and correct SQL patterns. Keep the space focused on a small number of well-understood tables, and use a Serverless SQL Warehouse when available for better performance and scalability.

See Curate an effective Genie space (AWS | Azure | GCP) for more best practices defining a new space.

Step 6: Authenticate per tenant at runtime

With provisioning complete, your runtime path becomes straightforward. When the application receives a request for a given tenant, it loads that tenant’s stored credentials and exchanges them for a Databricks access token using the OAuth client-credentials flow.

A simple example looks like this:

import requests

ACCOUNT_HOST = "https://accounts.cloud.databricks.com"
ACCOUNT_ID = "<your_account_id>"
CLIENT_ID = "<tenant_client_id>"
CLIENT_SECRET = "<tenant_secret>"

resp = requests.post(
    f"{ACCOUNT_HOST}/oidc/accounts/{ACCOUNT_ID}/v1/token",
    auth=(CLIENT_ID, CLIENT_SECRET),
    data={"grant_type": "client_credentials", "scope": "all-apis"},
    timeout=15
)

resp.raise_for_status()
token = resp.json()["access_token"]

These tokens are short-lived (roughly one hour), so your application should cache and refresh them automatically rather than requesting a new token for every call. If you are using the Databricks SDK, token handling is typically abstracted for you.

At this point, you have everything you need for secure tenant-scoped execution: a tenant-specific SP, a short-lived access token, and ABAC policies that evaluate that identity when the query runs.

Step 7: Call the Genie conversation API

You can now call Genie. The API is stateful, so the typical pattern is to start a conversation, receive a conversation ID and message ID, and then poll until the message completes and results are available.

The simplified flow looks like this:

import time
import requests

WORKSPACE_HOST = "https://<your-workspace-instance>.cloud.databricks.com"
WORKSPACE_TOKEN = os.environ["DATABRICKS_WORKSPACE_TOKEN"]  
SPACE_ID = "<your_genie_space_id>"

headers = {"Authorization": f"Bearer {WORKSPACE_TOKEN}"}

# 1. Start the Genie conversation
start = requests.post(
    f"{WORKSPACE_HOST}/api/2.0/genie/spaces/{SPACE_ID}/start-conversation",
    json={"content": "Top customers by revenue last month"},
    headers=headers,
    timeout=15
)
start.raise_for_status()

# Extract conversation and message IDs 
conversation_id = start.json()["conversation"]["id"]
message_id = start_data["message"]["id"]


# 2. Poll for the execution to finish
for _ in range(60):
    poll = requests.get(
        f"{WORKSPACE_HOST}/api/2.0/genie/spaces/{SPACE_ID}/conversations/"
        f"{conversation_id}/messages/{message_id}",
        headers=headers,
        timeout=15
    )
    poll.raise_for_status()

    status = poll.json()["status"]

    if status == "COMPLETED":
        break

    time.sleep(2)

In production, you will want to add robust retry logic, exponential backoff, timeout handling, and monitoring around failures or API limits. But conceptually, this is the full runtime loop: authenticate as the tenant, call Genie, let Genie generate SQL, and rely on ABAC to enforce tenant boundaries during execution.

Check out Best practices for using the Genie API (AWS | Azure | GCP) to maintain performance and reliability when using the Genie API.

Operating this pattern in production

Once your core flow is live, the heavy lifting shifts from initial implementation to ongoing operations. To keep your system secure, efficient, and easy to maintain, focus on four key areas.

1. Secret lifecycle management

Secrets are never a one-time setup step. They require regular rotation, with a reasonable rule of thumb being every 60 to 180 days. Fortunately, Databricks supports multiple active secrets per SP, enabling zero-downtime rotation. The overlap between old and new secrets is what keeps your production environment safe during the transition.

A step-by-step flow for rotating your secrets:

• Create a new secret and store it securely in your vault
• Update your application to use this newly generated secret
• Verify that all traffic has successfully switched over
• Revoke the old secret only after confirming the new one works

2. Clean deprovisioning

When a tenant is offboarded, you should carefully reverse the initial setup steps. Keeping this teardown process as part of your standard control-plane lifecycle makes the entire system much easier to reason about over time. Be sure to revoke active secrets, remove permissions, and optionally delete the SP entirely.

3. Observability and Genie space hygiene

As your platform usage grows, you need deep visibility into how the system is behaving. Keep a close eye on query patterns, performance bottlenecks, conversation volume, and audit signals tied to policy enforcement. You also need to maintain good space hygiene by deleting old records, especially since spaces often have strict conversation limits.

Note: As of the date of this publication, a Genie space has a limit of 10,000 conversations.

4. Evolving Your ABAC Model

Finally, expect your ABAC model to evolve as your data model changes. New tables, new sensitive fields, and new use cases will almost certainly require updates to your filters and masks. That is normal. The advantage of this architecture is that these changes occur in the governed policy and metadata layers, rather than by rewriting application filtering logic.

Operating this pattern in production

What makes this pattern highly effective is its clean separation of responsibilities, decoupling application logic from data governance. Service principals establish clear tenant identities, while OAuth provides a secure mechanism for your application to assume those identities at runtime. At the data layer, Unity Catalog ABAC policies dynamically enforce row and column-level access rules, freeing your application from managing complex, error-prone tenant filters. By combining these robust governance guardrails with the Genie Conversation API, you can confidently scale self-service analytics and securely deliver a rich, natural language data experience to external users directly within your application. Ultimately, this approach yields a multi-tenant architecture that is fundamentally easier to secure, scale, and maintain than pushing tenant filters deep into application code.

Resources

• Genie Conversation APIs/SDK
• For more on embedding Genie into your custom web apps with Genie Conversation APIs, check out Use Genie Everywhere with Enterprise OAuth.
• For more on building a Genie space from scratch to production-ready through benchmark evaluation and systematic optimization, check out How to Build Production-Ready Genie Spaces, and Build Trust Along the Way.

Embedding Genie API for a Multi‑Tenant Application was originally published in DBSQL SME Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

New features, new costs. In this article, we'll explore Genie Code pricing. We'll also look at when it can be free and when it can exceed the stated price.

The main difference between cloud solutions is that enabling a new feature often incurs additional costs. I've already written several articles about the cost of Databricks. And I was curious about the cost of using Genie Code.

Unfortunately, there's currently no convenient expense monitoring dashboard that would show exactly where money is being spent. There are only a few dashboards, but they provide a very high-level overview. However, if you're interested, you can find everything you need in Databricks Logs or system tables.

I ran some experiments with running Genie Code, and here's what the billing showed.

Experiment 1: Running Genie Code to write a text article.

I tried actively using Genie Code for 10 minutes without searching the Unity catalog or running queries. I asked him to write an article. It was clear he'd been poring over some Databricks documentation and wrote a 14,955-character article:

From the logs (system.access.assistant_events), it is clear that during this time, it managed to create 15 events, and all of this was almost free:

Experiment 2: Running Genie Code to write a script.

During the second experiment, I asked him to write a script, and he came up with a plan:

I also performed selections from tables and analyzed the results:

The experiment lasted 30 minutes, including adjustments, and, according to the logs, 79 events occurred.

In Databricks, all costs are displayed as DBU per hour, and we calculate the cost based on the pricing. When Genie starts, a separate cluster of the PREMIUM_ALL_PURPOSE_SERVERLESS_COMPUTE_* type is launched, and its cost is currently approximately $0.75 per DBU, depending on the region and cloud provider.

But in the end, the amount was approximately equal to 2 DBU:

After analyzing the detailed logs (system.access.audit), which contained over 900 events during this period, I found an interesting line:

{
  "virtual_cluster_size":"VirtualMedium",
  "billing_info":
    {"context":
      {
        "notebook_name":"/Users/myuser@gmail.com/.conversations/___DBCONVERSATION___(): Databricks ...",
        "notebook_id":"935626676034490"
      }
     },
     "custom_tags":{"NotebookId":"935626676034490"},
     "cluster_creator":"REPL_LAUNCHER",
     "kind":"SERVERLESS_REPL_VM",
     "autotermination_minutes":"15",
     "idempotency_token":"serverless-***********************",
     "platform_channel":"CLIENT-4"
}

Now we see that Genie Code spun up a separate VirtualMedium-sized cluster with an autotermination_minutes setting of 15 minutes to fulfill queries.

The tracking problem.

When I was running the experiment, I didn't run anything on Databricks that day except Genie Code to find its billing.

Currently, billing shows that the cluster was launched by the user themselves, not Genie Code. Unfortunately, the cost breakdown doesn’t include any additional labels or tags.
If your company uses clusters with tags and cost centers, you’ll need to add more users to accurately distribute them among teams.

Quite a big change coming to Databricks Genie. Starting July 8, 2026, Genie product usage beyond the free monthly allowance will move to pay-as-you-go billing. The usage is billed based on underlying LLM usage in DBUs, and budgets can be set up to track/control spend.

The transition from Assistant to Genie Code is a huge leap in productivity. Business users gain independence from data engineers for basic analytics, and engineers are freed from writing routine ad-hoc queries.

However, when an Agent appears instead of copying code from the Assistant, and can radically change your script, you need to be able to work with version control. Fortunately, Databricks already has built-in versioning for Notebooks, but it's best to connect to GitHub and commit successful scripts. Also, remember that if you give the Agent complete freedom, it can delete data in tables, jobs, and so on, or create a script that wastes your budget. It's a handy assistant that requires control.

Author: Maksim Pachkouski [Databricks MVP & Databricks Solutions Architect Champion]. Follow my posts on Medium and LinkedIn to stay up-to-date on the latest news in Data Engineering and learn about best practices.

Databricks Genie Code Pricing: The Real Cost of Agentic AI was originally published in DBSQL SME Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Databricks Genie is transforming how teams interact with data using natural language. But as adoption grows, so does the need for visibility: How are users interacting with your Genie spaces? What is the cost per query? Where are the friction points?

This blog (along with the accompanying Databricks Notebook) provides an approach to building a scalable, end-to-end observability workflow that answers these questions and much more. By leveraging Genie REST APIs and Databricks System Tables, this solution creates two powerful datasets that give you full transparency into your AI/BI environment. These tables can then be a part of your observability toolkit on Databricks!

How to use this blog

Consider this blog your launchpad for Genie observability techniques. It covers the essential logic you need to account for when building this workflow. The accompanying notebook contains all the code required to start experimenting immediately.

Pro Tip: Leverage Databricks Assistant to reverse engineer the code provided in the notebook to make it truly your own!

For Genie Space Administrators, these tables should be more than logs. They are a roadmap for evolving your data strategy. By analysing the genie_observability_main_table and dbsql_cost_per_query_table, you can drive continuous improvement in three key areas:

1. Refine Your Semantic Layer

User questions are the best indicator of what is missing from your data model. Use these insights to close the gap between user intent and available data:

• Curate with Precision: Identify questions where Genie struggled or failed. Use this feedback to add missing tables, define better join relationships, or provide specific SQL examples to guide Genie’s reasoning.
• Standardise Metrics: Spot recurring questions about specific KPIs (e.g., “churn rate” or “daily active users”). Centralise, promote and govern these common definitions into Databricks Metric Views, ensuring consistent logic across the entire organisation.

2. Optimise Engineering Patterns (“Shift Left”)

Observability helps you spot performance bottlenecks early. If you notice Genie frequently generating complex, expensive joins to answer simple questions, it might be time to “shift left”:

• Move semantic modeling logic upstream in your ETL pipeline.
• Pre-aggregate data or denormalise tables in the Silver/Gold layer to improve query performance and reduce latency for end users.

3. Master Operational Efficiency

With granular cost attribution, you can conduct more accurate cost analysis for chargebacks and usage monitoring:

• Attribute Costs: Accurately charge back compute costs to specific departments or Genie spaces based on actual DBU consumption.
• Manage Capacity: Identify peak usage times to ensure your SQL Warehouses are sized correctly, balancing performance with cost.

What this blog achieves

Throughout this post, we will demonstrate how to create two tables that are critical to Genie observability.

1. genie_observability_main_table

• What it tracks: Every message sent to your Genie spaces, including user questions, Genie’s natural language responses, and the generated SQL.
• Why it matters: It captures critical metadata like user feedback (thumbs up/down), error messages, and SQL statement IDs, etc, enabling you to pinpoint quality issues and improve Genie’s accuracy.

2. dbsql_cost_per_query_table

• What it tracks: Granular cost attribution for every SQL query executed across your workspaces.
• Why it matters: This allows you to attribute costs to Genie Spaces, by filtering out SQL queries and their costs that originate from a given Genie Space. You can now see the exact DBU consumption and USD cost for every question asked, simplifying chargebacks and ROI analysis.

Pre-requisites:

• This notebook/workflow is targeted towards a Genie space manager.
• Entitlements: You must have the Databricks SQL workspace entitlement. See Manage entitlements.
• Compute: CAN USE access on at least one pro or serverless SQL warehouse.
• Data access: SELECT privileges on the data used in the space.
• Genie space ACLs: At least CAN EDIT permissions on the Genie space. Genie space creators automatically have CAN MANAGE permissions on spaces they create. See Genie space ACLs.
• You should have enough permissions to create tables in a schema within a catalog of your choice
• Access to system tables is governed by Unity Catalog. Account admins have access to system tables by default. To allow a user to query system tables, an admin must grant that user USE and SELECT permissions on the system schemas.

Part 1: Capturing the Conversation (The API Layer)

To build the genie_observability_main_table, we need to extract the raw conversation history. We simplify this process by using the databricks-sdk for Python. The SDK handles authentication, pagination, and data typing automatically, making the extraction code cleaner and more robust than raw API calls. Note: The Databricks notebook also has an implementation that uses the python requests library.

Here is a snippet of the logic used to fetch and parse the messages using the WorkspaceClient:

from databricks.sdk import WorkspaceClient

def get_genie_observability_table(space_id: str):
    w = WorkspaceClient()
    
    # Get space metadata
    space = w.genie.get_space(space_id=space_id)
    space_name = space.title
    
    records = []
    
    # 1. List all conversations (SDK handles pagination automatically)
    # We set include_all=True to fetch conversations from all users
    conversations = w.genie.list_conversations(space_id=space_id, include_all=True)
    
    for conv in conversations:
        # 2. List all messages within the conversation
        messages = w.genie.list_conversation_messages(
            space_id=space_id, 
            conversation_id=conv.conversation_id
        )
        
        for msg in messages:
            # 3. Flatten the message object into a row for our table
            # (Helper function _extract_message_data parses attachments & feedback)
            record = _extract_message_data(msg, space_id, space_name, w)
            records.append(record)
            
    # Convert list of records to Spark DataFrame
    schema = _get_schema()
    return spark.createDataFrame(records, schema=schema)

Part 2: Attributing the Cost (The System Tables Layer)

Genie spaces incur costs based on the underlying compute (Databricks SQL Warehouse) used to execute the SQL generated from natural language queries. Because Databricks SQL bills for warehouse uptime rather than per-query execution, the “cost per query” is a derived approximation.

Accurately attributing this cost is complex, as multiple queries often run concurrently on a shared warehouse. To address this, we employ a “Time-Weighted” attribution strategy. This method calculates the total cost of a warehouse during a specific window and distributes it based on each query’s execution duration relative to the warehouse’s total active workload.

Link to the full SQL Script for calculating the Cost per Query is available here.

We run this script to create the dbsql_cost_per_query_table table.

To isolate costs for a specific Genie space, we utilise the query_source_id column in the dbsql_cost_query_table table. For queries originating from Genie, this ID corresponds directly to the unique space_id of the Genie space.

We attribute the final costs by performing an INNER JOIN between the cost table and the Genie space table (on query_source_id = space_id), followed by a SELECT DISTINCT operation to produce the final, accurate dataset.

Visualising Insights: Building the Dashboard & Meta-Genie

Once the notebook execution is complete, your two main tables are populated and ready to drive actionable insights. You can immediately leverage these to power a structured Databricks AI/BI Dashboard, or even create a “Meta-Genie” Space — pointing a new Genie space at these tables to ask questions about your Genie usage in natural language (Genie-ception).

To get started with a Dashboard, create a new AI/BI Dashboard and use the following SQL snippets to define your datasets. You can also view the dashboard JSON here.

Dataset 1: Engagement & Adoption

Use this dataset to visualise how end users are interacting with your Genie spaces. Key metrics to track include the number of unique users, conversation volume, and message distribution over time.

USE CATALOG IDENTIFIER(:catalog);
USE SCHEMA IDENTIFIER(:schema);
SELECT * FROM genie_observability_main_table
WHERE created_datetime >= :param.min
AND created_datetime <= :param.max
AND (
:space_id = 'ALL'
OR space_id = :space_id
)
ORDER BY
created_datetime DESC

Dataset 2: Cost Analysis

Use this dataset to visualise cost per user, cost per space, expensive query identification, and general ROI metrics.

USE CATALOG IDENTIFIER(:catalog);
USE SCHEMA IDENTIFIER(:schema);
SELECT DISTINCT
c.*
FROM dbsql_cost_per_query_table c
INNER JOIN genie_observability_main_table o
ON c.query_source_id = o.space_id
WHERE c.start_time >= :param.min
AND c.start_time <= :param.max
AND (
:genie_space_id = 'ALL'
OR o.space_id = :genie_space_id
);

Get Started

Ready to implement this in your environment? You can find the full source code, including the Python extraction logic and the complete SQL cost modelling in a Databricks Notebook format in our repository.

Link to Notebook: LINK

Explore Wider Observability

Genie is just one part of the Databricks Data Intelligence Platform. If you are interested in broader observability tooling, including Apps monitoring, Model Serving endpoint monitoring and more, explore our full Observability repository here: LINK

Repo Contributors: Sam Le Corre and Mike Dobing

Unlock Deep Observability & Cost Tracking for Databricks Genie was originally published in DBSQL SME Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

Level of detail expressions are now available in Databricks AI/BI Dashboards, which unlock a whole array of interesting analytical questions we can now answer directly within AI/BI.

Level of detail expressions let you specify a granularity at which aggregations are calculated which is different than the level of detail of your visualizations.

For me personally, this is one of those concepts that sounds really simple at first blush, but once you get into the details, dealing with calculations and visualizations that simultaneously involve multiple levels of granularity can get pretty tricky. Because of that, I wanted to walk through a basic, concrete example in a good amount of detail, to demonstrate the mechanics behind these expressions and ultimately build an intuition for when and how to use them properly.

To demonstrate, we will build a basic visualization based on the samples.bakehouse.sales_transactions table — this is a sample dataset that’s available within every Databricks workspace. The dashboard artifacts are available on this GitHub repo — feel free to try it on your own!

Note: This demo shows how to build level of detail expressions as custom calculations, which are local to an AI/BI dashboard. If these expressions need to be reused across multiple dashboards and use cases (which in a real-life context, they probably do!), the expressions can be defined in a Unity Catalog Metric View once available.

For the purposes of this blog, I’ll use a small sample of the dataset, just to make the math easier to follow. Here’s the dataset we’ll use for the following examples:

Setting the stage

Objective: I’m the VP of Operations at the Bakehouse company, a popular cookie franchise. I’m interested in doing an analysis on how the different franchises perform relative to one another, so I work with my team to come up with metrics to measure performance. Initially, I want to focus on Average Sale Prices of the product and how they compare across franchises:

• Average Sales Price (ASP) for a single franchise
• Average Sales Price (ASP) across all franchises (global)
• Difference between ASP for a single franchise and all franchises: Provides a relative measure of how a franchise’s ASP compares to the average

I see that these metrics require dealing with 2 different levels of granularity: at the franchise level and at the global level (all franchises). For me, that’s an indication that I should consider using a level of detail expression (from now on, I’ll abbreviate to LOD expression), since they’re designed for working with multiple levels of granularity (or, levels of detail; hence the name 🙂)

There are two ways we can go about this, which align with the Fixed and Coarse level-of-detail calculations available in AI/BI.

Primer on important concepts:

Before we get deep into the nuances of LOD concepts, let’s take a quick aside to explain three important, foundational concepts:

Fields vs. Measures:

There are generally two types of calculations in AI/BI (and any other BI tool, for that matter):

• Fields: Row-level transformations. Conceptually, these can be calculated whenever the source dataset is refreshed.
• Measures: Calculations that aggregate something to a different level of granularity (e.g. AVG, SUM, etc.). Conceptually, these can only be calculated at runtime, since the level of granularity that the aggregation needs to consider will depend on the visualization grouping or filters from the dashboard.

Visualization groupings vs. filters

• When I use the term “visualization groupings”, I’m referring to the fields that are used in the visualization, which tell the visualization the granularity at which to calculate the Measures.
• Filters, in this case, refers to a simple Filter that would apply to a visualization to control the subset of data the visualization can display.

Let me show you a concrete example to illustrate how these work. Here are two pivot tables, both calculating the Average of the totalPrice field (we’ll call this Average Sales Price):

Notice that in both visuals, the ASP of Biscotti is $4. They both effectively filter to product = “Biscotti”, but they achieve this differently:

Table 1 (Visualization Grouping):

• The product field is added to the Rows of the pivot table
• Each “row” in the table represents a different product
• For the Biscotti row specifically, only Biscotti transactions are aggregated
• The filtering happens implicitly based on which row we’re looking at

Table 2 (Filter):

• No fields are added to the Rows (no visualization grouping)
• A dashboard filter is applied: product = “Biscotti”
• All records in the underlying dataset are filtered to just Biscotti transactions
• Then the aggregation is applied to what remains

Key takeaway: From a calculation perspective, visualization groupings and filters operate the same way. They both control which subset of data gets aggregated. The difference is:

• Visualization groupings apply filtering per row in the visual (e.g., each product gets its own filtered aggregation)
• Filters apply filtering to all rows in the visual at once (e.g., the entire visual only sees Biscotti data)

Since they function equivalently for calculation purposes, I’ll use filters in the examples below to keep things simple.

Mental model for how visualization groupings & filters impact measures

The section above reveals a helpful mental model to think about how filters (and visualization groupings) impact a measure calculation:

Step 1 — Filter the data: The filter condition is applied to the underlying dataset, including any calculated fields. This can be:

• The whole dataset (in the context of a dashboard filter), or
• A subset relevant to a specific row (in the context of a visualization grouping)

Step 2 — Aggregate: With the filtered dataset in hand, apply the Measure’s aggregation logic (AVG, SUM, etc.)

Keep this two-step model in mind as we explore how LOD expressions work with our sample dataset.

Fixed Level-of-Detail Expressions:

Defining the expressions

We’ll start out by creating a Pivot Table, since it’s easier to see the values and how they’re calculated. I know it’s boring, but trust me, it will make it easier to see how the calculations work. Once we build a Pivot Table, we can use these expressions in any visual type we see fit.

Let’s get started:

1. First, let’s create the franchise-level Average Sales Price measure that we’ll compare against this global average. Create a Measure called asp (short for Average Sales Price) with the syntax:

AVG(totalPrice)

Create a pivot table with franchiseID on the Rows, and asp on the Values.

How it’s calculated: This is super straightforward. For each franchiseID in the dataset, the visualization layer calculates the average of the totalPrice field.

2. Next, we’ll calculate the Average Sales Price across all franchises, as a fixed level-of-detail expression:

Fixed level-of-detail expressions are defined as expression which:

Aggregate over pre-defined set of dimensions specified in the expression itself, ignoring all other dimensions in the visualization.

Let’s create the following fixed LOD expression to define the average sales price across all franchises.

Create a custom calculation called asp_all_franchises_fixed. This will create a “Field” (as opposed to a “Measure” in the Dataset), which means it can be calculated for each row in the dataset (whereas “Measures” are aggregated in the context of the visual).

AVG(totalPrice) OVER ()

Here’s what the dataset looks like with the new calculated field:

Intuitively, this tells us that the Average Sales Price across all franchises is $4.50.

3. Finally, let’s put it all together with the comparison between the Average Sales Price for this franchise vs. the global average.

Let’s create a custom calculation called asp_diff_fixed with the following syntax:

MEASURE(asp) - ANY_VALUE(asp_all_franchises_fixed)

This custom calculation produces a Measure, which means it is an aggregated value which is calculated at runtime, in the context of the visualization’s dimensions and any relevant filters. Because the asp_all_franchises_fixed field is fixed across all rows in this dataset, we can use the ANY_VALUE function to allow this custom calculation to be used in a measure calculation (see docs for reference).

Intuitively, this tells us the difference between the average sales price for this franchise ($4, for Franchise 1) and the average sales price for all franchises ($4.50), resulting in a value of -$0.50, meaning Franchise 1’s ASP is $0.50 below the global average.

Adding Filters

Ok, great. This is super helpful and shows that we can show different levels of granularity in the same dataset and visualization (the franchise grain and the global grain). But let’s take it a step further and add additional filters to our visualization and observe how the calculations adapt to the filter selections. We’ll add a filter to the visualization for the “Product” field.

What happens when we apply a filter for the “Biscotti” product? Let’s use the mental model above to think through this. Filters follow the two-step process of: 1) filter the underlying data and 2) aggregate the filtered data.

Concretely, let’s see how that works when we apply the filter for “Biscotti”.

1. After filtering the dataset for product = ‘Biscotti’, we get this:

2. Now, with this filtered dataset, we’ll re-apply the aggregations:

Average Sales Price:

• Franchise 1: $3
• Franchise 2: $5

Diff. Average Sales Price (using calculation MEASURE(asp) — ANY_VALUE(asp_all_franchises_fixed))

• Franchise 1: $3 — $4.50 = $-1.50
• Franchise 2: $5 — $4.50 = $0.50

Intuitively, this means that Franchise 1 has an ASP for the Biscotti product that’s $1.50 less than the global ASP for Biscotti, while Franchise 2 is $0.50 more than the global ASP.

Notice that in this calculation, the global ASP stays constant across all the various circumstances: It’s $4.50 when we’re looking at all transactions unfiltered, it’s $4.50 when we’re looking at just Biscotti, it’s $4.50 when we’re looking at both Franchise 1 and Franchise 2. Conceptually, this is why it’s a fixed level-of-detail calculation. It’s fixed on the dimensions that we’ve predefined in the expression, which in this case, is no dimension because we defined it as a global average (with the syntax: AVG(totalPrice) OVER ()). However, we could just as easily fix the calculation on any other dimension that is relevant for our analysis (more on this in the next section).

Takeaway

And that’s it! We’ve built out the Fixed LOD calculation to calculate the difference between the franchise-level ASP and the global-level ASP. We observed that the expression was calculated independently of the visualization groupings and filters in the dashboard: no matter what fields or filters we added, the global ASP across all franchises was always $4.50. In many use cases, this may be exactly the behavior you want to see.

However, in other cases, we may need the LOD expression to adapt to the visualization groupings or filters. As an example, let’s say we’re interested in answering the question:

“What is the Average Sales Price of the Biscotti product across all franchises?” With the fixed calculation, this doesn’t quite work. Since we’ve fixed the expression at the global level, we can only calculate that expression at the global granularity. To fix this problem (pun intended), we can turn to the other type of LOD expressions: Coarse.

Coarse Level-of-Detail Expressions

Why do we need another type of expression?

Considering the scenario above, where we’re trying to answer the question “What is the Average Sales Price of the Biscotti product across all franchises?”, we could certainly add a new fixed LOD expression that calculates the ASP for each product (e.g. with an expression like AVG(totalPrice) OVER (PARTITION BY product)).

But then, let’s say we want to ask an even more granular question like “What is the Average Sales Price of the Biscotti product across all franchises on Tuesdays?” Now, we’re adding another dimension to the equation (the date field). Again, we can certainly handle this by creating another expression, which calculates the ASP by product and by day of the week.

But let’s hold on just a darn minute.

We already have 3 expressions to handle 3 very similar (but slightly different) questions. As this dashboard gets into the hands of end users, I anticipate that there will be a lot of different variations on these questions and interesting ways to slice and dice this data. For each of those requests, I don’t want to need to create a new expression every time. What if there was a way to dynamically re-generate the LOD expressions to adapt to the visualization groupings and filters in the dashboard? That’s exactly the motivation for when to use a coarse level-of-detail expression (I prefer to think of these as “dynamic” LOD expressions).

Defining the expressions

Let’s go ahead and build one, we’ll follow the same three steps as above to calculate the different components of this Pivot Table:

1. Average Sales Price: This will be the same as in the Fixed section
2. Average Sales Price for all franchises: Create a custom calculation called asp_all_franchises with the following syntax:

AVG(totalPrice) AGGREGATE OVER (ORDER BY franchiseId ALL)

Whoa — we’ve got some new syntax! AGGREGATE OVER is the keyword to indicate that this is a coarser LOD expression (for more details on syntax, refer to the linked docs). This generates a measure, so it will be aggregated at query-time based on the context of the visualization.

Intuitively, what this means is that we want to calculate the Average Sales Price for all dimensions in the visualization grouping or filter, EXCEPT for franchiseId, where we want to ignore that filter and consider ALL values of franchiseId.

3. Diff. of Average Sales Prices: Putting it all together, we’ll create a simple custom calculation called asp_diff with the syntax:

MEASURE(asp) - MEASURE(asp_all_franchises)

Let’s build a quick Pivot Table with these 3 fields and see how it observes compared to the fixed LOD expressions.

Without any additional visualization groupings or filters, this works exactly the same way as it did for the Fixed LOD expression. Let’s add that filter for product and walk through how the difference of ASP measure will be calculated now with a filter on product = ‘Biscotti’. Again, measures are conceptually evaluated using a two step process: 1) Filter the dataset and 2) Aggregate the measures.

1. When the dataset is filtered to Biscotti, we get the same dataset as before:

2. Re-apply the aggregations on the filtered dataset:

Average Sales Price (same as before)

• Franchise 1: $3
• Franchise 2: $5

Average Sales Price All Franchises (Coarse LOD): With the filtered dataset, we’ll get the Average of the totalPrice field across all franchises (in other words, we’ll ignore any filters on the franchiseId field, but will respect all other filters). In this case:

• Franchise 1: $4
• Franchise 2: $4

Diff. Average Sales Price:

• Franchise 1: $3 — $4 = $-1
• Franchise 2: $5— $4 = $1

This means that the ASP for the Biscotti product across all franchises is $4, so Franchise 1 sells it for $1 less than the average and Franchise 2 sells it for $1 more than the average.

Takeaway

We’re getting a different result than with the fixed LOD expression because here, we’re recalculating the expression based on the context of the visual, which in this case includes the product filter. We could add multiple other dimensions (and combinations of dimensions) to this visual, and the expression will always recompute within the context of those dimensions. This is pretty powerful!

The point is not that one method (Fixed vs. Coarse) is better than the other: they are both valid approaches, it really just depends on the question we’re trying to answer.

Comparing against a global, fixed ASP makes sense if we’re doing a high-level analysis/benchmarking exercise to understand the general trend of the business. Comparing against a dynamic (or coarser) average, which adjusts to product dimension selections is helpful if we’re focused on understanding trends within a specific product line, where the global average isn’t useful, but understanding the ASP for that specific product across franchises is. Different types of analysis, but both are completely valid and useful in their own right. Use the right type of expression for the job!

Summary

We’ve walked through the motivation and the mechanics for how the fixed and coarse LOD expressions work in AI/BI. Next time you’re building a dashboard and need to represent various levels of granularity in the same visualization or dataset, keep these two methods in mind.

Use Fixed when the dimensions are pre-determined and do not need to be re-calculated in the context of the visualization. Use Coarse when the context of the visualization matters a lot, so the expressions should be recomputed to reflect that context.

Mental model: If a user adds a filter or groups by a new dimension, should my comparison point change or stay the same?

If you have any questions, reach out to your Databricks Account team or email me at justin.kolpak@databricks.com.

Ok, this blog made me hungry, I’m going to go eat a Biscotti! Thanks for reading.

Level of Detail Expressions in AI/BI: A Practical Guide was originally published in DBSQL SME Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Manually setting up access permissions for hundreds of tables is a tedious task. Databricks recently introduced the Manage Access Request Destinations (Public Preview) feature, which significantly simplifies the approval process. In part one, I covered how to audit and export current approvers. Now, let's move on: in this article, I'll show you how to close the automation loop—take a ready-made list from Excel and mass-assign approvers to schemas and tables in Unity Catalog via the API.

Typically, the Data Governance department maps tables and responsible parties using external registries (Collibra, Excel, etc.). To avoid routine manual configuration, we need a synchronization script.

As an example, we'll implement uploading from Excel. This is a general-purpose scenario, and suitable for most teams. This is what the automated workflow looks like:

Importing approvers from Excel

We're implementing a scenario in which a script reads mappings of schemas and approvers from an Excel file and applies them to Databricks.

We'll use email for notifications (the API allows up to 5 emails per object). Although the Manage access request destinations feature allows you to send requests to Slack or Teams, for simplicity of demonstration, we'll use email.

Let's assume we have an Excel file where a list of schemes and their owner approvers is listed on one tab:

Another tab contains a list of tables and also the owner approvers, since the entire "qa.new_schema" schema may be owned by "MissisClaus@gmail.com", while a table within the same schema may be owned by "elf1@gmail.com":

A connector from Databricks to SharePoint was recently released, and I'd suggest using this method to load an Excel file into Databricks. To simplify the example, I'll simply upload it to Volume.

As I wrote in the previous article, the API SDK needs to be updated:

%pip install -U databricks-sdk
dbutils.library.restartPython()

As always, first we make the necessary imports:

import pytz, requests, pyarrow as pa, json
from databricks.sdk import WorkspaceClient
from datetime import datetime, timezone
from pprint import pprint
from pyspark.sql import Row
from collections import defaultdict
from pyspark.sql.functions import *
from pyspark.sql.types import *
from databricks.sdk.service.catalog import (AccessRequestDestinations, Securable, NotificationDestination, SecurableType, DestinationType)

wc = WorkspaceClient()

Below, I will provide a code example if you want to install the approver for a specific table and test how it works (YOU CAN SKIP IT):

table_full_name = "qa.new_schema.new4" ### your table

payload = AccessRequestDestinations(
    securable=Securable(type=SecurableType.TABLE, full_name=table_full_name),
    destinations=[
        NotificationDestination(
            destination_type=DestinationType.EMAIL,
            destination_id="data-approvers@company.com", ### your email
        )
    ],
)

wc.rfa.update_access_request_destinations(payload, update_mask="destinations")

To load the data, I'll use Databricks' built-in capabilities for reading Excel directly into a Spark DataFrame. In our script, we'll separate the logic: we'll calculate the chart sheet separately from the table sheet.

The choice of tool here is not critical—if you prefer, this step can be implemented using Pandas or any other library.

df_excel_schemes = spark.read.option("headerRows", 1).option("dataAddress", "schemes").excel('/Volumes/dev/edl_test/volume_test/data_approvers_tst.xlsx')
df_excel_schemes.createOrReplaceTempView("EXCEL_SCHEMES")

df_excel_tables = spark.read.option("headerRows", 1).option("dataAddress", "tables").excel('/Volumes/dev/edl_test/volume_test/data_approvers_tst.xlsx')
df_excel_tables.createOrReplaceTempView("EXCEL_TABLES")

The Unity Catalog uses a three-level namespace: catalog.schema.table. This allows us to automatically determine the object type without explicitly querying Excel. Let’s write a small utility function that classifies an object (Catalog, Schema, or Table) simply by counting the number of dots in its full name:

def infer_securable_type_by_dots(full_name: str) -> SecurableType:
    dots = full_name.count(".")

    if dots == 2:
        return SecurableType.TABLE
    if dots == 1:
        return SecurableType.SCHEMA
    if dots == 0:
        return SecurableType.CATALOG

    raise ValueError(f"Error '{full_name}' (dots: {dots})")

In a real production environment, no one will likely install an approver for the entire catalog, so we’ll focus on tables and schemas.

We’ll create a function that takes a row from a DataFrame, clears the list of email addresses of empty values, and sends a request via the Databricks SDK. To ensure the process doesn’t stop due to a single invalid entry, we’ll add exception handling and save all errors in a separate list for the final report.

error_rows = []
success_counts = {
    "schema": 0,
    "table": 0
}

def set_approvers_from_row(row):
    securable_type = infer_securable_type_by_dots(row.TABLE)
    print(f"Processing: {row.TABLE}, inferred type: {securable_type}")

    if securable_type == SecurableType.SCHEMA:
        level = "schema"
    elif securable_type == SecurableType.TABLE:
        level = "table"
    else:
        level = "catalog"

    emails = [
        row.EMAIL_1,
        row.EMAIL_2,
        row.EMAIL_3,
        row.EMAIL_4,
        row.EMAIL_5,
    ]
    emails = [e.strip() for e in emails if e and e.strip()]

    if not emails:
        error_rows.append(Row(FULL_NAME=row.TABLE, LEVEL=level, ERROR="No EMAILs provided"))
        return

    payload = AccessRequestDestinations(
        securable=Securable(type=securable_type, full_name=row.TABLE),
        destinations=[NotificationDestination(destination_type=DestinationType.EMAIL, destination_id=e) for e in emails],
    )

    try:
        wc.rfa.update_access_request_destinations(payload, update_mask="destinations")
        success_counts[level] += 1

    except Exception as e:
        error_rows.append(Row(FULL_NAME=row.TABLE, LEVEL=level, ERROR=str(e)))

Now let’s create the final lists for the schemas and tables. At this stage, we do two things:

• We trim the fields of spaces (using the trim function), as Excel files often contain unnecessary indentation.
• We collect the data in local memory (.collect) to pass it to the Databricks API client.

schemas = df_excel_schemes.select(
    trim("SCHEMA").alias("SCHEMA"),
    trim("EMAIL_1").alias("EMAIL_1"),
    trim("EMAIL_2").alias("EMAIL_2"),
    trim("EMAIL_3").alias("EMAIL_3"),
    trim("EMAIL_4").alias("EMAIL_4"),
    trim("EMAIL_5").alias("EMAIL_5"),
).collect()

tables = df_excel_tables.select(
    trim("TABLE").alias("TABLE"),
    trim("EMAIL_1").alias("EMAIL_1"),
    trim("EMAIL_2").alias("EMAIL_2"),
    trim("EMAIL_3").alias("EMAIL_3"),
    trim("EMAIL_4").alias("EMAIL_4"),
    trim("EMAIL_5").alias("EMAIL_5"),
).collect()

For each row in our lists, the previously written function is called. This is when the actual API calls and access settings are updated.

for row in schemas:
    set_approvers_from_row(row)

for row in tables:
    set_approvers_from_row(row)

Finally, we display the work statistics. The script will show the number of successfully updated schemas and tables. If any errors occurred, we'll collect them in a separate DataFrame and display them in a table for easy analysis of incidents.

print("===== SUMMARY =====")
print(f"Schemas successfully updated : {success_counts['schema']}")
print(f"Tables successfully updated  : {success_counts['table']}")
print(f"Total failures               : {df_errors.count()}\n")
print("===== ERROR DETAILS:")

df_errors = spark.createDataFrame(error_rows)
df_errors.display()

I've demonstrated the basic automation logic. Of course, in production, you'll need to replace display() and add audit logs to the database table or send alerts to Slack when an update fails.

• Link to the Notebook with source code

Author: Maksim Pachkouski [Databricks MVP & Databricks Solutions Architect Champion]. Follow my posts on Medium and LinkedIn to stay up-to-date on the latest news in Data Engineering and learn about best practices.

Set Access Request Approvers in Databricks from Excel via API was originally published in DBSQL SME Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Databricks recently added the Manage access request destinations feature (Public Preview), but the documentation and YouTube video talk about managing it through the interface. This is a visual demonstration for purposes, but for real production and automation, we need a different solution — and that’s what I’ll tell you about.

To understand the value of the new approach, consider the legacy process used in most companies. If a user needs access to a table, they create a request in Jira, list the tables they need to access, specify the group they want to access, and attach an approval from the owner. The administrator then writes a script and grants permissions.

Below is an example of what the processes look like before and after, as well as how much time each action typically requires. As we can see, the new process consumes half as much time and resources:

Manage access request destinations

Take a quick look at how this new feature works in the Databricks interface. The administrator assigns an owner to a catalog, schema, or table:

In the interface, you can specify up to 5 emails (resource owners), and this can also be not only email, but also Slack, Teams, and other applications:

If an owner is configured for a directory, schema, or table, the settings are inherited down the hierarchy to the table. Now the user can request access. It’s best to request access for a group, for your entire team. Below is an example:

A notification is sent via email, and the owner grants access to read or modify the table in a couple of clicks:

I’ll explain how to create a script to extract a list of tables and view their assigned owners. In the next part, I’ll show you how to download a list of table owners and tables from Excel and install them in Databricks via the API. Subscribe to my blog on Medium to stay up to date.

How to extract current approvers via API?

To manage requests through code, we’ll need the rfa (Request for Access) API. Your current version of the Databricks SDK likely doesn’t have it yet, as the feature is quite new. So, first, update the package:

%pip install -U databricks-sdk
dbutils.library.restartPython()

Next, we make the necessary imports:

import pytz, requests, pyarrow as pa, json
from datetime import datetime, timezone
from pprint import pprint
from collections import defaultdict
from pyspark.sql import Row
from pyspark.sql.functions import *
from pyspark.sql.types import *
from databricks.sdk import WorkspaceClient
from databricks.sdk.service.catalog import (AccessRequestDestinations, Securable, NotificationDestination, SecurableType, DestinationType)

wc = WorkspaceClient()

Using the API, we extract a list of directories, schemas, and tables. Inheritance occurs hierarchically, and it’s difficult to determine at what level permissions are set, so we collect the effective_level. We also exclude system tables, as they are of no interest to us.

EXCLUDE_CATALOGS = {"samples", "system"}
EXCLUDE_SCHEMAS = {"information_schema"}
rows = []

def get_rfa(securable_type: str, full_name: str) -> dict:
    return wc.rfa.get_access_request_destinations(securable_type, full_name).as_dict()

def norm_destinations(rfa: dict):
    ds = (rfa or {}).get("destinations") or []
    return sorted({(d.get("destination_type"), d.get("destination_id")) for d in ds})

def pick_level(dt, ds, dc):
    if not dc and not ds and not dt:
        return None
    if dt and dt != ds:
        return "table"
    if ds and ds != dc:
        return "schema"
    if dc:
        return "catalog"
    return None

for cat in wc.catalogs.list():
    if cat.name in EXCLUDE_CATALOGS:
        continue

    rfa_catalog = get_rfa("catalog", cat.name)
    dc = norm_destinations(rfa_catalog)

    for sch in wc.schemas.list(catalog_name=cat.name):
        if sch.name in EXCLUDE_SCHEMAS:
            continue

        schema_full = sch.full_name
        rfa_schema = get_rfa("schema", schema_full)
        ds = norm_destinations(rfa_schema)

        for tbl in wc.tables.list(catalog_name=cat.name, schema_name=sch.name):
            table_full = tbl.full_name
            rfa_table = get_rfa("table", table_full)
            dt = norm_destinations(rfa_table)

            effective_level = pick_level(dt, ds, dc)

            if effective_level == "table":
                effective = rfa_table
            elif effective_level == "schema":
                effective = rfa_schema
            elif effective_level == "catalog":
                effective = rfa_catalog
            else:
                effective = None

            rows.append(Row(
                catalog=cat.name,
                schema=schema_full,
                table=table_full,
                effective_level=effective_level,
                effective_access_request_destinations_json=(json.dumps(effective, ensure_ascii=False) if effective is not None else None)
            ))

df_rfa = spark.createDataFrame(rows)
df_rfa.createOrReplaceTempView("TABLE_ACCESS_DESTINATIONS")

Using effective_level, we can see at what level the access query settings were set for each table:

Let’s assume we work for a large enterprise company, so we only allow email, and we don’t check other parameters. Let’s break out the five emails into separate columns.

df_dest_emails = spark.sql("""
    SELECT
        catalog,
        schema,
        table,
        effective_level,
        get(emails, 0) AS EMAIL_1,
        get(emails, 1) AS EMAIL_2,
        get(emails, 2) AS EMAIL_3,
        get(emails, 3) AS EMAIL_4,
        get(emails, 4) AS EMAIL_5
    FROM (
        SELECT
            *,
            TRANSFORM(
                FILTER(
                    FROM_JSON(
                        effective_access_request_destinations_json,
                        'STRUCT<destinations:ARRAY<STRUCT<destination_id:STRING,destination_type:STRING>>>'
                    ).destinations,
                    d -> d.destination_type = 'EMAIL'
                ),
                d -> d.destination_id
            ) AS emails
        FROM TABLE_ACCESS_DESTINATIONS
    )
""")

df_dest_emails.createOrReplaceTempView("TABLE_ACCESS_DESTINATIONS_EMAILS")

Next, we need to get a list of all tables. For this, we’ll use the Databricks system table.

df_all_tables = spark.sql("""
    SELECT
        CONCAT_WS('.', table_catalog, table_schema, table_name) AS table,
        table_owner,
        table_type   
    FROM system.information_schema.tables
    WHERE
        table_catalog NOT IN ('samples', 'system')
        AND table_schema <> 'information_schema'
    ORDER BY table
""")

df_all_tables.createOrReplaceTempView("ALL_TABLES")

All that remains is to make a join and find tables where the approver is not specified:

%sql
SELECT
  AT.table as TABLE,
  TAD.effective_level as LEVEL,
  TAD.EMAIL_1,
  TAD.EMAIL_2,
  TAD.EMAIL_3,
  TAD.EMAIL_4,
  TAD.EMAIL_5
FROM ALL_TABLES as AT
LEFT JOIN TABLE_ACCESS_DESTINATIONS_EMAILS as TAD
  ON AT.table = TAD.table
ORDER BY table

From the results, we see that there are rows where the approver columns are empty:

• Link to the full Notebook code

Why do we need a configuration export?
Downloading current settings is useful in several scenarios:

1. Migration to IaC (Infrastructure as Code). If access was configured manually for some time, exporting will help obtain the current system state (a “snapshot”) and translate it into automated scripts.
2. Security audit. For example, to ensure that users are requesting access only to Views, and not to raw tables.
3. Updating responsible persons. If an approver leaves, exporting will allow you to quickly find all tables where that employee is listed as the approver and assign a replacement.

Often, data on responsible parties is stored not in Databricks, but in Data Governance registries (Collibra, Excel). In the next post, I’ll show you how to automate the transfer of this information: we’ll learn how to download a configuration file from Excel and apply it to tables via the API.

Author: Maksim Pachkouski [Databricks MVP & Databricks Solutions Architect Champion]. Follow my posts on Medium and LinkedIn to stay up-to-date on the latest news in Data Engineering and learn about best practices.

Databricks Access Requests: Export Configured Approvers (Destinations) with the API in Unity… was originally published in DBSQL SME Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Author: Manuel Beuttler, Delivery Solutions Architect @ Databricks

Introduction: Schema Drift and Version Control Challenges

In modern data platforms like Databricks, managing schema changes can be a significant headache, often leading to schema drift and broken downstream pipelines. This challenge is a core problem faced by many data teams that seek a robust solution for version control and Data Product lifecycle management.

Without a disciplined approach, changes made manually in the UI or via ad hoc scripts create a “wild west” environment. A simple column rename in a development environment may go undocumented, only to cause a critical pipeline failure when the code is moved to production.

The solution lies in establishing a DevOps pipeline where every schema modification, from a simple column comment update to a major table alteration, is version-controlled, automated, and subject to a formal review and approval process.

While our previous articles (Database Change Management and Advanced Schema Management) focused on what Liquibase can do and how to execute commands, in this one, we will walk through an example architecture that leverages Liquibase within an Azure DevOps CI/CD pipeline to govern schema changes in a Databricks Unity Catalog (UC) environment and build production-ready data products.

About Liquibase

Liquibase is an open source database change management tool that helps development teams track, version, and deploy database schema changes in a controlled and automated way. It works by storing all database modifications (called “changesets”) in text-based files that are version-controlled alongside application code. Liquibase automatically translates these platform-independent change definitions (in SQL, XML, YAML, or JSON) into database-specific SQL. It supports various database systems as targets and offers additional features and capabilities, including integration with CI/CD pipelines for automated deployments across different stages, rollback support, drift detection to identify unauthorized database changes, and comprehensive tracking tables that ensure each change is applied only once.

Solution Architecture: The key components

The goal is to build an architecture that establishes a bridge between code repositories and the data warehouse, treating database schemas with the same rigor as application code. Our solution relies on four key components:

1. Source Control Repository (e.g., Git): Stores the “source of truth” for the database, specifically the Liquibase change log files that define the schema state.
2. CI/CD Pipeline (e.g., Azure DevOps): Our orchestration component and approval gateway. Including the binaries (Liquibase Docker Container) and execution environment that runs the schema migration logic, ensuring consistency across deployments.
3. Databricks Unity Catalog (UC): The central governance layer that serves as the target for all schema changes, managing tables, permissions, and metadata.
4. Databricks SQL Warehouse: Acts as the compute engine and connection endpoint, allowing Liquibase to execute JDBC commands against the Databricks environment.

Next, let’s look deeper into the individual components and how to tie them together.

Repository Structure and Organization

Our starting point is a well-organized repository as a foundation for maintainable schema management. The example structure below separates concerns: binaries for dependencies, templates for reusability, pipeline definitions for orchestration, and data products for actual schema changes.

Example Folder Structure

repository/
├── bin/
│   ├── DatabricksJDBC42.jar
│   └── liquibase-databricks-1.4.2.jar
│
├── templates/
│   └── liquibase-template.yml
│
├── pipelines/
│   └── azure-pipelines.yml
│
└── data_products/
    ├── changelogs/
    │   ├── fleet/
    │   │   ├── 20241215-01-create-fleet-table.sql
    │   │   ├── 20241220-01-add-mileage-column.sql
    │   │   └── 20241230-01-add-vehicle-type-comment.sql
    │   │
    │   ├── trips/
    │   │   ├── 20241215-01-create-trips-table.sql
    │   │   └── 20241222-01-add-driver-id-column.sql
    │
    └── root_changelog.xml

Folder Breakdown

bin/: Contains the necessary JDBC drivers and Liquibase extensions.

• DatabricksJDBC42.jar: The official Databricks JDBC driver that enables Liquibase to connect to the SQL Warehouse.
• liquibase-databricks-1.4.2.jar: The Liquibase extension specifically designed for Databricks, which handles Unity Catalog-specific SQL syntax.

templates/: Contains reusable YAML templates that abstract the Liquibase Docker execution logic. This keeps the main pipeline file clean and promotes consistency across multiple pipelines if you manage multiple catalogs or schemas.

pipelines/: Contains the Azure DevOps pipeline definition(s). This is where the orchestration logic lives, defining triggers, stages, variables, and approval gates.

data_products/: The heart of the repository. Each subfolder represents a logical data product (typically corresponding to a table or a related group of tables). Within each product folder we have:

• root_changelog.xml: The master changelog file that Liquibase reads. It references all individual SQL files in the changelogs/ folder, maintaining the correct execution order.
• changelogs/: By using this extra level in the folder hierarchy, we only have to reference it once in the root_changelog.xml and every new subfolder (f.e. fleet or trip) will be automatically added. If we don’t want to have this redundant level, we will have to reference each of them individually.
• [fleet|trip|{table}]/: Contains individual SQL files with the actual schema definitions, so called changelogs (we will cover them in detail later). Each file is named with a timestamp and sequence number for easy ordering (e.g., YYYYMMDD-NN-description.sql).

This structure easily scales: as new data products are added, simply create a new folder under changelogs/ and start committing SQL files.

CI/CD Pipeline Definition

For better readability and maintainability, we split the pipeline definition into two files, encapsulating the logic for running the Liquibase Docker container and other reusable parts into a YAML template.

The template mounts the necessary volumes (change logs and drivers), uses the master changelog as the entry point and constructs the JDBC connection string dynamically:

parameters:
 - name: action
   type: string
 - name: displayName
   type: string

steps:
 - script: |
     # Use latest docker liquibase image
     docker run \
       -v $(Build.SourcesDirectory)/data_products:/liquibase/changelog \
       -v $(Build.SourcesDirectory)/bin:/liquibase/lib \
       liquibase \
       --classpath="/liquibase/changelog:/liquibase/lib/DatabricksJDBC42.jar:/liquibase/lib/liquibase-databricks-1.4.2.jar" \
       --url="$(DATABRICKS_SQL_ENDPOINT)ConnCatalog=$(CATALOG);ConnSchema=$(SCHEMA);UID=token;PWD=$(PAT);" \
       --changeLogFile=root_changelog.xml \
       ${{ parameters.action }}
   displayName: ${{ parameters.displayName }}

After defining the template, we can reference and use it in our main pipeline definition: the azure-pipelines.yml file, which orchestrates the entire workflow. It is triggered on commits to the main branch and executes a two-stage process: validation followed by deployment with an approval gate.

trigger:
  - main

stages:
  - stage: validation
    displayName: 'Validate Schema Changes'
    dependsOn: []
    jobs:
      - job: reviewChanges
        displayName: 'Review Changes'
        pool:
          name: ${AGENT_POOL}
        
        variables:
          - name: CATALOG
            value: ${TARGET_CATALOG}
          - name: SCHEMA
            value: ${TARGET_SCHEMA}
          - name: SQL_ENDPOINT
            value: ${DATABRICKS_SQL_ENDPOINT_DEV}
          - name: DATABRICKS_CLIENT_ID
            value: $(DATABRICKS_CLIENT_ID_DEV)
          - name: DATABRICKS_CLIENT_SECRET
            value: $(DATABRICKS_CLIENT_SECRET_DEV)

        steps:
          - checkout: self

          - template: templates/template_liquibase.yml
            parameters:
              action: validate
              displayName: 'Validating changelog files'
          
          - template: templates/template_liquibase.yml
            parameters:
              action: status
              displayName: 'Show status on current and pending changelogs'

          - template: templates/template_liquibase.yml
            parameters:
              action: updateSQL
              displayName: 'Show SQL before it will be executed'
    
  - stage: deployment
    displayName: 'Deploy Schema Changes'
    dependsOn: [validation]
    jobs:
      - deployment: deployProd
        displayName: 'Deploy to Production'
        pool:
          name: ${AGENT_POOL}
        
        variables:
          - name: CATALOG
            value: ${TARGET_CATALOG}
          - name: SCHEMA
            value: ${TARGET_SCHEMA}
          - name: SQL_ENDPOINT
            value: ${DATABRICKS_SQL_ENDPOINT_DEV}
          - name: DATABRICKS_CLIENT_ID
            value: $(DATABRICKS_CLIENT_ID_DEV)
          - name: DATABRICKS_CLIENT_SECRET
            value: $(DATABRICKS_CLIENT_SECRET_DEV)

        environment: demo-prod-env

        strategy:
          runOnce:
            deploy:
              steps:
                - checkout: self
                
                - template: templates/template_liquibase.yml
                  parameters:
                    action: update
                    displayName: 'Executing changelog files'

Trigger: For our example, the pipeline automatically runs whenever changes are committed to the main branch, ensuring that every schema modification undergoes validation before deployment. For real-world production scenarios, we will likely want to implement more advanced trigger logic, depending on the branch used for staging and releases.

Variables: Each stage defines environment-specific variables for the target catalog, schema, and Databricks connection details. Using template variables (e.g., ${TARGET_CATALOG}) allows the same pipeline to be reused for different environments by simply overriding values at pipeline execution time.

Stage 1: The Validation Stage

The first stage of the pipeline is read-only and designed purely for validation/review. It executes several Liquibase commands to ensure the proposed changes are safe and correct, without modifying the target database.

1. validate: Scans the change logs for syntax errors, missing files, or malformed XML/SQL structures.
2. status: Reports which changesets have not yet been applied to the target database.
3. updateSQL: This is the most critical step for reviewers. It generates and prints the exact SQL statements that Liquibase intends to execute.

By inspecting the output of updateSQL, the data owners can verify exactly what will happen (e.g., “Ah, this script drops a column I didn’t expect”). This is vital for preventing accidental data loss.

The screenshot above shows an example of the updateSQL stage’s output. It is intended for users who are familiar with SQL. It not only shows which statements are executed by the changelog file, but also lists database locks and some additional information about the changeset.

If the validation state succeeds without errors, the deployment state will be triggered next. However, to enforce a manual check before deployment, the pipeline pauses. This is where the Approval Gate comes into play.

Stage 2: The Deployment Stage

In Azure DevOps, this is implemented using Environments. A specific environment (e.g., demo-prod-env) is configured to require approval from a designated Data Steward or Data Owner. The pipeline will not proceed to the Deployment stage until this person reviews the Validation output and explicitly clicks “Approve.”

Deployment Stage:

• dependsOn: [validation]: Only runs after Validation succeeds.
• environment: demo-prod-env: Configures the Approval Gate
• update: Executes the pending changesets, applying the schema changes to the target catalog and schema.

Once approved, the Deployment Stage triggers. It runs the update action, which:

1. Connects to the Databricks SQL Warehouse.
2. Acquires a lock on the database to prevent concurrent updates.
3. Executes the pending changesets.
4. Updates the tracking table to indicate that these changes are complete.

With this separation of concerns — validation without side effects, followed by human review and deployment — we ensure that only reviewed and approved changes reach production.

Change Logs: The core of our schema definitions

In this example, we define schema changes as a series of “changesets.” While Liquibase supports XML, JSON, and YAML, using Formatted SQL is the suggested best practice for data teams. We assume that data engineers are familiar with standard SQL, which lowers the entry barrier while still benefiting from Liquibase’s tracking capabilities. Liquibase, in general, supports various systems (not only Databricks), which allows us to use the same changesets for deployments on PostgreSQL (the core of Databricks Lakebase) or other databases.

Our recommended best practice is to use one changeset file for each version or logical change. This modular approach makes code reviews easier and simplifies the tracking of history.

The examples below illustrate this approach, showing two types of changelog files:

1. Master Changelog (XML): This file references all individual changeset files located under a specified path. We defined a filter to only consider for SQL files.
2. Individual SQL File: This is the actual SQL code being tracked and executed by the Liquibase command.

<?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog
   xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
       http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-latest.xsd">


   <includeAll path="changelogs" relativeToChangelogFile="true" endsWithFilter=".sql"/>
</databaseChangeLog>

- liquibase formatted sql
 - changeset Max.Mustermann:20251230–001
 - comment: Updating comments for vehicle_type to include examples.
ALTER TABLE fleet ALTER COLUMN vehicle_type COMMENT 'The type of the vehicle for example TRUCK or CAR';

Treating schema changes as modular, version-controlled artifacts brings the same discipline to data infrastructure that software engineering has long relied on. With formatted SQL and Liquibase, we can confidently evolve databases while preserving consistency and accountability across deployments.

Advanced Usage: Ideas for using each component to its full potential

Implementing this architecture requires us to consider certain nuances regarding how Liquibase interacts with Databricks Unity Catalog.

Tracking State with Governance

Liquibase maintains its own internal state using a DATABASECHANGELOG table created within the target schema. This table acts as the ledger of history, tracking:

• ID: The unique identifier of the changeset.
• Author: Who created the change.
• Filename: Where the change was defined.
• Checksum: A hash of the file content to detect if a deployed script has been retroactively modified (which throws an error to prevent inconsistencies).

Because this table resides within Unity Catalog, it is governed in the same manner as any other table in Unity Catalog. We can, and should, restrict access so that only the service principal running the CI/CD pipeline has write access to it, ensuring the audit trail remains tamper-proof.

Handling Existing Tables (Brownfield Projects)

In a real-world example, we would most likely not start from a blank slate; instead, we would have existing tables filled with valuable data. Luckily, bringing these under version control is straightforward.

When initializing the Liquibase project for an existing schema, we create a “baseline” changeset. Crucially, if the table already exists in the target environment, we can omit the CREATE TABLE statement or use Liquibase “preconditions” to check for existence. This allows Liquibase to “sync up” its tracking table without failing because the object already exists.

More information on diff-changelog can be found in the Liquibase documentation.

Managing Rollbacks

All quality gates, code reviews, or other safety measures will fail sometimes. So, what can we do if our changesets include errors and we need to revert them? Fortunately, Liquibase also offers advanced options to support you:

• Use the built-in rollback behavior for default operations, such as CREATE TABLE.
• Define custom undo logic directly in the change-log to make rollbacks more testable and predictable
• Tag important versions of our deployments to later roll back to a specific point in time

Before deploying such logic into production, it is essential to verify and test our rollback strategy. So we know that we are prepared when the time comes.

For more details and additional implementation examples, refer to the Liquibase documentation.

The Risk of Automated Change Generation

Liquibase offers a powerful feature called diff, which can compare two databases (e.g., Dev vs. Prod) and automatically generate a changelog to synchronize them. While tempting, this feature introduces significant risk, particularly regarding Column Renames.

The Issue:
If a developer renames a column from customer_id to client_id in the development environment, Liquibase’s diff tool does not recognize this as a rename operation. Instead, it interprets the change as two separate actions:

1. DROP column customer_id
2. ADD column client_id

The Implication:
If this auto-generated script is deployed to production, it will delete the column and all its data, then create a new, empty column.

Because of this risk, the best practice is to avoid automated changelog generation for production deployments. Instead, engineers should manually author SQL changesets. This ensures that a rename is explicitly written as ALTER TABLE … RENAME COLUMN, preserving the data.

Tool Selection: Why Liquibase and not Databricks Asset Bundles (DABs)

If you have already implemented Infrastructure as Code using Terraform or Databricks Asset Bundles, you might wonder why you should introduce a new technology and tool into your ecosystem, just for the sake of Schema Management.

While both tools manage Databricks resources, they serve fundamentally different purposes based on the lifecycle of the object and the consequences of replacing it.

1. The Case for Databricks Asset Bundles (DABs): Resource Definition
DABs follow an “Infrastructure as Code” (IaC) philosophy. They are designed for resources where the definition is the master, and the actual object can be updated, restarted, or replaced to match that definition.

Best For:

• Spark™ Declarative Pipelines with Materialized Views.
• Lakeflow Jobs and Pipelines: Jobs, tasks, and pipeline schedules.
• Compute Infrastructure: Job Clusters, SQL Warehouses, and Instance Pools.

Why: These resources are declarative.

• If you update a Job schedule in a DAB, the platform simply overwrites the old schedule.
• If you change a Compute configuration, the cluster restarts with the new settings.
• If you modify logic in a Spark Declarative Pipeline, the system handles the update automatically, often by refreshing the flow or recomputing the data.
  The state is either ephemeral (compute) or fully derived from code (pipelines).

2. The Case for Liquibase: Persistent State Evolution
Liquibase follows an “Evolutionary Database” philosophy. It is designed for resources where the data is the primary asset, and the structure must be carefully mutated around it.

Best For: Standard Managed Tables (Delta tables).

Why: These tables maintain a persistent state that cannot be simply recomputed or replaced. You cannot “re-deploy” a standard table containing 5 years of historical customer data just because you want to add a middle_name column; replacing the table would delete the history. Liquibase excels here because it executes granular, imperative modifications (ALTER TABLE…, UPDATE…) that evolve the schema version by version, while strictly preserving the underlying data.

The Rule of Thumb:

• If the resource describes how code runs (Jobs, Compute) or data derived from code (Declarative Pipelines) → Use DABs.
• If the resource describes stored data that must be preserved and evolved over time (Standard Tables) → Use Liquibase.

Conclusion

By combining the governance of Databricks Unity Catalog with Liquibase’s version control capabilities, we showed how data teams can bring software engineering rigor to their data platforms. Our example architecture replaces anxiety-inducing manual updates with a transparent, automated process where every change is tracked, reviewed, and reversible.

While it requires an initial investment in setup, creating the repository structure, configuring the Docker container, and defining the pipelines, the result is a stable, audit-ready data environment that scales with your team.

Liquibase offers a wide range of capabilities, allowing you to quickly define your production-ready Data Products. Some automations, however, require careful evaluation, and manual definitions and steps should be preferred.

Final remark

Tools like Liquibase for database change management, CI/CD platforms for automation, and unified governance layers such as Unity Catalog provide the technical foundation. However, sustainable success comes from pairing these tools with a culture of domain ownership, thorough documentation, rigorous testing, and accountability for data quality and accessibility. Without data owners and stewards living up to these expectations and managers supporting the organizational change to develop data products ready for production, it is hard to achieve, and the potential for shortening the path from idea to insight and from insight to action remains unutilized.

To successfully implement production-ready data products, you will need both people and technology.

About the author

Manuel Beuttler is a Delivery Solutions Architect at Databricks, helping customers across Central EMEA advance their data journey. Drawing on his experience in building data pipelines on Azure Databricks and establishing data-driven products and services in the sheet metal manufacturing industry, he brings a strong mix of hands-on engineering and practical governance expertise to every engagement.

Additional Resources

Documentation:

• Liquibase Official Documentation
• Databricks Unity Catalog Best Practices
• Azure DevOps Pipeline Templates
• Azure DevOps Environments and Approvals

Community Posts:

• Advanced Schema Management on Databricks with Liquibase
• Databricks Schema Versioning with Flyway and Liquibase: A Step-by-Step Guide

Governed Schema Evolution on Databricks was originally published in DBSQL SME Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Judging by the forums, some specialists have been waiting two years for a solution using various auto-replacements. However, I have an inside scoop for you: parameterization is finally working in the new release v0.281.0 !

The feature was released two weeks ago, so it's not yet documented. I tested it and wrote examples of how it works. Spoiler alert: it didn't quite work out that way…

Why is this a problem at all?

Typically, you first develop a Databricks Dashboard in the dev environment and then migrate it to prod. In the Databricks Unity Catalog world, this means your data is stored in different catalogs:

• dev_catalog.edl_schema.table1
• prod_catalog.edl_schema.table1

If your dashboard's code (file .lvdash.json) is hardcoded to dev_catalog, the dashboard will continue to look at dev data or crash with an access error when deployed to production. DABs can override parameters for Jobs and DLTs, but they support a few dashboard parameters. Engineers had to write Python scripts that parsed the dashboard's JSON and physically changed the text before deployment.

What happened?

Databricks has updated its API and expanded parameterization. The resource now dashboard accepts two new parameters: dataset_catalog and dataset_schema. This means we can finally ditch the hacks and use native variables.

How to transfer dashboards between development environments

In Databricks, a data engineer has five main options for migrating Lakeview dashboards (CI/CD) from Dev to Prod, in addition to manually working through the UI:

1. DABs (Databricks Asset Bundles) are a new native way to manage projects.
2. Terraform is a classic Infrastructure as Code (IaC).

You can also create and update a dashboard using separate commands:

1. SDK - for those who need complex migration logic.
2. REST API is the lowest-level and most powerful method.
3. Databricks CLI — for Bash scripting enthusiasts.

Let's see how this works in practice.

Option 1: Databricks Asset Bundles (DABs) — The Right Way

These parameters were especially welcomed by those using DABs. Now we can explicitly set dataset_catalog and dataset_schema, which are correctly passed to the dashboard context during deployment.

Don't forget to update your CLI to version 0.281.0. Here is an example databricks.yml :

bundle:
  name: project_2026_01_01

variables:
  dataset_catalog:
    description: Target Catalog for the environment
    default: dev
  dataset_schema:
    description: Target Schema for the environment
    default: edl_test
  warehouse_id:
    description: SQL Warehouse ID

resources:
  dashboards:
    my_dashboard:
      display_name: "Test NEW feature 2026 (${bundle.target})"
      warehouse_id: ${var.warehouse_id}
      file_path: ./src/dashboards/test1.lvdash.json
      embed_credentials: true
      dataset_catalog: ${var.dataset_catalog}   #🔥 NEW
      dataset_schema: ${var.dataset_schema}     #🔥 NEW

targets:
  dev:
    mode: development
    default: true
    variables:
      dataset_catalog: dev       #🔥 NEW
      dataset_schema: edl_test   #🔥 NEW
      warehouse_id: "123456789101112"
  
  prod:
    mode: production
    variables:
      dataset_catalog: prod   #🔥 NEW
      dataset_schema: edl     #🔥 NEW
      warehouse_id: "123456789101112"

When executed:

databricks bundle deploy -t prod

DABs don't modify the original JSON file. Instead, when creating a dashboard object in the workspace, it sets the default context (Catalog/Schema) at the level of all datasets. When the user opens the dashboard, all queries without an explicit catalog (for example, SELECT * FROM dash1 ) are executed in the prod.edl context:

Option 2: Terraform

Terraform is HashiCorp's Infrastructure as Code (IaC) tool that allows you to describe and manage cloud infrastructure through declarative configuration files. Instead of manually creating resources through the UI or REST API, you describe the desired infrastructure state in code, and Terraform automatically configures the actual infrastructure to that state. It also now supports dataset_catalog and dataset_schema for Databricks Dashboards starting with version 1.100.0 . Here's an example main.tf file :

terraform {
  required_providers {
    databricks = {
      source  = "databricks/databricks"
      version = ">= 1.100.0"
    }
  }
}

provider "databricks" {
  host  = var.workspace_url
  token = var.databricks_token
}

resource "databricks_dashboard" "sales" {
  display_name         = "NEW Terraform Dashboard - ${upper(var.environment)}"
  parent_path          = var.parent_path
  warehouse_id         = var.warehouse_id
  serialized_dashboard = file("${path.module}/test2.lvdash.json")
  
  dataset_catalog = var.dataset_catalog    # 🔥 NEW
  dataset_schema  = var.dataset_schema     # 🔥 NEW
}

output "dashboard_url" {
  value = "https://${var.workspace_url}/sql/dashboardsv3/${databricks_dashboard.sales.id}"
}

output "environment" {
  value = var.environment
}

Option 3: Python SDK

We can also create and update a Dashboard using the Python SDK. But first, we need to update the SDK to the latest version:

%pip install -U databricks-sdk
dbutils.library.restartPython()

Creating a dashboard through SDK, but in this example, the dashboard will actually be copied, replacing the catalog and schema:

from databricks.sdk import WorkspaceClient

wc = WorkspaceClient()

dashboard_path = "/Workspace/Users/my_email@gmail.com/test1.lvdash.json"

workspace_obj = wc.workspace.get_status(path=dashboard_path)
source_dashboard = wc.lakeview.get(dashboard_id=workspace_obj.resource_id)

new_dashboard = wc.lakeview.create(
    dashboard=Dashboard(
        display_name="NEW SDK 2026_01",
        serialized_dashboard=source.serialized_dashboard,
        warehouse_id=source_dashboard.warehouse_id  
    ),
    dataset_catalog="prod",  #🔥 NEW
    dataset_schema="edl"     #🔥 NEW
)

Updating a dashboard via SDK. For example, we have a dashboard with many datasets, and we want to change the catalog and schema for all of them:

from databricks.sdk import WorkspaceClient
from databricks.sdk.service.dashboards import Dashboard

wc = WorkspaceClient()

dashboard_path = "/Workspace/Users/my_email@gmail.com/test1.lvdash.json"

workspace_obj = wc.workspace.get_status(path=dashboard_path)
dashboard_id = workspace_obj.resource_id
current_dashboard = wc.lakeview.get(dashboard_id=dashboard_id)

updated_dashboard = wc.lakeview.update(
    dashboard_id=dashboard_id,
    dashboard=current_dashboard,
    dataset_catalog="prod",     #🔥 NEW
    dataset_schema="edl"        #🔥 NEW
)

Option 4: Working via API

For those writing their own deployment scripts in Python, there's the option to create and update dashboards. The code via API appears more cumbersome than via SDK. Below is an example of creating one by copying an existing one and replacing the directory and schema:

fimport requests
import json

browser_host =  spark.conf.get("spark.databricks.workspaceUrl")
host = f"https://{browser_host}"

new_token = wc.tokens.create(comment="Python SDK Generated Token", lifetime_seconds=9600)
token = new_token.token_value

DASHBOARD_PATH = "/Workspace/Users/my_email@gmail.com/test1.lvdash.json"

headers = {
    "Authorization": f"Bearer {token}",
    "Content-Type": "application/json",
    "Accept": "application/json"
}

workspace_response = requests.get(
    f"{host}/api/2.0/workspace/get-status",
    headers=headers,
    json={"path": DASHBOARD_PATH}
)
workspace_response.raise_for_status()
dashboard_id = workspace_response.json()["resource_id"]

source_response = requests.get(f"{host}/api/2.0/lakeview/dashboards/{dashboard_id}", headers=headers)
source_response.raise_for_status()
source_data = source_response.json()

serialized_dashboard = source_data["serialized_dashboard"]
warehouse_id = source_data["warehouse_id"]

create_payload = {
    "display_name": "NEW API 2026_01",
    "serialized_dashboard": serialized_dashboard,
    "warehouse_id": warehouse_id
}

create_response = requests.post(
    f"{host}/api/2.0/lakeview/dashboards",
    headers=headers,
    params={
        "dataset_catalog": "prod",     #🔥 NEW
        "dataset_schema": "edl"        #🔥 NEW
    },
    json=create_payload
)
create_response.raise_for_status()

Now let's update the catalog and diagram in the existing dashboard:

import requests
import json

browser_host = spark.conf.get("spark.databricks.workspaceUrl")
host = f"https://{browser_host}"

new_token = wc.tokens.create(comment="Python SDK Generated Token", lifetime_seconds=9600)
token = new_token.token_value

DASHBOARD_PATH = "/Workspace/Users/my_email@gmail.com/test1.lvdash.json"

headers = {
    "Authorization": f"Bearer {token}",
    "Content-Type": "application/json",
    "Accept": "application/json"
}

workspace_response = requests.get(
    f"{host}/api/2.0/workspace/get-status",
    headers=headers,
    json={"path": DASHBOARD_PATH}
)
workspace_response.raise_for_status()
dashboard_id = workspace_response.json()["resource_id"]

source_response = requests.get(
    f"{host}/api/2.0/lakeview/dashboards/{dashboard_id}", 
    headers=headers
)
source_response.raise_for_status()
source_data = source_response.json()

update_payload = {
    "display_name": source_data["display_name"],
    "serialized_dashboard": source_data["serialized_dashboard"],
    "warehouse_id": source_data["warehouse_id"]
}

update_response = requests.patch(
    f"{host}/api/2.0/lakeview/dashboards/{dashboard_id}",
    headers=headers,
    params={
        "dataset_catalog": "dev",       # 🔥 NEW
        "dataset_schema": "edl_test"    # 🔥 NEW
    },
    json=update_payload
)
update_response.raise_for_status()

Option 5: CLI

It is also possible to create and update dashboards via the CLI, but this is rarely used in practice, so I will omit examples.

What conclusions did I draw from testing the new parameters?

1. New parameters are not saved: After the request is completed, the API "forgets" these parameters. The dashboard simply retains the hard-coded catalog and schemas.
2. The current implementation is more like “Find & Replace”: When you send a request, the Databricks server takes your JSON (serialized_dashboard), goes inside the array datasets, finds the fields there, the catalog, and the schema physically overwrites them with the values ​​you passed.

I think it would be great if Databricks added the ability to set a default catalog and schema for the entire Dashboard. After adding a new dataset, the default schema is "workspace," and the default catalog is "default":

Author: Maksim Pachkouski [Databricks MVP & Databricks Solutions Architect Champion]. Follow my posts on Medium and LinkedIn to stay up-to-date on the latest news in Data Engineering and learn about best practices.

Dynamic Catalog & Schema in Databricks Dashboards (DABs, API, SDK, Terraform) was originally published in DBSQL SME Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.