What went GA and how it fits to the platform

Introduction

A card authorization usually has a few hundred milliseconds to come back with a fraud decision. The model making that decision needs features such as “total amount this card spent in the last 10 minutes,” and that number has to include the transaction that happened two seconds ago. A feature table refreshed every minute is already stale for this purpose.

Until now we solved this in two parts. Spark ran the ETL, the lakehouse tables and the training data. Apache Flink or a hand-written consumer service ran the operational path from Kafka to an online store. That worked but with great extra effort:

• The same business logic implemented twice, often in different languages.
• Different deployment and operational models for each path.
• Possible inconsistencies between the offline features used for training and the online features used for inference.
• Separate monitoring, failure recovery and governance for each environment.

On October 29, 2026, Databricks announced GA of real-time mode (RTM) in Spark Declarative Pipelines (SDP). RTM itself is not new: it shipped in Spark Structured Streaming earlier. What is new is that it now runs inside the declarative pipeline framework, with SDP’s orchestration, failure recovery and serverless compute.

What real-time mode changes inside the engine

The Databricks documentation describes three changes in RTM:

1. Long-running batches. A batch stays open for a long time (five minutes by default), and records are processed as they arrive in the source instead of being collected first.
2. Simultaneous stage scheduling. All stages of the query are scheduled at the same time. A stateful query with a source stage and a shuffle stage has both sets of tasks running at once.
3. Streaming shuffle. Data moves from one stage to the next as soon as it is produced, instead of waiting for the upstream stage to finish.

The result is that a record entering Kafka flows through the whole query without waiting for a batch boundary. Databricks quotes end-to-end latency as low as five milliseconds.

The problem it solves: one engine for the operational path

The case for RTM is less about raw speed and more about removing the second engine. When fraud features are computed in Flink and training features in Spark the same aggregation exists twice in two languages and two runtimes. Sooner or later the two versions drift and the model sees different numbers in production than it saw in training.

RTM lets the low-latency path use the same DataFrame API, the same Unity Catalog permissions, the same pipeline event log and the same deployment process as the rest of the data platform. In SDP, it also inherits dependency handling, automatic failure recovery and serverless compute.

On serverless, you don’t choose instance types and Databricks says you pay for the cores the query actually uses.

The announcement names fraud detection, live personalization, real-time recommendations and real-time feature computation as target workloads.

Databricks also publishes performance numbers, and they should be read as vendor claims:

• RTM is described as up to 100 times faster than the micro-batch architecture.
• On customer feature-computation workloads that Databricks selected, RTM is described as up to 92% faster than Apache Flink.
• The Databricks Feature Store, which uses this architecture, reports a p99 latency of 200 ms from Kafka to the online feature store.

Where RTM fits in the Databricks platform

RTM flows run beside the medallion architecture, not inside it. This is the most important thing to understand before designing with it and the announcement does not say it.

The supported sources and sinks table in the docs lists Delta as unsupported, both as a source and as a sink. An RTM flow reads from Apache Kafka, Azure Event Hubs through its Kafka interface, Amazon MSK or Amazon Kinesis.

It writes to Kafka, Event Hubs or MSK or to a ForEach sink that runs your own Python code for each row. Kinesis works only as a source.

So an RTM flow cannot read a silver table and cannot write a gold one.

It sits on the event bus and writes to operational systems: another topic, a key-value store, a Postgres database such as Lakebase or an API.

The components around RTM complete the picture. Lakebase, the managed Postgres service, is the natural target for features or decisions that applications query by key. The Feature Store online tables are built on this pattern. Model Serving endpoints read those features at inference time. Unity Catalog governs the lookup tables an RTM flow joins against and the connections it uses for Kinesis.

Building with it: card transactions to a fraud feature store

(1) The payment service publishes each transaction to the event bus.

(2) The RTM flow reads it and enriches it with the merchant lookup table.

(3) The flow upserts that card’s features into Lakebase.

(4) When the next authorization arrives, the payment service sends a scoring request to Model Serving.

(5) The fraud model reads the card’s features from Lakebase by key.

Configure the pipeline

RTM needs two settings at the pipeline level: continuous mode and the RTM Spark flag.

{
  "continuous": true,
  "spark_conf": {
    "spark.databricks.streaming.realTimeMode.enabled": "true"
  }
}

Example 1: the smallest real-time flow

An RTM flow must be an update flow, its trigger must be RealTime, and its target must be a sink.

This flow reads the raw topic and republishes it, which is enough to verify connectivity and measure baseline latency.

from pyspark import pipelines as dp

dp.create_sink("txn_passthrough_sink", "kafka", {
    "kafka.bootstrap.servers": broker_address,
    "topic": output_topic,
})

@dp.update_flow(
    name="txn_passthrough",
    target="txn_passthrough_sink",
    spark_conf={
        "pipelines.trigger": "RealTime",
        "pipelines.trigger.interval": "5 minutes",  # checkpoint interval, optional
    },
)
def txn_passthrough():
    return (
        spark.readStream
            .format("kafka")
            .option("kafka.bootstrap.servers", broker_address)
            .option("subscribe", input_topic)
            .option("startingOffsets", "latest")
            .load()
            .selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)", "timestamp")
    )

Example 2: write features to Lakebase with a ForEach sink

A Kafka topic is not something a fraud model can query by card ID. The features need to land in a store that supports point lookups. Inside SDP, the documented way to do this today is a ForEach sink.

Databricks does have a native Lakebase sink with built-in batching, retries and managed authentication, but its documentation lists it for Structured Streaming on classic compute (Databricks Runtime 18.3 and above) and it states that serverless and SDP are not supported.

The announcement lists a native Lakebase sink for SDP as upcoming work. Until it ships, the writer is yours to build.

Here is one example with upserts into a Postgres table with psycopg.

mport json
from pyspark import pipelines as dp

class CardFeatureWriter:
    def __init__(self, conninfo: str):
        # Plain strings are serialized to the executors; never pass a live connection.
        self.conninfo = conninfo

    def open(self, partition_id, epoch_id):
        import psycopg  # import on the executor
        self.conn = psycopg.connect(self.conninfo, autocommit=True, connect_timeout=5)
        return True

    def process(self, row):
        f = json.loads(row.value)
        self.conn.execute(
            """
            INSERT INTO fraud.card_features (card_id, window_end, spend_10m, txn_count_10m)
            VALUES (%s, %s, %s, %s)
            ON CONFLICT (card_id) DO UPDATE
              SET window_end = EXCLUDED.window_end,
                  spend_10m = EXCLUDED.spend_10m,
                  txn_count_10m = EXCLUDED.txn_count_10m
              WHERE EXCLUDED.window_end >= fraud.card_features.window_end
            """,
            (f["card_id"], f["window_end"], f["spend_10m"], f["txn_count_10m"]),
        )

    def close(self, error):
        if getattr(self, "conn", None) is not None:
            self.conn.close()

conninfo = dbutils.secrets.get("fraud", "lakebase_conninfo")  # resolved on the driver
dp.foreach_sink(name="card_features_sink", writer=CardFeatureWriter(conninfo))

What to check before production

The operator list has gaps that matter for fraud and personalization logic. According to the operator table and the limitations page:

• Stream-to-stream joins and session windows are not supported.
• dropDuplicates works but keeps unbounded state; dropDuplicatesWithinWatermark is not supported.
• foreachBatch, mapPartitions, mapInPandas, mapInArrow and flatMapGroupsWithState are not supported or not recommended.
• transformWithState works with differences: handleInputRows is called once per row, event-time timers are not supported, and transformWithStateInPandas is not supported.
• union cannot follow a stateful operator and cannot read the same source DataFrame twice.

My honest view: adopt it for features, keep Flink for event logic

My recommendation for teams on Databricks today: use RTM for single-flow feature pipelines and alerting where the source is already Kafka, Event Hubs, MSK or Kinesis and the target is a topic or a key-value store. Keep complex event processing where it runs now. Wait for multi-flow support to reach GA before consolidating many operational streams into one pipeline.

Databricks Real-time mode(RTM) in Spark Declarative Pipelines was originally published in Databricks Community Articles on Medium, where people are continuing the conversation by highlighting and responding to this story.

Connections, foreign catalogs, JDBC sources, secrets and bundles: what I learned federating PostgreSQL, Oracle, MariaDB, YugabyteDB, Snowflake and Denodo into one metastore

It started with a simple question I couldn't answer: which foreign catalog uses which connection?

I had created a handful of connections and catalogs in Unity Catalog, and a few days later I no longer remembered how they were linked. A Python snippet gave me the answer in seconds, but the real problem was the names. They said nothing.

Databricks doesn't impose a naming standard. If you don't set one, you end up with a metastore where postgres, pgsql and pg all mean the same thing, and where renaming a server breaks every downstream query.

So I built a convention, and then I stress-tested it on a lab with very different sources: public demo databases, an Oracle Free instance, a MariaDB server without TLS, a YugabyteDB cloud cluster requiring certificates and IP allow-lists, a vendor-managed Snowflake account, and two Denodo instances. Every rule below exists because something in that lab forced it.

The full convention, a deployable bundle, and the test queries are on GitHub: unity-catalog-federation-blueprint (link at the end).

Rule zero: logical names, not physical names

Server names, hostnames, cloud providers, and database products all change during a system's life. The business system they serve does not.

A catalog called fed_mssql_srvdb01_billing_prod looks precise until the billing database migrates to Azure SQL. Then you have two bad options: rename it and break every query, dashboard, and grant, or keep a name that now lies.

The fix is to separate two objects that are easy to confuse:

• The connection is physical. It stores how to connect: technology, host, credentials. Only admins see it, so it may carry technical details.
• The foreign catalog is logical. It is what users query, and its name appears in every query and lineage graph. It should speak the language of the business.

The source ID: one identifier to derive everything

The core of the convention is the source ID, a short, stable, logical identifier for each data source, assigned and recorded centrally.

For a SaaS application, it is the product name: salesforce, servicenow. For an application database, it is the business system: billing, crm. For a sandbox it is the lab name: oralab, yugalab. For a dataset replicated on several technologies, it is the dataset: pkdd_financial.

What to avoid: physical database names that recur everywhere (main, appdb, postgres), names that will clash with future systems (financial), and anything that describes where the data lives rather than what it is.

Every other name is derived from the source ID:

Connection        conn_<tech>_<source_id>[_<qualifier>]_<env>
Foreign catalog   fed_<source_id>[_<variant>]_<env>
JDBC view catalog jdbc_<source_id>[_<variant>]_<env>.<schema>.<table>
Secret            _platform_<env>.secrets.<connection without env>_<key>

So conn_pg_billing_prod feeds fed_billing_prod, and the link between them is obvious just from reading the names.

Why the environment suffix, if dev and prod are separate workspaces?

This was my first objection. Databricks recommends separate workspaces per environment, so why put dev or prod in names at all?

Because workspaces are separate, but the metastore is not. All workspaces in the same region attach to one Unity Catalog metastore, so catalog and connection names must be unique across environments. Workspace-catalog binding controls visibility, not naming.

The rule follows from object scope:

• Metastore-level objects (catalogs, connections, external locations, storage credentials) get the _<env> suffix.
• Objects inside a catalog (schemas, tables, views, volumes, secrets) don't: the catalog already carries it.
• Workspace objects (jobs, pipelines, clusters, warehouses) don't either: the workspace already carries it.

The environment always goes at the end. Listings then group by object type and system, with dev and prod side by side: fed_billing_dev, fed_billing_prod.

One connection per catalog

My first draft used one connection per server. The lab broke it immediately: one server hosted an Oracle instance and several PostgreSQL databases, each a different source. What source ID goes in the name of a connection serving five sources?

The answer is to stop sharing: one connection per foreign catalog, always, even when two connections point to the same server with the same credentials. It costs a few extra objects, and in exchange:

• the connection name always derives from the source ID, with no exceptions;
• a database can move to another server without touching its neighbours;
• the original question ("which catalog uses which connection?") answers itself.

Two refinements came later. When the source ID already starts with the technology, as with SaaS products or a Denodo instance, write the technology once: conn_denodo_29090_dev, not conn_denodo_denodo_29090_dev. And use a closed, documented list of technology codes (pg, ora, mssql, mariadb, yb, snowflake, denodo, sfdc…), or you'll soon have postgres, pgsql and pg side by side.

When the technology is the point

"No technology in catalog names" has one legitimate exception: when the technology is the subject.

In the lab I loaded the same banking dataset (PKDD'99 financial) on MariaDB and PostgreSQL to compare behaviour. Querying "the dataset" is meaningless here: you need to know which copy you're hitting. The convention adds a variant qualifier to every copy, the first one included:

fed_pkdd_financial_pg_dev
fed_pkdd_financial_mariadb_dev

The connection doesn't need it, because its technology code already makes it unique.

When federation isn't possible: JDBC view catalogs

Lakehouse Federation doesn't cover everything. Denodo has no federation connector. And one of my MariaDB servers had no TLS, which federation requires for MySQL. I checked the server handshake to be sure: the SSL capability flag simply wasn't there.

The workaround is a JDBC connection, a Unity Catalog connection that loads a driver JAR from a volume. JDBC connections don't create foreign catalogs, though. You query them through remote_query(), in the source's own SQL dialect.

To give these sources the same shape as a foreign catalog, I generate a JDBC view catalog: one view per source table, each a thin SELECT * over remote_query(), the same idea as Denodo base views.

jdbc_ctulab_dev.ctu_financial.loan

The jdbc_ prefix describes the access method, just like fed_. If the source becomes federable one day, consumers only change the prefix. Users need SELECT on the views only, because the connection privilege is checked on the view owner.

A platform catalog for the plumbing

Secrets, JDBC drivers and certificates need a home. The obvious candidate, the default workspace catalog, is the wrong one: it is usually bound to a single workspace, often grants broad permissions to all users, and carries no environment.

Instead, a dedicated catalog per environment:

_platform_dev
 ├─ secrets  (schema)          Unity Catalog secrets, no grants
 ├─ drivers  (schema)
 │   └─ jdbc (volume)          JDBC driver JARs, readable by connection users
 └─ certs    (schema)
     └─ ca   (volume)          server/CA certificates, admins only

The leading underscore marks it as infrastructure, and in most tools it sorts it above the data catalogs. The separate schemas exist for one reason: different grants. Everyone querying a JDBC connection needs to read the drivers, and nobody should read the secrets.

Secrets follow the newer Unity Catalog secrets model (catalog.schema.secret) instead of workspace secret scopes, and are named by intended use: conn_pg_ecom_user, conn_pg_ecom_password. I create connections with the Python SDK, reading values from secrets, so credentials never appear in a SQL statement or in the query history.

Deploying it with bundles

Databricks recommends Declarative Automation Bundles (formerly Asset Bundles) for multi-environment deployment. They fit the convention through one rule:

Bundle target names are the environment codes.

Then ${bundle.target} is the _<env> suffix, and no environment-specific name is ever written by hand:

variables:
  billing_catalog:
    default: fed_billing_${bundle.target}

Connections are not a bundle resource type, so a platform bundle deploys an idempotent setup job. It reads a versioned source registry file (config/sources_<env>.yml) and creates the platform catalog, connections, foreign catalogs, JDBC connections and view catalogs. The run fails if anything is missing, so a broken source surfaces in the deployment pipeline rather than in a user's query. Project bundles then consume the sources through variables.

One deliberate exception: development mode prefixes resources with the developer's name (dev_fred_…). That breaks the "environment at the end" rule, and it should, because a leading dev_<user>_ marks a personal sandbox while a trailing _dev marks the shared environment. The platform bundle never runs in development mode.

Lessons learned the hard way

Pushdown depends on the Spark dialect, not on federation vs JDBC. I ran the same aggregate query three ways. Through JDBC to MariaDB, the filter was pushed, but the GROUP BY ran in Databricks. Through JDBC to PostgreSQL and through federation to PostgreSQL, the plans were identical: filter, aggregates, and grouping all pushed to the source. Both paths use the same Spark JDBC engine, and Spark picks the dialect from the URL. A jdbc:mariadb: URL falls back to the generic dialect, which doesn't push aggregates.

MariaDB via the generic dialect needs ANSI_QUOTES. Spark quotes column names with double quotes, and MariaDB reads them as string literals. The symptom is baffling: value 'ok' cannot be decoded as Integer. Adding sessionVariables=sql_mode=ANSI_QUOTES to the JDBC URL fixes it.

'' is not an escape in Spark SQL. Adjacent string literals are concatenated, so a doubled quote silently vanishes. When you build remote_query() strings in code, escape with a backslash.

Server certificates go in as text. userProvidedServerCertificate expects PEM content, not a path, and must be paired with trustServerCertificate 'false'.

Serverless egress IPs are published, and they change. SELECT current_metastore() returns cloud and region. Filter Databricks' published IP ranges with those values to build the allow-list, and re-check it periodically.

Vendor-managed accounts can close the door. A Snowflake account managed by an ELT platform suddenly rejected every client IP through a network policy I couldn't change. Prefer pushing data from such platforms to a destination you control.

Semantic layers don't travel over JDBC, but their queries do. Denodo 9.5 introduced metric views, similar in spirit to Databricks metric views. Databricks can't recognise them as metric views, because no semantic metadata crosses a JDBC driver. But remote_query() passes the text to Denodo unchanged, so this works, with Denodo computing the metric:

SELECT * FROM remote_query('conn_denodo_29090_dev', query =>
  'SELECT trans_year, evaluate_metric(transaction_count) AS n
     FROM mv_yuga_fin_account_activity
    GROUP BY trans_year');

I validated it by comparing those yearly counts with the same counts computed by Databricks on a PostgreSQL copy of the data. The difference was zero for every year.

Takeaways

• Name sources by what they are, never by where they live.
• One connection per catalog: a few extra objects buy you obvious lineage and painless migrations.
• Put the environment only where the metastore is shared, and always at the end.
• Treat the source registry as code, and let bundles derive every name from it.
• When federation can't reach a source, JDBC view catalogs keep the same shape for consumers.
• Before blaming federation for slow queries, check the dialect in the plan.

The complete convention (with edge cases, lifecycle rules, and the full lab), the platform bundle, and the test queries are open source:

github.com/Capppo/unity-catalog-federation-blueprint

Contributions are welcome, especially new technology codes, table-listing queries for other JDBC sources, and real-world cases that break the rules.

This is an independent community project, not affiliated with Databricks or Denodo.

Naming Your Data Sources in Databricks Unity Catalog: A Convention Tested on a Real Multi-Source… was originally published in Databricks Community Articles on Medium, where people are continuing the conversation by highlighting and responding to this story.

The problem: a chatbot that doesn’t know your business

Imagine you call a company’s support line and ask:

“How do I fix file sync problems in CloudVault Pro?” then “What’s your return policy?” and finally “How many returns has customer alice@example.com made this year?”

These three questions need three different places to look:

• Question 1 lives inside product manuals (PDF files).
• Question 2 lives in a policy table (structured data).
• Question 3 lives in a customer history table (structured data).

A normal chatbot (ChatGPT-style) knows the internet, but it knows nothing about your PDFs, your policies, or your customers. If you ask, it may confidently make something up. That is called a hallucination, and in customer support it is dangerous.

In this article we build an AI agent that:

1. Reads the question.
2. Decides where to look (manuals, policy table, or customer table).
3. Fetches the real facts.
4. Writes a clear answer based only on those facts.

Key ideas

AI agent vs. agentic AI

• An AI agent is one worker: an LLM + tools + (optionally) memory. It breaks a task into steps and carries them out.
• Agentic AI is a team of such agents, often with an “orchestrator” manager deciding who works next.

RAG : Retrieve, Augment, Generate

1. Retrieve — find the most relevant passages for the question.
2. Augment — attach those passages to the question.
3. Generate — the LLM writes an answer using that evidence.

Why not paste all PDFs into the prompt? LLMs have a context window (a maximum number of tokens they can read at once). With ~500 PDFs you would blow past it, and you would pay for every token on every question. Retrieval sends only the few relevant passages.

What we will build

Step 0 — Workspace

We start by signing up for a free workspace. Think of it as an online office where all the work happens. We only need to know four rooms: Catalog (where data is stored), Compute (where things run), Playground (where we test the AI) and Serving (where the finished agent lives).

1. You land on the workspace home page.
2. In the left sidebar, find the AI/ML section (Playground, Serving) and Catalog, Compute, Workspace.

Step 1 — Talk to an LLM

Before building anything, we check that the AI model answers a question. The Playground needs no code. The Python and SQL versions show the same thing for people who want to do it in a notebook.

In the Playground

Open AI Playground, choose the model, and ask:

Why are AI agents getting popular?

Normal

Then add a system prompt, such as Use a funny language, and ask again. The tone changes completely.

By using Funny Language

In Python

Under the hood:

• chat.completions.create is the standard API shape popularised by OpenAI and now used by many providers, which is why this one client works with many Databricks-hosted models.
• max_tokens caps the reply length.
• Reasoning models may return content as a list of blocks: one with type: reasoning (the model’s internal thoughts) and one with type: text (the real answer).

In SQL (the easiest way)

ai_query is a built-in SQL function: first argument = model, second = prompt. For a data engineer this is powerful, because you can call an LLM on every row of a table.

Step 2 — Create the catalog, schema and volume

Upload your files in the Volume

Step 3 — Build the unstructured knowledge base (PDFs)

The AI can’t read PDF files directly, so we pull the text out of each PDF and save it as rows in a table. Now the manuals are data the system can search.

Extract text from every PDF

Save as a Delta table

We have created a product_document Table

What about long documents?

Each PDF was short enough to sit in one row. Real manuals are often 50+ pages. Then you chunk: split each document into passages of roughly 500–1,000 characters with a small overlap, one row per chunk. There is no universal best size; test retrieval quality and adjust.

Step 4 — Build the structured knowledge base

Some facts, such as policies and customer history, are already neat rows and columns. We save these as three small tables: products, policies and customer history.

We have a Sample Tables of Products, Policies, Customer_service_history

Step 5 — Enrich the documents

Before searching, we stamp each manual’s product name and category on top of its text. It’s like writing the department name on the front of every file folder, so a search for “storage software” can find the right one.

The search index will look at one text column. So we pack the most useful context into that single column: category, subcategory, product name, then the document itself.

Step 6 — Create the vector search (AI Search) index

Here we build a search that works by meaning, not exact words. Each manual becomes a list of numbers (an embedding), and the question becomes one too. The closest matches win. We test this search on its own first, so we know it works before the AI is involved.

What is an embedding?

Computers can’t “understand” sentences, but they are excellent with numbers. An embedding model converts a sentence into a long list of numbers that captures its meaning. Sentences with similar meaning get similar numbers.

• “Press Sync Now to refresh your files”
• “Use the refresh option if your files look out of date”

Different words, same meaning, so their number lists sit close together. A keyword search would miss the match; a meaning-based search finds it.

Create the endpoint

An endpoint is the always-ready address where our search lives. Free Edition allows only one, so create it once and reuse it.

Create the index

Click on AI Search Index

Test the index before involving the LLM

This is the most underrated debugging habit: always test retrieval alone first. If retrieval is bad, the LLM can’t save you.

In the index page choose Query index. You should see the CloudVault Pro document ranked first, with a score.

this is the “R” in RAG working on its own. It proves retrieval is correct before the AI is added.

Step 7 — Create the structured tools (Unity Catalog functions)

Language models can write SQL, but generated queries can be wrong, and running them freely would require broad access to your tables. For customer data, that is a risk we can avoid. So we don’t give the agent a free-form query tool.

Instead we give it two fixed lookups: one that returns a named policy and one that returns a customer’s support history. The agent decides when to call them and what value to pass in, but the query itself is written by us and tested in advance, and Unity Catalog controls who may run it.

Tool 1: policy lookup

Tool 2: customer service history

Why functions instead of free-form SQL?

1. The agent can only use what we built (controlled access). Think of a vending machine: you can pick only the items it lists. The agent can run our two lookups, but it cannot browse other tables or run a query nobody approved.
2. The same question gets the same answer (predictable results). The rule for counting a customer’s returns is written once, by us. Every time the agent calls it, that same rule runs, so two people asking the same thing see the same number. A model that writes fresh SQL each time might count slightly differently.
3. Someone decides who may use it (governance). Unity Catalog is the permission system in Databricks. An administrator can choose who is allowed to run each function, and the function is listed in Catalog Explorer like any table, so it can be reviewed.
4. Each lookup has a label the agent reads (clear contract). The agent chooses a tool by reading its COMMENT, a short description we write. It works like the label on a button, for example “Returns the official details of a named support policy.” Write these labels as if briefing a new colleague who has never seen your data; clearer labels lead to better choices.

Step 8 — Assemble the agent in AI Playground

Now we connect the AI to its tools: the manual search and our two lookups. We don’t tell it which one to use. It reads the question and decides. Try a question that needs two tools and watch it use both.

1. Open AI Playground and pick the model (one that supports tool calling).
2. Click Tools → Add tool.
3. Add the AI Search tool and select product_document_index.
4. Add the two UC function tools: get_policy_details and get_service_history

Ask Questions

Step 9 — Register and deploy the agent (Model Serving)

The Playground is a test bench, so to share the agent we turn it into a live service, which Databricks calls “serving.” When nobody is using it, it goes to sleep to save resources

When nobody is using the agent, it goes to sleep to save resources, so the first answer can take about 30 seconds. If you see an error saying scale-to-zero must be enabled

1. In Playground, clear the chat history
2. Click Get code → Create agent notebook.
3. A notebook opens, already containing the model, tools and MLflow logging code. You do not need to understand every line.
4. Find the TODO cell and set where to register the model:

5. Click Run all.

Registered Model

Serving endpoints

Deployed endpoint answering in Playground

In PDF

Step 10 — Give it a face: the Databricks App

A chat page is what a normal user would see. We create it from a ready-made template, with no coding. The user types a question, and they never need to know the answer may come from a PDF or a table.

Business users won’t use Playground. They need a simple chat page.

1. Compute → Apps → Create app.
2. Choose the template Chat UI for an existing agent.
3. Select your serving endpoint. Leave size and authorization defaults.
4. Name it, for example customer-portal, then Create app.
5. Wait a few minutes for the first deployment, then open the app URL.

If the app shows an error about a missing scope, open the app’s Settings → User authorization, add the Model Serving scope, redeploy, and open the app in a fresh window.

App creation

The final chat UI

Step 11 — Watch the agent think (MLflow Tracing)

A correct-looking answer can still be wrong underneath. Tracing records every step.

Open Serving → your endpoint → Traces and click a recent request. For “Summarize return policy for me”

1.The AI reads the question and asks for a tool

2.The tool runs and returns data

3.The AI reads that data and writes the answer.

If a customer says “your bot gave me wrong instructions,” you can open the replay and see: did the bot look in the wrong place, did the search return the wrong page, or did it misread the right page?

Step 12 — Check the agent’s answers (evaluation)

Think of a driving test. You don’t wait for a crash to learn whether someone can drive. You give them the same set of situations and check each result against what a correct response looks like. Evaluation is that set of situations for your agent, written down so you can re-run it after every change (a new prompt, a new chunk size, a different model).

Why it matters most for agents: an agent has several places to go wrong (wrong tool, wrong passage, wrong wording), and a change that fixes one question can quietly break another.

A simple, working evaluation script

In my run, 5 of 7 tests passed. One failure showed raw tool text inside the answer. That is exactly what the test list is for.

Letting an AI judge score the answers

Text matching only asks, “does the answer contain these words?” A better question is, “is this answer correct and well-formed?” For that, Databricks provides LLM judges through MLflow’s GenAI evaluation. A judge is another AI model that reads your agent’s answer and gives a verdict, usually with a written reason.

Two judges are useful for this project:

it shows readers the difference between “contains the right words” and “a judge says the answer is correct,” using the same agent.

Build Your Agentic AI App in Databricks: A Step-by-Step Guide was originally published in Databricks Community Articles on Medium, where people are continuing the conversation by highlighting and responding to this story.

I've noticed that most modern cloud platforms have an interesting feature: they display costs, but in an overly aggregated and complex manner. Sometimes they hide features behind SKU combinations. Databricks has a standard Dashboard for this, as well as the Governance Hub. The Hub displays costs at a high level, across all workspaces:

As a result, an entire industry of FinOps tools has grown around cloud spending. At DAIS 2026, I counted more than 10 similar products. There are solutions for monitoring, cost allocation, anomaly detection, and infrastructure optimization. But each new tool is another interface that needs to be learned, configured, and regularly accessed. I don't have time to learn the new features of Databricks, let alone third-party tools...

I had the idea to try making a widget, but Ilya came up with a more interesting solution. What if we added prices and costs directly to the Databricks interface via a plugin? It sounded crazy, but after a couple of experiments, we got it working.

This is how FinOpsWay was born —a free browser extension that displays costs directly alongside Databricks resources. Ultimately, it complements the Databricks Governance Hub.

Initially, we added a mini-panel with a list of expenses:

Then the ideas started coming one after another. First, we added a display next to the clusters. That was simple. Then, we added the cost when selecting a cluster, which was more interesting because now the engineer could see the actual cost, not just abstract DBUs:

But I was always interested in how much each launch cost. Copying the ID each time and going to the dashboard was too much. But now I can see it right next to each launch:

We gradually added cost display for various objects. At some point, someone asked to add costs for Notebooks. It wasn't difficult to implement, but ultimately, the LinkedIn post about this feature got the most likes. I still don't understand why they need to see it...

What else can you see? We've also added a cost display next to Databricks Apps. In the current Dashboard, they're hidden as a combination of SKU and Billing_SKU and displayed as All-purpose serverless compute:

Also, for fun, we added this to Lakebase Postgres during setup, as well as to branches and databases:

It was a funny effect. Ilya used to think Lakebase was expensive and avoided using it, but after this feature, he changed his mind.

Before launching this product, we discussed at length whether to release it as a paid product, a free product, or a subscription. Unfortunately, any Chrome extension can be downloaded from the browser and the full code can be viewed. Therefore, we decided to make this product free for everyone by activating sponsorship through GitHub . Currently, we have about 250 active users and no donations, so I'm not sure how long this initiative will last. We'd be interested in your opinions on the idea and what features might be missing.

This extension also works on Databricks Free Edition, and you can track the latest updates on the LinkedIn page .

Databricks costs directly within the interface — FinOpsWay extension was originally published in Databricks Community Articles on Medium, where people are continuing the conversation by highlighting and responding to this story.