How Alpha Level built autonomous threat detection on Databricks to catch what rules miss

By Derek Witt and Cody Anderson (Databricks), Joshua Neil and Mike Pozmantier (Alpha Level)

Twenty IPs. Eight countries. Thirty-three failed logins spread across several days. Zero alerts.

Not because nobody was watching. Because the attack used a legacy Microsoft authentication endpoint (OrgIdWsTrust2) that bypasses SigninLogs entirely. MFA never triggered. Conditional Access never triggered. The SIEM had nothing to write a rule against because it never saw the traffic.

In another customer environment, a Windows Server 2008 R2, end-of-life since January 2020, was serving 255 Citrix users every single business day. Listed in Microsoft Defender for Endpoint inventory as “Can be onboarded,” meaning it was known to exist but never enrolled. Zero process telemetry. Zero patches since 2020. An attacker who found it before anyone else did would have had an unmonitored server with active users and no detection coverage.

Neither of these showed up in an alert queue. They showed up in Autonomous Hunter, Alpha Level’s AI-driven anomaly detection platform built on Databricks, because behavioral anomalies don’t look like known attack signatures. They look like noise, or nothing at all.

The gap nobody talks about

The conversation around SOC tooling spends a lot of time on alert fatigue. Too many alerts, too much noise, analysts buried in false positives. That’s real. But there’s a quieter problem that gets less attention. That problem is the attacks that don’t generate any alert at all.

Rules-based detection (SIEM correlation rules, EDR signatures, Conditional Access policies, threat intel feeds) is great at catching what you anticipated. You wrote a rule for it, or a vendor shipped you a signature for it, and when it matches, it fires. Rules are good for identifying what is known, but adversaries are highly motivated to innovate, and you need protection for the unknown as well.

That works reasonably well for known attack patterns. It doesn’t work at all for the stuff designed to look normal: insider threats, lateral movement that uses legitimate admin tools, credential theft over legacy endpoints, C2 beaconing that mimics normal traffic cadences, ransomware precursors doing nothing but reconnaissance for weeks. These threats slip through not because your tools failed, but rather because the adversary is trying to blend in with normal users, and a rule can’t discriminate well. The net result is either too many false positives, or a missed breach.

A good human threat hunter, someone with deep knowledge of adversary behavior who sits down with raw data and actively looks for things that don’t add up, can find this stuff. Most organizations don’t have enough of them. And the ones they do have are spending most of their time on alert triage, not proactive hunting.

That’s the gap Autonomous Hunter is built for. Not alert triage. Finding the threats that aren’t generating alerts at all.

Why you can’t just dump logs into an LLM

Can a large language model do this? You’ve got the logs. The model is smart. Point it at the data.

The honest answer is no. Not from raw telemetry. This isn’t a hot take. Ask any AI chatbot directly (“Should I use an LLM for anomaly detection from raw telemetry in a cyber security context?”) and it’ll tell you the same thing. Use statistical and ML models. LLMs are trained on language patterns, not behavioral baselines. They can’t tell you that this login frequency is unusual for this specific user’s auth patterns at this time of year, because they don’t have that baseline. They can reason about what anomalies look like in general. They can’t measure them.

The second problem is context. If you feed an agent raw logs, you’ve given it a needle-in-haystack problem with the emphasis on the haystack. Too little context, and it hallucinates. Too much spurious context, and it burns through tokens pattern-matching against noise and still hallucinates. Either way you get high costs and low confidence in the output.

Josh Neil, Alpha Level’s co-founder and CTO, calls this the Goldilocks problem. The work isn’t in the agent. It’s in preparing exactly the right context before the agent ever sees it. Strip out everything that’s benign. Surface the things that are actually anomalous. Assemble the surrounding context that makes an anomaly meaningful. Get that right, and the agent can do real investigative work. Skip it, and you’re throwing compute at a problem the agent wasn’t designed to solve.

How Autonomous Hunter actually works

Alpha Level built a three-stage pipeline on Databricks that separates the anomaly detection work from the agentic work. Each stage does what it’s actually good at.

Stage 1: Statistical anomaly detection

Security telemetry from across the enterprise (network flows, authentication events, cloud API logs, endpoint data) lands in a Databricks lakehouse. That’s the foundation everything runs on. Autonomous Hunter’s self-learning time series models run as Spark jobs directly against those data, building a per-enterprise behavioral baseline across all four telemetry surfaces. These aren’t rules. They’re models that learn what normal looks like for your specific environment, including your users’ login hours, typical inter-host traffic patterns, and usual authentication volumes across your auth stack.

When something deviates from that baseline significantly enough, it surfaces as a rare event. The threshold is high by design. Most anomalies are explainable, like someone logging in from a new city because they’re traveling, or a server seeing unusual traffic because of a deployment. The statistical models do the first pass of separating the rare from the merely unusual.

Stage 2: Graph context assembly

Rare events don’t live in isolation. Thirty-three failed logins across 20 IPs aren’t just a list of authentication failures. They’re part of a story that includes which accounts were targeted, what those accounts have access to, what other activity surrounded them, and how the timing pattern compares to known attack patterns.

Autonomous Hunter assembles rare events into graphs of nodes and edges, storing the context packages in Delta tables alongside the raw telemetry. Nodes are entities: users, hosts, IPs, authentication endpoints. Edges are the relationships and events connecting them, things like authentication attempts, network connections, and API calls. The graph captures why an event is anomalous, not just that it is. Events that can be explained by surrounding context get filtered here. A legitimate admin doing a bulk operation, a service account doing its normal job. Those get explained and dropped.

What’s left after graph filtering is a small population of anomalies that are both statistically rare and contextually unexplained. That’s what goes to the agent.

Stage 3: Agentic investigation and the analyst interface

The AI agent receives a pre-packaged graph of exactly the right context, with no raw log dump and no irrelevant noise. Its job is to investigate, not hunt. It reasons about what the anomaly pattern could represent, what the likely impact is, what other data would confirm or rule out a threat, and what a SOC analyst needs to know.

The investigation summaries surface in the Autonomous Hunter SOC interface, a Databricks App that Alpha Level deploys for their customers. SOC analysts access it to review findings, dig into investigation summaries, and decide what to act on. The whole pipeline (ingest, anomaly detection, graph assembly, agentic investigation) and the analyst-facing interface all run on Databricks, which means Alpha Level isn’t stitching together a separate ML platform, a separate app hosting layer, and a separate data store. It’s one stack.

This is where the Goldilocks framing pays off. Because the agent is working from carefully prepared context, it can do the reasoning it’s actually good at: “here’s a pattern of behavior, here’s what’s anomalous about it, here’s what it probably means.” The output is an investigation summary a human analyst can act on, not another list of events to triage.

The agent is the last mile, not the first. All the heavy lifting that makes agentic investigation accurate and cost-effective happens upstream.

What this finds in practice

Case 1: 20 IPs, zero alerts, an invisible brute force

Microsoft’s OrgIdWsTrust2 legacy authentication endpoint is a documented blind spot. It’s a WS-Trust 2.0 flow that predates SigninLogs. Traffic through it simply doesn’t appear there. MFA doesn’t apply to it. Conditional Access doesn’t apply to it. Standard SIEM rules that fire on failed login volumes in AADRiskEvents or SecurityAlert won’t catch it either, because those tables are populated from SigninLogs.

Autonomous Hunter caught a distributed brute force campaign targeting this endpoint: 33 failed login attempts spread across 20+ source IPs in 8+ countries, sustained over multiple days. No velocity trigger because the attempts were slow and distributed. No location trigger because the IPs were already spread globally. Zero alerts in the customer’s existing stack.

Detection came from behavioral modeling. The pattern of attempts against that specific endpoint was anomalous against the enterprise’s baseline, even at the per-IP level. The attack was running indefinitely, undetected, with adversaries guessing passwords before MFA could ever enter the picture. Since MFA wasn’t hit, this means the adversary could determine which passwords worked, since only those would take the user to the MFA process, and failed logins were also transparent to the adversary.

Every Microsoft environment with legacy authentication enabled has this exposure and most of them have no visibility into it.

Case 2: 255 users, EOL since 2020, no one knew

The second case is less about an active attack and more about an attack surface nobody could see. A Windows Server 2008 R2 (end-of-life since January 2020, meaning no security patches for over five years) was serving 255 Citrix users every business day. It was visible in MDE inventory. Someone had just never enrolled it.

Because there was no agent on the server, there were no DeviceEvents, no ProcessEvents, no telemetry of any kind. The server existed in a monitoring gap.

Autonomous Hunter found it through behavioral network analysis alone. Traffic patterns from 255 hosts were converging on this one server in a way that was anomalous when viewed against the rest of the environment’s topology. No agent required.

Every environment has servers that fell through the cracks. The question is whether you find them before an attacker does.

Why this runs on Databricks

Josh has been building anomaly detection ML in security for a long time, going back to research environments where the team had access to real compute and used it. For a couple of decades, enterprise security standardized on SIEM platforms that traded raw compute capability for convenience and compliance. Those platforms dominate the market. They’re also not built for data science. They’re built for log storage and boolean query logic. You can’t run arbitrary ML workloads against your security telemetry in most of them.

The result is that a lot of detection capability that existed in high-compute research settings never made it into enterprise production at scale. Not because the algorithms were wrong. Because the platforms that enterprises bought weren’t capable of running them.

Databricks changes the equation in a few ways that matter specifically for what Alpha Level is building.

Compute that scales with the workload. The algorithms powering Autonomous Hunter’s anomaly detection (time series modeling, graph computation, unsupervised clustering) are memory-intensive by nature. Security telemetry at enterprise scale is millions of events per day. Databricks handles arbitrary memory allocation and autoscaling for Spark jobs, which means Alpha Level doesn’t need a different architecture for a small design partner versus a global enterprise. The platform scales the infrastructure.

A single platform from ingest to investigation. Data lands in the lakehouse, the ML runs on it as Spark jobs, the graph context gets assembled and stored in Delta tables, and the findings surface in a Databricks App. That’s the whole stack on one platform. For a five-person team landing enterprise security contracts, avoiding a separate ML platform, a separate application hosting layer, and a separate data store isn’t just a convenience. It’s the difference between a viable product and a maintenance nightmare.

Data ownership, built for Lakewatch. Databricks recently launched Lakewatch, an agentic SIEM built on the security lakehouse designed as a cost-effective, vendor-agnostic alternative to proprietary, legacy SIEMs. Alpha Level is built to complement it. Bring your security telemetry into Lakewatch, run Autonomous Hunter as the detection and investigation layer on top, and you’ve got a stack where you own the data, the detection doesn’t depend on the data platform staying put, and swapping out either component doesn’t require a year-long migration. That’s a meaningful shift from how security data has worked for the last decade.

Who should be paying attention

Autonomous Hunter is pre-release. Alpha Level is working with a select group of design partners, primarily enterprises that have existing security stacks and are increasingly concerned not about alert volume but about what they’re not detecting at all.

If you’re running security operations at an organization with real exposure and you’ve been assuming your SIEM coverage is complete, it probably isn’t. Legacy authentication endpoints, unmonitored servers, slow lateral movement that never triggers a velocity rule. These are the patterns that don’t show up until after something goes wrong.

Alpha Level will be demoing Autonomous Hunter at the 2026 Databricks Data and AI Summit. Demo videos are in the works. We’ll update this post with links when those are live. If you want to explore whether a design partnership makes sense for your organization, reach out to the team at alphalevel.ai (hello@alphalevelsec.com).

Derek Witt is a Solutions Architect at Databricks focused on the Startups segment. Cody Anderson is on the Databricks Startups and AI Accelerator team. Joshua Neil is co-founder and CTO at Alpha Level, with 26 years in threat detection research. Mike Pozmantier is co-founder and CEO of Alpha Level.

The Databricks for Startups Field Engineering team works closely with founders and engineers working on ambitious ideas. We love connecting on scalable architecture, data platforms, and the real challenges of bringing data and AI systems into production.

Working on something interesting? We’d be happy to compare notes and see where we can help. Email us at ask_startups@databricks.com or reach out on LinkedIn.

Zero Alerts Doesn’t Mean Zero Attacks was originally published in Databricks for Startups on Medium, where people are continuing the conversation by highlighting and responding to this story.

How Zerobus Ingest gets your event data into the lakehouse without the Kafka tax

By Derek Witt (Solutions Architect at Databricks)

You’ve got sensors on a factory floor, clickstream events from your app, or telemetry from a fleet of devices doing something awesome in the field. The data is flowing, and you need it in your lakehouse. So you spin up a message bus service, write a Spark Streaming job to read from it, and then you wait on both of those systems while the data lands in a Delta table.

You’ve got sensors on a factory floor, clickstream events from your app, or telemetry from a fleet of devices doing something awesome in the field. The data is flowing, and you need it in your lakehouse. So you spin up a message bus service, write a Spark Streaming job to read from it, and then you wait on both of those systems while the data lands in a Delta table.

When you have 10 engineers and half of them are building core product, you can’t dedicate two of them to plumbing an event streaming pipeline. But that’s exactly what happens. Somebody becomes the “message bus person.” Somebody else debugs consumer lag on a Saturday. Meanwhile, the data you actually care about is sitting three hops away from where it needs to be.

What if your app could just push records directly into a Delta table in your lakehouse?

What Zerobus Is (and Why It Beats a Message Bus)

Zerobus Ingest is a serverless, push-based ingestion service that’s part of Databricks Lakeflow Connect. Your application sends data directly to a Delta table in Unity Catalog. No brokers, no partitions, no consumer groups, and no infrastructure to manage at all. You don’t provision anything. You don’t pick instance sizes. You scale by opening more connections, and Zerobus handles the rest.

The architecture shift is straightforward. Before Zerobus, the standard pattern looked like this:

Before: App → Kafka → Spark Streaming → Delta Lake

After: App → Zerobus Ingest → Delta Lake

That middle layer disappears entirely. You go from three systems to one managed service. Your app opens a stream, sends records, and they show up in your table within seconds. The data is buffered durably on the server side and materialized into Delta on a cadence that keeps your table fresh without you thinking about compaction or file sizes.

Here’s what the numbers look like: each stream handles up to 100 MB/s and 15,000 rows/s. Need more? Open more streams. Zerobus supports thousands of concurrent clients writing to the same table, which means you can scale to 10+ GB/s aggregate throughput without managing a single partition. For a company running a few hundred IoT devices, you probably won’t even think about limits. For a company with thousands of devices across regions, you still won’t manage infrastructure, you’ll just open more connections.

A caveat worth mentioning: if you need fan-out to five different consumers reading from the same event stream, keep the message bus. Zerobus is purpose-built for getting your data into the lakehouse without extra hops. It’s a scalpel for the “data needs to land in Delta” problem, not a general-purpose message bus.

But for many startups and smaller enterprises building out their data platform, the lakehouse is the destination. That means one less message broker bill, one less structured streaming compute job, one less dashboard to watch. That’s real money back in your pocket.

How Does it Work?

The whole flow is: create a table, point the SDK at it, send records. Twenty-ish lines of code, no cluster configs, no YAML manifests. Here’s what that looks like.

1. Create your target table

Zerobus doesn’t create or alter tables for you. Define your schema in Unity Catalog first:

CREATE TABLE catalog.schema.sensor_events (
    device_id    STRING,
    temperature  DOUBLE,
    humidity     LONG,
    event_time   TIMESTAMP
);

2. Send a single record

Install the SDK (pip install databricks-zerobus-ingest-sdk) and you’re about 15 lines from ingesting data. SDKs also exist for Java, Go, TypeScript, and Rust, all built on a shared Rust core.

You’ll need a service principal with USE CATALOG, USE SCHEMA, MODIFY, and SELECT grants on your target table. You’ll also need to gather your Zerobus server endpoint and workspace URL. Set up your credentials and connection info as environment variables and go:

import os
from zerobus.sdk.sync import ZerobusSdk
from zerobus.sdk.shared import RecordType, StreamConfigurationOptions, TableProperties

sdk = ZerobusSdk(
    os.environ["ZEROBUS_SERVER_ENDPOINT"],
    os.environ["DATABRICKS_WORKSPACE_URL"],
)

options = StreamConfigurationOptions(record_type=RecordType.JSON)
table = TableProperties(os.environ["ZEROBUS_TABLE_NAME"])

stream = sdk.create_stream(
    os.environ["DATABRICKS_CLIENT_ID"],
    os.environ["DATABRICKS_CLIENT_SECRET"],
    table,
    options,
)

try:
    record = {
        "device_id": "sensor-001",
        "temperature": 22.5,
        "humidity": 55,
        "event_time": 1700000000000000,  # epoch microseconds
    }
    offset = stream.ingest_record_offset(record)
    stream.wait_for_offset(offset)  # blocks until durably written
finally:
    stream.close()

Side note: timestamps are epoch microseconds, not millisecond timestamps and definitely not ISO strings. Get this wrong and your queries will return dates in the year 55000.

JSON works great for prototyping. For production workloads, switch to Protobuf for type safety and better throughput.

3. High-throughput batch ingest

When you’ve got a fleet of devices or services pushing thousands of records per second, you don’t want to block on every ACK. Register an AckCallback handler instead and let the SDK handle durability confirmation in the background:

from zerobus.sdk.shared import AckCallback

class MyAckHandler(AckCallback):
    def on_ack(self, offset: int) -> None:
        print(f"Durable up to offset: {offset}")

    def on_error(self, offset: int, message: str) -> None:
        print(f"Error at offset {offset}: {message}")

options = StreamConfigurationOptions(
    record_type=RecordType.JSON,
    ack_callback=MyAckHandler(),
)

stream = sdk.create_stream(
    os.environ["DATABRICKS_CLIENT_ID"],
    os.environ["DATABRICKS_CLIENT_SECRET"],
    table,
    options,
)

try:
    for i in range(10_000):
        record = {
            "device_id": f"sensor-{i % 200}",
            "temperature": 21.0 + (i % 10),
            "humidity": 40 + (i % 20),
            "event_time": 1700000000000000 + (i * 1000000),
        }
        stream.ingest_record_nowait(record)  # fire-and-forget
    stream.flush()  # send all buffered records before closing
finally:
    stream.close()

That’s it — No cluster configs, no partition math, and no YAML files longer than your actual application code. The flush() call at the end makes sure every buffered record has been sent before you close the stream. Delivery is at-least-once, so if you need deduplication downstream, plan for that in your table design.

4. Monitor Ingestion via System Tables

Zerobus writes logging information to system tables so you can track ingestion without building your own monitoring. Records ingested, bytes written, last commit time, all queryable with SQL:

SELECT
    table_name,
    SUM(committed_records)    AS total_records,
    SUM(committed_bytes)      AS total_bytes,
    MAX(commit_time)      AS last_ingest
FROM system.lakeflow.zerobus_ingest
WHERE table_name = 'catalog.schema.sensor_events'
GROUP BY table_name;

You can also create an AI/BI Dashboard off of these tables to monitor the stream in real-time.

Where This Fits (and Where It Doesn’t)

IoT and manufacturing. Thousands of sensors on constrained networks pushing temperature readings, vibration data, location pings. This is the sweet spot. The SDK is lightweight enough to run on edge gateways, and the choice between gRPC (high throughput, long-lived connections) and REST (stateless, better for chatty low-frequency devices) means you can pick the right protocol for your hardware. Databricks has a partnership with Soracom to stream IoT data from cellular and satellite devices directly into the lakehouse. If your devices can make an outbound call, they can write to Zerobus.

Clickstream and app events. Web and mobile events for personalization, monitoring, experimentation. Every extra hop between “user clicked a button” and “data is queryable” is latency your product team complains about. With Zerobus, your backend pushes the event and it’s in a Delta table seconds later, ready for your analytics queries or ML pipelines.

Partner integrations. Instead of teaching a partner your message bus topology and handing them a runbook, give them an SDK and a service principal. They write to a table you own and govern. You control the schema, the access, and the audit trail. For startups building platforms where third parties send you data, this is a much simpler onboarding story.

When not to use it. As mentioned earlier, Zerobus writes to one destination: Delta tables in your lakehouse. If you need multiple consumers reading from the same event stream in real time, a message bus is still the right tool. Zerobus doesn’t replace the message bus for true fan-out patterns. It replaces the bus for the very common case where you were only using it as a waypoint to get data into the lakehouse anyway. Be honest with yourself about which pattern you actually need. Most of the time, it’s the second one.

Because data lands directly in Unity Catalog governed tables, you get lineage, access controls, and audit logging from the moment the first record arrives. No extra setup for governance. It’s just there.

Get Started

If your ingest pipeline has more moving parts than your actual product, give Zerobus a look. Pick one table, one device fleet, one stream of app events. Wire it up in an afternoon. Watch the data land. Then decide whether you still need that Kafka cluster.

Take a look at this lightweight demo on creating a Zerobus-based stream, which includes a Databricks App for monitoring and shows some tips for deploying with Databricks Asset Bundles.

Consider using the Databricks AI-Dev-Kit for a fast start — it includes a databricks-zerobus-ingest skill that can help your coding agent start with best practices.

Resources

• Zerobus Ingest product page
• Documentation
• Product tour
• Python SDK on GitHub

The Databricks for Startups Field Engineering team works closely with founders and engineers working on ambitious ideas. We love connecting on scalable architecture, data platforms, and the real challenges of bringing data and AI systems into production.

Working on something interesting? We’d be happy to compare notes and see where we can help. Email the author at derek.witt@databricks.com or reach out on LinkedIn.

Databricks Zerobus Ingest — Stop Paying for Pipes You Don’t Need was originally published in Databricks for Startups on Medium, where people are continuing the conversation by highlighting and responding to this story.

Your dashboard is slow. You do what any reasonable person would do: bump the warehouse size to X-Large. Dashboards get faster. Your cloud bill gets uglier.

We’ve all been there. But here’s the thing — “how big should my warehouse be?” is the wrong question. The real question is big how? Should you scale out (more clusters of a smaller size) or scale up (fewer clusters of a bigger size)?

If you’re a startup or smaller team just getting on the platform, this one’s for you. No enterprise-scale assumptions. Just the stuff that matters when you’re trying to move fast and keep costs low.

Three Flavors, One Recommendation

Databricks SQL warehouses come in three types:

• Serverless — Starts in 2–6 seconds, uses Intelligent Workload Management (IWM) to scale elastically, and includes compute cost in the per-DBU price. This is the one you want.
• Pro — Photon engine plus Predictive I/O, but no IWM. Use it when serverless isn’t available in your region, or you need custom networking.
• Classic — Entry-level. Most conservative scaling, no Predictive I/O. Fine for getting started.

If serverless is available in your workspace, you should use it.

One exception: If you’re running something that never stops, like an ingestion job firing every minute from a JDBC connection, serverless doesn’t help you. You’re not benefiting from elastic scaling if the warehouse never scales down. Put a Classic or Pro warehouse on and let it run continuously. That should be its own separate warehouse.

The Only Framework You Need

Forget complicated capacity planning spreadsheets. There are two scaling levers and two signals that tell you which one to pull.

Scale OUT (add more clusters) when…

Your problem is concurrency. Lots of users are hitting dashboards at the same time. Each query is fast on its own, but queries pile up waiting for a slot. This is the classic Monday-morning-dashboard-refresh pattern.

The signal: Peak Queued Queries > 0 on your warehouse’s Monitoring tab. If queries are waiting in line, they don’t need a bigger engine — they need more engines.

Here’s the intuition: Operations like sums, counts, and averages distribute well across nodes. If you have 30 people loading dashboards at 9 AM and all those queries are simple aggregations, giving them three small clusters is better than one big one.

Scale UP (increase warehouse size) when…

Your problem is query weight. You have a handful of heavy queries doing large joins, complex aggregations, or window functions — and they’re slow because the cluster runs out of memory and starts writing intermediate results to disk.

The signal: Open any slow query’s Query Profile and look for “Bytes spilled to disk.” If you see spill, the cluster needs more RAM. Bumping from Small to Medium keeps data in memory and often cuts runtime in half.

Your Warehouse Is Already Telling You What It Needs

You don’t need a third-party observability tool. The Monitoring tab on the SQL warehouse compute page has everything you need. Here’s what to look at:

1. Peak Queued Queries — If this is consistently above zero during business hours, you need more clusters. Period. Increase your max cluster count.
2. Running Clusters — This shows how autoscaling is responding to load. If the chart regularly hits your max cluster cap, your ceiling is too low.
3. Completed Query Count — Correlate query volume with time of day. You’ll find burst windows that explain queuing spikes.
4. Activity Details (beta) — Color-coded annotations showing busy vs. idle periods. Useful for spotting patterns at a glance.
5. Query Profile — This is where you find “Bytes spilled to disk.” It lives in the query history detail view. Click into a slow query and inspect the profile.

Two Scenarios, Startup-Sized

Let’s make it concrete with Serverless pricing at $0.70/DBU (AWS US list).

Scenario A: Scale Up Saves Money

A daily report runs on an X-Small (6 DBU/h). It takes 12 minutes and spills to disk. On a Small (12 DBU/h), spill disappears and runtime drops to 5 minutes.

• X-Small: 6 DBU/h x $0.70 x (12/60) = $0.84 per run
• Small: 12 DBU/h x $0.70 x (5/60) = $0.70 per run

Bigger warehouse, faster results, and actually cheaper per run. The monitoring data told you exactly what to do.

Scenario B: Scale Out Saves Money

A BI team of 25 opens dashboards twice a day in 20-minute bursts. The queries are light; they need admission slots, not horsepower.

• 1x Small: (12 DBU/h x $0.70 x 0.33h) x 2 bursts = ~$166/month
• 2x 2X-Small clusters: 2 x (4 DBU/h x $0.70 x 0.33h) x 2 bursts = ~$111/month

Same concurrency capacity, a third less cost by going smaller and wider.

Separate Your Warehouses

Don’t run everything through a single warehouse. Split them by team and workload. For example:

1. BI users get a small warehouse (2X-Small or X-Small) with a generous max cluster count. If you have a team loading dashboards, let them scale out wide and back down quickly.
2. Data engineering gets a slightly bigger warehouse (Small or Medium) with fewer clusters. These jobs tend to be heavier but less concurrent.
3. Always-on ingestion gets its own Classic or Pro warehouse, sized to the job, running continuously. No point paying for serverless elasticity if the thing never stops.

Start Small and Right-Size

Here’s where we break from the usual advice. If you’re a startup, start small. Go with an X-Small or Small and see what happens. Most teams don’t need big warehouses, and you can always scale up when the monitoring tells you to.

The conventional wisdom says to start big so nobody complains about slowness. That makes sense for a Fortune 500. But if you’re a team of 20, 50, or even 200, be scrappy. Save the money. Watch the signals. Scale when the data says so.

A few parting tips:

• Cap your max clusters to protect your budget. Autoscaling is smart, but it’s not your finance team.
• Set auto-stop aggressively for Serverless — 10 minutes in the UI, down to 1 minute via API. You’re not paying for idle time if there is no idle time.

Warehouse sizing isn’t about picking the biggest option you can afford. It’s about matching your workload shape to the right combination of cluster size and cluster count. Use the monitoring signals and let the data tell you which lever to pull.

Your dashboards will be faster. And your budget will thank you.

This article was written by Derek Witt and John Neil — Databricks Solutions Architects who work daily with Startups of all sizes and stages.

Scale Out or Scale Up? Sizing Your DBSQL Warehouses Without Wasting Money was originally published in Databricks for Startups on Medium, where people are continuing the conversation by highlighting and responding to this story.

The “Close Enough” Problem

Here’s a scenario you’ve probably lived: you ask your AI assistant to build a Databricks pipeline. Thirty seconds later, it hands you something that… works. Technically. It ingests data. It writes to Delta. It even has a comment or two.

But look closer. It’s using the legacy import dlt API instead of the new pyspark.pipelines module. It partitions by date instead of using Liquid Clustering. There’s no rescue data handling for malformed records — so when a malformed JSON blob shows up (and it will), it silently poisons your downstream tables. No schema hints, meaning your column types are one upstream change away from drifting. No file metadata for lineage. No ingestion timestamps. It’s generating code the way Databricks worked two years ago, and it doesn’t know what it doesn’t know.

Let me be honest: recent AI models are impressive. They do know Databricks fundamentals, and most of the time, they’ll get you something functional. The problem is that “close enough” works fine — until you’re scaling to production, debugging a silent data quality issue at 2 AM, or explaining to your CTO why you need two sprints for a refactor that “should have been built right the first time.”

Databricks Moves Fast (Even for AI)

Databricks ships platform changes at a pace that makes most LLM training data feel stale. Spark Declarative Pipelines (the modern replacement for Delta Live Tables), Liquid Clustering, Lakebase, Agent Bricks, ResponsesAgent. All relatively recent, all representing meaningful shifts in how you should build.

Models trained on documentation from six to eighteen months ago miss these paradigms. And field engineers at Databricks see the consequences daily: customers (and their AI assistants) reaching for patterns that worked fine in 2023 but now have strictly better alternatives.

Some quick examples of what’s shifted:

• PARTITION BY should be CLUSTER BY (Liquid Clustering). import dlt should be from pyspark import pipelines as dp.
• Manual cluster sizing should be serverless-first. @dlt.table() should be @dp.table() with cluster_by instead of partition_cols.
• The old dlt.read() should be spark.read.table() with explicit Unity Catalog paths.

Each of these is a small thing. Together, they’re the difference between a modern architecture and a legacy one that’ll need rework before you hit your Series B.

What the AI Dev Kit Actually Is

The Databricks AI Dev Kit was created by Databricks field engineers, people who work with customers every day and see which patterns succeed in production and which ones create support tickets.

It has three layers:

1. Skills (20+ curated pattern libraries). These aren’t documentation dumps. They’re field-tested playbooks organized by domain. The Spark Declarative Pipelines skill alone has 10 files covering ingestion patterns, streaming, SCD Type 2, performance tuning, DLT migration, and project initialization. Model Serving covers 9 stages from training to production deployment. Each skill encodes the judgment calls that typically require a senior Databricks engineer in the room.
2. MCP Server (50+ tools). Your AI assistant can actually talk to your Databricks workspace — create pipelines, deploy models, run SQL, manage Unity Catalog. It’s not just generating code; it’s executing it.
3. Python library + starter template. For programmatic access and scaffolding new projects with the right structure from day one.

It’s a certified Gold Project, fully open source, and actively maintained by 20+ Databricks Field Engineers. Works with Claude Code, Cursor, GitHub Copilot, and OpenAI Codex.

The Difference in Practice

Let’s say that you ask your AI something like: “Build me a streaming ingestion pipeline for JSON files landing in cloud storage.”

Without the AI Dev Kit, you get something like this:

import dlt

@dlt.table(
  name="bronze_events",
  partition_cols=["date"]
)

def bronze_events():
  return (
    spark.readStream
    .format("cloudFiles")
    .option("cloudFiles.format", "json")
    .load("/mnt/raw/events/")
  )

It runs. It ingests data. But it’s using the legacy DLT API (import dlt was replaced by pyspark.pipelines), partitioning instead of Liquid Clustering, reading from /mnt/ paths instead of Unity Catalog Volumes, and has no schema hints, no rescue data handling, no ingestion timestamps, and no file lineage tracking. Every one of those gaps is a future production incident waiting to happen.

With the AI Dev Kit Spark Declarative Pipelines skill, a similar request produces:

from pyspark import pipelines as dp
from pyspark.sql import functions as F
from pyspark.sql.types import StructType, StructField, StringType

# Schema hints prevent type drift
event_schema = StructType([
    StructField("event_id", StringType(), nullable=False),
    StructField("user_id", StringType(), nullable=False),
    StructField("event_type", StringType(), nullable=True),
    StructField("event_timestamp", StringType(), nullable=True),
    StructField("amount", StringType(), nullable=True),
    StructField("country_code", StringType(), nullable=True)
])

@dp.table(
    name="bronze_events",
    cluster_by=["event_type", "country_code"],  # Liquid Clustering
    comment="Raw events with rescue data for malformed records",
    table_properties={
        "quality": "bronze",
        "delta.enableChangeDataFeed": "true"
    }
)

def bronze_events():
    return (
        spark.readStream
        .format("cloudFiles")  # Auto Loader for incremental ingestion
        .option("cloudFiles.format", "json")
        .option("cloudFiles.schemaLocation", "schema_location/bronze_events")
        .option("rescuedDataColumn", "_rescued_data")  # Capture malformed records
        .schema(event_schema)  # Schema hints prevent type drift
        .load("/Volumes/main/analytics/landing_zone/events/")  # Unity Catalog Volume
        .withColumn("_source_file", F.col("_metadata.file_path"))
        .withColumn("_file_modified_at", F.col("_metadata.file_modification_time"))
        .withColumn("_ingested_at", F.current_timestamp())
    )

Liquid Clustering instead of partitioning. Schema hints to prevent type drift. Rescue data handling so malformed records get flagged instead of silently corrupting downstream tables. File metadata for lineage. Unity Catalog Volumes for storage instead of (now deprecated) DBFS. Serverless compute by default. Medallion architecture naming conventions baked in.

The first version ships today and creates a refactoring ticket for the next sprint. The second version ships today and you move on to the next thing.

Why Startups Should Care

Startups often optimize for two things: shipping speed and minimizing tech debt. The AI Dev Kit helps with both.

You ship better code because your AI assistant has access to the same patterns that Databricks field engineers recommend to enterprise customers. But you also learn faster as the skills are educational, not just templates. When your AI generates a pipeline with Liquid Clustering and rescue data handling, you’re absorbing modern Databricks patterns through the code itself.

Technical debt compounds. The wrong architectural choices made at speed become expensive migrations six months later when you’re trying to scale. Getting it right the first time isn’t perfectionism; it’s pragmatism. And let’s be real, you probably don’t have a dedicated Databricks platform engineer on staff yet.

The AI Dev Kit gives your team senior-engineer-level Databricks judgment without the senior engineer. It’s like having a staff architect review every pipeline before it ships, except it happens at generation time, not in code review.

There’s a cost angle too. Serverless-first patterns, proper resource sizing, and Liquid Clustering instead of over-partitioned tables directly affect your Databricks bill.

In the end, you still need to understand what you’re building and review what your AI generates, but it dramatically reduces wrong turns and dead ends.

Getting Started

To get started, take a look at the AI-Dev-Kit repo.

Navigate to your project root using your terminal and install in one line:

curl -sL https://raw.githubusercontent.com/databricks-solutions/ai-dev-kit/main/install.sh | bash

You can start with skills-only mode (just context, no workspace access) or go full MCP server (context + execution). Either way, setup takes less than two minutes, and you’re on your way to greatly enhancing your Databricks development experience!

Your AI Dev Tool Knows Databricks. The Databricks AI-Dev-Kit Makes It an Expert. was originally published in Databricks for Startups on Medium, where people are continuing the conversation by highlighting and responding to this story.

Day 9 of 12 Days of Demos

This post is part of our 12 Days of Demos holiday series with the North Pole Modernization Office, a hands-on journey showing how Santa and the elves run their data, analytics, and AI operations on Databricks. Each installment builds on the last as we assemble a full, end-to-end Lakehouse architecture that powers the magic of the season.

If you’re just joining us, start with the series introduction to get the full story and setup checklist, and explore the public GitHub repo where you’ll find all notebooks, datasets, and assets you’ll need to follow along. Then check back daily on the Startups at Databricks Medium blog for the next chapter in Santa’s Lakehouse transformation!

If you run into any issues, email g@databricks.com!

The North Pole Modernization Office has come a long way. We have functions that query workshop production, reindeer telemetry, delivery routes, and behavioral analytics. We built a semantic search system over Santa’s letters. But Santa’s questions are getting more sophisticated:

“Find letters from children who seem especially grateful and creative, then check if any of them are on the coal warning list — I want to make sure we’re not missing good kids who had a rough year.”

A single agent with tools can query the behavioral data. But understanding the content and sentiment of the letters? That requires semantic search over unstructured text. We need an agent that can reason over letters and one that can execute structured queries, working together.

That’s where Agent Bricks comes in.

What is Agent Bricks?

Agent Bricks provides a streamlined way to build and optimize domain-specific AI agent systems. Instead of writing custom orchestration code, you configure what you need and Databricks handles the rest — model selection, fine-tuning, optimization, and deployment.

Agent Bricks currently supports four patterns: Information Extraction for turning documents into structured tables, Custom LLM for summarization and classification, Knowledge Assistant for Q&A over your documents, and Multi-Agent Supervisor for coordinating multiple agents and tools.

Today we’ll build a Knowledge Assistant to answer questions about Santa’s letters (building on the semantic search work from Day 7), then combine it with our MCP-connected operational functions (from Day 8) in a Multi-Agent Supervisor.

Part 1: Building a Knowledge Assistant

The Knowledge Assistant creates a RAG-powered chatbot that can answer questions and cite its sources. In Day 7, we built a vector search index over Santa’s letters using databricks-gte-large-en. The Knowledge Assistant can use that index directly — we just need to point it at the right data.

Create the Knowledge Assistant

Navigate to Agents in the left navigation pane and click Knowledge Assistant.

Configure the agent with a name like npmo-letter-assistant and a description: "Answers questions about children's letters to Santa, including sentiment, gift requests, tone, and Christmas spirit."

Under Knowledge Source, click Add knowledge source and select Vector Search Index. Choose the santa_letters_index you created in Day 7. For the Doc URI Column, select the column that identifies each letter (like name). For the Text Column, select letter. Give it a name like santa-letters and describe the content: "Letters from children to Santa containing gift requests, expressions of gratitude, and holiday wishes."

Optionally add instructions to guide behavior:

You are Santa's Letter Analysis Assistant. Answer questions about the 
children's letters based on their content, tone, and sentiment. You can 
find letters by concept (grateful, excited, creative) not just keywords. 
Always cite which letters support your answers.

Click Create Agent and wait a few minutes for it to build.

Test the Knowledge Assistant

Once ready, a chat window will pop-up to test from or you can try it in Playground.

Try questions like:

“Find me letters from children who seem especially grateful and polite.”

“Which letters show the most Christmas spirit and enthusiasm?”

“Are there any letters requesting unusual or creative gifts?”

The agent responds with answers grounded in the actual letters, complete with citations. This is the same semantic understanding we built in Day 7, now wrapped in a conversational agent.

Part 2: Building a Multi-Agent Supervisor

Now the real magic. We’ll create a supervisor that coordinates the Knowledge Assistant (for letter analysis) with the Unity Catalog functions we built in Day 8 (for operational data).

Create the Multi-Agent Supervisor

Navigate to Agents → Multi-Agent Supervisor.

Configure the supervisor with a name like npmo-operations-supervisor and a description: "Coordinates North Pole operations by combining letter analysis with real-time operational data. Can find letters by sentiment or content, then cross-reference with behavioral scores, production metrics, and delivery data."

Under Configure Agents, add your sub-agents:

Sub-agent 1: Letter Analysis (Knowledge Assistant)

Select Agent Endpoint as the type, then choose npmo-letter-assistant (Note that the Agent Endpoint might be something a little different than the Agent Name). Name it something like Letter Analyst and describe it: "Searches and analyzes children's letters to Santa. Can find letters by tone, sentiment, gift preferences, or Christmas spirit. Use this when questions involve the content of letters." You can also let these default to the values we put into the Knowledge Assistant.

Sub-agent 2: Operational Functions (Unity Catalog)

Select Unity Catalog Function as the type. Add the functions we created in Day 8. In the future, this will leverage Managed MCP rather than individual functions.

• get_coal_warning_list — Children currently flagged for coal
• get_top_nice_children — Highest-scoring children
• get_child_naughty_nice_score — Look up a specific child's score
• get_behavior_events_by_type — Analyze behavioral patterns
• get_low_energy_reindeer — Reindeer needing rest
• get_team_quality_metrics — Elf team performance
• get_workshop_material_totals — Workshop material usage information

For each function, the description comes from the SQL COMMENT we added in Day 8.

Add instructions to guide coordination:

You are the North Pole Operations Supervisor. When answering questions:

1. Determine if the question requires letter analysis, operational data, 
   or both.
2. For questions about letter content, sentiment, or tone, consult the 
   Letter Analyst.
3. For questions about scores, metrics, or status, use the operational 
   functions.
4. For complex questions, gather information from multiple sources and 
   synthesize a complete answer.

Always explain which sources you consulted.

Click Create Agent and wait for it to build.

Test the Supervisor

Once deployed, test with questions that require coordination across both domains:

“Find letters from grateful, creative children, then tell me if any of them are on the coal warning list.”

Watch the supervisor work:

1. Route the letter question to the Knowledge Assistant
2. Extract child names from the results
3. Call get_coal_warning_list() to check their status
4. Synthesize both into a coherent answer

More scenarios to try:

“Which children wrote the most enthusiastic letters but have low naughty-nice scores? Something doesn’t add up.”

“Find letters mentioning art supplies or creative gifts. How is the workshop team handling craft-related production?”

“Show me the top 10 nicest children and find their letters so I can see what they asked for.”

What’s Happening Under the Hood

The Multi-Agent Supervisor uses advanced orchestration to parse intent and understand what domains the question touches, route to the right specialist (Knowledge Assistant or UC functions), manage context between sub-agents, synthesize results into a coherent response, and enforce access controls so users only see results they’re authorized to access.

You didn’t write any of this orchestration logic. Agent Bricks built it.

Improving Quality with Human Feedback

Both Knowledge Assistant and Multi-Agent Supervisor support improvement through labeling sessions. The workflow: add sample questions to the Improve tab, start a labeling session and share with subject matter experts, have reviewers provide feedback via the Review App, then merge feedback and the system retrains.

For the North Pole, this means Santa’s senior elves can flag when the agent misinterprets a letter’s tone or makes a bad routing decision, and the system improves over time.

Wrapping Up

Today we combined a Knowledge Assistant specializing in Santa’s letters and with our operational functions in a Multi-Agent Supervisor. The North Pole Modernization Office now has an AI system that can reason over letters and query operational data — and coordinate between the two.

The pattern applies beyond holiday logistics. Complex enterprise questions rarely live in a single system. Agent Bricks lets you build specialized agents for each domain and orchestrate them without writing custom orchestration code.

Following along with the series? We’d love to hear how it’s going. Reach out to your Databricks account team with questions, or connect with me on LinkedIn.

12 Days of Demos: The Supervisor Claus with Agent Bricks was originally published in Databricks for Startups on Medium, where people are continuing the conversation by highlighting and responding to this story.

Day 8 of 12 Days of Demos

This post is part of our 12 Days of Demos holiday series with the North Pole Modernization Office, a hands-on journey showing how Santa and the elves run their data, analytics, and AI operations on Databricks. Each installment builds on the last as we assemble a full, end-to-end Lakehouse architecture that powers the magic of the season.

If you’re just joining us, start with the series introduction to get the full story and setup checklist, and explore the public GitHub repo where you’ll find all notebooks, datasets, and assets you’ll need to follow along. Then check back daily on the Startups at Databricks Medium blog for the next chapter in Santa’s Lakehouse transformation!

If you run into any issues, email g@databricks.com!

The North Pole Modernization Office has made great progress. We’ve ingested data with Auto Loader, built reliable pipelines with SDP, enriched datasets with batch AI functions, and packaged business logic into Unity Catalog functions. But functions sitting in a catalog don’t help an AI agent answer questions.

That’s where Databricks Managed MCP Servers come in.

Model Context Protocol (MCP) is an open standard that lets AI agents access external data and tools. Think of MCP servers as bridges between your LLM and the systems that hold your business logic. Databricks provides managed MCP servers that work out of the box. Point an agent at a simple URL, and every function in your schema becomes a callable tool.

The URL pattern:

https://<workspace-hostname>/api/2.0/mcp/functions/<catalog>/<schema>

No containers to deploy. No infrastructure to manage. Just governed, version-controlled functions exposed as AI-ready tools.

Step 1: Initialize the Data

First, clone the 12 Days of Demos repository into your Databricks workspace. We’ll work through a demo notebook to prepare our data and functions.

Open the mcp-demo.ipynb notebook in the 08-mcp folder, optionally update the configuration, and run the following cells:

TARGET_CATALOG = "workspace"
TARGET_SCHEMA = "dbrx_12daysofdemos"
TARGET_VOLUME = "raw_data_volume"

%run "../00-init/load-data"

This step:

• Sets your Unity Catalog context
• Creates a Volume for staging files
• Ingests synthetic CSV data into managed Delta tables

The datasets cover core North Pole operations:

gift_requests Letters to Santa with extracted gift preferences reindeer_telemetry Flight metrics, heart rate, energy efficiency workshop_production Toy manufacturing by elf team and shift behavioral_analytics Naughty/nice scores and behavioral events delivery_optimization Routes, weather delays, chimney accessibility

Step 2: Create the Functions

This step registers SQL functions that encapsulate queries an AI agent can invoke.

Here’s an example that returns workshop material usage:

CREATE OR REPLACE FUNCTION get_workshop_material_totals()
RETURNS TABLE(total_plastic_kg DOUBLE, total_wood_kg DOUBLE, 
              total_metal_kg DOUBLE, total_fabric_kg DOUBLE)
COMMENT 'Returns total usage of all materials in the workshop.'
RETURN (
  SELECT 
    SUM(materials_plastic_kg) as total_plastic_kg,
    SUM(materials_wood_kg) as total_wood_kg,
    SUM(materials_metal_kg) as total_metal_kg,
    SUM(materials_fabric_kg) as total_fabric_kg
  FROM workshop_production
);

The COMMENT is important — it tells the AI agent what the function does, helping it decide which tool to call.

The notebook creates 20+ functions spanning the full operation:

• Workshop: get_team_quality_metrics(), get_shift_performance(), get_toy_defect_rate()
• Reindeer: get_reindeer_flight_metrics(), get_flying_reindeer_status(), get_low_energy_reindeer()
• Behavior: get_top_nice_children(), get_coal_warning_list(), get_child_naughty_nice_score()
• Delivery: get_weather_impact_on_delivery(), get_chimney_accessibility(), get_delivery_route_metrics()

Connect to Serverless compute and run all cells below Step 2 to register the functions in Unity Catalog.

Testing with AI Playground

Now we connect the pieces. Navigate to AI Playground in your workspace.

1. Select a model with the Tools enabled label
2. Click Tools → + Add tool → Hosted Function
3. Browse to your catalog and schema, then select the functions you want to expose

Let’s also set a system prompt for our Playground agent:

You are Santa’s Chief Planning Officer for his holiday season efforts. Santa is going to ask you plain text questions about elements of his toy production and delivery planning, along with other topics. Use the MCP tools provided to answer the questions.

With the tools attached and the system prompt created, try some natural language queries:

“Which elf team has the best quality scores?”

“Show me reindeer with energy efficiency below 0.7 — who needs a carrot break?”

“How much does a blizzard slow down our delivery times compared to clear weather?”

“Who are the top 10 nicest children this year?”

The agent recognizes which function to call, executes it against your data, and synthesizes the results into a natural response.

The Magic Behind the Scenes

When you add Unity Catalog functions as tools in AI Playground, it connects to the Managed MCP Server at:

https://<your-workspace>/api/2.0/mcp/functions/12daysofdemos/npmo

The server automatically:

• Exposes each function as a callable tool
• Uses the COMMENT as the tool description
• Handles parameter validation and type conversion
• Returns structured results the LLM can reason over

This is the value of managed infrastructure: you focus on the business logic, Databricks handles the plumbing.

Export to Deployable Code

Click Get code in AI Playground to generate a Python notebook with a Pydantic-based agent. This gives you code you can customize, test programmatically, and deploy to a Model Serving endpoint.

Connect External Clients

Managed MCP Servers work with any MCP-compatible client — including tools like Cursor and Claude Desktop. Your Unity Catalog functions become tools anywhere MCP is supported.

Combine Data Sources

Tomorrow we’ll explore Agent Bricks and multi-agent orchestration. But the pattern we’ve established today — governed functions exposed via MCP — is the foundation. You can combine:

• UC Functions MCP — Custom business logic (what we built today)
• Vector Search MCP — Semantic search over documents
• Genie MCP — Natural language queries against tables
• DBSQL MCP — Run AI-generated SQL.

An agent could search documentation, query structured data, and run business logic — all in one conversation.

Wrapping Up

Today we turned Unity Catalog functions into AI-ready tools with zero infrastructure work. The North Pole Modernization Office now has an AI operations center where agents can answer questions about workshop production, reindeer health, delivery logistics, and the all-important naughty-or-nice list.

The pattern applies beyond the North Pole. Your organization has business logic locked in dashboards, stored procedures, and tribal knowledge. Unity Catalog functions + Managed MCP lets you expose that logic to AI in a way that’s auditable, secure, and maintainable.

Up next…

We take the next step with Agent Bricks, orchestrating multi-agent systems that can reason, plan, and execute complex tasks. If you missed earlier days, catch up here.

Happy holidays, and happy data engineering! 🎄🦌

Following along with the series? We’d love to hear how it’s going! Reach out to your Databricks account team with questions, or connect with me on LinkedIn.

12 Days of Demos: Santa’s AI Toolkit with Managed MCP was originally published in Databricks for Startups on Medium, where people are continuing the conversation by highlighting and responding to this story.