By AJ Franchino (Solutions Architect at Databricks)

I’ve had the same conversation a few times these last few months. It usually starts with this pause. The kind you get when you ask someone how they liked the pizza place you recommended and they start with “so…”

“Yeah, I tried the Assistant a while ago. It was slow, the results were mid at best, and I turned it off. Haven’t touched it since.”

And they say it with this finality. Like the verdict is in. Case closed. Moving on.

I get it. If you sat there waiting for it to think, and what came back wasn’t worth the wait, I’m not going to press you. That was the product at the time.

But while you were busy writing it off, it got better. And then it got better again. Quietly, without making a scene or being showy about it, the team took a mountain of feedback and built it all in. New capabilities showed up month after month. Agentic features landed. The whole thing was rebuilt from the ground up.

And now it has a new name: Genie Code.

This isn’t a rebrand slapped on the same product. This is a different tool. And I’d argue it’s the most contextually aware, far-reaching free AI assistant on any data platform right now.

From Autocomplete to Autonomous Agent

The old Assistant was basically autocomplete with a chat box. Good for some middling SQL, but nothing to blog about.

Genie Code is an agent. It reasons through problems. Plans multi-step approaches. Writes code, runs it, checks the output, fixes its own mistakes. And it does this across notebooks, SQL editors, Lakeflow pipelines, and dashboards.

But listing features doesn’t really capture what changed. So let me walk you through something I watched play out last week. One scenario, start to finish, to show you how these pieces actually connect.

One Ask, Five Surfaces

A startup I work with has a customer support operation generating thousands of tickets a week. Their head of data wanted to understand which product areas were driving the most support volume, predict which open tickets were likely to escalate, and give the support leads a dashboard they could actually use every morning. That’s three problems. Used to be three separate work-streams.

She opened a notebook and started with Genie Code.

Finding the data. First thing she typed: “What tables do we have related to support tickets and product usage?” Genie Code searched Unity Catalog, surfaced the relevant tables, showed her the schemas, and flagged a product telemetry table she didn’t even know existed. The lineage view connected everything. Two minutes in and she already had a better picture of what data was available than she’d gotten from a week of Slack threads with the engineering team.

Exploring the data. She asked Genie Code to profile the support ticket table and look for patterns. It wrote the code, ran it, and came back with a breakdown: ticket volume by product area, resolution times by severity, a time-series showing a spike in one product category that correlated with a recent release. It laid out a plan for the analysis and worked through it step by step. She didn’t write a single line of code. She just asked questions and steered.

Training a model. “Can you build a model that predicts which tickets are likely to escalate?” Genie Code put together the whole experiment. Feature selection from both the ticket and telemetry tables. Data profiling and cleaning. Train-test split. It tried a few approaches, evaluated each one, logged everything to MLflow. When the first model underperformed, it iterated on the feature engineering without being asked. The whole thing ran in the notebook while she answered emails.

Building the dashboard. She switched to dashboards and typed “Build me a morning operations dashboard for support leads” with an @mention pointing to her notebook. Genie Code pulled context from the analysis she’d just done, created the datasets, and built a multi-panel dashboard. Ticket volume trends, escalation predictions, product area breakdowns, SLA tracking. Not a wireframe. A working dashboard her team could open the next morning.

Setting up the pipeline. The data needs to stay fresh. She asked Genie Code to create a Lakeflow pipeline to keep the ticket and telemetry data up to date. It generated the pipeline definition, validated it, and when it hit an error on the first run, it read the stack trace, fixed the transformation, and reran. She didn’t touch the code.

One person. One afternoon. From “what data do we have?” to a predictive model, a live dashboard, and an automated pipeline. That’s not a feature list. That’s a workflow that would have taken a small team a few weeks.

The Part Nobody Talks About

Here’s what makes this different from just having a really good coding assistant in your terminal. Genie Code moves with you across surfaces. The context carries. When she mentioned her notebook in the dashboard prompt, it knew what she’d built. When she set up the pipeline, it understood the tables she’d been working with. It’s not starting from scratch every time you switch tools.

And because it’s wired into Unity Catalog, every table it finds, every query it writes, every model it logs respects the permissions and governance your team already has in place. No separate security review. No data leaving the perimeter.

MCP Server Connections. This is the one people sleep on. Genie Code connects to external tools through MCP. Jira, Confluence, GitHub, Google Drive, custom internal tools. It lives in Databricks but reaches out to wherever your team actually works.

Custom Instructions and Agent Skills. You can teach it how your team does things. Your approach to PII masking. Your data validation framework. Your naming conventions. It learns and applies those patterns across sessions.

Documentation Search with Citations. Ask it “What is Unity Catalog?” or “How do I set up a streaming table?” and it pulls from the actual Databricks docs, with links. For new hires ramping up on the platform, this alone is worth opening the panel for.

Why This Hits Different for Startups

You don’t have a dedicated Databricks admin. You don’t have a senior data engineer who’s been on the platform for five years and knows where all the bodies are buried. You have a small team moving fast, figuring things out as they go.

And some of those people have never used an AI coding tool at all. No Copilot, no Claude Code, no Cursor. For them, Genie Code is the easiest possible on-ramp. No CLI to install. No config files. No API keys. It’s already sitting in their workspace. They just have to open the panel.

Genie Code fills that gap. It’s the experienced platform engineer sitting next to every person on your team, answering questions, building things, debugging problems. And it’s doing all of this for free. No additional cost. You pay for the compute you already use. That’s it. Probably even less given that it’ll finish in less time.

For a startup watching every dollar, that math is hard to argue with.

The Quiet Comeback

I think the reason Genie Code didn’t get a big victory lap sooner is that the team was too busy shipping. Every few weeks, something new would show up. Agent mode got more capable. MCP support landed. Custom instructions became more flexible. The context window got smarter about pulling in relevant tables and schemas.

There was no single moment where it went from “meh” to “holy shit.” It was a steady climb. But if you step back and compare where it was a year ago to where it is today, it’s not even close.

Databricks published a benchmark: 77% success rate on real-world data science tasks, versus 32% for leading coding agents. Take that however you want. I just know what I’ve been seeing in the room.

For the Skeptics

I’m talking directly to you. The person who tried the Databricks Assistant back when it was early, got burned, and hasn’t opened the panel since. I know you’re out there because I keep meeting you.

The product you remember doesn’t exist anymore. What replaced it is an autonomous agent that understands your data, your governance, your entire workspace. And it ships work. Not suggestions. Work.

So open a notebook. Click the Genie Code panel. Give it something hard. The thing that would take your team hours. See what happens.

It’s free. You’ve got nothing to lose except the grudge.

Got Hooked? Go Deeper.

If Genie Code grabs you the way it grabbed the skeptics I’ve been converting, check out the AI Dev Kit. It’s the open-source toolkit for building production AI applications on Databricks, and a natural next step once you see what’s possible when the platform is actually working with you instead of next to you.

The best comeback stories don’t start with a press release. They start with someone reluctantly trying the thing again and going “…oh.”

The Databricks for Startups Field Engineering team works closely with founders and engineers working on ambitious ideas. We love connecting on scalable architecture, data platforms, and the real challenges of bringing data and AI systems into production.

Working on something interesting? We’d be happy to compare notes and see where we can help. Email us at aj@databricks.com or reach out on LinkedIn.

The Genie Code Conversation I Keep Having (And How It Keeps Ending) was originally published in Databricks for Startups on Medium, where people are continuing the conversation by highlighting and responding to this story.

In Part 1, we covered how Beusa Energy went from concept to a deployed AI red-teaming platform in just a few weeks. Speed was the headline, but speed without structural integrity is just a prototype. For the platform to hold up at enterprise scale, the architecture underneath had to be airtight.

The Red Team Orchestrator: A Three-Actor Orchestration Loop

At the core of the platform is an orchestration pattern Nick calls “The Red Team Orchestrator.” Every assessment runs three actors in a loop:

Simulator (Attack Model) generates adversarial prompts designed to exploit a specific vulnerability. It uses 20 attack methods, from single-turn techniques like prompt injection and leetspeak encoding to sophisticated multi-turn strategies like crescendo jailbreaking, where a conversation is gradually escalated from benign to harmful over many turns.

Target (Defender) is the model under test. It receives each adversarial prompt and responds as it would in production. The target can be any model from any provider: Databricks-hosted foundation models, external models via AI Gateway, or custom endpoints.

Evaluator (Judge) scores whether the target model resisted or was compromised. It analyzes the target’s response against the specific vulnerability being tested and assigns a pass/fail verdict with a confidence score and reasoning.

This loop runs across 24 vulnerability categories and 92 granular sub-types, covering bias, toxicity, PII leakage, jailbreaking, industry frameworks like OWASP LLM Top 10, agentic safety risks like excessive agency and recursive hijacking, and API security concerns like SSRF and SQL injection.

“The Red Team Orchestrator isn’t just a scan. It’s a structured adversarial simulation. The simulator is actively trying to break the model using the same techniques a real attacker would. The evaluator is a separate model judging the result independently.”

Here’s the part that makes or breaks it: the Simulator and Evaluator must use unfiltered APIs. If the model generating attacks has its own content filters, it will refuse to produce adversarial prompts, and the entire assessment falls apart. The team routes these actors through direct API access (via AI Gateway external model endpoints) while the Target model can use whatever configuration it runs in production, filters and all.

Beyond Red Teaming: Industry Benchmarks and RAG Accuracy

Safety testing tells you whether a model can be broken. But teams also need to know how a model performs before they trust it in production.

At the beginning of each assessment, the platform pulls 17 industry benchmarks and three model performance metrics from Artificial Analysis. Benchmarks include LiveCodeBench, TerminalBench (Hard), Math500, MMLU Pro, and others. This gives teams an objective baseline: before you even run adversarial tests, you can see where a model ranks on general capability relative to the field.

For RAG-specific accuracy, the platform includes OfficeQA, a set of 246 grounded reasoning questions backed by real U.S. Treasury Bulletin documents spanning 1939 to 2025. Instead of testing whether a model can resist adversarial prompts, OfficeQA tests whether a model can accurately answer questions when provided with document context, catching hallucination and grounding failures that traditional red teaming misses.

Document data is stored in a Unity Catalog Volume in three formats (pre-processed text, parsed JSON, and raw PDF), and benchmark jobs process all 246 questions against the target model with configurable scoring tolerance. Results are tracked across easy and hard difficulty tiers, with a leaderboard that ranks models by accuracy over time.

Together, teams get three answers before a model hits production: is it capable, is it accurate, and is it safe.

Infrastructure as Code: DABs as the Enterprise Glue

One of the first questions any security-minded team asks about a new internal tool is: How do I deploy this consistently and control who has access to what?

The answer is Databricks Asset Bundles (DABs). The platform’s entire infrastructure is defined in a single databricks.yml file that provisions everything on deploy:

• A serverless SQL Warehouse for query execution (auto-scales, no idle cost)
• A Unity Catalog schema with scoped grants for the app’s service principal
• A Unity Catalog Volume for OfficeQA benchmark documents
• The Databricks App itself, with explicit resource bindings for the warehouse, jobs, and every model serving endpoint it needs to query
• Three serverless Jobs: the assessment runner, the OfficeQA benchmark runner, and a bootstrap job for initial data setup

The app handles its own credential lifecycle. Users can create Databricks-backed secret scopes and secrets directly from the app UI. Users can also add secrets to existing Azure Key Vault-backed secret scopes, ensuring that API keys for providers never appear in code, logs, or developer environments.

Every resource gets the minimum permissions it needs. The app’s service principal is granted CAN_QUERY on specific serving endpoints, CAN_MANAGE_RUN on its jobs, and ALL_PRIVILEGES on its own schema. Nothing more.

# Excerpt from databricks.yml - App resource bindings
apps:
  red_team_app:
    name: databricks-llm-red-teamer
    resources:
      - name: sql-warehouse
        sql_warehouse:
          id: ${var.warehouse_id}
          permission: CAN_USE
      - name: red-team-job
        job:
          id: ${resources.jobs.red_team_assessment_job.id}
          permission: CAN_MANAGE_RUN
      - name: endpoint-claude-sonnet-4-6
        serving_endpoint:
          name: databricks-claude-sonnet-4-6
          permission: CAN_QUERY

The bundle supports Dev, UAT, and Production targets. Deploying to a new environment is a single command: databricks bundle deploy -t prod. The same infrastructure, the same permissions, the same governance controls, every time.

DABs themselves has been evolving rapidly. The new Direct Deployment Engine (experimental, expected to become the default in 2026) eliminates the Terraform dependency entirely, and features like lifecycle prevent_destroy now protect critical resources — like audit trail tables — from accidental deletion. And with the recent addition of databricks bundle plan, teams can preview exactly what a deployment will change before executing it.

“DABs turned the deployment story from ‘let me walk you through the setup’ to ‘run this one command.’ That matters when the people who need this tool aren’t DevOps engineers.”

Dynamic Model Management: No Redeploy Required

New models land constantly. The team needed a way to onboard them for testing without redeploying the entire application.

The platform uses a tiered approach built on the Databricks Python SDK. From within the app, the EndpointService can create and manage AI Gateway endpoints for external models at runtime. When a user wants to test a new model from OpenAI or Anthropic, the app creates a Model Serving endpoint through the SDK, configures AI Gateway settings (guardrails, rate limits, usage tracking, inference table logging), and makes the model available for assessment. All without touching databricks.yml or triggering a redeploy.

This tiered approach means the platform can keep pace with the market. When a new model drops, it can be under adversarial testing the same day.

The timing here is notable. The Databricks AI Gateway recently expanded from a serving-endpoint add-on into a broader enterprise control plane for governing LLM endpoints and coding agents. The new system.ai_gateway.usage system table provides 30+ columns of telemetry, and a built-in dashboard tracks request volume, latency percentiles, error rates, and per-endpoint usage breakdowns. For a red-teaming platform that routes dozens of models through Gateway endpoints, this level of visibility is a meaningful upgrade to the governance story.

Lakehouse Record Keeping: Delta Tables as the Source of Truth

Every interaction the platform produces flows into Delta tables governed by Unity Catalog. The data model spans 13 tables:

The test_cases table is the most critical. For every test, it captures: the vulnerability type, the attack method used, the exact adversarial prompt sent, the model's full response, the pass/fail verdict, the confidence score, and the evaluator's reasoning for its judgment. This is the raw evidence that backs every risk score.

Results are saved in batches (100 at a time) to handle assessments that generate thousands of test cases without exceeding job output limits. A checkpoint system ensures that if a long-running assessment hits an API rate limit, it can resume from where it left off rather than starting over.

“When someone in leadership asks why a model got a particular risk score, I can pull the exact prompt that broke it, the exact response it gave, and the exact reasoning the evaluator used to flag it. That level of traceability is what turns a red team exercise into an audit trail.”

MLflow Integration: Tracing, Scoring, and the Path to Convergence

While Delta tables serve as the authoritative record for reporting and analytics, MLflow provides a deeper operational view across three dimensions.

Tracing. The platform enables mlflow.openai.autolog() during job execution, which captures every OpenAI SDK call made by the simulator and evaluator. Each trace records the full sequence of API calls, their inputs and outputs, latency, and token consumption. Traces are logged to a shared experiment (/Shared/red-team-assessments) accessible to the team, and token usage data flows back into Delta for cost tracking.

Quality scoring. After every assessment, six independent quality scorers run against the results: three built-in scorers from MLflow (Generic Scorer, Guidelines) and three custom red team judges. These act as a second opinion on each attack, validating whether the Evaluator’s pass/fail verdicts hold up under independent review. The scores are surfaced directly in the results UI alongside the primary risk assessment.

Run tracking and artifact logging. Each assessment is tracked as an MLflow run with full metrics, parameters, and artifacts. This gives teams a historical record that goes beyond raw test cases, capturing how assessment configurations evolved over time and what patterns emerged across runs.

Today, the MLflow integration and Delta records serve different but complementary roles. Delta is the system of record for business reporting. MLflow is the operational layer for diagnostics, quality validation, and run history.

This gap is narrowing. Databricks recently launched the ability to store MLflow traces directly in Unity Catalog Delta tables (currently in Beta on select regions). Traces land in OpenTelemetry-format tables and become queryable via SQL. For this platform, that means the diagnostic view and the audit trail can eventually converge into a single governed layer.

Decoupled Execution: Jobs Handle the Heavy Lifting

The Databricks App itself is intentionally lightweight: a React frontend and a FastAPI backend that handle configuration, job submission, and result visualization. The actual adversarial simulations run on serverless Databricks Jobs.

When a user kicks off an assessment, the app writes the job configuration to Delta, submits a run via the Databricks SDK, and then polls for status updates. The assessment runner job (running on serverless compute) reads its configuration, initializes rate limiters for each provider, executes the Red Team Orchestration loop, and writes results back to Delta as it goes.

This separation matters for three reasons. First, assessment workloads are compute-intensive and long-running (2 minutes to 3+ hours depending on scope). Running them on the app’s own compute would block the UI for other users. Second, serverless job compute scales independently and has no idle cost. Third, the job’s execution is fully traceable in the Databricks Jobs UI, with logs, run history, and automatic alerting if something fails.

The rate limiting system deserves a note. The platform maintains per-provider rate limiters with configurable tiers (matching OpenAI’s tier structure), shared limiters when the simulator and evaluator use the same provider and model, and automatic retry with jitter and exponential backoff. If an assessment hits a rate limit wall, it saves a checkpoint and resumes after a cooldown period, up to five retry attempts before failing gracefully.

What’s Ahead

The platform is in production and actively testing models at Beusa, but the roadmap has several items in flight:

Lakebase integration. Lakebase reached General Availability on AWS in January 2026, with Azure currently in Beta and expected to follow soon. The Autoscaling version includes features that are a natural fit for this platform: scale-to-zero compute, instant branching via copy-on-write (useful for testing against different configurations), automated backups, and a PostgREST-compatible Data API. DABs already supports Lakebase resources (postgres_project, postgres_branch, postgres_endpoint), so the path from Delta-only to a hybrid operational data layer is clear once Azure GA lands. The plan is to migrate the application's transactional data (providers, models, app configuration) to Lakebase while keeping the analytical data (test cases, results, benchmarks) in Delta where it belongs.

MLflow trace convergence. With the UC trace storage Beta now live in select regions, the team plans to bridge the gap between MLflow traces and Delta records. The goal is a single queryable layer for traces, quality scores, and business reporting.

Marketplace distribution. The long-term vision is to package the platform and its reference architecture for distribution through the Databricks Marketplace, giving other organizations a head start on building their own governed AI red-teaming capabilities.

“This started as a solution to our own problem. But every conversation I have with other companies tells me the same thing: nobody has a great answer for AI red teaming yet. If we can make this reusable, it’s bigger than Beusa.”

The platform is currently in production at Beusa Energy, with plans to distribute through the Databricks Marketplace. If you’re interested in early access or want to discuss the reference architecture, reach out to Nick Fornicola or AJ Franchino on LinkedIn.

Inside The Red Team Orchestrator: The Architecture Behind Beusa Energy’s AI Security Hub was originally published in Databricks for Startups on Medium, where people are continuing the conversation by highlighting and responding to this story.

Every enterprise deploying LLMs faces the same unspoken question: Is this actually safe? At Beusa Energy, Nick Fornicola was asking it louder than most.

Beusa’s AI platform gives teams access to models from multiple providers, often on the day they are released. The business is excited. The engineers are ready. But the process for vetting new models before they hit production was anything but rigorous.

“Our ‘procedure’ for new models was vibes-based. Use the new LLM internally for a period of time, see if it behaves, and then release it to production once we felt comfortable. That worked until it didn’t. As Beusa began deploying AI-powered products at a faster clip, that approach became a liability.”

As Beusa began deploying AI-powered products at a faster clip, the gap between “this model seems fine” and “we can prove this model is safe” became impossible to ignore. Nick recognized the need for a structured, objective, and defensible way to test LLMs. Not a feeling. A platform.

The options were familiar: buy a third-party red-teaming tool (most of which require sending data outside the environment and months of procurement), or build internally. Building seemed like the obvious path until the enterprise realities set in.

“Writing code is easy. Deploying it in an enterprise is not. Cybersecurity reviews, infrastructure provisioning, application approvals. I needed a way to shortcut all of that without cutting corners on governance.”

That’s where the collaboration between Nick and Databricks Solutions Architect AJ Franchino took shape. The question wasn’t just what to build, but how to get it deployed without the months of friction that typically kill projects like this before they ever reach a user.

Bypassing the Three Bottlenecks That Kill AI Projects

In most organizations, the distance between a great idea and a deployed solution is filled with friction. To build a red-teaming platform at enterprise speed, the team had to navigate three specific bottlenecks that routinely stall internal tools for weeks or months.

1. The Infrastructure Approval Loop

In a traditional enterprise deployment, moving code to production involves ticket submissions, security reviews, infrastructure provisioning, and application approvals. That cycle usually runs two to three weeks at a minimum.

By deploying as a Databricks App, the red-teaming platform stayed within an already-vetted environment. No new servers. No new firewall rules. No new security silo to evaluate. Because Databricks Apps run natively within the existing workspace, Nick went from “Hello World” to a deployed, governed application in hours.

“I stayed within the guardrails that were already approved. That’s the difference between a project that ships and one that dies in a queue.”

2. The Procurement Bottleneck

Onboarding a new AI model provider is its own obstacle course. NDAs come easy, but MSAs and IP agreements pass back and forth between legal teams for weeks. IT needs to assess security posture. Data privacy and retention policies need verification. And critically, organizations need to ensure their internal data isn’t being used for model training.

To navigate this, the team adopted a hybrid model strategy. For many state-of-the-art and open source models, the team uses Databricks Model Serving natively. For others, they leverage an existing provider relationship that grants access to new models the day they land. The Databricks AI Gateway served as the connective tissue: by routing both native and external models through the Gateway, every model was accessible from a single, governed environment. Same billing. Same Unity Catalog permissions. Zero additional procurement friction.

3. The Developer Productivity Multiplier

As a director split between strategy and execution, Nick needed to move fast without sacrificing quality. He knows Python and FastAPI well, and has some experience with React, but he wanted a polished, professional application, not a quick prototype.

By pair-coding with Claude Code (available through Databricks’ coding agent integrations), Nick offloaded much of the backend Python work and focused on upskilling in React. He learned modern frontend tools like Shadcn/ui and TanStack in a fraction of the time it would have taken him to learn them on his own.

“Without AI-assisted coding, I would have likely reverted to a simpler framework like Streamlit just to save time, sacrificing the quality the project deserved. Claude Code let me build the app I actually wanted to build.”

Why This Pattern Matters

For AJ, the work with Nick at Beusa reflects a pattern playing out across the Databricks customer base: teams with real ideas and the technical chops to execute, but held back by the operational tax of deploying new tools in an enterprise.

The core insight behind Databricks Apps is simple: the application logic comes to the data, not the other way around. Instead of moving data out to where the application lives (which triggers security reviews, data governance concerns, and infrastructure approvals), the compute, the governance, and the data all stay in the same environment. For IT, security, and compliance teams, this changes the default answer. The question “Can we build this?” shifts from “not until we complete a multi-week review” to “yes, because it already runs inside the perimeter.”

The same Unity Catalog permissions that govern access to tables, models, and endpoints also govern the application itself. There is no separate security silo to stand up and no new vendor to onboard. This is what it looks like when a platform lets organizations say yes more often: reducing the distance between an idea and a governed, deployed solution to near zero, without relaxing any of the controls that enterprises require.

Turning “Shadow AI” Into Governed AI

Nick went from concept to a deployed application in under a week. But speed was only part of the win. By building natively on the Lakehouse, every adversarial interaction, every token count, and every risk score is persisted in Delta tables. App configuration, test cases, test results, and provider information are all retained and governed by Unity Catalog.

Permissions are managed through Databricks Asset Bundles (DABs), ensuring only the right team members can access the adversarial content that red-team assessments are designed to provoke. AI Gateway tracks usage across API calls, while MLflow provides full visibility into each trace.

“When leadership asks ‘Is this safe?’, I don’t point to a spreadsheet. I point to a permanent, queryable audit trail.”

What’s Next

Speed to value is only half the story. To make a security hub truly enterprise-grade, the architecture underneath has to be airtight: how to orchestrate multi-turn adversarial simulations across models, how to handle unfiltered responses safely during testing, and how to trace every failure back to its root cause.

In Part 2, we take a deep dive into the technical reference architecture, showing exactly how Unity Catalog, Databricks Apps, Jobs, AI Gateway, and MLflow Tracing come together to build a security hub that is as robust as it is fast.

From Vibes to Verdicts: How Beusa Energy Built an AI Security Hub on Databricks was originally published in Databricks for Startups on Medium, where people are continuing the conversation by highlighting and responding to this story.