<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Abhimanyu Gabhrani on Medium]]></title>
        <description><![CDATA[Stories by Abhimanyu Gabhrani on Medium]]></description>
        <link>https://medium.com/@undextered?source=rss-840464f32dd9------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*_AMpbus3O2tmDWRRALDwkw.jpeg</url>
            <title>Stories by Abhimanyu Gabhrani on Medium</title>
            <link>https://medium.com/@undextered?source=rss-840464f32dd9------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 16:52:18 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@undextered/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[My Databricks release-notes bot went quiet. Here’s the state-tracking fix.]]></title>
            <link>https://medium.com/@undextered/my-databricks-release-notes-bot-went-quiet-heres-the-state-tracking-fix-15b9f64a3011?source=rss-840464f32dd9------2</link>
            <guid isPermaLink="false">https://medium.com/p/15b9f64a3011</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[pyspark]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Abhimanyu Gabhrani]]></dc:creator>
            <pubDate>Tue, 14 Jul 2026 12:52:03 GMT</pubDate>
            <atom:updated>2026-07-14T12:52:03.224Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bZwsKPmkQgKIjIlcz0_TGQ.png" /></figure><p>The first version of this bot worked right up until it didn’t — and the way it failed was the worst kind. It didn’t error. It didn’t send me a red flow run. It just quietly stopped catching releases, and I only noticed because someone in a standup mentioned a feature I <em>should</em> have seen posted days earlier.</p><p>If you read my first write-up on this, you know the setup: a Power Automate flow watches the Databricks release notes RSS feed and posts new items to Teams. And you know I flagged a bug I hadn’t fixed yet — the flow decided what was “new” by comparing timestamps, and that comparison was silently dropping items. This is the fix I promised. The short version: <strong>stop asking “is this newer than last time?” and start asking “have I already seen this exact item?”</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/299/1*EsPt6PoHallj2S_I9RlYsg.png" /></figure><h3>Why the timestamp approach is a trap</h3><p>The original logic filtered feed items by date: grab everything published after the last run time. It sounds reasonable. It falls apart for a few reasons that are specific to RSS and specific to Power Automate.</p><p>The RSS connector doesn’t give you one clean, reliable “published” field. Depending on the feed you’ll see a publishDate, and separately something the feed author populates as the pub date — and they don&#39;t always agree, and one of them can be empty. I was keying off a field that wasn&#39;t consistently populated on the Databricks feed, so my &quot;published after X&quot; filter was comparing against nulls and quietly returning nothing.</p><p>Even when the date <em>is</em> populated, time-based dedup is fragile. Feeds re-publish or bump timestamps when an entry gets edited. Runs overlap or drift. Time zones sneak in. Any one of these means you either miss items or re-post them. And because the failure mode is “the filter returns an empty array,” there’s nothing to alert on. An empty result looks exactly like “no new releases today.”</p><p>That last point is the real lesson. <strong>A dedup strategy that fails silent is worse than one that fails loud.</strong> The whole job of this bot is to not miss things, and the v1 design missed things in the one way I couldn’t see.</p><h3>The fix: track what you’ve seen, not when you saw it</h3><p>Every RSS item has a stable identifier — its link, or a guid/id when the feed provides one. That identifier doesn&#39;t move when a timestamp gets bumped. So instead of trusting time, I keep a durable list of the identifiers I&#39;ve already posted, and each run I post only the items whose identifier isn&#39;t in that list.</p><p>That needs somewhere to store the list between runs. Power Automate flows are stateless — nothing survives from one run to the next unless you put it somewhere. My options were roughly: a SharePoint list, a Dataverse table, a real database, or an Excel file.</p><p>I went with Excel. Not because it’s elegant — it isn’t — but because it’s a single table in a file I already have, it needs zero new infrastructure or approvals, and the Excel connector’s “list rows” and “add a row” actions do exactly what I need. For a low-volume feed that fires a few times a day, an Excel table as a key-value store is the right amount of engineering. I’ll come back to where this choice bites.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hgdwfvjQa7Zm1Xe59DEPuA.png" /></figure><h3>Walking the flow</h3><p>Here’s each step and, more importantly, why it’s there.</p><p><strong>1. Recurrence</strong> — the schedule trigger. Nothing clever.</p><p><strong>2. List rows present in a table</strong> — read the Excel state table. This pulls back every identifier I’ve posted so far. On the very first run this table is empty (more on that below).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/622/1*DD5l8DFPM1lQ4xi593L-hQ.png" /></figure><p><strong>3. Select (the one I named </strong><strong>Selectseen)</strong> — this is the step people skip, and then their filter never matches. &quot;List rows&quot; gives you an array of <em>row objects</em>, not an array of strings. To compare cleanly I project it down to a flat array of just the identifier values.</p><p>The trick: in the Select action, switch the <strong>Map</strong> field to text mode (the little icon that toggles between key/value and a single text box). Map it to the identifier column only. The output is now [&quot;link-a&quot;, &quot;link-b&quot;, &quot;link-c&quot;] instead of [{ &quot;guid&quot;: &quot;link-a&quot; }, ...]. That flat shape is what makes the next step&#39;s contains() actually work.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/632/1*MWMonlXw78cQQlBB023dTA.png" /></figure><p><strong>4. List all RSS feed items</strong> — fetch the current feed. This returns everything the feed currently exposes, seen or not.</p><p><strong>5. Filter array</strong> — the core of the whole thing. Keep only the feed items whose identifier is <em>not</em> already in my seen list. In the Filter array advanced/expression mode:</p><pre>@not(contains(body(&#39;Selectseen&#39;), item()?[&#39;primaryLink&#39;]))</pre><p>Two things to get right here. body(&#39;Selectseen&#39;) is that flat string array from step 3. And item()?[&#39;primaryLink&#39;] is the identifier on the current feed item — check your own dynamic content for the exact token. Use the item link if that&#39;s what&#39;s stable on your feed; use the feed item&#39;s id/guid if it exposes one. Whatever you pick, it has to be the <em>same</em> field you&#39;re storing in Excel, and it has to match exactly — trailing spaces, casing, and URL encoding will all quietly break a contains check.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zW39Y5iTKRg9h11X4L51nA.png" /></figure><p><strong>6. Apply to each</strong> — loop the filtered items (the genuinely new ones) and, for each:</p><ul><li>Post to my personal test bot — I keep this so I can watch the bot behave before it’s shouting into a shared channel.</li><li>Post to the people who want a direct heads-up.</li><li>Post to the team channel.</li><li><strong>Add a row into the table</strong> — write this item’s identifier back to Excel so the next run treats it as seen.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*edgwoaITqqlMs4wOMUE0gQ.png" /></figure><h3>The ordering decision that ties back to v1</h3><p>Notice the “Add a row” is the <em>last</em> step in the loop — after the posts, not before. That’s deliberate, and it’s the direct lesson from v1.</p><p>If I recorded the identifier first and then posted, a failed post would leave the item marked “seen” forever. It would never be retried. That’s the exact silent-drop failure I was trying to kill, just with a new cause.</p><p>By posting first and recording last, I’ve chosen at-least-once delivery. The trade-off is real: if the posts succeed but the “add row” fails, next run re-posts that item as a duplicate. I’ll take a visible duplicate over an invisible miss every single time — a duplicate I can see and laugh off; a miss I find out about in a standup a week later.</p><h3>Where this design bites</h3><p><strong>First run posts everything.</strong> With an empty state table, nothing is “seen,” so the first run posts the entire current feed. Either accept one noisy run, or pre-seed the table with the current identifiers before you go live.</p><p><strong>The state table grows forever.</strong> Every new release adds a row and never removes one. For a release-notes feed that’s slow growth, but eventually “List rows” is dragging back a large table on every run. A cleanup that trims anything older than the feed’s own window is on my list.</p><p><strong>Excel is not a database.</strong> No real concurrency control. Keep the Apply to each loop sequential (concurrency off) so two iterations don’t write at once, and don’t point a high-frequency, high-volume flow at this pattern — it’ll outgrow Excel fast.</p><p><strong>Exact-match brittleness.</strong> The entire mechanism rests on the identifier matching byte-for-byte between what’s in the feed and what’s in Excel. Pick a stable field, store exactly what you compare, and don’t transform it in between.</p><h3>What I’d actually tell a colleague</h3><p>If you’re deduping a feed, a queue, or anything you process incrementally: don’t dedup on time. Dedup on identity, and persist the identities somewhere that survives a run. Time-based logic feels simpler and it is, right up until it silently does nothing and you have no way to tell. Store the IDs you’ve handled, compare against that, and record <em>after</em> the work succeeds — so the failure mode is a duplicate you can see, not a gap you can’t.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=15b9f64a3011" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I Kept Missing Databricks Releases. Here’s the Power Automate → Teams Flow I Built to Fix It (v1)]]></title>
            <link>https://medium.com/@undextered/i-kept-missing-databricks-releases-heres-the-power-automate-teams-flow-i-built-to-fix-it-v1-f2a48b3b7dab?source=rss-840464f32dd9------2</link>
            <guid isPermaLink="false">https://medium.com/p/f2a48b3b7dab</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[pyspark]]></category>
            <category><![CDATA[sql]]></category>
            <dc:creator><![CDATA[Abhimanyu Gabhrani]]></dc:creator>
            <pubDate>Fri, 03 Jul 2026 13:25:53 GMT</pubDate>
            <atom:updated>2026-07-03T13:25:53.545Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j8OK90dXttTbuI-EYZ1Nzg.png" /></figure><p>A few months ago I spent the better part of an afternoon building a workaround for something in Unity Catalog, only to have a teammate mention — almost in passing — that the exact thing I needed had shipped in a release two weeks earlier. I just hadn’t seen it.</p><p>That’s the problem with running Databricks day to day. Release notes land constantly across the platform, the Runtime, SQL, and pipelines, and unless checking the docs is literally someone’s job, you find out about half of it late — usually from a LinkedIn post or a colleague.</p><p>If you also struggle to keep up with Databricks’ latest updates and key innovations, or if you’re like me and just want to stay updated (and, let’s be honest, flex that knowledge in the next standup) — this is a small thing you can set up in an afternoon that pays for itself immediately.</p><p>The idea is simple: Databricks publishes an RSS feed of its release notes. Power Automate can read that feed and drop each new item straight into a Teams channel. No infra, no cost beyond a standard Power Automate license, no code.</p><p>This is <strong>v1</strong>. It works, and it’s the version I’d tell you to start with. It also has one real bug that I’ll be honest about at the end — because I hit it, tested a fix, and v2 is coming.</p><h3>What we’re building</h3><p>One flow, two blocks:</p><ul><li><strong>Trigger:</strong> an item gets published to the Databricks release notes RSS feed.</li><li><strong>Action:</strong> post that item as a message in a Teams channel.</li></ul><p>That’s the whole thing.</p><h3>The feed</h3><p>Databricks exposes a documentation RSS feed that covers release notes across the platform, Runtime, SQL, and more. If you’re on Azure Databricks — which I am — the feed is:</p><pre>https://learn.microsoft.com/en-us/azure/databricks/feed.xml</pre><p>On AWS or GCP, the docs site has the equivalent feed. Open any release notes page and look for the RSS icon in the top-right corner — that links straight to the feed URL for your cloud.</p><p>Paste the URL into a browser first and confirm you get XML back before you wire anything up. If the feed doesn’t load, nothing downstream will work, and you’ll waste twenty minutes debugging the wrong layer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/927/1*4_SH5K0XlXfnQmwpKhm77w.png" /></figure><h3>Step 1 — Create the flow</h3><p>In Power Automate, create a new <strong>Automated cloud flow</strong>. When it asks you to pick a trigger, search for RSS and choose <strong>&quot;When a feed item is published.&quot;</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TQ6lhyq2EJDAnTQ-HQlDHA.png" /></figure><h3>Step 2 — Configure the trigger</h3><p>Two fields matter here:</p><ul><li><strong>The RSS feed URL</strong> — paste the feed from above.</li><li><strong>“Chosen property will be used to determine which items are new”</strong> — leave this on PublishDate for now. Remember this field. It&#39;s the entire reason v2 exists.</li></ul><h3>Step 3 — Post to Teams</h3><p>Add a new action, search for Teams, and choose <strong>&quot;Post message in a chat or channel.&quot;</strong></p><p>Configure it:</p><ul><li><strong>Post as:</strong> Flow bot (or yourself, if you’d rather it read as personal).</li><li><strong>Post in:</strong> Group Chat.</li><li><strong>Group Chat:</strong> pick wherever your team actually looks. Mine posts to a dedicated Internal Team channel so it doesn&#39;t drown in general chat.</li></ul><p>For the <strong>Message</strong>, build the body from the feed item’s dynamic content. Here’s the format I use:</p><pre>&lt;p&gt;&lt;strong&gt;🚀 &lt;a href=&quot;@{triggerOutputs()?[&#39;body/primaryLink&#39;]}&quot;&gt;@{triggerOutputs()?[&#39;body/title&#39;]}&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;<br>&lt;p&gt;📅 &lt;em&gt;@{triggerOutputs()?[&#39;body/publishDate&#39;]}&lt;/em&gt;&lt;/p&gt;<br>&lt;blockquote&gt;@{triggerOutputs()?[&#39;body/summary&#39;]}&lt;/blockquote&gt;<br>&lt;p&gt;&lt;a href=&quot;@{triggerOutputs()?[&#39;body/primaryLink&#39;]}&quot;&gt;&lt;strong&gt;Read the full release note →&lt;/strong&gt;&lt;/a&gt;&lt;/p&gt;</pre><p>Those tokens (Feed title, Primary feed link, and so on) come from the trigger output, so Power Automate fills them in per item. The exact labels vary slightly by connector version, but they&#39;re all there in the dynamic content picker under the RSS trigger.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HF6mAS8A1ivmb8VKQMVsfA.png" /></figure><h3>Step 4 — Test it</h3><p>Save, then either run the flow manually or wait for the next release to land. When a new item comes through, you’ll get something like this in your channel:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zhmLokBD0AYV17A4KciyVQ.png" /></figure><p>That’s v1. Genuinely useful, and most days it just works.</p><h3>The gotcha (and why there’s a v2)</h3><p>Here’s the honest part.</p><p>That “When a feed item is published” trigger decides what’s <em>new</em> by comparing the PublishDate of each feed item against the latest date it recorded on the previous run. Anything newer than that high-water mark gets posted. Anything that isn&#39;t, doesn&#39;t.</p><p>That assumption breaks on the Databricks feed. I’ve watched release notes show up with timestamps that aren’t newer than what the trigger already had — same-day batches, entries that look backfilled, revisions that don’t bump the date cleanly. When that happens, the trigger looks at the item, decides it isn’t “new,” and silently skips it.</p><p>So every so often, a release note just… never shows up in Teams. No error. No failed run. The flow is green. You only notice when you trip over a feature that <em>should</em> have been posted and wasn’t.</p><p>For a lot of people that’s a fine trade — you catch most releases for an afternoon of setup. But if you’re treating this as your actual source of truth (I am), missing the occasional release <em>quietly</em> is exactly the failure mode you can’t afford. A missed alert you know about is a bug; a missed alert you don’t know about is a trap.</p><p>I’ve tested a v2 that doesn’t lean on PublishDate to decide what&#39;s new, so it stops dropping items. I&#39;ll write that one up next — stay tuned.</p><h3>The takeaway</h3><p>If you want to stop finding out about Databricks features two weeks late, v1 gets you most of the way there fast: RSS trigger, Teams action, done. Just go in knowing the PublishDate logic will occasionally eat a release, and don&#39;t treat it as complete coverage until v2 lands.</p><p>Set it up, point it at a channel your team actually reads, and enjoy being the person who already knew.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f2a48b3b7dab" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I audited Databricks AI Functions for PII. A dozen were fine. One wasn’t.]]></title>
            <link>https://medium.com/@undextered/i-audited-databricks-ai-functions-for-pii-a-dozen-were-fine-one-wasnt-b2e55599abdd?source=rss-840464f32dd9------2</link>
            <guid isPermaLink="false">https://medium.com/p/b2e55599abdd</guid>
            <category><![CDATA[unity-catalog]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[compliance]]></category>
            <dc:creator><![CDATA[Abhimanyu Gabhrani]]></dc:creator>
            <pubDate>Mon, 22 Jun 2026 10:51:14 GMT</pubDate>
            <atom:updated>2026-06-22T10:51:14.529Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b4j-l3s-meocmq-EAnSK0g.png" /></figure><p>There was already a narrow sign-off on the books: ai_classify had been cleared to bucket a text column that might contain customer PII into a few labels. Databricks-hosted model, data stays in the tenant, fine.</p><p>Then security came back with a bigger ask. Teams across my pods were reaching for the rest of the ai_* family — ai_mask, ai_extract, ai_summarize, ai_query — on columns that absolutely held names, emails, and phone numbers. Could we say about all of them what we&#39;d already said about ai_classify? They wanted one answer for the whole family, not a ticket per function, and they wanted it backed by documentation, not vibes.</p><p>So I went and read the docs properly — not the marketing page, the data-protection and model-serving pages — and traced where the bytes actually go for each function. The short version: almost all of them collapse to a single, defensible answer. Exactly one doesn’t, and it’s the one everyone reaches for first.</p><h3>The part that’s boring in a good way</h3><p>The ai_* functions split into two groups, and the split is the whole story.</p><p>The first group is the <strong>task-specific</strong> functions: ai_classify, ai_mask, ai_extract, ai_parse_document, ai_analyze_sentiment, ai_summarize, ai_translate, ai_fix_grammar, ai_gen, ai_similarity, ai_forecast, vector_search, and the beta ai_prep_search. Each one is single-purpose, and — this is the load-bearing fact — each one runs on a model that Databricks hosts. You don&#39;t pick the model. Foundation Model APIs or a Databricks-managed system does the work, inside the Databricks tenant.</p><pre>-- Both of these run on Databricks-hosted models.<br>-- The text goes to a Databricks endpoint and the result comes back.<br>-- Nothing leaves the tenant.</pre><pre>SELECT ai_classify(notes, ARRAY(&#39;billing&#39;, &#39;technical&#39;, &#39;complaint&#39;)) AS category<br>FROM support_tickets;<br>SELECT ai_mask(notes, ARRAY(&#39;person&#39;, &#39;email&#39;, &#39;phone&#39;)) AS masked_notes<br>FROM support_tickets;</pre><p>Because all of them sit behind the same hosting model, they’re governed by one data-protection policy, and you only have to evaluate that policy once. Three things in it actually matter for a PII review:</p><p><strong>No training on your data.</strong> For paid accounts, Model Serving doesn’t use inputs or outputs to train any model or improve any Databricks service. This is the question every security reviewer asks first, and it’s a clean no.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*3Bh8lgoL9o2T4ktMYF499A.png" /><figcaption><a href="https://docs.databricks.com/aws/en/machine-learning/model-serving/">Databricks Documentation — Deploy Models using Model Serving</a></figcaption></figure><p><strong>Retention is “up to 30 days,” not zero.</strong> This is the bit I’d flag to anyone who skims. Foundation Model APIs may temporarily store inputs and outputs to detect and mitigate abuse. That storage is isolated from other customers, stays in your workspace’s region, and is only accessible for security/abuse purposes — but it exists, for up to thirty days. If you’ve been telling people “Databricks-hosted means nothing is stored,” you’re wrong, and the gap matters for a strict PII classification. It’s not zero-retention. It’s short, isolated, abuse-only retention.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*jcfQgqWWrwZtAAvaoVyO3w.png" /><figcaption><a href="https://docs.databricks.com/aws/en/machine-learning/model-serving/">Databricks Documentation — Deploy Models using Model Serving</a></figcaption></figure><p><strong>Residency stays in your Geo.</strong> Foundation Model APIs is a Databricks Designated Service, so it honors your workspace’s data-residency Geo. The catch: Databricks may process data outside the originating <em>region</em> while staying inside the Geo, and if your workspace is in a Model Serving region that isn’t US or EU, you have to explicitly enable cross-Geo processing. Worth confirming your workspace’s Geo and that cross-Geo isn’t switched on when you didn’t expect it. Traffic is TLS in transit either way.</p><p>That’s the entire risk picture for twelve-plus functions. One policy, three facts. If your PII tolerance accepts short abuse-only retention inside your region with no training, every task-specific function is in scope. That’s a genuinely nice place to land — you can give the pod one answer instead of thirteen.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*0Vgs2YOPnGQ2hAKdQmcnGg.png" /><figcaption><a href="https://docs.databricks.com/aws/en/machine-learning/model-serving/">Databricks Documentation — Deploy Models using Model Serving</a></figcaption></figure><h3>The exception</h3><p>ai_query is not like the others, and the difference is easy to miss because the SQL looks almost identical.</p><p>ai_query is the general-purpose one. You write the prompt, and <em>you</em> choose the endpoint. That single degree of freedom is what breaks the blanket answer, because its data handling isn&#39;t a property of the function — it&#39;s a property of whatever endpoint you point it at. ai_query supports four kinds:</p><ul><li><strong>Databricks-hosted foundation model endpoints</strong> (the ones prefixed databricks-). Same policy as everything above. In-tenant, no training, 30-day abuse-only retention, in-Geo.</li><li><strong>Provisioned throughput endpoints</strong> — dedicated/fine-tuned models, still on Databricks Model Serving, still in-tenant.</li><li><strong>Custom model serving endpoints</strong> — your own registered models, served on Databricks, still in-tenant.</li><li><strong>External model endpoints</strong> — and here’s the trapdoor.</li></ul><p>An external endpoint routes the call to a model hosted <em>outside</em> Databricks: OpenAI, Anthropic, Google, Azure OpenAI. When ai_query targets one of these, your prompt and the column values in it leave the Databricks tenant and land with a third-party provider, governed by <em>that provider&#39;s</em> terms — their retention, their residency, their safety handling. That&#39;s not a Databricks data-protection question anymore. It&#39;s third-party data sharing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1012/1*RyLV7Zeyj9YQL9SbDuWhAw.png" /></figure><p>Now look at the two calls side by side:</p><pre>-- Stays in the tenant. Governed by the Section-4 policy.<br>SELECT ai_query(<br>  &#39;databricks-meta-llama-3-3-70b-instruct&#39;,<br>  &#39;Summarize this ticket: &#39; || notes<br>) AS summary<br>FROM support_tickets;</pre><pre>-- Looks the same. Ships every value in `notes` to OpenAI.<br>SELECT ai_query(<br>  &#39;my-openai-endpoint&#39;,<br>  &#39;Summarize this ticket: &#39; || notes<br>) AS summary<br>FROM support_tickets;</pre><p>From the call site you cannot tell which one is a data-egress event. The only difference is the endpoint name, and a string in a SQL editor doesn’t announce that it’s about to send customer PII to a vendor.</p><p>The decision actually happened earlier — at endpoint creation:</p><pre>{<br>  &quot;name&quot;: &quot;my-openai-endpoint&quot;,<br>  &quot;config&quot;: {<br>    &quot;served_entities&quot;: [{<br>      &quot;external_model&quot;: {<br>        &quot;provider&quot;: &quot;openai&quot;,<br>        &quot;name&quot;: &quot;gpt-4&quot;,<br>        &quot;task&quot;: &quot;llm/v1/chat&quot;,<br>        &quot;openai_config&quot;: { &quot;openai_api_key&quot;: &quot;{{secrets/my_scope/openai_key}}&quot; }<br>      }<br>    }]<br>  }<br>}</pre><p>That external_model block is the moment your data boundary moves. Which is exactly why the control belongs <em>there</em>, not in code review of every ai_query call.</p><blockquote><em>One note so I don’t overclaim: Databricks’ built-in assistive features (Assistant, Genie) do use zero-data-retention endpoints from partners. That’s a different thing from an external endpoint you create yourself and call from </em><em>ai_query — for that one, the provider&#39;s standard terms apply unless you&#39;ve arranged otherwise. Don&#39;t let the assistive-feature ZDR language lull you into assuming your own external endpoint inherits it.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/598/1*_D38cntS9xfWPnWUrLG_Qg.png" /></figure><h3>What I’d actually tell a teammate</h3><p>ai_query&#39;s safety isn&#39;t a property of the function. It&#39;s a property of the endpoint. So don&#39;t read the SQL — read the endpoint.</p><p>Concretely, the gate that matters is <strong>who can create an external model serving endpoint</strong>. Lock that down with Unity Catalog privileges and treat every new external endpoint as a governance decision, not a config change. Once external endpoints can’t appear without approval, every ai_query in the codebase is safe by construction, because the only endpoints available are in-tenant. Route through Unity AI Gateway on top of that if you want usage tracking and guardrails.</p><p>And the line I keep repeating to my pods: the twelve task-specific functions give you one answer you can memorize. ai_query gives you a question you have to ask every time — <em>where does this endpoint send the data?</em> Both are fine to use. Only one of them is fine to stop thinking about.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b2e55599abdd" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>