<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Josh Rosenberg on Medium]]></title>
        <description><![CDATA[Stories by Josh Rosenberg on Medium]]></description>
        <link>https://medium.com/@rosenberg.josh34?source=rss-512ea28a2b18------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Vx5CrtwxuOuC_EL9AxiIAQ.png</url>
            <title>Stories by Josh Rosenberg on Medium</title>
            <link>https://medium.com/@rosenberg.josh34?source=rss-512ea28a2b18------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 17:13:35 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@rosenberg.josh34/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to Secure and Audit AI Agents That Access Enterprise Data]]></title>
            <link>https://medium.com/@rosenberg.josh34/how-to-secure-and-audit-ai-agents-that-access-enterprise-data-87e208eb80e6?source=rss-512ea28a2b18------2</link>
            <guid isPermaLink="false">https://medium.com/p/87e208eb80e6</guid>
            <category><![CDATA[databricks-unity-catalog]]></category>
            <category><![CDATA[databricks-genie]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Josh Rosenberg]]></dc:creator>
            <pubDate>Mon, 19 Oct 2026 16:17:44 GMT</pubDate>
            <atom:updated>2026-10-19T16:18:48.115Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>A field guide for securing Databricks Genie, Genie One and MCP-connected assistants, with the SQL to audit them.</em></h4><p><strong>Key points</strong></p><ul><li>Put permissions in Unity Catalog once and every AI agent that queries the data inherits them, so you’re not rebuilding access rules inside each tool.</li><li>Databricks Genie One and Genie Agents query data as the person asking. Row filters, column masks and attribute-based access control (ABAC) apply to Genie exactly as they apply to SQL.</li><li>Every Genie interaction lands in the system.access.audit system table, so you can answer &quot;who asked what, and what did it touch&quot; with a query.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5BQw-8KQdbRRSfephEK-nA@2x.png" /></figure><h3>How should you secure AI agents that access enterprise data?</h3><p>Govern the data layer instead of each agent. If every AI tool keeps its own copy of your access rules, you’ll have five versions of the policy within a year and none of them will match. Keep the rules where the data lives, and every agent, dashboard and SQL query picks them up automatically.</p><p>The first question security teams ask me in a Genie evaluation is almost never about the model. It’s usually something like, “So if a sales rep asks Genie about compensation, what happens?” Fair question. An AI coworker needs to say “you don’t have access to that” as reliably as a database does, and with Genie, a database is literally what’s answering underneath.</p><p>Here’s how that works in Databricks, and how you’d prove it to an auditor.</p><h3>What are Genie One, Genie Agents and Unity Catalog?</h3><ul><li><a href="https://docs.databricks.com/aws/en/genie-one/"><strong>Databricks Genie One</strong> </a>is Databricks’ AI coworker for business users. It answers questions in natural language by generating SQL against governed data, and it’s available on the web, in mobile apps and through MCP.</li><li><a href="https://docs.databricks.com/aws/en/genie"><strong>Genie Agents</strong></a> (fka Genie spaces) are curated, domain-specific setups where a data team defines the tables, metrics, instructions and example queries for one area of the business, like finance or sales.</li><li><a href="https://www.databricks.com/product/unity-catalog"><strong>Unity Catalog</strong></a> is the Databricks governance layer. It holds the permissions, row filters, column masks, tags and audit logs for every data asset.</li></ul><p>So Genie writes the SQL, and Unity Catalog decides what that SQL gets to see.</p><h3>How does Databricks Genie enforce data permissions?</h3><p>Databricks Genie enforces permissions through Unity Catalog, using the identity of the person asking the question. To get an answer from a Genie Agent, a user needs two things: access to the Genie Agent itself, and SELECT on the underlying tables.</p><p>Here’s the part that confuses people - every Genie Agent runs on a SQL warehouse, and the agent creator’s compute credentials are embedded in it and used for everyone’s queries. That sounds alarming until you realize it only covers compute. Row filters and column masks are still evaluated as the person asking, so a shared warehouse doesn’t mean shared data.</p><p>Something else that surprises people: Genie can query tables beyond the ones you explicitly added to an agent, because Unity Catalog controls access rather than the Genie Agent. Picking tables for an agent helps accuracy, but don’t treat it as a security boundary. Your Unity Catalog grants are the boundary.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8QPU6QJ7bWMnuIGvC-kn5g.png" /></figure><p>What does row- and column-level security look like with Genie?</p><p>Here’s a small example on sample travel data, similar to the samples.wanderbricks dataset. Copy a bookings table into a schema you own first, since you can&#39;t change the samples catalog.</p><pre>-- Only let users see bookings in regions they belong to<br>CREATE FUNCTION main.demo.region_filter(region STRING)<br>RETURN is_account_group_member(&#39;admins&#39;)<br>    OR is_account_group_member(concat(&#39;region_&#39;, lower(region)));<br><br>ALTER TABLE main.demo.bookings<br>  SET ROW FILTER main.demo.region_filter ON (region);<br>-- Mask guest emails for everyone outside support<br>CREATE FUNCTION main.demo.mask_email(email STRING)<br>RETURN CASE WHEN is_account_group_member(&#39;support&#39;) THEN email<br>            ELSE &#39;***@***&#39; END;<br>ALTER TABLE main.demo.bookings<br>  ALTER COLUMN guest_email SET MASK main.demo.mask_email;</pre><p>Now ask a Genie Agent built on that table, “Show me last month’s bookings with guest emails.” A user in region_west sees only western bookings, with masked emails. A support user sees the emails. Nobody changed anything inside Genie.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v9zkMFkcydS_D_ofZRmNJA.png" /><figcaption>Query results before column masking is applied</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kZH89xwNNHV0IP70gDl30w.png" /><figcaption>Query results after column masking is applied</figcaption></figure><p>For more than a handful of tables, Databricks recommends ABAC policies over per-table filters. ABAC applies row filters and column masks automatically to every object carrying a governed tag, such as pii, across whole catalogs and schemas. That&#39;s how you avoid hand-wiring masks on 400 tables.</p><h3>What happens when Claude, ChatGPT or Cursor connect to Genie through MCP?</h3><p>The Model Context Protocol (MCP) lets external AI assistants call tools in other systems. Databricks offers managed MCP servers, including a Genie One MCP server at https://&lt;workspace-hostname&gt;/api/2.0/mcp/genie and a per-agent server for a single Genie Agent.</p><p>The security model doesn’t change when the request comes from outside Databricks. Requests must be authenticated to your workspace, and Unity Catalog permissions are enforced on every request. Someone asking Claude Desktop a question through the Genie One MCP server sees exactly what they would see in Genie One itself.</p><p>Honestly, this is the part I like most. Without it, every new assistant would need its own credentials, its own copy of the data and its own access rules. With it, adding Claude or Cursor is just one more connection, and your governance team doesn’t have to start over.</p><p>A couple of things to watch:</p><ul><li><strong>Prefer on-behalf-of-user authentication.</strong> It keeps per-user permissions and per-user audit trails intact. Use a service principal only for automated agents, give it the minimum grants it needs (CAN RUN on the Genie Agent, SELECT on specific tables, CAN USE on the warehouse), and use a dedicated service principal per app.</li><li><strong>Check the release status.</strong> As of this writing, the Genie One MCP server requires the Managed MCP Servers workspace preview to be turned on.</li></ul><h3>How do you audit what an AI agent asked and accessed?</h3><p>You audit Databricks Genie the same way you audit everything else in Databricks: with system tables. Genie activity is recorded in system.access.audit under the service name aibiGenie. The SQL that Genie generates and runs shows up in system.query.history, attributed to the user who asked.</p><p>Between the two, an auditor gets the whole story: the audit log shows who talked to which Genie Agent and when, and query history shows exactly what SQL ran against which data.</p><p>These are the three queries I’d set up first. Test them on a demo workspace before relying on them, since audit event names can shift as features evolve.</p><p><strong>1. Who is using Genie, and how?</strong></p><pre>SELECT<br>  user_identity.email AS user_email,<br>  action_name,<br>  COUNT(*) AS events,<br>  MAX(event_time) AS last_seen<br>FROM system.access.audit<br>WHERE service_name = &#39;aibiGenie&#39;<br>  AND event_date &gt;= current_date() - INTERVAL 30 DAYS<br>GROUP BY ALL<br>ORDER BY events DESC;</pre><p><strong>2. Which Genie requests failed?</strong></p><p>A spike in failed requests from one user can mean someone is probing for data they can’t see, or simply that an agent is misconfigured. Either way, you want to know.</p><pre>SELECT<br>  event_time,<br>  user_identity.email AS user_email,<br>  action_name,<br>  response.status_code,<br>  response.error_message<br>FROM system.access.audit<br>WHERE service_name = &#39;aibiGenie&#39;<br>  AND response.status_code &lt;&gt; 200<br>  AND event_date &gt;= current_date() - INTERVAL 7 DAYS<br>ORDER BY event_time DESC;</pre><p><strong>3. What SQL did Genie actually run?</strong></p><pre>SELECT<br>  start_time,<br>  executed_by,<br>  query_source.genie_space_id AS genie_agent_id,<br>  execution_status,<br>  statement_text<br>FROM system.query.history<br>WHERE query_source.genie_space_id IS NOT NULL<br>  AND start_time &gt;= current_date() - INTERVAL 7 DAYS<br>ORDER BY start_time DESC;</pre><p>Query 3 is usually what wins over a skeptical security team. They don’t have to trust that the AI probably stayed in bounds. They can read every statement it ran, user by user.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DvS-pygrqo6dxniUeV-5uA.png" /><figcaption>Sample output of query 3</figcaption></figure><p>To move from auditing after the fact to catching issues as they happen, schedule queries 2 and 3 as Databricks SQL alerts. For example, alert when one user generates more than 20 failed Genie requests in an hour, or when Genie queries touch a table tagged restricted. You can also put the results on an AI/BI dashboard so the governance team has one place to look.</p><h3>What should a security review of an AI agent cover?</h3><p>Use this checklist before rolling Databricks Genie One or Genie Agents out to business users.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/710/1*hJxVE6-_FYZlV42A8yVsUQ.png" /><figcaption>What to check for, and what good looks like</figcaption></figure><p>Most teams skip that last row, and I’d push hard on it. Before launch, sign in as a low-access test user and ask the questions that should fail: salaries, another region’s revenue, customer emails. If nothing comes back that shouldn’t, you’ve got real evidence for the security review instead of a slide that just says “governed.”</p><h3>Frequently asked questions</h3><p><strong>Can Databricks Genie see data a user doesn’t have access to?</strong> No. Databricks Genie generates SQL, and Unity Catalog evaluates that SQL against the asking user’s permissions, row filters and column masks. If the user can’t query the data directly, Genie can’t return it to them.</p><p><strong>Is adding tables to a Genie Agent a security control?</strong> No. Choosing tables for a Genie Agent improves answer quality, but Unity Catalog permissions are the actual access boundary.</p><p><strong>Are AI assistants connected through MCP governed the same way?</strong> Yes. The Databricks managed MCP servers, including the Genie One MCP server, require authentication to your workspace and enforce Unity Catalog permissions on every request.</p><p><strong>How do I see what SQL an AI agent ran on my data?</strong> Query system.query.history and filter on Genie as the query source. Each statement is attributed to the user who asked the question.</p><p><strong>Do I need a separate governance tool for AI agents?</strong> Not for data access on Databricks. If agents reach your data through Unity Catalog, your existing grants, policies and audit logs already cover them.</p><h3>Sources</h3><ul><li><a href="https://docs.databricks.com/aws/en/genie-agents/set-up">Databricks docs: Set up and manage a Genie agent</a></li><li><a href="https://docs.databricks.com/aws/en/agents/mcp/genie-mcp">Databricks docs: Genie One MCP server</a></li><li><a href="https://docs.databricks.com/aws/en/ai-bi/admin/audit">Databricks docs: Monitor Genie usage with audit logs and alerts</a></li></ul><p><em>Related: my earlier guide on </em><a href="https://medium.com/dbsql-sme-engineering/embedding-genie-api-for-a-multi-tenant-application-d307bfbfc89b"><em>embedding the Genie API in a multi-tenant application with ABAC</em></a><em> goes deeper on per-tenant access control.</em></p><p><em>For a playbook on how to roll out Genie One for your enterprise, check out my other blog here: </em><a href="https://www.databricks.com/blog/how-roll-out-genie-one-step-step-enterprise-playbook"><em>How to roll out Genie One: A step-by-step enterprise playbook</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=87e208eb80e6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Who’s Actually Using Your Databricks Dashboards (and What Do They Cost)?]]></title>
            <link>https://medium.com/dbsql-sme-engineering/whos-actually-using-your-databricks-dashboards-and-what-do-they-cost-80bc0224b827?source=rss-512ea28a2b18------2</link>
            <guid isPermaLink="false">https://medium.com/p/80bc0224b827</guid>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[business-intelligence]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[dashboard]]></category>
            <dc:creator><![CDATA[Josh Rosenberg]]></dc:creator>
            <pubDate>Fri, 17 Jul 2026 18:06:46 GMT</pubDate>
            <atom:updated>2026-07-21T16:03:22.552Z</atom:updated>
            <content:encoded><![CDATA[<h4>A free, importable AI/BI dashboard that turns Databricks system tables into usage, performance, and cost analytics for every dashboard in your account</h4><p><strong>By Josh Rosenberg (Sr. Solutions Engineer @ Databricks) &amp; Cassi Sullivan (Sr. Solutions Engineer @ Databricks)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oXSqf3OIXBFdMz--T5Sv2g.png" /><figcaption>Usage &amp; Cost Dashboard</figcaption></figure><p>We spend a lot of energy building dashboards. We agonize over the right chart type, argue about color palettes, and rewrite the same aggregation four different ways to shave 200 milliseconds off a load. Then we publish, share the link, and move on.</p><p>And then… we mostly stop paying attention.</p><p>Six months later, nobody can answer the simple questions. Which dashboards do people actually open? Who are the power users? Which one has quietly become the most expensive object in the workspace because it re-scans a billion rows every time someone breathes near it? Are we paying to keep three near-identical “exec overview” dashboards alive when only one gets traffic?</p><p>Those aren’t dashboard-building questions. They’re dashboard <em>operations</em> questions, and Databricks already collects everything you need to answer them. It’s sitting in the system tables. You just have to assemble it.</p><p>So we built an <a href="https://www.databricks.com/product/business-intelligence">AI/BI</a> dashboard that does exactly that, and we’re sharing it so you can import it into your own account in a couple of minutes. This post walks through what it shows, how it works under the hood, how to set it up, and — importantly — the caveats you should understand before you present any of these numbers to a VP.</p><h3>What it actually shows</h3><p>The dashboard is organized into two analytical pages plus a shared filter panel, all driven by a single date range and a set of cross-cutting filters (viewer, dashboard ID, warehouse).</p><p><strong>The Overview page</strong> answers the adoption question. At a glance you get total dashboard views, unique viewers, the count of dashboards that saw any activity, and total queries fired. Below the headline counters, it breaks down daily query volume over time, ranks your most-used dashboards, surfaces your top viewers, and — the one that tends to start conversations — estimated cost by dashboard. There’s also a detailed query-level table for when you need to drill into specifics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a_og_FbjASct5th7QU3FRA.png" /><figcaption>Overview tab</figcaption></figure><p><strong>The Performance &amp; Attributed Costs page</strong> answers the efficiency question. Average query duration, cache hit rate, and total DBUs consumed sit up top. From there it trends cost and query duration over time, plots cache hits versus misses (a fast, honest read on how much your dashboards benefit from result caching), and slices cost and DBU consumption by user, by SKU, and by warehouse. The by-warehouse breakouts are useful for spotting a dashboard that’s pointed at an oversized or poorly-sized warehouse.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HlW5p3GOAcAa2aBBIEZ64Q.png" /><figcaption>Performance &amp; Attributed Costs Tab</figcaption></figure><p><strong>The Global Filters pane</strong> is the control panel. Everything is parameterized, so you can zoom from “the whole account, last 90 days” down to “this one dashboard, this one warehouse, last week” without editing a single query.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iiU7gmoP10I8x5kB__TG-Q.png" /><figcaption>Global FIlters Pane</figcaption></figure><p>The design goal was that someone should be able to open it cold and, within about thirty seconds, know whether their dashboard estate is healthy, loved, and reasonably priced.</p><h3>How it works under the hood</h3><p>The whole thing is built on <a href="https://www.databricks.com/product/unity-catalog">Unity Catalog</a> <a href="https://docs.databricks.com/aws/en/admin/system-tables/">system tables </a>— the read-only, account-wide operational logs Databricks maintains in the system catalog. No pipelines, no jobs, no copies of data. The dashboard queries the source directly. Four datasets do the work:</p><ul><li><strong>Views come from the audit log.</strong> The system.access.audit table records dashboard interactions under service_name = ‘dashboards’. We filter to the getDashboard, getPublishedDashboard, and executeDashboardWidget actions and pull the viewer’s email and the dashboard ID out of the request parameters. A left join to system.access.workspaces_latest swaps opaque workspace IDs for human-readable names.</li><li><strong>Performance comes from query history.</strong> system.query.history carries a query_source struct, and for any statement triggered by an AI/BI dashboard, query_source.dashboard_id is populated. That single field is the key that lets you attribute duration, rows read, cache behavior, and execution status back to a specific dashboard.</li><li><strong>Cost is the interesting part.</strong> Query history tells you <em>which</em> statements a dashboard ran; system.billing.attributed_usage tells you the DBUs each of those statements consumed, joined on the statement ID (usage_metadata.dbsql_statement_id = statement_id). To turn DBUs into dollars, we join system.billing.account_prices for any negotiated contract pricing on the SKU — if no contract price exists (as is the case for most accounts), we fall back to system.billing.list_prices, which always carries the public list rate. The result is an estimated cost per query that rolls up cleanly to per-dashboard, per-user, and per-warehouse totals. Note that attributed_usage lags by approximately 2 days, so cost figures for the most recent days may be incomplete until billing attribution settles.</li><li><strong>Daily activity is a pre-aggregation.</strong> A fourth dataset groups query history by day, dashboard, and warehouse to produce the trend lines efficiently — query counts, unique users, average duration, bytes read, and a cache-hit/miss split.</li></ul><p>That’s the entire architecture. Because it reads live from system tables, there’s nothing to schedule and nothing to maintain. When Databricks writes new audit and billing records, your dashboard reflects them on the next refresh.</p><h3>Getting it running</h3><p>The dashboard is importable directly from the repo (see below). Grab the .lvdash.json, <a href="https://docs.databricks.com/aws/en/dashboards/automate/import-export">import it as a new AI/BI dashboard</a> in your workspace, point it at a SQL warehouse, and refresh.</p><p>There’s one prerequisite that trips people up, and it’s not the import — it’s permissions. System tables have to be <a href="https://docs.databricks.com/aws/en/admin/system-tables/#enable"><strong>enabled by an account admin</strong></a>, and the schemas this dashboard reads (access, query, and billing) each have to be turned on. Whoever runs the dashboard also needs SELECT on those system schemas. If you import it and every tile comes back empty or throws a permissions error, that’s almost always the cause. Enabling system tables is a one-time account-level action; your platform or admin team can do it in a few minutes.</p><p>Once it’s importing cleanly, set the date range on the Global Filters page to something reasonable (we default to the last 30–90 days), pick a warehouse, and let it populate. From there it’s just filtering.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/305/1*uXSd-TygfkWv4XMa2sKQNg.png" /><figcaption>Setting the default date range</figcaption></figure><h3>Possible Upgrades</h3><p>If you want to take the dashboard a step further, there are four upgrades worth calling out.</p><ul><li><strong>How to find the dashboard ID: </strong>The fastest path is to grab it from the dashboard URL. For AI/BI dashboards, the dashboard ID is the value in the /sql/dashboardsv3/{id} URL, and if you are starting from the workspace file instead, workspace/get-status returns the dashboard resource_id, which is also the identifier used by the <a href="https://docs.databricks.com/api/workspace/lakeview/list">Dashboard API</a> and the published dashboard URL.</li><li><strong>How to automatically include the dashboard name:</strong> The main dashboard access audit events log dashboard_id, but not the human-readable dashboard name. A practical improvement is to resolve that ID through the <a href="https://docs.databricks.com/api/workspace/lakeview/get">dashboard API</a>, whose response includes display_name, so your reporting layer can show readable dashboard names without relying on a manual mapping file.</li><li><strong>Pull it via API into your own table:</strong> If you need longer retention or a cleaner reporting layer, snapshot the dashboard metadata into your own table on a schedule instead of depending only on the live retention window of system tables. The same dashboard_id can be used with the Dashboard API to pull dashboard details such as display_name, path, and warehouse_id, which gives you a lightweight dimension table you can reuse across joins and historical analysis.</li><li><strong>Add extra joins to show warehouse name explicitly: </strong>Query history records the warehouse used for a statement through compute.warehouse_id, while the warehouses system table exposes the readable warehouse_name. Joining on warehouse_id, and taking the latest warehouse snapshot for each ID, lets you replace opaque IDs with explicit warehouse names and makes cost and performance breakdowns much easier to interpret.</li></ul><h3>Where it earns its keep</h3><p>A few concrete ways we’ve seen this pay off:</p><ul><li><em>Rationalizing the estate.</em> Sort dashboards by activity and the long tail of zero-view dashboards becomes obvious. Those are candidates to archive — less clutter, fewer stale numbers floating around, and in some cases real savings.</li><li><em>Cost attribution and chargeback.</em> “Cost by dashboard” and “cost by user” give you a defensible, data-backed starting point for conversations about who’s driving warehouse spend, instead of guessing.</li><li><em>Performance triage.</em> The cache hit rate and average-duration trends make it easy to spot a dashboard that’s slow because it’s not benefiting from caching, or one whose performance has quietly degraded as its underlying tables grew.</li><li><em>Adoption tracking.</em> If you just shipped a dashboard to a new team, the viewer and query-volume trends tell you within days whether it landed or whether you need to go do some enablement.</li></ul><h3>Caveats — read these before you quote any numbers</h3><p>This is the part I’d want a colleague to read most carefully, because system-table data is powerful but it has sharp edges. Treat the dashboard as a strong directional signal, not a billing-grade source of truth.</p><p><strong>“Views” are an approximation.</strong> A single person opening one dashboard generates multiple audit events — an open event plus a widget-execution event for each tile on the page. Counting those actions gives you an excellent <em>engagement</em> signal, but it is not a literal count of unique human page loads, and it will run higher than a naive “one open = one view” intuition. If you need stricter semantics, narrow the action list (for example, count only getDashboard/getPublishedDashboard) and calibrate to your own environment.</p><p><strong>Cost is a list-price estimate.</strong> The dollar figures are computed from account_prices, which reflect list pricing. They do <strong>not</strong> reflect your organization’s actual negotiated or effective rate, discounts, or commit-based pricing. Use them to compare dashboards <em>against each other</em> and to spot outliers — not as a substitute for your real invoice.</p><p><strong>Cost attribution is cleanest on serverless SQL.</strong> Statement-level DBU attribution via attributed_usage lines up best with serverless SQL warehouses, where usage is tied to individual statements. On classic or pro SQL warehouses, billing is driven by cluster uptime rather than per-statement execution, so per-dashboard cost attribution can be partial or absent. Know which warehouse type your dashboards run on before you read too much into the cost tiles.</p><p><strong>System tables aren’t real-time.</strong> There’s ingestion latency between an event happening and it landing in the audit and billing tables — typically on the order of hours, and billing records can lag longer. Expect “today” to look sparse; judge trends on data that’s had time to settle.</p><p><strong>Retention is finite.</strong> System tables retain history for a bounded window, and the dashboard can only show you what’s still retained. If you need longer-horizon analysis, snapshot the data into your own table on a schedule.</p><p><strong>Query history is still in Public Preview.</strong> The system.query.history table — the backbone of the performance and cost datasets — is a Public Preview feature at the time of writing. Its schema can evolve, so keep an eye on the release notes if a field stops behaving the way you expect.</p><p>None of these should stop you from using the dashboard. They’re the difference between someone confidently presenting garbage and someone presenting a well-understood estimate with the right footnotes. The second person is the one who gets trusted with the next question.</p><h3>Grab it and make it yours</h3><p>Everything here is importable and fully editable. If your organization tags warehouses by team, add a tag breakdown. If you care about published-link traffic specifically, tighten the audit filter. If you want an alert when a dashboard’s cost crosses a threshold, wire one up on top of the cost dataset.</p><p>The point isn’t this exact set of tiles — it’s that the data to run your dashboards like a product, rather than shipping and forgetting them, is already in your account. This just connects the wires.</p><p><strong>Repo and import instructions: </strong><a href="https://github.com/Rosenberg-Josh/databricks-dashboard-usage-analytics"><strong>Repo Link</strong></a></p><p>If you end up extending it, we’d love to hear what you added.</p><p>*If there are any system tables referenced here that are not currently accesible, you can reach out to your account team for support.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=80bc0224b827" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dbsql-sme-engineering/whos-actually-using-your-databricks-dashboards-and-what-do-they-cost-80bc0224b827">Who’s Actually Using Your Databricks Dashboards (and What Do They Cost)?</a> was originally published in <a href="https://medium.com/dbsql-sme-engineering">DBSQL SME Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Agentic Era Needs a Trust Layer]]></title>
            <link>https://medium.com/@rosenberg.josh34/the-agentic-era-needs-a-trust-layer-d28be74276fe?source=rss-512ea28a2b18------2</link>
            <guid isPermaLink="false">https://medium.com/p/d28be74276fe</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[unity-catalog]]></category>
            <dc:creator><![CDATA[Josh Rosenberg]]></dc:creator>
            <pubDate>Fri, 22 May 2026 19:02:25 GMT</pubDate>
            <atom:updated>2026-05-22T19:02:25.655Z</atom:updated>
            <content:encoded><![CDATA[<h4>Why consolidating on a governed data platform will matter more than ever as AI moves from answering questions to taking action.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M-m90AwCAket-Ay0q4hmUw.png" /></figure><p>A few months ago, I was talking with a data leader who was trying to build a generative AI assistant for their business teams. This was early days of the Claude Code hype.</p><p>The prototype was impressive. You could ask a question in plain English, the assistant would retrieve context from internal documents, generate SQL against governed tables, and explain the answer in a way that felt polished enough for an executive demo.</p><p>But once we started talking about production, the conversation changed.</p><p>Who should the assistant be allowed to answer on behalf of? Which version of the customer table should it use? What happens if the same metric is defined differently in the BI layer and the transformation layer? Could the assistant see sensitive fields? Could it take an action in another system? If the answer was wrong, how would anyone know why?</p><p>The AI part was exciting. The platform questions were harder.</p><p>That conversation stuck with me because it reflects where a lot of companies are right now. They are not struggling to build impressive demos. They are struggling to build <strong>trusted systems</strong>.</p><p>And as we move from generative AI into the agentic era, that distinction is going to matter more than ever.</p><p>For the last decade, the modern data stack has largely been about specialization. You had one tool for ingestion, another for transformation, another for BI, another for ML, another for orchestration, another for governance, and another for monitoring. For a while, that tradeoff made sense. The goal was mostly to move data, model it, visualize it, and make it available to humans.</p><p>But the agentic era changes the equation.</p><p>When AI systems move from answering questions to taking actions, the cost of fragmentation goes up dramatically. Agents do not just need access to data. They need context, permissions, tools, business logic, lineage, evaluation, observability, and guardrails. They need to understand not only what the data says, but whether they are allowed to use it, where it came from, how fresh it is, and what action should happen next.</p><p>That is why consolidating on a data platform like Databricks is going to become increasingly important. Not because every company wants fewer tools for the sake of fewer tools, but because agentic systems require a level of connectedness, governance, and operational trust that fragmented architectures are poorly suited to provide.</p><h3>The agentic era is not just “chat with your data”</h3><p>A lot of early generative AI use cases were framed as conversational interfaces. Ask a question. Get an answer. Summarize a document. Generate some SQL. Search a knowledge base.</p><p>That was useful, but it was still mostly human-in-the-loop.</p><p>Agents are different. They can plan, call tools, collaborate across specialized workflows, and maintain enough state to complete multi-step tasks. That shift matters because the interface is no longer just conversational. It becomes operational.</p><p>A chatbot might answer, “Revenue was down in the West region because churn increased in mid-market accounts.”</p><p>An agent might go further. It could investigate the churn trend, check CRM notes, compare support tickets, look at product usage, draft a renewal risk summary, create a task for the account team, update a forecast, and monitor whether the intervention worked.</p><p>That is not just analytics. That is an operational workflow.</p><p>And once AI starts participating in operational workflows, the platform underneath it becomes much more important.</p><h3>Fragmented data stacks create fragmented agents</h3><p>Most companies already struggle with basic data fragmentation. Metrics mean different things in different dashboards. Permissions are managed differently across tools. Data lineage is incomplete. Business logic gets duplicated in notebooks, SQL models, dashboards, spreadsheets, and application code.</p><p>That is <em>annoying</em> when humans are consuming the data, but it becomes <strong>risky</strong> when agents are acting on it.</p><p>An agentic system needs to know which data is trusted, which business logic to apply, which permissions to respect, which tools it can call, and how to explain the path it took to reach an answer or complete a task. If those answers live across ten loosely connected systems, the agent becomes harder to trust, harder to debug, and harder to govern.</p><p>This is where consolidation becomes less of a platform preference and more of an architectural requirement.</p><h3>The real bottleneck is not the model. It is the system around the model.</h3><p>A lot of AI strategy conversations still start with the model: Which LLM should we use? Should we use open source or proprietary? How do we reduce token cost? Which benchmark is best?</p><p>Those are valid questions, but they are not the whole problem.</p><p>In enterprise environments, the harder questions are usually around the system surrounding the model. Can it access the right data? Can it access only the right data? Can we trace what happened? Can we evaluate quality? Can we monitor cost, latency, and failures? Can we enforce governance consistently enough for security and compliance teams to <strong>trust</strong> it?</p><p>This is why the data platform matters so much. The future will not be won by companies that simply connect an LLM to a pile of APIs. It will be won by companies that build trusted intelligence systems around their data, tools, and business processes.</p><p>Databricks is interesting here because it is not just a place to store data or run notebooks. The broader Data Intelligence Platform is converging around governed data, AI development, model serving, agent tooling, evaluation, lineage, and monitoring.</p><p>That convergence matters. In the agentic era, the platform is not just where data lives. It is where context, governance, and intelligence come together.</p><h3>Agents need governed access to tools, not just data</h3><p>One of the most important shifts in the agentic era is that tools become first-class assets.</p><p>A tool might be a SQL query, a function, a vector search endpoint, a forecasting model, a CRM update, a ticket creation workflow, or an MCP server exposing approved enterprise capabilities.</p><p>This means governance cannot stop at tables.</p><p>If an agent can answer a question, retrieve a document, query a table, call a model, and update a business system, then the organization needs a way to govern that entire chain. The user, the data, the model, the tool call, the output, and the action all need to fit inside a consistent security and observability model.</p><p>Enterprises do not need hundreds of ungoverned agents floating around the business. They need agents that operate inside the same security, compliance, and operational boundaries as the rest of the data estate.</p><p>This is one of the reasons a unified governance layer becomes so important. If data, models, functions, tools, and AI applications can be governed together, agentic systems become much easier to deploy responsibly.</p><h3>The semantic layer has to move closer to the governed data</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RzvxJdW74TETRPUyJCJTlw.png" /></figure><p>One of the most underestimated challenges in agentic AI is semantic consistency.</p><p>If an executive asks, “Why did revenue drop last quarter?” the agent needs to know what “revenue” means. Gross revenue? Net revenue? Bookings? Recognized revenue? Invoiced revenue? Revenue excluding refunds? Revenue by order date or payment date?</p><p>Humans can sometimes resolve that ambiguity through context. Agents need the business logic made explicit.</p><p>Historically, semantic logic has often been scattered across BI tools, dbt models, spreadsheets, notebooks, and application code. In the agentic era, that fragmentation becomes a major blocker. Agents need reliable, governed, reusable definitions they can call consistently.</p><p>The winning pattern is not “let every agent figure it out.” The winning pattern is curated data products, governed metrics, approved tools, and clear access boundaries.</p><p>This is where consolidating around a platform becomes powerful. If your governed data, metric definitions, lineage, AI tooling, and permission model are closer together, agents have a much better chance of producing answers that are not only technically correct, but business-correct.</p><h3>Evaluation becomes a core platform capability</h3><p>With traditional analytics, quality issues are often discovered by humans. Someone sees a number that looks wrong. A dashboard breaks. A stakeholder complains that a metric does not match another report.</p><p>With agents, that feedback loop is too slow.</p><p>Agentic systems need continuous evaluation. They need traces. They need human feedback. They need offline test sets. They need regression testing. They need monitoring for accuracy, latency, tool failures, cost, and unsafe behavior.</p><p>This is a key point for platform teams: <strong>agent quality cannot be managed with vibes.</strong></p><p>You need an engineering system around it. You need to know what the agent did, why it did it, what data it used, which tools it called, whether the answer was correct, and whether a new version made things better or worse.</p><p>That looks a lot more like software engineering, data engineering, and ML operations converging into one discipline.</p><h3>Customers do not just need a demo. They need an operating model.</h3><p>Most companies do not struggle to build an impressive prototype. They can connect an LLM to a document, build a RAG app, generate a SQL query, or create a workflow demo.</p><p>The hard part is production.</p><p>Production means identity, permissions, auditability, cost controls, latency, data freshness, monitoring, evaluation, CI/CD, rollback, human review, business ownership, and change management. All of those are manageable on their own. The challenge is making them work together.</p><p>That is why I think platform consolidation will become such a major theme. Not because every workload must run in one place, and not because every specialized tool goes away. But because the core control plane for enterprise intelligence needs to be unified.</p><p>The companies that succeed with agents will likely be the ones that build a governed data foundation, create reusable business logic, centralize identity and access controls, approve the tools agents are allowed to call, and treat tracing and evaluation as part of the development lifecycle.</p><p>That is not just an AI problem.</p><p>That is a platform problem.</p><h3>Consolidation does not mean eliminating every tool</h3><p>It is worth being clear: consolidation does not mean every company should abandon every point solution.</p><p>There will still be excellent tools for BI, reverse ETL, observability, experimentation, orchestration, activation, and application development. The point is not that one platform should replace every product in the enterprise.</p><p>The point is that the center of gravity needs to shift.</p><p>In the previous era, it was tolerable for governance, logic, data, and AI workflows to be loosely stitched together because humans were still doing much of the interpretation.</p><p>But in the agentic era, loose stitching becomes a liability.</p><p>The more agents are expected to reason over enterprise context and take action, the more important it becomes to have a governed, observable, interoperable foundation underneath them.</p><p>The best version of consolidation is not a closed monolith. It is an open, governed platform that allows other tools to plug in while keeping the most important controls centralized.</p><p>That is where platforms like Databricks are headed: open data formats, unified governance, scalable compute, AI tooling, model and agent lifecycle management, and integration with the broader ecosystem.</p><h3>The future data platform is an intelligence platform</h3><p>For years, the data platform was mostly judged by whether it could store data, transform data, and serve data to dashboards.</p><p>That bar is rising.</p><p>The next-generation data platform will be judged by whether it can help the business turn data into intelligent action safely.</p><p>That means supporting analytics, real-time and batch workloads, AI and ML, governed access to data and tools, evaluation, tracing, monitoring, and lineage. It means giving agents trusted context while still letting teams move quickly. It means making the intelligent thing easy, and the unsafe thing hard.</p><p>Agents are going to make data platforms more important, not less important. The more powerful the AI layer becomes, the more critical the underlying data and governance layer becomes.</p><p>A model without trusted context is a liability. An agent without governed tools is a risk. A workflow without observability is a black box. A fragmented platform makes all of this harder.</p><h3>The companies that win will build trust into the foundation</h3><p>The agentic era will create a lot of excitement. It will also create a lot of noise.</p><p>There will be demos everywhere. Some will be genuinely useful. Some will be fragile wrappers around APIs. Some will work beautifully in a controlled environment and fall apart in production.</p><p>The companies that create durable value will not be the ones with the flashiest prototype. They will be the ones that build the best foundation.</p><p>That is the real opportunity for consolidating on Databricks and platforms like it.</p><p>The agentic era is not just about giving every employee a chatbot. It is about rethinking how intelligence moves through the business.</p><p>And if intelligence is going to move through the business, then the platform it runs on has to be <strong>trusted</strong>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d28be74276fe" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Embedding Genie API for a Multi‑Tenant Application]]></title>
            <link>https://medium.com/dbsql-sme-engineering/embedding-genie-api-for-a-multi-tenant-application-d307bfbfc89b?source=rss-512ea28a2b18------2</link>
            <guid isPermaLink="false">https://medium.com/p/d307bfbfc89b</guid>
            <category><![CDATA[embedded-analytics]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[databricks-dashboard]]></category>
            <dc:creator><![CDATA[Josh Rosenberg]]></dc:creator>
            <pubDate>Sat, 21 Mar 2026 04:49:00 GMT</pubDate>
            <atom:updated>2026-03-31T16:40:09.517Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Implementation Guide for Attribute-Based Access Control (ABAC) for a Multi-Tenant Architecture for AI/BI Genie</em></p><p>By Josh Rosenberg (Sr. Solutions Engineer @ Databricks) &amp; Josh Bae (Delivery Solutions Architect @ Databricks)</p><h3><strong>Summary</strong></h3><ul><li>Embed the Genie Conversation API directly inside multi-tenant apps to deliver stateful analytics conversations backed by a curated Genie space.</li><li>Authenticate requests through your application using OAuth M2M with service principals, not individual Databricks user accounts or external workspace provisioning.</li><li>Securely personalize data with Unity Catalog ABAC policies and implement row-level security guardrails.</li></ul><h3>Introduction</h3><p>Since the launch of AI/BI Genie spaces, we have seen enterprise-scale adoption of self-service analytics using natural language for many Databricks users. While this served customers well for internal reporting and operational visibility, the lack of access for external audiences to the UI-native experience on Databricks made it difficult to share this capability more broadly.</p><p>However, with the <a href="https://www.databricks.com/blog/genie-conversation-apis-public-preview">Public Preview release</a> of the AI/BI Genie Conversation API across AWS, Azure, and GCP, users could now serve Genie-level insights from any application, even external or custom-built ones. The API provided a seamless way for business users to “talk to their data” from any surface, but it raised an immediate governance issue: how would the app ensure that only the data that is permitted for a particular viewer could be returned?</p><p>It is a problem we have seen many of our customers wrestle with, so this blog offers guidance on designing your multi-tenant Genie space apps with robust per-tenant guardrails via service principals (using OAuth M2M authentication) and Unity Catalog ABAC (attribute-based access control) policies.</p><h3>Reference Architecture</h3><p>This section outlines the high-level request flow and required infrastructure components for the integration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7QYdSa6IPik554bTO0Eztw.png" /></figure><p>At a high level, the runtime flow looks like this:</p><ol><li>Your application receives a question from a user in a multi-tenant app.</li><li>The app determines which tenant the request belongs to.</li><li>The app retrieves that tenant’s Databricks service principal credentials from a secure vault.</li><li>The app uses the OAuth client credentials flow to get an access token.</li><li>The app calls the Genie Conversation API using that tenant-specific identity.</li><li>Genie generates SQL against curated Unity Catalog tables (in a Genie space).</li><li>ABAC policies evaluate current_user() during query execution.</li><li>Only the rows and columns that the identity is allowed to access are returned.</li></ol><p>That basic runtime architecture is what makes the pattern powerful: every request runs as a tenant-specific principal, so identity becomes the control point for access.</p><h3>Why this pattern works</h3><p>The major design choice here is to use a dedicated <em>service principal (SP) per tenant</em> (which we will elaborate on in later sections). That way, your application no longer has to be the primary enforcement layer for data isolation. Instead, Databricks governance via Unity Catalog handles it at query time.</p><p>When a request runs, Unity Catalog evaluates policies using the current caller, and because each tenant has its own SP, you can write rules directly against that identity. This dramatically simplifies the mental model. Rather than sprinkling tenant filters throughout your app and hoping they are always applied correctly, you let the Databricks Platform evaluate access where it matters most: when the SQL actually runs. The tradeoff is operational: you now have many identities and secrets to manage, so automation is essential.</p><h3>Multi-Tenancy Design</h3><p>Tenant isolation hinges on how you structure service principals and how Unity Catalog ABAC evaluates row filters and column masks at query time. Let’s clarify these terms to better articulate how they will serve the implementation in the proposed solution later in this blog.</p><p><strong><em>What’s a service principal (SP)?</em></strong></p><p>Databricks recommends using SPs as specialized identities for accessing Databricks resources because they avoid relying on individual user credentials. It provides a programmatic way to grant/restrict access to the Genie space(s) without defining how each individual user will query the underlying data. Since all requests run under the tenant SP, filters/masks apply consistently without per-user identity.</p><p><strong><em>What’s an attribute-based access control (ABAC) policy?</em></strong></p><p>ABAC in Unity Catalog is a governance model that lets you centrally express access rules using attributes and have them enforced automatically at query time. In practice, ABAC policies typically take the form of row filters (to limit which rows are visible) and column masks (to redact sensitive fields), and those policies can be inherited down the object hierarchy (catalog → schema → table). Under our recommended design, applications can authenticate via SPs and rely on ABAC to enforce tenant boundaries at the data layer, rather than implementing security solely in application code.</p><p>We will cover authoring ABAC policies in a later section, but you can read more at <em>Unity Catalog attribute-based access control (ABAC).</em> (<a href="https://docs.databricks.com/aws/en/data-governance/unity-catalog/abac/">AWS</a> | <a href="https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/abac/">Azure</a> | <a href="https://docs.databricks.com/gcp/en/data-governance/unity-catalog/abac/">GCP</a>)</p><h3>Implementation Guide</h3><h4>Prerequisites</h4><p>Before implementing this pattern, make sure you have the following in place:</p><ul><li>Be an account admin and/or workspace admin (SP creation, generating OAuth secrets, etc.)</li><li>Have access to the following:<br>- A Databricks workspace with the Databricks SQL entitlement<br>- At least CAN EDIT permissions on the Genie space<br>- CAN USE access on at least one pro or serverless SQL warehouse<br>- SELECT privileges on the data used in the space<br>- A control-plane process or service for onboarding tenants<br>- A secure secret store or vault to manage SP credentials</li></ul><p>Additionally, to create OAuth secrets for SPs, you should also account for several ABAC-related constraints when designing this architecture. ABAC in Unity Catalog requires <em>Serverless compute</em> (<a href="https://docs.databricks.com/aws/en/compute/serverless/">AWS</a> | <a href="https://learn.microsoft.com/en-us/azure/databricks/compute/serverless/">Azure</a> | <a href="https://docs.databricks.com/gcp/en/compute/serverless/notebooks">GCP</a>) or <em>Standard compute running on Databricks Runtime (DBR) 16.4+</em>, and there are policy limits to consider, such as <strong>one row filter per table per user</strong> and <strong>one mask per column per user</strong>. Those constraints should help inform how you design your governance and policy layer from the outset, avoiding conflicts or unnecessary complexity later.</p><p>For best practices, limitations, and example user-defined functions (UDFs), see <em>UDFs for ABAC policies best practices</em> (<a href="https://docs.databricks.com/aws/en/data-governance/unity-catalog/abac/udf-best-practices">AWS</a> | <a href="https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/abac/udf-best-practices">Azure</a> | <a href="https://docs.databricks.com/gcp/en/data-governance/unity-catalog/abac/udf-best-practices">GCP</a>)</p><h4>Step 1: Provision a service principal for each tenant</h4><p>The first step is to treat tenant identity as a first-class component of your onboarding process.</p><p>When a new tenant is created, your provisioning system should create (or retrieve) a dedicated SP for that tenant, assign it to the appropriate workspace, and treat that identity as the credential your application will use whenever it needs to execute requests on the tenant’s behalf. Your app should not create identities dynamically at runtime. Instead, identity provisioning should live in a separate onboarding or control-plane process responsible for identity lifecycle management.</p><p>Maintaining this separation is important. Your application runtime should consume tenant credentials rather than manage tenant identities directly. Centralizing identity provisioning within a control plane ensures clearer lifecycle management, stronger security practices, and a more maintainable multi-tenant architecture.</p><p>You can leverage the Account SCIM REST API to do this.</p><p>To execute the HTTP requests, you will need your Databricks account ID, the account console URL (such as <a href="https://accounts.cloud.databricks.com/">https://accounts.cloud.databricks.com</a> for AWS), and an account-level access token. You will also need the exact SCIM ID of the group where you plan to assign the new service principal.</p><p><em>Note: The code can also be run with WorkspaceClient since workspace admins can also add SPs to the Databricks account.</em></p><pre>import os<br>import requests<br><br># Retrieve configuration from environment variables<br>ACCOUNT_ID = os.environ[&quot;DATABRICKS_ACCOUNT_ID&quot;]<br>ACCOUNT_HOST = &quot;https://accounts.cloud.databricks.com&quot;<br>TOKEN = os.environ[&quot;DATABRICKS_ACCOUNT_TOKEN&quot;]  # admin token<br><br># Define a function that creates a new service principal<br>def create_service_principal(display_name: str):<br><br>    # Build the account-level API endpoint for creating a service principal<br>    url = (<br>        f&quot;{ACCOUNT_HOST}/api/2.0/accounts/{ACCOUNT_ID}&quot;<br>        f&quot;/scim/v2/ServicePrincipals&quot;<br>    )<br><br>    # Send a POST request to Databricks to create the service principal<br>    response = requests.post(<br>        url,<br>        headers={<br>            &quot;Authorization&quot;: f&quot;Bearer {TOKEN}&quot;,<br>            &quot;Content-Type&quot;: &quot;application/scim+json&quot;<br>        },<br>        json={<br>            &quot;schemas&quot;: [&quot;urn:ietf:params:scim:schemas:core:2.0:ServicePrincipal&quot;],<br>            &quot;displayName&quot;: display_name<br>        },<br>        timeout=30,<br>    )<br><br>    # Raise an exception if Databricks returns an HTTP error status<br>    response.raise_for_status()<br>    return response.json()</pre><p>Not included above, but Databricks recommends assigning each SP in Unity Catalog to <em>groups</em>. They simplify access to workspaces, data, and other securable objects.</p><p>Check out <em>Manage service principals</em> (<a href="https://docs.databricks.com/aws/en/admin/users-groups/manage-service-principals">AWS</a> | <a href="https://learn.microsoft.com/en-us/azure/databricks/admin/users-groups/manage-service-principals">Azure</a> | <a href="https://docs.databricks.com/gcp/en/admin/users-groups/manage-service-principals">GCP</a>) for more explanations on how to manage SPs for your Databricks account and workspaces.</p><h4>Step 2: Generate and store the OAuth secret</h4><p>Once the tenant’s SP has been created and registered in Unity Catalog, the next step is to generate an OAuth secret that your application can use as the client_secret in the OAuth client-credentials flow.</p><p>You can create this secret programmatically using the Databricks API.</p><p><em>Note: A service principal can have up to five OAuth secrets, each with a configurable lifetime (up to two years).</em></p><pre>import os<br>import requests<br><br># Retrieve configuration from environment variables<br>ACCOUNT_ID = os.environ[&quot;DATABRICKS_ACCOUNT_ID&quot;]<br>ACCOUNT_HOST = &quot;https://accounts.cloud.databricks.com&quot;<br>TOKEN = os.environ[&quot;DATABRICKS_ACCOUNT_TOKEN&quot;]  # admin token<br><br># Define a function that creates an OAuth secret for a given SP ID<br>def create_oauth_secret(service_principal_id: str):<br><br>    # Build the account-level API endpoint for creating a SP secret<br>    url = (<br>        f&quot;{ACCOUNT_HOST}/api/2.0/accounts/{ACCOUNT_ID}&quot;<br>        f&quot;/servicePrincipals/{service_principal_id}/credentials/secrets&quot;<br>    )<br><br>    # Send a POST request to Databricks to create the OAuth secret<br>    response = requests.post(<br>        url,<br>        headers={&quot;Authorization&quot;: f&quot;Bearer {TOKEN}&quot;},<br>        json={&quot;lifetime&quot;: &quot;15552000s&quot;},  # 180 days<br>        timeout=30,<br>    )<br><br>    # Raise an exception if Databricks returns an HTTP error status<br>    response.raise_for_status()<br><br>    return response.json()</pre><p>One important detail is that the OAuth secret string value is displayed only once in the API response and cannot be retrieved again. As soon as it is shown, you should immediately store it in a secure secrets manager and never persist it in plaintext outside the vault. In practice, you will store the tenant’s client_id, client_secret, secret_id, and expires_at so you can authenticate at runtime and rotate secrets safely later.</p><p>Check out <em>Authorize service principal access to Databricks with OAuth </em>(<a href="https://docs.databricks.com/aws/en/dev-tools/auth/oauth-m2m">AWS</a> | <a href="https://learn.microsoft.com/en-us/azure/databricks/dev-tools/auth/oauth-m2m">Azure</a> | <a href="https://docs.databricks.com/gcp/en/dev-tools/auth/oauth-m2m">GCP</a>) for more information and instructions on how to do this via UI.</p><h4>Step 3: Grant the minimum permissions during provisioning</h4><p>After the SP and its associated OAuth secret are created, the next step is to grant only the permissions required for the tenant to use Genie and query the data it is authorized to access.</p><p>To use a Genie space, the SP must have two things:</p><ul><li>The consumer access or Databricks SQL entitlement</li><li>At least SELECT privileges on all of the Unity Catalog data objects used in the space</li></ul><p><em>Note: Queries run using the compute credentials embedded by the author who configured the warehouse. End users do not need direct warehouse permissions.</em></p><p>This should be part of your automated provisioning flow, not a manual admin step. A Python example using the Databricks SDK for Genie permissions looks like the following:</p><pre>import requests<br><br>WORKSPACE_HOST = &quot;https://&lt;your-workspace-instance&gt;.cloud.databricks.com&quot;<br>WORKSPACE_TOKEN = os.environ[&quot;DATABRICKS_WORKSPACE_TOKEN&quot;] <br><br>def provision_genie_client_permissions(<br>    host: str,<br>    token: str,<br>    sp_app_id: str,      # client_id of the tenant SP<br>    genie_space_id: str,<br>):<br>    headers = {&quot;Authorization&quot;: f&quot;Bearer {TOKEN}&quot;}<br><br>    # Allow the service principal to run the Genie space<br>    requests.patch(<br>        f&quot;{WORKSPACE_HOST}/api/2.0/permissions/genie/{genie_space_id}&quot;,<br>        headers=headers,<br>        json={<br>            &quot;access_control_list&quot;: [{<br>                &quot;service_principal_name&quot;: sp_app_id,<br>                &quot;permission_level&quot;: &quot;CAN_RUN&quot;<br>            }]<br>        },<br>        timeout=15<br>    ).raise_for_status()</pre><p>To grant SELECT permissions on the data, you can use SQL:</p><pre>GRANT USE CATALOG ON CATALOG main TO `&lt;service_principal&gt;`;<br>GRANT USE SCHEMA ON SCHEMA main.sales TO `&lt;service_principal&gt;`;<br>GRANT SELECT ON ALL TABLES IN SCHEMA main.sales TO `&lt;service_principal&gt;`</pre><p>At this point, the SP has sufficient permissions to function (to query the underlying tables of the Genie space). However, we still have not defined the guardrails that control what data it can actually see. This is where ABAC policies come into play, enforcing which rows and columns are visible to the SP at query time.</p><h4>Step 4: Enforce tenant isolation with ABAC</h4><p>With identities and permissions in place, you can now enforce the actual tenant boundary. Because each tenant has its own SP, you can write ABAC policies that evaluate the current identity directly.</p><p>A common pattern is to maintain a mapping table between SPs and tenant IDs, then use a function that checks whether the current caller is authorized for a given row. For example:</p><pre>CREATE OR REPLACE FUNCTION main.sec.filter_by_tenant(tenant_id STRING)<br>RETURNS BOOLEAN<br>RETURN EXISTS (<br>  SELECT 1<br>  FROM main.sec.tenant_acl t<br>  WHERE t.tenant_id = tenant_id<br>    AND t.principal = current_user()<br>);</pre><p>You can then apply that function as a row filter:</p><pre>ALTER TABLE main.sales.customer_orders<br>SET ROW FILTER main.sec.filter_by_tenant ON (tenant_id);</pre><p>At that point, queries are automatically scoped at execution time. It does not matter whether the SQL was written by hand or generated by Genie — if the current SP is not allowed to see a row, that row will not be returned.</p><p>This is one of the strongest aspects of the pattern: enforcement stays close to the data rather than in the application layer.</p><p>Column masking follows the same idea. If you have fields such as SSNs, account numbers, or other sensitive attributes, you can apply a masking policy so those values are exposed only when the SP is explicitly authorized.</p><h4>Step 5: Curate the Genie space for quality answers</h4><p>Once the security model for your tenants is in place, you can focus on improving the quality of responses in your Genie space. A Genie space is only as good as the data model and business context it understands, so it is worth spending time curating it.</p><p>Focus on adding clear table and column descriptions, defining joins and key measures, and including business-friendly synonyms or metadata so Genie can interpret questions accurately. Provide a small set of high-quality example queries — typically five to ten — to demonstrate common questions and correct SQL patterns. Keep the space focused on a small number of well-understood tables, and use a Serverless SQL Warehouse when available for better performance and scalability.</p><p>See <em>Curate an effective Genie space</em> (<a href="https://docs.databricks.com/aws/en/genie/best-practices">AWS</a> | <a href="https://learn.microsoft.com/en-us/azure/databricks/genie/best-practices">Azure</a> | <a href="https://docs.databricks.com/gcp/en/genie/best-practices">GCP</a>) for more best practices defining a new space.</p><h4>Step 6: Authenticate per tenant at runtime</h4><p>With provisioning complete, your runtime path becomes straightforward. When the application receives a request for a given tenant, it loads that tenant’s stored credentials and exchanges them for a Databricks access token using the OAuth client-credentials flow.</p><p>A simple example looks like this:</p><pre>import requests<br><br>ACCOUNT_HOST = &quot;https://accounts.cloud.databricks.com&quot;<br>ACCOUNT_ID = &quot;&lt;your_account_id&gt;&quot;<br>CLIENT_ID = &quot;&lt;tenant_client_id&gt;&quot;<br>CLIENT_SECRET = &quot;&lt;tenant_secret&gt;&quot;<br><br>resp = requests.post(<br>    f&quot;{ACCOUNT_HOST}/oidc/accounts/{ACCOUNT_ID}/v1/token&quot;,<br>    auth=(CLIENT_ID, CLIENT_SECRET),<br>    data={&quot;grant_type&quot;: &quot;client_credentials&quot;, &quot;scope&quot;: &quot;all-apis&quot;},<br>    timeout=15<br>)<br><br>resp.raise_for_status()<br>token = resp.json()[&quot;access_token&quot;]</pre><p>These tokens are short-lived (roughly one hour), so your application should cache and refresh them automatically rather than requesting a new token for every call. If you are using the Databricks SDK, token handling is typically abstracted for you.</p><p>At this point, you have everything you need for secure tenant-scoped execution: a tenant-specific SP, a short-lived access token, and ABAC policies that evaluate that identity when the query runs.</p><h4>Step 7: Call the Genie conversation API</h4><p>You can now call Genie. The API is stateful, so the typical pattern is to start a conversation, receive a conversation ID and message ID, and then poll until the message completes and results are available.</p><p>The simplified flow looks like this:</p><pre>import time<br>import requests<br><br>WORKSPACE_HOST = &quot;https://&lt;your-workspace-instance&gt;.cloud.databricks.com&quot;<br>WORKSPACE_TOKEN = os.environ[&quot;DATABRICKS_WORKSPACE_TOKEN&quot;]  <br>SPACE_ID = &quot;&lt;your_genie_space_id&gt;&quot;<br><br>headers = {&quot;Authorization&quot;: f&quot;Bearer {WORKSPACE_TOKEN}&quot;}<br><br># 1. Start the Genie conversation<br>start = requests.post(<br>    f&quot;{WORKSPACE_HOST}/api/2.0/genie/spaces/{SPACE_ID}/start-conversation&quot;,<br>    json={&quot;content&quot;: &quot;Top customers by revenue last month&quot;},<br>    headers=headers,<br>    timeout=15<br>)<br>start.raise_for_status()<br><br># Extract conversation and message IDs <br>conversation_id = start.json()[&quot;conversation&quot;][&quot;id&quot;]<br>message_id = start_data[&quot;message&quot;][&quot;id&quot;]<br><br><br># 2. Poll for the execution to finish<br>for _ in range(60):<br>    poll = requests.get(<br>        f&quot;{WORKSPACE_HOST}/api/2.0/genie/spaces/{SPACE_ID}/conversations/&quot;<br>        f&quot;{conversation_id}/messages/{message_id}&quot;,<br>        headers=headers,<br>        timeout=15<br>    )<br>    poll.raise_for_status()<br><br>    status = poll.json()[&quot;status&quot;]<br><br>    if status == &quot;COMPLETED&quot;:<br>        break<br><br>    time.sleep(2)</pre><p>In production, you will want to add robust retry logic, exponential backoff, timeout handling, and monitoring around failures or API limits. But conceptually, this is the full runtime loop: authenticate as the tenant, call Genie, let Genie generate SQL, and rely on ABAC to enforce tenant boundaries during execution.</p><p>Check out <em>Best practices for using the Genie API</em> (<a href="https://docs.databricks.com/aws/en/genie/conversation-api#best-practices-for-using-the-genie-api">AWS</a> | <a href="https://learn.microsoft.com/en-us/azure/databricks/genie/conversation-api">Azure</a> | <a href="https://docs.databricks.com/gcp/en/genie/conversation-api#best-practices-for-using-the-genie-api">GCP</a>) to maintain performance and reliability when using the Genie API.</p><h3>Operating this pattern in production</h3><p>Once your core flow is live, the heavy lifting shifts from initial implementation to ongoing operations. To keep your system secure, efficient, and easy to maintain, focus on four key areas.</p><h4>1. Secret lifecycle management</h4><p>Secrets are never a one-time setup step. They require regular rotation, with a reasonable rule of thumb being every 60 to 180 days. Fortunately, Databricks supports multiple active secrets per SP, enabling zero-downtime rotation. The overlap between old and new secrets is what keeps your production environment safe during the transition.</p><p>A step-by-step flow for rotating your secrets:</p><ul><li>Create a new secret and store it securely in your vault</li><li>Update your application to use this newly generated secret</li><li>Verify that all traffic has successfully switched over</li><li>Revoke the old secret only after confirming the new one works</li></ul><h4>2. Clean deprovisioning</h4><p>When a tenant is offboarded, you should carefully reverse the initial setup steps. Keeping this teardown process as part of your standard control-plane lifecycle makes the entire system much easier to reason about over time. Be sure to revoke active secrets, remove permissions, and optionally delete the SP entirely.</p><h4>3. Observability and Genie space hygiene</h4><p>As your platform usage grows, you need deep visibility into how the system is behaving. Keep a close eye on query patterns, performance bottlenecks, conversation volume, and audit signals tied to policy enforcement. You also need to maintain good space hygiene by deleting old records, especially since spaces often have strict conversation limits.</p><p><em>Note: As of the date of this publication, a Genie space has a limit of 10,000 conversations.</em></p><h4>4. Evolving Your ABAC Model</h4><p>Finally, expect your ABAC model to evolve as your data model changes. New tables, new sensitive fields, and new use cases will almost certainly require updates to your filters and masks. That is normal. The advantage of this architecture is that these changes occur in the governed policy and metadata layers, rather than by rewriting application filtering logic.</p><h3>Operating this pattern in production</h3><p>What makes this pattern highly effective is its clean separation of responsibilities, decoupling application logic from data governance. Service principals establish clear tenant identities, while OAuth provides a secure mechanism for your application to assume those identities at runtime. At the data layer, Unity Catalog ABAC policies dynamically enforce row and column-level access rules, freeing your application from managing complex, error-prone tenant filters. By combining these robust governance guardrails with the Genie Conversation API, you can confidently scale self-service analytics and securely deliver a rich, natural language data experience to external users directly within your application. Ultimately, this approach yields a multi-tenant architecture that is fundamentally easier to secure, scale, and maintain than pushing tenant filters deep into application code.</p><h3>Resources</h3><ul><li>Genie Conversation <a href="https://docs.databricks.com/api/workspace/genie">APIs</a>/<a href="https://databricks-sdk-py.readthedocs.io/en/latest/workspace/dashboards/genie.html">SDK</a></li><li>For more on embedding Genie into your custom web apps with Genie Conversation APIs, check out <a href="https://www.databricks.com/blog/access-genie-everywhere">Use Genie Everywhere with Enterprise OAuth</a>.</li><li>For more on building a Genie space from scratch to production-ready through benchmark evaluation and systematic optimization, check out <a href="https://www.databricks.com/blog/how-build-production-ready-genie-spaces-and-build-trust-along-way">How to Build Production-Ready Genie Spaces, and Build Trust Along the Way</a>.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d307bfbfc89b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dbsql-sme-engineering/embedding-genie-api-for-a-multi-tenant-application-d307bfbfc89b">Embedding Genie API for a Multi‑Tenant Application</a> was originally published in <a href="https://medium.com/dbsql-sme-engineering">DBSQL SME Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Filter Across Datasets in Databricks AI/BI Dashboards]]></title>
            <link>https://medium.com/@rosenberg.josh34/how-to-filter-across-datasets-in-databricks-ai-bi-dashboards-8db9e5d2f172?source=rss-512ea28a2b18------2</link>
            <guid isPermaLink="false">https://medium.com/p/8db9e5d2f172</guid>
            <category><![CDATA[dashboard]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[business-intelligence]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Josh Rosenberg]]></dc:creator>
            <pubDate>Fri, 16 Jan 2026 18:42:05 GMT</pubDate>
            <atom:updated>2026-01-16T18:42:05.884Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Design patterns, real SQL examples, and when to rethink your model</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*blZMPkc2HOclTX3HObJAqg.png" /><figcaption>Databricks AI/BI Dashboard</figcaption></figure><p>A common question I see come up when teams start building dashboards in Databricks AI/BI is deceptively simple:</p><blockquote><em>“How do I apply one filter across multiple datasets?”</em></blockquote><p>If you’re coming from tools like Tableau or Power BI, you might expect filters to automatically “just work” across charts. In Databricks, they don’t. And that’s not a bug. It’s a design choice that forces you to be explicit about your data model.</p><p>Once you understand how filters actually work under the hood, building clean, predictable dashboards becomes much easier. This article walks through the mental model, the most reliable patterns, concrete SQL examples, and the warning signs that your dashboard needs a redesign rather than another filter.</p><h3>Why cross-dataset filtering is harder than it sounds</h3><p>Databricks dashboards sit directly on top of your lakehouse. There is no hidden semantic layer quietly reconciling column names or inferring joins for you.</p><p>A dashboard filter does not join datasets or propagate logic. It only passes values.</p><p>Each chart runs its own query. The filter simply injects a parameter into that query. If the query does not explicitly use that parameter in a compatible way, nothing happens. Sometimes worse, it looks like it’s working while returning misleading results.</p><p>This is the tradeoff Databricks makes. You get transparency, performance, and governance. In return, you have to model your data intentionally.</p><h3>The mental model: how filters actually work</h3><p>At a mechanical level, Databricks AI/BI dashboard filters are just parameters.</p><ol><li>A filter is defined with a name and data type</li><li>The user selects one or more values</li><li>Each tile receives those values</li><li>Each query decides how to apply them</li></ol><p>There is no global awareness of relationships between datasets. Two charts will only respond to the same filter if:</p><ul><li>They reference the same parameter name</li><li>They apply it to semantically equivalent columns</li><li>The data types align</li></ul><p>Once you internalize this, most “why isn’t my filter working?” issues become obvious.</p><h3>Prerequisite: shared filter keys (and why naming matters)</h3><p>Before we talk patterns, this is non-negotiable.</p><p>If you want a filter to work across datasets, the datasets must share a compatible filter key. That means:</p><ul><li>Same business meaning</li><li>Same data type</li><li>Same grain</li></ul><p>For example, these are not interchangeable without normalization:</p><ul><li>customer_id vs licensing_customer_id</li><li>event_timestamp vs event_date</li><li>region_name vs region_code</li></ul><p>Dashboards are not the place to fix this. Views are.</p><p>If you fix it once in a view, every dashboard benefits.</p><h3>Pattern 1: Shared dimension table (recommended default)</h3><p>If you remember one thing from this article, let it be this:<br><strong>Most cross-dataset filtering problems are data modeling problems.</strong></p><p>The cleanest, most scalable approach is a shared dimension table or view that all fact queries join against.</p><h3>When to use this pattern</h3><ul><li>Multiple fact tables</li><li>Stable business dimensions like customer, provider, region, product</li><li>Dashboards that need to scale beyond one use case</li></ul><p>You can define these dimension tables upstream or in the dashboards themselves as their own datasets.</p><p>Let’s walk through an example:</p><p>I have some data related to ticket sales for a company that operates entertainment venues. Tables include customers, events, ticket sales, and venues.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/992/1*O8P23eQg5mWCdsa9wnXGaQ.png" /></figure><p>By default, these tables don’t have any relationship between them, so I can’t filter across them if I want to filter events and ticket sales at the same time.</p><h3>Enter Parameters</h3><p>By creating a specific table for a field I want to use as a parameter, I can use that field in multiple queries to cross filter across tables.</p><p>Let’s say I want to filter based on event_id. The first step would be to create a new dataset in here that contains unique event_ids.</p><pre>SELECT DISTINCT event_id FROM jr_demos.entertainment_venue.events</pre><p>Now, within my queries I can add a new parameter that will have the same values as the event_id’s I just created.</p><pre>SELECT * FROM jr_demos.entertainment_venue.ticket_sales<br>WHERE array_contains(:event_id, event_id)</pre><p>If I want the default to be all event_id’s rather than a specific one, I can update the query to be as follows:</p><pre>SELECT * FROM jr_demos.entertainment_venue.ticket_sales<br>WHERE <br>  CASE WHEN size(:event_id) = 0 THEN true <br>  ELSE array_contains(:event_id, event_id) END</pre><p>Once this WHERE statement is added to each query for each field you want to use to cross filter, you can then use this new parameter to filter across datasets.</p><p>To accomplish this, first add a filter to your dashboard. I’ll be adding one for Event ID.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6gMnTiDGF7-WB8kaquA1FA.png" /></figure><p>Next, in the configuration pane for the filter widget, you can add the event_id from the dim table as the field, and add each parameter you created from the datasets you want to filer across:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/354/1*gfx_ocicPtYD2AaymsHeMA.png" /></figure><p>I also set the default value to none so it defaults to all results.</p><p>And voila! I now have the ability to use a single filter across multiple datasets without having to join them together.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-b13piPaJUEBSCZpiL_log.gif" /></figure><h3>Why this works so well</h3><ul><li>One filter key, many facts</li><li>Filters bind to dimension attributes, not raw IDs</li><li>Easy to govern with Unity Catalog</li><li>Easy to extend later</li></ul><p>This is the pattern I default to unless I have a strong reason not to.</p><h3>Pattern 2: Pre-joined or wide tables (use sparingly)</h3><p>Sometimes simplicity wins. For executive dashboards or high-visibility reporting, a pre-joined table can be the right call.</p><h3>Example</h3><pre>CREATE OR REPLACE TABLE analytics.provider_summary AS<br>SELECT<br>  p.provider_id,<br>  p.provider_name,<br>  p.region,<br>  SUM(e.enrollment_count) AS total_enrollments,<br>  COUNT(i.inspection_id) AS inspection_count<br>FROM dim_provider p<br>LEFT JOIN fact_enrollments e<br>  ON p.provider_id = e.provider_id<br>LEFT JOIN fact_inspections i<br>  ON p.provider_id = i.provider_id<br>GROUP BY ALL;</pre><p>Filters become trivial:</p><pre>SELECT *<br>FROM analytics.provider_summary<br>WHERE region = :region;</pre><h3>The catch</h3><p>This is a UX optimization, not a modeling strategy. It trades flexibility for convenience and can become painful as requirements change.</p><h3>Looking ahead</h3><p>I’m excited about where this is heading.</p><p>As Databricks continues investing in AI/BI, metrics, and semantic modeling, there’s a lot of opportunity to reduce the boilerplate required to make cross-dataset filtering work, while still keeping the transparency and control that make the platform compelling in the first place.</p><p>If future features can make it easier to express intent once and reuse it everywhere, without hiding what’s actually happening under the hood, that’s a big win. Until then, being explicit and disciplined about modeling has been the most reliable way I’ve found to build dashboards that scale.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8db9e5d2f172" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Better Way to Think About Goal Setting This New Year]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@rosenberg.josh34/a-better-way-to-think-about-goal-setting-this-new-year-3718a1521f58?source=rss-512ea28a2b18------2"><img src="https://cdn-images-1.medium.com/max/1536/1*nEmgmCPRfiA46V2EZXNAGA.png" width="1536"></a></p><p class="medium-feed-snippet">The art of real accomplishment</p><p class="medium-feed-link"><a href="https://medium.com/@rosenberg.josh34/a-better-way-to-think-about-goal-setting-this-new-year-3718a1521f58?source=rss-512ea28a2b18------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@rosenberg.josh34/a-better-way-to-think-about-goal-setting-this-new-year-3718a1521f58?source=rss-512ea28a2b18------2</link>
            <guid isPermaLink="false">https://medium.com/p/3718a1521f58</guid>
            <category><![CDATA[new-years-resolutions]]></category>
            <category><![CDATA[motivation]]></category>
            <category><![CDATA[goals]]></category>
            <dc:creator><![CDATA[Josh Rosenberg]]></dc:creator>
            <pubDate>Sun, 23 Nov 2025 16:21:44 GMT</pubDate>
            <atom:updated>2025-11-23T16:21:44.701Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Why You’re Easier to Influence Than You Think]]></title>
            <link>https://medium.com/@rosenberg.josh34/why-youre-easier-to-influence-than-you-think-86a44dcd4dc0?source=rss-512ea28a2b18------2</link>
            <guid isPermaLink="false">https://medium.com/p/86a44dcd4dc0</guid>
            <category><![CDATA[influence]]></category>
            <category><![CDATA[book-recommendations]]></category>
            <category><![CDATA[psychology]]></category>
            <category><![CDATA[books]]></category>
            <category><![CDATA[persuasion]]></category>
            <dc:creator><![CDATA[Josh Rosenberg]]></dc:creator>
            <pubDate>Sat, 15 Nov 2025 22:07:16 GMT</pubDate>
            <atom:updated>2025-11-15T22:07:16.992Z</atom:updated>
            <content:encoded><![CDATA[<p>And other insights from Cialdini’s classic</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/225/0*VtuvQmhV8yXrkjUd" /></figure><p>Welcome to my Book of the Month series, where I take a dense, idea-heavy book and break it down into something you can read in a few minutes. I do this for two reasons. First, to share the most valuable insights in a way that feels clear and practical. Second, to force myself to actually remember what I read. There’s no better way to retain information than to explain it.</p><p>Think about the last time you were nudged into a decision without fully realizing why. Maybe you bought something you hadn’t planned to. Maybe you said yes to plans even though you wanted to stay home. Maybe you agreed to “just one more drink” after telling yourself you were done. Whatever the situation, you were influenced. We all are. Small nudges shape our behavior far more than we want to admit.</p><p>Cialdini’s <em>Influence</em> breaks down these forces into a handful of psychological levers. When you understand them, you get better at recognizing how people try to persuade you, and you also get better at communicating with others. This awareness won’t make you immune overnight, but it gives you just enough space to make decisions that match your real intentions instead of reacting on autopilot.</p><p>Below are the six major principles and how to use (or defend against) each one.</p><h3>Reciprocity</h3><p><strong>Example:</strong><br>You try a free sample at the grocery store, and suddenly you feel like you should buy the product, even if you didn’t want it in the first place.</p><p><strong>What it is:</strong><br>Humans are wired to repay what they receive. Even small gestures create an invisible pressure to return the favor.</p><p><strong>How to defend against it:</strong><br>Pause and ask yourself whether you actually want what is being offered. Appreciation does not require payback.</p><p><strong>How to use it ethically:</strong><br>Give value with no expectation attached. People can feel when it’s genuine.</p><h3>Commitment and Consistency</h3><p><strong>Example:</strong><br>You agree to a small request like signing up for a newsletter. Soon after, you’re asked to attend an event or make a donation. You feel pressure to keep saying yes because you “already started.”</p><p><strong>What it is:</strong><br>We work hard to stay consistent with our past statements and actions. Once we commit to something, we feel an internal push to follow through.</p><p><strong>How to defend against it:</strong><br>Ask yourself whether you would make the same decision if you weren’t already involved. If the answer is no, step back.</p><p><strong>How to use it ethically:</strong><br>Encourage people to commit to goals they already care about. Help them follow through rather than pushing them into something new.</p><h3>Social Proof</h3><p><strong>Example:</strong><br>You try a new restaurant because it keeps showing up in your feed. Thousands of others have checked it out, so you trust that it’s good.</p><p><strong>What it is:</strong><br>When we are uncertain, we look to others as a guide. Popularity becomes a shortcut for trust and safety.</p><p><strong>How to defend against it:</strong><br>Remind yourself that common choices are not automatically good ones. Choose based on your own needs, not the crowd’s excitement.</p><p><strong>How to use it ethically:</strong><br>Show real stories, real outcomes, and real success. Don’t exaggerate what others are doing.</p><p>This is the one that personally resonated the most. The ability to see past what everyone else is doing and create your own path grants you a level of freedom that is hard to achieve.</p><h3>Liking</h3><p><strong>Example:</strong><br>You’re more likely to buy from someone who is friendly, similar to you, or simply pleasant to be around.</p><p><strong>What it is:</strong><br>We say yes more easily to people we like. Similarity, warmth, and positive energy all boost persuasion.</p><p><strong>How to defend against it:</strong><br>Focus on the message instead of the messenger. Ask yourself if the idea would still make sense coming from someone you dislike.</p><p><strong>How to use it ethically:</strong><br>Build real connection. Find common ground. Authenticity is more persuasive than charm.</p><h3>Authority</h3><p><strong>Example:</strong><br>A person in a lab coat or someone with a title makes a recommendation, and you instinctively follow it.</p><p><strong>What it is:</strong><br>We are conditioned to defer to experts, confident speakers, and people who seem knowledgeable. Authority signals competence, even if it isn’t real.</p><p><strong>How to defend against it:</strong><br>Look for actual proof of expertise. A title or confident tone is not enough.</p><p><strong>How to use it ethically:</strong><br>Lead with transparency. Share your reasoning, show your work, and be honest about what you know and what you don’t.</p><h3>Scarcity</h3><p><strong>Example:</strong><br>“Only two left.” “Sale ends tonight.” You suddenly want the item much more than you did five minutes ago.</p><p><strong>What it is:</strong><br>We value things that seem limited. Scarcity triggers urgency and the fear of missing out.</p><p><strong>How to defend against it:</strong><br>Ask yourself whether you truly want the thing or whether you’re reacting to the idea of losing it.</p><p><strong>How to use it ethically:</strong><br>If something is truly limited, say so. If not, don’t manufacture artificial pressure.</p><h3>Unity</h3><p><strong>Example:</strong><br>You instantly trust someone when you learn they come from your hometown or share your background. Even small similarities create a sense of “us.”</p><p><strong>What it is:</strong><br>Unity is the feeling of shared identity. When someone feels like they are part of your group, their influence increases dramatically.</p><p><strong>How to defend against it:</strong><br>Shared identity does not mean shared goals. Familiarity can cloud judgment.</p><p><strong>How to use it ethically:</strong><br>Build on genuine common ground. Don’t pretend there is unity if it isn’t real.</p><h3>Final Thoughts: Influence Is Everywhere, But So Is Your Power</h3><p>Once you start noticing these principles out in the wild, it becomes hard not to see them. They show up in marketing emails, political campaigns, sales calls, your group chats, even the small interactions you have every day. Influence isn’t rare, it’s the default operating system of human behavior.</p><p>But here’s the part I find most reassuring:</p><p><strong>Being aware of these forces gives you more control, not less.</strong></p><p>Understanding influence doesn’t turn you into a cold, overly guarded skeptic. It makes you:</p><ul><li>slower to react</li><li>faster to reflect</li><li>and better equipped to make decisions that actually match your goals</li></ul><p>And when you use these principles <em>ethically</em>, you also become someone others naturally trust and follow. You become clearer, more compelling, and more persuasive. Not because you’re manipulating anyone, but because you understand how humans think and what they need to feel secure, seen, and supported.</p><p>At its core, Cialdini’s work is a reminder that influence isn’t about tricking people, it’s about understanding them. And understanding yourself.</p><p>If you take away one idea from this book, let it be this:</p><p><strong>The better you understand what moves people, the harder it becomes for anything to move you off your true path.</strong></p><p>Thanks for reading, and stay tuned for next month’s book. If there’s one you want to see covered, send me a message as I love recommendations.</p><p>A <a href="https://a.co/d/6bqdwr5">link</a> to Cialdini’s book:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/344/0*0SiA9-iFbPumVKOS.jpg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=86a44dcd4dc0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Declarative Pipelines: The Next Major Abstraction in Data Engineering?]]></title>
            <link>https://medium.com/@rosenberg.josh34/declarative-pipelines-the-next-major-abstraction-in-data-engineering-4d769d82e622?source=rss-512ea28a2b18------2</link>
            <guid isPermaLink="false">https://medium.com/p/4d769d82e622</guid>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[data-lakehouse]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Josh Rosenberg]]></dc:creator>
            <pubDate>Tue, 21 Oct 2025 15:23:32 GMT</pubDate>
            <atom:updated>2025-10-21T15:23:32.017Z</atom:updated>
            <content:encoded><![CDATA[<p>Stop writing DAGs, start declaring outcomes. The next wave of data engineering is here.</p><p><em>By Josh Rosenberg (Sr. Solutions Engineer @ Databricks)</em></p><h3>From Manual Data Pipelines to Modern Orchestration</h3><p>The history of data engineering is a history of abstraction.</p><p>In the early days, data pipelines were <em>procedural</em>: developers wrote scripts that explicitly told systems what to do and how to do it. Extract this CSV, transform it here, load it there. ETL tools like Informatica and Talend introduced higher-level visual workflows, but the underlying logic was still imperative: you told the computer exactly how to move data, step by step.</p><p>Then came <em>distributed computing. </em>Tools like Hadoop, Spark, and cloud-native warehouses were introduced. The new landscape meant new abstractions for scale. Spark’s DataFrame API let engineers express transformations without worrying about the underlying execution plan. The engine optimized the “how,” freeing engineers to focus more on the “what.”</p><p>That shift, from procedural to <em>declarative</em> intent, has been unfolding for a decade. And now, we’re entering the next stage: <strong>fully declarative data pipelines.</strong></p><h3>What “Declarative” Really Means for Data Engineering</h3><p>In simple terms, declarative programming is about expressing <em>what</em> you want, not <em>how</em> to get it done.</p><p>SQL is an example of a tool that has always been declarative. You tell the database <em>what</em> result you want, and the optimizer figures out the <em>how</em>. But outside of SQL, most data workflows are still stuck in procedural thinking. We define orchestration DAGs, write Spark jobs, manage retries, and specify dependencies. Even modern orchestration tools like Airflow and Dagster remain largely imperative. They are powerful, but verbose.</p><p>Declarative data engineering takes a different stance. Instead of defining the exact execution flow, you define <em>the end state of the data system</em> , the tables, quality expectations, dependencies, and freshness targets, and let the underlying engine plan and maintain that state for you.</p><p>Think of it like driving a car:</p><p>An <strong>imperative</strong> pipeline is a manual transmission: you control every shift, every clutch movement, and every transition between gears. It gives you precision, but it also demands constant attention.</p><p>A <strong>declarative</strong> pipeline is an automatic transmission: you tell it where you want to go, and it handles the mechanics of shifting for you, smoothly and efficiently. You still steer, but you no longer need to manage every motion of the engine.</p><h3>The Value of the Declarative Shift</h3><p>Declarative systems unlock three major benefits:</p><ol><li><strong>Simplicity and speed: </strong>Engineers can express intent in fewer lines of code, focusing on the business logic rather than orchestration glue.</li><li><strong>Reliability through automation: </strong>The system manages dependencies, retries, and re-computation automatically. It understands lineage and can react to data changes intelligently.</li><li><strong>Maintainability at scale: </strong>Declarative pipelines are <em>stateful</em>: they describe what the data estate should look like, making it easy to reason about drift, version control, and reproducibility.</li></ol><p>In other words, declarative data pipelines let teams scale impact without scaling operational complexity.</p><h3>Enter: Spark Declarative Pipelines on Databricks</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qquhBpnGSErUCcMfupWPlQ.png" /><figcaption>Lakeflow Designer on Databricks</figcaption></figure><p><a href="https://www.databricks.com/product/data-engineering/spark-declarative-pipelines"><strong>Spark</strong> <strong>Declarative Pipelines</strong> on Databricks</a> builds on this vision. They bring the same declarative mindset that transformed infrastructure and machine learning to the data engineering layer.</p><p>Instead of writing procedural code to manage ingestion, transformations, and dependencies, you define your tables and transformations as declarative specifications. Spark’s optimizer and the Databricks control plane take care of execution, retries, and incremental updates behind the scenes.</p><p>With <strong>Spark Declarative Pipelines</strong>, you can:</p><ul><li>Define data lineage through table relationships and incremental logic.</li><li>Use SQL, Python, or YAML to describe desired tables, views, and refresh rules.</li><li>Let Databricks handle orchestration, checkpointing, and optimization.</li><li>Build composable, governed pipelines that integrate directly with Unity Catalog.</li><li>Automate CDC and streamline ingestion pipelines.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hHyN2Q-NK4zee-Y6TNSlcQ.png" /></figure><p>This approach combines the flexibility of Spark with the predictability of a managed, declarative workflow engine , and it’s natively integrated into the Lakehouse architecture.</p><h3>A Brief History of Abstraction in Data Engineering</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/709/1*dZb1Kh3-7kmZNxDjcV_RkA.png" /></figure><p>Each leap has followed a familiar pattern: abstraction drives productivity. As platforms mature, the details of <em>how</em> data moves fade away, leaving room for creativity and governance around <em>why</em> it moves.</p><p>Declarative pipelines may well be the next major step, one that unifies ETL, data quality, and orchestration under a single, expressive framework.</p><h3>What This Means for Data Teams</h3><p>As data systems grow more complex, declarative models will become essential to managing them sustainably. Teams can:</p><ul><li>Describe pipelines as <strong>infrastructure-as-code. </strong>Portable, versioned, and testable.</li><li>Focus on <strong>business semantics</strong> instead of orchestration scripts.</li><li>Gain <strong>automatic observability</strong> through lineage and freshness guarantees.</li></ul><p>The result is a more intelligent and adaptive data platform. It’s a platform that’s self-managing, resilient, and ready for the AI-driven era of analytics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HVkQHoTwLh3aAToK" /><figcaption>Creating a Materialized View with Declarative Pipelines</figcaption></figure><h3>Looking Ahead</h3><p>Declarative programming represents the logical next step in the evolution of data engineering. Just as Spark abstracted away the complexity of distributed computation, <strong>Spark Declarative Pipelines</strong> abstract away the complexity of data workflow management.</p><p>It’s the same story: higher-level abstraction, less operational friction, more focus on insight.</p><p>The technology is still evolving, but the direction is clear: toward more automation, more intelligence, and more freedom for humans to focus on higher-value work. The teams that start adopting declarative systems today will be the ones setting the standard tomorrow. This isn’t a passing trend, it’s the next step in data engineering. Better to get ahead of it now than play catch-up later.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4d769d82e622" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Love Is Blind Can Make You Fluent (Seriously)]]></title>
            <link>https://medium.com/@rosenberg.josh34/how-love-is-blind-can-make-you-fluent-seriously-26a5d52b218d?source=rss-512ea28a2b18------2</link>
            <guid isPermaLink="false">https://medium.com/p/26a5d52b218d</guid>
            <category><![CDATA[netflix]]></category>
            <category><![CDATA[comprehensible-input]]></category>
            <category><![CDATA[love-is-blind]]></category>
            <category><![CDATA[language-learning]]></category>
            <category><![CDATA[language]]></category>
            <dc:creator><![CDATA[Josh Rosenberg]]></dc:creator>
            <pubDate>Fri, 17 Oct 2025 19:06:45 GMT</pubDate>
            <atom:updated>2025-10-17T19:06:45.384Z</atom:updated>
            <content:encoded><![CDATA[<p>Why Netflix’s most chaotic dating show might secretly be one of the best tools for learning authentic, real-world language.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1ZeyfKqeU31nMjjWBV1S8A.jpeg" /></figure><p>If you’ve ever tried learning a new language, you’ve probably heard the advice: <em>watch TV in your target language.</em></p><p>And sure, that sounds easy until you turn on a historical drama or a slow-paced thriller and realize that nobody actually talks that way in real life. The dialogue is too perfect, too polished.</p><p>That’s where <em>Love Is Blind</em> comes in.</p><p>The global dating show, where strangers get engaged without ever seeing each other, is available in multiple languages such as English, Japanese, French, Spanish, Brazilian Portuguese, and more. It’s a messy, emotional, completely unscripted social experiment that surprisingly makes it perfect for language learning.</p><p>Here’s why it works better than you’d expect.</p><h3>1. You hear real emotions, not scripted lines</h3><p>When someone on <em>Love Is Blind</em> says, “I didn’t expect to feel this way,” or “I just need you to listen,” you’re hearing spontaneous, emotional language. There’s no rehearsal, no polished take two.</p><p>That’s the kind of speech you actually need in real conversations — the pauses, the hesitations, the emotion behind the words.</p><p>Try watching a tense or romantic scene, writing down a few phrases, and repeating them <em>with</em> the emotion, not just the words. That’s how you internalize tone and rhythm, the true sound of communication.</p><h3>2. It’s full of the little words that make speech sound real</h3><p>Native speakers rely on small connecting words like “you know,” “I mean,” “honestly,” “like.” They rarely show up in textbooks, but they’re essential for sounding natural.</p><p>Every language has its version:</p><ul><li>Spanish: <em>o sea, es que, entonces</em></li><li>French: <em>bah, en fait, tu vois</em></li><li>Italian: <em>cioè, allora, tipo</em></li></ul><p>Listen for them and try repeating sentences as you hear them — this technique, called <em>shadowing</em>, helps your brain pick up rhythm and flow naturally.</p><h3>3. It’s messy, just like real conversation</h3><p>Unlike scripted shows, people on <em>Love Is Blind</em> interrupt each other, lose their train of thought, or change their minds halfway through a sentence.</p><p>That’s exactly what happens in real conversations. When you learn to follow those natural patterns such as pauses, restarts, and misunderstandings, it makes you better prepared for real-world interaction.</p><p>You’ll hear things like:</p><blockquote><em>“No, wait, that’s not what I meant.”<br> “Hold on, let me explain.”<br> “You’re twisting it!”</em></blockquote><p>It’s spontaneous, emotional, and unfiltered. The best kind of listening practice.</p><h3>4. The repetition helps you remember</h3><p>Every season has the same emotional rhythm: attraction, doubt, jealousy, family pressure, forgiveness. That repetition reinforces vocabulary naturally.</p><p>You’ll hear similar words and phrases again and again, but in new contexts. It’s spaced repetition, disguised as drama.</p><p>After a few episodes, you’ll instinctively understand how to express affection, set boundaries, apologize, or talk about trust — without needing to memorize anything.</p><h3>5. It’s entertaining enough to keep you consistent</h3><p>The biggest challenge in language learning isn’t finding resources, it’s staying consistent.</p><p>That’s why <em>Love Is Blind</em> works so well. The show is entertaining. You care about what happens next. You keep watching, which means you keep listening. And because it’s available in so many languages, you can even hop between versions to experience cultural differences. How Brazilians flirt, how the French argue, how the Japanese express hesitation.</p><p>It’s cultural immersion without effort.</p><h3>6. How to turn it into a learning habit</h3><p>Here’s a simple way to get started:</p><ol><li>Choose a <em>Love Is Blind</em> season in your target language.</li><li>Watch each episode twice, first for fun, then for focus.</li><li>Turn on subtitles in the target language (not English).</li><li>Write down natural phrases that show emotion or disagreement.</li><li>Practice saying them out loud or shadowing along.</li><li>Reflect on how <em>you</em> would express the same feeling.</li></ol><p>You’ll start to pick up natural phrasing, rhythm, and even cultural nuance, all while watching something genuinely engaging.</p><h3>The takeaway</h3><p>Language learning isn’t just about grammar and vocabulary. In reality, it’s about human connection.</p><p><em>Love Is Blind</em> may be chaotic, but beneath the drama are real conversations about vulnerability, conflict, and affection. It’s language in its most honest form.</p><p>When you learn through emotion and story, the words stop being something you study — they become something you feel.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=26a5d52b218d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A better way to learn with AI]]></title>
            <link>https://medium.com/@rosenberg.josh34/a-better-way-to-learn-with-ai-a5dd34369646?source=rss-512ea28a2b18------2</link>
            <guid isPermaLink="false">https://medium.com/p/a5dd34369646</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[productivity]]></category>
            <dc:creator><![CDATA[Josh Rosenberg]]></dc:creator>
            <pubDate>Wed, 28 May 2025 14:37:37 GMT</pubDate>
            <atom:updated>2025-05-28T14:37:37.523Z</atom:updated>
            <content:encoded><![CDATA[<p>Most folks I talk to who are using AI to learn are going about it the wrong way.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HrifSaYvcLM7txpGGO3YKw.png" /></figure><p>Don’t get me wrong. Tools like ChatGPT are incredible. But if you’re just treating them like a chattier version of Google and asking questions like “Tell me about X,” you might be missing the real utility.</p><p>Reading or being told something is only the first step. How many times have you read a book or watched a video, only to forget 90 percent of it a week later? That’s because passive learning, where you consume information without really interacting with it, doesn’t lead to long-term understanding.</p><h3>Passive Learning vs. Active Engagement</h3><p>Passive learning is easy. Scroll a few articles, skim a blog post, or binge some YouTube videos. You’ll pick up a few ideas, sure. But unless you actively work with that information, recall it, apply it, or explain it, it drifts off into the recesses of your mind.</p><p>I’ve come to believe this:</p><blockquote><strong>You don’t truly understand something until you can explain it clearly.</strong></blockquote><p>I’ve lost count of the times I thought I understood a concept until I had to explain it to someone else. That’s when the gaps show. You stumble, you search for the words, and you realize you didn’t understand it as deeply as you thought. And honestly, I’ve found that I was the biggest offender of this in my own learning.</p><p>That’s where AI, used right, can transform the way we learn.</p><h3>Using AI as an Active Learning Partner</h3><p>Here’s the trick. Don’t just ask ChatGPT to teach you. Teach it back. Use it like a study buddy who listens patiently, corrects gently, and never gets tired of your rambling explanations.</p><p>Let’s say you’re learning about neural networks. Instead of typing:</p><p>“What is a neural network?”</p><p>Try this:</p><p>“I’m trying to explain what a neural network is. Here’s my attempt: A neural network is a model that tries to mimic the brain by using layers of interconnected nodes to recognize patterns in data. Can you help me improve this explanation and point out what I might be missing?”</p><p>This flips your learning experience from passive to active. You’re forced to recall, synthesize, and articulate the concept, and then get feedback to refine it.</p><h3>Why This Works</h3><p>This approach works because it taps into how our brains actually learn best. There’s a concept called the <a href="https://en.wikipedia.org/wiki/Generation_effect"><strong>generation effect</strong></a>. It shows that people remember information better when they try to generate it themselves rather than just reading or hearing it. Another proven strategy is <a href="https://www.retrievalpractice.org/why-it-works"><strong>retrieval practice</strong></a>, where you recall something from memory instead of re-reading it. Both help strengthen understanding.</p><p>And then there’s the <a href="https://golden.com/wiki/Feynman_technique-XKEGJPA"><strong>Feynman Technique</strong></a>, one of my personal favorites. It’s simple but brutally effective. The idea is that if you can’t explain a concept in plain language, like you’re talking to a curious 12-year-old, then you probably don’t understand it well enough yet. The act of trying to explain something clearly reveals where your gaps are, and forces you to fill them.</p><p>When you use ChatGPT to explain something, you’re doing all of this at once. You’re retrieving the information from memory, generating your own version of it, and testing your ability to teach it. Then you get instant feedback to help close any knowledge gaps. It’s active learning in every sense.</p><h3>What to Avoid</h3><p>There are a few traps to watch out for when using AI to learn. The biggest one is letting the chatbot do all the thinking. If you’re just copying its answers, skimming the output, and nodding along, you’re not learning. You’re consuming.</p><p>Also, avoid asking super vague or shallow questions without following up. If you ask “What is machine learning?” and then move on to the next shiny thing, you’re not building depth. You’re staying at the surface.</p><p>Instead, struggle a bit. Try explaining a concept before you’re ready. Let yourself get it wrong. That moment where you realize what you misunderstood is where learning actually happens.</p><h3>The Benefits Stack Up</h3><p>Once I started using ChatGPT this way, everything changed. I began retaining concepts longer, noticing subtler details, and feeling more confident when discussing complex topics with others.</p><p>It works for anything. Biology, history, economics, programming, philosophy. And it’s especially powerful when you use follow-up prompts like:</p><ul><li><strong>“Can you quiz me on this?”</strong></li><li><strong>“What’s a good analogy for this concept?”</strong></li><li><strong>“What’s a common misunderstanding about this?”</strong></li><li><strong>“Explain this back to me as if I were 10 years old.”</strong></li></ul><h3>TLDR: Be the Teacher</h3><p>Learning isn’t just about input. It’s about output. If you can explain it, you’re learning it. And with tools like ChatGPT, you have a patient tutor and a feedback loop always ready.</p><p>So the next time you sit down to study something new, try flipping the script.</p><p>Don’t just ask the AI to teach you.</p><p>Try teaching it back.</p><p>You might be surprised how much more you retain and how much more fun learning becomes.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a5dd34369646" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>