<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Reader on Medium]]></title>
        <description><![CDATA[Stories by Reader on Medium]]></description>
        <link>https://medium.com/@singhanuj2803?source=rss-874748343425------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*o4y-lnE158gHv0Gw7QmqMg.jpeg</url>
            <title>Stories by Reader on Medium</title>
            <link>https://medium.com/@singhanuj2803?source=rss-874748343425------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 18:34:49 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@singhanuj2803/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Databricks Genie Costs Explained: How to Track Free Credits and Avoid a Surprise Bill]]></title>
            <link>https://medium.com/@singhanuj2803/databricks-genie-costs-explained-how-to-track-free-credits-and-avoid-a-surprise-bill-d6f3068c0f9a?source=rss-874748343425------2</link>
            <guid isPermaLink="false">https://medium.com/p/d6f3068c0f9a</guid>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[cost-optimization]]></category>
            <category><![CDATA[databricks-genie]]></category>
            <category><![CDATA[databricks-sql]]></category>
            <dc:creator><![CDATA[Reader]]></dc:creator>
            <pubDate>Sun, 04 Oct 2026 15:34:30 GMT</pubDate>
            <atom:updated>2026-10-04T15:38:37.400Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A simple, no-jargon guide that turns confusing DBU and SKU terminology into a clear monitoring plan you can actually use.</em></p><h3>Content</h3><ol><li>Why You Need to Understand Genie Usage</li><li>The One-Sentence Message</li><li>Your Meeting Cheat Sheet</li><li>Six Simple Concepts You Must Know</li><li>Your Two-Part Monitoring Plan</li><li>Step-by-Step: Checking Budgets in the UI</li><li>Free vs. Paid — How to Tell the Difference</li><li>The Main SQL Query: Per-User Usage</li><li>A Bonus SQL Query for Daily Trends</li><li>What to Put on Your Dashboard</li></ol><h3>1. Why You Need to Understand Genie Usage</h3><p>If you are a data engineer, <strong>Genie</strong> can feel intimidating at first. It is an AI capability inside Databricks, and every time someone interacts with it, usage is consumed.</p><p>That usage translates into <strong>DBUs</strong> — short for “Databricks Units” — which is simply the way Databricks measures and prices consumption.</p><p>The problem most teams face is not that Genie is hard to use. It is that <strong>costs can appear quietly</strong>. One month you are safely within your free allowance, and the next you see paid DBUs on the bill.</p><p>Your goal is simple:</p><ul><li>Understand how Genie usage works.</li><li>Avoid surprise costs.</li><li>Explain the plan clearly to your team and manager.</li></ul><h3>2. The One-Sentence Message</h3><p>Whenever your manager asks you what the plan is, you can fall back on a single, memorable sentence:</p><blockquote><strong><em>“We will use Databricks Budgets for alerts and the </em></strong><strong><em>system.billing.usage table for detailed, per-user Genie reporting.&quot;</em></strong></blockquote><p>That is your north star. Everything else in this guide supports that one sentence.</p><h3>3. Your Meeting Cheat Sheet</h3><p>When the questions start flying in a meeting, you do not want to fumble. Here are the answers you should memorise.</p><ol><li><strong>Where do I monitor Genie?</strong></li></ol><blockquote>Account Console → <strong>Usage → Budgets</strong> for alerts;</blockquote><blockquote>Databricks SQL → <strong>system.billing.usage</strong> for detail.</blockquote><p><strong>2. How do I know free credits are low?</strong></p><blockquote>Check each user&#39;s GENIE_FREE_USAGE DBUs against the monthly free allowance.</blockquote><p><strong>3. How do I know free credits are finished?</strong></p><blockquote>The user has reached the allowance, and/or paid Genie DBUs appear.</blockquote><p><strong>4. What happens next?</strong></p><blockquote>Genie keeps working, but extra eligible usage becomes <strong>paid</strong> unless a budget blocks it.</blockquote><p><strong>5. What should we alert on?</strong></p><blockquote><strong>80%</strong> = early warning</blockquote><blockquote><strong>90%</strong> = urgent warning</blockquote><blockquote><strong>100%</strong> = limit reached, and <strong>paid DBUs &gt; 0</strong> = cost has started.</blockquote><p><strong>6. What should I show my manager?</strong></p><blockquote>Active users, free DBUs used, people near the limit, paid DBUs, estimated cost, and the usage trend.</blockquote><p><strong>Easy memory rule:</strong> Free usage is the monthly cushion. Paid usage means the cushion has already been used up.</p><h3>4. Six Simple Concepts You Must Know</h3><p>Before you touch a single query, you need to speak the language. Here is the plain-English version of every important term.</p><blockquote><strong>a) Term</strong> <strong>Plain-English meaning</strong> <strong>Genie</strong> A Databricks AI capability that consumes usage when people interact with it.</blockquote><blockquote><strong>b) DBU</strong> Databricks Unit — the unit used to measure consumption and pricing.</blockquote><blockquote><strong>c) SKU</strong> A label for a specific kind of usage. For example, GENIE_FREE_USAGE identifies free Genie consumption.</blockquote><blockquote><strong>d) System table</strong> A Databricks-managed table holding account information, such as billing usage and audit data.</blockquote><blockquote><strong>e) Budget</strong> A monthly cost guardrail. It can notify people and may block extra usage if enforcement is enabled.</blockquote><blockquote><strong>f) Unity Gateway</strong> The governance layer Databricks uses for AI/LLM-related spend controls, including Genie budgets.</blockquote><h3>5. Your Two-Part Monitoring Plan</h3><p>You only need two places to do your job well. Think of them as a fast lane and a detailed lane.</p><p><strong>Place</strong> <strong>Use it for</strong> <strong>Why it matters</strong></p><blockquote><strong>1. Account Console → Usage → Budgets</strong> Fast operational monitoring and alerts It shows estimated month-to-date spending so you can act before costs surprise you.</blockquote><blockquote><strong>2. Databricks SQL → </strong><strong>system.billing.usage</strong> Detailed reporting by user, date, SKU, workspace, and DBUs It is the detailed billing record you use for dashboards and investigation.</blockquote><ul><li>Use <strong>Budgets</strong> when you need a quick health check.</li><li>Use <strong>SQL</strong> when you need to answer “who used what?”</li></ul><h3>6. Step-by-Step: Checking Budgets in the UI</h3><p>Follow these steps whenever you want a fast operational view of your Genie spend.</p><ol><li>Open the <strong>Databricks Account Console</strong>. This is different from working only inside one workspace.</li><li>In the left menu, select <strong>Usage</strong>, then open <strong>Budgets</strong>.</li><li>Open an existing Genie/Unity Gateway budget, or create a new one if you have the required account-admin permission.</li><li>Look for the <strong>current monthly estimated spend</strong>, the amount against thresholds, the budget scope, and the alert recipients.</li><li>Use this page for quick health checks. Use SQL for the detailed “who used what?” questions.</li></ol><blockquote><strong>Important:</strong> If you cannot access budgets or the billing system table, ask the account administrator for the appropriate access. Do not assume that no data means there was no Genie usage.</blockquote><h3>7. Free vs. Paid — How to Tell the Difference</h3><p>The moment you can read this table confidently, the whole topic unlocks for you.</p><p><strong>What I see</strong> <strong>What it means</strong> <strong>What I should say</strong> GENIE_FREE_USAGE is below the monthly allowance The user is still within the free monthly allowance. &quot;This person is still within the free allowance.&quot; GENIE_FREE_USAGE is near the allowance The user may move into paid usage soon. &quot;This person is running low; we should watch their trend.&quot; GENIE_FREE_USAGE has reached the allowance The free allowance is exhausted. &quot;The free allowance is now used up.&quot; Paid Genie DBUs are greater than zero Chargeable Genie usage has started. &quot;Paid Genie consumption has already started.&quot;</p><p><strong>Practical check:</strong> Paid DBUs greater than zero is the <strong>strongest signal</strong> that free usage has been exceeded. Always look for it first.</p><h3>8. The Main SQL Query: Per-User Usage</h3><p>Run this query in Databricks SQL. It gives you a current-month view by user: free DBUs used, estimated free DBUs remaining, percentage used, paid DBUs, key dates, and a simple status message.</p><blockquote><strong>Note:</strong> The 150.00 value below represents the assumed monthly free allowance. Replace it only after your organization confirms its actual figure<em>.</em></blockquote><pre>-- Per-user Genie free and paid DBU usage for the current calendar month.<br>-- Replace 150.00 only after your organization confirms a different free allowance.</pre><pre>WITH genie_monthly_usage AS (<br>  SELECT<br>    COALESCE(<br>      u.identity_metadata.run_as,<br>      u.identity_metadata.created_by,<br>      &#39;Unknown&#39;<br>    ) AS user_name,<br>    SUM(<br>      CASE<br>        WHEN u.sku_name = &#39;GENIE_FREE_USAGE&#39;<br>        THEN u.usage_quantity<br>        ELSE 0<br>      END<br>    ) AS free_dbus_used,<br>    SUM(<br>      CASE<br>        WHEN u.sku_name &lt;&gt; &#39;GENIE_FREE_USAGE&#39;<br>        THEN u.usage_quantity<br>        ELSE 0<br>      END<br>    ) AS paid_dbus_used,<br>    MIN(u.usage_date) AS first_usage_date,<br>    MAX(u.usage_date) AS last_usage_date,<br>    COUNT(*) AS usage_records<br>  FROM system.billing.usage AS u<br>  WHERE u.billing_origin_product = &#39;GENIE&#39;<br>    AND u.usage_date &gt;= DATE_TRUNC(&#39;MONTH&#39;, CURRENT_DATE())<br>    AND u.usage_date &lt; ADD_MONTHS(DATE_TRUNC(&#39;MONTH&#39;, CURRENT_DATE()), 1)<br>  GROUP BY<br>    COALESCE(<br>      u.identity_metadata.run_as,<br>      u.identity_metadata.created_by,<br>      &#39;Unknown&#39;<br>    )<br>)<br>SELECT<br>  user_name,<br>  ROUND(free_dbus_used, 2) AS free_dbus_used,<br>  150.00 AS monthly_free_dbu_allowance,<br>  ROUND(<br>    GREATEST(150.00 - free_dbus_used, 0.00),<br>    2<br>  ) AS estimated_free_dbus_remaining,<br>  ROUND(<br>    LEAST((free_dbus_used / 150.00) * 100, 100.00),<br>    1<br>  ) AS free_allowance_used_pct,<br>  ROUND(paid_dbus_used, 2) AS paid_dbus_used,<br>  first_usage_date,<br>  last_usage_date,<br>  usage_records<br>FROM genie_monthly_usage;</pre><h3>9. A Bonus SQL Query for Daily Trends</h3><p>Run this second query when you want to spot a <strong>sudden spike</strong>. A user may look safe today but still run out before month-end if they are burning through Genie quickly.</p><pre>SELECT<br>  u.usage_date,<br>  COALESCE(<br>    u.identity_metadata.run_as,<br>    u.identity_metadata.created_by,<br>    &#39;Unknown&#39;<br>  ) AS user_name,<br>  ROUND(SUM(<br>    CASE<br>      WHEN u.sku_name = &#39;GENIE_FREE_USAGE&#39;<br>      THEN u.usage_quantity<br>      ELSE 0<br>    END<br>  ), 2) AS free_dbus_used_that_day,<br>  ROUND(SUM(<br>    CASE<br>      WHEN u.sku_name &lt;&gt; &#39;GENIE_FREE_USAGE&#39;<br>      THEN u.usage_quantity<br>      ELSE 0<br>    END<br>  ), 2) AS paid_dbus_used_that_day<br>FROM system.billing.usage AS u<br>WHERE u.billing_origin_product = &#39;GENIE&#39;<br>  AND u.usage_date &gt;= DATE_TRUNC(&#39;MONTH&#39;, CURRENT_DATE())<br>  AND u.usage_date &lt; ADD_MONTHS(DATE_TRUNC(&#39;MONTH&#39;, CURRENT_DATE()), 1)<br>GROUP BY<br>  u.usage_date,<br>  COALESCE(<br>    u.identity_metadata.run_as,<br>    u.identity_metadata.created_by,<br>    &#39;Unknown&#39;<br>  )<br>ORDER BY<br>  u.usage_date DESC,<br>  paid_dbus_used_that_day DESC,<br>  free_dbus_used_that_day DESC;</pre><h3>10. What to Put on Your Dashboard</h3><p>When you build your first report, keep it simple. A clean dashboard beats a complex one every time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/1*oT7c7PKgztvtBirU568ppg.png" /><figcaption>Image</figcaption></figure><blockquote><strong>Pro tip:</strong> Build this as your first <strong>AI/BI dashboard</strong> or <strong>Power BI report</strong> within one to two weeks. It becomes the single source of truth your manager checks every week<em>.</em></blockquote><h3>Important Notes</h3><p>Keep these points in your back pocket — they make you sound sharp in any room.</p><ul><li><strong>Free usage is a monthly cushion.</strong> As soon as you see paid DBUs greater than zero, the cushion has been used up. Treat that as your clear, unmistakable signal.</li><li><strong>Budgets and SQL serve different jobs.</strong> Use Budgets for fast alerts and health checks; use system.billing.usage when you need the detailed &quot;who, when, and how much&quot; answer.</li><li><strong>Never assume silence equals zero usage.</strong> If you cannot see budgets or the billing table, ask the account admin for access. “No data” might just mean “no access.”</li><li><strong>The </strong><strong>150.00 allowance is a placeholder.</strong> Confirm your organization&#39;s actual monthly free allowance before you publish any report. Do not guess.</li><li><strong>Set alerts before you need them.</strong> Configure <strong>80%</strong> (early warning), <strong>90%</strong> (urgent warning), and <strong>100%</strong> (limit reached) notifications so costs never sneak up on you.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d6f3068c0f9a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Secure Azure Databricks CLI Authentication on Windows]]></title>
            <link>https://medium.com/@singhanuj2803/secure-azure-databricks-cli-authentication-on-windows-089ba8209b72?source=rss-874748343425------2</link>
            <guid isPermaLink="false">https://medium.com/p/089ba8209b72</guid>
            <category><![CDATA[databricks-cli]]></category>
            <dc:creator><![CDATA[Reader]]></dc:creator>
            <pubDate>Thu, 10 Sep 2026 02:18:34 GMT</pubDate>
            <atom:updated>2026-09-10T02:18:35.775Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><strong><em>A practical guide to OAuth, Windows Credential Manager, profiles, and secure configuration</em></strong></blockquote><h3>Introduction</h3><p>The Databricks CLI requires authentication before you can run account-level or workspace-level commands. On Windows, the recommended approach is to use OAuth-based authentication and allow the CLI to store tokens securely in <strong>Windows Credential Manager</strong> rather than exposing them in configuration files.[<a href="https://ppl-ai-file-upload.s3.amazonaws.com/web/direct-files/attachments/67217492/e29bd94e-5af3-48e4-9975-2aaab6c70d12/Authentication-for-the-Databricks-CLI-Azure-Databricks-_-Microsoft-Learn.pdf?AWSAccessKeyId=ASIA2F3EMEYE62XV3XGI&amp;Signature=0YQtDYVPT45ly1esKgzeWUf3DKw%3D&amp;x-amz-security-token=IQoJb3JpZ2luX2VjEHUaCXVzLWVhc3QtMSJHMEUCIA%2Fj51BuZGZyRwH2Rv07jLgKI2DHI0pRLDaNFerFuBaqAiEAsXUexb4WFjN6TGVsqTmKcY7xKDysyLI9kZWlfR0NpDYq8wQIPRABGgw2OTk3NTMzMDk3MDUiDAOw%2BvgomabT%2B%2F84LCrQBI1kajVIrrvmdj6G81tJ82bpUBjbTVU4cjXspSM13K0FdSSTYM9dysnnuplw5rIu9QTJ8l7LQBSUtby0PaX31h%2FCZQKeNKYcdb%2FZM20TB%2FUESSyxQsz4oG51%2BIrCLy4uh2wEla1KIK%2BtQ%2FMnMJzkKUR3mqjnoHOpuOJqUe8OMWro%2B5zQ7jaR6zYJS5IGHZEf8p1XUpQWAxQw7C59VEXmCskZm9nYzfgiTw5pxQmbQuvlOb3SWXCF3QhpuUXeBSFUoTS7rjaYxYztsvtTe%2BCChrmbxNvbBCoGzR8ciKxWp4obVHZRSAhI0Zi%2BPXIOFWTiSvh0jnb6MqaoUeVY%2FswOy%2FPBg7Bx3UvLQrVt35q9Imo9SjfxMOpIIoE5mvx8zqs62waMzww1k72liGMjvtgP3KLRbXQRADGRiUmdY1LHulkhte4Nxbg7cXRbZjuH%2F6lLeXq0AepS8F2oKXzk9m9xel%2Fcj8l1Ypc%2B3vdpGmHTh%2FDXpoGeXKwz%2BuaRtI1rykLpd%2BW6h5SdN7Wau%2BUugqA6No3gFnKtJI1pdGVF8kyqz5rLaMOxa4xTjqnM8a%2BCh4AstCz8LgonV64g6r6sbmDS%2BdHP6Zz70u8AnSk8vVYOYC2ABuEnXoGksZCPy%2BosMM2%2Bz20fCpDUuXTCHTmCo7bweSitGglazLa%2BPkELTlq%2Faq99Bo1nQ0ZHu6VjBaHtIgjz0FXvvV81ogGvvZMau9rWNnsfuE%2FxAK9h7HOBLD1s%2BfbVxUJclvVpM6KroI8M8hCuAWFsHOgl7Nux2JtrWVylBWgwveX61AY6mAEcZsMWJDhvQ0gO7L73USfMPNhbtzScjyi%2FMPmkmQDXUS9r1r71JJTJdVl1YGPEpm5PyZqY0o7srzGI9kHRbdsTMUpTFzniE%2FhWh0ipvM33Djq%2BIy37DtNTWGsPKO8ZQM6ODjp0ahF1g6d7k6U1FhHEoqPpeakFGbk4qelgQG6loMqgdBny8rFCovd05Sf9%2Faa8SgfA6hyinA%3D%3D&amp;Expires=1788788880">ppl-ai-file-upload.s3.amazonaws</a>]</p><p>This guide extracts only the Windows-relevant information from the Microsoft Learn document and converts it into a Medium-ready article.</p><h3>Why Authentication Matters</h3><p>Authentication determines how the Databricks CLI proves its identity when accessing Azure Databricks resources.</p><p>You may need authentication for:</p><ul><li>Workspace-level commands, such as listing clusters.</li><li>Account-level commands, such as listing account groups.</li><li>Local development from a Windows laptop.</li><li>Automation scripts and CI/CD pipelines.</li><li>Azure virtual machines running the Databricks CLI.</li></ul><p>Example commands:</p><pre>databricks clusters list</pre><pre>databricks account groups list</pre><p>The authentication method should match the way you use the CLI:</p><ul><li><strong>Interactive user work:</strong> OAuth U2M.</li><li><strong>Automation and applications:</strong> OAuth M2M.</li><li><strong>Azure VM workloads:</strong> Azure managed identity.</li><li><strong>Microsoft Entra service principal workflows:</strong> Microsoft Entra ID service principal authentication.</li><li><strong>Existing legacy setups:</strong> Personal access tokens.</li></ul><h3>Recommended Option: OAuth U2M</h3><p>OAuth user-to-machine authentication is usually the most convenient option for a Windows developer.</p><p>With OAuth U2M:</p><ul><li>You sign in interactively through a browser.</li><li>The CLI manages short-lived OAuth tokens.</li><li>Tokens expire in under an hour.</li><li>The risk associated with accidentally exposed long-lived tokens is reduced.</li><li>The CLI creates or updates a local configuration profile.</li></ul><h3>Workspace Login</h3><p>Open <strong>PowerShell</strong> or <strong>Windows Terminal</strong> and run:</p><pre>databricks auth login --host &lt;workspace-url&gt;</pre><p>Replace &lt;workspace-url&gt; with your Azure Databricks workspace URL.</p><p>Example:</p><pre>databricks auth login --host https://adb-1234567890123456.7.azuredatabricks.net</pre><h3>Account Login</h3><p>For account-level commands, run:</p><pre>databricks auth login `<br>  --host &lt;account-console-url&gt; `<br>  --account-id &lt;account-id&gt;</pre><p>Example:</p><pre>databricks auth login `<br>  --host https://accounts.azuredatabricks.net `<br>  --account-id &lt;account-id&gt;</pre><p>The CLI opens a browser-based sign-in flow. After authentication, you can accept the suggested profile name or provide your own.</p><h3>Windows Credential Manager</h3><p>Starting with Databricks CLI version 1.0.0, OAuth U2M tokens are stored in operating-system-native secure storage.</p><p>On Windows, this storage is <strong>Credential Manager</strong>. The .databrickscfg file stores only non-secret information, such as the workspace host and profile name; the OAuth token itself is not written to that file</p><p>This is safer than storing credentials directly in a text file.</p><h3>Windows Security Benefits</h3><ul><li>Tokens are protected by Windows-native credential storage.</li><li>Secrets are not exposed in .databrickscfg.</li><li>Each profile has its own cached token.</li><li>Logging out of one profile does not automatically log out other profiles pointing to the same workspace.</li><li>Short-lived OAuth tokens reduce the impact of accidental exposure.</li></ul><h3>Working with Profiles</h3><p>A profile stores the connection details and authentication configuration for a Databricks account or workspace.</p><p>To list available profiles, run:</p><pre>databricks auth profiles -o text</pre><p>To use a specific profile:</p><pre>databricks clusters list --profile &lt;profile-name&gt;</pre><p>The shorter form is:</p><pre>databricks clusters list -p &lt;profile-name&gt;</pre><p>For account-level commands:</p><pre>databricks account groups list -p &lt;profile-name&gt;</pre><p>In PowerShell, you can often press <strong>Tab</strong> after -p or --profile to display available profile names.[<a href="https://ppl-ai-file-upload.s3.amazonaws.com/web/direct-files/attachments/67217492/e29bd94e-5af3-48e4-9975-2aaab6c70d12/Authentication-for-the-Databricks-CLI-Azure-Databricks-_-Microsoft-Learn.pdf?AWSAccessKeyId=ASIA2F3EMEYE62XV3XGI&amp;Signature=0YQtDYVPT45ly1esKgzeWUf3DKw%3D&amp;x-amz-security-token=IQoJb3JpZ2luX2VjEHUaCXVzLWVhc3QtMSJHMEUCIA%2Fj51BuZGZyRwH2Rv07jLgKI2DHI0pRLDaNFerFuBaqAiEAsXUexb4WFjN6TGVsqTmKcY7xKDysyLI9kZWlfR0NpDYq8wQIPRABGgw2OTk3NTMzMDk3MDUiDAOw%2BvgomabT%2B%2F84LCrQBI1kajVIrrvmdj6G81tJ82bpUBjbTVU4cjXspSM13K0FdSSTYM9dysnnuplw5rIu9QTJ8l7LQBSUtby0PaX31h%2FCZQKeNKYcdb%2FZM20TB%2FUESSyxQsz4oG51%2BIrCLy4uh2wEla1KIK%2BtQ%2FMnMJzkKUR3mqjnoHOpuOJqUe8OMWro%2B5zQ7jaR6zYJS5IGHZEf8p1XUpQWAxQw7C59VEXmCskZm9nYzfgiTw5pxQmbQuvlOb3SWXCF3QhpuUXeBSFUoTS7rjaYxYztsvtTe%2BCChrmbxNvbBCoGzR8ciKxWp4obVHZRSAhI0Zi%2BPXIOFWTiSvh0jnb6MqaoUeVY%2FswOy%2FPBg7Bx3UvLQrVt35q9Imo9SjfxMOpIIoE5mvx8zqs62waMzww1k72liGMjvtgP3KLRbXQRADGRiUmdY1LHulkhte4Nxbg7cXRbZjuH%2F6lLeXq0AepS8F2oKXzk9m9xel%2Fcj8l1Ypc%2B3vdpGmHTh%2FDXpoGeXKwz%2BuaRtI1rykLpd%2BW6h5SdN7Wau%2BUugqA6No3gFnKtJI1pdGVF8kyqz5rLaMOxa4xTjqnM8a%2BCh4AstCz8LgonV64g6r6sbmDS%2BdHP6Zz70u8AnSk8vVYOYC2ABuEnXoGksZCPy%2BosMM2%2Bz20fCpDUuXTCHTmCo7bweSitGglazLa%2BPkELTlq%2Faq99Bo1nQ0ZHu6VjBaHtIgjz0FXvvV81ogGvvZMau9rWNnsfuE%2FxAK9h7HOBLD1s%2BfbVxUJclvVpM6KroI8M8hCuAWFsHOgl7Nux2JtrWVylBWgwveX61AY6mAEcZsMWJDhvQ0gO7L73USfMPNhbtzScjyi%2FMPmkmQDXUS9r1r71JJTJdVl1YGPEpm5PyZqY0o7srzGI9kHRbdsTMUpTFzniE%2FhWh0ipvM33Djq%2BIy37DtNTWGsPKO8ZQM6ODjp0ahF1g6d7k6U1FhHEoqPpeakFGbk4qelgQG6loMqgdBny8rFCovd05Sf9%2Faa8SgfA6hyinA%3D%3D&amp;Expires=1788788880">ppl-ai-file-upload.s3.amazonaws</a>]</p><h3>OAuth M2M for Windows Automation</h3><p>OAuth machine-to-machine authentication is designed for scripts, services, and applications that must access Azure Databricks without an interactive user login.</p><p>It uses:</p><ul><li>A service principal.</li><li>An OAuth client ID.</li><li>An OAuth client secret.</li><li>The OAuth client credential flow.</li></ul><h3>Account-Level Profile</h3><p>A Windows .databrickscfg profile for account-level access contains:</p><pre>[account-automation]<br>host = &lt;account-console-url&gt;<br>account_id = &lt;account-id&gt;<br>client_id = &lt;service-principal-client-id&gt;<br>client_secret = &lt;service-principal-oauth-secret&gt;</pre><h3>Workspace-Level Profile</h3><p>For workspace-level automation:</p><pre>[workspace-automation]<br>host = &lt;workspace-url&gt;<br>client_id = &lt;service-principal-client-id&gt;<br>client_secret = &lt;service-principal-oauth-secret&gt;</pre><p>Use the profile from PowerShell:</p><pre>databricks clusters list -p workspace-automation</pre><p>Avoid placing client secrets directly in source code, Git repositories, scripts committed to version control, or shared documents.</p><h3>Plaintext Storage Fallback</h3><p>If Windows Credential Manager cannot be used — for example, in a restricted or headless environment — you can configure the Databricks CLI to use plaintext JSON token storage.</p><p>Set the environment variable in PowerShell:</p><pre>$env:DATABRICKS_AUTH_STORAGE = &quot;plaintext&quot;</pre><p>Then run the login command again:</p><pre>databricks auth login --host &lt;workspace-url&gt;</pre><p>You can also configure plaintext storage in .databrickscfg:</p><pre>[__settings__]<br>auth_storage = plaintext</pre><p>The environment variable takes precedence over the configuration-file setting. Plaintext storage should be treated as a fallback because credential files may be readable by other users or processes on the machine.</p><h3>Azure CLI Authentication on Windows</h3><p>Azure CLI authentication allows the Databricks CLI to use the identity authenticated through Azure CLI.</p><p>First, sign in:</p><pre>az login</pre><p>If you work with multiple Azure tenants or subscriptions, you can select the appropriate subscription:</p><pre>az account set --subscription &quot;&lt;subscription-name-or-id&gt;&quot;</pre><p>You can then configure a Databricks profile with the required host and account or workspace settings and use it with the -p option. The Microsoft Learn document identifies Azure CLI authentication as one of the supported authentication approaches.[<a href="https://ppl-ai-file-upload.s3.amazonaws.com/web/direct-files/attachments/67217492/e29bd94e-5af3-48e4-9975-2aaab6c70d12/Authentication-for-the-Databricks-CLI-Azure-Databricks-_-Microsoft-Learn.pdf?AWSAccessKeyId=ASIA2F3EMEYE62XV3XGI&amp;Signature=0YQtDYVPT45ly1esKgzeWUf3DKw%3D&amp;x-amz-security-token=IQoJb3JpZ2luX2VjEHUaCXVzLWVhc3QtMSJHMEUCIA%2Fj51BuZGZyRwH2Rv07jLgKI2DHI0pRLDaNFerFuBaqAiEAsXUexb4WFjN6TGVsqTmKcY7xKDysyLI9kZWlfR0NpDYq8wQIPRABGgw2OTk3NTMzMDk3MDUiDAOw%2BvgomabT%2B%2F84LCrQBI1kajVIrrvmdj6G81tJ82bpUBjbTVU4cjXspSM13K0FdSSTYM9dysnnuplw5rIu9QTJ8l7LQBSUtby0PaX31h%2FCZQKeNKYcdb%2FZM20TB%2FUESSyxQsz4oG51%2BIrCLy4uh2wEla1KIK%2BtQ%2FMnMJzkKUR3mqjnoHOpuOJqUe8OMWro%2B5zQ7jaR6zYJS5IGHZEf8p1XUpQWAxQw7C59VEXmCskZm9nYzfgiTw5pxQmbQuvlOb3SWXCF3QhpuUXeBSFUoTS7rjaYxYztsvtTe%2BCChrmbxNvbBCoGzR8ciKxWp4obVHZRSAhI0Zi%2BPXIOFWTiSvh0jnb6MqaoUeVY%2FswOy%2FPBg7Bx3UvLQrVt35q9Imo9SjfxMOpIIoE5mvx8zqs62waMzww1k72liGMjvtgP3KLRbXQRADGRiUmdY1LHulkhte4Nxbg7cXRbZjuH%2F6lLeXq0AepS8F2oKXzk9m9xel%2Fcj8l1Ypc%2B3vdpGmHTh%2FDXpoGeXKwz%2BuaRtI1rykLpd%2BW6h5SdN7Wau%2BUugqA6No3gFnKtJI1pdGVF8kyqz5rLaMOxa4xTjqnM8a%2BCh4AstCz8LgonV64g6r6sbmDS%2BdHP6Zz70u8AnSk8vVYOYC2ABuEnXoGksZCPy%2BosMM2%2Bz20fCpDUuXTCHTmCo7bweSitGglazLa%2BPkELTlq%2Faq99Bo1nQ0ZHu6VjBaHtIgjz0FXvvV81ogGvvZMau9rWNnsfuE%2FxAK9h7HOBLD1s%2BfbVxUJclvVpM6KroI8M8hCuAWFsHOgl7Nux2JtrWVylBWgwveX61AY6mAEcZsMWJDhvQ0gO7L73USfMPNhbtzScjyi%2FMPmkmQDXUS9r1r71JJTJdVl1YGPEpm5PyZqY0o7srzGI9kHRbdsTMUpTFzniE%2FhWh0ipvM33Djq%2BIy37DtNTWGsPKO8ZQM6ODjp0ahF1g6d7k6U1FhHEoqPpeakFGbk4qelgQG6loMqgdBny8rFCovd05Sf9%2Faa8SgfA6hyinA%3D%3D&amp;Expires=1788788880">ppl-ai-file-upload.s3.amazonaws</a>]</p><h3>Microsoft Entra Service Principal Authentication</h3><p>Microsoft Entra ID service principal authentication is useful for Windows-based scripts and automation that require a non-interactive identity.</p><p>A workspace-level profile can include:</p><pre>[entra-automation]<br>host = &lt;workspace-url&gt;<br>azure_tenant_id = &lt;tenant-id&gt;<br>azure_client_id = &lt;application-id&gt;<br>azure_client_secret = &lt;client-secret&gt;</pre><p>An account-level profile additionally includes:</p><pre>account_id = &lt;account-id&gt;</pre><p>Use it with:</p><pre>databricks clusters list -p entra-automation</pre><p>The service principal must have the necessary permissions in the relevant Azure Databricks workspace or account. Keep the client secret protected and rotate it according to your organization’s security policy.</p><h3>Authentication Precedence</h3><p>When the Databricks CLI searches for authentication settings, it evaluates configuration sources in this order:</p><ol><li>Bundle settings files, when running inside a Databricks bundle directory.</li><li>Environment variables.</li><li>Profiles in .databrickscfg.</li></ol><p>Once the CLI finds the required setting, it stops searching other locations.</p><p>This behavior is important on Windows because an environment variable can override a profile without visibly changing .databrickscfg.</p><p>For example, if this environment variable exists:</p><pre>$env:DATABRICKS_TOKEN</pre><p>the CLI uses that token even if other tokens are present in .databrickscfg.</p><p>To inspect whether the variable is set:</p><pre>Get-ChildItem Env:DATABRICKS_TOKEN</pre><p>To remove it from the current PowerShell session:</p><pre>Remove-Item Env:DATABRICKS_TOKEN</pre><h3>Choosing the Right Method</h3><p>ScenarioRecommended methodWindows-specific considerationLocal interactive developmentOAuth U2MTokens are stored in Credential ManagerScheduled scriptsOAuth M2MProtect the service-principal secretCI/CD pipelineOAuth M2M or service principalUse the pipeline secret storeAzure VM workloadManaged identityAssign the identity appropriate Azure Databricks permissionsExisting legacy implementationPATPrefer OAuth where possibleAzure administrator workflowAzure CLI authenticationRun az login before Databricks CLI commands</p><p>Databricks recommends OAuth instead of personal access tokens where possible because OAuth provides stronger security for user authentication.</p><h3>Important Notes</h3><ul><li>Install the Databricks CLI before configuring authentication.</li><li>Decide whether your commands target an account or a workspace.</li><li>Use OAuth U2M for normal Windows user activity.</li><li>Use OAuth M2M or a service principal for automation.</li><li>Windows stores OAuth U2M tokens in Credential Manager by default.</li><li>The .databrickscfg file should not contain unnecessary secrets.</li><li>Use -p or --profile to select the intended authentication profile.</li><li>Check environment variables when the CLI appears to ignore a profile.</li><li>Treat plaintext storage as a fallback, not the default.</li><li>Avoid committing .databrickscfg or secret-containing files to Git.</li><li>Use least-privilege permissions for service principals and managed identities.</li><li>Prefer the workspace host and explicitly assign the identity to the workspace where possible.[<a href="https://ppl-ai-file-upload.s3.amazonaws.com/web/direct-files/attachments/67217492/e29bd94e-5af3-48e4-9975-2aaab6c70d12/Authentication-for-the-Databricks-CLI-Azure-Databricks-_-Microsoft-Learn.pdf?AWSAccessKeyId=ASIA2F3EMEYE62XV3XGI&amp;Signature=0YQtDYVPT45ly1esKgzeWUf3DKw%3D&amp;x-amz-security-token=IQoJb3JpZ2luX2VjEHUaCXVzLWVhc3QtMSJHMEUCIA%2Fj51BuZGZyRwH2Rv07jLgKI2DHI0pRLDaNFerFuBaqAiEAsXUexb4WFjN6TGVsqTmKcY7xKDysyLI9kZWlfR0NpDYq8wQIPRABGgw2OTk3NTMzMDk3MDUiDAOw%2BvgomabT%2B%2F84LCrQBI1kajVIrrvmdj6G81tJ82bpUBjbTVU4cjXspSM13K0FdSSTYM9dysnnuplw5rIu9QTJ8l7LQBSUtby0PaX31h%2FCZQKeNKYcdb%2FZM20TB%2FUESSyxQsz4oG51%2BIrCLy4uh2wEla1KIK%2BtQ%2FMnMJzkKUR3mqjnoHOpuOJqUe8OMWro%2B5zQ7jaR6zYJS5IGHZEf8p1XUpQWAxQw7C59VEXmCskZm9nYzfgiTw5pxQmbQuvlOb3SWXCF3QhpuUXeBSFUoTS7rjaYxYztsvtTe%2BCChrmbxNvbBCoGzR8ciKxWp4obVHZRSAhI0Zi%2BPXIOFWTiSvh0jnb6MqaoUeVY%2FswOy%2FPBg7Bx3UvLQrVt35q9Imo9SjfxMOpIIoE5mvx8zqs62waMzww1k72liGMjvtgP3KLRbXQRADGRiUmdY1LHulkhte4Nxbg7cXRbZjuH%2F6lLeXq0AepS8F2oKXzk9m9xel%2Fcj8l1Ypc%2B3vdpGmHTh%2FDXpoGeXKwz%2BuaRtI1rykLpd%2BW6h5SdN7Wau%2BUugqA6No3gFnKtJI1pdGVF8kyqz5rLaMOxa4xTjqnM8a%2BCh4AstCz8LgonV64g6r6sbmDS%2BdHP6Zz70u8AnSk8vVYOYC2ABuEnXoGksZCPy%2BosMM2%2Bz20fCpDUuXTCHTmCo7bweSitGglazLa%2BPkELTlq%2Faq99Bo1nQ0ZHu6VjBaHtIgjz0FXvvV81ogGvvZMau9rWNnsfuE%2FxAK9h7HOBLD1s%2BfbVxUJclvVpM6KroI8M8hCuAWFsHOgl7Nux2JtrWVylBWgwveX61AY6mAEcZsMWJDhvQ0gO7L73USfMPNhbtzScjyi%2FMPmkmQDXUS9r1r71JJTJdVl1YGPEpm5PyZqY0o7srzGI9kHRbdsTMUpTFzniE%2FhWh0ipvM33Djq%2BIy37DtNTWGsPKO8ZQM6ODjp0ahF1g6d7k6U1FhHEoqPpeakFGbk4qelgQG6loMqgdBny8rFCovd05Sf9%2Faa8SgfA6hyinA%3D%3D&amp;Expires=1788788880">ppl-ai-file-upload.s3.amazonaws</a>]</li></ul><h3>Action Items</h3><h3>For Windows Developers</h3><ol><li>Open PowerShell or Windows Terminal.</li><li>Confirm that the Databricks CLI is installed:</li></ol><pre>databricks -h</pre><ol><li>Log in interactively:</li></ol><pre>databricks auth login --host &lt;workspace-url&gt;</pre><ol><li>Complete the browser-based authentication flow.</li><li>List profiles:</li></ol><pre>databricks auth profiles -o text</pre><ol><li>Test the selected profile:</li></ol><pre>databricks clusters list -p &lt;profile-name&gt;</pre><h3>For Automation Engineers</h3><ol><li>Create or identify a Microsoft Entra service principal.</li><li>Grant it the required Azure Databricks permissions.</li><li>Configure OAuth M2M or Microsoft Entra service principal authentication.</li><li>Store secrets in a secure secret manager or CI/CD secret store.</li><li>Avoid hardcoding secrets in PowerShell scripts.</li><li>Test the connection with a non-destructive CLI command.</li><li>Rotate secrets and review permissions periodically.</li></ol><h3>For Troubleshooting</h3><ol><li>Confirm that the workspace URL is correct.</li><li>Verify that the account ID is included for account-level commands.</li><li>Check the active profile with:</li></ol><pre>databricks auth profiles -o text</pre><ol><li>Check whether DATABRICKS_TOKEN is overriding your profile.</li><li>Confirm that the signed-in user or service principal has sufficient permissions.</li><li>Check Windows Credential Manager if OAuth token storage fails.</li><li>Use plaintext storage only when secure storage is unavailable.</li></ol><h3>Final Takeaway</h3><p>For most Windows users, the cleanest Azure Databricks CLI setup is <strong>OAuth U2M with Windows Credential Manager</strong>. Use OAuth M2M, Microsoft Entra service principals, or managed identities when the CLI runs unattended, and always verify which profile or environment variable is supplying the credentials</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=089ba8209b72" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Installing the Databricks CLI on Windows: Four Paths, One Working Terminal]]></title>
            <link>https://medium.com/@singhanuj2803/installing-the-databricks-cli-on-windows-four-paths-one-working-terminal-d6a3c1bb8e65?source=rss-874748343425------2</link>
            <guid isPermaLink="false">https://medium.com/p/d6a3c1bb8e65</guid>
            <category><![CDATA[azure-databricks]]></category>
            <category><![CDATA[databricks-cli]]></category>
            <dc:creator><![CDATA[Reader]]></dc:creator>
            <pubDate>Mon, 07 Sep 2026 09:34:04 GMT</pubDate>
            <atom:updated>2026-09-07T09:34:04.060Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A no-fluff walkthrough for getting </em><em>databricks -v to return 0.205.0 or above — whichever install method fits your setup.</em></p><h3>Why This Matters to You</h3><p>If you’re working with Azure Databricks from a Windows machine, the CLI is your bridge between your terminal and your workspace — bundles, jobs, clusters, secrets, all scriptable without opening the browser. Microsoft’s official install page covers macOS, Linux, and Windows together, which makes it easy to get lost hunting for the parts that actually apply to your OS.</p><p>You’re going to walk through exactly four ways to get the Databricks CLI running on Windows, plus how to keep it current once it’s there. No macOS Homebrew steps, no Linux-only detours — just what you need.</p><h3>Method 1: WinGet (the fastest route)</h3><p>If you’re on a reasonably modern Windows build, WinGet is already sitting on your machine, and it’s the path of least resistance.</p><ul><li>Open Command Prompt and search for the package first, then install it:</li></ul><pre>winget search databricks<br>winget install Databricks.DatabricksCLI</pre><ul><li>Restart your Command Prompt after installation — this matters, because the updated PATH won’t reflect in your current session.</li><li>Confirm the install worked:</li></ul><pre>databricks -v<br># Or:<br>databricks version</pre><ul><li>You’re looking for <strong>0.205.0 or above</strong>. Anything below that means something went sideways.</li></ul><p>This is the method you should default to unless you have a specific reason not to.</p><h3>Method 2: Chocolatey (experimental)</h3><p>Chocolatey is Windows’ other major package manager, but Microsoft explicitly flags this path as <strong>Experimental</strong> — worth knowing before you commit to it for a production setup.</p><ul><li>Install with a single command:</li></ul><pre>choco install databricks-cli</pre><ul><li>Verify the same way as before:</li></ul><pre>databricks -v</pre><ul><li>If you ever need to move to a newer release later, you’ll use choco upgrade databricks-cli (covered below).</li></ul><p>Use this if your team already standardizes on Chocolatey for tooling — otherwise, WinGet is the steadier choice.</p><h3>Method 3: curl via WSL</h3><p>This one’s for you if you’re comfortable in a Linux-style shell or your workflow already leans on WSL.</p><p>A few things to know before you start:</p><ul><li>Microsoft recommends <strong>Windows Subsystem for Linux (WSL)</strong> specifically for this method on Windows.</li><li>You’ll need unzip installed — check your OS documentation if it&#39;s missing.</li><li>If WSL isn’t an option for you, skip straight to the source build method below instead.</li></ul><p>The install command itself is case-sensitive, so copy it exactly:</p><pre>curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sh</pre><p>A couple of practical notes:</p><ul><li>Run this as an <strong>administrator</strong> on Windows.</li><li>It installs to C:\Windows\databricks.exe.</li><li>If you get a “Target path already exists” error, that means a previous install is still sitting there — delete it manually, then re-run the command.</li></ul><h3>Method 4: Source Build (manual .zip)</h3><p>No package manager, no WSL — just you, a browser, and a zip file. This is your fallback when nothing else fits.</p><p>Here’s the process:</p><ol><li>Head to the Databricks CLI repository’s Releases section on GitHub and grab the release matching your architecture:</li></ol><ul><li>databricks_cli_X.Y.Z_windows_386.zip → Windows, Intel 32-bit</li><li>databricks_cli_X.Y.Z_windows_amd64.zip → Windows, Intel 64-bit</li><li>databricks_cli_X.Y.Z_windows_arm64.zip → Windows, ARM 64-bit</li></ul><p>2. Not sure which architecture you’re on? Run this in Command Prompt or PowerShell:</p><ul><li>echo %PROCESSOR_ARCHITECTURE%</li></ul><p>3. Extract the zip. Inside, you’ll find the CLI executable — leave it in place or move it wherever you keep your tools.</p><p>4. Confirm the version:</p><ul><li>databricks -v</li></ul><p>This method also gives you access to pre-release builds if you want to experiment with upcoming features early.</p><h3>Keeping It Updated on Windows</h3><p>Once you’re set up, staying current is just as easy — and it mirrors whichever install method you used.</p><ul><li><strong>WinGet users:</strong></li><li>winget upgrade Databricks.DatabricksCLI</li><li><strong>Chocolatey users:</strong></li><li>choco upgrade databricks-cli</li><li><strong>curl/WSL users:</strong> delete the existing install at C:\Windows\databricks.exe, then re-run the same install command from Method 3.</li><li><strong>Source build users:</strong> optionally delete the old executable and extracted folder, then repeat the download-and-extract steps from Method 4.</li></ul><p>After any update, run databricks -v again to confirm you&#39;re on the version you expect.</p><h3>Important Notes</h3><ul><li><strong>Version floor matters:</strong> every method here assumes you’re targeting 0.205.0 or above — that’s the threshold Microsoft’s documentation uses throughout.</li><li><strong>Chocolatey is experimental</strong> for both install and update — treat it accordingly if you’re setting up shared or production tooling.</li><li><strong>WSL is the recommended path for curl on Windows</strong> — if WSL isn’t available to you, don’t force it; go straight to the source build instead.</li><li><strong>The curl install script is case-sensitive.</strong> A stray lowercase letter in the flags will break it.</li><li><strong>“Target path already exists” is a known snag</strong> with the curl method — it just means you need to clear out the old binary first.</li><li><strong>Authentication is a separate step.</strong> Every method above ends the same way: once the CLI is installed, you still need to configure authentication before you can actually talk to your workspace.</li></ul><h3>Action Items</h3><ul><li>Decide which method fits your environment: WinGet for simplicity, Chocolatey if your team already uses it, curl+WSL if you’re shell-comfortable, or source build if you want full manual control.</li><li>[Run the install command for your chosen method.</li><li>Restart your terminal session if you used WinGet.</li><li>Verify with databricks -v — confirm you&#39;re at 0.205.0 or above.</li><li>Set a reminder to periodically re-run your method’s upgrade command so you don’t drift onto a stale version.</li><li>Move on to configuring authentication for the CLI — that’s the next step before you can run any real commands against your workspace.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d6a3c1bb8e65" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Commit to Production: How You Can Automate Databricks Deployments with CI/CD Asset Bundles]]></title>
            <link>https://medium.com/@singhanuj2803/from-commit-to-production-how-you-can-automate-databricks-deployments-with-ci-cd-asset-bundles-b2d4833e3731?source=rss-874748343425------2</link>
            <guid isPermaLink="false">https://medium.com/p/b2d4833e3731</guid>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[databricks-asset-bundles]]></category>
            <category><![CDATA[github-actions]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Reader]]></dc:creator>
            <pubDate>Mon, 07 Sep 2026 09:14:54 GMT</pubDate>
            <atom:updated>2026-09-07T09:14:54.095Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A hands-on walkthrough of building a complete, gated pipeline — from your laptop to a live production workspace</em></p><h3>Content Index</h3><ul><li>Why This Matters to You</li><li>Setting Up Your Databricks Asset Bundle Project</li><li>Developing Locally and Getting Under Version Control</li><li>The Continuous Integration (CI) Pipeline</li><li>Merging and Triggering Continuous Delivery</li><li>The Approval Gate and Production Deployment</li><li>Reading the Pipeline: Left Side vs. Right Side</li><li>Key Notes You’ll Want to Remember</li><li>Action Items — Your Next Build</li><li>Topics</li></ul><h3>Why This Matters to You</h3><p>If you’ve ever manually deployed a Databricks notebook to production and held your breath while doing it, this post is for you. What you’re about to walk through is a complete CI/CD workflow for Databricks Asset Bundles — one that takes your code all the way from local development on your own machine to a fully automated, gated deployment in production.</p><p>By the end, you’ll understand exactly how a single bundle — your notebooks, libraries, cluster settings, and scripts — moves through initialization, version control, automated testing, a human approval checkpoint, and finally deployment, all orchestrated by GitHub Actions.</p><h3>Setting Up Your Databricks Asset Bundle Project</h3><p>Everything starts with initializing a new Databricks Asset Bundle project. This one step generates two things for you:</p><ul><li>A starter manifest</li><li>A folder layout</li></ul><p>From there, you edit that manifest to define exactly what your workflow needs:</p><ul><li>Notebooks</li><li>Libraries</li><li>Cluster settings</li><li>Scripts</li></ul><p>Think of the manifest as the single source of truth for your project — everything downstream in this pipeline traces back to what you define here.</p><h3>Developing Locally and Getting Under Version Control</h3><p>Once your manifest is configured, you bring your project under version control by pushing your bundle to a remote GitHub repository, straight from your local development environment.</p><p>From that point forward, you have real flexibility:</p><ul><li>You can keep developing code and making changes.</li><li>You can run unit tests locally before anything touches a shared environment.</li><li>You can optionally deploy the bundle to your own personal development workspace, so you can verify end-to-end logic without stepping on other developers’ work.</li></ul><p>When you’re happy with where things stand, you commit your changes and push them to the remote repository. That push is the trigger — it’s what kicks off the entire CI/CD process.</p><h3>The Continuous Integration (CI) Pipeline</h3><p>The moment you push to the remote GitHub repository, the continuous integration pipeline fires — automated end-to-end using GitHub Actions. Here’s what it does for you, automatically:</p><ol><li>Checks out your code</li><li>Installs dependencies</li><li>Runs your unit tests</li></ol><p>If anything breaks during this phase, the process stops immediately and flags the exact broken assertion — you don’t have to go hunting for it. A successful CI build is your signal that your code meets the project’s quality gates and is genuinely ready to move forward.</p><h3>Merging and Triggering Continuous Delivery</h3><p>Once CI passes and you’re satisfied with your changes, the next step is yours to initiate: you submit a request to merge everything into the main branch — the branch that holds your stable, tested code.</p><p>The moment that merge completes, the continuous delivery pipeline picks up automatically. Its first task deploys the version-controlled bundle to the test environment, which exists specifically for integration and data quality checks under conditions that closely mirror production.</p><h3>The Approval Gate and Production Deployment</h3><p>Here’s where the workflow deliberately slows down. After deployment to the test environment, the pipeline pauses at a protected approval gate — and it stays paused until a stakeholder actively gives the green light.</p><p>Once that approval comes through:</p><ul><li>The pipeline resumes on its own.</li><li>The next task deploys the identical bundle to production.</li></ul><p>That word “identical” matters. Nothing gets rebuilt or modified between test and production — what you approved is exactly what ships.</p><h3>Reading the Pipeline: Left Side vs. Right Side</h3><p>When you’re looking at the pipeline visualization in practice, orient yourself with a simple rule:</p><ul><li><strong>Left side of the screen</strong> → continuous integration</li><li><strong>Right side of the screen</strong> → continuous delivery</li></ul><p>Once this mental map clicks, scanning any pipeline run becomes much faster — you’ll instantly know which phase you’re looking at.</p><h3>Key Notes You’ll Want to Remember</h3><ul><li>Every component — from notebooks to cluster definitions — travels the exact same scripted path, every single time.</li><li>Everything is version-controlled in Git, automatically built and tested, and deployed unchanged from test to production.</li><li>CI and CD are triggered by different events: CI fires on every push; CD fires only after a merge into the main branch.</li><li>The approval gate is non-negotiable — production deployment cannot happen without explicit stakeholder sign-off.</li><li>The test environment exists to catch integration and data quality problems before they ever reach production, under conditions that represent the real thing.</li></ul><h3>Action Items — Your Next Build</h3><ol><li>Initialize a new Databricks Asset Bundle project and review the generated manifest and folder layout.</li><li>Edit the manifest to define your notebooks, libraries, cluster settings, and scripts.</li><li>Push your bundle to a remote GitHub repository to bring it under version control.</li><li>Develop locally, run unit tests, and optionally deploy to your personal development workspace to verify logic in isolation.</li><li>Commit and push your changes to trigger the CI pipeline, and confirm it passes all quality gates.</li><li>Submit a merge request into the main branch once CI succeeds — this is what triggers continuous delivery.</li><li>Monitor the deployment to the test environment and confirm integration and data quality checks pass.</li><li>Coordinate with your stakeholder to clear the approval gate before production deployment.</li><li>Verify that the identical, tested bundle reaches production without modification.</li></ol><p>Once you’ve run through this once, you’ll have real, hands-on experience with a complete CI/CD workflow for Databricks Asset Bundles — one you can apply to any Databricks project going forward.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b2d4833e3731" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[One Cell at a Time: Mastering Remote Execution with Databricks Connect]]></title>
            <link>https://medium.com/@singhanuj2803/one-cell-at-a-time-mastering-remote-execution-with-databricks-connect-5eb449378640?source=rss-874748343425------2</link>
            <guid isPermaLink="false">https://medium.com/p/5eb449378640</guid>
            <category><![CDATA[pyspark]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[vscode-extension]]></category>
            <category><![CDATA[vscode]]></category>
            <category><![CDATA[azure-databricks]]></category>
            <dc:creator><![CDATA[Reader]]></dc:creator>
            <pubDate>Sat, 05 Sep 2026 15:09:24 GMT</pubDate>
            <atom:updated>2026-09-05T15:09:24.819Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RQwti3oFLVIcJ-Q58w_2Mg.png" /></figure><h3>Your Notebook, Their Cluster: How Databricks Connect Bridges Local Code and Remote Power</h3><p><em>A hands-on walkthrough for running individual cells and scripts on a live remote cluster — without wrapping everything into a workflow first</em></p><h3>Content Index</h3><ul><li>Why This Matters to You</li><li>Setting Up a Virtual Environment</li><li>Installing Databricks Connect</li><li>Activating and Deactivating Your Environment</li><li>Selecting the Correct Kernel in Your Notebook</li><li>Running Notebook Cells on the Remote Cluster</li><li>Running Python Scripts via Databricks Connect</li><li>Key Notes You’ll Want to Remember</li><li>Action Items — Your Next 15 Minutes</li><li>Topics</li></ul><h3>Why This Matters to You</h3><p>If you’ve been working with the Databricks extension in VS Code, you’ve probably hit this wall already: you can connect to a remote cluster just fine, but the moment you try to run a single notebook cell on its own — instead of the whole notebook as a workflow — it fails. You get an error, because that cell was never actually reaching the remote cluster in the first place.</p><p>That’s frustrating when you’re iterating on code and just want to test one thing at a time.</p><p>The fix is a package called <strong>Databricks Connect</strong>. Once it’s installed and wired up correctly, you can run individual cells — and even standalone Python scripts — directly against your remote cluster, without wrapping everything into a full workflow first. Let’s walk through exactly how you set that up.</p><h3>Setting Up a Virtual Environment</h3><p>Here’s the first thing you need to know: you can’t just install Databricks Connect globally. If you do, it will conflict with any local PySpark installation you already have sitting on your machine. So the very first step is creating a dedicated virtual environment for this.</p><p>A couple of things to get right before you run any commands:</p><ul><li>Pick a Python version that matches your Databricks runtime version. In this walkthrough, that’s Python 3.11 — check your own cluster’s runtime before you proceed.</li><li>You can name your environment anything you like. Here, it’s called venv_dbc.</li></ul><p>On a Mac, you’d type:</p><pre>python3.11 -m venv venv_dbc</pre><p>On Windows, the equivalent is:</p><pre>py -3.11 -m venv venv_dbc</pre><p>Once you run that, you’ll see the virtual environment folder appear in your project.</p><h3>Installing Databricks Connect</h3><p>This next part matters more than it might seem: the version of Databricks Connect you install needs to match your cluster’s runtime version exactly. If your cluster is on runtime 15.4, you install Databricks Connect 15.4 — not the latest version, not whatever’s convenient.</p><pre>pip install databricks-connect==15.4</pre><p>Once that finishes, confirm it landed correctly:</p><pre>pip list</pre><p>You should see Databricks Connect listed at the version you just installed.</p><h3>Activating and Deactivating Your Environment</h3><p>Now that Databricks Connect is installed, you’ll be switching your virtual environment on and off as you work. Here’s how you do that on each platform.</p><p><strong>On Mac:</strong></p><pre>source venv_dbc/bin/activate</pre><p><strong>On Windows:</strong></p><pre>venv_dbc\Scripts\activate</pre><p><em>(If you’re using PowerShell, you may need to append </em><em>ps1 to that command.)</em></p><p><strong>To deactivate, on either OS:</strong></p><pre>deactivate</pre><p>You can toggle this as often as you need — deactivating and reactivating won’t touch any of the packages you’ve already installed.</p><h3>Selecting the Correct Kernel in Your Notebook</h3><p>Here’s the step that trips people up: activating your virtual environment in the terminal is <em>not</em> the same as telling your notebook to use it. You have to do both.</p><ul><li>Open your notebook’s kernel selector and choose your virtual environment (venv_dbc) from the list.</li><li>If it asks you to restart, go ahead. If it prompts you to install the ipykernel package, accept that too.</li><li>If your cluster happens to be shut down at this point, restart it and wait until it’s fully running before you try executing anything.</li></ul><p>Once your kernel is set correctly and your cluster is live, you’re ready to actually run something on it.</p><h3>Running Notebook Cells on the Remote Cluster</h3><p>With your virtual environment activated <em>and</em> selected as your notebook’s kernel, this is where it clicks into place: individual notebook cells now execute directly on your remote Databricks cluster. No more errors, no more needing to package everything as a full workflow just to test a line of code.</p><p>This is the core loop you’ll use for day-to-day iterative development — write a cell, run it against the remote cluster, see the result, adjust, repeat.</p><h3>Running Python Scripts via Databricks Connect</h3><p>Notebooks aren’t the only thing Databricks Connect supports — you can run standalone .py scripts against your remote cluster too. Here&#39;s the flow:</p><ol><li>Create a new folder for your scripts (for example, scripts) and add a file — say, temp.py.</li><li>Write your code. Even something as simple as a spark.sql query works to test the connection.</li><li>Save the file, then click the Databricks icon in VS Code and select <strong>“Run Current File with Databricks Connect.”</strong></li><li>If you’re prompted to select a Python environment, choose the virtual environment you created earlier.</li><li>Want to double-check or change your interpreter manually? Open the <strong>Command Palette</strong>, search for <strong>“Python: [interpreter selection]”</strong>, and pick your virtual environment from there.</li><li>Run the file. You’ll see the output appear in your terminal — confirmation that the execution actually happened on your remote cluster, not locally.</li></ol><h3>Key Notes You’ll Want to Remember</h3><ul><li>Databricks Connect must live inside its own dedicated virtual environment — installing it globally will conflict with local PySpark.</li><li>Match your Databricks Connect version to your cluster’s Databricks Runtime version exactly.</li><li>Terminal activation and notebook kernel selection are two separate steps. You need both, every time.</li><li>The same virtual environment interpreter also needs to be selected when you’re running standalone scripts, not just notebooks.</li><li>None of this works if your cluster isn’t running — check that first when something fails.</li></ul><h3>Action Items — Your Next 15 Minutes</h3><ol><li>Check your Databricks cluster’s runtime version before you install anything.</li><li>Create a virtual environment using a matching Python version.</li><li>Install the matching version of databricks-connect inside that environment.</li><li>Activate the environment, then confirm the install with pip list.</li><li>Select your virtual environment as your notebook’s kernel — restart and install ipykernel if it asks.</li><li>Make sure your remote cluster is actually running before you test anything.</li><li>For script-based work, select the same virtual environment as your Python interpreter via the Command Palette, then run using “Run Current File with Databricks Connect.”</li></ol><p>Once you’ve worked through these steps, you’ll have a setup where you can freely iterate — cell by cell, or script by script — directly against your remote Databricks cluster, without the friction of packaging everything into a workflow first.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5eb449378640" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mastering Spark Memory Management in Azure Databricks: Executor Memory, Unified Memory, and…]]></title>
            <link>https://medium.com/@singhanuj2803/mastering-spark-memory-management-in-azure-databricks-executor-memory-unified-memory-and-8373521d895b?source=rss-874748343425------2</link>
            <guid isPermaLink="false">https://medium.com/p/8373521d895b</guid>
            <category><![CDATA[databricks-certification]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[databricks-basics]]></category>
            <category><![CDATA[azure-databricks]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Reader]]></dc:creator>
            <pubDate>Thu, 09 Jul 2026 04:15:32 GMT</pubDate>
            <atom:updated>2026-07-09T04:15:32.273Z</atom:updated>
            <content:encoded><![CDATA[<h3>Mastering Spark Memory Management in Azure Databricks: Executor Memory, Unified Memory, and Off-Heap Optimization</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*UDaPLNxU-kenxwv9ielIVA.jpeg" /></figure><h3>1. Introduction to Spark Memory Management</h3><p>Understanding <strong>memory management in Apache Spark</strong> is essential for optimizing performance and troubleshooting bottlenecks in distributed data processing systems.</p><p>Spark workloads rely heavily on how memory is structured and allocated within <strong>executors</strong>. When developers understand how Spark organizes executor memory internally, they can make more effective decisions about caching strategies, join operations, shuffle performance, and garbage collection behavior.</p><p>Key reasons memory management matters:</p><ul><li>Prevents <strong>OutOfMemory errors</strong></li><li>Reduces <strong>garbage collection pauses</strong></li><li>Improves <strong>query execution performance</strong></li><li>Helps optimize <strong>large-scale data pipelines</strong></li></ul><p>A Spark executor contains several memory regions. The most important region is <strong>on-heap memory</strong>, which is managed by the <strong>Java Virtual Machine (JVM)</strong>.</p><p>Most Spark operations run here, including:</p><ul><li>Joins</li><li>Shuffles</li><li>Sorting</li><li>Aggregations</li></ul><p>Executor memory is further divided into multiple logical sections that control how Spark uses memory during computation.</p><h3>2. Spark Executor Memory Layout</h3><p>Spark divides executor memory into several components to balance computation, caching, and system overhead.</p><h3>Major Memory Components</h3><p>Spark executors contain three primary memory categories:</p><h3>1. On-Heap Memory</h3><ul><li>Managed by the <strong>JVM</strong></li><li>Used for most Spark operations</li><li>Contains execution memory, storage memory, user memory, and reserved memory</li></ul><h3>2. Off-Heap Memory</h3><ul><li>Managed directly by the <strong>operating system</strong></li><li>Optional and <strong>disabled by default</strong></li><li>Used to reduce JVM garbage collection overhead</li></ul><h3>3. Overhead Memory</h3><ul><li>Used for <strong>internal Spark operations</strong></li><li>Includes network buffers, JVM metadata, and other system processes</li></ul><h3>On-Heap Memory Sections</h3><p>On-heap memory is logically divided into four sections.</p><h3>Execution Memory</h3><p>Used for computational operations such as:</p><ul><li>Joins</li><li>Shuffles</li><li>Sorting</li><li>Aggregations</li></ul><p>These operations require temporary data structures during execution.</p><h3>Storage Memory</h3><p>Storage memory supports caching.</p><p>Used for:</p><ul><li>Cached <strong>RDDs</strong></li><li>Cached <strong>DataFrames</strong></li><li><strong>Broadcast variables</strong></li></ul><h3>User Memory</h3><p>Used to store user-defined objects such as:</p><ul><li>Variables</li><li>Lists</li><li>Sets</li><li>Dictionaries</li><li>User-Defined Functions (UDFs)</li></ul><h3>Reserved Memory</h3><p>Reserved memory is allocated for <strong>internal Spark operations</strong> and cannot be used by the application</p><h3>3. Executor Memory Calculation Example</h3><p>Consider the following configuration:</p><pre>spark.executor.memory = 10 GB</pre><p>Spark uses memory fractions to divide executor memory.</p><h3>Unified Memory Calculation</h3><p>Configuration:</p><pre>spark.memory.fraction = 0.6</pre><p>Unified memory equals <strong>60% of executor memory</strong>.</p><p>Calculation:</p><pre>10 GB × 0.6 = 6 GB</pre><p>This 6 GB becomes the <strong>unified memory pool</strong>.</p><h3>Storage Memory Calculation</h3><p>Configuration:</p><pre>spark.memory.storageFraction = 0.5</pre><p>Storage memory equals <strong>50% of unified memory</strong>.</p><p>Calculation:</p><pre>6 GB × 0.5 = 3 GB</pre><p>Therefore:</p><ul><li>Storage Memory = <strong>3 GB</strong></li><li>Execution Memory = <strong>3 GB</strong></li></ul><h3>Remaining Memory Breakdown</h3><p>Total executor memory = <strong>10 GB</strong></p><p>Unified memory = <strong>6 GB</strong></p><p>Remaining memory = <strong>4 GB</strong></p><p>Breakdown:</p><ul><li>User Memory = <strong>3.7 GB</strong></li><li>Reserved Memory = <strong>300 MB</strong></li></ul><p>This structure ensures Spark always retains memory for system-level tasks.</p><h3>4. Overhead Memory Explained</h3><p>Executor memory applies only to <strong>on-heap memory</strong>.</p><p>Spark also allocates <strong>overhead memory</strong>, which is calculated separately.</p><p>Formula:</p><p>The overhead memory is the <strong>maximum of</strong>:</p><ul><li><strong>384 MB</strong>, or</li><li><strong>10% of executor memory</strong></li></ul><p>Example:</p><p>Executor memory = <strong>10 GB</strong></p><p>10% of 10 GB = <strong>1 GB</strong></p><p>Therefore:</p><pre>Overhead Memory = 1 GB</pre><p>Total memory requested from the cluster manager:</p><pre>Executor Memory + Overhead Memory</pre><pre>10 GB + 1 GB = 11 GB</pre><p>This memory is requested from cluster managers such as:</p><ul><li>YARN</li><li>Kubernetes</li><li>Standalone Spark clusters</li></ul><h3>5. Unified Memory Management</h3><p>Execution memory and storage memory together form <strong>unified memory</strong>.</p><p>Unified memory introduces <strong>dynamic memory allocation</strong>.</p><p>Instead of fixed boundaries, Spark dynamically adjusts memory usage depending on workload requirements.</p><h3>Dynamic Memory Behavior</h3><p>Key principles:</p><ul><li>Execution memory can borrow unused storage memory.</li><li>Storage memory can use unused execution memory.</li><li>Execution memory always has <strong>higher priority</strong>.</li></ul><p>This design prevents idle memory from being wasted.</p><h3>Memory Behavior Scenarios</h3><h3>Scenario 1 — Storage Has Free Memory</h3><p>If execution requires additional memory:</p><ul><li>Execution borrows unused storage memory.</li></ul><p>This improves computational performance.</p><h3>Scenario 2 — Storage Contains Cached Data</h3><p>If execution needs more memory:</p><ul><li>Spark evicts cached storage blocks.</li><li>The eviction policy used is <strong>LRU (Least Recently Used)</strong>.</li></ul><h3>Scenario 3 — Storage Requires More Memory</h3><p>If storage needs more space:</p><ul><li>It cannot evict execution memory.</li><li>Instead, it must evict its own cached blocks.</li></ul><p>Execution memory remains protected.</p><h3>6. Spark Memory Before and After Version 1.6</h3><p>Spark memory management changed significantly after version <strong>1.6</strong>.</p><h3>Before Spark 1.6</h3><p>Memory was divided into <strong>fixed partitions</strong>.</p><p>Problems included:</p><ul><li>Execution memory could not borrow storage memory.</li><li>Storage memory remained unused in many workloads.</li><li>This caused <strong>inefficient memory utilization</strong>.</li></ul><h3>Spark 1.6 and Later</h3><p>Spark introduced <strong>Unified Memory Management</strong>.</p><p>Key improvements:</p><ul><li>Execution and storage share dynamic boundaries.</li><li>Memory adapts to workload demands.</li><li>Overall memory utilization improves significantly.</li></ul><p>This design remains the standard in modern Spark versions.</p><h3>7. Off-Heap Memory in Spark</h3><p>Off-heap memory exists outside the JVM heap.</p><p>Instead of relying on JVM memory management, it uses <strong>native memory controlled by the operating system</strong>.</p><h3>Key Characteristics</h3><ul><li>Managed by the <strong>operating system</strong></li><li>Not subject to <strong>JVM garbage collection</strong></li><li>Reduces <strong>GC pauses</strong></li><li>Helps stabilize memory usage in large workloads</li></ul><h3>Trade-Offs</h3><h3>Advantages</h3><ul><li>Reduces garbage collection overhead</li><li>Improves performance in memory-intensive workloads</li><li>Helps avoid large JVM heap pressure</li></ul><h3>Limitations</h3><ul><li>Developers must manage memory manually</li><li>Risk of <strong>memory leaks</strong></li><li>Slower than on-heap memory</li></ul><p>However, off-heap memory is still <strong>faster than disk spilling</strong>.</p><h3>8. Enabling Off-Heap Memory</h3><p>To enable off-heap memory in Spark:</p><pre>spark.memory.offHeap.enabled = true</pre><p>Then define the memory size:</p><pre>spark.memory.offHeap.size</pre><p>Recommended starting point:</p><ul><li>Allocate <strong>10% — 20% of executor memory</strong></li></ul><p>Example:</p><p>Executor memory = <strong>10 GB</strong></p><p>Recommended off-heap memory:</p><pre>1 GB – 2 GB</pre><h3>9. Important Notes</h3><p>Key insights every Spark developer should remember:</p><ul><li>Most Spark operations run in <strong>JVM on-heap memory</strong>.</li><li>Execution memory handles <strong>joins, shuffles, sorting, and aggregations</strong>.</li><li>Storage memory stores <strong>cached datasets and broadcast variables</strong>.</li><li>Unified memory enables <strong>dynamic sharing between execution and storage memory</strong>.</li><li>Execution memory always receives <strong>higher priority</strong>.</li><li>Cached blocks are evicted using the <strong>LRU algorithm</strong>.</li><li>Excessive caching wastes valuable storage memory.</li><li>Off-heap memory reduces <strong>garbage collection overhead</strong>.</li><li>Spark requests cluster memory based on:</li><li>Executor memory</li><li>Overhead memory</li><li>Optional off-heap memory</li></ul><h3>10. Action Items</h3><h3>1. Understand Executor Memory Layout</h3><p>Review how Spark divides memory into execution, storage, user, and reserved regions.</p><h3>2. Avoid Unnecessary Caching</h3><p>Cache datasets only when they are reused multiple times.</p><h3>3. Monitor Execution Memory Usage</h3><p>Track joins, shuffles, and aggregations since they consume the most memory.</p><h3>4. Tune Memory Fractions When Necessary</h3><p>Adjust Spark configurations:</p><pre>spark.memory.fraction<br>spark.memory.storageFraction</pre><h3>5. Monitor Garbage Collection Behavior</h3><p>Frequent GC pauses may indicate excessive JVM heap usage.</p><h3>6. Experiment with Off-Heap Memory</h3><p>Enable it cautiously for memory-intensive workloads.</p><h3>7. Use the Spark UI for Diagnostics</h3><p>Inspect executor memory usage, caching behavior, and shuffle performance.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8373521d895b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Spark DAG in Azure Databricks: Reading Files, Transformations, Joins, and…]]></title>
            <link>https://medium.com/@singhanuj2803/understanding-spark-dag-in-azure-databricks-reading-files-transformations-joins-and-0333a35c8a6b?source=rss-874748343425------2</link>
            <guid isPermaLink="false">https://medium.com/p/0333a35c8a6b</guid>
            <dc:creator><![CDATA[Reader]]></dc:creator>
            <pubDate>Tue, 07 Jul 2026 20:40:33 GMT</pubDate>
            <atom:updated>2026-07-07T20:40:33.202Z</atom:updated>
            <content:encoded><![CDATA[<h3>Understanding Spark DAG in Azure Databricks: Reading Files, Transformations, Joins, and Aggregations</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z3JBO_rZIQGLNjPnV9iZTg.png" /></figure><h3>Introduction</h3><p>Apache Spark processes large-scale data using a <strong>Directed Acyclic Graph (DAG)</strong> execution model. Instead of executing operations sequentially like traditional systems, Spark builds a DAG that represents all transformations and actions required to produce a result.</p><p>Understanding the Spark DAG is critical because it reveals:</p><ul><li>How Spark <strong>breaks jobs into stages and tasks</strong></li><li>When <strong>data shuffling occurs</strong></li><li>Which operations are <strong>expensive or optimized</strong></li><li>How <strong>Spark UI visualizes execution plans</strong></li></ul><p>When working in <strong>Azure Databricks</strong>, the Spark UI and query plans make it possible to analyze this DAG and understand exactly how Spark executes operations such as file reads, transformations, joins, and aggregations.</p><h3>How Spark Builds a DAG</h3><p>Spark constructs a DAG whenever a job is executed. This DAG represents the <strong>logical and physical execution plan</strong> for a computation.</p><h3>Key Characteristics of a Spark DAG</h3><ul><li>A DAG represents <strong>all transformations and actions</strong> in a workflow.</li><li>Nodes represent <strong>operations</strong>, such as filters or joins.</li><li>Edges represent <strong>data dependencies</strong> between operations.</li><li>The DAG is divided into <strong>stages</strong>, separated by shuffle boundaries.</li></ul><h3>Example Execution Flow</h3><p>A typical Spark workflow may look like:</p><ol><li>Read a dataset from storage.</li><li>Apply transformations such as filters or column operations.</li><li>Perform joins between datasets.</li><li>Execute aggregation operations.</li><li>Trigger an action such as .show() or .collect().</li></ol><p>Spark translates this workflow into <strong>jobs → stages → tasks</strong>.</p><h3>Reading Files and Initial DAG Creation</h3><p>When Spark reads a dataset (for example, a <strong>Parquet file</strong>), the process begins with <strong>metadata discovery</strong>.</p><h3>What Metadata Includes</h3><ul><li>Number of partitions</li><li>Dataset size</li><li>Number of columns</li><li>Column data types</li><li>File locations</li></ul><p>Even though metadata retrieval is <strong>not an action</strong>, Spark still launches a <strong>job</strong> to gather this information.</p><h3>Why Metadata Matters</h3><p>Metadata helps Spark determine:</p><ul><li>How to <strong>distribute tasks across executors</strong></li><li>How many <strong>partitions to process</strong></li><li>How to <strong>optimize query execution</strong></li></ul><h3>Example .show() Execution</h3><p>When .show() is executed:</p><ul><li>Spark creates another job.</li><li>Only the required rows are retrieved.</li><li>Operations in the DAG may include:</li><li>Parquet file scan</li><li>Columnar-to-row conversion</li><li>Collect limit operation</li></ul><p>This ensures efficient data retrieval.</p><h3>Understanding Data Batches vs Partitions</h3><p>Spark processes data using <strong>batches</strong>, not individual rows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/733/1*AkTMqfQSG1a641-mO3f_fQ.png" /></figure><h3>How Spark Uses Batches</h3><p>When .show() requests a few rows:</p><ul><li>Spark <strong>does not scan all partitions</strong></li><li>Only <strong>minimal batches are processed</strong></li><li>Computation and cluster resources are reduced</li></ul><p>This design significantly improves <strong>performance and efficiency</strong>.</p><h3>Narrow Transformations in Spark</h3><p>Narrow transformations occur when <strong>each output partition depends on exactly one input partition</strong>.</p><h3>Characteristics</h3><ul><li>No shuffle required</li><li>Executed within a <strong>single stage</strong></li><li>Faster and more efficient</li></ul><h3>Common Narrow Transformations</h3><p>Examples include:</p><ul><li>Filtering rows</li><li>Modifying columns</li><li>Creating new columns</li><li>Selecting specific columns</li></ul><h3>Example Workflow</h3><p>A typical narrow transformation pipeline might include:</p><ul><li>Filtering records by city</li><li>Splitting a name column into first and last name</li><li>Modifying an existing column</li><li>Selecting relevant columns</li></ul><p>In the DAG, these operations appear as:</p><ul><li><strong>Filter</strong></li><li><strong>Projection</strong></li></ul><p>Because there is no shuffle, Spark executes them in <strong>one stage</strong>.</p><h3>Wide Transformations and Join Operations</h3><p>Wide transformations require <strong>data movement across partitions</strong>, which introduces <strong>shuffle operations</strong>.</p><h3>Why Shuffle Happens</h3><p>For operations like joins, Spark must ensure that:</p><ul><li>Rows with the <strong>same key are in the same partition</strong></li></ul><p>This requires redistributing data across executors.</p><h3>Example: Sort-Merge Join</h3><p>A sort-merge join involves multiple steps:</p><ol><li>Read both datasets</li><li>Shuffle data by join key</li><li>Sort partitions</li><li>Perform the join</li></ol><p>This creates <strong>multiple stages</strong> in the DAG.</p><p>Spark may also include additional steps, such as:</p><ul><li>Filtering null values in join keys</li><li>Sorting partitions before merging</li></ul><h3>Broadcast Join Optimization</h3><p>Spark can optimize joins using <strong>broadcast joins</strong> when one dataset is significantly smaller.</p><h3>How Broadcast Join Works</h3><ol><li>The smaller dataset is <strong>broadcast to all executors</strong></li><li>Each executor performs the join locally</li><li>The large dataset <strong>does not shuffle</strong></li></ol><h3>Benefits</h3><ul><li>Eliminates expensive shuffle operations</li><li>Reduces the number of stages</li><li>Improves overall job performance</li></ul><h3>Typical Use Case</h3><p>Broadcast joins work best when:</p><ul><li>One dataset is <strong>small enough to fit in memory</strong></li><li>The other dataset is <strong>large</strong></li></ul><h3>Aggregation Operations and Shuffle Behavior</h3><p>Aggregation operations such as groupBy() are <strong>wide transformations</strong>.</p><p>This is because records must be <strong>grouped by a key across partitions</strong>.</p><h3>Spark Aggregation Workflow</h3><p>Spark performs aggregation in two phases:</p><p><strong>Phase 1 — Partial Aggregation</strong></p><ul><li>Executed locally within partitions</li><li>Reduces the amount of data before shuffle</li></ul><p><strong>Phase 2 — Final Aggregation</strong></p><ul><li>Data is shuffled</li><li>Records with the same key are grouped together</li><li>Final aggregation is computed</li></ul><h3>Adaptive Query Execution (AQE)</h3><p>Spark can further optimize execution using <strong>AQE</strong>, which can:</p><ul><li>Reduce shuffle partitions</li><li>Handle data skew</li><li>Adjust execution strategies dynamically</li></ul><h3>Count Distinct and Multi-Stage DAG Execution</h3><p>The countDistinct operation is more complex than simple aggregations.</p><p>It requires multiple steps to ensure correctness across distributed partitions.</p><h3>Execution Steps</h3><ol><li><strong>Local distinct operation</strong></li><li><strong>Shuffle to group identical key combinations</strong></li><li><strong>Another distinct step across partitions</strong></li><li><strong>Partial aggregation</strong></li><li><strong>Second shuffle</strong></li><li><strong>Final aggregation</strong></li></ol><p>Because of multiple shuffle boundaries, the DAG contains <strong>more stages</strong> than simple aggregations.</p><h3>Important Notes</h3><h3>Key Concepts to Remember</h3><ul><li>Spark DAG represents the <strong>complete execution flow</strong> of transformations and actions.</li><li>Metadata reading may <strong>trigger a Spark job</strong> even if it is not an action.</li><li>Spark stores data in <strong>columnar format</strong> but may convert it to <strong>row format</strong> during execution.</li><li>.show() retrieves only the <strong>required data batches</strong>, not the entire dataset.</li><li><strong>Batches and partitions are different concepts</strong> in Spark execution.</li><li><strong>Narrow transformations</strong> do not require shuffle operations.</li><li><strong>Wide transformations</strong> require shuffles and create multiple stages.</li><li>Join operations commonly trigger <strong>shuffle boundaries</strong>.</li><li><strong>Broadcast joins</strong> eliminate shuffle when one dataset is small.</li><li>Aggregations use <strong>partial aggregation before shuffle</strong>.</li><li><strong>Adaptive Query Execution (AQE)</strong> dynamically optimizes execution.</li><li>countDistinct introduces <strong>multiple shuffle stages</strong>.</li></ul><h3>Action Items for Practicing Spark DAG Analysis</h3><h3>Hands-On Learning Steps</h3><ol><li>Practice analyzing <strong>Spark DAGs in the Spark UI</strong> when running jobs in Azure Databricks.</li><li>Run .show() on a dataset and observe <strong>minimal batch processing</strong>.</li><li>Execute <strong>narrow transformations</strong> such as filters and column modifications.</li><li>Verify that narrow transformations create <strong>only one stage</strong>.</li><li>Perform joins on large datasets and observe <strong>shuffle stages</strong>.</li><li>Enable and disable broadcast joins using:</li></ol><pre>spark.sql.autoBroadcastJoinThreshold</pre><p>7. Compare DAG structures between:</p><ul><li>Sort-merge joins</li><li>Broadcast joins</li></ul><p>8. Run aggregation queries such as:</p><pre>groupBy().count()</pre><p>and analyze <strong>partial vs final aggregation stages</strong>.</p><ol><li>Execute <strong>countDistinct queries</strong> and observe additional shuffle stages.</li><li>Explore the <strong>Spark UI SQL tab</strong> to understand how query plans map to DAG execution.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0333a35c8a6b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Traditional Data Lakes Break Your Data Pipelines (And How Delta Lake Fixes It)]]></title>
            <link>https://medium.com/@singhanuj2803/why-traditional-data-lakes-break-your-data-pipelines-and-how-delta-lake-fixes-it-dc6eb61dfccd?source=rss-874748343425------2</link>
            <guid isPermaLink="false">https://medium.com/p/dc6eb61dfccd</guid>
            <category><![CDATA[databricks-delta-lake]]></category>
            <category><![CDATA[azure-databricks]]></category>
            <category><![CDATA[data-lake]]></category>
            <dc:creator><![CDATA[Reader]]></dc:creator>
            <pubDate>Tue, 07 Jul 2026 20:40:17 GMT</pubDate>
            <atom:updated>2026-07-07T20:40:17.236Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5LvUUgBxHHXnF2ZXIHCuJw.png" /></figure><h3>1. Introduction: Why You Must Understand the Limits of Data Lakes</h3><p>When you start working with modern data platforms, one of the first things you learn is that <strong>data lakes are extremely powerful for storing large volumes of data</strong>.</p><p>A data lake allows you to:</p><ul><li>Store <strong>structured data</strong></li><li>Store <strong>semi-structured data</strong></li><li>Store <strong>unstructured data</strong></li><li>Ingest data from <strong>multiple systems at massive scale</strong></li></ul><p>Because of this flexibility, many modern architectures rely heavily on <strong>Azure Data Lake Storage or Amazon S3</strong> as their primary storage layer.</p><p>But here is something important you should understand:</p><blockquote><em>A </em><strong><em>data lake is designed for storage, not for transactional reliability.</em></strong></blockquote><p>This distinction becomes critical once you start building <strong>data pipelines, analytics platforms, and machine learning workflows</strong>.</p><p>To truly understand why <strong>Delta Lake exists</strong>, you first need to recognize the <strong>missing capabilities in traditional data lakes</strong>.</p><h3>2. Data Lakes vs Databases: The Critical Architectural Difference</h3><p>One of the biggest misconceptions in modern data engineering is assuming that <strong>a data lake behaves like a database</strong>.</p><p>In reality, the two systems are fundamentally different.</p><h3>Databases</h3><p>A <strong>database management system (DBMS)</strong> provides built-in guarantees that protect your data.</p><p>For example:</p><ul><li>Transactions are reliable</li><li>Data consistency is enforced</li><li>Concurrent operations are managed safely</li><li>Updates and deletes are straightforward</li></ul><p>Because of these guarantees, databases are trusted for <strong>mission-critical applications</strong>.</p><h3>Data Lakes</h3><p>A <strong>data lake</strong>, on the other hand, is simply a <strong>distributed storage system</strong>.</p><p>Examples include:</p><ul><li><strong>Azure Data Lake Storage Gen2</strong></li><li><strong>Amazon S3</strong></li><li><strong>Google Cloud Storage</strong></li></ul><p>These systems excel at <strong>scalable storage</strong>, but they do <strong>not provide transactional database features</strong>.</p><p>When you store files inside a data lake, you lose several protections that traditional databases provide.</p><h3>3. ACID Properties: The Reliability Guarantee You Expect From Databases</h3><p>When you learned about relational databases, you likely encountered the concept of <strong>ACID properties</strong>.</p><p>These properties guarantee <strong>data reliability and correctness</strong>.</p><p>The four ACID principles are:</p><h3>Atomicity</h3><ul><li>A transaction either <strong>completes fully</strong> or <strong>fails completely</strong></li><li>Partial operations are <strong>never committed</strong></li></ul><p>Example:</p><p>If a financial transaction fails halfway through, <strong>the system rolls back everything</strong>.</p><h3>Consistency</h3><ul><li>Data must always follow defined <strong>rules and constraints</strong></li></ul><p>Example:</p><p>A database will reject data that violates:</p><ul><li>Primary keys</li><li>Foreign keys</li><li>Data type rules</li></ul><h3>Isolation</h3><ul><li>Multiple transactions can run simultaneously without interfering with each other.</li></ul><p>Example:</p><p>Two users updating a system at the same time <strong>will not corrupt the dataset</strong>.</p><h3>Durability</h3><ul><li>Once a transaction is committed, <strong>the data is permanently stored</strong>.</li></ul><p>Even if a system crashes, the committed data <strong>remains safe</strong>.</p><p>These properties make databases <strong>trustworthy systems for critical workloads</strong>.</p><h3>4. What Happens When ACID Does Not Exist in a Data Lake</h3><p>Now imagine you are working in a real data engineering pipeline.</p><p>You are using:</p><ul><li><strong>Azure Data Factory</strong> to load data</li><li><strong>Azure Data Lake Storage</strong> as your storage layer</li><li><strong>PySpark</strong> for processing</li></ul><p>Here is a scenario you might encounter.</p><h3>Example Pipeline Failure</h3><p>A pipeline starts overwriting a dataset. During execution:</p><ul><li>The job writes <strong>50% of the data successfully</strong></li><li>The pipeline <strong>fails halfway</strong></li></ul><p>At the same time a <strong>PySpark job </strong>attempts to read the dataset. Since the data lake has <strong>no transactional guarantees</strong>, the dataset now contains:</p><ul><li>Half <strong>new data</strong></li><li>Half <strong>old data</strong></li></ul><p>This creates:</p><ul><li><strong>Incomplete datasets</strong></li><li><strong>Incorrect analytics</strong></li><li><strong>Broken downstream pipelines</strong></li></ul><p>This type of issue is extremely common in traditional data lake architectures</p><h3>5. Major Drawbacks of Azure Data Lake Storage</h3><p>While data lakes provide incredible scalability and cost efficiency, they also introduce several major limitations.</p><p>Understanding these limitations helps you design <strong>more reliable data architectures</strong>.</p><h3>Lack of ACID Transactions</h3><p>Traditional data lakes <strong>do not support ACID transactions</strong>.</p><p>This means:</p><ul><li>Partial writes can occur</li><li>Failed pipelines leave inconsistent data</li><li>Concurrent reads and writes create conflicts</li></ul><p>In other words, <strong>atomic operations are not guaranteed</strong>.</p><h3>No Schema Enforcement</h3><p>Another limitation is that <strong>data lakes do not enforce schema rules</strong>.</p><p>This allows:</p><ul><li>Different file structures in the same dataset</li><li>Inconsistent column definitions</li><li>Broken analytics queries</li></ul><p>Over time, this leads to <strong>messy and unreliable datasets</strong>.</p><h3>No Native Updates or Deletes</h3><p>Unlike databases, data lakes <strong>do not support direct record-level operations</strong>.</p><p>You cannot simply run:</p><ul><li>UPDATE</li><li>DELETE</li></ul><p>Instead, you must:</p><ul><li>Reprocess the dataset</li><li>Rewrite files</li><li>Implement custom logic</li></ul><p>This makes <strong>data management much more complex</strong>.</p><h3>No Built-in Data Versioning</h3><p>Data lakes also lack <strong>native dataset versioning</strong>.</p><p>If data is overwritten:</p><ul><li>You cannot easily restore older versions</li><li>Auditing becomes difficult</li><li>Data recovery becomes complicated</li></ul><p>This limitation creates challenges in <strong>enterprise data governance</strong>.</p><h3>Data Quality Risks</h3><p>Because of these limitations, several data quality issues can appear:</p><ul><li>Partial data writes</li><li>Inconsistent schema structures</li><li>Incomplete datasets</li><li>Difficult debugging of pipelines</li></ul><p>These issues can quickly make <strong>large data platforms unreliable</strong>.</p><h3>6. Why the Industry Created Delta Lake</h3><p>To solve these problems, the data engineering community needed something new.</p><p>The goal was simple:</p><blockquote><em>Combine the </em><strong><em>scalability of data lakes</em></strong><em> with the </em><strong><em>reliability of databases</em></strong><em>.</em></blockquote><p>This is exactly what <strong>Delta Lake</strong> was designed to achieve.</p><p>Delta Lake introduces critical capabilities on top of existing data lakes:</p><ul><li><strong>ACID transactions</strong></li><li><strong>Schema enforcement</strong></li><li><strong>Time travel (data versioning)</strong></li><li><strong>Reliable concurrent reads and writes</strong></li></ul><p>This transformation turns a traditional data lake into a <strong>transactional data lake</strong>.</p><p>Platforms like <strong>Databricks</strong> use Delta Lake to make large-scale data pipelines <strong>reliable, consistent, and production-ready</strong>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dc6eb61dfccd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understand Lakehouse Architecture]]></title>
            <link>https://medium.com/@singhanuj2803/understand-lakehouse-architecture-2da9b4d862ba?source=rss-874748343425------2</link>
            <guid isPermaLink="false">https://medium.com/p/2da9b4d862ba</guid>
            <category><![CDATA[lakehouse-architecture]]></category>
            <category><![CDATA[azure-databricks]]></category>
            <dc:creator><![CDATA[Reader]]></dc:creator>
            <pubDate>Mon, 06 Jul 2026 19:08:47 GMT</pubDate>
            <atom:updated>2026-07-07T03:19:11.691Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kkT5sVJHTYpjdZ_nNM48Qw.png" /></figure><h3>Why You Need to Understand Lakehouse Architecture</h3><p>When you start working with modern analytics platforms like <strong>Azure Databricks</strong>, you quickly realize that data architecture has evolved through several generations.</p><p>To understand the <strong>Lakehouse architecture</strong>, you first need to understand the systems that existed before it.</p><p>Data platforms evolved through three major stages:</p><ol><li><strong>Traditional Data Warehouses</strong></li><li><strong>Data Lakes + Warehouses (Two-Tier Architecture)</strong></li><li><strong>Lakehouse Architecture</strong></li></ol><p>Each generation attempted to solve the limitations of the previous one.</p><p>When you understand this progression, you clearly see <strong>why the Lakehouse architecture was introduced and why it has become the foundation of modern data platforms</strong>.</p><h3>The First Generation: Traditional Data Warehouse Architecture</h3><p>The earliest analytical systems were <strong>traditional data warehouses</strong>.</p><p>These systems were designed primarily to process <strong>structured data stored in tables</strong>.</p><h3>How the Architecture Worked</h3><p>The workflow typically looked like this:</p><ol><li>Data arrives from <strong>structured operational systems</strong></li><li>Data is processed through an <strong>ETL pipeline</strong></li><li>Data is cleaned and transformed</li><li>Data is stored in a centralized <strong>data warehouse</strong></li><li>BI tools generate <strong>reports and dashboards</strong></li></ol><h3>Core Strengths of Data Warehouses</h3><p>Traditional warehouses provided powerful enterprise capabilities:</p><ul><li><strong>ACID transaction guarantees</strong></li><li>Ability to perform <strong>updates and deletes</strong></li><li>Strong <strong>data governance</strong></li><li>Strict <strong>access control</strong></li><li>Reliable support for <strong>Business Intelligence (BI)</strong></li></ul><p>However, these systems had a critical limitation.</p><h3>The Biggest Limitation</h3><p>Data warehouses were built primarily for <strong>structured data</strong>.</p><p>This meant they struggled to support:</p><ul><li>Images</li><li>Logs</li><li>Sensor data</li><li>JSON</li><li>Machine learning datasets</li></ul><p>As organizations began generating massive volumes of diverse data, this limitation became a major problem.</p><h3>The Next Evolution: Modern Data Warehouse with Data Lakes</h3><p>To solve this problem, organizations introduced <strong>data lakes</strong> alongside their warehouses.</p><p>This created a <strong>two-tier architecture</strong>.</p><h3>How the Two-Tier Architecture Works</h3><p>In this setup:</p><p><strong>Data Lake</strong></p><ul><li>Stores large volumes of raw data</li><li>Supports <strong>structured and unstructured data</strong></li><li>Enables <strong>machine learning workloads</strong></li></ul><p><strong>Data Warehouse</strong></p><ul><li>Stores processed and curated data</li><li>Supports <strong>BI reporting and analytics</strong></li></ul><p>This model allows organizations to benefit from both systems.</p><h3>Advantages of Data Lakes</h3><p>Data lakes introduced several important improvements:</p><ul><li>Storage for <strong>any type of data</strong></li><li><strong>Massive scalability</strong></li><li><strong>Low-cost object storage</strong></li><li>Faster <strong>data ingestion</strong></li><li>Support for <strong>data science and machine learning</strong></li></ul><p>This architecture became very popular in modern data platforms.</p><p>But it also introduced new challenges.</p><h3>Limitations of the Two-Tier Architecture</h3><p>Although the two-tier architecture increased flexibility, it also created <strong>operational complexity</strong>.</p><p>A <strong>data lake alone cannot provide the same guarantees as a warehouse</strong>.</p><h3>Key Limitations of Data Lakes</h3><p>Without additional layers, data lakes lack:</p><ul><li><strong>ACID transaction guarantees</strong></li><li><strong>Schema enforcement</strong></li><li>Built-in <strong>governance</strong></li><li>Reliable support for <strong>BI queries</strong></li></ul><p>Because of this, organizations still depended on data warehouses for reporting.</p><h3>Operational Problems Introduced</h3><p>Maintaining two separate systems creates several challenges:</p><ul><li>Additional <strong>infrastructure cost</strong></li><li><strong>Data duplication</strong></li><li>Complex <strong>data synchronization</strong></li><li>Higher <strong>maintenance effort</strong></li></ul><p>Organizations began looking for a way to <strong>unify these architectures</strong>.</p><h3>Databricks Introduced the Lakehouse Concept</h3><p>To solve these challenges, <strong>Databricks</strong> introduced the <strong>Lakehouse architecture</strong>.</p><p>Their goal was simple but powerful:</p><blockquote><em>Create a single system that supports both </em><strong><em>BI analytics and machine learning workloads</em></strong><em>.</em></blockquote><p>The key technology that enabled this idea is <strong>Delta Lake</strong>.</p><h3>What Delta Lake Adds to a Data Lake</h3><p>Delta Lake introduces a <strong>metadata and transaction layer</strong> on top of the data lake.</p><p>This layer includes:</p><ul><li><strong>Transaction logs</strong></li><li><strong>Metadata management</strong></li><li><strong>Schema enforcement</strong></li><li><strong>Data versioning</strong></li></ul><p>Even though the data still resides in object storage, this additional layer enables <strong>database-like reliability</strong>.</p><h3>What Lakehouse Architecture Actually Means</h3><p>The <strong>Lakehouse architecture</strong> combines the strengths of two major systems:</p><ul><li><strong>Data Lakes</strong></li><li><strong>Data Warehouses</strong></li></ul><h3>What You Gain from Data Lakes</h3><ul><li>Storage for multiple data formats</li><li>Massive scalability</li><li>Low-cost object storage</li><li>Support for large datasets</li></ul><h3>What You Gain from Data Warehouses</h3><ul><li>ACID transactions</li><li>Governance</li><li>Query optimization</li><li>Data consistency</li></ul><p>By combining these capabilities, the Lakehouse architecture provides <strong>a unified platform for analytics and AI workloads</strong>.</p><p>This is exactly what modern data platforms require.</p><h3>Key Capabilities of Lakehouse Architecture</h3><p>When you implement a Lakehouse architecture, you gain capabilities from both worlds.</p><h3>Data Lake Advantages</h3><ul><li>Storage for <strong>structured, semi-structured, and unstructured data</strong></li><li>Highly scalable storage</li><li>Efficient ingestion pipelines</li><li>Large data volume support</li></ul><h3>Data Warehouse Capabilities</h3><ul><li><strong>ACID transactions</strong></li><li><strong>Data versioning</strong></li><li><strong>Auditing</strong></li><li><strong>Indexing</strong></li><li><strong>Caching</strong></li><li><strong>Query optimization</strong></li></ul><h3>Another Critical Advantage</h3><p>Lakehouse architectures support <strong>both batch and streaming workloads</strong>.</p><p>This means you can process:</p><ul><li><strong>Historical datasets</strong></li><li><strong>Real-time data streams</strong></li></ul><p>in the <strong>same platform</strong>.</p><h3>Implementing Lakehouse Using Delta Lake</h3><p>In practice, implementing a Lakehouse architecture means <strong>adding a transactional layer on top of a data lake</strong>.</p><p>This is exactly what <strong>Delta Lake</strong> does.</p><h3>How Delta Lake Works</h3><p>Delta Lake operates using three key components:</p><ol><li><strong>Data Lake Storage-</strong>Stores raw data files</li><li><strong>Transaction Log Layer-</strong>Tracks all changes to data</li><li><strong>Metadata Management-</strong>Maintains schema and version control</li></ol><h3>What This Enables</h3><p>Because of the transaction log system, you gain:</p><ul><li>Reliable <strong>updates and deletes</strong></li><li><strong>ACID guarantees</strong></li><li><strong>Time travel for data versioning</strong></li><li><strong>Schema enforcement</strong></li></ul><p>This allows your data lake to behave more like a <strong>data warehouse</strong>.</p><p>When you build modern platforms in <strong>Azure Databricks</strong>, this architecture becomes the foundation of your data system.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2da9b4d862ba" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ Databricks Genie Billing Starts July 6: What This Means for Enterprises]]></title>
            <link>https://medium.com/@singhanuj2803/databricks-genie-billing-starts-july-6-what-this-means-for-enterprises-4464bdfda7e9?source=rss-874748343425------2</link>
            <guid isPermaLink="false">https://medium.com/p/4464bdfda7e9</guid>
            <category><![CDATA[databricks-genie]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[azure-databricks]]></category>
            <category><![CDATA[databricks-certification]]></category>
            <dc:creator><![CDATA[Reader]]></dc:creator>
            <pubDate>Mon, 06 Jul 2026 19:07:59 GMT</pubDate>
            <atom:updated>2026-07-06T19:07:59.342Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/384/0*0Zba8XsUAeP6nek4" /></figure><p>On <strong>July 6, 2026</strong>, Databricks will begin billing for Genie products (Genie, Genie Spaces, Genie Code) under a <strong>pay-as-you-go model</strong>. Each user gets <strong>150 DBUs free per month</strong> (~$10.50 in US East), but usage beyond that will now incur charges. Service principals (automation identities) will be billed from the very first DBU. Compute resources like SQL warehouses remain billed separately.</p><p>This shift marks a turning point: Genie is no longer “free experimentation at scale.” Enterprises must now treat LLM usage as a <strong>governed resource</strong>.</p><h3>🏢 Enterprise Implications</h3><h3>1. Cost Governance Becomes Mandatory</h3><ul><li>Genie usage is now a <strong>metered service</strong>. Without budgets, costs can spiral.</li><li>Enterprises must configure <strong>Unity AI Gateway budgets</strong> to monitor and control spend.</li></ul><h3>2. Experimentation vs. Production</h3><ul><li>The free tier supports light experimentation.</li><li>Production workloads (pipelines, automation, recruiter search, etc.) will quickly exceed the allowance.</li></ul><h3>3. Shared Pool Risk</h3><ul><li>Genie products share one budget pool.</li><li>Without segmentation, teams may compete for the same allowance, leading to “shadow billing.”</li></ul><h3>⚙️ Recommendations for Enterprises</h3><p><strong>Set Budgets Immediately</strong></p><ul><li>Use Unity AI Gateway budgets scoped to workspaces, user groups, or individuals.</li><li>Tag Genie usage with databricks-product: genie.</li></ul><p><strong>Educate Teams</strong></p><ul><li>Engineers should know the free tier limit and billing model.</li><li>Encourage responsible usage and monitoring.</li></ul><p><strong>Segment Workloads</strong></p><ul><li>Separate experimentation from production.</li><li>Assign budgets per team to avoid cross-team conflicts.</li></ul><p><strong>Automate Alerts</strong></p><ul><li>Configure alerts for shared pools.</li><li>Use blocks for per-user limits to prevent runaway costs.</li></ul><h3>👨‍💻 How Data Engineers Can Use Genie Code Effectively</h3><p>Data engineers are the <strong>frontline users</strong> of Genie Code. Here’s how they can maximize value without worrying about usage limits:</p><h3>🔹 Best Practices</h3><ul><li><strong>Batch Requests:</strong> Instead of multiple small calls, batch prompts to reduce DBU consumption.</li><li><strong>Cache Results:</strong> Store outputs locally when possible to avoid repeated calls.</li><li><strong>Use Service Principals Wisely:</strong> Since they have no free tier, reserve them for production pipelines only.</li><li><strong>Optimize Prompts:</strong> Cleaner, more precise prompts reduce token usage and DBU costs.</li><li><strong>Leverage Free Allowance:</strong> Encourage engineers to use their personal 150 DBUs for experimentation before moving workloads to production identities.</li></ul><h3>🔹 Team-Level Strategies</h3><ul><li><strong>Shared Libraries:</strong> Build reusable Genie Code snippets so teams don’t reinvent the wheel.</li><li><strong>Governed Sandboxes:</strong> Create “sandbox workspaces” where engineers can experiment within free limits.</li><li><strong>Monitoring Dashboards:</strong> Track Genie usage per engineer/team to identify heavy consumers early.</li><li><strong>Hybrid Approach:</strong> Use Genie Code for prototyping, then migrate stable workflows to Spark/SQL for cost efficiency.</li></ul><h3>🌐 The Bigger Picture</h3><p>Databricks is signaling that <strong>Genie is enterprise-grade</strong>. Free experimentation is still possible, but governance is now non-negotiable. For enterprises, this is an opportunity to:</p><ul><li>Reinforce <strong>cost accountability</strong> in AI usage.</li><li>Empower data engineers with <strong>structured experimentation</strong>.</li><li>Align Genie adoption with <strong>production-ready governance frameworks</strong>.</li></ul><h3>✨ Closing Thought</h3><p>Genie Code is powerful, but like any compute resource, it must be managed. Enterprises that set budgets, educate teams, and optimize usage will unlock Genie’s full potential — without surprise bills.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4464bdfda7e9" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>