<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Łukasz Król on Medium]]></title>
        <description><![CDATA[Stories by Łukasz Król on Medium]]></description>
        <link>https://medium.com/@lukkrool?source=rss-121ae2e93541------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*yYslai13oQJSXB7w_thANw.jpeg</url>
            <title>Stories by Łukasz Król on Medium</title>
            <link>https://medium.com/@lukkrool?source=rss-121ae2e93541------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 11:19:34 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@lukkrool/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Using Databricks Serverless Warehouse with Power BI’s DirectQuery and Dual Modes]]></title>
            <link>https://medium.com/@lukkrool/using-databricks-serverless-warehouse-with-power-bis-directquery-and-dual-modes-05d0f9fe1649?source=rss-121ae2e93541------2</link>
            <guid isPermaLink="false">https://medium.com/p/05d0f9fe1649</guid>
            <category><![CDATA[direct-lake]]></category>
            <category><![CDATA[direct-query]]></category>
            <category><![CDATA[power-bi]]></category>
            <category><![CDATA[tabular-model]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Łukasz Król]]></dc:creator>
            <pubDate>Sun, 19 Oct 2025 17:41:31 GMT</pubDate>
            <atom:updated>2025-10-19T17:41:31.015Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/665/1*Oi9BG6NET-jVOjrG8scybw.png" /></figure><p><em>Although Databricks supports </em><a href="https://docs.databricks.com/aws/en/dashboards/"><em>basic dashboards</em></a><em>, it is far from being a complete Business Intelligence tool. While it is possible to use </em><a href="https://learn.microsoft.com/en-us/azure/databricks/genie/"><em>AI/BI Genie</em></a><em> directly on clean well-annotated models, there is still a gap when it comes to classic BI — whenever dealing with rich and appealing Reports or self-service Semantic Models allowing for drag and drop experience. Dedicated BI tools such as Power BI or Tableau are directly promoted by Databricks. In this post, I’d like to provide an overview of a powerful combination of Power BI and Databricks that allows to greatly decrease Power BI costs for a slight increase in Databricks costs.</em></p><h3>Why Databricks is not always enough</h3><p>Unity Catalog allows creation of user-friendly table and column names, descriptions, tags and even <a href="https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-syntax-ddl-alter-table-add-constraint">relationships</a>. Furthermore is is possible to create so called <a href="https://learn.microsoft.com/en-us/azure/databricks/dashboards/datasets/custom-calculations">Custom Calculations</a> bearing some resemblance to DAX measures inside Databricks Dashboards.</p><p>Unfortunately these elements cannot be used to create an analog to Power BI’s Semantic model. A visual in Databricks Dashboard can only reference a single flat dataset, view or SQL Query, never full model. This means that the dashboard creator is forced to combine the tables again and again depending on the visual’s context and that the table relationships won’t be followed automatically. There is also no Databricks equivalent of hierarchies allowing for drill-down nor drill-through between the dashboard pages. Finally, Custom Calculations exist as part of the Dashboards and are not included in Unity Catalog.</p><h3>Overview of Power BI Partition Modes</h3><p>Power BI is a popular choice for reporting and visual analytics. Power BI developers often configure the connection to underlying data sources (ie Databricks), simple Power Query ETLs, Semantic Model and the visuals in a single pbix file or <a href="https://learn.microsoft.com/en-us/power-bi/developer/projects/projects-overview">pbip</a> project.</p><p>In our approach, we need to extend this thinking to a <a href="https://learn.microsoft.com/en-us/power-bi/connect-data/service-datasets-understand">Power BI Semantic Model</a> deployed to a Fabric Capacity and one or more Power BI Reports using Live Connection to that Semantic Model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oQBW5miqAOTGMAIl711h4w.png" /><figcaption>Chain of relationships between a report end user and underlying data. A user interacts with a visual. That visual can reference a number of columns and measures in a Semantic Model. The Semantic Model allows for automatic connection and aggregation of data from multiple tables thanks to relationships, hierarchies and measures. Data inside Semantic Model’s partitions is loaded from Databricks Unity Catalog schema. Managed or External, their data is stored from Delta (or Iceberg, Hudi etc.) tables on a dedicated storage account.</figcaption></figure><p>Semantic model enhances a SQL schema with specific definitions of <strong>relationships</strong> (how to join tables), <strong>hierarchies</strong> (ordering of columns for drill-down) and <strong>measures</strong> (aggregation rules). Furthermore it contains the mapping between Semantic Model tables and underlying Unity Catalog tables. This mapping includes information on how the underlying data is loaded (<strong>Partition Mode</strong>) and optionally transformed (<strong>Power Query</strong>). This rigidness allows to unambiguously translate the user’s drag &amp; drop journey into DAX queries executed against the Semantic Model data or translated to SQL and executed against Databricks Unity Catalog. Whether it is the former or latter depends on the settings of the Partition Mode.</p><p>Partition Modes that one will typically encounter are Import, DirectQuery, Dual and DirectLake. These settings are all about tradeoff between UX latency and data latency. Low UX latency means that data is presented fast. Low data latency means that it is up to date.</p><p>In Import Mode, row-level data is loaded from Databricks to Power BI following a schedule and then served directly from Power BI. This offers the lowest UX latency and full power of DAX. However this also means that data is only as new as the most recent refresh. This typically means daily latency or hourly latency in case of hourly refreshes. More importantly, this means long refresh times and looming Fabric costs for larger datasets.</p><p>In DirectQuery Mode, Semantic Model serves as additional metadata on top of a SQL engine. In our case this means a Serverless SQL Warehouse in a Databricks Workspace connected to Unity Catalog (classic or pro Warehouse is impractical for DirectQuery Mode because of the start-up time). Each unique user journey with the Semantic Model results with SQL queries being executed against Databricks. While the data is always up to date (at least up to date with the state of Databricks), the UX latency suffers and the report might overall feel a bit clunky compared to Import Mode, even for small tables. Full set of DAX will not be available. Measures should be kept simple and model clean, so that produced SQL is not to complicated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HzFa1IzsjGzChcNrbeTdYg.png" /><figcaption>Breakdown of different Partition Modes for Semantic Model tables on top of Databricks. Darker table color signals where the data available through interacting with the Semantic Model actually resides.</figcaption></figure><p>It is possible to specify different Partition Modes for tables in the Semantic Model. This is where the <strong>Dual </strong>mode comes into play. A <strong>Dual </strong>partition can act either as an Import partition or a DirectQuery partition, depending on the context. This is the key for mixing Import and DirectQuery partitions effectively.</p><p>When discussing Import Modes, it is impossible not to mention <a href="https://learn.microsoft.com/en-us/fabric/fundamentals/direct-lake-overview">DirectLake</a>. With DirectLake, the Semantic Model is built directly on top of Delta tables, without the step of translating DAX into SQL. This is possibly your target approach if you wish to invest into Fabric heavily. However if your center of gravity is Databricks, and you only wish to build a full-fledged Business Intelligence without going to deep into Fabric territory, a combination of Import, DirectQuery and Dual might bre preferred.</p><p>Having said that, there are a lot of good architecture examples of Databricks-Fabric hybrids, like <a href="https://medium.com/@piethein/integrating-azure-databricks-and-microsoft-fabric-0030d3cf5156">the following article</a>. The idea is usually to create Delta tables with Databricks and then refer to these tables through DirectLake. The limitation of such approach would be that:</p><ul><li>DirectLake expects V-ordered Delta tables — directly on OneDrive or accessible through shortcuts. It won’t be able to read Delta tables using Liquid Clustering, Z-Ordering, or other Open Table Formats supported by Databricks.</li><li>These approaches might require a large Fabric capacity in order to perform more complex queries — it is contrary to the idea of using Databricks SQL Warehouse for the cost-effective heavy lifting.</li></ul><h3>Designing Dual Mode Datamart</h3><p>Let’s consider a simple datamart that could benefit from Direct Query. It is going to be used to model events across time and geographical hierarchy. It contains 2 dimensions — <strong>dimDatetime </strong>and <strong>dimGeography</strong>. In addition to that, it has 2 facts— <strong>fctEvents </strong>(several million records)<strong> </strong>and <strong>fctEventStatisticsDaily </strong>(365–366 records per year). <strong>fctEvents</strong> is fed by a stream and is near-realtime. <strong>fctEventStatisticsDaily</strong> contains aggregation of events enriched by other sources. Both facts connect to both dimensions. The dimensions almost never change.</p><p>Setting all of the tables to Import Mode would be feasible with a smart partitioning of <strong>fctEvents</strong>, ex. partitioning by month and never refreshing older partitions. This would however still result in data latency and large Fabric costs.</p><p>Setting all of the tables to DirectQuery will always work, however the report will be a bit clunky. While acceptable for the large tables, the degregated UX on the small tables is not justified.</p><p>What we want is to set the smaller <strong>fctEventStatisticsDaily</strong> to Import and larger <strong>fctEvents</strong> to DirectQuery. <strong>The dimensions need to work with both facts, so we set them to Dual.</strong> This way, the dimensions can act as Import Mode tables when they need to be connected with <strong>fctEventStatisticsDaily </strong>and Direct Query tables when they need to be connected with <strong>fctEvents</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/483/1*cqJiwjhoNdQIheQf2nmN9Q.png" /><figcaption>Data of dual partitions is recognized by Power BI as being available both in the Semantic Model and in the backend.</figcaption></figure><p>What should happen when we execute a DAX query combining Direct Query tables, like filtering <strong>fctEvents</strong> by <strong>dimGeography</strong> and aggregating by a level of <strong>dimDatetime,</strong> is <strong>query push-down</strong>. DAX should be translated to a single SQL query and sent to the Databricks backend. <strong>dimGeography</strong> should contribute to the WHERE clause and <strong>dimDatetime</strong> to the GROUP BY clause. Data sent back to Power BI should already be filtered and aggregated.</p><p>If we force Power BI to combine data from Import and Direct Query tables directly, with non of them using the Dual mode, a peculiar disaster will occur. Data needs to be joined somewhere before aggregation, and since the engine can’t push the data of the Import Mode table back to Databricks, it will attempt to load the whole Direct Query table to Power BI for the specific query. This will most likely result in overloading the Fabric capacity or hitting the 1 million record row limit. This behavior is one of the reasons of the reputation that Direct Query has with some Data Engineers.</p><p>To be on the safe side, the easiest approach when working with DirectQuery is to set all dimensions and small facts to Dual and leave large facts in DirectQuery.</p><h3>Conclusion</h3><p>While Databricks has some reporting capabilities, it currently misses Semantic Model features such as relationships, measures and hierarchies. This forces Databricks dashboard creators to write manual or AI-generated SQL queries for each context that a Dashboard has to present.</p><p>A Databricks project might have to be supplemented with full-fledged Business Intelligence tool such as Power BI. Introduction of Power BI brings in the question of where should the data be stored — Databricks or in Power BI. DirectQuery mode is a good option for large tables, as long as some latency is accepted and the DAX measures don’t have to be too complex. All the tables that participate in queries referencing a DirectQuery table need to be either DirectQuery or Dual.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=05d0f9fe1649" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Databricks® and Snowflake® interoperability]]></title>
            <link>https://medium.com/@lukkrool/databricks-and-snowflake-interoperability-f33ecea04011?source=rss-121ae2e93541------2</link>
            <guid isPermaLink="false">https://medium.com/p/f33ecea04011</guid>
            <category><![CDATA[lake-house]]></category>
            <category><![CDATA[delta-tables]]></category>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[iceberg-table]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Łukasz Król]]></dc:creator>
            <pubDate>Sun, 18 May 2025 08:23:39 GMT</pubDate>
            <atom:updated>2025-05-18T08:23:39.789Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G_AJk0fCVeA4I5e8icWFzg.png" /></figure><p><em>Databricks and Snowflake are competing data platforms, both providing a comprehensive set of capabilities. In most cases it doesn’t make sense to include both stacks in a project, as the potential synergies are not worth the added complexity. It is however technically possible to integrate the two if necessary, allowing for efficient data exchange. Databricks is the main driver for unlocking the integration, with the process being agnostic from Snowflake’s perspective.</em></p><h3>Basic comparison</h3><p>If considering Databricks and Snowflake as standalone data platforms, an oversimplified selection criteria between the two would be:</p><ul><li>Choose Snowflake for data warehousing excellence.</li><li>Choose Databricks for flexibility and customization.</li></ul><p>Snowflake traditionally excels at warehousing and data sharing, while Databricks is centered around distributed data processing covering various data formats and processing paradigms. Both Databricks and Snowflake can be used to build a cloud data warehouse. In case of Databricks this usually means a number of (properly modelled) tables, with their metadata residing in <a href="https://www.databricks.com/product/unity-catalog">Databricks Unity Catalog</a> and actual data being stored as <a href="https://delta.io/">Delta Tables</a> on a dedicated storage service, ex. an Azure Storage Account (Data Lake Storage Gen2). In case of Snowflake, it is about storing a number of tables — once again properly modelled — using Snowflake’s <a href="https://docs.snowflake.com/en/user-guide/tables-micro-partitions">proprietary storage format</a>. Both Databricks and Snowflake serve their data to Business Intelligence tools via SQL-engines, which they both refer to as “SQL Warehouses”.</p><p>While it is possible to create basic dashboards in both <a href="https://docs.databricks.com/aws/en/dashboards/">Databricks</a> and <a href="https://docs.snowflake.com/en/user-guide/ui-snowsight-dashboards">Snowflake</a>, these should not be considered full-blown Business Intelligence solutions. For a full data platform, they need to be supplemented with tools such as Power BI or Tableau. Both Databricks and Snowflake fall into the “lower” or “backend” layers of a data platform — where most of the actual data integration work takes place.</p><p>The scope of Databricks is broader than pure cloud warehousing and includes areas such as Generative AI Model Serving, Classic ML, Data Streaming, Unstructured Data Processing and Data Catalog.</p><p>Both platforms can make use of<a href="https://medium.com/@manojkumar.vadivel/apache-iceberg-vs-delta-lake-for-your-data-lake-house-e0fdc8b0d199"> Open Table Formats</a> in order to reduce vendor lock-in, at least when it comes to storage. In case of Databricks the answer has always been Delta. Iceberg is on the other hand the open table format preferable by Snowflake. As you will see in in the following sections Iceberg or more specifically Iceberg Metadata is the key to make Databricks-generated data readable by Snowflake.</p><h3>Motivation for integrating Databricks with Snowflake</h3><p>What could be the motivations for integrating both stacks?</p><h4>Enterprise scale</h4><p>For small and stable companies, it might be possible to implement a consistent data policy, clearly embracing one of the platforms. For larger enterprises, trying to enforce a single policy might not always be possible. A large company’s data landscape will always involve a portfolio of projects using different cloud and on-prem technologies, in different stages of development. These “silos” are not something to promote, but are also not to be considered an outright data mismanagement. Rather than spending resources on attempts to migrate them to a single data platform or even unified data model using today’s “top technology”, it might be more practical to encourage data exchange between the projects and plug them into modern data governance concepts such as Data Mesh.</p><h4>Vendor-specific technical skills</h4><p>A team skilled with a suboptimal platform will most likely deliver better results than a team that learns the optimal tool on a project. Maintaining a technical culture around a given stack is the key to deliver successfully and without exceeding a realistic budget.</p><p>When discussing vendor lock-in, one typically focuses on data storage formats. The real lock-in is however not in the data, but in the logic around data transformation processes and their related costs. To a degree, different flavors of SQL used across the platforms can be maintained with tools such as <a href="https://www.getdbt.com/">DBT</a>, allowing to deploy the same project to different SQL engines. This however does not include all the areas of a data project, and might not allow the tools to really shine (ex. a complicated but useful Python notebook running on an all-purpose Databricks cluster). Finally, DBT itself might be considered a lock-in.</p><p>Effective migration between the stacks requires adequate knowledge of both the source and target platform. This does not mean that all the developers in a given company need to master all existing technologies. What it means is that it could be a good idea to keep a number of projects using an alternative technology to preserve skills specific to that technology.</p><h4>Technical synergies in a single data project</h4><p>One can also argue for some technical reasons justifying usage of both stacks simultaneously in a single data project:</p><ul><li>Snowflake excels in relational data warehousing and handling of multiple heavy queries originating from BI Tools, ex. Power BI running in <a href="https://learn.microsoft.com/en-us/power-bi/connect-data/desktop-use-directquery">direct query mode</a>. For such scenarios, it is conceivable to implement Databricks <a href="https://www.databricks.com/glossary/medallion-architecture">golden-layer</a> using Snowflake.</li><li>A Snowflake project might come-up with requirements calling for complicated operations such as dynamic metadata handling, parsing of unstructured data or streaming analytics. For such cases, Databricks could be used to handle these data sources and present the outcome to Snowflake as a relational domain-specific Data Product for further enhancement.</li><li>A company might embrace Databricks Unity Catalog as its default Data Catalog. In such cases it will be necessary for Snowflake projects to be able to access its metadata.</li></ul><h3>Data Sharing Implementation</h3><h4>Core Concept</h4><p>Different approaches can be applied depending if Snowflake is to access Databricks data or the other way around. For both methods described in this post, data is being served back in response to a request — there are no redundant steps involved such as scheduling curated data extracts to be actively pushed from source to target. Proposed methods are <a href="https://docs.databricks.com/aws/en/query-federation/"><strong>Lakehouse Federation</strong></a> to make Snowflake data available in Databricks and <a href="https://docs.databricks.com/en/delta/uniform.html"><strong>Delta UniForm</strong></a> to make Snowflake able to access Databricks generated data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/836/1*N2FmVV_AAFQu5PdJXqjd2A.png" /><figcaption>Two methods to integrate Databricks and Snowflake — arrows represent data transfer direction.</figcaption></figure><p>Both engines can share relational data residing on a storage account. For warehousing scenarios requiring ACID transactions, this should ideally be a Delta <a href="https://docs.databricks.com/aws/en/delta/uniform">UniForm</a> with Iceberg compatibility enabled. With this approach, the same table will be visible as Delta for Databricks and as Iceberg for Snowflake. As <a href="https://medium.com/@kywe665/unity-catalog-vs-apache-polaris-522b69a4d7df">Iceberg requires a Catalog</a>, the Databricks Workspace needs to be upgraded to Unity Catalog and Snowflake needs to access the table through <a href="https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration-rest-unity">Catalog Integration</a>. While both engines access the same storage account, <strong>only Databricks can use Delta-Uniform in write mode</strong>. Thus Delta-Uniform can be used to make Databricks-created data readable in Snowflake.</p><p>With Lakehouse Federation it is possible to map Snowflake as a Foreign Catalog in Databricks Unity Catalog, and thus be able to read Snowflake native tables and views directly.</p><p>Physical data storage is handled differently in both platforms. While all of Databricks data — managed or not — is recommended to be external and typically resides on a storage account (ex. ADLS Gen2), Snowflake will most likely still keep most of its data internal using its proprietary format. This is reflected on the following picture — while both platforms can access an external storage account, Snowflake needs to be accessed directly to fetch data stored internally in proprietary format or accessible through views.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/889/1*_PIYrrlVLTwFd7iwCHPe0A.png" /><figcaption>Two methods to integrate Databricks and Snowflake. Arrows point from the side initiating the connection to the side serving or storing the data.</figcaption></figure><h4>Network setup</h4><p>Network connectivity needs to be established from both sides before subsequent configuration steps can be performed. As a baseline for both methods, the following points need to be accomplished:</p><ul><li>Storage Account needs to be accessible from the computes of both Databricks and Snowflake.</li><li>Databricks Workspace needs to be accessible from Snowflake for Catalog Integration.</li><li>Snowflake endpoint needs to be accessible from Databricks for Lakehouse Federation.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/687/1*goOKyVgMZaUUa6Q-jmh1QQ.png" /><figcaption>Simplified Azure Cloud Network setup for Databricks-Snowflake integration. Arrows represent possibility to reach a target from a source. Solution can be implemented using IP access lists (service endpoints) or private endpoints.</figcaption></figure><h4>Shared external tables (Delta UniForm)</h4><p>Configurations steps for enabling the connection on both sides are well described in <a href="https://www.databricks.com/blog/read-unity-catalog-tables-in-snowflake">the following Databricks article</a>. The <a href="https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration-rest-unity">following Snowflake documentation</a> can serve as reference for Catalog Integration.</p><p>Additional high-level takeaway:</p><ul><li>The common Open Table Format used by both engines is not Iceberg but Delta with additional metadata making it readable by Iceberg clients (Delta-Uniform Iceberg).</li><li>This connection is one way. While Iceberg clients can read the Delta UniForm tables, they cannot write to these tables.</li><li>Instead of creating an explicit extract, it is possible to enable Iceberg client compatibility for an area of the Databricks Lakehouse, ex. all of the silver layer. There is a minor performance degradation related with this setting, but is should still be much cheaper than an explicit export — compute and developer timewise.</li></ul><h4>Implementing Lakehouse Federation</h4><p>Please refer to Databricks documentation when it comes to g<a href="https://docs.databricks.com/aws/en/query-federation/">eneral Lakehouse Federation</a> as well as its <a href="https://docs.databricks.com/aws/en/query-federation/snowflake">Snowflake specifics</a>. What’s worth highlighting is that:</p><ul><li>Foreign Catalogs created using Lakehouse Federation are read only. Databricks can read from Snowflake, but cannot write to Snowflake with this method.</li><li>Databricks performs push-down of queries, translating the logic coming from Spark SQL or PySpark. Push-down capabilities include filters and joins. It is possible to filter a Snowflake table using another Snowflake table without having to pull both tables into a Spark session. For those familiar with SAS, this concept will seem similar to implicit passthrough. You can check <a href="https://medium.com/@lukkrool/sas-and-databricks-a-former-sas-consultants-comparison-27866a7faa8a">my other article</a> for more details.</li><li>This approach requires double compute, as both Databricks and Snowflake will be involved in fetching the data, even if Snowflake makes use of external Iceberg tables. So there will be compute costs involved on both sides. However because of the push-down mechanism the Snowflake costs can remain low if filters are applied correctly.</li></ul><h3>Conclusion</h3><p>It is worth remembering that not everything that is technically possible is worth doing. Snowflake and Databricks are comprehensive ecosystems. Combining them together in a single project brings additional complexity and potential redundancy.</p><p>While it might seem cumbersome to look for motivation fur such integration in a small project, it could make sense in a larger organization with a strong presence of both stacks. When sufficient technical and administration skills are available on both sides, it is possible to exchange the data between the two easily.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f33ecea04011" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SAS and Databricks — a former SAS consultant’s comparison]]></title>
            <link>https://medium.com/@lukkrool/sas-and-databricks-a-former-sas-consultants-comparison-27866a7faa8a?source=rss-121ae2e93541------2</link>
            <guid isPermaLink="false">https://medium.com/p/27866a7faa8a</guid>
            <category><![CDATA[data-platforms]]></category>
            <category><![CDATA[comparison]]></category>
            <category><![CDATA[sasinstitute]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[migration]]></category>
            <dc:creator><![CDATA[Łukasz Król]]></dc:creator>
            <pubDate>Sat, 07 Dec 2024 13:25:10 GMT</pubDate>
            <atom:updated>2024-12-07T13:25:10.108Z</atom:updated>
            <content:encoded><![CDATA[<h3>SAS and Databricks — a former SAS consultant’s comparison</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*voenDeIA9Ydo9p4WTZ8cDA.png" /><figcaption>SAS and Databricks logos</figcaption></figure><p><em>Databricks needs no introduction — along with Snowflake and Microsoft Fabric it is one of the “big three” Cloud Data Platforms, providing Lakehouse, Data Integration, Streaming, Machine Learning and Model Serving Capabilities. SAS®, the analytics software suite by SAS Institute, is a different kind of animal. It is one of the most venerable data platforms, having a history of reverse compatibility all the way back till 1976.</em></p><h3>Why to compare SAS with Databricks</h3><p>While SAS Customers enjoy unchallenged stability and high quality of service provided by an experienced network of SAS Consultants, the conservative strategy of SAS Institute has its drawbacks. The software was originally designed to make the most of sequential processing of structured data on mainframe systems. Today’s pattern of choice is to perform distributed processing and serving of mixed structured and unstructured data — the Lakehouse approach. While the two main overlapping languages of SAS (SAS 4GL, SAS Macro Language) are quite powerful when dealing with structured data on a local hard drive, these languages will feel alien to Python or even SQL Users. This creates a sort hermeticity around SAS, isolating it from dominating trends. SAS Users, Programmers and Consultants need to choose if they wish to embrace SAS Viya — the cloud offering of SAS Institute, or master an alternative platform.</p><p>This is where Databricks comes into play. As someone who worked with both platforms for over 3 years each, I find Databricks to possess some features that SAS Developers might find friendly. This comparison is aimed to serve as a mind-map for SAS Developers wishing to master a second technology. First I am going to highlight where the major differences that can not be forgotten are. It is going to include a comparison of different environment components, followed by Databricks SQL/Python analogs for SAS 4GL/Macro. <strong>Please keep in mind that this comparison is for classic on-prem SAS 9.4, not SAS Viya.</strong></p><h3>Core differences</h3><p>SAS 9.4 is an on-prem client-server system, while Databricks is PaaS therefore this comparison should not be interpreted directly. There are many articles providing an overview of Databricks from the Cloud perspective. The aim of this article it to provide an SAS angle. This is not a one to one comparison — more like the closest think of the Databricks side resembling a SAS concept. Before we get there, let’s highlight what’s incomparable.</p><h4>Roots and foundation</h4><p>SAS originated as proprietary software for mainframe systems. These days SAS runs on z/OS, UNIX, Linux and Windows. Originally it consisted solely of an interpreter allowing to run batch SAS programs as well as to work interactively after logging-in to a machine having the software previously installed. In the early 2000s came SAS 9 with its client-server architecture, exposing the existing interpreter with a plethora of Application Servers accessible from an even bigger list of desktop client tools, Data Integration Studio or Enterprise Guide to name a few. SAS Viya, released in 2016 is aimed to provide a cloud experience for SAS Users.</p><p>Databricks is a commercial PaaS offering built around the open source Spark framework. Much like SAS is available on different operating systems, Databricks is available on different cloud platforms — Azure, AWS and GCP. While a Developer’s or Analyst’s experience is similar, provisioning and administering Databricks in the cloud environment requires knowledge specific for a given cloud platform. While Databricks itself is only available in the cloud, pure open-source Spark could be run on-prem or as IaaS to avoid vendor lock-in.</p><p>Full Databricks landscape requires more than the Workspace. A complete system requires a number of components coming from a specific cloud provider, ex. a Key Vault, Storage Account and a Virtual Network when implemented in Azure:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*U2QfriuY-s8AIdca.png" /><figcaption>Simple Cloud Architecture for Databricks on Azure</figcaption></figure><h4>Computation paradigm</h4><p>SAS makes the best our of sequential disk to disk processing. This is an artifact of times when compute was local and RAM was expensive. Most SAS DATA STEPs and PROC STEPs cause a disk read and disk write. Databricks on the other hand is designed as an in-memory distributed processing engine, coupled with an external distributed storage. Writing or reading to storage is an explicit operation, and for most of the duration of a Spark job or interactive session data is typically distributed between the operating memories of cluster nodes.</p><p>SAS programming embraces the sequential style, like with the following RETAIN and BY statements:</p><pre>/*Sort the data in place.*/<br>proc sort data=input_data;<br>  by input_grouping;<br>run;<br><br>/* Count the number of groups in a dataset.*/<br>/* Put it in a macro variable.*/<br>data _null_;<br>  set input_data end=last_record;<br>  by input_grouping;<br><br>  /*Declare a variable that will keep value when iterating through rows.*/<br>  retain n_groups 0; <br>  <br>  /* +1 group */<br>  if last.input_grouping then do;<br>    n_groups = n_groups + 1;<br>  end;<br><br>  /*Put the result in a macro variable.*/<br>  if eof then do;<br>    call symput(&quot;macro_variable&quot;,n_groups);<br>  end;<br>run;</pre><p>Spark is not MapReduce and provides abstractions for sequential processing, ex. through <a href="https://spark.apache.org/docs/3.5.1/sql-ref-syntax-qry-select-window.html">Spark SQL Window Functions</a>. Nevertheless operations requiring sorting should be avoided, as they require to shuffle data between nodes of the computation cluster. If a SAS Program contains too many sequential operations, it might be wiser to implement it from scratch rather than to try to perform a 1–1 rewrite.</p><p>SAS operations are eager, meaning that they are executed the moment a part of code is interpreted. Spark is lazy — as long as data is not directly referenced or cached, new transformations are just added to a lineage tree, only to be executed when needed.</p><h4>User interface</h4><p>Databricks doesn’t come with any graphical desktop tools. There is also obviously no concept of an explicit server. A <strong>Databricks Workspace</strong> is an integrated environment available in the browser through a unique URL providing access to all of data, code and compute. In addition to the Databricks Workspace, there is the <strong>Account Console</strong>, allowing to administer <strong>Unity Catalog</strong>. Unity Catalog acts as a metadata and security umbrella above different <strong>Databricks Workspaces</strong>. Does it sounds similar to Metadata Server?</p><p>Most Workspace configuration can be performed visually, and some graphical design can be done, ex. for Workflows or Dashboards, but the primary experience is through code. This involves manipulating data with SQL and Python (more rarely with R and Scala) as well as the Databricks infrastructure itself with Databricks API, Databricks CLI and Terraform.</p><h4>Handling Metadata</h4><p>SAS provides a double naming for the schema by separating physical column names from those visible to Business Users through the concept of Information Maps. In Databricks, a physical column name is equivalent with its metadata name. Moreover SAS introduces ambiguity to column interpretation through so called formats, separating actual values from how they are displayed, making formats an inseparable element of physical datasets. In Databricks, data is displayed the way it is stored, say STRING, DATETIME or DECIMAL (10,2). DATE presentation and other formatting can be introduced in the Business Intelligence reports, but not in metadata or physical datasets.</p><h4>Strategy</h4><p>For 50 years SAS thrived on reverse compatibility. Databricks on the other hand pushes forward with new versions of Databricks Runtime, often braking the compatibility and forcing (mostly minor) updates to the code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/605/1*P5i5OWlhJMAT2CNNWmkyZA.png" /><figcaption>Databricks Runtime 9.1 is the oldest one currently available. It was released in September 2021.</figcaption></figure><p>SAS Institute provides its Customers not only with technology, but also with complete solutions dealing with a specific business area, such as risk management. Databricks on the other hand is a purely technological company.</p><p>Lastly, there is a big difference in terms of the entry cost and personal investment. SAS is quite closed, with paid trainings and limited open exchange. Core of Databricks is open-source, and it is possible to get started for several EUR per month or even for free when having access to a trial Azure Subscription.</p><h3>Environment and architecture comparison</h3><h4>SAS Session -&gt; Interactive Spark Application</h4><p>SAS statements are executed in the context of a SAS session. A SAS session keeps track of LIBNAME assignments and macro variables. It executes DATA STEPs and PROC STEPs after substituting enough macro variables to assemble executable SAS code. A SAS session is local to the machine on which it is running. SAS macro processor can be characterized as <strong>on-line macro processor</strong> — both the macros and the actual code are executed in the same run, and it is possible to influence the macro variables from the underlying SAS 4GL.</p><p>In Spark, <a href="https://spark.apache.org/docs/3.5.3/cluster-overview.html">distributed is the norm and local is the exception</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/596/0*uQqaGIHKyvq8BwBk.png" /><figcaption>Spark Architecture</figcaption></figure><p>An interactive Spark Application (ex. a Databricks Notebook) is controlled from a Driver Program, but actual operations on distributed in-memory datasets (<a href="https://spark.apache.org/docs/3.5.3/sql-programming-guide.html#content">data frames</a>) are executed in parallel on Worker Nodes. It is also possible to execute Python statements (ex. with Pandas) outside of Spark — such code will be executed entirely in the Driver Program and won’t affect worker nodes.</p><p>In Databricks, Spark can be interacted with from Python, SQL, Scala and R. However contrary to SAS, their interpreters are not working at the same time. While it it possible to execute SQL statements from Python or Scala, a Spark program is either Python, Scala, R or SQL. It is however possible to mix languages in a single Databricks notebook through multiple cells.</p><h4>SAS Program -&gt; Databricks Notebook</h4><p>Whenever working directly with the SAS interpreter, or using graphical tools, SAS Developers ultimately create SAS Programs — text files with *.sas extension. SAS Programs consist of SAS 4GL enriched with the SAS Macro layer giving it some flexibility.</p><p>Databricks Notebooks are the most common way of working with Spark in Databricks. A Notebook needs to have a default language. A Notebook consists of <strong>cells</strong> acting as execution units. Each cell can be written in a different language.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/463/1*ntfz1Nt3DRFkdWjIS2otiA.png" /><figcaption>Displaying the classic sashelp.class dataset in Python, Scala and SQL</figcaption></figure><p>Scala and Python can’t access each other’s variables, but can use the same SQL schema.</p><p>SAS can be used for ETLs, but traditionally it is also used as an interactive statistical tool. In Databricks, this can be achieved with Python and R.</p><h4>SAS Dataset -&gt; Delta Format</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*nTWqlP9QDMip92xvxeAsQA.png" /><figcaption>Delta Lake</figcaption></figure><p>Much like a LIBNAME is a reference to a SASXBDAT file residing on a disk, a Unity Catalog table references a <a href="https://delta.io/">Delta Format</a> file residing on a Data Lake (hence Delta Lake). Delta Format itself is an extension to <a href="https://parquet.apache.org/">Parquet</a>. While Parquet is a purely analytical column-oriented immutable data format, Delta enhances it by enabling updates and deletions through ACID transactions (it isn’t an OLTP-grade transactionality, but it’s enough for warehose/lakehouse operations). Delta is not really a file but a folder with multiple parquet files with a JSON text file. Each transaction writes new parquet files to the delta and updates the JSON file. Inserts, updates, deletions — their data and information about type of opertion is all contained in the newly appended file. During fetching, after the actual version of a record is found, its older versions are ignored. Once in a while, the VACUUM command needs to be run on the Delta Folder to defragment it and reduce the number of older parquet files inside.</p><p>The way that UPDATES and DELETIONS are implemented in Delta are much different than with SAS datasets. While SAS operates on individual rows and either modifies them or flags them as deleted, Delta only appends new data and operates on whole batches rather than individual records.</p><h4>SAS Index -&gt; Liquid Clustering or Partitioning and Z-Order</h4><p>SAS allows to create external indexes on SASXBDAT files, in the form of another file sitting along the main data file. It is also possible to keep SASXBDAT datasets in a predefined sort order, thus creating an internal index.</p><p>Distributed nature of Delta is in contrast with the idea of a centralized index allowing to retrieve a single record. It is however still possible to organize the data in a way that limits the number of files that need to be fetched and scanned. Delta makes use of data skipping — the JSON index contains minimal and maximal values in each column of each parquet file. This allows the fetching process to assess if the parquet file should be scanned at all. It is recommended to distribute the data between the different files in a way that will <strong>cluster</strong> it by up to four data columns through <a href="https://learn.microsoft.com/en-us/azure/databricks/delta/clustering">Liquid Clustering</a>. For really large tables (&gt;10 TB) it is still a good idea to implement partitioning, thus having separate subfolders for specific values of a given column, and enforce <a href="https://learn.microsoft.com/en-us/azure/databricks/delta/data-skipping">Z-Ordering</a> on the partitions.</p><h4>Metadata Server -&gt; Unity Catalog</h4><p>Originally in SAS, libraries (lists of tables) were created individually in each SAS program through LIBNAME statements pointing to folders containing SASXBDAT (SAS9BDAT, SAS8BDAT etc.) files, or even z/OS datasets containing members. After a LIBNAME is assigned to a <strong>libref</strong>, it is possible to reference a table in a SAS program through the LIBNAME.FILENAME table reference. Later came Metadata Server, containing centralized definitions for such libraries, as well as definitions of other objects such as Reports or Information Maps.</p><p>In addition to libraries consisting of SAS datasets, it it also possible to point to other data sources, in particular external databases such as SQL Server, Oracle, DB2 etc. A powerful and often underappreciated feature of SAS LIBNAMES is <strong>implicit passthrough </strong>— ability to translate filters in SAS DATA STEP WHERE statements as well as PROC SQL into native SQL of a relational database and executing it directly against that database.</p><p>In Databricks, Metadata for relational tables is stored in <a href="https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/">Unity Catalog</a>. There can be only one UC Account per (ex. Azure) Subscription. In an Account, there can be multiple Metastores defined, with Metastore acting as a complete metadata repository. Inside a Metastore there can be defined Catalogs, Schemas, Tables, Views and <a href="https://learn.microsoft.com/en-us/azure/databricks/database-objects/">other objects</a>. Databricks relational data typically consists of files in the <a href="https://learn.microsoft.com/en-us/azure/databricks/introduction/delta-comparison">Delta Format</a> stored on an external storage service, ex <a href="https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction">Data Lake Storage Gen2</a>. Unity Catalog supports the concept of <a href="https://learn.microsoft.com/en-us/azure/databricks/query-federation/">Lakehouse Federation</a>, in which it is possible to create UC Catalog pointing to a cloud service, such as Snowflake or Azure SQL. To an extent it <strong>also supports implicit passthrough which is here referred to as push-down</strong>.</p><p>Unity Catalog contains mostly definitions related to data. Definitions of Computes and Code to be executed on those Computes as well as other Databricks objects such as Queries, Dashboards or MLFlow experiments are stored locally in Databricks Workspaces.</p><h4>Workspace Server -&gt; All-Purpose Cluster</h4><p>When working with SAS 9 in client-server architecture, it is possible to execute code remotely on a Workspace Server form a client tool such as Enterprise Guide. In Databricks, you run Notebooks mainly on All-Purpose Clusters defined in the Workspace:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/854/1*_nBpIFVNuaJtLPN7Yi0jiQ.png" /><figcaption>All-purpose compute — Databricks Workspace</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/953/1*-61Gn7mSDUuKnYlk0wdYYA.png" /><figcaption>Attaching a Databricks Notebook to an All-purpose compute.</figcaption></figure><h4>Batch Server -&gt; Job Cluster</h4><p>SAS Batch servers are abstractions for running SAS Programs using physical server’s local interpreter on a given schedule. Databricks supports batch processing through Job Clusters — cheaper analogs of all-purpose clusters that exist only for the duration of a Databricks Job (or DLT Pipeline).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/910/1*LTJU2iDg3kbmPGbtD72jDA.png" /><figcaption>Configuring Tasks in a Databricks Job to run on a Job Cluster.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/1*wvDnNXZ--8amPFQLn4zz5Q.png" /><figcaption>Interface to monitor running and expired job clusters.</figcaption></figure><h4>SAS SPDS, SAS LASR-&gt; Databricks SQL Warehouse</h4><p>SAS SPDS and later SAS LASR are engines aimed at reducing the bottleneck of SASXBDAT datasets in order to support tools requiring high data throughput such as Visual Analytics.</p><p>Databricks provides SQL Warehouses — specialized SQL-only computes that allow to fetch and cache data in parallel in an efficient manner. There are three types of SQL Warehouses — <strong>classic</strong>, <strong>pro</strong> an <strong>serverless</strong>. While classic and pro are computes running in user’s Subscription much like All-purpose and Job clusters, serverless runs in subscription of Databricks and offers similar experience to Snowflake, that is no-latency start time of the resource.</p><h4>Visual Analytics, Web Report Studio -&gt; Databricks Dashboards</h4><p>While Databricks offers Business Intelligence in the form of Dashboards, it must be said that it is quite limited if compared to full-blown solutions such as Power BI or Tableau. Nevertheless it is there, offering simple BI capabilities.</p><h3>Coding comparison</h3><h4>PROC SQL -&gt; SPARK SQL</h4><p>This one is rather obvious. SAS allows to create SQL statements through the PROC SQL interactive procedure. Spark SQL can be executed either directly in SQL cells or through the spark.sql function in Python cells.</p><pre>proc sql;<br>    create table work.class as<br>    select sex, count(*) as count<br>    from sashelp.class<br>    group by sex;<br>quit;<br><br>proc print data=work.class;<br>run;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/523/1*cdIUnHQ7_o1i3OdxSB6cdQ.png" /><figcaption>Running Spark SQL on sashelp.class</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/511/1*KqSK_tsQ2ecp3K4mn9wqyQ.png" /><figcaption>Running Spark SQL on sashelp.class through a Python session</figcaption></figure><h4>DATA STEP -&gt; SPARK SQL or PySpark</h4><p>Simple DATA STEPs can be rewritten to SQL, more complex DATA STEP might require Python or mixing Python and SQL in multiple cells.</p><pre>data M F X;<br>  set sashelp.class;<br>  if sex = &quot;F&quot; then output F;<br>  else if sex = &quot;M&quot; then output M;<br>  else output X;<br>run;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*ExHwQG16-lrJsNir8DHHGA.png" /><figcaption>Three ways to filter a table — Spark SQL, Spark SQL embedded in PySpark and pure PySpark</figcaption></figure><h4>MACRO LANGUAGE -&gt; Python</h4><p>In SAS, 4GL statements are often wrapped in SAS Macros that give them some flexibility. In the same way that SAS Macro Language wraps around SAS 4GL, Python can wrap around SQL. Dynamic SQL should not be abused however, as most operations can be most likely performed in pure Python.</p><pre>%macro filter_class(sex);<br>  data class_&amp;sex.;<br>    set sashelp.class(where sex=&quot;&amp;sex.&quot;);<br>  run;<br>%mend filter_class;<br><br>%filter_class(M);<br>%filter_class(F);<br>%filter_class(X);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/710/1*JodLkCxhWe6ovdq1fkyIgw.png" /><figcaption>Emulating a SAS Macro with Dynamic SQL</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/707/1*nJ08xmKDo5jXUzyjm_GP0g.png" /><figcaption>Rewriting a SAS Macro to Python</figcaption></figure><p>An alternative way of handling macros would be to use <a href="https://www.getdbt.com/">DBT </a>and its <a href="https://jinja.palletsprojects.com/en/stable/templates/">Jinja </a>macros. These are however “offline” macros compiled before deploying code to the environment. It is not possible to influence the Jinja context from within Python or access values of Python variables and contents of Unity Catalog with Jinja.</p><h4>Automated rewrite</h4><p>These days we are obviously not forced to rewrite code manually. The built-in Databricks Assistant comes handy when it comes to generating PySpark, Spark SQL or doing rewrites:</p><pre>%macro class_multiplier(N);<br>    data class_multi;<br>        set <br>            %do &amp;i. = 1 %to &amp;N.;<br>                sashelp.class<br>            %end;<br>        ;<br>    run;<br>%mend class_multiplier;<br><br>%class_multiplier(10);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/730/1*YySZmx-J5LpazyTQc4Yn_A.png" /><figcaption>Automatically rewriting simple SAS Code to PySpark. Notice that the output is slightly different as it contains the additional iteration column.</figcaption></figure><p>Think how much more could be achieved with dedicated models, or when performing finetuning or supplying a model with more SAS Corpus with RAG. This is of course a simple example. As with all AI-generated code, the prompt user needs to understand the output, ex. to point the model to areas still needing correction after the first run, and choose what pieces of SAS code to present to the model. Last but not least, there are some tools on the market that claim automated SAS code rewrite which could be worth checking.</p><h3>Conclusion</h3><p><em>If it works don’t touch it</em> is a common wisdom in IT, and it worked well for SAS. However the on-prem to cloud move we see is forcing some SAS Installations to be lifted to SAS Viya anyway. This might be a good moment to consider migrating to a cloud-native tool such as Databricks. Databricks is obviously not the only tool on the market, so a detailed comparison and a couple of POCs with different technologies should be performed before picking a strategy. Databricks however has some features where some direct analogies can be pointed out.</p><p>While it is important to keep in mind the core differences…</p><ul><li>SAS is not only technology, but also business solutions and consultancy</li><li>Databricks is only technology, but has a bigger scope than SAS including streaming, unstructured data and Generative AI</li><li>SAS is local and sequential</li><li>Databricks is distributed and parallel</li></ul><p>…migration of SAS Code to Databricks is feasible.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=27866a7faa8a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Databricks Workspace Security Explained]]></title>
            <link>https://medium.com/@lukkrool/databricks-workspace-security-explained-52c62c197430?source=rss-121ae2e93541------2</link>
            <guid isPermaLink="false">https://medium.com/p/52c62c197430</guid>
            <category><![CDATA[metadata]]></category>
            <category><![CDATA[data-lakehouse]]></category>
            <category><![CDATA[workspace]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[security]]></category>
            <dc:creator><![CDATA[Łukasz Król]]></dc:creator>
            <pubDate>Fri, 08 Nov 2024 11:35:09 GMT</pubDate>
            <atom:updated>2024-11-25T20:18:48.267Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Boo77gjeRuAoBEdCA7vEMQ.png" /></figure><p><em>Unity Catalog greatly improves the way </em><strong><em>metadata</em></strong><em> and </em><strong><em>physical data</em></strong><em> can be secured and accessed from Databricks Workspaces, however this does not cover all there is to secure. There remains a number of important objects that are still defined at the level of a Databricks Workspace and need to be configured locally. Let’s get through these objects, their configurable permissions and see what the consequences of their misuse could be.</em></p><h3>Databricks securables</h3><p>On the highest level, Databricks securables could be classified as follows:</p><ol><li>Metadata</li><li>Physical data</li><li><strong>Workspace Objects</strong></li></ol><p>An obvious example of <em>metadata </em>is a SQL definition of a table, whether in Unity Catalog or in Hive Metastore. There are many more types of <em>metadata </em>securables. Some of them resemble classic warehouse or database objects such as <em>Catalogs</em>,<em> Schemas</em> or V<em>iews</em>, there are also new lakehouse objects such as <em>Volumes</em> or <em>External Credentials</em>. Check <a href="https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/">Unity Catalog documentation</a> for a full list.</p><p>Behind <em>metatada</em> stands <em>physical data</em>, most often in the form <a href="https://learn.microsoft.com/en-us/azure/databricks/introduction/delta-comparison">Delta-format parquet files</a> on a dedicated <em>Storage Account</em>. In the perfect world, both <em>metadata </em>and <em>physical data</em> should be secured and accessed in the same way, it is however not always the case. In a classic relational warehouse, the filesystem is off-limits for most users of the SQL interface. In the case of a lakehouse, users and developers are encouraged to work with the cloud filesystem directly, so we are dealing with a mixed security model — we need to secure both the relational schema and the underlying filesystem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BtmwXozdWR_OulijPQJfGA.png" /><figcaption>Three ways of accessing physical data residing on a storage account.</figcaption></figure><p><em>Databricks Workspace</em> Users access physical data (relational <em>Tables</em> and file <em>Volumes)</em> typically through secured definitions in Unity Catalog. In older Databricks configurations, it is often possible to bypass <em>Metadata </em>(local <em>Hive Metastore</em>)<em> </em>and work with Delta and other file formats directly. Lastly, data on the Storage Account might have to be accessed using tools not connected to Unity Catalog, such as Azure Data Factory, Azure Storage Explorer or a Fabric Workspace. For more information on how to access a Storage Account from Databricks, you can check <a href="https://medium.com/@lukkrool/databricks-azure-storage-account-connection-explained-7ed0b6ab32ff">my other Medium article</a>.</p><p>To even start thinking about accessing the <em>physical</em> data or <em>metadata</em>, we need to be able to use the <em>Databricks Workspace</em>. This is where various <a href="https://learn.microsoft.com/en-us/azure/databricks/workspace/workspace-assets"><em>Databricks Workspace Objects</em></a> come into play.</p><h3>Databricks Workspace Objects</h3><p>One way to classify the <em>Workspace Objects</em> could be as follows:</p><p><strong>Core Configuration</strong> — this includes the Principals allowed to use the <em>Workspace</em> (<em>Users</em>, <em>Groups</em>, <em>Service Principals</em>), their general entitlements and general workspace settings</p><p><strong>Computes</strong> — different types of computes available in the workspace, such as <em>Clusters, SQL Warehouses, Vector Search Endpoints, Model Serving Endpoints and Apps</em></p><p><strong>Code &amp; other instructions </strong>— <em>Notebooks</em>, <em>Queries</em>, <em>Jobs, Delta Live Table Pipelines</em>, Dashboards, <em>Alerts</em> and more</p><p><strong>Secret Scopes</strong> — cached credentials for accessing external data without Unity Catalog</p><p>That’s quite a lot of securables. I’ll go through all of them and explain what permissions can be applied on them and what could be the consequences of their misuse. What’s important to remember, they are all local to the <em>Workspace</em>, meaning that they can’t be managed with Unity Catalog.</p><h3>Core Configuration</h3><h4>Workspace Principals</h4><p>Even though principals are defined globally in Unity Catalog for all the <em>Workspaces</em> in a given <em>Account</em> and <em>Subscription</em>, they need to be explicitly added to a Databricks Workspace. Individual user account permissions should be avoided, and all permissions should be granted through groups to decrease the amount of administrative work related with adding and deleting users and to make the permissions consistent for users of the same type.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/622/1*dSqpL82oDf6opa0UkLB9-A.png" /><figcaption>Databricks Workspace Principals — UI view</figcaption></figure><p>In general, principals are the object <strong>to</strong> which permissions are assigned, with the exception of <em>Service Principals</em> used to run <em>Workflows</em> and other automation tasks. Users and Groups can get the following permissions <strong>on</strong> a <em>Service Principal</em> (at the Account level):</p><ul><li><strong>Service principal: Manager </strong>— can change security settings of a Service Principal</li><li><strong>Service principal: User</strong> — can use the Service Principal in a Workflow</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/1*oEqjcBtVAPffBpCjUySKJQ.png" /><figcaption>Permissions on a Service Principal —Databricks UI View</figcaption></figure><p>Using a <em>Service Principal</em> means having access to all of its permissions through a <em>Workflow</em>. <strong>It is possible for a User/Group to bypass its own permissions this way, including permissions to data. </strong>Permissions to use a Service Principal should only be granted to Users/Groups having similar security scope.</p><h4>General Entitlements</h4><p><em>Workspace Principals</em> can get the following general permissions:</p><ul><li>Workspace access</li><li>Databricks SQL Access</li><li>Unrestricted Cluster Creation (Users and Groups)</li><li>Admin access</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/447/1*1wiudcIUlfZZGcjldq54kg.png" /><figcaption>General permissions for a User (Admin)— Databricks UI View</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/442/1*9HtUVU-1Gru3CzRqFJbOFg.png" /><figcaption>General permissions for a Group — Databricks UI View</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/177/1*SjDswaXVu-b6cmbMgeAgsg.png" /><figcaption>General permissions for a Service Principal — Databricks UI View</figcaption></figure><p><em>Workspace Access </em>and <em>Databricks SQL</em> Access are enabled by default for all users. These permissions control the capabilities users have in the <em>Databricks Workspace</em> and allow to create different experiences for Data Scientists and Data Warehouse Specialists respectively. For most projects it is a good idea to combine both perspectives and control access in a more fine grained way, so it is ok to keep these permissions allowed by default.</p><p>Configuring <em>Admin access</em> for a user has the same effect as adding that user to the built-in Admin group. This has the effect of giving that user a <strong>full control</strong> <strong>over all of the <em>Databricks Workspace</em></strong>.</p><p><strong><em>Unrestricted cluster creation</em> is a strong permission allowing to create clusters outside of cluster policies as well as other types of <em>Computes</em></strong>. A principal with this permission is only limited by the size of a vNET and cloud provider limits (ex. Azure Quotas). It should be ideally limited to a small trusted group. You can check <a href="https://medium.com/@lukkrool/databricks-compute-costs-explained-e672f6204d5a">my following article</a> for an overview on Databricks Compute Costs.</p><h4>General Workspace Configuration</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/177/1*jnQwOuGKbgeEZcJTnVsUQA.png" /><figcaption>General Workspace Configuration — Databricks UI</figcaption></figure><p>These are the general settings available for <em>Workspace Admins,</em> it is a good idea to get through all of them. The following entries are worth checking:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/591/1*39JS2OA4tfEwHPX1YwiWQQ.png" /><figcaption>Security / Data Security / Table Access Control</figcaption></figure><p>This is a setting for non-UC Workspaces that allows to enforce permissions on metadata in legacy Hive Metastore.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*gKK5RDpAeQnY42XsI43VwQ.png" /><figcaption>Security / Egress and ingress / Store interactive notebook results in customer account</figcaption></figure><p>This setting allows to store logs on a dedicated Storage Account rather than the built-in account as an additional security, ex. to avoid password leaking</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/581/1*dZv-pMEkPQ0FjEZrq1SAkg.png" /><figcaption>Compute / SQL Warehouses</figcaption></figure><p>This setting allows to configure <em>SQL Warehouses</em>, ex. set the <em>Service Principal</em> for <em>Storage Account</em> access when not using Unity Catalog.</p><h3>Computes</h3><p><em>Computes</em> lifecycle and access is controlled differently depending on its type.</p><p><strong>All-Purpose Compute &amp; Job Compute</strong></p><p>Creation of <em>All-Purpose</em> and <em>Job Computes</em> is controlled through <em>Cluster Policies</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/689/1*F0QeTMxs9_HTVENo3CAr5w.png" /><figcaption>Default Cluster Policies — Databricks UI</figcaption></figure><p>A <em>Cluster Policy</em> specifies limits on clusters such as:</p><ul><li>How large Cluster can the user create</li><li>How many Clusters can the user create</li><li>Which Databricks runtimes can be used on the created Clusters</li><li>Is the Cluster allowed to use Photon</li><li>Security mode for the Cluster</li></ul><p><em>Cluster Policies</em> are best for interactive analysis and development where Analysts or Data Engineers need to experiment with different cluster settings or create ad-hoc computes. There is a number of default <em>Cluster Policies</em> available, with <a href="https://learn.microsoft.com/en-us/azure/databricks/admin/clusters/personal-compute"><em>Personal Compute</em></a> being always available for all the Workspace Users as a global Account setting. Users with <em>Unrestricted cluster creation</em> entitlement can create clusters outside of cluster policies.</p><p>From permission perspective, creation of All-Purpose Computes and Job Computes is the same. However while a job compute is only accessible from the workflow in which it is defined, an all-purpose (interactive) compute is persistent and its access can be controlled for different principals:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/490/1*xhFVrBHhf4L6XEQ_tIWV1A.png" /><figcaption>Cluster Permissions — Databricks UI</figcaption></figure><ul><li><strong>Can Attach To </strong>— a principal can use the cluster if it’s already running</li><li><strong>Can Restart</strong> — in addition to using the cluster, a principal can start and stop it</li><li><strong>Can Manage</strong> — can grant the above permissions to other users and edit other cluster properties</li></ul><p><em>Cluster Pools</em>, also called <em>Instance Pools</em> can speed-up start of <em>Clusters</em> by providing a set of already running VMs with pre-installed Databricks runtime:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/615/1*jwe8UuW2U6XcjAJRS7LT2w.png" /><figcaption>Cluster Pool Permissions — Databricks UI</figcaption></figure><ul><li><strong>Can Attach To </strong>— a principal can use VMs from the pool</li><li><strong>Can Manage</strong> —a principal can manage the pool</li></ul><h4>SQL Warehouses</h4><p><em>SQL Warehouses</em> are much more simple to configure. A SQL Warehouse can only be created by a principal with <em>Unrestricted cluster creation</em>. The default size configuration for a SQL Warehouse is already quite large and costs 80 DBUs.</p><p>Access to created <em>SQL Warehouse</em> is controlled slightly differently:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/662/1*w7yzBT7YdhqHKErw35vZdg.png" /><figcaption>Permissions to access a SQL Warehouse — Databricks UI</figcaption></figure><ul><li><strong>Can use </strong>— a principal can start and use the compute, but can’t stop it</li><li><strong>Can monitor</strong>— a principal can use the <em>Monitoring Tab</em> to view query history and aggregated <em>SQL Warehouse</em> statistics</li><li><strong>Can manage</strong> — can grant the above permissions to other users and modify cluster properties, ex. <em>SQL Warehouse</em> size</li></ul><h4>Model Serving Endpoints and Vector Search Endpoints</h4><p>While <em>Clusters</em> and <em>SQL Warehouses</em> are mainly used for Data Engineering and Data Science, there exist a number of newer computes designed strictly with ML in mind. <em>Model Serving Endpoints</em> allow to serve whole models, while the Vector Search Endpoints are designed specifically for <em>Retrieval-Augmented Generation (RAG).</em></p><p><em>Model Serving Endpoints:</em></p><ul><li><strong>Can view</strong> — a principal can display model’s metadata</li><li><strong>Can query</strong> — a principal can use the model</li><li><strong>Can manage</strong> — a principal can modify settings of the model</li></ul><p><em>Vector Search Endpoints:</em></p><ul><li><strong>Can use </strong>— a principal can use the endpoint</li><li><strong>Can manage</strong> — a principal can modify the endpoint</li></ul><p><strong>Apps</strong></p><p>This is a new type of compute being in preview at the moment of writing. It is going to have its own sets of permissions.</p><p>When it comes to dangers resulting from misconfigured <em>Compute</em>, they can be summarized as follows:</p><ul><li>Overprovisioning — paying for unnecessary compute</li><li>Stolen compute power — paying for and contributing to malicious activities</li><li>Stability — lack of control of properties of computes, ex. unauthorized modifications to compute size or Databricks runtime</li></ul><h3>Code &amp; other instructions</h3><p>The code or other types of instructions that can run on the <em>Computes</em> have their own sets of permissions. They can be divided into <em>Workspace Folder Elements</em> and <em>Workflows</em>.</p><p><strong>Workspace Folder Elements:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/494/1*xTo2mM41Aoxu9wfZTDGqZw.png" /><figcaption>Different kinds of objects available in Workspace Folder Tree — Databricks UI</figcaption></figure><p>Permissions across the folder tree are <strong>propagated downward</strong>, meaning that a permission set on the root folder will be active on all the child folders and objects unless overridden with a heavier permission. This allows to control the permissions across whole depth of the tree with a few sets of permissions on top-level folders. Permissions across different objects are consistent…</p><ul><li><strong>Can View</strong> — a principal can display the object</li><li><strong>Can Run</strong> — a principal can run the object</li><li><strong>Can Edit</strong> — a principal can display, run and modify the object</li><li><strong>Can Manage</strong> — a full set of permissions over the object</li></ul><p><em>…</em> with the exception of <em>Queries</em> and <em>Dashboards</em>. <em>Queries</em> can be configured in two ways — <strong>run as viewer</strong> and <strong>run as owner</strong>. With <strong>run as owner</strong>, the query is run using the owner’s permissions, so the owner should be careful of what data is being exposed with her query. An even broader exception — <strong>anyone in my account can view</strong> — is available for the Dashboards.</p><p><strong>Workflows</strong></p><p>There are two types of Workflows — <em>Jobs</em> and <em>Delta Live Tables</em> pipelines. Jobs are the universal automation abstractions that can be used to run code/instruction elements described in the previous section, as well as <em>Delta Live Tables</em> pipelines themselves. Permissions across the two are similar:</p><ul><li>Can View — a principal can see the workflow and its logs</li><li>Can (Manage) Run — a principal can operate the workflow</li><li>Can Manage — a principal can operate and manage the workflow</li><li>Is Owner — a principal <strong>can set the workflow to run using own permissions</strong></li></ul><p>What’s important to remember about the <em>Workflows </em>is that that the permissions used by a <em>Workflow </em>are often different from the permissions of the principal triggering the <em>Workflow</em> — <em>Workflows</em> are usually running using specified <em>Service Principals.</em></p><p>When it comes to the risks related with Code stored on a Databricks Workspace, it can be put into two categories. The first category is hijacking other principal’s permissions, ex. through a Job, Notebook or Query. Another risk is simply damaging the objects — this could mean production code but also other developer’s work in progress.</p><h3><strong>Secret scopes</strong></h3><p><em>Secret scopes</em> are a Databricks mechanism for storing credentials to external systems when Unity Catalog connection is not available. Secret scopes are only manageable through the Databricks API or CLI. Possible values are READ, WRITE and MANAGE. Any Databricks Principal with at least the READ permission can use the Secret scope and all the secrets stored in that scope. The danger of misconfiguring a secret scope is that a principal might be able to access external systems directly using the cached password.</p><h3>Summary</h3><p>As the saying goes, one picture is worth a thousand words.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BsgOveEMELtu0j38nooHIA.png" /><figcaption>Simplified overview of Databricks Workspace security. Critical or not obvious securables are highlighted in red.</figcaption></figure><p>With Unity Catalog and enforced Storage Account security, one could argue that <em>Databricks Workspace</em> risks can be neglected, as with CI/CD the contents of the <em>Workspace </em>can always be recreated from a repository. However if the Workspace is to be widely accessible and serve as part of a Data Platform and not only a standalone ETL engine, the following risks need to be accounted for:</p><p><strong>Hijacking other user’s permissions.</strong> While permissions for <em>Users </em>and <em>Groups</em> to use <em>Service Principals</em> are managed globally in Unity Catalog, it is possible for a <em>User</em> to locally modify a <em>Job </em>or a <em>Notebook </em>referenced by the <em>Job</em> while keeping the <em>Job’s</em> original <em>Service Principal</em>, even if the User misses permissions to use that <em>Service Principal</em> directly. Similarly, an owner of a <em>Query</em> can configure that <em>Query</em> to run using her own permissions, thus possibly bypassing UC permissions.</p><p><strong>Overprovisioning.</strong> Sky (or cloud provider quotas) is the limit when it comes to how much money can be wasted by provisioning unnecessary computes, especially with the default size of <em>SQL Warehouses</em>. As <em>Workspace</em> objects, <em>Computes</em> need to be secured locally — by Admins not being too generous with <em>Unrestricted cluster creation</em> and by preparing correct <em>Cluster Policies</em>.</p><p><strong>Damaging the Workspace objects.</strong> While all of the Workspace configuration, including security, should be deployable from the repository, it is better to avoid situations where the production environment is temporarily disabled or several hours of developer’s time are lost.</p><p>Databricks security is rather complex, even when including only <em>Databricks Workspace Security</em> and skipping other areas such as metadata security (<em>Unity Catalog</em> or <em>Hive Metastore</em>), <em>Storage Account</em> security or virtual networking. There is however no other way than to understand it and to implement a consistent security model. Databricks Workspace security itself should be configured as Infra as Code, whether through <a href="https://learn.microsoft.com/en-us/azure/databricks/dev-tools/terraform/">Terraform</a>, <a href="https://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/">Databricks CLI</a> or <a href="https://learn.microsoft.com/en-us/azure/databricks/reference/api">Databricks API</a> for consistent configuration across all of the environments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/1*2tAkkWnE6BZ7xSd7omCC5g.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=52c62c197430" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Databricks (Azure) Storage Account Connection Explained]]></title>
            <link>https://medium.com/@lukkrool/databricks-azure-storage-account-connection-explained-7ed0b6ab32ff?source=rss-121ae2e93541------2</link>
            <guid isPermaLink="false">https://medium.com/p/7ed0b6ab32ff</guid>
            <category><![CDATA[unity-catalog]]></category>
            <category><![CDATA[data-lake]]></category>
            <category><![CDATA[azure-storage-account]]></category>
            <category><![CDATA[data-lakehouse]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Łukasz Król]]></dc:creator>
            <pubDate>Fri, 25 Oct 2024 19:40:28 GMT</pubDate>
            <atom:updated>2024-11-08T12:05:59.905Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/766/1*MCSUCBB6YfhegQy8P9NC2A.png" /></figure><p><em>Databricks is not Snowflake or Fabric and requires the strongest technical and administrative background of the three. While Databricks pushes strongly to establish itself as a Universal Data Platform by encouraging its customers to use Unity Catalog and various managed data control features, its open-source Spark roots still show at times. Configuring a connection between a Databricks environment and its data has been greatly improved with Unity Catalog, but it still requires some administrative background. As with all things Databricks, it offers more control than the other platforms but requires some know-how.</em></p><h3>Core concepts</h3><p>A Databricks environment is divided into <a href="https://learn.microsoft.com/en-us/azure/databricks/getting-started/overview">Control Plane and Compute Plane</a>. There is no “Data Plane” that could be used by default — the <em>Workspace Storage Account</em> delivered with the environment is <strong>not to be used to store production data.</strong></p><p>The <em>Workspace Storage Account </em>is home to <em>DBFS — Databricks File System</em>. It is always available on a <em>Databricks Workspace </em>and is <strong>not securable </strong>— it is a “better than nothing” filesystem available as default temporary space for all Workspace Users. Tables stored on DBFS are referred to as <em>internal</em> tables, contrary to <em>external</em> tables residing on dedicated Storage Accounts. DBFS filesystem is accessible through the <strong>dbfs:/</strong> prefix.</p><p>A Databricks environment is not complete without a separate dedicated Storage Account. In case of Azure that would be <em>Data Lake Storage Gen2, </em>obtained by enabling <em>Hierarchical Namespace</em> on Storage Account creation, thus configuring the Blob Service as Data Lake, allowing to use folders and set permissions on those folders (RWX <a href="https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-access-control">ACLs</a>). DLSG2 is accessible through the <strong>abfss:/</strong> prefix.</p><p>While there are multiple ways to configure the connection between Databricks and a Storage Account…</p><ol><li><strong>Unity Catalog</strong></li><li><strong>Spark Config</strong></li><li>DBFS Mountpoints</li><li>Credential Passthrough</li></ol><p>…only Unity Catalog should be considered for new production (non-POC) environments.</p><p>Whenever making a connection from Databricks to a Storage Account, the following 3 levels need to be covered:</p><h4>Networking — can Databricks get through Storage Account’s firewall or access its Private Endpoint</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xor6w8WFd3n5NBmKgiUKww.png" /><figcaption>Three network options for connecting to a Storage Account — unsecured Public Endpoint, secured Public Endpoint (whitelisting a vNET with Service Endpoint on Storage Account firewall) and Private Endpoint. Storage Account Firewall configuration is independent of Databricks Workspace Firewall.</figcaption></figure><h4>Authentication — how does Databricks present itself to the Storage Account</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/992/1*5Mns3SCeCceFhl0ZXeIizA.png" /><figcaption>Different ways to Authenticate to a Storage Account with Databricks. Non-Entra ID methods (Shared Access Signatures and Storage Account Access Keys) are not really authentication, as they provide access to the Storage Account without any user context. Whenever dealing with passwords, Key Vault backed secret-scopes need to be used in Databricks.</figcaption></figure><p><strong>Authorization — what permissions does Databricks have on the Storage Account</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/988/1*8Z6KiSNAQ2jG2w2WncS5Cg.png" /><figcaption>Two ways of granting permissions on a Storage Account are <a href="https://learn.microsoft.com/en-us/azure/role-based-access-control/overview"><strong>Role Based Access Controls</strong></a> And <strong>A</strong><a href="https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-access-control"><strong>ccess Control Lists</strong></a>. RBACs are general Azure permissions and the lowest scope they can be applied to is the whole Storage Account. ACLs are folder-level RWX permissions. It is possible to mix both methods, keeping in mind that RBACs override ACLs.</figcaption></figure><h3>Accessing Data Lake Storage Gen2 through Unity Catalog</h3><p>While the primary feature of <em>Unity Catalog</em> is presence of one subscription-level <em>Metastore</em> in place of legacy local <em>Hive Metastores, t</em>here is also a fundamental difference in how Databricks Computes that integrate with Unity Catalog authenticate to a Storage Account.</p><p>Unity Catalog introduces the <a href="https://learn.microsoft.com/en-us/azure/databricks/connect/unity-catalog/storage-credentials"><em>Storage Credential</em></a> concept. In case of Azure, a Storage Credential is a reference to an Azure resource called <em>Databricks Access Connector</em>, which is in turn a container for a <em>Managed Identity</em> (System or User Assigned). <em>Storage Credential</em> is securable, meaning that it is possible to specify which Unity Catalog principals can use it.</p><p>Another Unity Catalog object element necessary to reference the Storage Account from Databricks is an <a href="https://learn.microsoft.com/en-us/azure/databricks/connect/unity-catalog/external-locations"><em>External Location</em></a><em>.</em> <em>External Location</em> combines the <em>Storage Credential</em> with an <strong>:/abfss</strong> path to a Data Lake Folder. It is possible to specify multiple External Locations specifying the same Storage Account as long as they don’t overlap. A Unity Catalog principal (user, group or service principal) needs UC permissions on the <em>External Location</em> to be able to use it<em>.</em></p><p>There are more Unity Catalog objects such as <em>Catalogs, Schemas </em>and <em>Volumes </em>that reference the data available through <em>External Locations</em>. These can be either <em>Managed</em> or <em>External.</em> <em>Managed </em>objects<em> </em>are stored in locations specified when defining a <em>Catalog or Schema </em>and their lifecycle is fully controlled by UC<em>. External</em> objects can be created with an explicit Data Lake <strong>:/abfss</strong> URL, as long as there exists an <em>External Location</em> that covers that URL. For full understanding of the Unity Catalog object tree, refer to the <a href="https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/">documentation</a>.</p><p>From the point of view of pure Storage Account accessibility, an <em>External Location</em> is enough to access the Storage Account in an ad-hoc manner, ex. with <em>dbutils.fs.*</em>. However, we still need to make sure that the Storage Account itself recognizes the <em>Managed Identity</em> behind our <em>External Location</em>, and that our Databricks cluster can connect at all! This means that we need to:</p><ol><li>Allow the <em>public subnet</em> used by the Databricks compute through the storage account firewall (<em>Networking -&gt; Add IP ranges to allow access from the internet or your on-premises networks.</em>).</li><li>Additionally, allow the Databricks Access Connector / Managed Identity through the firewall (<em>Networking -&gt; Specify resource instances that will have access to your storage account based on their system-assigned managed identity</em>).</li><li>Grant the Managed Identity permissions on the Storage Account (RBACs or ACLs).</li></ol><p>The approach with RBACs is simpler than that with ACLs, as it is enough to grant the Managed Identity <a href="https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles"><em>Storage Blob Data Contributor</em></a> over the whole storage account. This way we practically entrust Unity Catalog with all the contents of the Storage Account. If the Storage Account needs to be accessed independently of Databricks and Unity Catalog, we need to apply additional fine-grained security on the Storage Account, ex. with ACLs.</p><p>One last piece of the puzzle is that the User needs to use a Databricks Compute that is compatible with Unity Catalog, meaning one of the following:</p><ul><li>Shared (USER_ISOLATION)</li><li>Single User (SINGLE_USER)</li><li>SQL Warehouse</li></ul><p>A Compute is a workspace-level object, and the permissions for Users and Jobs to use Clusters and Warehouses need to be granted locally in the Workspace — they are not defined in UC.</p><p>An easy way to check if a Cluster works with Unity Catalog is to look for the Unity Catalog tag on Cluster properties:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/327/1*rjqS4xp55V1DIGKHxdIZdQ.png" /><figcaption>A small Unity Catalog Compute</figcaption></figure><p>Let’s wrap the whole connection chain up:</p><ol><li>The User is registered in Databricks Unity Catalog.</li><li>The User connects to a Workspace that is enabled for Unity Catalog.</li><li>In that Workspace, the User has permissions to connect to a Unity Catalog Cluster or SQL Warehouse, thus gaining access to the UC Metastore.</li><li>In the Metastore, the User has permissions to use an <em>External Location.</em></li><li>The <em>External Location</em> is referencing a <em>Storage Credential</em>.</li><li>The <em>Storage Credential</em> references an instance of the <em>Databricks Access Connector</em> Service.</li><li>The <em>Databricks Access Connector Service</em> contains an Azure <em>Managed Identity</em>.</li><li>The <em>Managed Identity</em> has permissions to access the Storage Account.</li><li>The Databricks Compute is allowed through the Storage Account firewall.</li></ol><p>It might be a bit confusing at first, but it is the only way to secure the data at both the schema (table) level and filesystem level — see legacy methods below. Unity Catalog itself is free, and the Databricks Computes that work with UC are priced in the same way as those on Legacy Databricks Workspaces. You can check my <a href="https://medium.com/@lukkrool/databricks-compute-costs-explained-e672f6204d5a">story</a> for an overview of Databricks pricing.</p><h3>Legacy methods</h3><p>One might often encounter systems that implement connection to the Storage Account using one of the legacy approaches.</p><h4>Spark Config</h4><p>This is a <a href="https://docs.gcp.databricks.com/en/connect/storage/azure-storage.html">pure Spark connection method</a>. The differences with respect to Unity Catalog can be summarized as follows:</p><ul><li>The connection is configured per-Cluster in Cluster’s Spark Config, or for all SQL Warehouses on a legacy Workspace through <em>Global Warehouse Config</em>. This is different than a Metastore-level UC connection usable in multiple Workspaces.</li><li>A Service Principal is used instead of a Managed Identity.</li><li>A Key Vault backed secret scope and secret rotation procedure should be implemented for the Service Principal Key.</li><li>When interacting with a Storage Account, all users having access to the Service Principal share the same filesystem permissions. This situation is different than with UC’s Managed Identity, as it is not possible to control permissions through multiple <em>External Locations</em>. The only option is to globally allow or disallow filesystem operations for a user group with the legacy <a href="https://learn.microsoft.com/en-us/azure/databricks/data-governance/table-acls/any-file">ANY FILE</a> permission.</li></ul><p>Overall this approach is inferior to UC, as the users of a Cluster that connects using this method share the same filesystem permissions as that of the <em>Service Principal </em>used for the connection. If ANY FILE permission is granted, they can bypass <em>Hive Metastore</em> table permissions and work with underlying files directly. If it is not granted, they cannot create their own tables or work with the filesystem. However the approach is valid, if the limitations are not a problem from requirements point of view. Even in Databricks Workspaces that are enabled for Unity Catalog, it is still possible to use legacy Clusters that can connect with this method, they won’t have access to UC Metastore though and will be limited to local <em>Hive Metastore</em>. Databricks Security is a wide topic and inolves securing file layer (physical data), schema layer (metadata) and the <a href="https://medium.com/@lukkrool/databricks-workspace-security-explained-52c62c197430">Databricks Workspace</a> itself.</p><h4>DBFS Mountpoints</h4><p><a href="https://learn.microsoft.com/en-us/azure/databricks/dbfs/mounts">Mountpoints </a>are a deprecated access pattern in which an <strong>:/abfss</strong> (or other Cloud) path is permanently attached in DBFS, much like a disk can be mounted on a Linux filesystem. The problem with this approach is that DBFS is not securable, is available for everyone who can login into the Workspace, and is not available from any other Workspace. Mountpoints persist as long as they are not unmounted, and cache the credentials of the account that was used for the mount. In this approach the filesystem is not secured at all — anyone who can use the Workspace has access to the mountpoint.</p><h4>Credential passthrough</h4><p><a href="https://learn.microsoft.com/en-us/azure/databricks/archive/credential-passthrough/adls-passthrough">Credential passthrough</a> is another deprecated access pattern. In this approach, the identity of the user connected to the Databricks Workspace was passed directly to the Storage Account. It was designed for users working with their data interactively and required them to have permissions (RBAC/ACL) directly on the Storage Account. It is not possible to use it with Workflows, and it is incompatible with <em>Hive Metastore</em>. Table ACLs. While it allowed to secure the data on the filesystem, it provided no protection for the contents of local <em>Hive Metastore</em>.</p><h3>Summary</h3><p>While it would seem that there is a lot of access methods available, there isn’t really much of a choice when it comes to how a Storage Account can be connected to a Databricks Workspace. The only way that allows to secure the schema layer (metadata) and file layer (physical data) at the same time is Unity Catalog, either with the <em>Managed </em>or<em> External</em> approach. Alternatively, if the schema does not have to be shared between the workspaces and you wish to stay closer to original Spark, using Clusters Spark Config is still an option — at least for now.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*GzOuJZTqeT4rpCIT7rJWlA.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7ed0b6ab32ff" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Databricks Compute Costs Explained]]></title>
            <link>https://medium.com/@lukkrool/databricks-compute-costs-explained-e672f6204d5a?source=rss-121ae2e93541------2</link>
            <guid isPermaLink="false">https://medium.com/p/e672f6204d5a</guid>
            <category><![CDATA[price]]></category>
            <category><![CDATA[cost]]></category>
            <category><![CDATA[estimations]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Łukasz Król]]></dc:creator>
            <pubDate>Mon, 14 Oct 2024 18:58:43 GMT</pubDate>
            <atom:updated>2024-11-08T12:03:06.425Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zx4b0EneYv5jKTOgKoPnLg.png" /></figure><p><em>It is usually a good idea to estimate how much a given system will cost in terms of cloud infrastructure and data processing pipelines. One shouldn’t be too obsessed with infrastructure costs though — people’s time is much more valuable than CPU time. Being too frugal when it comes to cloud infrastructure will increase your overall cost. Nevertheless you need to be able to predict the evolution of cost of your system over time to avoid unpleasant surprises. It might also be your customer’s requirement — not necessarily to know the exact value, but to assess your level of control of the costs.</em></p><h3>Scope</h3><p>Cost predictions and comparisons are often done to explore different alternatives. These might include:</p><ul><li>data platform vendors (ex. Databricks, Snowflake, Fabric)</li><li>general cloud service vendors (ex. Azure, AWS, Google)</li><li>reliance on additional services (ex. Storage Accounts, Data Factory, Power BI, Virtual Networking)</li><li>comparing cloud with on-premises</li><li>hybrid scenarios involving combinations of the above</li></ul><p>The selection process is complex, and it is important to keep in mind that cost is not the only factor. A more expensive architecture might be chosen for any of the following reasons:</p><ul><li>skillset of the team</li><li>SLA requirements</li><li>data classification — is the data allowed to leave the country, or be lifted to the cloud at all</li><li>size and availability of the on-prem infrastructure team</li><li>avoiding vendor lock in</li></ul><p>In this entry I’ll focus <strong>exclusively on Databricks costs</strong>. Such estimate can be included as part of an estimate of a larger system, but might be already sufficient in a simplistic scenario as the following (you can check <a href="https://medium.com/@lukkrool/databricks-azure-storage-account-connection-explained-7ed0b6ab32ff">my story</a> for details on how to connect Databricks to a Storage Account):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/726/1*XODG6fOS5xFVA08v5-H6Sw.png" /><figcaption>Simple Cloud Architecture for Databricks on Azure</figcaption></figure><p>Unless you plan to use streaming or use lots of file-arrival triggers for Databricks workflows (which generates a lot of metadata operations on the storage account), non-Databricks costs are neglectable in the bigger picture.</p><h3><strong>Cost structure</strong></h3><p>The basic pricing unit in Databricks is a <strong>DBU — A Databricks Unit</strong>. This is the fee for using Databricks — virtual machines needed to run the <strong>non-serverless</strong> computes are <strong>charged separately</strong> depending on the vendor. So the difference in total price between serverless and non-serverless is smaller than would seem from just looking at DBUs.</p><p>DBUs cost differently depending on the type of computations you run. The <a href="https://www.databricks.com/product/pricing">following</a> Databricks pricing page describes different DBU offerings:</p><ul><li><a href="https://www.databricks.com/product/pricing/datascience-ml">Data Science &amp; Machine Learning</a><br> All-Purpose Clusters<br> All-Purpose Clusters with Photon<br> Serverless Clusters</li><li><a href="https://www.databricks.com/product/pricing/jobs">Workflows &amp; Streaming</a><br> Job Clusters<br> Job Clusters with Photon<br> Serverless Clusters<br> Delta Live Tables</li><li><a href="https://www.databricks.com/product/pricing/databricks-sql">Data Warehousing</a><br> SQL Warehouse Classic<br> SQL Warehouse Pro<br> SQL Warehouse Serverless</li><li><a href="https://www.databricks.com/product/pricing/foundation-model-serving">Generative AI</a><br> Pay-Per-Token<br> Provisioned Throughput</li></ul><p>We already see 12 different types of computes! To make it more complex, depending on the cloud vendor, the computes can be <em>Standard</em>, <em>Premium </em>or <em>Enterprise</em> with DBUs priced differently according to the Databricks Workspace tier. In this article I focus on <em>Premium</em>. As of the moment of writing, <em>Standard</em> misses security features like <a href="https://docs.databricks.com/en/data-governance/table-acls/index.html">table ACLs</a> or <a href="https://www.databricks.com/product/unity-catalog">Unity Catalog</a> and <em>Enterprise</em> is only available on AWS.</p><p>So, what roles do these computes play in a project?</p><h4>All-Purpose Clusters</h4><p>These are the workhorses of Databricks and are the closest to open-source Spark. All-Purpose Clusters are <em>interactive</em> clusters to which you connect with your Python, SQL, Scala or R notebooks. Though suggested for Data Science and ML Specialists, they can also significantly speed-up the work of Data Engineers if they write complicated ETLs and prefer on-line over off-line development.</p><p>All-Purpose clusters come in different security-related flavors like <em>Unrestricted</em>, <em>Single User</em>, or <em>Shared (Unity Catalog or Legacy Table ACL). </em>Shared is a good default option. Databricks security is a bigger topic and deserves an article of it’s own.</p><p><em>Photon</em> is an optional feature allowing to use optimized binaries for twice the DBUs. This might be useful if running time is critical, or you manage to achieve more than twice the speed you get without <em>Photon</em>.</p><p>An all-purpose cluster runs with a selected version of Databricks runtime. This run-time can be <em>Standard</em> or <em>ML </em>(don’t confuse with <em>Standard</em> Workspace tier). ML Runtimes are pre-loaded with common Machine Learning libraries and are suitable for training AI Models on nodes with GPUs.</p><p>All-purpose Clusters can run both interactive notebooks and job workflows.</p><h4><strong>Job Clusters</strong></h4><p><em>Job Clusters</em> are like <em>All-Purpose Clusters </em>except they cannot be connected to from an interactive session. They are also not defined globally in the workspace — instead they are contained in the definition of a <em>Databricks Workflow</em> or a <em>Delta Live Tables</em> pipeline. They are created when a workflow starts and terminated afterwards. Job Cluster DBUs are roughly 2 times cheaper than All-Purpose DBUs. So, why the option to use all-purpose compute for <em>Workflows</em>? There are some cases where it can be worth the extra cost, like multiple workflows reusing the same compute to avoid start-up latency or keeping the compute open after the workflow terminates to load a Business Intelligence model immediately.</p><h4>SQL Warehouses</h4><p>SQL Warehouses are the computes optimized for handling SQL queries. They offer increased performance for SQL, while they can’t handle other languages. They come as <em>Classic</em>, <em>Pro</em> and <em>Serverless</em>. You can think of Databricks SQL Warehouse (especially the <em>Serverless</em> version) as the closest thing to a Snowflake Warehouse in the Databricks World. They are typically used to load Business Intelligence tools or to handle ad-hoc SQL queries, but Pro can be also used with Notebooks.</p><p>Security-wise SQL Warehouses work as Databricks Shared Computes (Unity Catalog or Legacy Table ACL).</p><h4>Serverless Computes</h4><p>Serverless Computes are the versions of Databricks computes with Virtual Machines running in Databricks Cloud Subscription rather than your own subscription. The difference can be summarized as:</p><ul><li>DBUs are more expensive.</li><li>You are not directly charged for VM’s needed to run Databricks.</li><li>Computes behave like they start in a few seconds.</li><li>Network architecture needs to be configured to open for Databricks Virtual Networks where the “serverless servers” run.</li></ul><p>These are the computes to choose if few minutes of waiting for the compute to start is not acceptable, ex. a Business Intelligence Tool running in Direct Query mode (SQL pushdown — executing joins, filters and aggregations in the backend).</p><h4><strong>AI Model Training &amp; Serving</strong></h4><p>These are the specialized computes to serve trained AI models, or to perform massively parallel model training.</p><p>AI model training can be still performed the classic way, that is using an All-Purpose Cluster with GPU nodes.</p><h3>Sample estimation</h3><p>Let us put this information together to estimate costs for running a small to medium size project:</p><ul><li>Azure, West Europe, Databricks Premium</li><li>2–3 Data Engineers</li><li>1–2 Data Scientists</li><li>3h batch running once per day</li><li>daily load of aggregated data of a few hundred million records to a Business Intelligence tool</li><li>a few queries per day on detailed data of several billion records</li><li>external business intelligence tool (Power BI, Tableau, etc.) not considered in estimation</li></ul><p>To simplify the calculations, we are going to assume that:</p><ul><li>Databricks is the only source of costs (we exclude Storage Account, Business Intelligence, Key Vaults, DevOps pipeline VMs)</li><li>Nightly batch runs on a single environment (ex. Production)</li><li>Data Engineers work on a single environment (ex. Development)</li><li>Data Analysts and Scientists work on a single environment (ex. ML Environment allowing to develop models and interactively explore the production data)</li><li>Business Intelligence Model is being loaded once per day form the <em>Gold Layer</em></li><li>Business Intelligence Model generates a few queries to the <em>Gold Layer</em> per day through direct query.</li></ul><p>We are going to need the following computes:</p><h4>All-purpose cluster for Data Engineers</h4><p>We assume that all Data Engineers will use the same all-purpose cluster and keep it running for 8 hours per day during working hours. Assuming 20 working days per month this yields 160 hours.</p><figure><img alt="A Standard_D16ds_v5 cluster with recent runtime (15.4 LTS), without photon, with autoscaling (2–4 worker nodes) is 12–20" src="https://cdn-images-1.medium.com/max/388/1*K0biRAl1PGa9A3RApcpzYw.png" /><figcaption>DBU summary for All-purpose cluster</figcaption></figure><p>Let’s say that the extra 2 nodes are a reserve and that for the majority of time the cluster will run with 1 driver node and 2 worker nodes, so 12 DBUs. An all-purpose DBU in Azure / West Europe is 0.55$, so the estimated cost for DBUs is:</p><p><strong>20(day)</strong> x <strong>8(hour)</strong> x <strong>12(DBU)</strong> x <strong>0.55($/DBU) = 1056 $</strong></p><p>We still need to pay for the Virtual Machines. We can get an estimate for the VMs using <a href="https://azure.microsoft.com/en-us/pricing">Azure Price Calculator</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ANPEpO8LzVYnJ0APJPGeBA.png" /></figure><p>Combining DBUs and VMs yields a total of <strong>1578.24$ per month</strong> for the All-purpose cluster.</p><p>If using Azure, we can calculate the full DBU+VM cost directly by pricing a whole Databricks service:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5kaD6EyXDvhkleVgn3U_ag.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J_Kd6ecQGebhuvEI5qTVQg.png" /></figure><p>It is however a good idea to be able calculate the costs directly to fully understand the price structure and different kinds of computes (ex. to not get confused by Microsoft and Databricks documentation calling the computes slightly differently, or by the inconsequential DBU calculation showing per-node DBUs instead of total DBUs).</p><h4>Job cluster for nightly processing</h4><p>In our simple scenario there’s no point in running the regular production workflow with an all-purpose clusters. Instead we’re going to use a job cluster with the same specification as the all-purpose cluster so that the Data Engineers have the same experience with the performance as in the production workload. We assume that the batch will run for 3 hours per day, so the monthly DBU cost is going to be:</p><p><strong>30(day)</strong> x 3<strong>(hour)</strong> x <strong>12(DBU)</strong> x <strong>0.30($/DBU) = 324 $</strong></p><p>of course, we still need to pay for the VMs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w5X1bJPTaPfvGr4qYjFtcw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iHcGztC2aDxJVqTy7ScutA.png" /></figure><p>If Data Engineers decide to use Delta Live Tables instead of Notebooks, the price calculation for the DLT Compute will be the same as for Job Cluster, however it should be expected to be running more often as they would use it for development and not only to run the production workload.</p><h4>Serverless SQL Warehouse for Data Analysts and Business Intelligence Users</h4><p>We are going to initially channel all ad-hoc and Business Intelligence queries through the smallest available (2X-small ~ 4 DBU/h) <em>Serverless SQL Warehouse</em>. The nature of this workload except the refresh of the Business Intelligence model is less predictable — let’s assume between 0.5 and 2 hours per work day. In the extreme scenario of having the Serverless Warehouse running for several hours per day we could even replace it with a Pro or Classic Warehouse that would be permanently running on fixed hours.</p><p><strong>20(day) x (0.5 to 2)(hour) x 4(DBU) x 0.91($/DBU) = (36.4 to 145.6) $</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DJuV92uoUoeKFxjLMVblHg.png" /></figure><h4>GPU-enabled all-purpose compute for Data Scientists</h4><p>Our Data Scientists might be interested in retraining and adjusting Large Language Models to our own data, or even training their own models from scratch. We can provide them with a large GPU-enabled single-node cluster as a starting point for their activities:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/460/1*eGEEodCStII7IQvNNWjxZg.png" /><figcaption>DBU summary for single-node GPU Cluster</figcaption></figure><p>Assuming that it would be kept running for 4 hours per work day, we can expect the following DBU costs…</p><p><strong>20(day) x 4(hour) x 10(DBU) x 0.55($/DBU) = 440 $</strong></p><p>…and the following total costs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wbaDJVWSpo9SNLammG7EWQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mi5WW-gcjk3zQpNuGJ-cCQ.png" /></figure><p>This is an assumption for an ML POC. If the Data Scientists are really up to something, they might want to spend several days on serious model training.</p><h3>Summary</h3><p>The estimate as the one presented in this article is reasonable for a system which is small in terms of number of involved people. It is sufficient to handle data ranging in several billion records, as long as it’s only the <strong>volume</strong> that is big, not necessarily <strong>velocity</strong> and <strong>volatility</strong> — velocity would call for use of <em>Spark Streaming</em> or <em>Delta Live Tables, </em>while <em>variety </em>could mean having to involve more Data Engineers, Data Architects or rely on AI.</p><p>It is however already possible draw some conclusion from the cost structure:</p><ul><li>Data Engineering Development: 1600 $</li><li>Nightly Batch: 600 $</li><li>Business Intelligence, Data Analysts: 150$ (Databricks only)</li><li>ML Engineering POC: 800 $</li></ul><p>Most costs come from interactive compute, that is clusters keeping active sessions with which the human actors interact. It is by no means a suggestion to have a policy to reduce that time — it’s simply an information on where the costs are. It is the work that is actually needed to make the project happen. The production environment is cheaper than the development, as it only performs actual computation and never has to wait for people — cheap predictable job compute or serverless compute running for short unpredictable bursts.</p><p>It has to be made clear that this structure doesn’t have to scale with the size of the project. Cost structure in a streaming project or a project involving several Data Engineers or intensive AI training would be much different. Finally, in Databricks the infrastructure including <em>Computes</em> is by no means fixed and allows to delegate their creation and management to Developers. You can check <a href="https://medium.com/@lukkrool/databricks-workspace-security-explained-52c62c197430">my other article</a> for details on controlling Compute creation.</p><p>Performing a price estimation can give you an idea of the range of expenses needed to keep the system running. It requires understanding of different types of computes available in Databricks, and some intuition on the expected performance and patterns of of the workloads. It should be performed in a a few scenarios, as it has to be based on assumptions like expected time per day the system will have to be running. The range of these scenario will give you an idea of the mean expected cost and range of uncertainty.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/985/1*AnwIKSe9Y80SgaEGcusR5w.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e672f6204d5a" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>