<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by JD Braun on Medium]]></title>
        <description><![CDATA[Stories by JD Braun on Medium]]></description>
        <link>https://medium.com/@jdbraun?source=rss-c22ed87dd65------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*k5S0p5rJqPulnyWY1eCxKw.jpeg</url>
            <title>Stories by JD Braun on Medium</title>
            <link>https://medium.com/@jdbraun?source=rss-c22ed87dd65------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 14:14:46 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jdbraun/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Isolake, Revisited: Isolating Databricks Serverless Workloads from the Public Internet]]></title>
            <link>https://medium.com/databricks-platform-sme/isolake-revisited-isolating-databricks-serverless-workloads-from-the-public-internet-1fe58f107611?source=rss-c22ed87dd65------2</link>
            <guid isPermaLink="false">https://medium.com/p/1fe58f107611</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[JD Braun]]></dc:creator>
            <pubDate>Fri, 26 Sep 2025 16:00:27 GMT</pubDate>
            <atom:updated>2025-09-26T16:13:38.762Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tWze5ujtjhPFF7mCK7bMKw.png" /><figcaption>Image: The Isolated Lakehouse: Foundation for no Public Internet Access on Databricks.</figcaption></figure><h3><strong>TL;DR — Extending Isolake for Serverless</strong></h3><ul><li><strong>Enforce egress control:</strong> Create a restrictive <a href="https://docs.databricks.com/aws/en/security/network/serverless-network-security/manage-network-policies">serverless network policy</a>, start in dry-run mode to analyze impact in the <a href="https://docs.databricks.com/aws/en/admin/system-tables/outbound-network">network access events</a> system table, then attach it to target workspaces.</li><li><strong>Go private by default:</strong> Use <a href="https://docs.databricks.com/aws/en/security/network/serverless-network-security/#what-is-a-network-connectivity-configuration-ncc">network connectivity configuration</a> (NCC) <a href="https://docs.databricks.com/aws/en/security/network/serverless-network-security/pl-to-internal-network">private endpoint rules</a> to reach cloud resources over PrivateLink using serverless compute.</li><li><strong>Mitigate supply-chain risk:</strong> Pin third-party packages or scan and mirror them to <a href="http://medium.com/r?url=https%3A%2F%2Fdocs.databricks.com%2Faws%2Fen%2Flibraries%2Fvolume-libraries">Unity Catalog Volumes</a> (or your internal artifact repository) instead of pulling from public repositories.</li><li><strong>Aim for zero-internet:</strong> Assess usage patterns and plan a path to no direct internet access for serverless.</li></ul><h3>Why now?</h3><p>Two years ago this December, we kicked off this entire <a href="https://medium.com/databricks-platform-sme">Databricks Platform Medium Publication</a> with the post: <a href="https://medium.com/databricks-platform-sme/isolake-a-simplistic-deployment-design-to-an-isolated-databricks-lakehouse-on-aws-c0f98b5bbba0">Isolake — A simplistic deployment design to an isolated Databricks Lakehouse on AWS</a>.</p><p>In that maiden blog post we discussed how Databricks administrators or infrastructure and security teams could isolate Databricks from the public internet by using Amazon Web Services (AWS) PrivateLink. Effectively, this would allow classic clusters to not need any outbound internet connectivity and inbound connections from users or machines could be configured to only access Databricks over the cloud backbone.</p><p>Since then, there has even been a <a href="https://www.youtube.com/watch?v=XFQSNRgXTbc">YouTube video</a> on the topic, and not to mention our <a href="https://github.com/databricks/terraform-databricks-sra">Security Reference Architecture (SRA) — Terraform Templates for AWS</a> implemented the Isolake architecture as the default deployment methodology.</p><p>However, there have been three unrelated events that pushed me to write this as a follow-up to the original post.</p><ul><li><strong>General availability of serverless security controls.</strong> Databricks has released two core features that can allow you to control <strong>what</strong> resources serverless compute can connect to (network policies) and <strong>how</strong> serverless connects to resources (NCC). These features provide the same network controls and integration capabilities that are available using cloud primitives for classic clusters.</li><li><strong>Release of serverless workspaces for new and existing Databricks accounts.</strong> Databricks has released fully serverless workspaces on AWS. If someone gets started with Express Setup, then your first workspace will not have classic cluster available, which makes the back-end connectivity section of the original Isolake post irrelevant.</li><li><strong>Shai-Hulud Supply Chain Attack.</strong> This was one of the largest and most dangerous software supply chain attacks involving JavaScript packages, publicly known as <a href="https://www.cisa.gov/news-events/alerts/2025/09/23/widespread-supply-chain-compromise-impacting-npm-ecosystem">Shai-Hulud</a>.</li></ul><p>The Shai-Hulud attack was the final motivator to write this follow-up to my original post in 2023.</p><p>Since September 14th, 2025, GitHub has removed over 500 compromised packages, and npm continues to block uploads containing the malware’s IoCs (indicators of compromise), showing the scale of the event.</p><blockquote><strong><em>I cannot stress enough how important it is that we continue to make strides in isolating data platforms from the public internet.</em></strong></blockquote><p>While Shai-Hulud was relegated to npm packages, it isn’t out of the realm of possibility that a similar incident could impact third-party packages hosted on PyPI or other public repositories used primarily by data professionals.</p><p>Supply-chain attacks, typosquatting, poisoned AI models, unapproved APIs, the list is long. The public internet’s attack surface is simply too large to allow unrestricted egress or blanket approvals to third-party sites (e.g., all of PyPI).</p><p>Given the severity of Shai-Hulud, availability of fully serverless workspaces, and developments in security controls for serverless compute, this is the perfect opportunity to:</p><ul><li>Revisit the core components of the original Isolake architecture.</li><li>Walk through how you can isolate serverless compute from the public internet using network policies.</li><li>Provide a pattern to connect serverless compute to cloud resources over AWS PrivateLink using private endpoints in the NCC.</li></ul><p>Now, let’s head on back to our quiet house out by the lake.</p><p><strong>Additional Resources:</strong></p><ul><li>If you’re a Databricks administrator or security architect, read <a href="https://www.databricks.com/blog/unified-approach-data-exfiltration-protection-databricks">A Unified Approach to Data Exfiltration Protection on Databricks</a>! The story doesn’t stop at “no public internet”; make sure your broader controls are covered. (Applies across all Databricks clouds.)</li><li>If you’re just getting started with Databricks, check out the <a href="https://medium.com/databricks-platform-sme/six-practical-steps-to-lock-down-your-databricks-workspace-after-getting-started-with-express-setup-46f1ed646f25">Six Practical Steps to Lock Down Your Databricks Workspace After Getting Started with Express Setup</a>! This blog will make security a day zero priority.</li></ul><h3>Revisiting How to Isolate (Iso) the Databricks Lakehouse (Lake)</h3><p><strong><em>Before we continue, as with the first iteration of Isolake, this architecture assumes that a customer is on Databricks’ enterprise tier.</em></strong></p><p>There were two core architecture components of the original Isolake architecture for Databricks.</p><ul><li><strong>Back-end Connectivity — Classic Clusters to Databricks and AWS</strong></li><li><strong>Front-end Connectivity — Users or Machines to Databricks</strong></li></ul><p>Let’s break both of these down by walking through a short summary and outlining any changes since the original posting.</p><p><strong>Back-end Connectivity — Classic Clusters to Databricks and AWS:</strong></p><p><strong>What is the Back-End Connectivity Architecture?</strong></p><p>When a user or a machine initiates creating a classic cluster in Databricks via the workspace UI or API, Databricks assumes a cross-account role to create the underlying EC2 instances on the initiator’s behalf.</p><p>These clusters then need to make connections with Databricks and three AWS services: AWS Security Token Service (STS), Amazon Kinesis, and Amazon S3. Databricks, STS, and Kinesis connect via interface endpoints, while S3 uses a gateway endpoint.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FdNlqDdlJxzGFaDlDIP8og.png" /><figcaption>Image: Architecture for Back-End Private Connectivity.</figcaption></figure><p><strong>What’s changed?</strong></p><p>This back-end architecture has remained entirely the same from a networking perspective.</p><p>From a Databricks perspective, the only major change is that Databricks now supports <a href="https://docs.databricks.com/aws/en/data-governance/unity-catalog/disable-hms">disabling the legacy Hive Metastore</a> entirely. At the time of the original blog post, this wasn’t an option, so there were additional steps needed to use Apache Derby on classic clusters. This is no longer required.</p><p><strong>Front-end Connectivity — Users or Machines to Databricks:</strong></p><p><strong>What is the Front-End Connectivity Architecture?</strong></p><p>In a standard setup, when a user or machine attempts to access a Databricks workspace, they’ll be routed to a public IP address. Databricks allows you to instead route all traffic to the workspace UI and subsequent APIs via front-end PrivateLink.</p><p>Instead of routing to a public IP, network administrators will <a href="https://medium.com/@alyssonmarquesdesouza/configuring-dns-resolution-for-private-databricks-workspaces-aws-1341abf90139">re-route the traffic to &lt;region&gt;.privatelink.databricks.com</a> to the IP of the private endpoint in AWS through Direct Connect or Site-to-Site VPN, keeping traffic on the cloud backbone.</p><p>This network path is enforced in Databricks through the Private Access Setting (PAS) on the workspace.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*oGXfag6XRo5Izj6b.png" /><figcaption>Image: Architecture for Front-End Private Connectivity.</figcaption></figure><p><strong>What’s changed?</strong></p><p>Nothing has changed involving front-end PrivateLink since the original post.</p><p>I want to clarify that in the original blog post, we positioned Amazon AppStream 2.0 in front of Databricks because the infrastructure we built this architecture on lacked Direct Connect or Site-to-Site networking configurations. I want to be clear that AppStream should be seen as optional if pre-existing network paths from the user or machine to AWS are already configured.</p><p>It should be noted that as of September 2025, Databricks serverless workspaces do not support front-end PrivateLink. In lieu of front-end PrivateLink, users can configure IP ACLs. For more information, be sure to check out the blog: <a href="https://medium.com/databricks-platform-sme/six-practical-steps-to-lock-down-your-databricks-workspace-after-getting-started-with-express-setup-46f1ed646f25">Six Practical Steps to Lock Down Your Databricks Workspace After Getting Started with Express Setup</a>.</p><h3>Isolating Serverless Compute from the Public Internet</h3><p><a href="https://docs.databricks.com/aws/en/security/network/serverless-network-security/manage-network-policies"><strong><em>Documentation</em></strong></a><strong><em> | </em></strong><a href="https://docs.databricks.com/api/account/networkpolicies/createnetworkpolicyrpc"><strong><em>API</em></strong></a><strong><em> | </em></strong><a href="https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/account_network_policy"><strong><em>Terraform</em></strong></a><strong><em> | </em></strong><a href="https://github.com/databricks/terraform-databricks-sra/blob/main/aws/tf/modules/databricks_account/network_policy/main.tf"><strong><em>SRA</em></strong></a></p><p>When we wrote the original blog post, we focused on how simple the overall architecture was.</p><p>I’m excited to say that with serverless compute, it’s even simpler than before. To dictate what resources serverless compute can talk to, Databricks has released <a href="https://docs.databricks.com/aws/en/security/network/serverless-network-security/manage-network-policies">network policies</a> for serverless egress controls.</p><p>This is a single pane of glass that can be used to control serverless egress for any number of workspaces within a single account.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xO8aWn8yRAd1U17Bi43Mgg.png" /><figcaption>Image: Network Policy Configuration Showing Restrictive Egress Controls</figcaption></figure><pre># Terraform Documentation: https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/account_network_policy<br><br>resource &quot;databricks_account_network_policy&quot; &quot;restrictive_network_policy&quot; {<br>  account_id        = var.databricks_account_id<br>  network_policy_id = &quot;${var.resource_prefix}-np&quot; # Must not be more than 32 characters<br><br>  egress = {<br>    network_access = {<br>      restriction_mode = &quot;RESTRICTED_ACCESS&quot;<br>      policy_enforcement = {<br>        enforcement_mode = &quot;ENFORCED&quot;<br>      }<br>    }<br>  }<br>}</pre><p>Network policies allow you to configure allowed egress rules for both allowed domains (e.g., google.com) and storage destinations (e.g., S3 buckets).</p><p>If you’re worried about the implications of enabling the restrictive policy on an existing workspace, then you can use the policy enforcement mode to select a dry run option. Any violating outbound connections will be logged in the <a href="https://docs.databricks.com/aws/en/admin/system-tables/outbound-network">network access events system table</a>.</p><h3>Outbound Connectivity for Serverless Compute to Private Resources</h3><p><a href="https://docs.databricks.com/aws/en/security/network/serverless-network-security/pl-to-internal-network"><strong><em>Documentation</em></strong></a><strong><em> | </em></strong><a href="https://docs.databricks.com/api/account/networkconnectivity/createnetworkconnectivityconfiguration"><strong><em>API</em></strong></a><strong><em> | </em></strong><a href="https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/mws_network_connectivity_config"><strong><em>Terraform</em></strong></a><strong><em> | </em></strong><a href="https://github.com/databricks/terraform-databricks-sra/blob/main/aws/tf/modules/databricks_account/network_connectivity_configuration/main.tf"><strong><em>SRA</em></strong></a></p><p>Preventing any outbound internet connectivity from serverless compute is great, but when I’ve talked about this before, there is always a follow-up question, “how do I access my cloud resources then?”</p><p>It’s a fair question. When using classic clusters, the cloud resources are provisioned in your cloud account where you control the networking patterns that they follow. For example, you can easily peer the workspace’s classic compute virtual private cloud (VPC) with the VPC of an Amazon Relational Database Service (RDS) instance.</p><p>However, for serverless, compute is deployed in your Databricks account in the Databricks network. That meant that connections from serverless had to be made to your private resources through the public internet, which required changing the security posture of your underlying resource.</p><p>That was, until recently!</p><p><a href="https://docs.databricks.com/aws/en/security/network/serverless-network-security/#what-is-a-network-connectivity-configuration-ncc">NCC</a> now allows you to configure <a href="https://docs.databricks.com/aws/en/security/network/serverless-network-security/pl-to-internal-network">PrivateLink connectivity</a> between serverless and your private cloud resource through private endpoint rules.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BT7zjFe4Dk1YpftFfyqfHg.png" /><figcaption>Image: Architecture Diagram for Serverless PrivateLink Connectivity</figcaption></figure><p>When you configure private endpoint rules in the NCC object, you’ll specify the domain name as well as the corresponding endpoint name. This will effectively route all requests to the domain through a VPC endpoint, instead of out the public internet.</p><p>You’ll see this on display if you run the following command after the status of the endpoint rule has changed to ESTABLISHED.</p><pre>%sh<br><br>nslookup &lt;endpoint_name&gt;</pre><p>This command’s output will be a private IP address found within the Databricks network, instead of the same IP you’d see if you did the same command in your console.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/615/1*TvNOv_WotUQ6d6aaXcTOLA.png" /><figcaption>Image: The Private Endpoint Rule UI with Example Values.</figcaption></figure><p>In the context of Shai-Hulud-style supply chain incidents, if you’re not interested in using <a href="https://docs.databricks.com/aws/en/libraries/volume-libraries">Unity Catalog volumes</a> as your source of vetted third-party packages, you can create private artifactories that can be set up to have serverless compute pull directly from them over PrivateLink.</p><p>Serverless compute on Databricks will utilize gateway endpoints to access in-region S3 buckets. You can directly configure <a href="https://docs.databricks.com/aws/en/security/network/serverless-network-security/pl-aws-resources">private connectivity to S3 using a dedicated interface endpoint</a>.</p><p>The patterns for private connectivity don’t stop there! Be sure to check out this two-part series on Private and Dedicated Connectivity Patterns for Databricks Serverless Using PrivateLink (<a href="https://community.databricks.com/t5/technical-blog/private-and-dedicated-connectivity-patterns-for-databricks/ba-p/91134">Part 1</a>, <a href="https://community.databricks.com/t5/technical-blog/private-and-dedicated-connectivity-patterns-for-databricks/ba-p/91690">Part 2</a>).</p><h3>A Very Familiar Sight — Isolake Revisited Two Years Later</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PP8P2I2RUUMG-pyHc8o6Fw.png" /><figcaption>The Updated Isolated Lakehouse Architecture</figcaption></figure><p>After revisiting Isolake nearly two years later, you can see the picture still looks very much the same.</p><p>Serverless egress controls and the network policies make it extremely simple to isolate from public internet access, while the network connectivity configuration gives you the ability to connect to private resources without opening any firewalls or changing the security posture of any downstream assets.</p><p>Whether you’re just getting started with serverless workspaces or run an enterprise Databricks set-up, both of these features are readily available to you to ensure that your serverless workloads are not interacting with the public internet in any capacity.</p><p>Until next time, I’ll be squirreled away in my Isolated Lakehouse away from any giant space sand worms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZgfaWERBJ0mskDfK" /><figcaption>Image: Pixel Image of Desert Sand Worm and Isolated Lakehouse. Generated by ChatGPT.</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1fe58f107611" width="1" height="1" alt=""><hr><p><a href="https://medium.com/databricks-platform-sme/isolake-revisited-isolating-databricks-serverless-workloads-from-the-public-internet-1fe58f107611">Isolake, Revisited: Isolating Databricks Serverless Workloads from the Public Internet</a> was originally published in <a href="https://medium.com/databricks-platform-sme">Databricks Platform SME</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Six Practical Steps to Lock Down Your Databricks Workspace After Getting Started with Express Setup]]></title>
            <link>https://medium.com/databricks-platform-sme/six-practical-steps-to-lock-down-your-databricks-workspace-after-getting-started-with-express-setup-46f1ed646f25?source=rss-c22ed87dd65------2</link>
            <guid isPermaLink="false">https://medium.com/p/46f1ed646f25</guid>
            <category><![CDATA[platform]]></category>
            <category><![CDATA[security]]></category>
            <dc:creator><![CDATA[JD Braun]]></dc:creator>
            <pubDate>Thu, 29 May 2025 03:14:11 GMT</pubDate>
            <atom:updated>2025-05-29T03:45:00.542Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><strong><em>You can follow along by signing up at </em></strong><a href="https://www.databricks.com/try-databricks"><strong><em>Try Databricks</em></strong></a><strong><em>.</em></strong></blockquote><p>Express setup is the <strong>fastest</strong> way to get started with Databricks on AWS.</p><p>In a minute or two, you’re up and running with a fully serverless workspace that you can use for exploratory analysis, processing your own data from S3, running SQL queries, and ultimately have the entirety of the Databricks Data Intelligence Platform available to you.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/449/1*0RmOnB2uVt96iMhBkEw9LA.png" /><figcaption>Serverless workspace in seconds with express setup, no cloud infrastructure required!</figcaption></figure><p>Now, a question I frequently get from folks is what comes next from a security perspective?</p><p>This comes from individuals who are using their new serverless workspace day in and day out, but they want to go further and make sure it’s locked down for either production use cases, sensitive non-regulated data, or to ensure it’s up to the standards of their other SaaS platforms.</p><p>So, in this blog post, we’ll walk through exactly that: a guided path for hardening your first serverless workspace created through express setup.</p><p>In all honesty, I won’t be adding much narrative to this particular blog post, so after we walk through an overview of each step, it’ll be only a rough architecture diagram, instructions to implement, and documentation links.</p><p>Even if you’re not planning to upgrade to Databricks Enterprise Tier, be sure to check out the end of the article for a fun little walkthrough of a small feature that I’ve been asked about countless times.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vklaXvp2oU3oNfXQkHyRLQ.png" /><figcaption>Guided pathway for hardening your first workspace.</figcaption></figure><p>The steps to lockdown your workspace will be broken into two sections.</p><p>The first section is based on the Databricks <strong>Premium Tier</strong>, which is the tier you start with when signing up through express setup. These steps include:</p><ul><li><strong>Configure Single-Sign On (SSO) and System for Cross-Domain Identity Management Integration (SCIM)</strong>: I’ll walk through how to set up SSO and SCIM for your new workspace using your existing identity provider.</li><li><strong>Update Workspace Ingress and Egress Controls: </strong>I’ll walk you through the security section of the workspace admin console, specifically the ingress and egress controls available to you.</li><li><strong>Familiarize Yourself with the Audit Log Table in System Tables using Genie: </strong>Last, I’ll show you how to analyze your audit log table and get familiar with how to observe your users’ behavior in Databricks.</li></ul><p>After we’ve wrapped up the Premium Tier section, I’ll walk through how to upgrade your account to <strong>Enterprise Tier</strong> to continue on our hardening journey. Then, we’ll go through another three steps:</p><ul><li><strong>Implement a Restrictive Network Policy: </strong>Network policies provide egress controls for serverless compute, allowing you to restrict outbound traffic to specific internet domains and storage locations.</li><li><strong>Implement Account and Workspace IP Access Lists (ACLs): </strong>IP access lists allow you to restrict the IP address ranges that can interact with both your Databricks account console and workspace.</li><li><strong>Update Workspace with a Customer-Managed Key for Managed Services:</strong> A customer-managed key for managed services encrypts several services in the Databricks control plane, as well as the workspace’s root storage and the initial catalog backed by Default Storage.</li></ul><p>With all that said, let’s dive straight in.</p><h3>Configure Single-Sign On (SSO) and System for Cross-Domain Identity Management (SCIM) Integration:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/815/1*kMCUa0IqyVTXqCt9AreWQg.png" /><figcaption>A very common first step for customers is to implement SSO and SCIM for Databricks!</figcaption></figure><p><strong>Configure SSO [</strong><a href="https://docs.databricks.com/aws/en/admin/account-settings-e2/single-sign-on/"><strong>Documentation</strong></a><strong>]:</strong></p><ol><li>Navigate to the account console</li><li>Follow step-by-step instructions for SSO, provided in reprise demos: <a href="https://app.getreprise.com/launch/W68jb2X/">OIDC SSO</a>, <a href="https://app.getreprise.com/launch/ryND0on/">SAML SSO</a>.</li></ol><p><strong>Configure SCIM [</strong><a href="https://docs.databricks.com/aws/en/admin/users-groups/scim/aad"><strong>Documentation</strong></a><strong>]:</strong></p><ol><li>Navigate to account console. Follow step-by-step instructions for SCIM, provided in provider-specific documentation.</li></ol><h3>Update Workspace Ingress and Egress Controls:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/827/1*b16EbWH_VHuSwongNLJqvg.png" /><figcaption>An easily changeable configuration, the workspace ingress and egress controls give you the ability to control what users can export from the workspace to their local machine</figcaption></figure><p><strong>[</strong><a href="https://docs.databricks.com/aws/en/admin/workspace-settings/"><strong>Documentation</strong></a><strong>]:</strong></p><ol><li>Navigate to the workspace and sign in as an administrator of the workspace</li><li>In the top right corner, click your name and select <strong><em>Settings</em></strong></li><li>Underneath <strong><em>Settings</em></strong>, click <strong><em>Security</em></strong></li><li>Scroll down to the <strong><em>Egress and ingress</em></strong> section</li></ol><p><strong>Disable the following</strong>: <em>Notebook results download, Notebook and File exporting, SQL results download, MLflow run artifact download, and Results table clipboard features</em></p><h3><strong>Familiarize Yourself with the Audit Log Table in System Tables using Genie:</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/818/1*8YtFfwfr2BgRSBhYAFDDLQ.png" /><figcaption>The audit table, pre-configured for serverless workspaces, gives you insight into what your users are doing throughout the Data Intelligence Platform.</figcaption></figure><p><strong>[</strong><a href="https://docs.databricks.com/aws/en/admin/system-tables/audit-logs"><strong>Documentation</strong></a><strong>]:</strong></p><ol><li>Navigate to the workspace and click on <strong><em>Catalog </em></strong>on the lefthand pane</li><li>Once on the catalog pane, click on the chevron next to <strong><em>system</em></strong> to expand the system database</li><li>Once expanded, click on the chevron next to <strong><em>access </em></strong>to expand the access schema</li><li>Click on the table named <strong><em>audit</em></strong></li><li>After the audit table has been selected, on the righthand side select <strong><em>Sample Data</em></strong></li><li>If compute is not available, click on <strong>Select compute</strong> and start the serverless starter warehouse</li><li>Once the warehouse is started, you’ll either see ‘No data’ or populated data fields. If you see the former, continue your journey investigating the platform and revisit this step in a few hours.</li><li>After you see data, click on <strong><em>Ask Genie</em></strong>.</li><li>Enter into the chat box: <strong><em>What are the most common actions performed by users?</em></strong></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/919/1*JoRHQRDaIT0DlrLKTC3fOQ.png" /><figcaption>Genie allows you to ask questions about your data using natural language, saving you time on understanding the audit log schema and writing SQL.</figcaption></figure><p>✍️ <strong>NOTE</strong>: If you’re interested in exploring audit logs further, or creating custom events on potentially suspicious activities, be sure to check out this blog post!</p><p><a href="https://www.databricks.com/blog/improve-lakehouse-security-monitoring-using-system-tables-databricks-unity-catalog"><strong><em>Improve Lakehouse Security Monitoring using System Tables in Databricks Unity Catalog</em></strong></a></p><h3><strong>Upgrade from Premium Tier to Enterprise Tier:</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*-iuXFdSlumNPvszW.png" /><figcaption>Managing your subscription and billing is simple with the unified view found in the account console.</figcaption></figure><p>⚠️ As the naming convention suggests, Enterprise Tier does cost more per Databricks Unit (DBU) than the Premium Tier. You can review the pricing differences in our documentation <a href="https://www.databricks.com/product/pricing">here</a>.</p><p><strong>[</strong><a href="https://docs.databricks.com/aws/en/admin/account-settings/account"><strong>Documentation</strong></a><strong>]:</strong></p><ol><li>Navigate to the account console</li><li>Click on <strong><em>Settings</em></strong></li><li>Click on <strong><em>Change plan</em></strong></li><li>Select <strong><em>Enterprise</em></strong></li><li>Click<strong> Save</strong></li></ol><h3><strong>Implement Network Policies:</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SbjyrrqomzVCjv4Oxbm-yw.png" /><figcaption>Network policies allow admins to easily configure what internet or storage destinations are accessible by serverless compute, if any!</figcaption></figure><p><strong>[</strong><a href="https://docs.databricks.com/aws/en/security/network/serverless-network-security/manage-network-policies"><strong>Documentation</strong></a><strong>]:</strong></p><ol><li>Navigate to the account console and sign-in</li><li>Click <strong><em>Cloud resources</em></strong></li><li>Click the <strong><em>Network</em></strong><em> </em>tab, then on the <strong><em>Network policies</em></strong><em> sub-tab</em></li><li>Click the <strong><em>Create new network policy</em></strong><em> </em>button</li><li>Insert a policy name</li><li>Select the radial button <strong><em>Restricted access to specific destinations</em></strong></li><li>Add allowed fully qualified domain names (e.g., google.com)</li><li>Add in existing S3 bucket names identified from the previous segment</li><li>Select the radial button <strong><em>Enforced for all products</em></strong></li><li>Click <strong><em>Create</em></strong></li><li>Navigate back to the <strong><em>Workspace</em></strong> tab and select the existing workspace</li><li>Click <strong><em>Update network policy</em></strong><em> </em>under the <strong><em>Network Policy</em></strong> box and select the newly created policy</li><li>Restart any existing serverless compute in the notebook</li><li>After a couple of minutes, test the network policy (e.g. %sh ping <em>allowed_domain_name</em>).</li></ol><p>✍️ <strong>NOTE</strong>: If you’d prefer to analyze outbound traffic prior to shutting off access to the public internet or storage locations, you can analyze the traffic by enacting <a href="https://docs.databricks.com/sap/en/security/network/serverless-network-security/manage-network-policies#configure-network-policies"><strong>Dry Run Mode</strong></a><strong> </strong>and<strong> </strong>viewing potential denials in the <a href="https://docs.databricks.com/aws/en/admin/system-tables/outbound-network"><strong>network access events</strong></a> system table.</p><h3>Implement Account and Workspace IP ACLs:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NEEiOVAhUX7gsE7AopI8DQ.png" /><figcaption>IP ACLs are a first line of defense against malicious actors. It can help ensure that the only traffic that’s allowed for the account console or the workspace is from your corporate VPN, not a local coffee shop’s internet!</figcaption></figure><p>⚠️ It’s <strong>highly</strong> recommended that a break glass IP is added to the IP access list while testing to prevent account or workspace lockout!</p><p><strong>Implement Account IP ACL [</strong><a href="https://docs.databricks.com/aws/en/security/network/front-end/ip-access-list-account"><strong>Documentation</strong></a><strong>]:</strong></p><ol><li>Navigate to account console and sign-in</li><li>Click <strong><em>Settings</em></strong></li><li>Along the top bar, click <strong><em>Security and Compliance</em></strong></li><li>Click <strong><em>IP Access List</em></strong></li><li>On the right-hand side, click <strong><em>Add Rule</em></strong></li><li>Enter a label name such as “Corp Network”</li><li>Enter in relevant IP addresses</li><li>Select the radial button for <strong><em>Status</em></strong> to <strong><em>Enable</em></strong></li><li>Click Add Rule</li></ol><p><strong>Implement Workspace IP ACLs [</strong><a href="https://docs.databricks.com/aws/en/security/network/front-end/ip-access-list-workspace"><strong>Documentation</strong></a><strong>]:</strong></p><p><em>Part 1 — Generate Admin Level Service Principal:</em></p><p>⚠️ <strong>Warning! Service principal IDs, secrets, OAuth tokens should always be kept secret and never shared publicly.</strong></p><ol><li>Navigate to account console and sign-in</li><li>Click <strong><em>User management</em></strong></li><li>Along the top bar, click <strong><em>Service principals</em></strong></li><li>On the right-hand side, click <strong><em>Add service principal</em></strong></li><li>Enter a service principal name, such as “workspace-admin-sp”</li><li>Click the newly created service principal and along the top bar navigate to <strong><em>Roles</em></strong></li><li>Click the radial button for <strong><em>Account admin</em></strong></li><li>Along the top bar navigate back to <strong><em>Principal information</em></strong></li><li>On the bottom pane, click <strong><em>Generate secret</em></strong></li><li>Copy down the secret and Client ID onto a separate tab</li><li>Navigate back to the workspaces tab and click the designated workspace</li><li>Click the newly created service principal and along the top bar navigate to <strong><em>Permissions</em></strong></li><li>On the right-hand side, click <strong><em>Add Permissions</em></strong></li><li>Search the newly created service principal and select <strong><em>admin</em></strong> from the permission list, then click <strong><em>Save</em></strong></li><li>Construct the token endpoint by replacing the workspace endpoint in the following URL: <strong>https://&lt;workspace-endpoint&gt;/oidc/v1/token</strong></li><li>Use the client ID and secret to create a temporary authentication token. This token is valid for ten minutes. The code is provided below.</li></ol><pre>export CLIENT_ID=&lt;client-id&gt;<br>export CLIENT_SECRET=&lt;client-secret&gt;<br><br>curl --request POST \<br>--url &lt;token-endpoint-URL&gt; \<br>--user &quot;$CLIENT_ID:$CLIENT_SECRET&quot; \<br>--data &#39;grant_type=client_credentials&amp;scope=all-apis&#39;</pre><p><em>Part 2— Enable Workspace IP ACLs:</em></p><ol><li>Construct the enabling IP ACL endpoint by replacing the workspace endpoint in the following URL: <strong>https://&lt;workspace-endpoint&gt;/api/2.0/workspace-conf</strong></li><li>To enable IP ACLs on the workspace, use the bearer token provided in step 16. The code is provided below</li></ol><pre>curl --location --request PATCH &#39;&lt;enable-IP-ACL-URL&gt;&#39; \<br>--header &#39;Content-Type: application/json&#39; \<br>--header &#39;Authorization: &lt;token&gt;&#39; \<br>--data &#39;{&quot;enableIpAccessLists&quot; : &quot;true&quot;}&#39;</pre><p><em>Part 3— Add IPs to Workspace ACL:</em></p><ol><li>Construct the IP ACL list endpoint by replacing the workspace endpoint in the following URL: <strong>https://&lt;workspace-endpoint&gt;/api/2.0/ip-access-lists</strong></li><li>Use the bearer token provided in step 16 to set an IP access list for the workspace. The code is provided below</li></ol><pre>curl --location &#39;&lt;enable-IP-ACL-URL&gt;&#39;&#39; \<br>--header &#39;Content-Type: application/json&#39; \<br>--header &#39;Authorization:  &lt;token&gt;&#39; \<br>--data &#39;{<br>  &quot;label&quot;: &quot;Office VPN&quot;,<br>  &quot;list_type&quot;: &quot;ALLOW&quot;,<br>  &quot;ip_addresses&quot;: [<br>    &quot;IP Address #1&quot;,<br>    &quot;IP Address #2&quot;<br>  ]<br>}&#39;</pre><h3>Update Workspace with a Customer-Managed Key for Managed Services:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VdmfeYGrztdQvYIkFsJ_AA.png" /><figcaption>A customer-managed key adds on another layer of control by giving you the ability to remove it at any time, thus rendering data and various control plane services unencryptable.</figcaption></figure><p><strong>[</strong><a href="https://docs.databricks.com/aws/en/security/keys/configure-customer-managed-keys#step-1-create-or-select-a-key-in-aws-kms"><strong>Documentation</strong></a><strong>]:</strong></p><p><em>Part 1— Create KMS Key:</em></p><ol><li>Navigate to the AWS console</li><li>Once authenticated, search for ‘KMS’ and click <strong><em>Key Management Service</em></strong></li><li>In the top right-hand corner, click <strong><em>Create key</em></strong></li><li>Leave the initial radial button selections of <strong><em>Symmetric</em></strong> and <strong><em>Encrypt and decrypt, </em></strong>then click <strong><em>Next</em></strong></li><li>In the alias, add in ‘workspace-managed-services’ or other appropriate naming convention based on the customer’s standards</li><li>Add in an optional description of, ‘KMS for Databricks workspace managed services’</li><li>Add in tags as appropriate, then click <strong><em>Next</em></strong></li><li>Add the user’s role, or another role, as the key administrator, then click <strong><em>Next</em></strong></li><li>Add the user’s role, or another role, as the key user, then click <strong><em>Next</em></strong></li><li>At the bottom of the key policy role, add the following policy, ensuring to fill in the required information</li></ol><pre>{<br>    &quot;Sid&quot;: &quot;Allow Databricks to use KMS key for managed services in the control plane&quot;,<br>    &quot;Effect&quot;: &quot;Allow&quot;,<br>    &quot;Principal&quot;: {<br>        &quot;AWS&quot;: &quot;arn:aws:iam::414351767826:root&quot;<br>    },<br>    &quot;Action&quot;: [<br>        &quot;kms:Encrypt&quot;,<br>        &quot;kms:Decrypt&quot;<br>    ],<br>    &quot;Resource&quot;: &quot;*&quot;,<br>    &quot;Condition&quot;: {<br>        &quot;StringEquals&quot;: {<br>            &quot;aws:PrincipalTag/DatabricksAccountId&quot;: &quot;&lt;account ID&gt;&quot;<br>        }<br>    }<br>}</pre><p><em>Part 2— Register KMS Key with Databricks [</em><a href="https://docs.databricks.com/aws/en/security/keys/configure-customer-managed-keys#step-3-create-a-new-key-configuration"><em>documentation</em></a><em>]:</em></p><ol><li>Navigate to account console and sign-in</li><li>Click <strong><em>Cloud Resources</em></strong></li><li>Along the top bar, click <strong><em>Encryption key configuration</em></strong></li><li>On the right-hand side, click <strong><em>Add encryption key</em></strong></li><li>Select the radial button for managed services</li><li>Scroll to the bottom and enter the key ARN and key alias of the newly created key then click <strong><em>Add</em></strong></li><li>Once the key has been added to the account console note the ID of the key for future use</li></ol><p><em>Part 3— Retrieve OAuth Token for Admin Service Principal [</em><a href="https://docs.databricks.com/aws/en/dev-tools/auth/oauth-m2m"><em>documentation</em></a><em>].</em></p><p>⚠️ <strong>Warning! Service principal IDs, secrets, OAuth tokens should always be kept secret and never shared publicly.</strong></p><ol><li>Navigate to account console and sign-in</li><li>Click <strong><em>User management</em></strong></li><li>Along the top bar, click <strong><em>Service principals</em></strong></li><li>On the right-hand side, click <strong><em>Add service principal</em></strong></li><li>Enter a service principal name, such as “admin-sp”</li><li>Click the newly created service principal and along the top bar navigate to <strong><em>Roles</em></strong></li><li>Click the radial button for <strong><em>Account admin</em></strong></li><li>Along the top bar navigate back to <strong><em>Principal information</em></strong></li><li>On the bottom pane, click <strong><em>Generate secret</em></strong></li><li>Input a set number of days the secret will be valid for and click <strong><em>Generate</em></strong>.</li><li>Copy down the secret and Client ID onto a separate tab</li><li>Navigate to Postman or to a terminal to run relevant API calls</li><li>Run an initial API call to generate an OAuth token with a service principal ID and secret as the username and password, see code below. Copy the OAuth Token for future use, this has a <strong>10 minute</strong> expiration</li></ol><pre>curl --location &#39;https://accounts.cloud.databricks.com/oidc/accounts/&lt;account id&gt;/v1/token&#39; \<br>--header &#39;Content-Type: application/x-www-form-urlencoded&#39; \<br>--header &#39;Authorization: &lt;token&gt;&#39; \<br>--data-urlencode &#39;grant_type=client_credentials&#39; \<br>--data-urlencode &#39;scope=all-apis&#39;</pre><p><em>Part 4— Patch the Workspace with the CMK for Managed Services [</em><a href="https://docs.databricks.com/aws/en/security/keys/configure-customer-managed-keys#step-4-add-the-key-configuration-to-a-workspace"><em>documentation</em></a><em>]:</em></p><p>⚠️ It is <strong>highly</strong> recommend to shut-off all compute prior to attempting this step.</p><ol><li>Navigate to the account console and collect the <strong>workspace </strong>ID. You can find this in the navigation bar: accounts.cloud.databricks.com/workspaces/<strong>&lt;workspace id&gt;</strong></li><li>Navigate to Postman or to a terminal to run relevant API calls, using the OAuth token as authorization, see code below</li></ol><pre>curl --location --request PATCH &#39;https://accounts.cloud.databricks.com/api/2.0/accounts/&lt;account_id&gt;/workspaces/&lt;workspace_id&gt;&#39; \<br>--header &#39;Content-Type: application/json&#39; \<br>--header &#39;Authorization: &lt;token&gt;&#39; \<br>--data &#39;{<br>  &quot;managed_services_customer_managed_key_id&quot;: &quot;&lt;key id&gt;&quot;<br>}&#39;</pre><ol><li>If the API is successful, navigate back to the workspace pane in the account console and observe the newly added managed services CMK box</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/265/1*QvjVuDOiUk9MsuuTc8lx_w.png" /><figcaption>CMK has been added to our existing workspace.</figcaption></figure><h3><strong><em>Congratulations!</em></strong></h3><p>If you’ve made it this far, whether you followed all the steps or only some of them, you’ve made immense progress in hardening your Databricks security posture.</p><p>Let’s talk about that small feature that I alluded to before, the ability to change your initial workspace name.</p><h3>How to Change Your Workspace Name in the Account Console:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U3Pr9QM99yEb193_qaKW-g.png" /><figcaption>Naming convention purists run in terror at the name: <strong>workspace</strong>.</figcaption></figure><p>After signing-up for express setup, the first workspace you’ll get is named <strong>workspace. </strong>If you’re anything like me, there is no way I’m going to let that naming convention stay.</p><p>So, here’s how you can go ahead and change that workspace name!</p><p><strong>Steps to change your initial serverless workspace name:</strong></p><ol><li>Navigate to the top-right corner of the workspace UI and click on <strong><em>Settings</em></strong></li><li>Navigate to the left-hand list and select <strong><em>Appearance</em></strong></li><li>On the bottom of the screen select the pencil next to <strong><em>Workspace Name</em></strong></li><li>Update the workspace name and click <strong><em>Save</em></strong></li><li>Navigate to the account console and observe the change</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kMsDOWKOmB9JpqOtuPOxYQ.png" /><figcaption>And yes... emojis are accepted characters.</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=46f1ed646f25" width="1" height="1" alt=""><hr><p><a href="https://medium.com/databricks-platform-sme/six-practical-steps-to-lock-down-your-databricks-workspace-after-getting-started-with-express-setup-46f1ed646f25">Six Practical Steps to Lock Down Your Databricks Workspace After Getting Started with Express Setup</a> was originally published in <a href="https://medium.com/databricks-platform-sme">Databricks Platform SME</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Must Watch YouTube Videos for Databricks Platform Administrators]]></title>
            <link>https://medium.com/databricks-platform-sme/must-watch-youtube-videos-for-databricks-platform-administrators-177a72def63c?source=rss-c22ed87dd65------2</link>
            <guid isPermaLink="false">https://medium.com/p/177a72def63c</guid>
            <category><![CDATA[platform]]></category>
            <category><![CDATA[administration]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[JD Braun]]></dc:creator>
            <pubDate>Wed, 17 Jul 2024 17:41:30 GMT</pubDate>
            <atom:updated>2024-07-17T17:44:39.961Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/429/0*EZGCdJsiYsCDPFCR.jpg" /><figcaption>stock.adobe.com</figcaption></figure><p>While written word is clearly the medium of choice for this platform, sometimes a picture or a video can be worth 1,000 words.</p><p>Below are <strong>16 videos</strong> I’d recommend for any Databricks administrators covering the topics of general platform administration, automation, networking, and security.</p><p><strong>Administration</strong>:</p><ul><li><strong>Administrator Best Practices and Tips for Future-Proofing your Databricks Account</strong> by Siddharth Bhai, Gaurav Bhatnagar, and Vicky Avison</li><li><strong>Best Practices for Setting up Databricks SQL at Enterprise Scale</strong> by Siddharth Bhai, Samrat Ray, and Jeremy Lewallen</li><li><strong>A Technical Deep Dive into Unity Catalog’s Practitioner Playbook</strong> by Zeashan Pappa</li><li><strong>Migration from Hive Metastore to Unity Catalog </strong>by Arthur Dooner</li></ul><p><strong>Automation</strong>:</p><ul><li><strong>Databricks Asset Bundles: A Standard, Unified Approach to Deploying Data Products on Databricks</strong> by Rafi Kurlansik and Pieter Noordhuis</li><li><strong>Terraform Templates for Secure Databricks Deployments from Day Zero</strong> by JD Braun</li><li><strong>Getting Started with Databricks Terraform Modules</strong> by Yassine Essawabi and Youssef Mrini</li><li><strong>How to Setup Databricks Unity Catalog with Terraform</strong> by Himanshu Arora and Youssef Mrini</li><li><strong>CI/CD in Databricks with Azure DevOps </strong>by Ravi Shankar</li></ul><p><strong>Networking:</strong></p><ul><li><strong>Deploying a Databricks Workspace on AWS with PrivateLink</strong> by JD Braun</li><li><strong>Azure Databricks Networking Security</strong> by Arthur Dooner</li></ul><p><strong>Security:</strong></p><ul><li><strong>Security Best Practices for the Lakehouse</strong> by David Veuve</li><li><strong>Security Best Practices and Tools to Build a Secure Lakehouse</strong> by Anindita Mahapatra and Arun Pamulapati</li><li><strong>Enterprise-Grade Security and Compliance on the Databricks Data Intelligence Platform </strong>by Filippo Seracini and Alex Esibov</li><li><strong>Data Warehousing Performance, Scale and Security with Databricks SQL</strong> by Alex Esibov and Jeremy Lewallen</li><li><strong>Isolate the Lakehouse: Deploying an Isolated Architecture on Databricks</strong> by JD Braun</li></ul><h3><strong>Administration — Videos:</strong></h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FQPTi-aFH7fs%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DQPTi-aFH7fs&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FQPTi-aFH7fs%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/4b279e49dfad5e62b8cfe686f3aba7a3/href">https://medium.com/media/4b279e49dfad5e62b8cfe686f3aba7a3/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FCLBq4fBE2Ro%3Fstart%3D1%26feature%3Doembed%26start%3D1&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DCLBq4fBE2Ro&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FCLBq4fBE2Ro%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/9bb1f545cdf596bca8083eef44e1b097/href">https://medium.com/media/9bb1f545cdf596bca8083eef44e1b097/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FLzmmObc_Bmw%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DLzmmObc_Bmw&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FLzmmObc_Bmw%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/78a2b0d10eeb1bb630ca19d6b940e79e/href">https://medium.com/media/78a2b0d10eeb1bb630ca19d6b940e79e/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fr-LlER-IWCM%3Fstart%3D8%26feature%3Doembed%26start%3D8&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dr-LlER-IWCM&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fr-LlER-IWCM%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/79837528653a4d409daffd482b1ea2c0/href">https://medium.com/media/79837528653a4d409daffd482b1ea2c0/href</a></iframe><h3><strong>Automation — Videos:</strong></h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F9HOgYVo-WTM%3Fstart%3D25%26feature%3Doembed%26start%3D25&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D9HOgYVo-WTM&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F9HOgYVo-WTM%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/c9c23152d1dbe10eb08a4292856a6889/href">https://medium.com/media/c9c23152d1dbe10eb08a4292856a6889/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F7RZabe2xUbM%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D7RZabe2xUbM&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F7RZabe2xUbM%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/aea5ebb6a9f5e17626a6cc500b83eece/href">https://medium.com/media/aea5ebb6a9f5e17626a6cc500b83eece/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FZBTp9Iz0QYA%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DZBTp9Iz0QYA&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FZBTp9Iz0QYA%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/927f72a51387dab3ac7898266c83d011/href">https://medium.com/media/927f72a51387dab3ac7898266c83d011/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FYv8qHK-nzh0%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DYv8qHK-nzh0&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FYv8qHK-nzh0%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/06cb73f6f33953248d84416258e73ec6/href">https://medium.com/media/06cb73f6f33953248d84416258e73ec6/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FlBbOQvX9yN4%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DlBbOQvX9yN4&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FlBbOQvX9yN4%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/0e0d8cd6237477a0afb85d3d89228b0b/href">https://medium.com/media/0e0d8cd6237477a0afb85d3d89228b0b/href</a></iframe><h3>Networking Videos:</h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FjwooOK5h_vQ&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DjwooOK5h_vQ&amp;image=http%3A%2F%2Fi.ytimg.com%2Fvi%2FjwooOK5h_vQ%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/9ccfefc36113c717731be921ba7207a9/href">https://medium.com/media/9ccfefc36113c717731be921ba7207a9/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FVbK0KYgpE0U%3Fstart%3D1%26feature%3Doembed%26start%3D1&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DVbK0KYgpE0U&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FVbK0KYgpE0U%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/9a7755ab69d75f7892e7a78fc3e2d0ff/href">https://medium.com/media/9a7755ab69d75f7892e7a78fc3e2d0ff/href</a></iframe><h3><strong>Security — Videos:</strong></h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F7ap9LzpO5t4&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D7ap9LzpO5t4&amp;image=http%3A%2F%2Fi.ytimg.com%2Fvi%2F7ap9LzpO5t4%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/34f7c563f88f3b59d3b0dca876c4bece/href">https://medium.com/media/34f7c563f88f3b59d3b0dca876c4bece/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FB73Jvzr8zPA%3Flist%3DPLTPXxbhUt-YW5ryIvQo2W5qN4stKMQ5yL&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DB73Jvzr8zPA&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FB73Jvzr8zPA%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/991e88f30b5aa4e6238b74459920da9c/href">https://medium.com/media/991e88f30b5aa4e6238b74459920da9c/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FI9lxH6pzask%3Flist%3DPLTPXxbhUt-YW5ryIvQo2W5qN4stKMQ5yL&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DI9lxH6pzask&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FI9lxH6pzask%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/8daf0ac0094aa2043b8fa476339bf371/href">https://medium.com/media/8daf0ac0094aa2043b8fa476339bf371/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FVzVIT1yPdnw%3Flist%3DPLTPXxbhUt-YW5ryIvQo2W5qN4stKMQ5yL&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DVzVIT1yPdnw&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FVzVIT1yPdnw%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/cda8d3fb4291abd9c0428da6dab741a3/href">https://medium.com/media/cda8d3fb4291abd9c0428da6dab741a3/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FXFQSNRgXTbc%3Fstart%3D5%26feature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DXFQSNRgXTbc&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FXFQSNRgXTbc%2Fhqdefault.jpg&amp;key=d04bfffea46d4aeda930ec88cc64b87c&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/a3bcf8389bd222425d941f4f10bfb4fa/href">https://medium.com/media/a3bcf8389bd222425d941f4f10bfb4fa/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=177a72def63c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/databricks-platform-sme/must-watch-youtube-videos-for-databricks-platform-administrators-177a72def63c">Must Watch YouTube Videos for Databricks Platform Administrators</a> was originally published in <a href="https://medium.com/databricks-platform-sme">Databricks Platform SME</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How we’d set-up our Databricks deployment pipelines with Terraform repositories in 2024]]></title>
            <link>https://medium.com/databricks-platform-sme/how-wed-set-up-our-databricks-deployment-pipelines-with-terraform-repositories-in-2024-ac5f816c3dda?source=rss-c22ed87dd65------2</link>
            <guid isPermaLink="false">https://medium.com/p/ac5f816c3dda</guid>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[automation]]></category>
            <dc:creator><![CDATA[JD Braun]]></dc:creator>
            <pubDate>Wed, 07 Feb 2024 18:54:10 GMT</pubDate>
            <atom:updated>2024-02-07T18:57:14.938Z</atom:updated>
            <content:encoded><![CDATA[<p><em>The video and blog post have been co-authored by Tony Bo. A video representing this blog post content can be found </em><a href="https://www.youtube.com/watch?v=OgxQop9fB70&amp;list=PLeK0Tsm_E67QFwU4trdGZSjcQiTbcsBoz&amp;index=2&amp;t=889s"><em>here</em></a><em>.</em></p><p>There are countless ways an organization can architect their Terraform repositories to deploy cloud infrastructure, from a 400-line main.tf file all the way to everything in tidy modules to be called downstream.</p><p>Some of these architectures stand the test of time, while others become brittle over the years and introduce rate limits, circular dependencies, or the accidental coupling of unrelated resources.</p><p>In this post, Tony and I outline how in 2024 we’d architect our Databricks Terraform repository for long-term scalability and usability, driven by two basic principles: isolation and reusability.</p><p>These principles are not unique to a Terraform repository with Databricks, but we wanted to walk through practical examples of how they’re applied.</p><p>You can find the example code base used to deploy these resources <a href="https://github.com/JDBraun/dbx_mws_example">here</a>. No warranties or guarantees are associated with this repository, but please leave Git Issues as needed.</p><p><strong>NOTE</strong>: This blog post assumes a base level of familiarity with Terraform and Databricks. Before proceeding, you should be familiar with the basics of Terraform (e.g., plan, apply, etc.) and Databricks (Unity Catalog, workspaces, etc.).</p><h3>Isolating Your Environments:</h3><p>The first key point we architect for is the isolation of Terraform resources that have no relationship to each other. The atomic unit we’ll be working with, from a Databricks perspective, is a workspace.</p><p>This could mean separation by designation (e.g., development, QA, and production) or by business unit (finance, human resources, analytics, etc.). We aim to ensure that the fate of one workspace is not tied to another, providing downstream flexibility to iterate without fear.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-R3q_olcqOv3dJSjUKGbrw.png" /><figcaption>Example of Breaking out State Files</figcaption></figure><p>As illustrated in the above picture, our state files look like this:</p><ul><li><strong>Unity Catalog — State File</strong>: A state dedicated to a single resource, the Unity Catalog metastore. This is a linchpin in the architecture that we don’t want to risk by integrating it into other parts of the repository.</li><li><strong>Logging — State File:</strong> A state file containing the Databricks and cloud resources for audit and billing logs. Again, this is something that should be entirely unconnected to the rest of the infrastructure.</li><li><strong>N+1 Workspace — State File(s)</strong>: A state file for each workspace environment containing the Databricks and cloud resources needed for the underlying workspace. Examples include Databricks account resources (networking, credential, and storage config), cloud resources (VPC, subnets, security groups), and Databricks workspace resources (Workspace isolated catalog, cluster policies, etc.).</li></ul><p>In the Databricks account console, you’ll see a neatly organized list of workspaces like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zyDZh94aPE-dRzeeOL2o9w.png" /><figcaption>List of workspace examples</figcaption></figure><p>However, each of them is its own individual pipeline, completely independent of the others. Adding components to the development workspace will eventually get promoted into a higher environment. Independent pipelines for each environment means faster iteration and faster time to narrow down terraform errors or miscellaneous issues.</p><p>Within each workspace, there are resources that can be controlled and tuned to that specific environment, such as cluster policies, workspace isolated catalogs, and more.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/489/1*p_PavVsCaf0amqXvPzbt0A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/488/1*HFO1XiYnYB3QSvoot5S6dg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/483/1*iob45eQpXcpZ84YfoSUiog.png" /><figcaption>Catalog Isolation — Across Three Workspaces</figcaption></figure><p>For example, despite having full privileges to access these three catalogs, I’m only allowed to see the catalog in the workspace it’s bound to.</p><p>Enforcing isolation in both Terraform and user behavior — a win-win situation.</p><h3>Reusability for Writing Once and Using Everywhere:</h3><p>The second principle focuses on reducing the amount of work needed to spin up new resources. Let’s illustrate this by creating a new Databricks environment as an example.</p><p><em>Happy Path — Creating a New Databricks Workspace:</em></p><ol><li>A team requests the creation of a Databricks workspace.</li><li>The DevOps team creates a new Terraform repository and clones relevant content from previous environments.</li><li>The DevOps team updates the .tfvars file with appropriate variables for the new environment and performs a terraform init to initialize the state file.</li><li>Deploy!</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hj2B1jXGKstTF5zH5AhiJg.png" /><figcaption>Example of a new_environment.tf calling various endpoints with its modules.</figcaption></figure><p>n each module, there can be sub-modules that are then called for specific purposes. For example, our first module, found in development, QA, or production, calls three different modules. We call our cloud_provider module, which, as mentioned above, handles the creation of networking, credential, and storage resources for the workspace. Then, using outputs from the cloud provider, we call the databricks_account, which will create the workspace, assign the metastore, and add any users. Finally, we interact with databricks_workspace, where we create cluster policies, workspace configurations, and a workspace catalog.</p><pre>module &quot;cloud_provider&quot; {<br>    source = &quot;./cloud_provider&quot;<br>    providers = {<br>        aws = aws<br>        databricks = databricks.mws<br>    }<br>    aws_account_id        = var.aws_account_id<br>    databricks_account_id = var.databricks_account_id<br>    resource_prefix       = var.resource_prefix<br>    region                = var.region<br>    vpc_cidr_range        = var.vpc_cidr_range<br>    availability_zones    = var.availability_zones<br>    public_subnets_cidr   = var.public_subnets_cidr<br>    private_subnets_cidr  = var.private_subnets_cidr<br>    sg_ingress_protocol   = var.sg_ingress_protocol<br>    sg_egress_ports       = var.sg_egress_ports<br>    sg_egress_protocol    = var.sg_egress_protocol<br>    dbfsname              = var.dbfsname<br>  <br>}<br><br>module &quot;databricks_account&quot; {<br>    source = &quot;./databricks_account&quot;<br>    providers = {<br>        databricks = databricks.mws<br>    }<br><br>      databricks_account_id  = var.databricks_account_id<br>      region                 = var.region<br>      resource_prefix        = var.resource_prefix<br>      cross_account_role_arn = module.cloud_provider.cloud_provider_credential<br>      bucket_name            = module.cloud_provider.cloud_provider_storage<br>      vpc_id                 = module.cloud_provider.cloud_provider_network_vpc<br>      subnet_ids             = module.cloud_provider.cloud_provider_network_subnets<br>      security_group_ids     = module.cloud_provider.cloud_provider_network_security_groups<br>      metastore_id           = var.metastore_id<br>      user_name              = var.user_name<br>  <br>}<br><br>module &quot;databricks_workspace&quot; {<br>    source = &quot;./databricks_workspace&quot;<br>     providers = {<br>        aws = aws<br>        databricks = databricks.workspace<br>    }<br><br>    aws_account_id        = var.aws_account_id<br>    databricks_account_id = var.databricks_account_id<br>    resource_prefix       = var.resource_prefix<br>    workspace_id          = module.databricks_account.workspace_id<br>    uc_catalog_name       = &quot;${var.resource_prefix}-catalog-${module.databricks_account.workspace_id}&quot;<br>    workspace_catalog_admin = var.user_name<br>    team = var.team<br>  <br>}</pre><p>As we go deeper into one of these modules, the cloud_provider, you can see that these are further broken down into sub-modules. We break it out by the network, credential, and storage - or in other terms, the VPC, subnets, security groups of the workspace, the cross-account role for cluster spin-up, and the root S3 bucket for the workspace.</p><pre>module &quot;cloud_provider_network&quot; {<br>  source = &quot;../../common_modules_cloud_provider/cloud_provider_network&quot;<br><br>  vpc_cidr_range       = var.vpc_cidr_range<br>  availability_zones   = var.availability_zones<br>  resource_prefix      = var.resource_prefix<br>  public_subnets_cidr  = var.public_subnets_cidr<br>  private_subnets_cidr = var.private_subnets_cidr<br>  sg_ingress_protocol  = var.sg_ingress_protocol<br>  sg_egress_ports      = var.sg_egress_ports<br>  sg_egress_protocol   = var.sg_egress_protocol<br><br>}<br><br>module &quot;cloud_provider_credential&quot; {<br>  source = &quot;../../common_modules_cloud_provider/cloud_provider_credential&quot;<br><br>  aws_account_id        = var.aws_account_id<br>  databricks_account_id = var.databricks_account_id<br>  resource_prefix       = var.resource_prefix<br>  region                = var.region<br>  vpc_id                = module.cloud_provider_network.cloud_provider_network_vpc<br>  security_group_ids    = module.cloud_provider_network.cloud_provider_network_security_groups<br><br>  depends_on = [ module.cloud_provider_network ]<br>}<br><br>module &quot;cloud_provider_storage&quot; {<br>  source = &quot;../../common_modules_cloud_provider/cloud_provider_storage&quot;<br><br>  dbfsname = var.dbfsname<br>}</pre><p>Then, if we open up the cloud_provider_network module, we can see the reusable code that is going to be called by each environment. This gives us flexibility to parameterize any of the variables needed for these resources.</p><pre>module &quot;vpc&quot; {<br>  source  = &quot;terraform-aws-modules/vpc/aws&quot;<br>  version = &quot;5.1.1&quot;<br><br>  name = &quot;${var.resource_prefix}-data-plane-VPC&quot;<br>  cidr = var.vpc_cidr_range<br>  azs  = var.availability_zones<br><br>  enable_dns_hostnames   = true<br>  enable_nat_gateway     = true<br>  single_nat_gateway     = false<br>  one_nat_gateway_per_az = true<br>  create_igw             = true<br><br>  public_subnet_names = [for az in var.availability_zones : format(&quot;%s-public-%s&quot;, var.resource_prefix, az)]<br>  public_subnets      = var.public_subnets_cidr<br><br>  private_subnet_names = [for az in var.availability_zones : format(&quot;%s-private-%s&quot;, var.resource_prefix, az)]<br>  private_subnets      = var.private_subnets_cidr<br>}<br><br>// SG<br>resource &quot;aws_security_group&quot; &quot;sg&quot; {<br>  vpc_id     = module.vpc.vpc_id<br>  depends_on = [module.vpc]<br><br>  dynamic &quot;ingress&quot; {<br>    for_each = var.sg_ingress_protocol<br>    content {<br>      from_port = 0<br>      to_port   = 65535<br>      protocol  = ingress.value<br>      self      = true<br>    }<br>  }<br><br>  dynamic &quot;egress&quot; {<br>    for_each = var.sg_egress_protocol<br>    content {<br>      from_port = 0<br>      to_port   = 65535<br>      protocol  = egress.value<br>      self      = true<br>    }<br>  }<br><br>  dynamic &quot;egress&quot; {<br>    for_each = var.sg_egress_ports<br>    content {<br>      from_port   = egress.value<br>      to_port     = egress.value<br>      protocol    = &quot;tcp&quot;<br>      cidr_blocks = [&quot;0.0.0.0/0&quot;]<br>    }<br>  }<br>  tags = {<br>    Name = &quot;${var.resource_prefix}-data-plane-sg&quot;<br>  }<br>}<br><br>module &quot;vpc_endpoints&quot; {<br>  source  = &quot;terraform-aws-modules/vpc/aws//modules/vpc-endpoints&quot;<br>  version = &quot;3.11.0&quot;<br><br>  vpc_id             = module.vpc.vpc_id<br>  security_group_ids = [aws_security_group.sg.id]<br><br>  endpoints = {<br>    s3 = {<br>      service         = &quot;s3&quot;<br>      service_type    = &quot;Gateway&quot;<br>      route_table_ids = module.vpc.private_route_table_ids<br>      tags = {<br>        Name = &quot;${var.resource_prefix}-s3-vpc-endpoint&quot;<br>      }<br>    },<br>    sts = {<br>      service             = &quot;sts&quot;<br>      private_dns_enabled = true<br>      subnet_ids          = length(module.vpc.private_subnets) &gt; 0 ? slice(module.vpc.private_subnets, 0, min(2, length(module.vpc.private_subnets))) : []<br>      tags = {<br>        Name = &quot;${var.resource_prefix}-sts-vpc-endpoint&quot;<br>      }<br>    },<br>    kinesis-streams = {<br>      service             = &quot;kinesis-streams&quot;<br>      private_dns_enabled = true<br>      subnet_ids          = length(module.vpc.private_subnets) &gt; 0 ? slice(module.vpc.private_subnets, 0, min(2, length(module.vpc.private_subnets))) : []<br>      tags = {<br>        Name = &quot;${var.resource_prefix}-kinesis-vpc-endpoint&quot;<br>      }<br>    }<br>  }<br>  depends_on = [<br>    module.vpc<br>  ]<br>}</pre><p>There are a number of ways to architect this. However, we’ve chosen this path to reduce writing as much novel Terraform code as possible.</p><h3>Wrapping It All Up:</h3><p>As we mentioned at the beginning of this blog post, there are numerous ways to structure Terraform repositories for deploying Databricks environments, and this is just one of many examples.</p><p>However, regardless of your structure, we recommend adhering to the two principles of isolation and reusability in 2024.</p><p><strong>Isolation</strong>: Isolate your environment to prevent collisions between environments. This approach helps avoid rate limits or circular dependencies. Importantly, we want to ensure that an update in a development environment does not inadvertently force a refresh of the production state.</p><p><strong>Reusability</strong>: Reduce the workload for your DevOps team over the year by writing common, reusable modules that can be leveraged across all environments. This not only ensures consistency within your Databricks workspaces but also allows for the easy deployment of new resources and the updating of existing ones.</p><p>As noted above, we’ve published a working example repository for this blog post <a href="https://github.com/JDBraun/dbx_mws_example">here</a>. No warranties or guarantees, but GitHub issues are appreciated.</p><p>So, what are your thoughts? Do you think we missed anything? Are there any other key components essential for deploying scalable infrastructure on Databricks that we should include?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ac5f816c3dda" width="1" height="1" alt=""><hr><p><a href="https://medium.com/databricks-platform-sme/how-wed-set-up-our-databricks-deployment-pipelines-with-terraform-repositories-in-2024-ac5f816c3dda">How we’d set-up our Databricks deployment pipelines with Terraform repositories in 2024</a> was originally published in <a href="https://medium.com/databricks-platform-sme">Databricks Platform SME</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Isolake — A simplistic deployment design to an isolated Databricks Lakehouse on AWS]]></title>
            <link>https://medium.com/databricks-platform-sme/isolake-a-simplistic-deployment-design-to-an-isolated-databricks-lakehouse-on-aws-c0f98b5bbba0?source=rss-c22ed87dd65------2</link>
            <guid isPermaLink="false">https://medium.com/p/c0f98b5bbba0</guid>
            <category><![CDATA[security]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[JD Braun]]></dc:creator>
            <pubDate>Fri, 08 Dec 2023 20:54:45 GMT</pubDate>
            <atom:updated>2024-12-28T17:35:00.466Z</atom:updated>
            <content:encoded><![CDATA[<h3>Isolake — A simplistic deployment design to an isolated Databricks Lakehouse on AWS</h3><p>A massive thank you to my co-contributors to this blog post and Terraform repo: <a href="https://www.linkedin.com/in/wenxin-liu-22b33492/">Wenxin Liu</a> and <a href="https://www.linkedin.com/in/ryan-gordon-0781055/">Ryan Gordon</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bAqw3jm47CidVWmc" /><figcaption>ChatGPT generated image of an isolated Lakehouse</figcaption></figure><h3><strong><em>TL;DR — Can we isolate the Lakehouse?</em></strong></h3><p>For readers who want to dive directly into the code without the background, architecture, and demo walkthrough.</p><p><strong>What is Isolake</strong>:</p><ul><li>Isolake is a simple and specialized Databricks workspace deployment design on AWS that isolates users and workloads from the public internet, utilizing Unity Catalog and AWS PrivateLink as its foundational architectural components. While this approach is already adopted by many Databricks customers, I am very excited to document, Terraform, and share this pattern with the broader tech community.</li></ul><p><strong>Highlights:</strong></p><ul><li>No outbound internet connectivity.</li><li>Scoped-down policies for VPC endpoints, storage credentials, and the cross-account role.</li><li>Integrated with Databricks enterprise security features like customer-managed keys.</li><li>Optional access solely via Amazon AppStream.</li></ul><p><strong>Total Cost of Ownership (TCO):</strong></p><ul><li>Databricks enterprise tier pricing based on uptime for Databricks clusters.</li><li>Uptime on underlying AWS compute infrastructure (EC2, EBS, etc).</li><li>Uptime and throughput on AWS networking components (e.g., PrivateLink endpoints).</li><li>KMS costs for two customer-managed keys.</li><li>Data storage costs in S3.</li><li><strong>Optional</strong> — Uptime on AWS AppStream Fleet.</li></ul><p><strong>UPDATE (12/24)</strong>: For the most up-to-date code for an isolated architecture, please use the <a href="https://github.com/databricks/terraform-databricks-sra/tree/main/aws">Security Reference Architecture — Terraform Templates</a>.</p><p>Front-end connectivity through AppStream not provided. Please reach out directly through <a href="https://www.linkedin.com/in/jdbraun1/">LinkedIn</a> if you’re interested in this front-end architecture.</p><h3>Background — Complete Flexibility to Utter Isolation</h3><p>Ever since I started using Databricks in 2018, I’ve been a fan of the flexibility it offered. I enjoyed how easy it was to spin up a cluster, open a notebook, start coding away in SQL or Python, download any and all packages from PyPI, and use whichever public API was available to me.</p><p>However, while I was enjoying this total freedom to analyze, enrich, and publish datasets, I wasn’t considering the trade-off I was making with security. I wasn’t vetting Python packages, verifying the public APIs I was using, or auditing my actions.</p><p>Was this because I was a malicious user? Absolutely not. I was simply unaware of the potential risks when given the flexibility to execute code in an environment that can connect to the public internet.</p><p>But, a lot has changed since 2018. Security teams are now much more aware of how to handle data tooling in a cloud environment. Over the past two years, I’ve worked closely with security teams to harden their Databricks Lakehouses. From implementing data exfiltration firewalls and writing Terraform scripts for deploying Databricks with security best practices to analyzing CloudTrail logs, the plethora of tools to mitigate risk has been continually growing.</p><p>However, during this time, I’ve seen some recurring use cases:</p><ul><li>A data analyst for an insurance company running SQL queries on highly sensitive data with read-only permissions.</li><li>A data scientist training a large language model on GPU clusters and needing to de-identify PII data in the process.</li><li>A data engineer writing native PySpark code to process customer data, with a stringent requirement that all customer data be processed within their virtual private cloud (VPC).</li><li>A consultant or a third party handling sensitive data that isn’t allowed to be copied onto their local machine.</li></ul><p>In each of these use cases, there is a key trend — absolutely no interaction with the public internet. If that’s the case, why do we still offer these personas the opportunity to go outbound to the World Wide Web?</p><p>Can we simplify this architecture and mitigate data leakage from non-security-conscious users to a significant extent?</p><p>Can we <strong>iso</strong>late the <strong>lake</strong>house?</p><h3>An Isolated Deployment Design — Isolake</h3><p>The answer is: <strong>yes</strong>.</p><p>The Isolake deployment design attempts to do just that by hardening a Databricks workspace on AWS with no outbound internet connectivity, scoped-down policies, enterprise security Databricks features, and optionally accessible only through AWS AppStream.</p><p>Let’s talk about why Databricks makes this level of isolation so simple to achieve. It boils down to three areas:</p><ul><li>Extensive built-in features for every data persona like a SQL interface, machine learning training, multi-language capability, and flexibility in cluster selection.</li><li>As a platform-as-a-service, a very simple networking story which includes no public IPs on instances, no inbound communication to the clusters, and generally available support for backend connectivity to the Databricks control plane through PrivateLink.</li><li>A centralized governance tool in Unity Catalog that handles fine-grained controls and short-term credential generation to access data from S3 buckets.</li></ul><p>These cornerstones are what enable this solution to be so straightforward. Without this feature set, here’s what we are looking at:</p><ul><li>Piecing together multiple tools to fit each persona like a data warehouse, a notebook service, etc., or forcing a round circle into a square hole by making personas do their day-to-day in environments that aren’t familiar to them.</li><li>If the underlying instances required public IPs, we’d be looking at a much different networking design story. We’d be managing complex security group rules and have concerns over inbound connections from the public internet.</li><li>Without Unity Catalog, managing data permissions would become burdensome. Instead of defining the Identity Access Management (IAM) permissions once and then using Unity Catalog’s fine-grained controls, we’d either be adding very permissive IAM roles to cover each use case or adding a lot of IAM roles for each persona.</li></ul><p>If our goal is to satisfy the requirements of someone like past data engineer JD from accidentally leaking data or interacting with malicious public internet assets, we’ve hit the mark.</p><p>In our next sections, we’ll be taking a deeper look at this architecture from the backend perspective — clusters interacting with Databricks and AWS services. Then from the frontend perspective — users interacting with the workspace.</p><p>At the end of the blog post, I outline experimental hardening methods. These include scoped-down network access control lists (NACLs) on the private subnets and bucket policies for the root storage of Databricks, Unity Catalog, and the data bucket.</p><h3><strong>Before we continue — Is this design for everyone?</strong></h3><p>The answer is: <strong>no</strong>.</p><p>Clearly, this type of design will not fit many use cases, and I would never recommend it in all scenarios. There are obvious downsides to isolating users from the internet, as there are countless safe assets that can be used to enrich datasets.</p><p>If you’re looking to harden your Databricks deployments but do not require this level of isolation, take a look at the following amazing resources:</p><ul><li><a href="https://www.databricks.com/blog/2021/02/02/data-exfiltration-protection-with-databricks-on-aws.html"><strong>Data Exfiltration Protection with Databricks on AWS</strong></a>: A blog post outlining a standard Data Exfiltration protection strategy featuring a firewall to restrict outbound traffic.</li><li><a href="https://github.com/databricks/terraform-databricks-sra"><strong>Security Reference Architecture — Terraform Templates</strong></a>: A Databricks field engineering and customer collaborative effort to create reusable Terraform templates that integrate security best practices.</li><li><a href="https://www.databricks.com/sites/default/files/2023-03/Databricks-AWS-Security-Best-Practices-and-Threat-Model.pdf"><strong>Security Best Practices for Databricks on AWS</strong></a>: A list of best practices for Databricks on AWS, including common threat scenarios and how to resolve them.</li><li><a href="https://www.databricks.com/blog/2023/02/03/announcing-multi-cloud-support-security-analysis-tool-sat.html"><strong>Security Analysis Tool</strong></a>: A multi-cloud tool that can be used to help harden your Databricks environment by reviewing current deployments against best practices.</li></ul><h3>Building Isolake — Backend Architecture:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7yRpZAUw6idkeLf0Cipbig.png" /><figcaption>Image of backend network diagram of the Isolake architecture</figcaption></figure><p>Backend network diagram of the Isolake architecture — So, how does a Databricks cluster spin up in the Isolake design?</p><p>Let’s use the situation where a user logs into Databricks and spins up a cluster in the console:</p><ul><li>A cross-account role is assumed by Databricks to spin up the Elastic Compute Cloud (EC2) instances in a private subnet.</li><li>The EC2 instance, with a Databricks Amazon Machine Image (AMI), grabs artifacts from a Databricks-hosted Simple Storage Service (S3) bucket over the S3 gateway endpoint.</li><li>The cluster resolves connections to the control plane and to Security Token Service (STS) and Kinesis using the Virtual Private Cloud (VPC) interface endpoints.</li></ul><p>These are three easy steps with traffic only ever leaving the boundaries of the VPC to head to the S3 gateway endpoint. Once the cluster shows the green checkmark in the workspace console, the user can start going about their day-to-day job. All the code they write will be passed through the secure cluster connectivity relay, and Unity Catalog will handle any credential generation, given they have the permissions.</p><p>You might be thinking, <strong>wait a second</strong>, what about the Hive Metastore? That’s clearly called out in the Databricks documentation as not being supported by the PrivateLink endpoint.</p><p>And that is correct, it’s not. Instead, to cover this gap, in the classic cluster and SQL warehouse configuration, we’ll be using Apache Derby.</p><p><a href="https://kb.databricks.com/metastore/set-up-embedded-metastore">Apache Derby</a> is an embedded metastore, which can be used when you only need to retain table metadata during the life of the cluster. Since we’re going to be using Unity Catalog for all things metadata and governance, we don’t need that outbound connection to a Hive metastore — maintaining our isolation from the public internet.</p><p>Using Derby lets us move past any errors that we may see from Spark’s dependencies on the Hive metastore, like not being able to see the sample data that’s included in workspace deployments.</p><p>To include Apache Derby, you only need to add the following lines to either the Spark configuration of classic compute or the data access configuration of the SQL warehouse and click run:</p><pre>spark.hadoop.javax.jdo.option.ConnectionUserName admin<br>spark.hadoop.javax.jdo.option.ConnectionURL jdbc:derby:memory:myInMemDB;create=true<br>spark.hadoop.javax.jdo.option.ConnectionDriverName org.apache.derby.jdbc.EmbeddedDriver</pre><ul><li><strong>WARNING — Temporary Hive Metastore: </strong>Please note that since we are using a temporary Hive metastore with Apache Derby, any data saved to the Hive metastore will be lost when the cluster is terminated. It’s crucial to use Unity Catalog for all data-related activities to avoid data loss. Alternatively, consider using an external hive metastore within the same VPC as a more permanent solution until Unity Catalog becomes the only option within the platform.</li><li><strong>WARNING — Serverless SQL and External Hive Metastore Capabilitiy: </strong>Currently, Serverless SQL does not support an external hive metastore. Adding the Derby configuration to the data access settings in the SQL admin will lead to issues in spinning up Serverless SQL warehouses. If you are using SQL warehouses, we recommend opting for serverless by default and not including the Derby configuration. This has numerous benefits, including maintaining the isolation boundary by avoiding internet connectivity. If your corporate policy dictates that serverless cannot be used, it is advisable to add the Derby configurations.</li></ul><p>Let’s go ahead and walk through a demo of this backend connection in a series of pictures. I’ll show the networking configuration on the EC2 instance, the S3 gateway endpoint policy, the interface endpoints, and bring it together by performing a basic task that a data engineer and analyst would do — query data, write it to Unity Catalog, and then query it with SQL.</p><p>This demo will visually guide you through the set-up, but if at this point you want to dig into the code and get going, be sure to check out my Terraform repo.</p><h3>A quick note on the high availability of AWS services:</h3><p>As is common with cloud services, most API calls typically start by trying to reach their local destination, such as s3.us-east-1.amazonaws.com. However, should that not be available, they will fall back to the global URL, s3.amazonaws.com.</p><p>In this deployment design, we are dependent on regional endpoints for a functioning workspace. Therefore, we recommend that if your workspace needs an added layer of availability, follow the Data Exfiltration example above and allowlist those AWS services’ global names as needed.</p><p>In rare cases, some Databricks platform APIs may attempt to use the global name of AWS services. If this occurs, please set variables appropriately. For example, with MLFlow:</p><pre>%sh<br>MLFLOW_S3_ENDPOINT_URL= &#39;s3.&lt;region&gt;.amazonaws.com&#39;</pre><h3>Buidling Isolake — Backend Demo:</h3><p><strong>Databricks cluster — EC2 instances:</strong></p><p>Let’s start at our lowest common denominator, the underlying EC2 instance. I’ve gone ahead and spun up a cluster using the workspace console to check out the associated security group.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KcRCztK9q9H3cRW3ozkg_w.png" /><figcaption>Image of the Security Groups</figcaption></figure><p>The security groups, or the firewall around the EC2 instance, only allow for communication between assets within the same security group and outbound to the prefix list for the S3 gateway endpoint. This means any traffic trying to reach anything but these assets is blocked — a meaningful first step in keeping our traffic isolated.</p><p><strong>Going outbound </strong>— <strong>route tables:</strong></p><p>Even though we have the security group preventing traffic outside the security group, let’s take a look at the route table and where the network traffic will be routed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h2T-nKOw29kz6BT_XbqW5A.png" /><figcaption>Image of the Route Table</figcaption></figure><p>In the route table, we see traffic can only go to private IPs within the same VPC, as well as the prefix list for the S3 gateway endpoint. In most route tables in an AWS setting, you’ll see a route for 0.0.0.0/0. This is a catch-all that routes the remaining IP addresses to follow. We don’t have this in Isolake, so effectively any traffic not routing to the local VPC CIDR of 10.0.0.0/23 or the prefix list of IPs for the S3 gateway endpoint is going up a cul-de-sac.</p><p><strong>Going to the data — S3 gateway endpoint policy:</strong></p><p>With our security group rules and our route tables, we’ve covered outbound public internet traffic. But, despite not being able to access the public internet, what about accessing rogue buckets to exfiltrate data?</p><pre>{<br> &quot;Version&quot;: &quot;2012-10-17&quot;,<br> &quot;Statement&quot;: [<br>  {<br>   &quot;Sid&quot;: &quot;Grant access to Databricks Root Bucket&quot;,<br>   &quot;Effect&quot;: &quot;Allow&quot;,<br>   &quot;Principal&quot;: {<br>    &quot;AWS&quot;: &quot;*&quot;<br>   },<br>   &quot;Action&quot;: [<br>    &quot;s3:PutObject&quot;,<br>    &quot;s3:ListBucket&quot;,<br>    &quot;s3:GetObjectVersion&quot;,<br>    &quot;s3:GetObject&quot;,<br>    &quot;s3:GetBucketLocation&quot;,<br>    &quot;s3:DeleteObject&quot;<br>   ],<br>   &quot;Resource&quot;: [<br>    &quot;arn:aws:s3:::jd-dbfs-uc/*&quot;,<br>    &quot;arn:aws:s3:::jd-dbfs-uc&quot;<br>   ],<br>   &quot;Condition&quot;: {<br>    &quot;StringEquals&quot;: {<br>     &quot;aws:PrincipalAccount&quot;: &quot;414351767826&quot;<br>    },<br>    &quot;StringEqualsIfExists&quot;: {<br>     &quot;aws:SourceVpc&quot;: &quot;vpc-0cedbf325cfe57e27&quot;<br>    }<br>   }<br>  },<br>  {<br>   &quot;Sid&quot;: &quot;Grant access to Databricks Unity Catalog Metastore Bucket&quot;,<br>   &quot;Effect&quot;: &quot;Allow&quot;,<br>   &quot;Principal&quot;: {<br>    &quot;AWS&quot;: &quot;*&quot;<br>   },<br>   &quot;Action&quot;: [<br>    &quot;s3:PutObject&quot;,<br>    &quot;s3:ListBucket&quot;,<br>    &quot;s3:GetObjectVersion&quot;,<br>    &quot;s3:GetObject&quot;,<br>    &quot;s3:GetBucketLocation&quot;,<br>    &quot;s3:DeleteObject&quot;<br>   ],<br>   &quot;Resource&quot;: [<br>    &quot;arn:aws:s3:::jd-isolake-uc/*&quot;,<br>    &quot;arn:aws:s3:::jd-isolake-uc&quot;<br>   ]<br>  },<br>  {<br>   &quot;Sid&quot;: &quot;Grant read-only access to Data Bucket&quot;,<br>   &quot;Effect&quot;: &quot;Allow&quot;,<br>   &quot;Principal&quot;: {<br>    &quot;AWS&quot;: &quot;*&quot;<br>   },<br>   &quot;Action&quot;: [<br>    &quot;s3:ListBucket&quot;,<br>    &quot;s3:GetObjectVersion&quot;,<br>    &quot;s3:GetObject&quot;,<br>    &quot;s3:GetBucketLocation&quot;<br>   ],<br>   &quot;Resource&quot;: [<br>    &quot;arn:aws:s3:::&lt;bucket name&gt;/*&quot;,<br>    &quot;arn:aws:s3:::&lt;bucket name&gt;&quot;<br>   ]<br>  },<br>  {<br>   &quot;Sid&quot;: &quot;Grant Databricks Read Access to Artifact and Data Buckets&quot;,<br>   &quot;Effect&quot;: &quot;Allow&quot;,<br>   &quot;Principal&quot;: {<br>    &quot;AWS&quot;: &quot;*&quot;<br>   },<br>   &quot;Action&quot;: [<br>    &quot;s3:ListBucket&quot;,<br>    &quot;s3:GetObjectVersion&quot;,<br>    &quot;s3:GetObject&quot;,<br>    &quot;s3:GetBucketLocation&quot;<br>   ],<br>   &quot;Resource&quot;: [<br>    &quot;arn:aws:s3:::databricks-prod-artifacts-us-east-1/*&quot;,<br>    &quot;arn:aws:s3:::databricks-prod-artifacts-us-east-1&quot;,<br>    &quot;arn:aws:s3:::databricks-datasets-virginia/*&quot;,<br>    &quot;arn:aws:s3:::databricks-datasets-virginia&quot;<br>   ]<br>  },<br>  {<br>   &quot;Sid&quot;: &quot;Grant access to Databricks Log Bucket&quot;,<br>   &quot;Effect&quot;: &quot;Allow&quot;,<br>   &quot;Principal&quot;: {<br>    &quot;AWS&quot;: &quot;*&quot;<br>   },<br>   &quot;Action&quot;: [<br>    &quot;s3:PutObject&quot;,<br>    &quot;s3:ListBucket&quot;,<br>    &quot;s3:GetBucketLocation&quot;<br>   ],<br>   &quot;Resource&quot;: [<br>    &quot;arn:aws:s3:::databricks-prod-storage-virginia/*&quot;,<br>    &quot;arn:aws:s3:::databricks-prod-storage-virginia&quot;<br>   ],<br>   &quot;Condition&quot;: {<br>    &quot;StringEquals&quot;: {<br>     &quot;aws:PrincipalAccount&quot;: &quot;414351767826&quot;<br>    }<br>   }<br>  }<br> ]<br>}</pre><p>This is where our restrictive S3 gateway endpoint policy takes over. This prevents users from writing to buckets not covered in the list. In this policy, we include the needed S3 buckets from a Databricks perspective, as well as the data bucket, in this case, my CloudTrail logs bucket.</p><p><strong>How is this even working— AWS PrivateLink:</strong></p><p>AWS PrivateLink allows traffic to traverse the AWS backbone to native and third-party services, by creating endpoints that exist within your VPC.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7hNK8Mbitv-yM6B3u3pXUQ.png" /><figcaption>Image of the list of the VPC endpoints</figcaption></figure><p>The endpoints needed for a functioning cluster are:</p><ul><li><strong>S3</strong> — gateway endpoint</li><li><strong>Kinesis</strong> — interface endpoint</li><li><strong>STS</strong> — interface endpoint</li><li><strong>Databricks backend: REST API </strong>— interface endpoint</li><li><strong>Databricks backend: Secure Cluster Connectivity Relay</strong> — interface endpoint</li></ul><p>The interface endpoints are located within the cluster subnets and have the same security group, shown above, attached to them. For Kinesis and STS, we have included scoped-down policies that you can find in their respective Terraform files in the <a href="https://github.com/JDBraun/isolake">GitHub repo</a>.</p><p><strong>Now what — working with your data:</strong></p><p>Let’s embrace the full Lakehouse and explore and clean a dataset (CloudTrail logs), create a table in Unity Catalog, and use a SQL warehouse to query it. The first step is to triple-check we’re not making any outbound connections with an attempt to use a GET call against <strong>api.ipify.org</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U423cqtOUPreuE_L5YEHqg.png" /><figcaption>Image of a failed called to a public facing IP</figcaption></figure><p>Nothing, we’re not making our way to the internet. Next up, utilizing Unity Catalog as our centralized governance tool, we’ll query our CloudTrail logs and create a cleaned-up DataFrame.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rA4a_LD1Gcfu9RiNG7DMZA.png" /><figcaption>Image of queried CloudTrail logs in a tabular format</figcaption></figure><p>Now that we have queried the data, let’s create a table in Unity Catalog and use the great new AI features to summarize and add comments to the column.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G5OOSEErSzcFZmzYmIPe4Q.png" /><figcaption>Image of writing the table to Unity Catalog</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ge07db6xxHb1NVL6OK1D2Q.png" /><figcaption>Image of the table in Unity Catalog</figcaption></figure><p>The data has been cleansed, a managed table has been created in Unity Catalog, and now let’s use a SQL warehouse to run standard ANSI-SQL to query it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6PuuqMw0B9z5QSf55JHnYg.png" /><figcaption>Image of querying the data with a SQL warehouse</figcaption></figure><p><strong>Where do we go from here — continuing on the Databricks Lakehouse Journey</strong>:</p><p>With our isolated Databricks Lakehouse, we’re free to let data analysts, engineers, and scientists make key business decisions with a drastically reduced risk that a non-security-conscious user could exfiltrate data or use malicious public internet access.</p><p>If you’re sitting here wondering about bucket policies and network access control lists, be sure to check out my experimental section at the end of the blog post.</p><p>But…</p><p>What if we wanted to take it a step further? What if we only wanted users to access Databricks from a machine that we can control, monitor, and prevent common activities like copying, pasting, or admins downloading results onto their local machines?</p><p>Onto building the optional frontend architecture!</p><h3>Building Isolake — Frontend Architecture:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zonKRPyIBHgeYTVX5n4sRQ.png" /><figcaption>Image of a frontend network diagram of the Isolake architecture</figcaption></figure><p>Now that we’ve covered the backend of Isolake, let’s dive into the optional frontend of Isolake and why it’s important for certain organizations.</p><p>While working within a web browser, a user typically still has administrative control of their local machine. This means they can copy code, paste data from their clipboard, or download the results of a DataFrame if they have enough permissions.</p><p>This level of flexibility, similar to running arbitrary code at the intersection of the public internet, isn’t inherently a bad thing. It can only create accidental use cases, like someone moving a subset of code to a spreadsheet to share with coworkers or moving code to non-version-controlled systems.</p><p>So, this type of solution will be specific to organizations where this is a clear concern. A few use cases have included:</p><ul><li>Remote data personas that need to access sensitive data to perform analysis, but managing the DNS resolution off VPNs or on VPNs becomes too burdensome or complex.</li><li>Consultants or other third parties that need to access the workspace but do not have a corporate-approved laptop.</li><li>Intra-organization data sharing and analysis. If there are analysts that extend across business groups, access needs to be granted for a short period to the workspace to run certain jobs or analyze certain data points.</li></ul><p>And let me be clear, while this may mitigate copying and pasting from the environment, it will not stop a screenshot from the computer, a phone picture, or someone looking over their shoulder and writing notes. This is meant to be another mitigation tactic to isolate the frontend of the Lakehouse.</p><p>Now, as a final reminder, this is entirely optional in the Terraform code. Amazon AppStream was selected based on my prior experience in building these secure data environments. This can be interchangeable based on your toolkit. Before we do the demo of this, let’s talk a little more about AppStream.</p><p>AppStream is a secure, fully managed application streaming service that lets you deploy streaming URLs that can be used directly from a user’s desktop. While I’m only using Firefox embedded in the sample image, these images can be fully customizable to fit your organization’s specific needs.</p><p>Now, onto the demo!</p><h3>Building Isolake — Frontend Demo:</h3><p><strong>A wrapper — Amazon AppStream</strong></p><p>AppStream has two key concepts: <a href="https://docs.aws.amazon.com/appstream2/latest/developerguide/managing-stacks-fleets.html">Fleets and Stacks</a>.</p><ul><li>A <strong>Fleet</strong> is the streaming instances that run the image you specify.</li><li>A <strong>Stack</strong> consists of an associated fleet, user access policies, and storage configurations.</li></ul><p>In our Fleet example below, we specify an on-demand type with a stream.standard.small instance type. This instance is configured to run within a given networking setup pictured below.</p><p>The Stack is where we configure specific user settings that are key to our Isolake isolation principles like disabling file download, disabling file upload, and preventing clipboard copying from local devices.</p><p>From the Fleet pane, we can generate a streaming URL to access the image. You can also use a <a href="https://docs.aws.amazon.com/appstream2/latest/developerguide/use-client-start-streaming-session.html#use-client-start-streaming-session-user-pool">User Pool</a> to generate a URL for the user as well as a welcome message.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2cM1dB-7GDipmkW6n4f2wA.png" /><figcaption>Image of an AppStream Fleet</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7NFzcFIjwOjUoQ8buJMsFg.png" /><figcaption>Image of an AppStream Network Details</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t47V8qjmEdbuG73Lp5aDUA.png" /><figcaption>Image of an AppStream Stack</figcaption></figure><p>Once we’ve logged into our AppStream instance, as pictured below with the workspace, we can use Firefox to navigate to our workspace URL. But there are two crucial steps to ensure that connection works: PrivateLink and DNS resolution.</p><p><strong>How is this working — PrivateLink and DNS Resolution</strong></p><p>There are two key features that need to be enabled for an individual entering through AppStream to be set up correctly so that a workspace can be accessed:</p><ol><li>An interface endpoint for the Databricks REST Endpoint. This interface endpoint, similar to how it works for backend connectivity, allows traffic to access the Databricks workspace WebApp on the AWS backbone, without needing to access the WebApp through a public IP address.</li><li>A private hosted zone (PHZ) associated with the AppStream VPC. This resolves any records going out to the workspace to the Private IP of the interface endpoint, instead of to the public IP.</li></ol><p>With this configured, you’ll be able to access the workspace by going into AppStream, opening up Firefox, and entering your domain name.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EAxmhpcW4QTzE4GhWXUZgg.png" /><figcaption>Image of PrivateLink endpoints</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vtugMIpt-mUi7Z4_ePUqgA.png" /><figcaption>Image of a Private Hosted Zone</figcaption></figure><p><strong>Locking it up — Private Access Setting</strong></p><p>This is great, but what about individuals trying to access the workspace from the public internet? Won’t that still be available?</p><p>This is where the Databricks Private Access Setting (PAS) object comes into play. This object is a must-have when using a PrivateLink-enabled workspace. Within it, there is a configurable setting called “Public access”. When set to enabled, the workspace will accept inbound connections not coming from the PrivateLink endpoint. If it’s set to disabled, then it will either accept connections coming from any endpoint in the account or only specified endpoints.</p><p>In the following images, we can see the private access setting object, accessing the workspace from the AppStream instance, and the denial that I got when I tried to access the workspace from a browser over the public internet.</p><p><strong>Note</strong>: As I mentioned, this is an optional setting in the Terraform code. If you switch it to true to enable it, be sure to swap out the private access setting from <strong>isolake-pas</strong> to <strong>isolake-pas-lockdown</strong> within the Databricks account console.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*unGa-TweBWof1yq8bGYJSQ.png" /><figcaption>Image in Private Access Setting Object in Databricks</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vz12eIhPv7rHDnCfisUm9w.png" /><figcaption>Image of accessing a Databricks workspace in an AppStream Instance</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*enzCVndJQmXLR9G_oMDxrQ.png" /><figcaption>Image of accessing Databricks in a normal web browser</figcaption></figure><p>Now with that settled, we have implemented isolation from both the backend and frontend of the Lakehouse.</p><p>Can we have more though? Can we implement more hardening into the platform features? I mentioned it above, so let’s dig into it with the next section on <strong>experimental hardening methods for Databricks workspaces</strong>.</p><h3><strong>Experimental Hardening Methods</strong></h3><p><strong>WARNING:</strong></p><p>In this section, we’ll be discussing experimental hardening methods. These are methods that have a distinct possibility of breaking certain functionalities of the workspace or causing user lockout from certain assets (e.g., S3 buckets).</p><p><strong>PROCEED WITH CAUTION.</strong></p><p><strong>Cluster Private Subnets — Network Access Control Lists (NACL)</strong>:</p><p>While security groups manage traffic at the instance level, NACLs manage traffic at the subnet level. NACLs, being stateless, can become difficult and challenging to manage at scale, which is why there is a preference toward managing network traffic with security groups and egress firewalls.</p><p>However, some organizations have policies that prohibit 0.0.0.0/0 allows on NACLs, despite the traffic not being routable to any public internet destination.</p><p>In this experimental feature, after the workspace has been deployed, I restrict the NACLs to only allowing traffic out of the VPC to the IPs found in the prefix list for the S3 gateway endpoint.</p><p>Since Databricks workspace validation checks if NACLs are set up correctly, this action can only be taken post-deployment. Thus, in your account console, you will see error messages around the networking configuration.</p><p><strong>Unity Catalog Bucket Policy</strong>:</p><p>Unity Catalog buckets for the metastore typically come without bucket policies. In this feature, we restrict the bucket policy to only allow certain actions from both the Databricks control plane, the VPC of the clusters, as well as the role and system IP of an admin who is responsible for creating and deleting the bucket.</p><p>This policy can cause lockout if the system IP and role are incorrect on deployment.</p><p><strong>Databricks Workspace Root Storage Bucket Policy</strong>:</p><p>The Databricks root bucket has a specific bucket policy that allows the Databricks control plane to write various notebook assets like job results, long query results, MLFlow, DBFS, etc. In this bucket policy, post-deployment, we restrict it to certain prefixes that can be written to.</p><p>This list of S3 prefixes is expanding based on what feature set may write to these buckets, thus can cause downstream issues if a new feature requires a new pathway.</p><p><strong>Data Bucket Policy</strong>:</p><p>To further enforce a layer of read-only access, the bucket policy for the data bucket in question is updated to only allow reading from the VPC that houses the clusters. This, paired along with the restrictive S3 gateway endpoint policy, creates a defense-in-depth scenario.</p><p>This policy can cause lockout if the system IP and role are incorrect on deployment.</p><h3>Wrapping it all up:</h3><p>In conclusion, Databricks’ flexibility in executing code can be strenuous for security teams to manage in certain situations. In the Isolake deployment design, we simplify that pattern to isolate the backend and optionally the front end from public assets while enabling the power of the built-in features of the Databricks Lakehouse.</p><p>Please take a look at our <a href="https://github.com/JDBraun/isolake">GitHub</a> repository for the Terraform code to implement this today. As always, if you see something, say something — I’d gladly respond to any <a href="https://github.com/JDBraun/isolake/issues">GitHub</a> issues.</p><p>Till next time… In the meantime, I’ll be hiding away in my isolated Lakehouse.</p><p><strong>Disclaimer</strong>: This blog post is my personal opinion and in no way represents the beliefs of my employer.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c0f98b5bbba0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/databricks-platform-sme/isolake-a-simplistic-deployment-design-to-an-isolated-databricks-lakehouse-on-aws-c0f98b5bbba0">Isolake — A simplistic deployment design to an isolated Databricks Lakehouse on AWS</a> was originally published in <a href="https://medium.com/databricks-platform-sme">Databricks Platform SME</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>