<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Kaustubh Mondal on Medium]]></title>
        <description><![CDATA[Stories by Kaustubh Mondal on Medium]]></description>
        <link>https://medium.com/@kaustubhwb?source=rss-c643e5805514------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*r4u8N0P7a_jtGFKB</url>
            <title>Stories by Kaustubh Mondal on Medium</title>
            <link>https://medium.com/@kaustubhwb?source=rss-c643e5805514------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 08:13:11 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@kaustubhwb/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Azure Databricks Networking Architecture — Part 1]]></title>
            <link>https://medium.com/@kaustubhwb/azure-databricks-networking-architecture-part-1-4921cfb92755?source=rss-c643e5805514------2</link>
            <guid isPermaLink="false">https://medium.com/p/4921cfb92755</guid>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[networking]]></category>
            <category><![CDATA[enterprise-architecture]]></category>
            <dc:creator><![CDATA[Kaustubh Mondal]]></dc:creator>
            <pubDate>Sat, 02 May 2026 12:18:44 GMT</pubDate>
            <atom:updated>2026-05-02T12:54:27.684Z</atom:updated>
            <content:encoded><![CDATA[<h4>Introduction and Options</h4><p>· <a href="#6d4a">Who is This Article For?</a><br>· <a href="#989a">Introduction</a><br>· <a href="#80a2">Basic Concepts</a><br> ∘ <a href="#4d5f">V-Net Injection</a><br> ∘ <a href="#d5c4">Serverless</a><br> ∘ <a href="#0874">Secure Cluster Connectivity (SCC)</a><br> ∘ <a href="#6ac3">Private Endpoints</a><br> ∘ <a href="#8596">Service Endpoints</a><br>· <a href="#f7e0">Key Decision Points</a><br> ∘ <a href="#e6f3">Decision Point 1: Hub and Spoke Architecture</a><br> ∘ <a href="#9dbc">Decision Point 2: Workspace Access Control</a><br> ∘ <a href="#0b8a">Decision Point 3: Network Traffic Regulation</a><br> ∘ <a href="#0a1b">Decision Points 4.1 and 4.2: Backend Access Control</a><br> ∘ <a href="#b577">Decision Point 5: Data Access Control</a><br>· <a href="#771a">Putting It All Together</a><br>· <a href="#fa1b">Making Your Decision</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7FXUZGw6cBvdOA5J1CJwJw.png" /></figure><p><a href="https://medium.com/@kaustubhwb/4921cfb92755?source=friends_link&amp;sk=81ceef544413c64ac64af2cccf03fcd4"><strong>Click here</strong></a><strong> to access this article for free</strong></p><h3>Who is This Article For?</h3><p>This article is the first in a series of four designed for anyone involved in the planning, implementation, and governance of Azure Databricks deployments. It is intended for, but not restricted to:</p><ul><li><strong>Solution/Cloud Architects</strong>: Determining the optimal approach for designing secure and scalable Databricks environments, with considerations for network isolation, private connectivity, and enterprise-grade security controls.</li><li><strong>Network &amp; Security Engineers:</strong> Understanding the basics of configuring secure cluster connectivity (SCC) and UDRs, managing NSG rules and service endpoints, enforcing egress lockdown or private endpoint connectivity</li><li><strong>Developers &amp; Data Engineers</strong>: Understanding how networking impacts library installation, data access, and integration with external services.</li><li><strong>Executives &amp; Decision Makers:</strong> Gaining strategic insight into architectural options and the factors that drive value, security, and total cost of ownership</li></ul><p>Whether you’re building a secure analytics platform or evaluating Databricks for an enterprise rollout, this series is for you. This first article provides a high-level overview and introduces you to the basics and the options available to you for building a scalable and secure Databricks system.</p><h3>Introduction</h3><p>When working with Azure Databricks, networking is a critical component that directly impacts security, scalability, and connectivity. Understanding how data flows between the control plane, data plane, and external services is essential for designing secure and high-performance architectures.</p><p>At a high level, Azure Databricks operates on a <strong>dual-plane architecture</strong>:</p><ul><li><strong>Control Plane</strong>: Managed by Databricks. It handles jobs, notebooks, REST APIs, and cluster management.</li><li><strong>Compute Plane</strong>: Runs in your Azure subscription (V-Net injection) or Databricks managed instances (Serverless). It is where the Databricks clusters actually process data.</li></ul><p>The challenge — and opportunity — lies in managing how these two planes communicate, and how you control access to users, applications, and other resources like Azure Data Lake, Blob Storage, PowerBi, etc.</p><h3>Basic Concepts</h3><p>The basic Concepts of Databricks Networking are:</p><h4>V-Net Injection</h4><ul><li>Databricks is deployed into a <strong>user-managed virtual network (V-Net)</strong></li><li>Users control subnets, NSGs, route tables, and network flow.</li><li>Allows integration with private networks, firewalls, and service &amp; private endpoints.</li></ul><h4>Serverless</h4><ul><li><strong>Databricks Serverless compute </strong>does not run inside the customer’s virtual network<strong> </strong>— it runs in Databricks-managed infrastructure</li><li>It provides instant, fully managed compute clusters that auto-scale and auto-terminate, eliminating the need to manage infrastructure.</li><li>Each workspace runs on isolated compute managed by Databricks, with built-in security controls — ideal for multi-tenant environments</li></ul><h4>Secure Cluster Connectivity (SCC)</h4><ul><li>When <strong>Secure Cluster Connectivity (SCC)</strong> is enabled, Azure Databricks clusters communicate with the Databricks control plane through an outbound-only, secure channel over HTTPS via Databricks relay.</li><li>The clusters poll the SCC relay for responses to their outbound requests to the control plane, effectively blocking any unsolicited inbound traffic.</li></ul><h4>Private Endpoints</h4><ul><li>A <strong>Private Endpoint</strong> creates a dedicated private IP address from your V-Net that maps directly to the Azure service (storage, SQL, etc.). The service appears as an actual network resource within your virtual network, with its own private IP that routes through your V-Net infrastructure.</li><li>Data never leaves the Microsoft Azure backbone and never traverses the public internet. The connection is entirely private, isolated from public networks. Other resources cannot access your endpoint even if they know the service name.</li><li>It provides true network-level isolation. Only resources within your V-Net (or peered V-Nets) can access the service. You can disable the public endpoint entirely. Works with NSGs and works across VNet peering, ExpressRoute, and VPN connections.</li></ul><h4>Service Endpoints</h4><ul><li>Azure service endpoints extend your V-Net identity to Azure services through route optimization. The service retains its public endpoint, but traffic is routed through Microsoft’s backbone network instead of the public internet. Your subnet is registered with the service.</li><li>Traffic flows through Azure’s internal backbone network rather than the public internet, providing better security than internet routing. However, the service endpoint itself remains publicly addressable — other services can potentially reach it if properly authenticated.</li><li>It allows you to implement subnet-level access control through service firewall rules. The service is still reachable from other Azure regions and services unless explicitly blocked. You must maintain both network rules and identity-based access controls. Less granular than Private Endpoints.</li></ul><h3>Key Decision Points</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7FXUZGw6cBvdOA5J1CJwJw.png" /></figure><h4>Decision Point 1: Hub and Spoke Architecture</h4><p>On the left side of the diagram, you’ll notice users connecting through an optional hub infrastructure. This hub-and-spoke topology serves as a centralized gateway for network traffic, housing shared services like Network Security Groups (NSGs), firewalls, and VPN gateways.</p><p><strong>When do you need a hub?</strong> If you’re managing multiple Databricks workspaces, require centralized security policy enforcement, or need to integrate with on-premises networks through ExpressRoute, a hub architecture becomes essential. The hub acts as your single point of control for traffic inspection, logging, and governance across all spoke networks.</p><h4>Decision Point 2: Workspace Access Control</h4><p>The first major security decision addresses how users and applications reach your Databricks workspace. You have two mutually exclusive options:</p><p><strong>Public Network Access</strong> allows users to connect via the internet, with security enforced through IP whitelisting and access control lists. This approach works well for distributed teams or scenarios requiring third-party tool integration.</p><p><strong>Private Network Access</strong> routes all traffic through Azure Private Link, ensuring the workspace URL is only accessible from within your virtual network. This eliminates internet exposure entirely but requires VPN or ExpressRoute for remote users.</p><p>Although hybrid approaches exist for specific use cases, it is recommended to choose one primary access method.</p><h4>Decision Point 3: Network Traffic Regulation</h4><p>These Network Security Groups (NSG) define precisely which traffic can flow in and out of the drivers and workers in the compute plane V-Net, implementing micro-segmentation within your compute environment. The NSG configurations at <strong>Decision Point 3</strong> can provide dual-layer traffic control based on how rigid or how flexible you want the controls to be:</p><p><strong>Perimeter Security</strong>: Controls all traffic crossing the compute plane V-Net boundary — inbound connections from the control plane and outbound access to storage, package repositories, and external services.</p><p><strong>Internal Segmentation</strong>: Manages communication between driver nodes (public subnet) and worker nodes (private subnet), enabling necessary orchestration while restricting the network flow from other resources in the V-Net.</p><h4>Decision Points 4.1 and 4.2: Backend Access Control</h4><p>The backend communication between the control plane and your compute clusters involves two distinct mechanisms, hence the split into 4.1 and 4.2:</p><p><strong>Decision 4.1</strong> addresses the compute plane’s connection architecture. Traditional deployments assigned public IP addresses to cluster nodes, creating potential exposure. Secure Cluster Connectivity (SCC) eliminates these public IPs, forcing clusters to initiate outbound-only connections to the control plane’s relay service. This inverted connection model means no inbound ports are exposed on your cluster nodes.</p><p><strong>Decision 4.2</strong> introduces the concept of <strong>serverless compute</strong> — a newer deployment option. Serverless compute runs on Databricks-managed infrastructure with NCC (Network Connectivity Configuration), abstracting away much of the network complexity. The trade-off? Less control over networking specifics in exchange for simplified operations and faster cluster startup times.</p><h4>Decision Point 5: Data Access Control</h4><p>The final decision addresses how your compute clusters access Azure Blob Storage. You have three primary options:</p><ol><li><strong>Storage Firewalls with IP restrictions</strong> — No additional cost, but the Databricks clusters require fixed egress public IP addresses or NAT gateways so that you can whitelist them in your Azure storage firewall.</li><li><strong>Private Links</strong> — Creates a private endpoint for the Azure storage in your V-Net, ensuring full private connectivity. Only resources within your V-Net (or peered V-Nets) can access the storage endpoint. The traffic never leaves your network for accessing the data, giving you complete network isolation.</li><li><strong>Service Endpoints</strong> — With Service Endpoints, your Azure storage service is still separate from your V-Net. Although traffic will still leave your V-Net to access the data, it will be routed over the Azure network backbone. Azure storage will allow traffic coming from your V-Net, without needing fixed egress IP addresses.</li></ol><p>The choice here significantly impacts both your security compliance posture and your monthly Azure and Databricks bill. Organizations with strict data residency requirements often mandate Private Links despite the cost premium.</p><h3>Putting It All Together</h3><p>The beauty of this architecture lies in its modularity. Each decision point operates somewhat independently, allowing you to mix and match based on your specific requirements. This comparison evaluates three network architecture approaches for connecting users, workspaces, compute clusters, and Azure storage in a cloud environment.</p><ul><li><strong>Private network</strong>: for maximum security but higher cost and complexity</li><li><strong>Balanced hybrid approach</strong>: Reduces cost while maintaining strong security</li><li><strong>Public network</strong>: Reduces the cost and complexity. Acceptable for non-sensitive workloads</li></ul><p>Each architecture presents different trade-offs between security, cost, and complexity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1RNnC5mLq9h9CMXTfsTCnw.png" /></figure><h3>Making Your Decision</h3><p>Start by asking these questions:</p><ol><li><strong>Compliance requirements</strong>: Do regulations mandate private connectivity? (HIPAA, PCI-DSS, etc.)</li><li><strong>Existing infrastructure</strong>: Do you already have a hub architecture for other Azure resources that you can leverage?</li><li><strong>User distribution</strong>: Are users remote/global, or all within corporate networks?</li><li><strong>Operational capacity</strong>: Can your team manage v-net complexity, or do you need the simplicity of serverless?</li><li><strong>Budget constraints</strong>: Can you justify Private Link costs, or are Service Endpoints or firewalls acceptable?</li><li><strong>Data sensitivity</strong>: What’s the impact if data is intercepted or exposed?</li><li><strong>Technology stack</strong>: Does your environment include services, tools, or dependencies that lack private link or service endpoint support?</li></ol><p>The answers to these questions will guide you toward the right combination across these decision points, creating an architecture that balances security, maintainability, and cost for your specific context.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4921cfb92755" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Delta sharing in Azure Databricks]]></title>
            <link>https://medium.com/@kaustubhwb/understanding-delta-sharing-in-azure-databricks-3f101156ed1d?source=rss-c643e5805514------2</link>
            <guid isPermaLink="false">https://medium.com/p/3f101156ed1d</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-architecture]]></category>
            <category><![CDATA[azure-databricks]]></category>
            <category><![CDATA[azure-networking]]></category>
            <category><![CDATA[delta-sharing]]></category>
            <dc:creator><![CDATA[Kaustubh Mondal]]></dc:creator>
            <pubDate>Mon, 16 Jun 2025 20:09:06 GMT</pubDate>
            <atom:updated>2025-06-16T21:16:55.193Z</atom:updated>
            <content:encoded><![CDATA[<h4>From Basics to Advanced</h4><h3>Table of Contents</h3><p>· <a href="#053e">Who is this Article for?</a><br>· <a href="#b084">Introduction</a><br>· <a href="#40d8">What are Provider, Recipient, and Share?</a><br>· <a href="#98f4">Open sharing versus Databricks-to-Databricks sharing</a><br>· <a href="#c1c2">Databricks-to-Databricks Sharing</a><br>· <a href="#85f6">How does Delta sharing work in Azure?</a><br>· <a href="#40c1">Part 1: Data access without network restriction</a><br> ∘ <a href="#5337">Useful links for Part 1</a><br>· <a href="#77d6">Part 2: Data access with network restrictions</a><br> ∘ <a href="#0560">Option 1: Provider enables storage firewall</a><br> ∘ <a href="#74ba">Useful Links for Option 1</a><br> ∘ <a href="#5ec6">Option 2: Provider enables storage private link</a><br> ∘ <a href="#8d47">Useful links for Option 2</a></p><h3>Who is this Article for?</h3><p>This article provides insights into Delta Sharing, an open protocol by Databricks for secure, real-time data sharing across platforms and clouds. The post is intended for, but not restricted to, the following audiences</p><ul><li><strong>Solution Architects</strong>: Provides architecture guidelines and technical capabilities of cross-region, cross-cloud, or inter-departmental data collaboration with the different variations of Delta sharing, with an emphasis on Databricks-to-Databricks Delta sharing in Azure</li><li><strong>Data Engineers &amp; Administrators</strong>: Gives a technical overview of the protocol, with links directing them to a step-by-step setup guide for the implementation of Delta sharing in Azure</li><li><strong>Security/Network Engineers</strong>: Advanced concepts are included for Network and security engineers for integrating Delta Sharing in Azure with enterprise architecture, with options ranging from fully public to secured and private data sharing.</li></ul><h3><strong>Introduction</strong></h3><p>Delta Sharing is an open protocol developed by Databricks for secure data sharing with other organizations regardless of the computing platforms they use.</p><p>It can be used for sharing live, read-only Delta Lake tables irrespective of the tools, cloud storage, or region using open standards (uses REST APIs and shares data in delta format).</p><p>Delta sharing enforces fine-grained access control and a strong data governance backbone when used together with Unity Catalog (now open source)</p><h3>What are Provider, Recipient, and Share?</h3><p>A <strong>Delta Sharing provider</strong> is an entity that hosts and securely shares live Delta Lake data by running a Delta Sharing server. The provider controls which data is shared, manages access permissions, and serves data directly from cloud storage without copying it. A provider can be on a Databricks workspace on Unity catalog or can be using Delta tables on a Delta sharing server.</p><p>A <strong>Delta Sharing recipient</strong> is a user or system that accesses and reads shared Delta Lake data from a Delta Sharing provider. Recipients use a secure credential file to connect and can query the live, read-only data directly with their preferred tools, without needing to copy or store the data locally.</p><p>A <strong>share</strong> in Delta Sharing is a defined set of data objects — like tables or files, that a provider makes available to one or more recipients. It specifies what data is shared<strong> </strong>and who has access to the data.</p><h3>Open sharing versus Databricks-to-Databricks sharing</h3><p>This section describes the two protocols for data sharing from a Databricks workspace that is enabled for Unity Catalog</p><p>The way a provider uses Delta Sharing in Azure Databricks depends on who they are sharing data with:</p><ol><li><em>Open sharing</em> lets you share data with any user, whether or not they have access to Azure Databricks.</li><li><em>Databricks-to-Databricks sharing</em> lets you share data with Azure Databricks users whose workspace is attached to a Unity Catalog metastore that is different from the provider.</li></ol><blockquote><strong>Note:</strong> Delta sharing can be made to work with Hive metastore. However, the architecture of the Open Delta Sharing protocol must be followed in this case.</blockquote><h3>Databricks-to-Databricks Sharing</h3><p>Databricks-to-Databricks sharing lets you share data with users in other Databricks accounts, whether they are on AWS, Azure, or GCP. It is also a great way to securely share data across different Unity Catalog metastores in your own Databricks account.</p><p><strong>Advantages of being on Databricks?</strong> <em>Delta sharing is indeed an open protocol.</em> <em>However, if you are contemplating Databricks for an organization with global operations or multiple divisions requiring seamless data sharing between them </em>— <em>There are 3 distinct advantages of using Databricks-to-Databricks Delta sharing over the open protocol:</em></p><ol><li>The application level security of the Delta sharing connection is <strong>managed entirely by the Databricks platform</strong>.</li><li>Apart from tables, it has the ability to share <strong>streaming tables</strong>, <strong>notebooks</strong>, <strong>views</strong>, <strong>volumes</strong>, and <strong>AI models</strong>.</li><li>Customers have the flexibility of <strong>enhancing security through network policies and restrictions</strong></li></ol><h3>How does Delta sharing work in Azure?</h3><p>Delta sharing protocol can work in both intra-cloud and cross-cloud setups. In the next sections, we will discuss intra-cloud <strong>Databricks-to-Databricks Delta sharing</strong> in <strong>Azure </strong>in more detail.</p><p>The Delta sharing mechanism can be categorized into 2 parts:</p><ol><li><strong>Data access without network restrictions —</strong> This part of the Databricks-to-Databricks delta sharing protocol is common for all setups, irrespective of the organizational network or security policies. In Figure 1, this part is depicted in sections <strong>[1], [2], and [5]</strong>.</li><li><strong>Data access with network restrictions —</strong> This part of the Databricks-to-Databricks delta sharing protocol has some variances that depend on the network and security policies of both the provider and the consumer. Network restrictions are optional for delta sharing, but are highly recommended. In Figure 1, this part is depicted in sections <strong>[3] and [4]</strong>.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/942/1*77Lf-YKaLRcyrC4nF10Edg.png" /><figcaption><strong>Figure 1: Databricks-to-Databricks Delta sharing in Azure</strong></figcaption></figure><h3>Part 1: Data access without network restriction</h3><p>This section is for the <strong>Data engineers </strong>and A<strong>dministrators</strong> wanting to understand the concepts of Delta sharing and how Databricks manages data access between the recipient and the provider. This part is common for all Databricks-to-Databricks delta sharing setups in Azure, irrespective of how the network is set up.</p><p><strong>Share recipient registration [1]: </strong>The share recipient registration is initiated by the provider’s metastore admin once the recipient’s account id, metastore ID, and cloud region are available. The provider’s metastore admin can then use this information to create a recipient in the provider system and add a share to the recipient.</p><p><strong>Metadata exchange [2]: </strong>Metadata information of the share is exchanged between the Unity catalog (control plane) and the compute plane for both the provider and the recipient</p><p><strong>Data exchange [5]: </strong>Once the registration is successful, the clusters in the receiver’s v-net can access the Parquet files in the provider’s storage account. The files, along with the metadata information passed through the Unity catalog, enable the recipient to read the data that was shared from the recipient’s Databricks system.</p><blockquote><strong>Note:</strong> The provider does not need to grant access to the recipient to its storage location. Access is entirely managed by Databricks under the hood by short-lived access tokens using end-to-end TLS encryption. The tokens are not visible to the provider or recipient, and the access is governed by Unity catalog.</blockquote><h4><strong>Useful links for </strong>Part 1</h4><p><a href="https://learn.microsoft.com/en-us/azure/databricks/delta-sharing/share-data-databricks"><em>Share data using the Delta Sharing Databricks-to-Databricks protocol (for providers)</em></a><em>,</em></p><p><a href="https://learn.microsoft.com/en-us/azure/databricks/delta-sharing/set-up"><em>Set up Delta Sharing for your account (for providers)</em></a></p><p><a href="https://learn.microsoft.com/en-us/azure/databricks/delta-sharing/grant-access"><em>Manage access to Delta Sharing data shares (for providers)</em></a></p><p><a href="https://learn.microsoft.com/en-us/azure/databricks/delta-sharing/create-recipient"><em>Create and manage data recipients for Delta Sharing</em></a></p><p><a href="https://learn.microsoft.com/en-us/azure/databricks/delta-sharing/recipient"><em>Access data shared with you using Delta Sharing (for recipients)</em></a></p><h3>Part 2: Data access with network restrictions</h3><p>Although Databricks internally manages the application layer security, it does not automatically secure the system at the network layer.</p><p>This section is intended for <strong>network engineers</strong> and <strong>security professionals</strong> who want to understand how Delta Sharing functions from a networking perspective. It also outlines steps to enhance security by regulating the flow of traffic in and out of Databricks and its underlying storage</p><h4>Option 1: Provider enables storage firewall</h4><p><strong>Step [3] </strong>of Figure 1 illustrates Option 1, where the <strong>provider </strong>enables a storage firewall that can whitelist specific IP addresses that are allowed to communicate with the storage container.</p><p>However, the public IP of the virtual machines that run the compute are dynamically assigned by Azure and can change over time, especially when the clusters are restarted, scaled, or new clusters are created. If the consumer enables <strong>secure cluster connectivity (no Public IP)</strong>, the VMs in the compute plane no longer have public IPs, and the clusters don’t have open ports. It ensures that clusters initiate outbound-only traffic through an HTTPS relay.</p><blockquote><strong>Note:</strong> All Azure Databricks network traffic between the classic compute plane VNet and the Azure Databricks control plane goes across the Microsoft network backbone, not the public internet, even when SCC is not enabled.</blockquote><blockquote><strong>Sounds confusing?</strong> I will explain this in more detail in another article, where I will do a deep dive into the Databricks networking.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zYGkGZ1cODZ8hjurh2jnjQ.png" /><figcaption><strong>Figure 2: Azure blob storage firewall with fixed egress IP in Databricks</strong></figcaption></figure><p>Let’s try to understand this option with a diagram. In <strong>Figure 2, </strong>the Provider enables the firewall in their storage account and allows access from fixed public IPs.</p><p>The <strong>provider</strong> has used a <strong>user-defined route</strong> to direct all outbound traffic to an<strong> Azure firewall</strong> behind an NVA. This firewall has a <strong>fixed public IP</strong>, which can be whitelisted in the storage firewall. SCC is also enabled, which ensures outbound-only traffic to the control plane. The provider also needs to allow the control plane traffic to the storage account. This is done by allowing traffic from the resource instance of the <strong>access connector for Databricks</strong> which acts as a managed identity for unity catalog to access external locations like azure storage.</p><blockquote><strong>Note:</strong> An easier setup for the provider would be to allow the traffic from the entire provider compute plane v-net into the storage container. This would not need setting up a azure firewall or a NAT gateway. However traffic from any resource within the provider v-net would be able to reach the storage.</blockquote><p>The <strong>consumer </strong>has configured a <strong>NAT gateway</strong> through which the traffic flows from the Databricks subnets. The NAT gateway is capable of producing a <strong>fixed egress IP</strong>, which can be whitelisted in the storage firewall. SCC is enabled, which means there is only outbound traffic to the control plane</p><p>Both the NAT gateway and Azure firewall can produce a fixed public IP address from the Databricks clusters, and step-by-step guides to implement both setups have been provided in the <strong>Useful Links</strong> section.</p><blockquote><strong>Note:</strong> With the default Databricks deployment (Azure managed v-net), Azure creates a NAT gateway in the Azure managed resource group when SCC is enabled, which can provide a stable public IP. This IP address can be whitelisted in the storage firewall (<strong>Figure 3</strong>).</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lqUf0a7iU4yHC-rDA-WcWQ.png" /><figcaption><strong>Figure 3: Azure managed NAT gateway with public IP address in managed resource group (default Databricks Deployment)</strong></figcaption></figure><p><strong>Configure firewall for serverless compute — </strong>Connectivity from serverless compute is managed with <strong>network connectivity configurations (NCCs)</strong>. Account admins can create NCCs in the account console, and an NCC can be attached to one or more workspaces. NCCs are account-level regional entities that are used to create and manage private endpoints and firewall rules. An NCC defines network identities for Azure resources (in the <strong>default rules</strong> section). When an NCC is attached to a workspace, serverless compute in that workspace uses one of the networks to connect to the Azure resource. The provider can “whitelist” these networks in the firewall of the storage account. Links for a step-by-step guide are provided in the <strong>Useful Links</strong> section.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Rm_coHNKw-aIUy5sxjafpw.png" /><figcaption><strong>Figure 4: Network Identities in serverless compute, which can be whitelisted in the storage firewall</strong></figcaption></figure><h4><strong>Useful Links for </strong>Option 1</h4><p><a href="https://learn.microsoft.com/en-us/azure/databricks/security/network/classic/vnet-inject"><em>Deploy Azure Databricks in your Azure virtual network (VNet injection)</em></a></p><p><a href="https://learn.microsoft.com/en-us/azure/storage/common/storage-network-security?tabs=azure-portal#grant-access-from-an-internet-ip-range"><em>Configure Azure Storage firewalls</em></a></p><p><a href="https://learn.microsoft.com/en-us/azure/databricks/security/network/classic/secure-cluster-connectivity"><em>Enable secure cluster connectivity</em></a></p><p><a href="https://learn.microsoft.com/en-us/azure/architecture/example-scenario/gateway/firewall-application-gateway#azure-firewall-only-design"><em>Route egress traffic to firewall</em></a></p><p><a href="https://learn.microsoft.com/en-us/azure/virtual-network/ip-services/configure-public-ip-nat-gateway"><em>Manage a public IP address with a NAT gateway</em></a></p><p><a href="https://learn.microsoft.com/en-us/azure/databricks/security/network/serverless-network-security/serverless-firewall"><em>Configure storage firewall for serverless compute access</em></a></p><h4><strong>Option 2: </strong>Provider enables storage private link</h4><p>In <strong>step[4]</strong> of Figure 1, the recipient would use Azure Private Link (cross or same tenant/subscription) to access the Provider’s storage account using private IPs within the consumer’s network. This means that the provider’s control plane also needs to access the storage location using private IPs within the provider’s network.</p><p><strong>Storage public access disabled </strong>— Once public access to the storage account is disabled, neither the provider nor the consumer can now reach the parquet files (which contain the actual data) over the internet using public IPs. How do we resolve this? By using <strong>private endpoints</strong></p><p><strong>Private endpoints </strong>— Private endpoints bring your Azure services into your v-net, allowing clients to access the service by using a private IP address from your v-net’s address space.</p><p><strong>Secure cluster connectivity (no Public IP) enabled — </strong>Enabling SCC ensures that the VMs in the compute plane do not have public IPs, and the clusters don’t have open ports. It ensures that the compute plane clusters initiate outbound-only traffic through an HTTPS relay, thus negating the need for inbound traffic routed to public IPs of the clusters.</p><p><strong>Private DNS Zones —</strong> Private DNS Zones are the entities that resolve the FDQN or Fully Qualified Domain Name of Azure services (like mystorageaccount.blob.core.windows.net) into their private IPs (like 10.xxx.22.0). Here, we have a Private DNS zone for each private endpoint — Storage-blob (for users to access blob storage), and storage-dfs (for Databricks to access storage). Each DNS zone contains a mapping of the FQDN and its private IP, thus telling the trusted networks which IP to resolve to for the services they are trying to communicate with.</p><blockquote><strong>Note:</strong> Private DNS Zones are created as part of the private endpoint creation unless a custom DNS server is used. In case of a custom DNS server, the DNS resolution needs to be additionally handled.</blockquote><blockquote>Go with Private DNS Zones if you are using Azure provided DNS service</blockquote><p><strong>Storage private endpoint in classic compute — </strong>Let’s try to understand this option with a diagram. <strong>Figure 5 </strong>shows a setup with Azure storage private endpoints with public access disabled<strong>.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zA5lAOB78aWh5tsTEAVS8A.png" /><figcaption><strong>Figure 5: Provider disabled storage public access and only allowed access over private link from classic compute</strong></figcaption></figure><p>The <strong>Provider </strong>has a storage container with <strong>private endpoints</strong> for dfs and blob services in a separate subnet with a<strong> Network Security Grou</strong>p (default rules + outbound internet traffic denied). The dfs service is required for Databricks to access the ADLS file system, and the blob service is used for the users to access the blob storage. Since <strong>Secure Cluster Connectivity (SCC)</strong> is enabled, there is only outbound traffic initiated by the clusters in the compute plane relayed through the SCC relay. Thus, it does not allow any unsolicited traffic to reach the clusters, other than the responses to the outbound traffic from the clusters.</p><blockquote><strong>Note:</strong> By enabling SCC, the Databricks cluster in the provider’s compute plane is secured. However, there are 2 drawbacks to this architecture:</blockquote><blockquote>(a) The control plane still uses public IP to communicate with the clusters,</blockquote><blockquote>(b) Users can access the Databricks workspace via the internet using a public URL</blockquote><blockquote>I will discuss how to address the drawbacks in another article, where I will do a deep dive into the network components of Azure Databricks</blockquote><p>The <strong>Consumer </strong>has a <strong>private endpoint</strong> (dfs service) for the same storage account placed in their compute plane v-net in a dedicated subnet with a similar <strong>Network Security Group</strong> (default rules + outbound internet traffic denied). The private endpoint is created in the consumer network using the resource ID of the provider’s storage account and is then approved by the provider. Since <strong>SCC is enabled</strong>, there is only outbound traffic initiated by the clusters in the compute plane relayed through the SCC relay.</p><p>So, by creating a set of private endpoints for the <strong>provider’s storage account</strong> in both the provider’s and the consumer’s v-net, we can ensure that both Databricks systems communicate privately with the storage account over a secure channel (Private link) using private IPs.</p><blockquote><strong>Note:</strong> For this option to work, both the consumer and the provider have to be on a Databricks workspace with v-net injection. With the default Databricks deployment (Azure managed v-net), the compute plane is entirely managed by Azure and does not integrate with private endpoints.</blockquote><p><strong>Configure Private endpoint for serverless compute — </strong>If either the provider or the consumer wants to use serverless compute for Delta sharing, then private endpoints have to be created in the respective Databricks account. Within the <strong>Network Connectivity Configuration(NCC)</strong>, private endpoints can be created for the storage account by providing the <strong>resource ID</strong> of the storage account and the <strong>sub-resource type </strong>(blob or dfs). Once created, a request is lodged from the Databricks account, which needs to be approved from the storage account ( networking → private endpoints). Links for a step-by-step guide are provided in the <strong>Useful Links</strong> section.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LnYFMoAVSsDoyCrjTsL-9A.png" /><figcaption><strong>Figure 5: Private endpoint in serverless compute for Azure storage account</strong></figcaption></figure><blockquote><strong>Note: Why not network peering? —</strong> It is a valid question, and yes, peering the provider and consumer v-nets will work (provided DNS mapping is resolved). However, peering allows bi-directional flow of traffic to the entire v-net as a whole instead of the specific services we require to communicate with (in this case Azure Storage). Yes, we can restrict access with network security groups and firewalls to some extent, but remember that not all resources within the v-net can be restricted with strict NSGs or firewalls.</blockquote><h4><strong>Useful links for </strong>Option 2</h4><p><a href="https://learn.microsoft.com/en-us/azure/databricks/security/network/classic/secure-cluster-connectivity"><em>Enable secure cluster connectivity</em></a></p><p><a href="https://learn.microsoft.com/en-us/azure/storage/common/storage-private-endpoints"><em>Use private endpoints for Azure Storage</em></a></p><p><a href="https://learn.microsoft.com/en-us/azure/private-link/private-endpoint-overview#access-to-a-private-link-resource-using-approval-workflow"><em>Access to a private-link resource using approval workflow</em></a></p><p><a href="https://www.eannaoceallaigh.com/blog/azure-cross-tenant-storage/"><em>Accessing Azure storage accounts in another tenant using private endpoints</em></a></p><p><a href="https://learn.microsoft.com/en-us/azure/virtual-network/network-security-groups-overview"><em>Azure Network security groups</em></a></p><p><a href="https://learn.microsoft.com/en-us/azure/virtual-network/manage-network-security-group?tabs=network-security-group-portal"><em>Create, change, or delete a network security group</em></a></p><p><a href="https://learn.microsoft.com/en-us/azure/databricks/security/network/serverless-network-security/serverless-private-link"><em>Configure private connectivity to Azure resources from serverless compute</em></a></p><p><a href="https://azure.microsoft.com/en-us/pricing/details/private-link/"><em>Azure Private Link pricing</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3f101156ed1d" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>