<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Scott Grzybowski on Medium]]></title>
        <description><![CDATA[Stories by Scott Grzybowski on Medium]]></description>
        <link>https://medium.com/@scottgriz_66302?source=rss-acb3a812b683------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*0OrbIWW3MEV8QXUb</url>
            <title>Stories by Scott Grzybowski on Medium</title>
            <link>https://medium.com/@scottgriz_66302?source=rss-acb3a812b683------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 03:00:07 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@scottgriz_66302/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[One Subnet to Reach Them All: Azure Databricks Private Network Gateway]]></title>
            <link>https://medium.com/databricks-platform-sme/one-subnet-to-reach-them-all-azure-databricks-private-network-gateway-4486304a9f99?source=rss-acb3a812b683------2</link>
            <guid isPermaLink="false">https://medium.com/p/4486304a9f99</guid>
            <category><![CDATA[networking]]></category>
            <category><![CDATA[azure-databricks]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Scott Grzybowski]]></dc:creator>
            <pubDate>Fri, 30 Oct 2026 20:34:20 GMT</pubDate>
            <atom:updated>2026-10-30T23:05:47.862Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QQHvGn6BZ0AkyUMBRDa0fQ.png" /></figure><h3>One Subnet to Reach Them All: Azure Databricks Private Network Gateway</h3><p><em>How Private Network Gateway (Public Preview) replaces per-resource load balancer plumbing with a single gateway into your VNet — and everything your network can reach from it.</em></p><p>Serverless compute is the fastest way to run workloads on Azure Databricks: no clusters to size, no VNet injection to plan, instant start-up. But serverless compute runs in the Databricks-managed network, not yours. The moment a notebook, job, or agent needs to talk to a PostgreSQL server on a VM, a Git server in the data center, or an Artifactory instance behind ExpressRoute, the question becomes: <em>how do I get from there to here, privately?</em></p><p>Until now, the answer was one private connection per resource. <strong>Private Network Gateway (PNG)</strong> changes that. You delegate one subnet in your VNet to Azure Databricks, create a gateway once, and serverless compute can reach whatever that subnet can reach — your VNet, peered spokes, and on-premises networks over ExpressRoute or VPN — following your own routes, DNS, and firewall rules.</p><p>This post covers how PNG works, when to use it, what I learned building it end to end, and (in the appendix) the complete code to recreate my lab environment.</p><h3>How We Got Here: The Load Balancer Era</h3><p>My colleagues <strong>Kiran Anand</strong> and <strong>Suraj Karuvel</strong> wrote the definitive series on reaching customer resources from Azure Databricks serverless before PNG existed. If you have an existing deployment, these patterns are still valid and worth reading:</p><ul><li><a href="https://medium.com/databricks-platform-sme/azure-databricks-serverless-private-connectivity-to-customers-vnet-through-load-balancer-d8e3ae3bab2b"><strong>Part 1: Serverless private connectivity through a Load Balancer</strong></a> — the “first mile”: an NCC private endpoint rule to a Private Link Service in front of an internal Standard Load Balancer.</li><li><a href="https://medium.com/databricks-platform-sme/azure-databricks-serverless-private-connectivity-to-customers-resources-part-2-b1c43ed8fcd4"><strong>Part 2: Reaching on-premises and other networks</strong></a> — the “last mile”: a forwarder (HAProxy, Nginx stream, or Linux IP forwarding) on the load balancer’s backend VM to relay traffic over ExpressRoute or VPN, because an Azure Standard Load Balancer backend pool can’t hold on-premises IPs.</li><li><a href="https://medium.com/@anand.kiran/azure-databricks-serverless-private-connectivity-to-customer-resources-through-azure-application-32ea023b2b26"><strong>Part 3: Private connectivity through Azure Application Gateway</strong></a> — Application Gateway v2 with Private Link as a managed Layer 4/Layer 7 alternative, with FQDN and IP backends.</li></ul><p>These designs work, and they’re the right answer for some cases today. But each one is per-resource plumbing — and that’s where the scaling problem starts.</p><h3>The Problem at Scale</h3><p>Private Link is excellent for cloud-managed services and a handful of fixed endpoints. Every target, though, needs its own NCC private endpoint and approval. Anything that isn’t a PaaS service — a database on a VM, an MCP server, an on-premises system — also needs a Private Link Service and a load balancer, and on-premises targets add a forwarder VM on top. Multiply that by tens or hundreds of resources and the setup, approvals, port juggling, and ongoing operations add up fast.</p><h3>Enter Private Network Gateway</h3><p>A private network gateway connects Azure Databricks serverless compute to resources in your VNet through <strong>a single managed gateway</strong>. When you create one, Azure Databricks injects the gateway into a subnet you’ve delegated to Microsoft.Databricks/workspaces. Serverless traffic for the destinations you configure flows through an encrypted tunnel into that subnet, then onward to anything your VNet can route to — including on-premises systems over ExpressRoute or VPN.</p><p>Three things make this different from the load balancer era:</p><ul><li><strong>Broad connectivity without per-resource onboarding.</strong> One gateway, many targets. No Private Link Service, load balancer, or forwarder VM per resource.</li><li><strong>Your network, your rules.</strong> Traffic lands in your VNet and follows your route tables, DNS, NSGs, and firewalls. Your network team keeps control, and your existing inspection and logging see the traffic.</li><li><strong>Stable, identifiable source IPs.</strong> Traffic arrives from the gateway subnet’s addresses, so downstream systems can allowlist it at the network layer — and with a NAT Gateway or firewall, SaaS providers can allowlist your own public IP.</li></ul><p>PNG <strong>complements</strong> Private Link and network security perimeters rather than replacing them. For a direct private connection to a cloud-managed resource such as storage or Azure SQL, use an NCC private endpoint rule. For same-region storage accounts that serverless reaches over Azure service endpoints, associate the account with an Azure network security perimeter (NSP) and allow the AzureDatabricksServerless service tag — this replaces the old practice of allowlisting Databricks serverless subnet IDs. Use the gateway for everything in your VNet and beyond.</p><h3>How It Works</h3><p><strong>It reuses your existing NCC.</strong> There’s no new object model. You create the gateway inside a network connectivity configuration (NCC) and attach that NCC to your workspaces; serverless products in those workspaces use the gateway automatically.</p><p><strong>Two traffic modes:</strong></p><ul><li><strong>SPECIFIC_DESTINATIONS</strong> — only the DNS names you list go through the gateway; everything else keeps its existing path. Suffix matching is supported, so contoso.internal also matches db.contoso.internal. This is the recommended mode for most cases.</li><li><strong>ALL_TRAFFIC</strong> — all remaining serverless egress goes through the gateway, so you can send it through your own firewall (Azure Firewall, Palo Alto, or another appliance) or a dedicated egress path.</li></ul><p><strong>Routing priority.</strong> Azure Databricks evaluates each outbound connection against your network paths in order and uses the first match:</p><ol><li><strong>NCC private endpoint rules (Private Link)</strong> — always win for their resource, even in ALL_TRAFFIC mode.</li><li><strong>Azure service endpoints</strong> — stay on the Azure backbone. Blob and ADLS storage always take this path, and the gateway can’t override it.</li><li><strong>Private network gateway</strong> — your listed destinations, or all remaining egress in ALL_TRAFFIC mode.</li><li><strong>Default serverless egress</strong> — everything else.</li></ol><p>One exception: Azure SQL (*.database.windows.net) normally uses its service endpoint. If you add your server’s FQDN as a gateway destination, the gateway carries that traffic instead. That’s useful when the FQDN points to SQL Server you host yourself behind your network.</p><p><strong>Serverless egress control.</strong> In SPECIFIC_DESTINATIONS mode, gateway destinations are automatically allowed by your serverless egress policy, just like Private Link domains. In ALL_TRAFFIC mode, you must add each destination to your network policy explicitly.</p><h3>What You Can Do With It</h3><ul><li><strong>Data sources inside your network</strong> — SQL Server, PostgreSQL, MySQL, or Oracle on VMs in your VNet or peered spokes, plus your own private endpoints (e.g., SQL Managed Instance).</li><li><strong>On-premises connectivity</strong> — reach data center databases, Git servers, and artifact repositories over your existing ExpressRoute or VPN, with no forwarder VMs.</li><li><strong>SaaS with dedicated egress (bring your own NAT)</strong> — reach SaaS such as Salesforce from your own static public IP via a NAT Gateway.</li><li><strong>Custom egress and security</strong> — inspect, filter, and log all serverless egress at Azure Firewall or a third-party network virtual appliance.</li><li><strong>External model providers</strong> — reach model APIs behind your own API gateway (such as internal-mode APIM or Kong).</li><li><strong>External MCP servers</strong> — let agents call private MCP tools hosted on App Service, AKS, or VMs through Unity Catalog connections.</li></ul><h3>Hands-On: Building It End to End</h3><p>To test PNG I built a small but realistic environment in westus2 (full code in the appendix):</p><ul><li>A VNet, <strong>scott-griz-png</strong> (172.25.0.0/23), with a dedicated <strong>private-network-gateway</strong> subnet (172.25.0.0/25) delegated to Microsoft.Databricks/workspaces.</li><li>A private Ubuntu VM running <strong>PostgreSQL</strong> at 172.25.0.132 with no public IP.</li><li>A <strong>private DNS zone</strong>, scott-griz-png.internal.com, with an A record for the database, linked to the gateway VNet.</li><li>A second, peered VNet with a jump host for administration.</li><li>A Databricks notebook, run on <strong>serverless compute</strong>, that tests DNS, TCP, and a PostgreSQL connection through the gateway.</li></ul><h3>Step 1: Choose the NCC</h3><p>In the account console, go to <strong>Security → Networking → Network connectivity configurations</strong> and select an NCC in the <strong>same region</strong> as your subnet (or create one).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mQAgivajB_jz5vIly3E6hg.png" /></figure><h3>Step 2: Open the Private network gateway tab</h3><p>On the NCC page, open the <strong>Private network gateway</strong> tab and click <strong>Add private network gateway</strong>. Once created, the gateway appears here with its subnet, traffic mode, and state.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2C8VEtaFGQz0l431qNBemw.png" /></figure><h3>Step 3: Confirm the prerequisites</h3><p>The wizard reminds you of the three Azure-side prerequisites: a dedicated /28-or-larger subnet in the NCC’s region, delegation to Microsoft.Databricks/workspaces, and a DNS choice (a custom resolver IP, or blank for Azure-provided DNS).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/1*YlYui9wpuX0U9pUfik2M9g.png" /></figure><h3>Step 4: Create the gateway</h3><p>Provide a name, the full subnet resource ID, an optional DNS resolver, a bandwidth tier (10, 30, 50, or 100 Gbps; the console defaults to 30), the traffic mode, and your destinations. I used SPECIFIC_DESTINATIONS with the database’s FQDN. Then attach the NCC to your workspace and wait for the gateway state to show <strong>Established</strong>.</p><p>You can also create the gateway with the Networking v1 REST API or Terraform.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*kWz56EYdVDZfPJcrPS1Xiw.png" /></figure><h3>Step 5: Test from a serverless notebook</h3><p>Create a notebook in the workspace the NCC is attached to and connect it to <strong>serverless compute</strong> (choose <strong>Serverless</strong> in the notebook’s compute selector). This matters: the gateway only carries traffic from serverless compute — a classic cluster won’t use it, and the test will fail. To skip the copy-paste, import <a href="https://github.com/ScottGriz/databricks-tools/blob/main/notebooks/Azure%20PNG%20Testing.ipynb">the test notebook from GitHub</a> — it has the same cells as below.</p><p><strong>Cell 1 — check DNS and the network path:</strong></p><pre>%sh<br># Replace the hostname (and 5432 if needed) with your PNG destination FQDN.<br>nslookup scott-griz-postgresql.scott-griz-png.internal.com<br>nc -vz -w 4 scott-griz-postgresql.scott-griz-png.internal.com 5432</pre><p>nslookup should return a NAT64 address such as 64:ff9b:1::100:ac19:84 — the IPv6 address the gateway maps your VM’s private IPv4 address to. That single answer proves the gateway, the destination entry, and the private DNS zone link are all working (the “recursion not available” lines are harmless resolver warnings).</p><p>Don’t be alarmed if nc fails. nslookup queries the DNS server directly and accepts the AAAA answer, but nc and Python’s default getaddrinfo ask for both A and AAAA records. The gateway’s resolver only gives a useful answer for AAAA, so the A lookup fails with “Resource temporarily unavailable.”</p><p>Cell 2 — raw TCP check with getaddrinfo:</p><pre># Test connection: raw TCP check through the PNG<br>import socket<br># Replace HOST with the FQDN you listed in your PNG destinations and PORT with the port number if not 5432<br>HOST = &quot;scott-griz-postgresql.scott-griz-png.internal.com&quot; # FQDN listed in the PNG destinations<br>PORT = 5432<br><br># PNG only routes IPv6, so resolve with getaddrinfo (not gethostbyname)<br>infos = socket.getaddrinfo(HOST, PORT, proto=socket.IPPROTO_TCP)<br>for fam, _, _, _, addr in infos:<br>    print(fam.name, addr[0])<br>fam, stype, proto, _, addr = infos[0]<br>s = socket.socket(fam, stype, proto)<br>s.settimeout(10)<br>try:<br>    s.connect(addr)<br>    print(&quot;TCP connect OK&quot;)<br>except Exception as e:<br>    print(&quot;TCP connect failed:&quot;, e)<br>finally:<br>    s.close()</pre><p>If this cell raises socket.gaierror (Python’s address-lookup error), it’s the same A-record problem as nc — which is why the next cell asks for IPv6 explicitly.</p><p><strong>Cell 3 — install psycopg2 and connect to PostgreSQL:</strong></p><p>Store the database user and password as secrets first. I used an Azure Key Vault-backed secret scope named key-vault-secret with the keys pvt-pg-user and pvt-pg-password; a Databricks-backed scope works too.</p><pre>%pip install psycopg2-binary</pre><pre>dbutils.library.restartPython()</pre><p># PostgreSQL connection</p><pre>import socket<br>import psycopg2</pre><pre># Replace HOST with the FQDN you listed in your PNG destinations and PORT with the port number if not 5432<br>HOST = &quot;scott-griz-postgresql.scott-griz-png.internal.com&quot;<br>PORT = 5432</pre><pre>addr = socket.getaddrinfo(HOST, PORT, socket.AF_INET6, socket.SOCK_STREAM)[0][4][0]</pre><pre>conn = psycopg2.connect(<br>    host=HOST, hostaddr=addr, port=PORT, dbname=&quot;my_test_db&quot;,<br>    user=dbutils.secrets.get(&quot;key-vault-secret&quot;, &quot;pvt-pg-user&quot;),<br>    password=dbutils.secrets.get(&quot;key-vault-secret&quot;, &quot;pvt-pg-password&quot;),<br>    sslmode=&quot;prefer&quot;, connect_timeout=10,<br>)<br>with conn.cursor() as cur:<br>    cur.execute(&quot;SELECT version(), inet_client_addr()&quot;)<br>    print(cur.fetchone())<br>conn.close()</pre><p>The output should look like this:</p><pre>(‘PostgreSQL 16.15 (Ubuntu 16.15–0ubuntu0.24.04.1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 13.3.0–6ubuntu2~24.04.1) 13.3.0, 64-bit’, ‘172.25.0.4’)</pre><p>Look at the second value: inet_client_addr() reports <strong>172.25.0.4</strong> — an address inside the gateway subnet (172.25.0.0/25), not a Databricks IP. That’s the gateway’s source NAT in action, and it’s exactly why your firewall and pg_hba.conf rules must allow the gateway subnet CIDR.</p><h3>Field-Tested Gotchas</h3><p>These are the lessons that cost me (and others in the field) the most time:</p><ul><li><strong>Test on serverless compute.</strong> The gateway only applies to serverless workloads. A notebook attached to a classic cluster will never use it, no matter how well the network is configured.</li><li><strong>Allow the gateway subnet CIDR, not Databricks IPs.</strong> The gateway source-NATs traffic out of its NICs in the delegated subnet, and there are several gateway nodes. Allow the whole subnet CIDR in NSGs, host firewalls, and database rules such as pg_hba.conf.</li><li><strong>DNS is make-or-break.</strong> The gateway resolves destinations with your resolver. Link private DNS zones to the VNet that holds the gateway subnet (and to any VNet that also needs the name), or point the gateway at a custom resolver it can reach. If you configure two custom resolvers (the maximum), make sure both return the same records and both are reachable from the gateway subnet on port 53. The gateway queries them round-robin, not primary-then-secondary, unlike VNet DNS settings, so a record that exists on only one resolver resolves only some of the time.</li><li><strong>IPv6 matters on the client side.</strong> Serverless-to-gateway traffic is IPv6/NAT64. Tools like nslookup and IPv4-only lookups give false negatives; in my testing, I needed an explicit AF_INET6 lookup in Python, as shown above. Test with a real client, not ping or nc.</li><li><strong>Destinations are FQDNs only.</strong> IP addresses aren’t accepted, and there’s no *. wildcard syntax; a parent domain matches all of its subdomains instead.</li><li><strong>Peering goes both ways.</strong> For targets in peered VNets, create the peering on both sides with forwarded traffic allowed. A successful ping doesn’t prove TCP works.</li><li><strong>Pick one path per FQDN.</strong> An NCC private endpoint rule and the gateway don’t fail over to each other for the same name.</li><li><strong>Give changes time.</strong> Allow about 10 minutes for changes to propagate, and restart SQL warehouses to drop cached connections.</li><li><strong>Storage never goes through the gateway.</strong> ADLS Gen2 and Blob stay on their Private Link or service endpoint path.</li><li><strong>Lifecycle is asynchronous.</strong> Deleting a gateway can take 30 minutes to 2 hours, so wait before removing the subnet. A FAILED gateway is terminal: fix the cause, delete it, and recreate it.</li></ul><h3>Prerequisites and Limits</h3><ul><li><strong>Premium plan</strong>, and an <strong>account admin</strong> to enable the preview from the account console <strong>Previews</strong> page.</li><li>The gateway, its subnet, the NCC, and the workspaces must be in the <strong>same region</strong>.</li><li>A <strong>dedicated subnet of /28 or larger</strong>, delegated to Microsoft.Databricks/workspaces, with downstream connectivity to every target.</li><li>Up to <strong>2 gateways per NCC</strong>, and up to <strong>2 DNS resolvers</strong> and <strong>100 destinations</strong> per gateway.</li><li>Works with <strong>serverless Databricks Runtime products and AI services</strong> — not classic compute.</li><li><strong>Billing</strong> begins on or after December 1, 2026. Gateways are billed by capacity tier while they exist, plus data processed — check the Azure Databricks pricing page for current rates.</li></ul><h3>When to Keep Using Private Link</h3><p>PNG isn’t a replacement for every private connection:</p><ul><li><strong>Cloud-managed PaaS</strong> (storage, Azure SQL, Azure Database for PostgreSQL, Azure OpenAI) should keep its NCC private endpoint rule — it’s the recommended path, and it takes priority anyway.</li><li><strong>Just one or two private resources?</strong> A private endpoint rule (with a load balancer, as in Kiran and Suraj’s series) may cost less than a gateway, which is billed 24/7 while it’s provisioned.</li><li><strong>Existing load balancer and Application Gateway designs</strong> keep working; you can migrate destinations to the gateway incrementally.</li></ul><h3>Wrapping Up</h3><p>Private Network Gateway turns serverless private connectivity from a per-resource project into a one-time network design decision. Delegate a subnet, create a gateway, list your destinations — and serverless compute reaches your VNet, your peered networks, and your data center through the controls your network team already trusts.</p><p>Start with SPECIFIC_DESTINATIONS, get DNS right first, allow the gateway subnet CIDR on your targets, and test with real clients. Then retire some load balancers.</p><p><strong>Resources:</strong></p><ul><li><a href="https://learn.microsoft.com/en-us/azure/databricks/security/network/serverless-network-security/private-network-gateway">Private network gateway (Microsoft Learn)</a></li><li><a href="https://learn.microsoft.com/en-us/azure/databricks/security/network/serverless-network-security/private-network-gateway/configure-private-network-gateway">Configure a private network gateway (Microsoft Learn)</a></li><li><a href="https://learn.microsoft.com/en-us/azure/databricks/security/network/serverless-network-security/serverless-firewall-config">Configure a firewall for serverless compute access (Microsoft Learn)</a></li><li><a href="https://github.com/ScottGriz/databricks-tools/blob/main/notebooks/Azure%20PNG%20Testing.ipynb">Azure PNG test notebook (GitHub)</a></li></ul><h3>Appendix: Recreate the Lab Environment</h3><p>The following az CLI commands build the environment used in this post. Replace the placeholders (&lt;subscription-id&gt;, &lt;your-email&gt;, &lt;admin-ip-range&gt;, &lt;YOURPASSWORD&gt;) with your own values. Create the resources in this order: VNet, private DNS zone, VM, PostgreSQL, then the gateway.</p><h3>A.1 VNet</h3><p>The VNet has four subnets: the delegated gateway subnet, a subnet for VMs, a utility subnet, and a subnet for private endpoints. Default outbound access is disabled everywhere, so a NAT Gateway provides internet egress for the VM subnets. The VNet is peered with a second VNet that hosts a jump host.</p><pre>az account set --subscription &lt;subscription-id&gt;<br>RG=scott-griz-utility<br>LOC=westus2<br>VNET=scott-griz-png</pre><pre># 1. NAT gateway (public IP first)<br>az network public-ip create -n nat-pip -g $RG -l $LOC \<br> --sku StandardV2 --allocation-method Static</pre><pre>az network nat gateway create -n scott-griz-png-nat -g $RG -l $LOC \<br>--public-ip-addresses nat-pip — idle-timeout 4 \<br>--tags Owner=&lt;your-email&gt;</pre><pre># 2. VNet with the first (delegated) subnet<br>az network vnet create -n $VNET -g $RG -l $LOC \<br>--address-prefixes 172.25.0.0/23 \<br>--subnet-name private-network-gateway --subnet-prefixes 172.25.0.0/25 \<br>--tags Owner=&lt;your-email&gt;</pre><pre>az network vnet subnet update -g $RG --vnet-name $VNET -n private-network-gateway \<br>--delegations Microsoft.Databricks/workspaces \<br>--disable-private-endpoint-network-policies \<br>--default-outbound-access false</pre><pre># 3. Remaining subnets<br>az network vnet subnet create -g $RG --vnet-name $VNET -n machines \<br>--address-prefixes 172.25.0.128/25 \<br>--nat-gateway scott-griz-png-nat \<br>--disable-private-endpoint-network-policies \<br>--default-outbound-access false</pre><pre>az network vnet subnet create -g $RG --vnet-name $VNET -n utility \<br>--address-prefixes 172.25.1.0/25 \<br>--nat-gateway scott-griz-png-nat \<br>--disable-private-endpoint-network-policies \<br>--default-outbound-access false</pre><pre>az network vnet subnet create -g $RG --vnet-name $VNET -n private-link \<br>--address-prefixes 172.25.1.128/25 \<br>--disable-private-endpoint-network-policies \<br>--default-outbound-access false</pre><pre># 4. Peering to the jump host VNet (create on BOTH VNets)<br>UTILITY_VNET_ID=$(az network vnet show -n scott-griz-utility-vnet -g $RG --query id -o tsv)<br>az network vnet peering create -n scott-griz-png-utility-peer -g $RG<br>--vnet-name $VNET \<br>--remote-vnet $UTILITY_VNET_ID --allow-forwarded-traffic</pre><pre>PNG_VNET_ID=$(az network vnet show -n $VNET -g $RG--query id -o tsv)<br>az network vnet peering create -n scott-griz-png-utility-link -g $RG \<br>--vnet-name scott-griz-utility-vnet \<br>--remote-vnet $PNG_VNET_ID --allow-forwarded-traffic</pre><p>Don’t place anything else in the delegated private-network-gateway subnet — Azure Databricks deploys the gateway’s resources there.</p><h3>A.2 Private DNS Zone</h3><p>A custom private zone with one A record for the database, linked to both VNets so the gateway subnet and the jump host can both resolve it.</p><pre>RG=scott-griz-utility<br>ZONE=scott-griz-png.internal.com</pre><pre>az network private-dns zone create -n $ZONE -g $RG</pre><pre>az network private-dns record-set a add-record -g $RG -z $ZONE \<br>-n scott-griz-postgresql -a 172.25.0.132</pre><pre>az network private-dns link vnet create -g $RG -z $ZONE \<br>-n scott-griz-png-dns-link -v scott-griz-png -e false</pre><pre>az network private-dns link vnet create -g $RG -z $ZONE \<br>-n scott-griz-png-utility-link -v scott-griz-utility-vnet -e false</pre><h3>A.3 PostgreSQL VM</h3><p>A private Ubuntu 24.04 VM with no public IP and a static private IP that matches the DNS record. Its NSG allows SSH only from your admin range and the jump host.</p><p>A fresh subscription must accept the Ubuntu Pro marketplace terms first:</p><pre>az vm image terms accept --urn Canonical:ubuntu-24_04-lts:ubuntu-pro:24.04.202610160</pre><pre>RG=scott-griz-utility<br>LOC=westus2<br>VMNAME=scott-griz-ubuntu-pg-pvt<br>VNET=scott-griz-png<br>SUBNET=machines</pre><pre># 1. NSG<br>az network nsg create -n ${VMNAME}--nsg -g $RG -l $LOC</pre><pre>az network nsg rule create -g $RG --nsg-name ${VMNAME}--nsg -n admin-ssh \<br>--priority 100 --direction Inbound --access Allow --protocol Tcp \<br>--source-address-prefixes &lt;admin-ip-range&gt; \<br>--destination-port-ranges 22</pre><pre>az network nsg rule create -g $RG — nsg-name ${VMNAME}--nsg -n jump-host-ssh \<br>--priority 110 --direction Inbound --access Allow --protocol Tcp \<br>--source-address-prefixes 172.28.197.4/32 \<br>--destination-port-ranges 22</pre><p># 2. VM (builds the NIC in the existing subnet with a static private IP, no public IP)</p><pre>az vm create \<br>--resource-group $RG --name $VMNAME --location $LOC --zone 1 \<br>--size Standard_D2ds_v5 \<br>--image Canonical:ubuntu-24_04-lts:ubuntu-pro:24.04.202610160 \<br>--plan-publisher Canonical --plan-name ubuntu-pro --plan-product ubuntu-24_04-lts \<br>--admin-username azureuser \<br>--ssh-key-values ~/.ssh/id_rsa.pub \<br>--vnet-name $VNET --subnet $SUBNET \<br>--nsg ${VMNAME}--nsg \<br>--private-ip-address 172.25.0.132 \<br>--public-ip-address “” \<br>--os-disk-size-gb 30 --storage-sku StandardSSD_LRS --os-disk-delete-option Delete \<br>--security-type TrustedLaunch --enable-secure-boot true --enable-vtpm true \<br>--assign-identity \<br>--tags Owner=&lt;your-email&gt;</pre><p># 3. Managed boot diagnostics</p><pre>az vm boot-diagnostics enable -g $RG -n $VMNAME</pre><h3>A.4 PostgreSQL Setup</h3><p>Run these on the VM.</p><pre>sudo apt update<br>sudo apt install postgresql postgresql-contrib<br>sudo systemctl enable postgresql<br>sudo su — postgres<br>psql</pre><pre>CREATE USER my_test_user WITH PASSWORD &#39;&lt;YOURPASSWORD&gt;&#39; CREATEDB;<br>CREATE DATABASE my_test_db OWNER my_test_user TEMPLATE template0 ENCODING &#39;UTF8&#39;;<br>GRANT ALL PRIVILEGES ON DATABASE my_test_db TO my_test_user;</pre><p>Connect as the new user (psql -h localhost -U my_test_user -d my_test_db) and create sample data:</p><pre>CREATE TABLE pet (name VARCHAR(20), owner VARCHAR(20), species VARCHAR(20), sex CHAR(1), birth DATE, death DATE);<br>INSERT INTO pet VALUES (&#39;Puffball&#39;,&#39;Diane&#39;,&#39;hamster&#39;,&#39;f&#39;,&#39;1999–03–30&#39;,NULL);<br>INSERT INTO pet VALUES (&#39;Tigger&#39;,&#39;Tommy&#39;,&#39;hamster&#39;,&#39;m&#39;,&#39;2001–03–30&#39;,NULL);<br>INSERT INTO pet VALUES (&#39;Zack&#39;,&#39;Scott&#39;,&#39;dog&#39;,&#39;m&#39;,&#39;2001–03–30&#39;,NULL);<br>INSERT INTO pet VALUES (&#39;Tera&#39;,&#39;Sharron&#39;,&#39;cat&#39;,&#39;f&#39;,&#39;1991–03–30&#39;,NULL);<br>INSERT INTO pet VALUES (&#39;Ichi&#39;,&#39;Sharron&#39;,&#39;cat&#39;,&#39;f,&#39;2003–03–30&#39;,NULL);</pre><p>Allow the <strong>gateway subnet CIDR</strong> (not Databricks IPs) and open the listener:</p><pre># /etc/postgresql/16/main/pg_hba.conf, under &quot;# IPv4 local connections&quot;:<br>host    all    all    172.25.0.0/25    scram-sha-256</pre><pre># /etc/postgresql/16/main/postgresql.conf:<br>listen_addresses = &#39;*&#39;</pre><pre>sudo systemctl restart postgresql</pre><h3>A.5 Gateway-Specific Requirements Checklist</h3><ul><li>Allow TCP 5432 from the gateway subnet CIDR (172.25.0.0/25) in pg_hba.conf, the NSG, and ufw if enabled. Same-VNet traffic is allowed by the default AllowVnetInBound NSG rule.</li><li>Add the database FQDN (scott-griz-postgresql.scott-griz-png.internal.com) as a gateway destination — not its IP.</li><li>Make sure the FQDN resolves to the private IP from the gateway subnet (private DNS zone linked to the gateway VNet).</li><li>In SPECIFIC_DESTINATIONS mode, the destination is automatically allowed by serverless egress control; in ALL_TRAFFIC mode, add it to your network policy.</li><li>Attach the NCC to the workspace and wait for the gateway state to show Established before testing.</li><li>Run the test notebook from Step 5 on <strong>serverless compute</strong> — classic clusters don’t use the gateway. A successful run returns the gateway subnet address (172.25.0.4 in my lab) from inet_client_addr().</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4486304a9f99" width="1" height="1" alt=""><hr><p><a href="https://medium.com/databricks-platform-sme/one-subnet-to-reach-them-all-azure-databricks-private-network-gateway-4486304a9f99">One Subnet to Reach Them All: Azure Databricks Private Network Gateway</a> was originally published in <a href="https://medium.com/databricks-platform-sme">Databricks Platform SME</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploying Azure Databricks with RFC-6598 Shared Address Space: Zero Routable IP Consumption]]></title>
            <link>https://medium.com/databricks-platform-sme/deploying-azure-databricks-with-rfc-6598-shared-address-space-zero-routable-ip-consumption-f25f17d48e2f?source=rss-acb3a812b683------2</link>
            <guid isPermaLink="false">https://medium.com/p/f25f17d48e2f</guid>
            <category><![CDATA[azure-databricks]]></category>
            <category><![CDATA[azure-networking]]></category>
            <category><![CDATA[networking]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Scott Grzybowski]]></dc:creator>
            <pubDate>Wed, 08 Jul 2026 02:05:42 GMT</pubDate>
            <atom:updated>2026-07-08T02:05:42.808Z</atom:updated>
            <content:encoded><![CDATA[<p><em>How to run large VNet-injected Databricks workspaces without spending a single routable IPv4 address — using an RFC-6598 “island” network, Azure Firewall forced tunneling, and local SNAT.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E7jlYvXu6fOFIfXz7Jnqyw.png" /><figcaption>Zero Routable IPs</figcaption></figure><p>If you’ve ever tried to stand up a large Azure Databricks deployment inside a mature enterprise network, you’ve probably hit the same wall I have: there’s no routable IPv4 space left to give it.</p><p>VNet-injected Databricks isn’t shy about addresses. Every workspace needs a minimum of two delegated subnets — a container/host subnet and a private subnet — and high-concurrency clusters scale scale fast. In an organization that has already carved up most of its RFC-1918 space (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16) across on-prem and cloud, finding contiguous, routable blocks for a new Databricks environment becomes a genuine blocker.</p><p>This post walks through a pattern that sidesteps the problem entirely. We deploy Databricks into a (100.64.0.0/16) <strong>“island” network which is part of the RFC-6598 network range of </strong>(100.64.0.0/10), then use Azure Firewall with <strong>forced tunneling</strong> and <strong>local SNAT</strong> to translate all egress behind a small block of routable RFC-1918 addresses.</p><p>The payoff: your Databricks workspaces consume <strong>effectively zero</strong> routable IPv4 space, while retaining controlled, inspected access to the internet, on-premises networks, and Azure PaaS dependencies.</p><h3>Why IPv4 conservation matters in Azure</h3><p>Corporate networks typically draw from the private IPv4 ranges defined by <strong>RFC 1918</strong>. On-premises, those ranges are usually more than enough. In the cloud, though, large networks are easy to build, and patterns like microservices, containerization, and elastic compute (Databricks clusters very much included) drive IPv4 consumption up quickly. Microsoft’s own guidance is explicit: treat IPv4 addresses as a <strong>limited resource</strong> and adopt conservative address-management practices.</p><p>Microsoft’s Azure Architecture Center documents two primary methods to minimize routable IPv4 consumption in large Azure networks:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DBx3mlOaFXfcoXEEFyzDzQ.png" /><figcaption>Two methods to minimize routeable IPv4 consumption</figcaption></figure><p>The Databricks RFC-6598 pattern in this post is a specialized application of <strong>Method 1</strong>: Databricks runs in an isolated, non-routable RFC-6598 VNet, and Azure Firewall provides the SNAT boundary back to routable space.</p><h3>Why RFC-6598 (100.64.0.0/10) specifically</h3><p>RFC 6598 — <em>“IANA-Reserved IPv4 Prefix for Shared Address Space”</em> — reserves 100.64.0.0/10 (100.64.0.0–100.127.255.255) for Carrier-Grade NAT (CGNAT) deployments. It&#39;s well-suited to this use case for three reasons:</p><ul><li><strong>No collision with existing private space.</strong> It doesn’t overlap the RFC-1918 ranges you almost certainly already use on-premises and across other Azure VNets, eliminating the routing conflicts that reusing 10.x space would introduce.</li><li><strong>Designed for intermediate, NATted traffic.</strong> The range is intended for traffic that is neither routable on the corporate LAN nor on the public internet — exactly the role Databricks cluster traffic plays before it’s SNATted at the firewall.</li><li><strong>A </strong><strong>/10 is enormous.</strong> 100.64.0.0/10 provides ~4 million addresses. Carving a /16 (65,536 addresses) per Databricks environment leaves ample room for even very large, multi-workspace deployments without ever touching your routable inventory.</li></ul><blockquote><strong>Note:</strong> Azure supports RFC-6598 space in VNets, but it isn’t part of the default “private” ranges Azure documents as recommended. Treat 100.64.0.0/10 as <strong>non-routable</strong> within your network — it must not be advertised over ExpressRoute/VPN to on-prem, and it must not overlap any address space that needs to reach Databricks directly. All cross-boundary reachability is handled via SNAT and Private Link, described below.</blockquote><h3>Reference architecture</h3><p>The design uses two resource groups and a two-firewall topology:</p><ul><li><strong>ip-conserve</strong> — the isolated <strong>island network</strong>. Contains the RFC-6598 VNet (100.64.0.0/16) where Azure Databricks, supporting VMs, Bastion, and private endpoints for the control plane live. This network is non-routable to the rest of the enterprise.</li><li><strong>ip-conserve-fw</strong> — the firewall resource group. Contains <strong>Firewall 1</strong> (local SNAT / force-tunnel gateway) and <strong>Firewall 2</strong> (controlled egress to internet and on-premises via a routable transit hub).</li></ul><p>Traffic flow at a glance:</p><ul><li>Databricks clusters and VMs in the RFC-6598 VNet send all non-local traffic (0.0.0.0/0) — via a user-defined route (UDR) — to <strong>Firewall 1</strong>.</li><li><strong>Firewall 1</strong> performs <strong>local SNAT</strong>, translating RFC-6598 source addresses behind its routable RFC-1918 IP, and force-tunnels the traffic onward.</li><li><strong>Firewall 2</strong> applies network and application rules for <strong>controlled egress</strong> to the internet, on-premises (via the routable transit hub / ExpressRoute or VPN gateway), and Azure PaaS dependencies.</li><li>Azure PaaS dependencies (ADLS / Blob storage, artifacts, logs, Event Hubs) are reached over <strong>Private Link / private endpoints</strong> through the tenant-routable hub, with Private DNS zones for resolution.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wLklFXjV65860k2iAS8kag.png" /></figure><p><em>Figure 1 — RFC-6598 island-network architecture: Databricks in the non-routable VNet, with Firewall 1 (local SNAT / force tunnel) and Firewall 2 (controlled egress).</em></p><h3>Address plan</h3><p>The RFC-6598 VNet is provisioned as a /16, with infrastructure subnets carved from the <strong>top</strong> of the range so the bulk of the space stays available for Databricks workspace subnets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ptTJ4r9LGnG5MUE3KEkVPQ.png" /><figcaption>Address plan</figcaption></figure><p>Databricks workspace subnets (public/private) can be sized per workspace from the remaining 100.64.x.x space. Because this space is non-routable and dedicated, subnets can be generous (e.g., /24 per role) without impacting your routable inventory.</p><h3>Prerequisites</h3><ul><li>An Azure subscription with permissions to create VNets, Azure Firewall (Standard or Premium SKU, managed via Firewall Policy), route tables, and VNet peerings.</li><li>A small block of <strong>routable RFC-1918 space</strong> for the two firewall VNets (two /24s in this design). This is the only routable space the deployment consumes.</li><li>A decision on <strong>one-firewall vs. two-firewall</strong> topology (see <em>Alternate method</em> below). Two firewalls is recommended.</li><li>A connectivity design for on-premises reachability (ExpressRoute or VPN gateway) attached to the routable transit hub, if required.</li><li>Familiarity with the Databricks VNet-injection (“bring your own VNet”) deployment model.</li></ul><h3>Step-by-step deployment (two-firewall — recommended)</h3><p>This procedure creates an RFC-6598 VNet at /16, connects it to an Azure Firewall (Standard or Premium SKU) with forced tunneling and local SNAT, then peers that environment to a second hub firewall for controlled internet and on-premises access.</p><p>Relevant Microsoft references:</p><ul><li>Forced Tunneling — <a href="https://learn.microsoft.com/en-us/azure/firewall/forced-tunneling">learn.microsoft.com/azure/firewall/forced-tunneling</a></li><li>Local SNAT (SNAT private ranges) — <a href="https://learn.microsoft.com/en-us/azure/firewall/snat-private-range">learn.microsoft.com/azure/firewall/snat-private-range</a></li><li>Azure Firewall rule processing — <a href="https://learn.microsoft.com/en-us/azure/firewall/rule-processing">learn.microsoft.com/azure/firewall/rule-processing</a></li></ul><p><strong>1. Create the RFC-6598 VNet</strong> in a resource group (100.64.0.0/16). At creation, add subnets at the <strong>end</strong> of the CIDR range for private endpoints, VMs, and Bastion:</p><ul><li>PE: 100.64.253.0/24</li><li>Utility (for VMs): 100.64.254.0/24</li><li>Bastion (for VMs): 100.64.255.0/26</li></ul><p><strong>2. Create a separate RFC-1918 </strong><strong>/24 VNet</strong> for the Azure Firewall that will act as the local NAT gateway (SNAT) for the RFC-6598 network. During firewall install you&#39;ll need two subnets:</p><ul><li>AzureFirewallManagementSubnet — /26 (required for forced tunneling)</li><li>AzureFirewallSubnet — /26</li></ul><p><strong>3. Deploy an Azure Firewall</strong> (Standard or Premium SKU) with an associated Firewall Policy, following the procedure for forced tunneling. Make note of the firewall’s private IP address.</p><p><strong>4. Configure local SNAT.</strong> In the associated Firewall Policy, open the SNAT (Private IP ranges) settings and set the private range to 255.255.255.255/32. Azure Firewall does <strong>not</strong> SNAT traffic destined to its configured private ranges, so a single /32 means almost no destination is treated as local — and the firewall SNATs effectively all traffic.</p><blockquote>Do <strong>not</strong> use 0.0.0.0/0 here — that designates every destination as private and disables SNAT entirely, which breaks the local-SNAT design. The setting and its meaning are identical on Classic and Standard/Premium; on Standard/Premium it lives in the Firewall Policy.</blockquote><p><strong>5. Peer the RFC-6598 VNet to the Firewall 1 VNet.</strong></p><p><strong>6. Create a route table</strong> for the RFC-6598 VNet’s Databricks and VM subnets that sends all non-local traffic (0.0.0.0/0) to the <strong>Firewall 1</strong> private IP (next hop type: <em>Virtual Appliance</em>). Associate it with the Databricks public/private subnets.</p><p><strong>7. Create a pass-through network rule</strong> on Firewall 1 that collects all traffic from the RFC-6598 VNet and, using the route from the previous step, forwards it to <strong>Firewall 2</strong>.</p><p><strong>8. Create a second RFC-1918 </strong><strong>/24 VNet</strong> for the egress firewall. During install you&#39;ll need one subnet:</p><ul><li>AzureFirewallSubnet — /26</li></ul><p><strong>9. Deploy Azure Firewall</strong> (with Firewall Policies) using the new VNet/subnet. This is <strong>Firewall 2</strong>, your controlled egress point to the internet and on-premises.</p><p><strong>10. Create the required network rules</strong> on Firewall 2 for Databricks assets (minimum set):</p><ul><li>adb-storage — storage (ADLS / Blob)</li><li>adb-eh — Event Hubs</li><li>adb-sql — SQL (control plane metastore / external)</li><li>adb-cp — Azure Databricks control plane</li></ul><p><strong>11. Create application rules</strong> for egress to the internet (optional, as required by your policy — e.g., for library/package installation).</p><p><strong>12. Deploy the Databricks workspace(s)</strong> into the RFC-6598 VNet (VNet injection, using the public/private subnets), then validate (see below).</p><h3>Validation</h3><p>From a Databricks notebook, run:</p><pre>%sh<br>curl https://ipinfo.io/ip</pre><p>The command should return the <strong>public IP of Firewall 2</strong> — confirming that cluster egress is being SNATted and force-tunneled through the firewall chain rather than originating from a routable Databricks address. If it returns anything else (or fails), revisit the UDR (Step 6), the local SNAT setting (Step 4), and the pass-through rule (Step 7).</p><h3>Alternate method (single firewall)</h3><p>A simpler topology uses only Firewall 1. Follow Steps 1–7 above (creating Firewall 1 <strong>with forced tunneling enabled</strong> and local SNAT), then <strong>skip</strong> the second firewall and deploy Databricks directly.</p><p>In this variant, because non-local traffic is <strong>not</strong> forwarded to a second firewall via an explicit 0.0.0.0/0 rule, Azure Firewall&#39;s <strong>default NAT</strong> sends the traffic to the internet, allowing clusters to come up and reach external dependencies.</p><p><strong>Recommendation:</strong> Use <strong>two firewalls</strong>. The second firewall provides a clean separation of concerns — it’s dedicated to egressing traffic to the internet or pointing at an ExpressRoute/VPN gateway for on-premises access — and gives you a single, well-defined point at which to monitor and inspect all egress traffic. The single-firewall method is best reserved for proofs of concept.</p><h3>Design considerations &amp; best practices</h3><ul><li><strong>Keep RFC-6598 strictly non-routable.</strong> Don’t advertise 100.64.0.0/10 (or the /16 used here) over ExpressRoute or VPN. Its conservation benefit depends on it never consuming or colliding with routable space.</li><li><strong>PaaS access via Private Link.</strong> Reach ADLS/Blob, artifacts, logs, and Event Hubs through private endpoints in the PE subnet with the appropriate Private DNS zones, rather than over public egress, to keep data-plane traffic private and inspected.</li><li><strong>Control-plane reachability.</strong> Use the Private DNS zone for the Databricks control plane (and Private Link where applicable) so the workspace resolves and reaches the control plane from inside the island network.</li><li><strong>SNAT capacity.</strong> Azure Firewall SNAT uses non-privileged source ports per instance. For large, bursty Databricks egress, monitor SNAT port utilization and scale the firewall SKU accordingly (start modest; move to Standard/Premium if egress volume is high).</li><li><strong>Right-size firewall SKUs.</strong> SNAT itself isn’t resource-intensive; choose the egress firewall SKU based on the inspection features (TLS inspection, IDPS) you require, not just throughput.</li><li><strong>TLS inspection (Premium SKU).</strong> Enable TLS inspection on the egress firewall (Firewall 2) to decrypt, inspect, and re-encrypt outbound HTTPS from Databricks clusters — surfacing threats and data movement that would otherwise be opaque inside the SNAT boundary. TLS inspection (and IDPS) requires the Azure Firewall Premium SKU, so select Premium for Firewall 2 where policy mandates deep packet inspection of egress. Pair it with a Firewall Policy that trusts your internal CA and excludes endpoints that pin certificates (certain Azure PaaS / Databricks control-plane calls) from interception.</li><li><strong>Monitoring.</strong> Centralize firewall logs (the dedicated egress firewall makes this clean) for audit and security review of all Databricks outbound traffic.</li><li><strong>This pattern repeats.</strong> Because the RFC-6598 space is dedicated and non-routable, you can stamp this island-network pattern out for additional Databricks environments without re-consuming routable IPs each time.</li><li><strong>Use Service Endpoint Policies (SEP) to minimize firewall costs.</strong> All traffic passes through the firewall in this type of deployment. Azure virtual network service endpoint policies filter outbound traffic from the classic compute plane, ensuring connections are only made to specific Azure Storage accounts. For example, Databricks pulls container images from artifact storage for each node in a cluster — which can be 12 GB per image. SEPs let you filter this traffic through the Azure backbone instead of routing it through the firewall.</li></ul><h3>References</h3><ul><li>Microsoft Learn: <em>Prevent IPv4 Exhaustion in Azure</em> — <a href="https://learn.microsoft.com/en-us/azure/architecture/networking/guide/internet-protocol-version-4-exhaustion">learn.microsoft.com/azure/architecture/networking/guide/internet-protocol-version-4-exhaustion</a></li><li>Microsoft Learn: <em>Azure Firewall forced tunneling</em> — <a href="https://learn.microsoft.com/en-us/azure/firewall/forced-tunneling">learn.microsoft.com/azure/firewall/forced-tunneling</a></li><li>Microsoft Learn: <em>Azure Firewall SNAT private IP ranges</em> — <a href="https://learn.microsoft.com/en-us/azure/firewall/snat-private-range">learn.microsoft.com/azure/firewall/snat-private-range</a></li><li>Microsoft Learn: <em>Azure Firewall rule processing</em> — <a href="https://learn.microsoft.com/en-us/azure/firewall/rule-processing">learn.microsoft.com/azure/firewall/rule-processing</a></li><li>Microsoft Learn: <em>VNet address ranges FAQ</em> — <a href="https://learn.microsoft.com/en-us/azure/virtual-network/virtual-networks-faq#what-address-ranges-can-i-use-in-my-virtual-networks">learn.microsoft.com/azure/virtual-network/virtual-networks-faq</a></li><li>IETF: <em>RFC 6598 — IANA-Reserved IPv4 Prefix for Shared Address Space</em> — <a href="https://www.rfc-editor.org/rfc/rfc6598.html">rfc-editor.org/rfc/rfc6598.html</a></li><li>Microsoft Learn: <em>Configure Azure virtual network service endpoint policies for storage access from classic compute</em> — <a href="https://learn.microsoft.com/en-us/azure/databricks/security/network/classic/service-endpoints">learn.microsoft.com/azure/databricks/security/network/classic/service-endpoints</a></li><li>GitHub Repository with Terraform code to build this example — <a href="https://github.com/ScottGriz/databricks-tools/tree/main/rfc6598-databricks-terraform">github.com/ScottGriz/databricks-tools/tree/main/rfc6598-databricks-terraform</a></li></ul><p><em>Have you run into IPv4 exhaustion with cloud data platforms? I’d love to hear how you’ve tackled it — drop a comment below.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f25f17d48e2f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/databricks-platform-sme/deploying-azure-databricks-with-rfc-6598-shared-address-space-zero-routable-ip-consumption-f25f17d48e2f">Deploying Azure Databricks with RFC-6598 Shared Address Space: Zero Routable IP Consumption</a> was originally published in <a href="https://medium.com/databricks-platform-sme">Databricks Platform SME</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How do I limit internet access but allow developers to install Python packages on Databricks…]]></title>
            <link>https://medium.com/databricks-platform-sme/how-do-i-limit-internet-access-but-allow-developers-to-install-python-packages-on-databricks-bdffc0efe503?source=rss-acb3a812b683------2</link>
            <guid isPermaLink="false">https://medium.com/p/bdffc0efe503</guid>
            <category><![CDATA[platform]]></category>
            <category><![CDATA[python-libraries]]></category>
            <category><![CDATA[databricks-unity-catalog]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Scott Grzybowski]]></dc:creator>
            <pubDate>Wed, 22 Oct 2025 22:07:37 GMT</pubDate>
            <atom:updated>2025-10-22T22:07:59.003Z</atom:updated>
            <content:encoded><![CDATA[<h3>How do I limit internet access but allow developers to install Python packages on Databricks Serverless?</h3><p>UC Volumes, Serverless Compute and Python Packages</p><h3>Overview</h3><p>Databricks users need the ability to install various Python packages in order for their notebook code to run. Developers would normally use “pip” to install these packages from <a href="http://pypi.org/">pypi.org</a> but many enterprise customers want to control where and when the packages are installed and is usually handled by connecting to an internal artifactory. This is a challenge in a SaaS solution since you do not own the networking.</p><p>I am providing a workaround in this article when using a serverless based compute cluster since networking is not owned by the customer as compared to a classic compute cluster.</p><p>I recommend that developers use notebook-scoped libraries. My example in the following article is doing just that using<em> !pip </em>or <em>%pip</em> from a notebook.</p><h3>Reference Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A8A-wKE__V8P1te6Zs-czQ.png" /></figure><h3>Before you begin</h3><h4>Workspace requirements</h4><p>Workspace enabled for Unity Catalog.</p><h4>Compute requirements:</h4><p>Serverless Compute enabled</p><p><strong>NOTE</strong>: I am using the latest version of the serverless environment (version “4”). You can review the installed Python packages on the cluster <a href="https://learn.microsoft.com/en-us/azure/databricks/release-notes/serverless/environment-version/four">here</a></p><h4>Permissions required:</h4><p>Databricks Account Administrator privileges</p><p>To create a Volume, you must be a metastore admin or a user with the CREATE VOLUME privilege on the Unity Catalog metastore attached to the workspace.</p><p>Additional permission requirements are specified in each task-based section that follows.</p><h3>Checklist</h3><ol><li>Create a Network Connection Configuration (NCC) and assign to the workspace</li><li>Create an Egress Policy and assign to the workspace</li><li>Identify and download the Python packages</li><li>Create a shared Unity Catalog Volume</li><li>Upload the package .whl files to the volume</li><li>Install packages via a notebook using local option</li></ol><h3>Serverless Security Controls</h3><p>Login to the databricks account administrator portal</p><p><a href="https://accounts.azuredatabricks.net/">Azure</a><br><a href="https://accounts.cloud.databricks.com/">AWS</a><br><a href="https://accounts.gcp.databricks.com/">GCP</a></p><p>I am using Azure Databricks for this article. The screens may differ between clouds.</p><h3>Network Connectivity Configuration (NCC)</h3><p>If you haven’t configured an NCC for your workspace, this will be our first step.</p><p>Choose “Cloud Resources” from the Account Portal.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2AtJhbITPfLRNdU_Xg9vrA.png" /><figcaption>Account Console Cloud Resources</figcaption></figure><p>Click “Add Network Connectivity Configuration”<br><strong>NOTE</strong>: If you already have an NCC, just search.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V1ZM5oTOyvaN3vColJr5AQ.png" /><figcaption>Account Create Network Connectivity Config (NCC)</figcaption></figure><p>Choose the region of your workspace and add any private endpoint connections for storage access.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y44SiB43EMkxfAbkErKQrQ.png" /><figcaption>Account Private Endpoints NCC</figcaption></figure><p>Choose “Workspaces” and assign the NCC to your workspace.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/945/1*QfA1FXDVdKdor-6D9M2apQ.png" /><figcaption>Account Assign NCC to Workspace</figcaption></figure><h3>Egress Controls</h3><p>Navigate to “Cloud Resources” again and choose “Network Policies — Serverless” and click on “Create new network policy”. <br><strong>NOTE</strong>: If you already have a policy, just search.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gKnVkbHFwOpIB4p7kTeDGA.png" /><figcaption>Account create netwrok Policy</figcaption></figure><p>Provide a name and then choose “Restricted access to specific destinations”. If you would like to understand more about serverless egress controls, please read our <a href="https://learn.microsoft.com/en-us/azure/databricks/security/network/serverless-network-security/network-policies">documentation</a>. I have chosen to only allow access to “database.windows.net” and <a href="http://microsoft.com/">microsoft.com</a>. Therefore, <a href="http://pypi.org/">pypi.org</a> and any other domains are off limits.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iJaf8qEPRMY3x0i5wJTmzA.png" /><figcaption>Account Egress Restrictions</figcaption></figure><p>Once done, choose “next” for the ingress controls. I am leaving “allow access from all sources” enabled for this article.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8DFNdxjzR9Ek-JrnI9zhaw.png" /><figcaption>Account Ingress Controls</figcaption></figure><p>Finally, choose “next” and assign the workspace to use this policy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I3NlbwVBOZnddIxWXwTMhg.png" /><figcaption>Account Assign Policy to Workspace</figcaption></figure><p><strong>NOTE</strong>: If you already have a serverless cluster running, you must terminate the cluster and restart to pick up the changes from above.</p><h3>Downloading packages from <a href="http://pypi.org/">pypi.org</a></h3><p>You will need to determine the common packages needed by your developers and download these with all dependencies to your local machine first. As I mentioned above, databricks serverless images have many <a href="https://learn.microsoft.com/en-us/azure/databricks/release-notes/serverless/environment-version/four">preinstalled</a> packages</p><p>There are several ways to download a package and all it’s dependencies to .whl files</p><p>To download a specific package (e.g., <em>msal</em>) and all its dependencies to the current directory, use the following command from a command window or terminal:</p><pre>pip download msal</pre><p>Download to a specific directory<br>If you want to download the packages to a different directory, use the — dest or — download flag:</p><pre>pip download msal - dest /path/to/downloaded/files</pre><p>If you have a requirements.txt file listing multiple packages, you can download all of them and their dependencies using:</p><pre>pip download -r requirements.txt - dest /path/to/downloaded/files</pre><p>Sample output from the download above</p><p>Collecting msal<br>Downloading msal-1.34.0-py3-none-any.whl.metadata (11 kB)<br>Collecting requests,&gt;=2.0.0 (from msal)<br>Downloading requests-2.32.5-py3-none-any.whl.metadata (4.9 kB)<br>Collecting PyJWT,&gt;=1.0.0 (from PyJWT[crypto],&gt;=1.0.0-&gt;msal)<br>Using cached PyJWT-2.10.1-py3-none-any.whl.metadata (4.0 kB)<br>Collecting cryptography&lt;49,&gt;=2.5 (from msal)<br>Downloading cryptography-46.0.2-cp311-abi3-macosx_10_9_universal2.whl.metadata (5.7 kB)<br>Collecting cffi&gt;=2.0.0 (from cryptography&lt;49,&gt;=2.5-&gt;msal)<br>Downloading cffi-2.0.0-cp312-cp312-macosx_11_0_arm64.whl.metadata (2.6 kB)<br>Collecting charset_normalizer&lt;4,&gt;=2 (from requests,&gt;=2.0.0-&gt;msal)<br>Downloading charset_normalizer-3.4.3-cp312-cp312-macosx_10_13_universal2.whl.metadata (36 kB)<br>Collecting idna&lt;4,&gt;=2.5 (from requests,&gt;=2.0.0-&gt;msal)<br>Using cached idna-3.11-py3-none-any.whl.metadata (8.4 kB)<br>Collecting urllib3,&gt;=1.21.1 (from requests,&gt;=2.0.0-&gt;msal)<br>Using cached urllib3–2.5.0-py3-none-any.whl.metadata (6.5 kB)<br>Collecting certifi&gt;=2017.4.17 (from requests,&gt;=2.0.0-&gt;msal)<br>Downloading certifi-2025.10.5-py3-none-any.whl.metadata (2.5 kB)<br>Collecting pycparser (from cffi&gt;=2.0.0-&gt;cryptography&lt;49,&gt;=2.5-&gt;msal)<br>Downloading pycparser-2.23-py3-none-any.whl.metadata (993 bytes)<br>Downloading msal-1.34.0-py3-none-any.whl (116 kB)<br>Downloading cryptography-46.0.2-cp311-abi3-macosx_10_9_universal2.whl (7.3 MB)<br>━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.3/7.3 MB 4.7 MB/s eta 0:00:00<br>Using cached PyJWT-2.10.1-py3-none-any.whl (22 kB)<br>Downloading requests-2.32.5-py3-none-any.whl (64 kB)<br>Downloading charset_normalizer-3.4.3-cp312-cp312-macosx_10_13_universal2.whl (205 kB)<br>Using cached idna-3.11-py3-none-any.whl (71 kB)<br>Using cached urllib3–2.5.0-py3-none-any.whl (129 kB)<br>Downloading certifi-2025.10.5-py3-none-any.whl (163 kB)<br>Downloading cffi-2.0.0-cp312-cp312-macosx_11_0_arm64.whl (181 kB)<br>Downloading pycparser-2.23-py3-none-any.whl (118 kB)<br>Saved ./msal-1.34.0-py3-none-any.whl<br>Saved ./cryptography-46.0.2-cp311-abi3-macosx_10_9_universal2.whl<br>Saved ./PyJWT-2.10.1-py3-none-any.whl<br>Saved ./requests-2.32.5-py3-none-any.whl<br>Saved ./charset_normalizer-3.4.3-cp312-cp312-macosx_10_13_universal2.whl<br>Saved ./idna-3.11-py3-none-any.whl<br>Saved ./urllib3–2.5.0-py3-none-any.whl<br>Saved ./certifi-2025.10.5-py3-none-any.whl<br>Saved ./cffi-2.0.0-cp312-cp312-macosx_11_0_arm64.whl<br>Saved ./pycparser-2.23-py3-none-any.whl</p><p>Successfully downloaded msal cryptography PyJWT requests charset_normalizer idna urllib3 certifi cffi pycparser</p><h3>Unity Catalog object set up</h3><p>We need to create a central location for all our packages. In my case I am using the default schema in my <em>scott_griz</em> catalog, named <em>python_packages</em>. You may create your Volume anywhere but keep in mind that this should be a global shared catalog/schema so that developers can access the files. Permissions will be unique to your organization and beyond the scope of this article.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/605/1*ljKNBZnVJtfZ36H3aUrlIQ.png" /><figcaption>Unity catalog explorer</figcaption></figure><p>If you don’t have a volume created, use the catalog explorer on the left navigation pane and choose your catalog / schema and then click “Create =&gt; Volume”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A6G7TKgF21rpMkuPDBcUZA.png" /><figcaption>UC Create Volume</figcaption></figure><h3>Upload the packages to the volume</h3><p>Upload the .whl files to the newly created volume <em>python_packages. </em>Navigate to the volume and choose <em>Upload to this volume </em>and choose the downloaded .whl files. I am keeping this article simple and did not create directories. In the real world, organizing your packages using directories is preferred.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7azIoqrvYXQ76G4LrKtHCA.png" /><figcaption>UC Volume upload</figcaption></figure><h3>Installing packages using a serverless notebook</h3><p>The latest Serverless image Version 4 already has the Python <em>msal</em> package installed. The version included in the image is 1.32.4 as noted in our <a href="https://learn.microsoft.com/en-us/azure/databricks/release-notes/serverless/environment-version/four">documentation</a>. I am going to install the newer version 1.34.0 in my notebook scope for the purpose of this article.</p><p>If I were to try and install the package using the command:</p><pre>%pip install msal==1.34.0</pre><p>The command fails because SEG does not whitelist the pypi.org domain. You will see an error message similar to:</p><p>WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by ‘NewConnectionError(‘&lt;pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f464fe52c00&gt;: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution’)’: /simple/msal/<br>WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by ‘NewConnectionError(‘&lt;pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f46502163c0&gt;: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution’)’: /simple/msal/<br>WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by ‘NewConnectionError(‘&lt;pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f464fe53620&gt;: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution’)’: /simple/msal/<br>WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by ‘NewConnectionError(‘&lt;pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f464fe53860&gt;: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution’)’: /simple/msal/<br>WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by ‘NewConnectionError(‘&lt;pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f464fe53a70&gt;: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution’)’: /simple/msal/</p><p>ERROR: Could not find a version that satisfies the requirement msal==1.34.0 (from versions: none)</p><p>ERROR: No matching distribution found for msal==1.34.0</p><p>However, If I were to try and install the package using the command:</p><pre>%pip install msal==1.34.0 - find-links /Volumes/scott_griz/default/python_packages</pre><p>The command succeeds after a few warnings:</p><p>Looking in links: /Volumes/scott_griz/default/python_packages</p><p>WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by ‘NewConnectionError(‘&lt;pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f442ed11430&gt;: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution’)’: /simple/msal/<br>WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by ‘NewConnectionError(‘&lt;pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f442dc3be30&gt;: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution’)’: /simple/msal/<br>WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by ‘NewConnectionError(‘&lt;pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f442dc4c170&gt;: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution’)’: /simple/msal/<br>WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by ‘NewConnectionError(‘&lt;pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f442dc4c380&gt;: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution’)’: /simple/msal/<br>WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by ‘NewConnectionError(‘&lt;pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f442dc4c590&gt;: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution’)’: /simple/msal/</p><p>Processing /Volumes/scott_griz/default/python_packages/msal-1.34.0-py3-none-any.whl<br>Requirement already satisfied: requests,&gt;=2.0.0 in /databricks/python3/lib/python3.12/site-packages (from msal==1.34.0) (2.32.3)<br>Requirement already satisfied: PyJWT,&gt;=1.0.0 in /databricks/python3/lib/python3.12/site-packages (from PyJWT[crypto],&gt;=1.0.0-&gt;msal==1.34.0) (2.10.1)<br>Requirement already satisfied: cryptography&lt;49,&gt;=2.5 in /databricks/python3/lib/python3.12/site-packages (from msal==1.34.0) (43.0.3)<br>Requirement already satisfied: cffi&gt;=1.12 in /databricks/python3/lib/python3.12/site-packages (from cryptography&lt;49,&gt;=2.5-&gt;msal==1.34.0) (1.17.1)<br>Requirement already satisfied: charset-normalizer&lt;4,&gt;=2 in /databricks/python3/lib/python3.12/site-packages (from requests,&gt;=2.0.0-&gt;msal==1.34.0) (3.3.2)<br>Requirement already satisfied: idna&lt;4,&gt;=2.5 in /databricks/python3/lib/python3.12/site-packages (from requests,&gt;=2.0.0-&gt;msal==1.34.0) (3.7)<br>Requirement already satisfied: urllib3,&gt;=1.21.1 in /databricks/python3/lib/python3.12/site-packages (from requests,&gt;=2.0.0-&gt;msal==1.34.0) (2.3.0)<br>Requirement already satisfied: certifi&gt;=2017.4.17 in /databricks/python3/lib/python3.12/site-packages (from requests,&gt;=2.0.0-&gt;msal==1.34.0) (2025.1.31)<br>Requirement already satisfied: pycparser in /databricks/python3/lib/python3.12/site-packages (from cffi&gt;=1.12-&gt;cryptography&lt;49,&gt;=2.5-&gt;msal==1.34.0) (2.21)<br>Installing collected packages: msal<br>Attempting uninstall: msal<br>Found existing installation: msal 1.32.3<br>Not uninstalling msal at /databricks/python3/lib/python3.12/site-packages, outside environment /local_disk0/.ephemeral_nfs/envs/pythonEnv-13fa1266-a13e-4114–97e7-d123d3189edb<br>Can’t uninstall ‘msal’. No files were found to uninstall.<br>Successfully installed msal-1.34.0<br>Note: you may need to restart the kernel using %restart_python or dbutils.library.restartPython() to use updated packages.</p><p>As I mentioned before, <em>msal</em> 1.32.4 is already installed in the base image so many of the dependency version requirements are already satisfied.</p><h3>Conclusion</h3><p>Installing python packages from a UC Volume may not be the most elegant solution but it does solve the problem of blocking the wild west of package installations in a serverless environment.</p><p>Let’s recap the steps:</p><ul><li>Unity Catalog Enabled Workspace</li><li>Serverless notebooks enabled</li><li>Establish Serverless Egress controls and only whitelist necessary domains in which <a href="http://pypi.org/">pypi.org</a> is not</li><li>Download the python packages and dependencies locally</li><li>Create a Unity Catalog Volume and grant permissions to necessary users or groups</li><li>Upload package .whl files to the Volume location</li><li>Install packages using:</li></ul><pre>%pip install msal==1.34.0 - find-links /Volumes/scott_griz/default/python_packages</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bdffc0efe503" width="1" height="1" alt=""><hr><p><a href="https://medium.com/databricks-platform-sme/how-do-i-limit-internet-access-but-allow-developers-to-install-python-packages-on-databricks-bdffc0efe503">How do I limit internet access but allow developers to install Python packages on Databricks…</a> was originally published in <a href="https://medium.com/databricks-platform-sme">Databricks Platform SME</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Mitigate Secret Spread on Azure Databricks]]></title>
            <link>https://medium.com/databricks-platform-sme/how-to-mitigate-secret-spread-on-azure-databricks-3b13f9f324f2?source=rss-acb3a812b683------2</link>
            <guid isPermaLink="false">https://medium.com/p/3b13f9f324f2</guid>
            <category><![CDATA[azure-devops]]></category>
            <category><![CDATA[azure-managed-identities]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Scott Grzybowski]]></dc:creator>
            <pubDate>Thu, 14 Nov 2024 00:02:59 GMT</pubDate>
            <atom:updated>2024-11-14T00:03:27.618Z</atom:updated>
            <content:encoded><![CDATA[<p><em>This blog and related content was a collaboration between my amazing colleague </em><a href="https://medium.com/u/1a5ade229d6e"><em>Alfeu Duran</em></a><em> Santos and me</em></p><h3>“What is the best way to authenticate with Databricks and where do I store my secrets?”</h3><p>Let’s be honest, no one wants to store user identities, passwords or secrets in order to programmatically authenticate to run a job or anything else for that matter. Personal Access Tokens (PAT) are even harder to manage securely and you end up with a complex maze of secret management and the “spread of secrets”.</p><h3><strong>Introducing a better way</strong></h3><p>When working with cloud platforms like Azure, securing access to critical resources is paramount. Traditionally, many teams rely on <strong>Service Principals (SP)</strong> to manage authentication between services. While this approach works well, it introduces several challenges related to secret management, such as regularly rotating passwords or certificates and ensuring these secrets remain safe from exposure.</p><p>Expanding on our SP example above, the Entra ID or Databricks SP will have a client ID and secret associated with it. We will use these 2 values to hit an OAuth endpoint and get a short lived OAuth Token and all is good, right? Not exactly; we need to store this client ID and secret securely in Azure Key Vault or other mechanism and grant permissions to the process to retrieve the secret and so on.</p><p>An alternative solution to boost security is the use of <strong>User Managed Identities (UMI)</strong>. Unlike Service Principals, UMIs are managed by Azure, eliminating the need for manual secret handling. This approach offers key advantages, such as:</p><ul><li><strong>No need for password or secret rotation</strong>, since Azure takes care of the identity lifecycle.</li><li><strong>Reduced risk of credential exposure</strong>, as secrets are never stored or shared.</li><li><strong>Simplified access management</strong> through direct integration with Databricks Workspace permissions, enabling precise control over <strong>who can access clusters, notebooks, jobs, and data</strong>.</li></ul><p>Before we dive into the details, a shout out to Christos Matskas | @ChristosMatskas and Varun Karandikar | @varundikar<a href="https://www.youtube.com/watch?v=vYUKC0mZFqI"> podcast “On.Net Show”</a> where they walk us through Managed Identities. I would recommend watching the video. <a href="https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/overview">Microsoft documentation</a> can also come in handy.</p><p>You may also want to review our previous blog “<a href="https://medium.com/databricks-platform-sme/integrating-databricks-asset-bundles-into-a-ci-cd-pipeline-on-azure-7b181b26d9ae">Integrating Databricks Asset Bundles into a CI/CD Pipeline on Azure</a>” which shows another approach to using SP with Databricks Asset Bundles (DAB)</p><p>In this blog post, we’ll walk through the <strong>step-by-step process of integrating Azure Databricks with a User Managed Identity (UMI)</strong> using <strong>Azure DevOps pipelines</strong>. This identity-based setup not only enhances security but also simplifies managing <strong>workspace-level permissions</strong> in Databricks, ensuring that only authorized entities and processes can perform actions.</p><p>Below is an architecture diagram illustrating this integration flow in Azure DevOps and Azure</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*m3D6rthxzaXVtqIa" /></figure><p>Let’s dive into how you can <strong>make your Databricks environment more secure and scalable</strong> with this streamlined identity approach</p><h3>1. Setting up the User Managed Identity (UMI)</h3><p>Let’s start by creating the <strong>User Managed Identity (UMI)</strong>.</p><ol><li>Go to the <strong>Azure Portal</strong> and search for <strong>Managed Identities</strong>.</li><li>Click <strong>+ Create</strong> and fill in the basics:</li></ol><ul><li><strong>Subscription</strong>: Pick the correct one.</li><li><strong>Resource Group</strong>: Use an existing one or create a new one.</li><li><strong>Name</strong>: Something like “databricks-umi” works!</li></ul><p>3. Select <strong>User-assigned</strong>, hit <strong>Review + Create</strong>, and click <strong>Create</strong>.</p><p>After creating the UMI, <strong>don’t forget to assign the necessary permissions</strong> to access your Databricks Workspace — like the <strong>Contributor role</strong> or a custom one with limited privileges.</p><h3>2. Configuring the Agent Virtual Machine to Use the UMI</h3><p>Now that we’ve got the UMI, let’s set things up so our Azure DevOps pipeline can authenticate with Databricks using that UMI. We’ll run the pipeline on a virtual machine (VM), which will act as an Azure DevOps Agent Pool. This VM will handle authentication securely using the UMI.</p><p>Here’s the process:</p><h3>Create a Virtual Machine (VM)</h3><ol><li>You can use any OS you like (I’m going with <strong>Ubuntu</strong> for this example).</li><li>Once the VM is up, <strong>assign the UMI</strong> to it:</li></ol><ul><li>Go to <strong>Settings</strong> in the VM menu.</li><li>Under <strong>Security</strong>, click on <strong>Identity</strong>.</li><li>In the <strong>User assigned</strong> tab, click <strong>Add</strong> and select the UMI you just created.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VL2liX-iHwiRhd7d" /></figure><h3>Install Required Tools on the VM</h3><p><strong>To ensure the VM can interact with Databricks and Azure, we need to install both the Databricks CLI and the Azure CLI.</strong></p><ol><li><strong>SSH into your VM:<br>ssh your-user@vm-ip</strong></li><li><strong>Install Azure CLI </strong><a href="https://learn.microsoft.com/en-us/cli/azure/install-azure-cli"><strong>documentation</strong></a><strong>:<br>curl -sL </strong><a href="https://aka.ms/InstallAzureCLIDeb"><strong>https://aka.ms/InstallAzureCLIDeb</strong></a><strong> | sudo bash</strong></li><li><strong>install the Databricks CLI </strong><a href="https://docs.databricks.com/en/dev-tools/cli/install.html"><strong>documentation</strong></a><strong>:<br>curl -fsSL </strong><a href="https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh"><strong>https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh</strong></a><strong> | sh</strong></li><li><strong>Verify the installations: Run the following commands to make sure both tools are installed correctly:<br>az — version<br>databricks — version</strong></li></ol><h3>3. Setting Up the Agent Pool in Azure DevOps</h3><p>Now that our <strong>VM</strong> is ready with the UMI attached, the next step is to create an <strong>Agent Pool</strong> in <strong>Azure DevOps</strong> and link our VM to it. This way, the VM will securely run our pipelines.</p><h4>Step 1: Create a New Agent Pool</h4><ol><li>In <strong>Azure DevOps</strong>, go to <strong>Organization Settings</strong> → <strong>Agent Pools</strong>.</li><li>Click <strong>Add Pool</strong> and give it a name.</li><li>Click <strong>Create</strong>.</li></ol><h4>Step 2: Add a New Agent</h4><ol><li>After creating the pool, click <strong>New Agent</strong>.</li><li>Follow the installation instructions provided by Azure DevOps, which will guide you based on the <strong>OS of your VM</strong>(in our case, Ubuntu).</li><li>Make sure you meet the <strong>prerequisites</strong> listed for the chosen OS, like installing necessary packages or enabling certain services.</li><li>Once the agent is set up on the VM and running, it will automatically connect to your <strong>Agent Pool</strong>.</li></ol><h3>4. Creating and Running the Azure DevOps Pipeline</h3><p>In this example, I’ll be using <strong>Databricks Asset Bundles</strong> and generating code based on the <strong>default Python template</strong>. This is a great way to quickly set up the structure we need for deployments. If you want to explore other templates, check out the <a href="https://docs.databricks.com/en/dev-tools/bundles/templates.html">official documentation</a>.</p><p>The code for this example is available in the following repository: <a href="https://github.com/alfeuduran/databricks-bundles-umi"><strong>https://github.com/alfeuduran/databricks-bundles-umi</strong></a>. After generating the template, you’ll find an example of the pipeline configuration in the <strong>azure-pipelines.yml</strong> file. Let’s walk through the key pieces that you’ll need to tweak for your environment.</p><h3>Key Changes in the azure-pipelines.yml File</h3><p>Once you’ve generated the template, here’s what you need to adjust to ensure the pipeline works with the <strong>VM agent</strong> we configured earlier.</p><p><strong>Set the Pool Name and Agent Name:<br></strong>In the YAML file, you’ll need to specify the <strong>pool name</strong> and <strong>agent name</strong> to match the setup from our previous steps. Here’s an example snippet:<br>yaml<br>jobs:</p><p>- job: onReleaseJob</p><p>pool:</p><p>name: poolName # Replace with your pool name</p><p>demands:</p><p>- agent.name -equals agentName # Replace with your VM agent name</p><ol><li><strong>Note:</strong> Make sure the pool name matches exactly with the one you created in Azure DevOps, and the agent name matches the one registered on the VM.</li></ol><p><strong>Change the variable group name, in my case i created the umiVariableGroup with the following variables:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3Ua8IIg9OhiuE-iP" /></figure><p>In azure-pipelines.yml make sure to use the same name for the variable group.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/930/0*-zkYjOKd9pSGrmEp" /></figure><h3>5. Running the Pipeline and Deploying to Databricks</h3><p>Once the <strong>pipeline is configured</strong> with the correct pool and agent names, and the variables are set (in my case, I created the <strong>umiVariableGroup</strong> with the required variables), the hard part is over! 🎉</p><p>Now, all that’s left is to <strong>run the pipeline</strong>. You can trigger it directly from <strong>Azure DevOps</strong> or through a <strong>GitHub repository</strong> if you’re using source control.</p><h3>Expected Outcome</h3><p>If everything is set up correctly, running the pipeline should:</p><ul><li><strong>Deploy the project</strong> using the <strong>Databricks Asset Bundles</strong> with the <a href="https://docs.databricks.com/en/dev-tools/bundles/templates.html">default-python</a> template.</li><li><strong>Deploy a Delta Live Table</strong> (DLT) pipeline and a <strong>job</strong>, automating the deployment within Databricks.</li></ul><p>And that’s it! With the <strong>UMI</strong> securely handling authentication, the VM agent pool running smoothly, and the pipeline ready to deploy, you’ve created a streamlined, secure, and automated workflow. 🚀</p><h3>Conclusion</h3><p>UMIs may not fit every situation but they certainly mitigate the secret spread and the need to store client credentials.</p><p>This setup makes it easy to manage deployments without worrying about secrets, passwords, or manual interventions. Now go ahead, hit that <strong>Run Pipeline</strong> button, and watch your Databricks deployment come to life! 🎯</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3b13f9f324f2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/databricks-platform-sme/how-to-mitigate-secret-spread-on-azure-databricks-3b13f9f324f2">How to Mitigate Secret Spread on Azure Databricks</a> was originally published in <a href="https://medium.com/databricks-platform-sme">Databricks Platform SME</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>