Passwordless Cloud-to-Cloud Data Ingestion

How we connected a Spark job running in one Azure tenant to a blob storage account in another tenant — without a single password, connection string, or secret ever touching our config.

The Power of Native Cloud Trust

Securing data transfers across organizational boundaries usually forces a compromise: you must exchange long-lived secrets, API keys, or certificates between parties. Every shared secret is a potential leak waiting to happen, creating operational burden and audit headaches. However, when both source (tenant A) and destination (tenant B) operate on the same platform (Microsoft Azure in our case), you gain a unique advantage: a shared root of trust.

Instead of tenant A and tenant B needing to directly trust each other with a shared password, both parties simply express their consent to Azure. tenant A delegates authorization to its managed identity, and tenant B grants that identity access to its target resource (Azure Data Lake Storage in our example). Microsoft Entra ID handles the identity assertion behind the scenes, effectively acting as an impartial security guard verifying every request.

To ensure network traffic never leaves the Microsoft backbone, the entire pipeline is wired through an Azure private endpoint, completely isolating the data flow from the public internet.

The Challenge

We had a straightforward-sounding requirement: a Spark job using SDLB, running inside an Azure Container App, needed to write Parquet files to an Azure Data Lake Storage Gen2 account.

Simple enough — except for two details that turned this into a real project:

1. The Container App and the storage account live in different Azure Entry ID tenants. Tenant A’s identity, subscription, network config — none of it exists inside tenant B.
2. No secrets allowed. Everything needed to be backed by a real, verifiable identity.

The instinct, early on, is to treat “secure access to a resource” as one problem. It isn’t. It’s two:

• Identity and access — proving who you are, and whether you’re allowed to touch the resource.
• Network reachability — whether your request can even get to the resource’s front door in the first place.

Part 1: Identity and Access

The diagram below summarizes the steps we took to set up authentication and authorization:

Steps 1–4: Setting up the identity chain

Here’s the actual sequence of trust, in the order it happens:

1. A user-assigned managed identity (UAMI), attached to the Container App, lives entirely in tenant A.
2. An app registration, also in tenant A, is configured with a federated identity credential (FIC) that names the UAMI as a trusted token issuer.
3. Because the target resource lives in a different tenant, the app registration is made multitenant, then provisioned into tenant B via an admin-consent flow. That consent step creates a service principal — a local presence for the container app — inside tenant B.
4. The admin of tenant B grants that service principal a Role-based access control (RBAC) role (Storage Blob Data Contributor) directly on the storage account.

Step 3 basically means that the admin of tenant B needs to click on a link that looks like this:

https://login.microsoftonline.com/<tenantB>/adminconsent?client_id=<clientId>

In our case, the admin clicked on the link and got an error “No reply address is registered for the application” which could look like the setup did not work. However, it’s only a cosmetic issue: the /adminconsent endpoint tries to redirect the browser back to a reply URL (redirect URI) registered on the app after consent completes — and since the app registration has none configured, that final redirect step fails with a blank error page.

Let’s pause for a second and compare the different objects that were set up here:

FICs aren’t locked to one tenant by design — they trust external OIDC issuers like GitHub or Kubernetes just fine, verified via the issuer’s discovery document. The lock here is a specific Microsoft rule: when the issuer is a managed identity, the app registration and the managed identity must be in the same tenant.

Get Timo Babst’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

This means that trust between the UAMI and the app registration can only be established in tenant A. But authorization on the storage account can only be granted to something tenant B recognizes locally — a tenant B service principal. Since a single-tenant app registration can’t be authorized in a tenant it doesn’t exist in, the only way to bridge that gap is: make the app multitenant, and let it acquire a service principal in tenant B, via consent.

Steps 5–6: Hooking up our Job to the Identity Chain

Azure’s WorkloadIdentityTokenProvider (available from Hadoop 3.4.1's hadoop-azure module) reads a JWT from a local file and exchanges it with Entry ID for a storage access token. On AKS, that JWT file is populated automatically by a workload-identity webhook. Azure Container Apps has no such webhook — so step one was building the exchange ourselves.

At runtime, two token requests chain together:

1. We fetch a token from Azure’s local identity endpoint, scoped to the special audience “api://AzureADTokenExchange” — this is the UAMI proving its own identity.
2. Present that token as a client_assertion to the tenant B’s OAuth endpoint, requesting a token scoped to https://storage.azure.com — this is the app registration exchanging trust for an actual storage-access token.

Because Container Apps doesn’t expose a Kubernetes-style federated-token file the way AKS does, we wrote a small refresh loop that calls the identity endpoint directly and writes the resulting JWT to a file WorkloadIdentityTokenProvider can read. It looked similar to this:

refresh_token() {
  RESPONSE=$(curl -s -f "$IDENTITY_ENDPOINT?resource=api://AzureADTokenExchange&api-version=2019-08-01&client_id=$AZURE_MANAGED_IDENTITY_CLIENT_ID" \
    -H "X-IDENTITY-HEADER: $IDENTITY_HEADER") || { echo "Failed to fetch token" >&2; return 1; }

  ACCESS_TOKEN=$(echo "$RESPONSE" | jq -e -r '.access_token') || { echo "Failed to parse access_token from JSON" >&2; return 1; }

  printf '%s' "$ACCESS_TOKEN" > /tmp/token-file
}
refresh_token
( while true; do sleep 1500; refresh_token; done ) &

And the Spark/Hadoop side of the config, once all of the above exists:

"spark.hadoop.fs.azure.account.auth.type.<account>.dfs.core.windows.net" = "OAuth"
"spark.hadoop.fs.azure.account.oauth.provider.type.<account>.dfs.core.windows.net" = "org.apache.hadoop.fs.azurebfs.oauth2.WorkloadIdentityTokenProvider"
"spark.hadoop.fs.azure.account.oauth2.msi.tenant.<account>.dfs.core.windows.net" = "<tenant-B-id>"
"spark.hadoop.fs.azure.account.oauth2.client.id.<account>.dfs.core.windows.net" = "<app-registration-client-id>"
"spark.hadoop.fs.azure.account.oauth2.token.file.<account>.dfs.core.windows.net" = "/tmp/token-file"

One detail worth calling out because it’s easy to get backwards: the client_id in the identity-endpoint call is the managed identity's client ID; the client.id in the Hadoop config is the app registration's client ID. They are two different GUIDs doing two different jobs, and mixing them up produces confusing, silent-looking failures rather than a clean error.

Part 2: Network reachability

With identitity management handled, we still needed to make sure that the data is copied securely.

The tricky part: most of Azure storage’s narrower network options — resource instance rules, VNet service endpoint rules — are explicitly restricted to the same tenant as the storage account. Cross-tenant, they don’t apply. That leaves two real options:

• A NAT Gateway for a static outbound IP, allowlisted on the storage account’s firewall. Cheap, fast to set up, but the IP is a blunt instrument — anything behind that gateway can reach the storage account.
• A private endpoint, which removes the public IP from the equation entirely and routes traffic over Microsoft’s private backbone.

We went with the Private Endpoint. It’s the more involved option, but pairs naturally with the identity work already done — RBAC still governs who can act, the private endpoint governs whether the request can arrive at all.

The private endpoint, step by step

1. In tenant A, create the Private Endpoint resource, targeting the storage account by its full ARM resource ID which is unique across tenants.
2. Create a private DNS zone named exactly privatelink.dfs.core.windows.net, and link it to the VNet.
3. Attach a DNS zone group to the Private Endpoint so it auto-registers the A record.
4. The admin of tenant B approves the pending connection request under their storage account’s networking settings.
5. The admin of tenant B makes sure public network access on the storage account is disabled.

In our first attempt we had missed creating the private DNS Zone and requests started failing with a 403 not authorized — the exact same error text we’d seen during a RBAC misconfiguration earlier, which was misleading. The problem was that without setting up a private DNS Zone, the network traffic was still resolving to a public IP and bypassing the private endpoint. Once the zone existed, was linked to the VNet, and had a zone group attached to the endpoint, DNS resolved privately and the same request succeeded.

The lesson buried in that detour: Azure Blob Storage reuses the same generic “this request is not authorized” message for both RBAC failures and network-level denials. When that error resurfaces after something else was already working, check the network path before assuming your permissions regressed.

The mental model, distilled

If there’s one idea worth carrying away from all of this, it’s the separation of concerns:

• When working with managed identities, identity trust is local — a federated credential can only verify what it can cryptographically check, which means it only spans one tenant. Cross-tenant access isn’t one trust relationship; it’s two single-tenant ones, chained by provisioning an app into both places.
• Network reachability and authorization are independent locks on the same door. RBAC decides who’s allowed in; the network layer decides whether they can knock in the first place. A 403 doesn’t tell you which lock is engaged — you have to check both.

Get the mental model right, and the actual Azure configuration — for all the CLI flags and portal clicks along the way — turns out to be the easy part.