Azure Databricks Data Sharing and Partner Ecosystem : Delta Sharing, Marketplace, Clean Rooms, Power BI, Tableau, and dbt
Introduction to Governed Data Collaboration
Hello and welcome. Modern organizations rarely create value from data in complete isolation. They exchange information with business units, customers, suppliers, research institutions, regulators, analytics platforms, and technology partners. The challenge is that traditional sharing often requires copying files, building custom interfaces, exchanging credentials, and maintaining duplicate datasets. Every copy increases operational cost, delays access, and creates another place where sensitive information can be exposed or become outdated. Azure Databricks addresses this challenge through an ecosystem built around Unity Catalog, Delta Sharing, Databricks Marketplace, Clean Rooms, Partner Connect, business intelligence tools, ingestion platforms, transformation services, and governance solutions.
Internal Access and External Sharing
Data sharing is different from ordinary data access. Internal access usually occurs among users and workloads governed within the same organizational boundary. External sharing crosses organizational, account, cloud, or technology boundaries. A data engineering team might share trusted sales data with an internal finance group, while a pharmaceutical company might collaborate with an external research organization. A retailer might distribute product intelligence to suppliers, and a data provider might commercialize curated datasets through a marketplace. Each scenario requires a deliberate combination of identity, permissions, governance, auditability, performance, cost control, and contractual rules.
Four Patterns for Data Collaboration
A useful sharing strategy begins by separating four patterns. Internal sharing grants governed access to people and workloads within the enterprise. Direct external sharing delivers selected assets to known recipients. Marketplace distribution packages data products for broad or targeted discovery. Clean-room collaboration allows multiple parties to perform approved analysis without freely exposing their underlying data. These patterns can work together, but they solve different problems. Choosing the correct pattern prevents organizations from treating every collaboration as a file transfer or creating an unnecessarily complex solution for a straightforward access requirement.
Unity Catalog as the Governance Foundation
Unity Catalog provides the governance foundation for many of these capabilities. It organizes data and artificial intelligence assets into governed objects such as catalogs, schemas, tables, views, volumes, functions, and models. Permissions can be assigned to users, groups, and service principals, while auditing and lineage help administrators understand how assets are accessed and used. When data sharing is integrated with this governance layer, teams can manage the original asset and its sharing relationship through a consistent control model instead of building a disconnected security system for every partner.
Copy-Free Sharing Within a Metastore
Internal sharing is simplest when participating workspaces are attached to the same Unity Catalog metastore. Administrators can grant access to governed assets without creating another physical copy or configuring a Delta Sharing recipient. A catalog can contain approved business data, while schemas and object-level permissions separate domains and use cases. Views can present only the columns or records required by a particular audience. This approach supports centralized governance while allowing multiple engineering, analytics, data science, and business teams to work from consistent information.
Disciplined Internal Access Design
Internal sharing still requires disciplined design. Giving every user broad access to raw data may be convenient initially, but it weakens accountability and makes future policy changes difficult. A better model uses groups that reflect job responsibilities, grants access at the appropriate catalog or schema level, and reserves direct user permissions for exceptional cases. Curated views or data products can hide technical complexity and sensitive attributes. Owners should document the intended purpose, refresh behavior, quality expectations, support contact, and permitted uses of each important asset.
Cross-Boundary Sharing with Delta Sharing
When data must cross metastore, account, cloud, or platform boundaries, Delta Sharing becomes a central option. Delta Sharing is an open protocol for securely sharing live data without requiring recipients to use the same computing platform. Instead of routinely exporting a complete dataset and delivering it through another storage system, the provider defines what can be accessed and authorizes specific recipients. The recipient then reads the shared data using a compatible client. Updates made by the provider can become available to authorized recipients without operating a separate recurring file-distribution pipeline.
Providers, Shares, and Recipients
Delta Sharing is built around three important concepts. A provider is the organization or entity that makes assets available. A share is a read-only collection of assets selected for distribution. A recipient represents the organization or party allowed to access one or more shares. In Unity Catalog, shares and recipients are governed securable objects. This separation allows providers to reuse a share for several recipients or give one recipient access to multiple shares. Access can also be revoked without redesigning the source tables or deleting the provider’s authoritative data.
Databricks-to-Databricks Sharing
Azure Databricks supports Databricks-to-Databricks sharing for recipients that use a Unity Catalog-enabled Databricks environment. The recipient provides a sharing identifier, and the provider uses it to establish the relationship. Databricks manages the secure connection without requiring the provider to distribute a long-lived bearer token. Depending on current feature support, this model can share governed tables, views, streaming tables, materialized views, volumes, models, and notebook files. It is especially useful for collaboration across Databricks accounts, cloud environments, or separate Unity Catalog metastores.
Open Delta Sharing and Federated Identity
Open Delta Sharing addresses recipients that do not use Databricks or do not have an appropriate Unity Catalog-enabled workspace. Compatible tools can include Apache Spark, pandas, Power BI, and other clients implementing the protocol. Azure Databricks supports token-based access and OpenID Connect federation for eligible open-sharing scenarios. Long-lived bearer tokens require careful storage, rotation, expiration, secure delivery, and revocation. Identity federation can reduce dependence on static credentials by exchanging trusted identity-provider assertions for short-lived access tokens, although organizations must confirm that their recipient tools support the required flow.
Provider Setup and Production Validation
A typical provider workflow begins by confirming that the source assets are governed appropriately and that Delta Sharing is enabled where required. The provider creates a share, adds approved assets, creates or registers a recipient, and grants the recipient access to the share. Open-sharing recipients receive the information needed to activate or connect to the share. Databricks-to-Databricks recipients can see the shared assets after the relationship and permissions are established. Before production use, the provider should test the experience with a controlled recipient and verify data visibility, performance, auditing, and revocation.
Recipient Access and Downstream Use
The recipient experience depends on the sharing method. A Databricks recipient creates or receives a catalog that represents the provider’s shared data and then grants local users permission to access the relevant objects. An open-sharing recipient uses a credential profile or federated connection with a supported client. Shared assets are generally read-only from the recipient’s perspective. If recipients need to enrich, transform, or permanently retain information, they can create authorized downstream copies in their own environment, subject to the provider’s terms, technical controls, and applicable data-use agreement.
The Real Costs Behind Zero-Copy Sharing
Delta Sharing is frequently described as zero-copy sharing because it can avoid creating a complete duplicate dataset simply to begin collaboration. However, zero-copy should not be interpreted as zero processing, zero network traffic, or zero cost. Queries still consume compute somewhere, data must still travel to the recipient, and cross-region or cross-cloud access may create network egress charges. Some shared views or derived assets may require provider-side filtering or materialization depending on the recipient environment. Architecture teams should validate compute attribution, storage behavior, data residency, and network costs before scaling a sharing program.
Share-Specific Filtering with Views
Providers can use views to control the information exposed through a share. A base table might contain every customer and attribute, while a shared view returns only the approved columns and rows. Dynamic views can tailor results using recipient properties in supported Databricks-to-Databricks scenarios. This can support regional segmentation, partner-specific records, or removal of confidential attributes. Designers should confirm current limitations because row filters and column masks applied directly to source tables are not automatically equivalent to share-specific filtering, and support varies by asset type and sharing protocol.
Streaming Shared Data Across Boundaries
Streaming extends sharing beyond periodic analytics. Delta Sharing can support shared tables as sources for incremental Structured Streaming workloads when the required provider and recipient capabilities are available. A recipient might continuously process updated supply-chain events, telemetry, or reference data rather than downloading a complete snapshot on every cycle. Providers can also share table history for compatible use cases, enabling change data feed, streaming, or time-travel patterns where supported. Teams must align retention, checkpointing, schema evolution, latency expectations, and failure recovery across organizational boundaries.
Monitoring and Lifecycle Governance for Shares
Governance does not end when a share is created. Providers should monitor who creates shares, changes recipients, adds assets, removes assets, retrieves credentials, and queries shared data. Azure Databricks audit logs and system tables can help examine sharing activity and usage. Recipients can monitor how their own users access shared catalogs. These technical records should connect to a broader operational process containing ownership reviews, periodic access recertification, token expiration, incident response, data classification, contractual restrictions, and a documented method for terminating a sharing relationship.
Marketplace as a Data Product Channel
Databricks Marketplace builds on Delta Sharing to support the discovery and distribution of data and related products. A provider can package an offering as a listing that explains what the product contains, who it serves, how frequently it changes, and how a consumer can obtain access. Products may include datasets and other supported assets, depending on the listing and platform capabilities. Marketplace turns sharing from a purely technical configuration into a product experience with discovery, documentation, ownership, access workflows, and provider-to-consumer relationships.
Building Trustworthy Marketplace Listings
Marketplace providers should treat listings as maintained products rather than static advertisements. A strong listing uses a clear title, business description, sample information, geographic and temporal coverage, refresh frequency, data dictionary, limitations, and terms of use. It identifies the responsible owner and explains how consumers can request assistance. Providers should publish only assets that have passed quality, security, privacy, licensing, and legal review. A technically accessible dataset can still create risk if its provenance, permitted uses, accuracy limitations, or support commitments are unclear.
Evaluating Marketplace Products
Consumers should evaluate Marketplace products with the same care used for any external dependency. They need to understand who produced the data, how it was collected, how often it is refreshed, what entities and time periods it covers, and whether it can be used for the planned purpose. They should inspect schema stability, missing values, identifiers, geographic restrictions, licensing terms, and support expectations. Before integrating a product into critical reporting or machine learning, consumers should create validation tests and design for the possibility that a provider changes, delays, or withdraws the offering.
Public Listings and Private Exchanges
Public and private distribution are not the same. A broadly discoverable Marketplace listing can help providers reach many potential consumers. A private exchange creates a controlled marketplace experience for an invited group of consumers. An organization might use a private exchange to distribute governed products to subsidiaries, strategic suppliers, clinical research partners, or selected customers. This offers the product-oriented discovery experience of a marketplace while limiting who can view or request the listings. The provider still needs recipient governance, entitlement management, monitoring, and a process for removing organizations that should no longer participate.
Standardizing Partner Distribution Through Private Exchanges
Private exchanges are useful when an organization has several products or many partner relationships. Without a private exchange, every data owner may invent a different onboarding process, naming convention, support model, and access method. A curated exchange can standardize product descriptions, approval workflows, legal conditions, contacts, and consumer expectations. It should not become an uncontrolled catalog containing every experimental table. A review board or federated governance process can establish minimum requirements while allowing domain teams to remain responsible for the quality and operation of their products.
Get Chris Kaspar’s stories in your inbox
Join Medium for free to get updates from this writer.
Controlled Collaboration with Clean Rooms
Clean Rooms address collaborations where parties need joint insight but should not receive unrestricted access to each other’s underlying records. Databricks Clean Rooms use a privacy-protecting environment in which collaborators make approved assets and code available for defined analyses. The goal is controlled computation rather than general data exchange. Potential applications include privacy-conscious audience measurement, advertising analysis, fraud intelligence, research collaboration, supplier analysis, and healthcare or life-sciences studies. A clean room reduces unnecessary disclosure, but it does not remove the need for legal, ethical, privacy, and statistical review.
Defining Clean Room Questions, Assets, and Outputs
A clean-room collaboration begins with clearly defined questions and permitted outputs. One party creates the clean room and invites collaborators. Each participant contributes approved tables or other supported assets and controls what can be used. Approved notebooks or analysis logic perform the collaboration, and output tables capture authorized results. Participants should understand which party can run each notebook, which data it can access, what outputs it can create, and who can retrieve those outputs. These decisions should be established before sensitive data is added to the collaboration.
Preventing Inference and Minimizing Data
Privacy protection requires more than hiding source tables. Analysts can sometimes infer confidential information from small groups, repeated queries, excessive dimensions, or the combination of outputs with outside datasets. A mature clean-room design may require minimum aggregation thresholds, suppression rules, approved functions, query review, limited output schemas, differential privacy techniques where appropriate, and restrictions on repeated analysis. Data minimization remains essential. If an attribute is not required for the approved purpose, it should not enter the clean room simply because the platform can technically govern it.
Clean Rooms, Masking, and De-Identification
Clean rooms, data masking, and de-identification are related but distinct. Masking changes how a value appears to a user. De-identification attempts to reduce the connection between records and individuals, but re-identification risk can remain. A clean room constrains the collaboration environment and the analyses that can occur. Depending on the use case, organizations may need all three approaches as well as encryption, identity controls, contractual safeguards, and output review. Highly regulated collaborations should involve privacy, security, compliance, legal, and domain experts rather than relying on platform administrators alone.
Connecting Power BI to Databricks
Business intelligence tools connect governed lakehouse data to business users. Power BI can connect to Azure Databricks through the supported Databricks connector and SQL warehouses. Users can establish connections from Power BI Desktop, and Partner Connect can simplify parts of the initial setup. Authentication options and deployment requirements depend on the selected connection mode and environment. Organizations should prefer modern identity-based authentication and service principals for automated machine-to-machine scenarios rather than distributing personal credentials or embedding individual user tokens in shared reporting solutions.
Import, DirectQuery, and Composite Models
Power BI architects must choose deliberately between imported and direct-query patterns. Importing data into the Power BI semantic model can provide responsive user interaction and reduce repeated warehouse queries, but it introduces another persisted copy and requires refresh management. DirectQuery can keep queries closer to the governed source and support fresher data, but dashboard performance depends on generated SQL, network latency, model design, concurrency, and SQL warehouse capacity. Composite models can combine patterns. The correct decision depends on freshness, volume, security, user experience, and cost requirements.
Governing and Optimizing Power BI
Effective Power BI integration requires more than connecting a table. Teams should expose curated schemas, stable views, or well-designed data products rather than asking every report author to interpret raw operational structures. Unity Catalog permissions should align with the intended security model, and Power BI semantic-model permissions should not be mistaken for source governance. Query history and profiles can reveal expensive or inefficient patterns. Aggregation, selective columns, appropriate warehouse sizing, serverless SQL where suitable, caching, and well-designed semantic models can improve performance without sacrificing governance.
Connecting Tableau to Databricks
Tableau can connect to Azure Databricks using its native Databricks connector, supported drivers, and SQL warehouse connection details. Partner Connect can generate or simplify connection configuration for supported experiences. Tableau Cloud can also explore eligible Unity Catalog tables or schemas through integrated workflows. Authentication choices include user-based identity and supported OAuth or service-principal patterns, subject to product versions and deployment architecture. Organizations should verify current compatibility requirements before standardizing because connector, driver, Tableau, and authentication capabilities continue to evolve.
Optimizing Tableau and Databricks SQL
Tableau performance depends on the interaction between workbook design and the Databricks SQL layer. A dashboard containing many worksheets and interactive filters can generate numerous concurrent queries. Returning excessive detail can also overwhelm visualization processing even when the warehouse produces results quickly. Designers should aggregate at useful business levels, reduce unnecessary marks, limit redundant filters, use actions for guided drill-down, and examine Tableau performance recordings alongside Databricks query profiles. Caching and extracts can help, but they introduce freshness and governance considerations that must be intentionally managed.
dbt Analytics Engineering on Databricks
dbt provides a transformation and analytics-engineering workflow on top of Databricks. Teams can define SQL models, tests, documentation, dependencies, and deployment practices while Databricks performs the underlying execution. dbt Cloud can connect through Partner Connect, and dbt Core can use the appropriate Databricks adapter and authentication configuration. Unity Catalog gives dbt projects governed locations for their source data and generated models. Service principals and OAuth are preferable for production automation because build pipelines should not depend on a developer’s personal identity.
Environment Separation, Testing, and Lineage with dbt
A dbt integration should clearly separate development, testing, and production. Developers may build isolated models in dedicated schemas, while controlled jobs promote validated definitions into production catalogs and schemas. Source freshness tests, uniqueness tests, relationship tests, and business-rule tests can detect important failures, although they do not replace comprehensive data-quality engineering. Model contracts and versioning can protect downstream consumers from unexpected changes. Lineage from transformation tools should complement Unity Catalog lineage and operational metadata, giving teams a more complete view of how business data is produced.
Choosing Ingestion Partners
Ingestion partners bring operational, software-as-a-service, database, event, and file data into the lakehouse. Azure Databricks technology integrations include solutions such as Fivetran, Informatica, Matillion, Rivery, RudderStack, Snowplow, and other supported partners, although the available Partner Connect catalog changes over time. Managed ingestion can reduce connector maintenance and handle source-specific details such as incremental extraction, schema drift, retries, and API limits. Teams should compare these solutions with Lakeflow Connect and custom pipelines based on source coverage, latency, scale, control, cost, and operational ownership.
Securing and Operating Ingestion Connectors
Connecting an ingestion tool is only the beginning. Administrators must determine what network path it uses, where credentials are stored, which catalog and schema it can write to, whether it supports Unity Catalog, and how failures are reported. A connector should usually write through a service identity with narrowly scoped privileges. Raw ingestion zones should remain distinct from trusted products, and schema drift should trigger controlled handling rather than silently changing production reports. Usage, compute, storage, and vendor charges should be observed together because convenience can conceal rapidly growing cost.
Evaluating Governance and Observability Partners
Governance partners add capabilities such as data quality monitoring, observability, modeling, security, discovery, and policy integration. Current Azure Databricks integrations can include platforms in categories such as data governance, security, semantic layers, and observability. These tools may read metadata, query system tables, monitor pipelines, analyze lineage, or enforce complementary controls. Integration teams should understand exactly what permissions a partner needs and whether it accesses metadata, sample values, full records, or credentials. Third-party governance software must itself be governed, monitored, and included in vendor-risk assessments.
Designing a Coherent Solution Ecosystem
Solution integrations extend beyond ingestion, transformation, and business intelligence. Machine learning tools, reverse ETL platforms, semantic layers, customer applications, and industry solutions can connect to Databricks through drivers, SQL interfaces, REST APIs, open protocols, or validated partner integrations. Reverse ETL tools can activate curated data in customer engagement or operational platforms. Semantic-layer technologies can standardize business metrics. Machine learning partners can support specialized development workflows. The architectural objective is not to collect as many tools as possible, but to create a coherent ecosystem with clear ownership and minimal duplication.
Production Controls for Partner Connect
Partner Connect accelerates configuration for supported products by guiding users through connection setup and, in some cases, creating required Databricks resources or permissions. This can shorten an evaluation, but it should not bypass enterprise architecture and security review. Administrators should inspect every created service principal, token, SQL warehouse, catalog permission, network connection, and vendor-side resource. Temporary evaluation objects should be removed when trials end. Production connections should use approved naming, tagging, credential rotation, ownership, monitoring, and disaster-recovery practices.
Least-Privilege Partner Architecture
A secure partner architecture follows least privilege and separation of duties. Human users authenticate through enterprise identity, while automated integrations use dedicated service principals or workload identities where supported. Each integration receives only the catalogs, schemas, tables, volumes, shares, or SQL warehouses required for its purpose. Secrets should be held in approved secret-management systems and never placed in notebooks, source repositories, or dashboard files. Network controls, private connectivity, firewall rules, encryption, audit logs, and incident-response procedures should reflect the sensitivity of the data and the trust level of the partner.
End-to-End Integration Lifecycle Governance
Organizations also need lifecycle governance. Every share, marketplace listing, clean room, connector, and partner integration should have a business owner, technical owner, purpose, data classification, approval date, review interval, support process, and retirement plan. Changes to schemas, identifiers, refresh schedules, or quality rules should be communicated to consumers. Contracts should address permitted use, retention, redistribution, service expectations, breach notification, and deletion obligations. When a relationship ends, administrators must revoke identities, remove recipient access, expire tokens, validate downstream obligations, and preserve appropriate audit evidence.
Healthcare Research Collaboration Pattern
Consider a healthcare research collaboration as a practical example. A healthcare organization could expose approved, de-identified population measures to an internal analytics team through Unity Catalog. It could share non-sensitive reference data directly with an external research partner through Delta Sharing. A private exchange could distribute curated products to an approved consortium. A clean room could support controlled cohort analysis without unrestricted disclosure of participant-level data. Power BI or Tableau could communicate authorized aggregate results, while dbt and ingestion partners maintain tested pipelines. Every layer would still require privacy, legal, ethical, and security oversight.
Retail Partner Ecosystem Pattern
A retail ecosystem offers another example. The retailer could ingest sales, inventory, advertising, and supplier data through managed connectors. dbt could transform those sources into certified product and demand models. Power BI and Tableau could serve internal decision-makers. Delta Sharing could provide selected performance data to individual suppliers, with recipient-specific views limiting each supplier to its own products. A private exchange could organize multiple supplier-facing products. A clean room could support joint advertising measurement while restricting access to customer-level records. Governance tools could monitor quality, lineage, access, and operational reliability across the ecosystem.
Selecting the Right Collaboration Pattern
The most important architectural decision is choosing between access, sharing, replication, federation, and controlled collaboration. Use ordinary Unity Catalog permissions when participants operate under the same metastore and governance boundary. Use Delta Sharing when known recipients need governed, read-only access across boundaries. Use Marketplace when products need discovery and repeatable distribution. Use a private exchange when discovery should be limited to invited consumers. Use Clean Rooms when approved computation is safer than broad data exposure. Use ingestion when the consumer genuinely needs an independent operational copy or the source interface is better suited to managed replication.
Starting with a Low-Risk Data Product
Start implementation with a small, valuable, and low-risk data product. Identify its owner and consumers, classify the data, document the intended use, and select the correct distribution pattern. Build a curated interface rather than exposing raw complexity. Test authentication, authorization, revocation, performance, cost, monitoring, schema evolution, and consumer onboarding. Add automated checks and an operational support process. Once the model works, convert the experience into a reusable blueprint for other domains. Standardization should cover controls and lifecycle expectations while still allowing product teams to choose technologies appropriate to their workloads.
Unifying Data Sharing as a Product Strategy
Azure Databricks data sharing is ultimately about creating trusted connections between producers and consumers. Delta Sharing provides an open foundation for cross-platform access. Marketplace brings discovery and product distribution. Private exchanges create curated ecosystems. Clean Rooms support controlled multi-party analysis. Power BI and Tableau bring governed information to business users. dbt, ingestion partners, governance platforms, and solution integrations help build a wider operational ecosystem. The greatest value appears when these capabilities are managed as one data-product strategy rather than a collection of isolated connectors. Thank you for watching, and I hope this gives you a practical foundation for designing secure and scalable data collaboration.
