Efficient Data Sharing with Databricks on AWS: A Decision Framework
Organizations running Databricks on AWS routinely need to exchange data across teams, accounts, and platforms. Imagine a scenario where your finance team wants real-time access to revenue data sitting in a core engineering bucket, while an external partner needs the same data for an analytics project.
Historically, this required complex ETL pipelines, manual SFTP setups, or brittle cross-account AWS IAM roles. Today, Unity Catalog and Delta Sharing provide native, secure alternatives.
But how do you choose the right approach? Use this definitive decision framework to match your business scenario with the correct architecture.
Produce — Which Pattern Shares Your Data Out?
When Databricks on AWS is sharing data out, the decision maps to one of the patterns
These patterns collapse into two shapes:
Bulk file access — Delta Sharing (in all its flavors), Delta UniForm + Iceberg REST, and shared S3 hand the other side the underlying Parquet/Delta files, read via short-lived pre-signed URLs. The provider spends zero compute.
Live query — JDBC/ODBC and the REST API run queries through a SQL Warehouse. The consumer always sees current data and needs no copy — but the provider pays the compute for every query.
There’s also a choice of protocol for the bulk-file shape when the consumer isn’t on Databricks. Delta Sharing is Databricks’ own protocol; Delta UniForm + the Unity Catalog Iceberg REST catalog publishes an Apache Iceberg metadata view over the same Parquet, so any Iceberg-native engine — Snowflake, Trino, Flink, Athena — reads it as a native Iceberg table. Reach for UniForm when the consumer wants open-table-format interop or you want one manifest to serve many engines; reach for Delta Sharing when they’re happy speaking the Delta Sharing protocol. Both are zero-copy and UC-governed.
Get Manish’s stories in your inbox
Join Medium for free to get updates from this writer.
This split yields two rules worth keeping front of mind:
Delta Sharing is read-only. The moment someone needs to write, you’re no longer sharing — you’re running shared storage with carefully scoped IAM roles.
JDBC/ODBC and the REST API bill the producer, not the consumer. They run on your SQL Warehouse, so use Serverless with auto-stop and watch the compute. That’s the price of zero-copy, always-fresh access.
Consumer patterns
When your Databricks on AWS lakehouse is the consumer, you have options. In nearly all cases, reading the data in place is preferable to copying it in.
Federation comes in two flavors, and it’s worth knowing which you’re using. Lakehouse Federation connects to a query engine (Snowflake, Postgres, Redshift, BigQuery) over a JDBC-style CONNECTION and pushes queries down to it. Catalog federation connects to a metastore — most commonly an AWS Glue Data Catalog or Hive Metastore — and surfaces every table it has registered under a single UC foreign catalog. The Glue case is the common one on AWS: a “unified layer” where some tables are native UC Delta and others are Glue-registered S3 tables, all readable through the same catalog.schema.table namespace with governance enforced identically. The consumer never needs to know which physical reality backs a given table.
The consumer’s default should be “query in place,” not “copy in.” D2D shares, Lakehouse Federation, and external-location reads all leave the data where it lives. Ingestion is the exception you justify — not the reflex you start with.
Architectural & Financial Considerations on AWS
Before finalizing your architecture, keep these two critical AWS infra rules in mind:
- Data Egress Fees: While Delta Sharing eliminates storage duplication, reading data out of your AWS region still incurs standard AWS network egress costs. If a recipient pulls terabytes of data to another cloud or region, plan for network charges.
- Governance & Auditing: Every query run by an external recipient is fully logged in the Unity Catalog audit logs. You maintain full lineage and can revoke access instantly by dropping the recipient from the share.
For more information, see Delta Sharing best practices, Lakehouse Federation, and column masks and row filters in Unity Catalog in the Databricks documentation, and the AWS Well-Architected Framework
