How Snowflake Apache Iceberg table Fit in data mesh. PART 1

Basic

Apache Iceberg is an open-source data table format that was developed to address some of the limitations of Apache Hive and Apache Hadoop’s HDFS file system. Iceberg tables are designed to provide a scalable, portable, and performant way of storing and querying large-scale data sets.

Some of the key features of Iceberg tables include:

1. Schema evolution: Iceberg tables allow for schema evolution without requiring data migration or rewriting. This means that you can add or remove columns to your data set over time without having to reprocess your entire data set.
2. Data versioning: Iceberg tables provide versioning capabilities, allowing you to track changes to your data over time.
3. Query performance: Iceberg tables are designed to optimize query performance by using partitioning and column-based storage.
4. Portable: Iceberg tables are portable across different storage systems, such as Hadoop Distributed File System (HDFS), Amazon S3, and Azure Blob Storage.
5. Transactions: Iceberg tables support transactions, which enables multiple users to access the same data set simultaneously, ensuring data consistency and integrity.

Data Mesh

Get Danish shaikh’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

In the context of data mesh, this feature can be useful in managing the data lake for a domain. By separating the hot and cold data, the iceberg table can help optimize the data storage and query performance. The hot data can be stored in the main table while the cold data can be stored in the iceberg table.

This separation of data can also help with managing access control and permissions, as the cold data can be stored in a separate table with restricted access. This can help ensure that only the necessary users and services have access to the cold data, reducing the risk of data breaches or unauthorized access.

Iceberg Tables can help customers save costs in two ways:

1. Storage costs: Since Iceberg Tables use customer-supplied cloud storage, customers no longer need to pay Snowflake for storage costs. Instead, they are billed by their cloud provider for storage used. This can result in significant cost savings for customers with large data sets.
2. Query costs: Iceberg Tables use the Parquet file format, which is highly optimized for analytical queries. This means that queries on Iceberg Tables can be executed more efficiently, resulting in faster query performance and lower query costs. In addition, Iceberg Tables use column-level metadata to skip over irrelevant data during queries, further reducing query costs.

To create an iceberg table use the below sample SQL .

-- Create an External Volume to hold Parquet and Iceberg data
create or replace external volume volume_name
STORAGE_LOCATIONS = 
(
 (
  NAME = 'domain_id_cold_bucket'
  STORAGE_PROVIDER = 'S3'
  STORAGE_BASE_URL = 's3://domain_data/data/domain_id_cold_bucket/'
  STORAGE_AWS_ROLE_ARN = '***'
 )
); 

create or replace iceberg table my_iceberg_table
 with EXTERNAL_VOLUME = 'volume_name'
 as select * from sample_data;

In the Next part, I will add code on how to set up hot and cold tables and add row-level security on top of it.