BigQuery
Supports table formats like Apache Iceberg, Delta, and Apache Hudi.
Open table formats sit on top of raw storage files (like Parquet or ORC) to organize massive datasets into logical tables. They add a metadata layer that enables:
- ACID Transactions: Ensures data reliability by preventing partial writes or file corruption.
- Time Travel: Allows you to query older snapshots or historical versions of your data.
- Schema Evolution: Makes it safe to add, drop, or rename columns without breaking tables or rewriting all underlying data. [2, 4, 5]
Comparing the Three Formats
- Apache Iceberg
Design: Engine-agnostic, tree-based metadata layout.
Strengths: Excellent multi-engine compatibility (works seamlessly with Trino, Flink, Spark, and Snowflake) and advanced schema/partition evolution at scale.
Get Ronel Solomon’s stories in your inbox
Join Medium for free to get updates from this writer.
Best For: Multi-cloud, multi-engine analytics architectures.
- Delta Lake
Design: Transaction-log-based structure.
Strengths: Deep integration with the Apache Spark and Databricks ecosystem, highly optimized batch-and-stream processing.
Best For: Spark-native pipelines and organizations standardized on Databricks.
- Apache Hudi
Design: Timeline-based architecture with built-in record-level indexing.
Strengths: Exceptional performance for frequent record-level updates and deletes (upserts) and real-time incremental change data capture (CDC) pipelines via Merge-On-Read (MoR) storage.
Best For: Streaming-first architectures and high-mutation workloads.
BigQuery’s RPC API to transform and manage data. a storage layer.
ODBC and JDBC drivers provide interaction with existing applications, including third-party tools and utilities.
ODBC (Open Database Connectivity) and JDBC (Java Database Connectivity) are standard software interfaces (drivers) that allow application programs to connect and communicate with databases like BigQuery.
