Understanding Data Lakehouses — Apache Hive

This is the beginning of a series of posts that will explore the concept of data lakehouses — a relatively new term in the data world. In upcoming posts, I’ll dive deeper into specific implementations of new-age data lakehouses, such as Delta Lakehouse and Apache Iceberg.

What is Apache Hive?

The textbook definition of Apache Hive is: “a distributed, fault-tolerant data warehouse system that enables analytics at a massive scale.” However, to break it down further, Hive itself doesn’t store any data. Instead, it manages the metadata of tables and interprets user queries based on this schema. The actual data is retrieved from storage systems like HDFS, S3, or ADLS Gen2 when queries are executed. Hive acts as the middle layer, connecting your queries to the underlying storage.

Why Apache Hive?

To manage massive volumes of data (commonly known as big data), HDFS was introduced for storage, and MapReduce for distributed data processing. However, the complexity and steep learning curve of MapReduce limited its widespread adoption. The introduction of Apache Hive marked a pivotal moment in the evolution of big data, as it simplified the process by providing an SQL-like interface. This made large-scale data processing much more accessible, allowing users to leverage Hadoop without needing to master the intricacies of MapReduce.

HQL query Execution

A hive table comprises two major components :

1. Actual Data — presented in Distributed storage.
2. Metadata — Schema or information of data stored in Metastore DB.

Any query executed is written in user-friendly Hive Query Language. Before execution, the table schema is verified using the meta store. The execution engine then translates the query into a sequence of MapReduce jobs, assuming that MapReduce is the chosen execution engine.

Apache Hive Architecture

Figure 1. Hive Architecture

The architecture of Hive, as shown in the figure, illustrates the query execution process with HDFS as the storage system (in its older version). The flow of query execution is well-defined, and here are some key components worth exploring in more detail:

1. Driver: The Driver is responsible for managing the lifecycle of a Hive session. It receives the user’s query, interacts with the compiler, and oversees the execution of the query.
2. Compiler: The Compiler translates the HiveQL query into an execution plan. It breaks down the query into manageable tasks and optimizes it for performance.
3. Execution Engine: The Execution Engine takes the execution plan generated by the Compiler and runs it. It submits the job to the underlying storage system (like HDFS) for processing.
4. Metastore: The Metastore stores metadata about Hive tables, including their schema (structure) and other information needed for query execution.

Additionally, while Hive traditionally used MapReduce as its execution engine, alternatives like Tez and Apache Spark have emerged to improve performance and efficiency. Each execution engine offers different advantages, making Hive more versatile in handling data processing tasks.

Get Pradeep Gontla’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Hive Data Model

1. Tables: In Hive, tables function similarly to those in relational databases. They are structured with rows and columns, and users can create, drop, and modify them using HiveQL commands.

2. Schemas: Hive utilizes DDL commands to define schemas, create tables, specify data types, and set various table properties.

3. Partitions: Partitions enable Hive to segment tables into smaller, more manageable sections. This segmentation facilitates faster querying of large datasets by allowing Hive to scan only the relevant partitions. Each partition corresponds to a directory in the HDFS file system (e.g., tables/partitions).

4. Buckets: Buckets provide an additional layer of organization within partitions by further dividing them into smaller segments based on the hash of a specific column. Each bucket is stored as a separate file within the partition directory.

Disadvantages of HIVE:

1. Limited Support for ACID (Atomicity, Consistency, Integrity, and Durability): Hive’s traditional architecture offers limited support for ACID transactions, making it less suitable for scenarios that require reliable and concurrent data updates. This limitation has been effectively addressed by technologies like Delta Lakehouse and Apache Iceberg, which enhance ACID compliance for big data processing. More details on this will be discussed in my upcoming posts.
2. Outdated Hive Execution Layer: The Hive compiler generates an execution plan, and tasks are executed sequentially. This approach can lead to additional overhead compared to the Catalyst optimizer utilized by Spark, which significantly enhances query performance through advanced optimization techniques. I will provide evidence of this in my next post.
3. Limited to Batch Processing: Hive was primarily designed for batch processing jobs, which restricts its effectiveness for real-time analytics or interactive querying. While improvements have been made with frameworks like Hive on Tez or Hive on Spark, its foundational design is not optimized for low-latency workloads.
4. No Time Travel Capabilities: Traditional Hive does not support time travel capabilities, which means users cannot easily query historical versions of data. This feature is critical in scenarios where tracking changes over time or reverting to previous states is necessary. In contrast, technologies like Delta Lakehouse and Apache Iceberg offer time travel functionality, allowing users to access and analyze past versions of their data effortlessly.

Conclusion

While Apache Hive has played a pivotal role in the evolution of big data processing, it is essential to acknowledge its limitations. The restricted support for ACID transactions, reliance on an outdated execution layer, focus on batch processing, and lack of time travel capabilities present challenges for modern data applications. As the landscape of big data continues to evolve, technologies like Delta Lakehouse and Apache Iceberg offer enhanced features, improved performance, and greater flexibility, addressing many of the shortcomings of Hive

In the upcoming post, I will discuss HIVE LLAP (Live Long and Process) and compare the query run times for Hive with MapReduce, Hive with Spark, and Spark as Execution Engine.

References :

1. https://cwiki.apache.org/confluence
2. Apache Hive Essentials by Dayong Du — Packt Publishing
3. https://medium.com/@vinciabhinav7/apache-hive-a-primer-c732d44038d8