<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Steve Voros on Medium]]></title>
        <description><![CDATA[Stories by Steve Voros on Medium]]></description>
        <link>https://medium.com/@stevejvoros?source=rss-e64442c7deb7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*YTTwEAdjDUqnieAr</url>
            <title>Stories by Steve Voros on Medium</title>
            <link>https://medium.com/@stevejvoros?source=rss-e64442c7deb7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 17:05:30 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@stevejvoros/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Liquid Clustering: An Innovative Approach to Data Layout in Delta Lake]]></title>
            <link>https://medium.com/@stevejvoros/liquid-clustering-an-innovative-approach-to-data-layout-in-delta-lake-1a277f57af99?source=rss-e64442c7deb7------2</link>
            <guid isPermaLink="false">https://medium.com/p/1a277f57af99</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[delta-lake]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[clustering]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Steve Voros]]></dc:creator>
            <pubDate>Wed, 20 Sep 2023 20:03:08 GMT</pubDate>
            <atom:updated>2023-09-27T18:20:57.182Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>Announced at the 2023 Data + AI Summit [<a href="https://www.databricks.com/blog/announcing-delta-lake-30-new-universal-format-and-liquid-clustering">1</a>], Delta Lake liquid clustering introduces an innovative optimization technique aimed at streamlining data layout in Delta Lake tables. Its primary goal is to enhance the efficiency of read and write operations while minimizing the need for tuning and data management overhead. Liquid clustering is specifically designed to address the challenges posed by Hive-style partitioning and Z-ordering. It provides a highly adaptive solution, particularly in the context of evolving data patterns, scaling demands, and data skew complexities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q7T3A_GDswARU0-RfdZCZw.png" /></figure><p>In this article, we will compare partitioning, Z-ordering, and liquid clustering using a practical dataset example. We will examine the inner workings, strengths, and weaknesses of traditional data-layout methods before contrasting them with the benefits of liquid clustering. Additionally, we will provide examples to guide the reader through the implementation of liquid clustering.</p><h3>Background</h3><h4>Example Dataset</h4><p>To better illustrate our discussion, we will refer to the following example of a transaction dataset throughout this write-up:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/1*ftwCGwGhQU7eIGkGzBnwUA.png" /><figcaption>Example Transaction Dataset</figcaption></figure><h4>What is a Data Layout? Why is It Important?</h4><p>Data layout is the way data is stored in memory or on disk. It is the physical arrangement of the data, which can have a great impact on the performance of the queries and data operations.</p><p>A data layout is considered balanced when the data is evenly distributed across the storage medium. In the context of Apache Parquet and Delta Lake [<a href="https://www.databricks.com/glossary/what-is-parquet">2</a>][<a href="https://docs.databricks.com/en/delta/index.html">3</a>], the goal of optimizing a data layout is to arrive at a uniform file size and an appropriate number of files across a given dataset. A balanced data layout is essential for efficient data skipping during query execution. Data skipping ensures irrelevant data files are ignored and not read into memory for query processing. Data skew and other inconsistencies can lead to suboptimal data skipping, increased execution times, and potential failures in execution.</p><h4>Hive-style Partitioning</h4><p>Hive-style partitioning is a common way to improve the performance of queries on large datasets stored in data lakes [<a href="https://docs.databricks.com/en/sql/language-manual/sql-ref-partition.html">4</a>]. It involves dividing the data into smaller partitions, i.e., sets of files, where the partition information is stored as part of each file’s path (see example below). This technique can help speed up queries significantly by enabling data skipping during scanning.</p><pre>/transactions/month=2023-1/country=Germany/{1.parquet, 2.parquet,...}<br>/transactions/month=2023-2/country=USA/{30.parquet, 31.parquet,...}<br>/transactions/month=2023-3/country=Mexico/{53.parquet, 54.parquet,...}<br>...</pre><p>Partitioning also comes with a few drawbacks. Storing the partition information as part of the file path introduces physical boundaries between partitions and makes partitioning overall a less flexible, static layout technique. As a result, it can be difficult to manage and update partitions, where a complete rewrite of the data may be necessary.</p><p>Partitioning can also lead to data skew, meaning some partitions are much larger than others. Furthermore, partitioning can be effectively applied on low-cardinality fields, i.e., columns with a low number of distinct values (<em>country</em> and <em>month</em> in our example) [<a href="https://docs.databricks.com/en/tables/partitions.html">5</a>]. Partitioning by high cardinality columns (<em>transactionId, transactionTime, senderId/recipientId</em>, etc.) would create many small files that cannot be combined, resulting in poor scan performance.</p><p>The following examples demonstrate some of the typical challenges that can arise with partitioning:</p><p>1. Some partitions may have small data or no data at all.</p><p>2. Other partitions may have multiple, large files that should ideally be broken down further.</p><p>3. Yet another group of partitions may each have one smaller file that should be ideally merged across these partitions.</p><p>4. Data may get ingested on a frequent basis, resulting in small, fragmented data files. A background maintenance job needs to be scheduled on a regular basis to optimize these files.</p><p>The diagram below illustrates these scenarios. Please note that the number of files is simplified for demonstration and may not be truly representative of an actual transaction dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u1w9IsT3qmZqnk9NghQ-fA.png" /><figcaption>Partitioning (Example: Unbalanced/Unoptimized Transaction Dataset)</figcaption></figure><h4>Z-ordering</h4><p>Z-ordering is a data-layout optimization technique that can be used to cluster data within partitions for improved query performance (note that using partitions is not a prerequisite though) [<a href="https://docs.databricks.com/en/delta/data-skipping.html">6</a>]. Z-ordering is recommended to be used on high-cardinality columns. Thus, it is complementary to partitioning. Z-ordering co-locates related data in the same set of files and this co-locality is in turn leveraged to further improve data skipping during scanning.</p><p>In order to further explore Z-ordering, we should first review the building block of this clustering method. A Z-cube is a collection of clustered files produced as a result of Z-ordering. Once a Z-cube reaches a certain threshold in size, the cube is considered stable (sealed). This means that a stable Z-cube’s data files are not considered for clustering when the dataset is Z-ordered next time. Furthermore, an unstable Z-cube gets combined and repeatedly re-clustered with newly ingested data files until the Z-cube reaches a stable state.</p><p>Although Z-ordering is a highly effective data-clustering technique, the above attributes lead to a few implications:</p><ol><li>Z-ordering is not idempotent; nevertheless, it aims to be an incremental operation [<a href="https://docs.databricks.com/en/delta/data-skipping.html">6</a>]. Consequently, the time it takes to Z-order is not guaranteed to lessen over multiple runs.</li><li>Z-ordering clusters data within partitions and not across partition boundaries.</li><li>Stable Z-cubes may still need to be opened up for future clustering when DML operations (merge, update, delete) are performed on existing records.</li><li>Query patterns may change over time, requiring a new set of clustering (and partitioning) keys. Applying a new set of keys can be compute-intensive for large datasets.</li></ol><p>While the maintenance time of large tables can be significantly reduced by Z-ordering targeted partitions, data management overhead can increase in parallel:</p><pre>-- Example of Z-ordering in targeted partitions<br>OPTIMIZE transactionsTable<br>WHERE month &gt;= to_date(&#39;2023-02-01&#39;)<br>ZORDER BY (senderId)</pre><p>Revisiting our example of the transaction table, file sizes are optimized and data is co-located by <em>senderId</em> within partitions after performing Z-ordering:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mhKMq2fDDpqHQjxenzEZBQ.png" /><figcaption>Partitioning plus Z-ordering (Example: Optimized Transaction Dataset)</figcaption></figure><p>The combination of partitioning and Z-ordering can overall still be considered a static layout technique. We will see how liquid clustering can overcome the above limitations next.</p><h3>How is Liquid Clustering Different?</h3><p>Unlike Hive partitioning and Z-ordering, liquid clustering keys can be chosen purely based on the query predicate pattern, with no worry about future changes, cardinality, key order, file size, and potential data skew. Liquid clustering is flexible and adaptive (hence, its name), meaning that</p><ol><li>Clustering keys can be changed without necessarily rebuilding the entire table</li><li>It eliminates the concept of partitions and can dynamically merge or further divide files in order to arrive at a balanced dataset with the ideal number and size of files.</li></ol><p>Liquid clustering accomplishes the above by leveraging a tree-based algorithm to incrementally map the data layout and maintain the associated metadata as part of the table’s Delta Lake logs. Consequently, liquid clustering is stateful and does not need to be recomputed each time an OPTIMIZE command is executed.</p><p>This property also makes a truly incremental clustering possible, meaning that newly ingested data gets clustered as necessary — ignoring previously clustered data. Operating on larger blocks compared to Z-ordering also makes table maintenance significantly more efficient. Moreover, cluster-on-write (also called eager clustering) becomes possible following this approach, with clustering performed as part of the ingestion.</p><p>Eliminating the need for physical partition boundaries and dynamically merging/dividing data files can dramatically boost data skipping. Persisting the metadata also allows for skew detection and correction as well as better concurrency support at the record rather than partition level.</p><p>Revisiting our earlier example of the transaction dataset, liquid clustering provides a much higher degree of flexibility across previous partition boundaries, achieving the desired file size in a more consistent way:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/943/1*Lr4lfzNGILCxrP1XWf7UYw.png" /><figcaption>Liquid Clustering (Example: Balanced Transaction Dataset)</figcaption></figure><p>Databricks recommends liquid clustering for all new Delta tables [<a href="https://docs.databricks.com/en/delta/clustering.html">7</a>]. Based on the above, it can be particularly beneficial for the following scenarios:</p><ul><li>Tables are often filtered by columns with different cardinalities</li><li>Tables have a significant skew in data distribution</li><li>Tables grow quickly and require maintenance and tuning effort</li><li>Tables have concurrent write requirements</li><li>Tables have access patterns that change over time</li><li>Tables where a typical partition key could leave the table with too many or too few partitions.</li></ul><h3>Benefits</h3><p>To recap, liquid clustering offers a number of benefits over Hive-style partitioning and Z-ordering:</p><ul><li><strong>Improved performance</strong>: Liquid clustering can significantly improve the performance of read and write operations on Delta Lake tables. It also provides better concurrency support and is more efficient at data maintenance due to its incremental approach [<a href="https://medium.com/closer-consulting/liquid-clustering-first-impressions-113e2517b251">8</a>].</li><li><strong>Easier management</strong>: Liquid clustering is easier to manage and update than partitioning or Z-ordering (see next section for examples). It supports the evolution of clustering keys and reduces the required maintenance on tables.</li><li><strong>Reduced data skew and fragmentation</strong>: Liquid clustering can minimize data skew and fragmentation, which can significantly improve query performance.</li></ul><h3>How to Use It?</h3><p>To use liquid clustering [<a href="https://docs.databricks.com/en/delta/clustering.html">7</a>], you need to create a Delta Lake table with the CLUSTER BY phrase specifying the clustering columns:</p><pre>CREATE TABLE transactions (<br>  id BIGINT,<br>  country STRING,<br>  month DATE,<br>  transactionTime TIMESTAMP,<br>  senderId INT,<br>  recipientId INT,<br>  amount DECIMAL,<br>  note STRING<br>)<br>CLUSTER BY (country, month)</pre><p>Ingesting data to the table (for example, using CTAS, INSERT or COPY INTO statements) can automatically trigger clustering as long as the ingestion size does not exceed 512 GB. This is also called eager clustering:</p><pre>INSERT INTO transactions WHERE country = &quot;Germany&quot; <br>SELECT * FROM transactions_staging</pre><p>In order to ensure sufficient clustering across the dataset regardless of ingestion size, you can trigger clustering by running OPTIMIZE as the following:</p><pre>OPTIMIZE transactions</pre><p>For tables with frequent updates or inserts, a backgroundOPTIMIZE job can be scheduled to optimize data layout on a regular basis. Due to the incremental nature of clustering, table maintenance is computationally cheap and runs in a short time. Downstream processes and users can continue accessing the table simultaneously without an impact due to Delta Lake’s ACID transactions [<a href="https://docs.databricks.com/en/lakehouse/acid.html">9</a>].</p><p>When query predicates change over time, you can set new clustering keys on the table as below. Subsequent OPTIMIZE will cluster newly ingested data according to the new set of clustering keys.</p><pre>ALTER TABLE transactions CLUSTER BY (country, month, senderId);<br>OPTIMIZE transactions;</pre><h3>Current Limitations</h3><p>As of writing this article, Delta liquid clustering is in Public Preview. The following limitations exist [<a href="https://docs.databricks.com/en/delta/clustering.html">7</a>]:</p><ul><li>You can only specify columns with statistics collected for clustering keys. By default, the first 32 columns in a Delta table have statistics collected.</li><li>You can specify up to 4 columns as clustering keys.</li><li>Structured Streaming workloads do not support clustering-on-write.</li></ul><h3>Conclusion</h3><p>This write-up has compared Hive-style partitioning, Z-ordering, and Delta Lake liquid clustering using a practical example with a transaction dataset. Liquid clustering, an innovative approach to data layout optimization in Delta Lake tables, stands out as a superior option. It offers enhanced efficiency and flexibility compared to Hive-style partitioning and Z-ordering, leading to significant improvements in query performance and data maintenance. In addition, liquid clustering excels in handling data skew and evolving query patterns. Databricks recommends adopting liquid clustering for all new Delta tables.</p><h3>References</h3><p>[1] Databricks, <a href="https://www.databricks.com/blog/announcing-delta-lake-30-new-universal-format-and-liquid-clustering">Announcing Delta Lake 3.0 with New Universal Format and Liquid Clustering</a></p><p>[2] Databricks, <a href="https://www.databricks.com/glossary/what-is-parquet">What is Parquet?</a></p><p>[3] Databricks, <a href="https://docs.databricks.com/en/delta/index.html">What is Delta Lake?</a></p><p>[4] Databricks, <a href="https://docs.databricks.com/en/sql/language-manual/sql-ref-partition.html">Partitions</a></p><p>[5] Databricks, <a href="https://docs.databricks.com/en/tables/partitions.html">When to partition tables on Databricks</a></p><p>[6] Databricks, <a href="https://docs.databricks.com/en/delta/data-skipping.html">Data skipping with Z-order indexes for Delta Lake</a></p><p>[7] Databricks, <a href="https://docs.databricks.com/en/delta/clustering.html">Use liquid clustering for Delta Table</a></p><p>[8] Gustavo Martins (Closer Consulting), <a href="https://medium.com/closer-consulting/liquid-clustering-first-impressions-113e2517b251">Liquid Clustering: First Impressions</a></p><p>[9] Databricks, <a href="https://docs.databricks.com/en/lakehouse/acid.html">What are ACID guarantees on Databricks?</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1a277f57af99" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>