In the fast-paced world of data engineering, Delta Lake has become a cornerstone for building reliable, scalable data pipelines on platforms like Databricks. It brings ACID transactions to data lakes, enabling robust analytics and machine learning workflows. However, when dealing with concurrent operations — such as multiple jobs or streams writing to the same table — users often encounter challenges that can disrupt performance and reliability. This article explores common concurrency issues in Delta Lake, practical mitigation strategies using partitioning and clustering, and experimental insights to help you optimize your setups.

In my early days working on pipelines that updated Delta tables, I frequently encountered Delta errors that halted our concurrent write jobs. While Delta Lake excels in analytical workloads, it’s not ideally suited for extremely high-transaction-volume scenarios, as handling massive concurrency remains a known challenge for columnar storage formats. That said, there are effective workarounds to these limitations without introducing new technologies like a dedicated transactional database into your stack.

To mitigate “concurrency error” messages, leverage partitioning and clustering wisely. Without proper partitioning or clustering on the the updated columns, operations often clashed over the same files, leading to conflicts. Below, I’ll outline the pros and cons of these two options. For in-depth details on Delta Lake’s concurrency and isolation mechanisms, refer to the official Databricks documentation on Isolation Levels and Write Conflicts.

Related errors: `ConcurrentAppendException`, `ConcurrentDeleteReadException`, `ConcurrentTransactionException`

Understanding the Basics

PARTITIONED BY

Partitioning in Delta Lake physically divides your table data into separate directories based on the values of one or more columns. This technique enables efficient query pruning by reducing the amount of data scanned, which improves query performance.

Use cases:

• When you have low to moderate cardinality columns frequently used as filters (e.g., region, customer ID).
• When you want to optimize for query pruning and reduce I/O.
• When your data ingestion or query patterns align well with partition keys.

Important note: By default, Spark often partitions files by date (e.g., year, month, day) in time-series workloads. Because of this, you might not always need to manually specify partitioning in your Delta Lake tables.

Considerations:

• Over-partitioning (too many small partitions) can lead to many small files and metadata overhead, which degrades performance.
• Changing partition keys after table creation is complex and generally discouraged.
• Partition pruning only benefits queries that filter on the partition columns.

Liquid Clustering (Automated CLUSTER BY)

Liquid Clustering is an automated, adaptive implementation of the traditional CLUSTER BY operation in Databricks Delta Lake. Instead of manually running clustering commands, Liquid Clustering continuously optimizes the data layout based on query patterns and data distribution, reducing manual tuning and maintenance.

Use cases:

• When you want automated, continuous optimization of data layout without manual intervention.
• When your query patterns evolve over time and you want the system to adapt dynamically.
• When you want to reduce operational overhead associated with manual clustering and partitioning.

Considerations:

• Requires Databricks Runtime versions that support Liquid Clustering.
• May introduce some additional compute overhead during optimization.
• Best suited for large, frequently queried tables with diverse and evolving query patterns.
• Liquid Clustering improves read performance by sorting data within files, enabling better data skipping, but does not isolate concurrent writes like partitioning does.

Note: The older ZORDER BY optimization is not longer recommended. Databricks prefers advanced and automated CLUSTER BY (Liquid Clustering) approach in Databricks Delta Lake.

Concurrency Experiment Setup

To validate concurrency behavior and performance, I ran a series of tests simulating concurrent writes to Delta Lake tables using three different data layout strategies.

• No partitions
• Liquid clustering (CLUSTER BY)
• Partitions (PARTITIONED BY)

Creating and Initializing a Clustered Delta Table

To explore concurrency in Delta Lake, I created a Delta table with clustering on the id column. The table schema includes id and status. I initialized the table with 100 records, each having a unique id and an initial status of "INIT".

from pyspark.sql.types import StructType, StructField, StringType, TimestampType
from datetime import datetime

# intialie tables with or without partitions
schema = StructType([
    StructField("id", StringType(), True),
    StructField("status", StringType(), True),
])
spark.sql("""
CREATE TABLE IF NOT EXISTS test_concurrent_writes_test_1 (
    id STRING,
    status STRING,
)
-- Note that we have NO PARTITIONS/CLUSTER and this will fail!
""")
spark.sql("""
CREATE TABLE IF NOT EXISTS test_concurrent_writes_test_2 (
    id STRING,
    status STRING,
)
PARTITION BY id
""")
spark.sql("""
CREATE TABLE IF NOT EXISTS test_concurrent_writes_test_3 (
    id STRING,
    status STRING,
)
CLUSTER BY id
""")

# populate tables with 100 ids and "INIT" status
ids = [f"{i:08d}" for i in range(1, 100)]
init_data = [(bid,"INIT") for bid in ids]
df_init = spark.createDataFrame(init_data, schema)
df_init.write.format("delta").mode("overwrite").saveAsTable(
   "test_concurrent_writes_test_1"
)
df_init.write.format("delta").mode("overwrite").saveAsTable(
   "test_concurrent_writes_test_2"
)
df_init.write.format("delta").mode("overwrite").saveAsTable(
   "test_concurrent_writes_test_3"
)

Concurrent Job Setup Using Joblib Parallelism

To simulate concurrent updates, I divided the id into chunks processed by multiple parallel jobs. Each job updates the status field for its assigned id using Delta Lake’s update API.

from pyspark.sql import SparkSession
from pyspark.sql.types import StructType, StructField, StringType
from delta.tables import DeltaTable
from joblib import Parallel, delayed
import time
import traceback

# Define update function
def update_status(id_val, table_num):
    deltaTable = DeltaTable.forName(spark, f"test_concurrent_writes_test_{table_num}")
    deltaTable.update(
        condition=f"id = '{id_val}'",
        set={"status": "'UPDATED'"}
    )

# Function to run test for a given table
def run_concurrency_test(table_num):
    print(f"Starting test for table {table_num}")
    ids = [f"{i:08d}" for i in range(1, 101)]
    start = time.time()
    status = "SUCCESS"
    try:
        Parallel(n_jobs=6)(delayed(update_status)(id_val, table_num) for id_val in ids)
    except Exception as e:
        status = "FAILED"
        print(f"Error: {traceback.format_exc()}")
    end = time.time()
    total_time_min = (end - start) / 60 if status == "SUCCESS" else "N/A"
    time_per_id_sec = (end - start) / 100 if status == "SUCCESS" else "N/A"
    print(f"Test {table_num}: Status={status}, Total Time (min)={total_time_min}, Time per ID (s)={time_per_id_sec}")

# Run tests
run_concurrency_test(1)  # Expected to fail
run_concurrency_test(2)  # Partitioned
run_concurrency_test(3)  # Clustered

This approach leverages 6 concurrent jobs, each handling a subset of id, to test the concurrency behavior of clustered Delta tables.

Experiment Results

+------+-----------------+---------+-----------------+------------+-----------------+---------------+
| Test | Method.         | Status  | Total Time (min)| Concurrency | Time per ID (s) | Total Ids    |
+------+-----------------+---------+------------------+-------------+----------------+--------------+
| 1    | No Partitions   | FAILED! | N/A             | 6           | N/A.            | N/A          |
| 2    | CLUSTER BY id   | SUCCESS | 4.32            | 6           | 12.95           | 100          |
| 3    | PARTITION BY id | SUCCESS | 1.5             | 6           | 4.5             | 100          |
+------+-----------------+---------+-----------------+-------------+-----------------+--------------+|

Key Findings

• No Partitioning (Test 1): Concurrent writes failed due to Delta Lake concurrency control errors (ConcurrentAppendException). This confirms that without partitioning or clustering, Delta Lake cannot safely handle concurrent appends at scale.
• Clustering (Test 2): Enabled concurrency with 6 parallel tasks, but total copy time was significantly higher (4.32 minutes) compared to partitioning. Clustering helps concurrency but does not isolate writes as effectively as partitioning.
• Partitioning (Test 3): Provided the best concurrency and performance, with the shortest total copy time (1.5 minutes) and lowest time per ID (4.5 seconds). Partitioning by ID and table name isolates concurrent writes to separate partitions, minimizing conflicts and maximizing throughput.

Best Practices

• When writes are more prevalent and reads are not so important, use partitions. Use PARTITIONED BY when you have natural partition keys that align with your query and write patterns, especially for high concurrency workloads.
• Use liquid clustering using CLUSTER BY to improve query performance on non-partition columns but be aware it does not isolate concurrent writes as well as partitioning. Consider liquid clustering for large, dynamic tables where automated, adaptive optimization can reduce manual maintenance. When reads are just as frequent as writes, then Liquid clustering might provide a better in between.

That said, partitioning and even advanced features like liquid clustering aren’t a silver bullet. Partitions and clustering come at a tradeoff that many files will slow down your queries! Even with optimal partitioning, updates targeting the same partition or clustered data (e.g., via liquid clustering) can still trigger conflicts, making truly concurrent writes challenging.

Concurrent writes in Databricks Delta Lake was originally published in Data Science Collective on Medium, where people are continuing the conversation by highlighting and responding to this story.

Earlier this week, I had the opportunity to attend the Insurtech Insights conference, an event that brings together a diverse range of innovators and leaders in the insurance technology sector. I was pleased to join the conference on short notice and found the experience both insightful and inspiring. Below, I share some of the key takeaways from the event.

Credit: Investopedia / Sydney Saporito

The conference primarily focused on Insurtech vendors and was designed to connect insurance professionals with cutting-edge proprietary software solutions. Established technology providers such as Oracle Cloud showcased their latest AI capabilities alongside a vibrant array of emerging startups leveraging advancements in AI and IoT. While many solutions were tailored to insurance carriers and underwriters, particularly in areas like site inspections, the event offered a valuable overview of innovations across the broader insurance ecosystem, including property insurance.

1. Hyperexponential Pricing tool

Hyperexponential has an intuitive and simple insurance pricing tool for actuaries and catastrophe modeling. Some interesting features are:

• End to end tool with portfolio ingestion, to actuarial CAT model rating tools like RMS plugin, all the way to a final pricing result
• Great flexibility to build the models

2. Instabase Document platform

• Instabase is a document platform app for unstructured data — you can try their tool here Instabase. The app learns from the uploaded document.

3. Data platform

• Cognisure.ai is an insurance data platform, like some projects at Data Strategy, trying to automate the insurance manual processes that are siloed across the organization like data ingestion from emails into actionable insights and profile developments.
• Majesco is another IT data platform. A relevant tool was a “loss control” product that they have built for insurance carriers and underwriters with very granular property fields, some sort of fine-grained portfolio ingestion that allows you to track losses.

4. Remote Sensing and IoT

• There were interesting startups in the property and casualty space leveraging image recognition and remote sensing technologies like Nearmap, a flood data provider Previsco, another geocoding vendor smarty, a Cyber IoT monitor platform Denexus, and even a booth with McKenzie Intelligence, event data provider.

The conference provided a highly engaging commercial platform where software vendors demonstrated their latest products and solutions. While detailed technical discussions on open-source tools and backend architectures were limited, the event offered valuable insights into the evolving landscape of insurance technology and the practical applications being developed.

Panel discussions reinforced the importance of strategic planning, subject matter expertise, stakeholder engagement, and continuous product exploration when integrating AI into insurance workflows. Overall, attending Insurtech Insights was a rewarding experience that broadened my understanding of the dynamic insurance technology space and highlighted promising directions for future innovation.

Towards a Collaborative Data Repository of the Subsurface

Underground data represents a multi-billion dollar asset for stakeholders that deal with the ground subsurface. During the last Sustainable Underground Systems Workshop at NYU-CUSP, Alan Leidner, president of NYC’s Geospatial Information System and Mapping Organization (GISMO), explained how this data can help us to run more effectively our physical infrastructure. It is estimated that 1% of excavations result in damages that cost an average $4,000 per incident nationally, but can be many times higher in dense urban environments. High quality, comprehensive and interoperable data about underground infrastructure (water, sewer, gas, electric, steam, telecommunications, transit networks, plus soils information) can have an annual benefit for capital construction and utility asset management operations that can be in the hundreds of millions of dollars for large sized cities like New York [1][2][3][4]. According to Leidner’s estimate, improving this data can save half a billion dollars per year to New York City government, city utilities and the city’s real estate and construction sectors.

Existing Information Models of the Built Environment

Tools like Building Information Modeling (BIM) can create digital representations with physical and functional details of the underground in a single file. However, the files are scattered among different parties in all sorts of formats and conventions, from public utilities to private telecom companies that are not always willing to share their precious information. This generates a large uncertainty for the underground works that translates into large risks and costs of infrastructure projects.

Underground Data Platforms Around the World

Some European regions have created collaborative platforms of underground data. The British Geological Society (BGS) has developed a nation wide borehole sharing platform during the past years (see image below). The Flanders region initiative led by Informatie Vlaanderen went even further since 2002 to include more soil tests and details through their open portal Databank Ondergrond Vlaanderen.

Back to the United States, the American Society of Civil Engineers (ASCE) leads their own initiative towards data standards. For example, the ASCE’s Utility Engineering & Surveying Institute offer civil engineering & surveying professionals standards and guidelines to work collaboratively. ASCE developed DIGGS, a project that involves development of a GML (XML-based) geospatial standard schema for the transfer of geotechnical and geoenvironmental data between multiple organizations. This schema allows a fluent transfer of data between project stakeholders, but the data is still likely to end in private hands.

No One Owns the Underground

This scattered environment of data aggregators reveals a weakness of the United States society where proprietary culture rules over a more collaborative model found in Europe, but it also reveals an opportunity of disruption. Amid this situation, a common-sense solution is an open-sourced collaborative data platform.

Security Concerns or Lack of Political Will?

New York city has particular issues with underground data availability. This was even more evident after 9/11 and hurricane Sandy when remediation works struggled to obtain inter-agency cooperation and underscored the need for planning (and data!) before a disaster strikes. According to Erika Kamptner, from NYC 3D Underground POC at NYC DoITT, there is a major security concern among agencies. Water utilities, for instance, suffer a great risk of vulnerability given their interconnected nature to the system as a whole. Certainly, disclosing this data can pose a major threat to public safety. However, many other industry leaders believe the unwillingness of some stakeholders to share information might be motivated by lack of incentives and the retention of political power.

Reaching Common Ground Towards a Collaborative Data Platform

The Open Geospatial Consortium (OGC) proposes the Model for Underground Data Definition and Integration (MUDDI). It designs a model to enable sharing between planning, design, construction, and operation phases through standards that ensure interoperability. Their website details a sophisticated systems-engineering approach to what can become leading collaborative data standards.

Underground Version 2.0

In my opinion, there is a need for a collaborative platform with servers, maps, and info-graphics built on top of existing standards. Given the failure among government agencies to create a common platform, what if we develop a portal that manages our underground using the same principles of modern Data Ops?

To make the product scalable, the platform must be open to any user. For those parties concerned with privacy, users should have the option of having either public or private data repositories. To achieve order and legal liability, verified accounts could be given to certified professionals and regulators.

Underground data advocates should fight for better collaborative platforms in order to keep working towards a more dynamic city that makes way to arteries of services fueled by readily available data as we excavate through the tissues of the urban subsurface. To improve the current status, the problem calls for leaders to adopt open database architectures, visualizations, and story-telling and work closely with innovators to incorporate smarter technologies. Only this way we will maximize the benefits of our underground world as a massive asset that improves our communities.

References

[1] Common Ground Alliance (2017). Damage Information Reporting Tool (DIRT). Retrieved from https://commongroundalliance.com/programs/damage-information-reporting-tool-dirt

[2] Alan C. O’Connor, J L. Dettbarn, L T. Gilday (2004). Cost Analysis of Inadequate Interoperability in the U.S. Capital Facilities Industry. NIST. Retrieved from https://www.nist.gov/publications/cost-analysis-inadequate-interoperability-us-capital-facilities-industry

[3] Rajat Agarwal, Shankar Chandrasekaran, and Mukund Sridhar (June, 2016). Imagine Construction’s Digital Future. McKinsey and Company. https://www.mckinsey.com/industries/capital-projects-and-infrastructure/our-insights/imagining-constructions-digital-future

[4] Federal Highway Administration Washington, DC. (1999). Cost Savings On Highway Projects Utilizing Subsurface Utility Engineering. Retrieved by https://www.fhwa.dot.gov/programadmin/pus.cfm

Underground Version 2.0 was originally published in data.tale() on Medium, where people are continuing the conversation by highlighting and responding to this story.

Learning Like Babies

Image Classification Using Semi-Supervised Learning

Convolutional Neural Nets (CNNs), a concept that has achieved the greatest performance for image classification, was inspired by the mammalian visual cortex system. Yet, state-of-the-art artificial intelligence models work different to the natural learning process of mammals. AI models often learn from extensive human labels, like a picture of a dog with a label of the word ‘dog’. This is called ‘supervised learning’. The problem is that most of the real world data is not labeled. The majority of our learning of the world comes from self learning.

According to Yann LeCun, father of CNNs and professor at NYU, the next ‘big thing’ in artificial intelligence is semi-supervised learning — a type of machine learning task that makes use of unlabeled data for training — typically a small amount of labeled data with a large amount of unlabeled data. That is why recently a large research effort has been focused on unsupervised learning without leveraging a large amount of expensive supervision.

“ The revolution will not be supervised” — Alyosha Efros

Inspired on this concept, our class at NYU hosted a competition to design a semi-supervised algorithm from 512k unlabelled images and 64k labelled images from ImageNet.

Semi-supervised learning is in fact the learning method mostly used by infants. When we are born, we don’t know how the world works: we don’t distinguish gravity, we don’t understand depth, or much less do we recognize human expressions. The human brain absorbs data mostly in an unsupervised or semi-supervised manner.

Method: Semi-supervised Representation Learning by Predicting Image Rotations

For the competition, the method employed by our algorithm recognizes the 2d rotation that is applied to the input image [2]. In a way, this assimilates to how babies learn to see the world through experience. For example, through time babies get used to how objects stand and how mountains lie below the sky. As it turns out, even this simple task of image rotation allows the network to learn features that are relevant for supervised tasks.

We used ResNet architecture (more specifically ResNet-50) for our final submission. ResNet models achieve state-of-the-art performance on many classification data sets. Moreover, ResNet residual units are invertible under certain conditions, which might help to preserve information from the early layers of the network obtained as a result of pre-training. We utilised a large set of unlabeled images to pre-train a ResNet-50 model on the rotation pre-task. The goal of the model is to predict one of the four rotation angles (0, 90, 180, 270) for each input image. As a result of training the model on this auxiliary task, we expected it to learn features that would be helpful in the main classification task.

After pre-training, we trained a linear classifier on top of this feature extractor for the main task, and fine-tuned the whole network by gradually unfreezing the layers starting from the top. We also used data augmentation. This strategy led to a significant boost in accuracy as compared to the previous model.

Results

The figure below shows the training and validation loss curves, and accuracy for the final model over the course of training. Red line shows the moment of switching from training only linear classifier to fine-tuning the whole network.

The figure below visualizes nearest neighbors for 4 random images in the validation dataset (first column). As it can be seen, semi-supervised rotation model clearly managed to learn meaningful representations of the images.

The final top-1 validation accuracy was 32.4%. Overall, we can conclude that semi-supervised pretraining on the unlabeled data helped to improve accuracy on the main classification task. However, it is not clear how much more beneficial semi-supervised training is as compared to pre-training on existing large labeled datasets such as ImageNet.

For more information, you may refer to our paper.

This project was developed by Evgenii Nikitin, @sunor.yavuz, and Manrique

References

[1] Carl Doersch, Abhinav Gupta, and Alexei A. Efros. Unsupervised visual representation learning by context prediction. In International Conference on Computer Vision (ICCV), 2015.
[2] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. CoRR, abs/1803.07728, 2018.
[3] Aidan N. Gomez, Mengye Ren, Raquel Urtasun, and Roger B. Grosse. The reversible residual network: Backpropagation without storing activations. CoRR, abs/1707.04585, 2017.
[4] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CoRR, abs/1512.03385, 2015.
[5] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. CoRR, abs/1603.05027, 2016.
[6] Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer. Revisiting self-supervised visual representation learning. CoRR, abs/1901.09005, 2019.
[7] Sebastian Ruder. An overview of gradient descent optimization algorithms. CoRR, abs/1609.04747, 2016.
[8] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. CoRR, abs/1605.07146, 2016.