PinnedInData Engineer ThingsbyAkashdeep Gupta·Dec 16, 2024How withColumn Can Degrade the Performance of a Spark Job?Reasons and Solutions to Avoid Performance Degradation due to excessive use of .withColumn() in Apache SparkA response icon8A response icon8
PinnedAkashdeep Gupta·Jan 24, 2024Copy-on-Write or Merge-on-Read? What, When, and How?This blog post was originally published on guptaakashdeep.comA response icon6A response icon6
PinnedAkashdeep Gupta·Oct 13, 2024Enhancing Spark Job Performance with MultithreadingThis blog post was originally published in The Pragmatic Data Engineer’s Playbook Newsletter.A response icon1A response icon1
InData Engineer ThingsbyAkashdeep Gupta·Jan 27, 2025Optimizing Iceberg MERGE StatementsHow can you eliminate shuffling, sorting, and push-down filters to optimize the Apache Iceberg merge statements?
InData Engineer ThingsbyAkashdeep Gupta·Dec 30, 2024Selecting between Double and Decimal Data Type To Avoid Unexpected ResultsHow do you choose between Double and Decimal data types for your tables/datasets on Data Lake, why does it matter, and when do you choose…
InData Engineer ThingsbyAkashdeep Gupta·Dec 25, 2024Building A Lightweight Spark Exception LoggerHow to Create a Spark Exception Logger and Test It Locally Using DuckDB, MinIO, and a Standalone Spark ClusterA response icon1A response icon1
InData Engineer ThingsbyAkashdeep Gupta·Dec 10, 2024Shuffle-less Join, a.k.a Storage Partition Join in Apache Spark — Why, How and Where?A Deep Dive into Shuffle-less joins (Storage Partitioned Joins) in Apache Spark to improve Join performance when using V2 Data Sources…
Akashdeep Gupta·Feb 13, 2024EMRFS S3 Optimized Committer and Committer Protocol for Improving Spark Write Performance — Why…This post was originally published at guptaakashdeep.com
Akashdeep Gupta·Jan 17, 2024Apache Iceberg — Architecture DemystifiedThis blog post details what happens under the hood when interacting with Apache Iceberg Tables. It explains how the different components…A response icon4A response icon4
Akashdeep Gupta·Jan 3, 2024How to Implement Write-Audit-Publish Pattern with Apache Iceberg on AWS using WAP idThis blog will cover how to implement a Write-Audit-Publish (WAP) Data Quality Pattern in AWS using Apache Spark on EMR that uses AWS Glue…