InTowards Data EngineeringbyThink Data·Sep 16The One-Line Spark Change That Can Turn a Slow Join Into a Fast OneThe optimization worked the last time someone checked. Nobody had checked in months.A response icon2A response icon2
Adhwaith Shankara Narayanan·Jan 6Apache Spark — Push Based ShuffleIn the previous blogs I had explained as to how Apache spark shuffles work. If you have not checked it, you can find it here → Apache…
Adhwaith Shankara Narayanan·Dec 15, 2025Apache Spark — Nuts & Bolts of Shuffle Part 2In my previous post, I explained what shuffle is in Apache Spark, why it occurs, and provided a high-level overview of how it works. In…A response icon1A response icon1
InArt of Data EngineeringbySantosh Joshi·Feb 1, 2025How Spark Performs Joins: A Quick Look into Small and Large Table JoinsOptimizing join operations in Spark for different table sizes and distributions.
Feruz Urazaliev·Aug 27, 2024How Can You Optimize Spark Join Operations for Better Performance in Big Data?Unlock the secrets to speeding up your Spark joins and take your data pipelines to the next level
Nethaji Kamalapuram·May 15, 2024Spark Dynamic Partition PruningDynamic Partition Pruning for Optimized Joins
Ajaykumar Dev·Mar 21, 2024Spark Interview Questions 5what is the difference between rdd and dataframe and dataset in spark ?
InTowards DevbyGuilherme Eid·Feb 13, 2024Spark Joins: A Comprehensive GuideUnderstanding data sharing among spark executors
Ashwin·Dec 23, 2023Broadcast Join in SparkAre you struggling to optimize your data processing in Spark? Look no further! This article will introduce you to the powerful and…A response icon1A response icon1
Venkatakrishnan·Sep 18, 2023How Apache Spark decides on the join strategyApache Spark uses a cost-based optimizer to decide on the join strategy. The optimizer takes into account a number of factors, including…
