InTowards Data EngineeringbyYeedu·6d agoWhy Spark Shuffle Breaks Down at Petabyte ScalePast a few hundred terabytes of daily shuffle, tuning spark.sql.shuffle.partitions or flipping on adaptive query execution stops moving the…A response icon1A response icon1
Filip Kosowski·Feb 3Apache Spark Shuffle: What It Is, Why We Need It, and What Triggers It?What is Shuffle?A response icon1A response icon1
Adhwaith Shankara Narayanan·Jan 6Apache Spark — Push Based ShuffleIn the previous blogs I had explained as to how Apache spark shuffles work. If you have not checked it, you can find it here → Apache…
Adhwaith Shankara Narayanan·Dec 15, 2025Apache Spark — Nuts & Bolts of Shuffle Part 2In my previous post, I explained what shuffle is in Apache Spark, why it occurs, and provided a high-level overview of how it works. In…A response icon1A response icon1
Seonggil Jeong·Dec 2, 2025Apache Spark Deep Dive: repartition & coalesceSpark를 통해 연산을 하다 보면 파티션의 수를 변경할 때 repartition과 coalesce를 사용할 수 있습니다. repartition, coalesce의 차이와 각각 어떻게 동작하는지 알아보겠습니다.
Seonggil Jeong·Dec 1, 2025Apache Spark Deep Dive: ShuffleSpark에서 Shuffle이 어떤 단계를 거쳐서 일어나고 각 단계마다 파티션 수가 어떻게 변하는지 알아보겠습니다.
Adhwaith Shankara Narayanan·Nov 24, 2025⚡Apache Spark — Nuts & Bolts of Shuffle Part 1Why shuffling occurs?
InAnalytics VidhyabyTharun Kumar Sekar·Jan 1, 2020Understanding Spark UIAt the end of this article, you will be able to look at the spark UI and understand the basic stuff.
Ai Deng·Nov 19, 2016Learn from Facebook 60 TB Spark use caseAfter read the Facebook’s engineer blog about Spark 60TB use case…
