DED(Data Engineering Digest)·Aug 5When Spark can SPILL to a disk, Why does it still OOM?Spill is the process of temporarily writing in memory data structures (from shuffle or aggregation buffers) to local disk when an executor…
InData And BeyondbySriw World of Coding·May 11Data Engineering Interview Question : How Much of Your 32GB Executor Heap is Actually for Caching…Imagine launching a Spark job on a beefy executor with 32GB heap memory, only to watch it crash with OOM errors during a simple cache…
Pulumatirambabu·Jun 5, 2025Understanding Spark’s df.cache() Heap vs. Off-Heap Memory StorageApache Spark’s in-memory caching mechanism is a powerful feature for optimizing iterative algorithms and interactive data analysis. One of…
Omkar·Feb 17, 2025Understanding & Fixing Spark Driver and Executor Out of Memory (OOM) ErrorsIntroductionA response icon1A response icon1
Ido Levi·Feb 2, 2025Taming the Spark MemoryA while ago, I was troubleshooting one of our Spark jobs — and I noticed something odd. Whether the job processed 10GB or 300GB of data…
Avin Kohale·Aug 1, 2024Spark — Beyond basics: Required Spark memory to process 100GB fileProcessing 100GBs file is a cake walk for spark ONLY if you know how to assign spark memory efficiently! Read to know more.A response icon12A response icon12
Nethaji Kamalapuram·May 13, 2024Spark Memory allocation & ManagementLets talk about how memory allocation works for spark driver and executors. Understanding driver and executor memory allocation is crucial…A response icon1A response icon1
