InT3CHbySriw World of Coding·Jun 21df.show() vs df.collect(): The One-Letter Mistake That Crashes Spark ClustersA junior data engineer adds one innocent line to “just peek at the data” before running a 200 GB pipeline:
InT3CHbySriw World of Coding·Jun 16Spark Interview Questions : Spark Driver OOM Nightmare - How to Calculate Driver Memory for…
Sriw World of Coding·Jun 4Spark Interview Question : Why on calling collect() action your Spark Job Crashes After 100%Imagine this: You’ve just processed 10 TB of data across 500 executors. The Spark UI shows “Job: 100% complete”. You smile… and then your…
Omkar·Feb 17, 2025Understanding & Fixing Spark Driver and Executor Out of Memory (OOM) ErrorsIntroductionA response icon1A response icon1
Bhavya Joshi·Jul 9, 2024Monitoring Apache Spark Metrics with PrometheusDeveloping a Spark application is a fundamental task; however, optimizing its resource utilization poses a substantial challenge. By…
Nethaji Kamalapuram·May 13, 2024Spark Memory allocation & ManagementLets talk about how memory allocation works for spark driver and executors. Understanding driver and executor memory allocation is crucial…A response icon1A response icon1
DED(Data Engineering Digest)·Oct 31, 2022How JMX metrics from spark applications will help to configure driver/executor memory correctly…What are jmx metrics → Java Management Extensions (JMX) is a specification for monitoring and managing Java applications.
Mahendran·Apr 8, 2021Spark YARN [static] Resource AllocationHere’s the basic spark-submit command you might be using in production
