nimish jain·Oct 5Everything about Spark-Submit in PysparkWhat is spark submit? 1) Runs spark job end to end 2) It is a command line tool used to run or submit spark app/job on spark clusterA response icon1
keisuke nebashi·Sep 4Two Ways to Add Packages When Using SparkWhen loading data from BigQuery with Spark, I need to use the Spark BigQuery Connector.
InData And BeyondbySriw World of Coding·May 11Data Engineering Interview Question : How Much of Your 32GB Executor Heap is Actually for Caching…Imagine launching a Spark job on a beefy executor with 32GB heap memory, only to watch it crash with OOM errors during a simple cache…
InTowards Data EngineeringbySriw World of Coding·May 1Spark Interview Question : Spark Executor Tuning Masterclass: Process 2TB Parquet Like a Pro“Your Spark job on 2TB data is crawling at 1% done after 2 hours. The cluster looks healthy, but everything’s spilling to disk. Sound…A response icon5
InData And BeyondbySriw World of Coding·Apr 28Data Engineering Interview Question : Fix Your Spark Output: From 50,000 Tiny Files to Fast…Imagine this: your 10 TB Spark job finishes cleanly at 3 AM. By 10 AM, the dashboard is red, and the downstream team is livid: “Your 50,000…
