PinnedVed Prakash·Aug 220 Data Engineering Interview Questions You Should Know for Databricks & PySpark Roles- Part 4for non members, read the blog for free hereA response icon3A response icon3
PinnedVed Prakash·Aug 2, 2023How to List and Print the Content of a S3 Bucket in AWS using PythonSince this is my first blog so if you’re reading this don’t forget to leave your comments below.A response icon1A response icon1
Ved Prakash·Oct 24Spark UI: How to Debug and Optimize Apache Spark JobsA practical guide to Spark’s explain(): logical plans, physical operators, join strategies and AQE, with real plans and a join that got…
Ved Prakash·Oct 22Explain Plan in Apache Spark: How to Read and Optimize Query Execution PlansApache Spark Learning Path — Progress TrackerA response icon1A response icon1
Ved Prakash·Oct 17I Changed Two Numbers and My AI Snake Learned to Kill Itself :Part 2Reward hacking, explained with five Q-learning snakes — and the five rules I now follow when writing a reward function
Ved Prakash·Oct 17Serialization in Apache Spark: Java vs Kryo, Performance, and PySparkUnderstand how Spark converts objects into bytes, why serialization affects shuffle performance and memory usage, and when Kryo can make…A response icon1A response icon1
Ved Prakash·Oct 15Reinforcement Learning: Teaching a Snake to Eat — Part 1: Building the World and a First BrainA five-part series on building a reinforcement learning Snake agent from scratch: no game engine, no RL library, just Python, NumPy and a…
Ved Prakash·Oct 15Checkpointing in Apache Spark: How to Break Long Lineage and Improve ReliabilityLearn how checkpoint() works, why long execution plans become expensive, and when reliable or local checkpointing makes sense in production…A response icon1A response icon1
Ved Prakash·Oct 8Persistence in Apache Spark: The Complete Guide to persist() and Storage LevelsLearn how Spark stores intermediate DataFrames and RDDs in memory, disk, or both — and when persistence can dramatically improve…A response icon1A response icon1
Ved Prakash·Oct 8Caching in Apache Spark: How to Avoid Recomputing the Same Data Again and AgainHow cache() works, when it actually improves performance, and why caching everything can make your Spark job slowerA response icon1A response icon1