InTowards AIbychakshu_salgotra·4d agoPart XXII — Caching Is Not Free: Persist Levels and the Memory You LoseThe fastest way to make a Spark job slower is to cache everything “just in case.” Here is the engine-level accounting of what `.cache()`…
InTowards AIbychakshu_salgotra·Nov 8Part XXI — Streaming State: Watermarks, State Stores, and the 146GB CheckpointEvery Structured Streaming query is a stateful program wearing a batch costume. The watermark is the only thing standing between your state…
InTowards AIbychakshu_salgotra·Nov 2Part XX — Delta Lake’s Transaction Log: Time Travel, VACUUM, and the Files That Never DieYour table never deletes anything. The Delta log is an append-only ledger, and every file you “removed” is still sitting on S3 until the…
InTowards AIbychakshu_salgotra·Oct 30Part XIX — Spark on Kubernetes: Executor Sizing and Dynamic AllocationKubernetes killed our executors every night for a week while the JVM insisted memory was fine. The JVM was telling the truth — it just…
InTowards AIbychakshu_salgotra·Oct 23Part XVIII — The Small-Files Tax: Compaction Strategies That Don’t StallYour lakehouse doesn’t slow down because the data grew. It slows down because the file count did — and every one of those files is a tax…
InTowards AIbychakshu_salgotra·Oct 22Part XVII — Checkpointing vs Lineage: The Fault-Tolerance Trade You MisconfigureSpark’s lineage is the most elegant recovery mechanism in distributed computing — until your iterative job’s DAG grows so long that…A response icon2A response icon2
InTowards AIbychakshu_salgotra·Oct 21Part XVI — Whole-Stage Codegen: The Tungsten Trick Behind 2x SpeedupsYour Spark job is not slow because of your data. It is slow because Spark is interpreting your query one virtual function call at a time —…
InTowards AIbychakshu_salgotra·Oct 20Part XV — Adaptive Query Execution: What AQE Actually Rewrites at RuntimeSpark plans your query before it has seen your data. AQE is the engine’s admission that the planner was guessing — and the machinery that…A response icon1A response icon1
InTowards AIbychakshu_salgotra·Oct 14Part XIV — Broadcast Variables: The Driver Bottleneck Nobody ProfilesBroadcast joins are sold as free money: skip the shuffle, ship the small table everywhere. But “small” is measured against the driver’s…
InTowards AIbychakshu_salgotra·Oct 12Part XIII — Internal vs External Tables: The DROP TABLE That Deleted 40TBManaged tables own your data; external tables borrow it. Confusing the two is how a schema cleanup script becomes a data-lake extinction…A response icon1A response icon1