Ayush Joshi·Sep 28Coalesce vs Repartition in Spark: A Stage-by-Stage BreakdownWhat really happens to your Spark partitions when you use coalesce() or repartition()?
Ayush Joshi·Sep 21SQL Query Execution Order: What Actually Happens When You Run a QueryYou write SQL like this:
Ayush Joshi·Sep 16Databricks Auto Loader: The Deep-Dive I Wish Someone Had Handed Me on Day OneMost data engineers learn Auto Loader like this:A response icon3A response icon3
Ayush Joshi·Sep 12Apache Airflow Architecture Explained: Every Component, End to EndApache Airflow is usually introduced as “just a workflow scheduler.” But the moment you go past a single DAG, you run into a wall of…A response icon1A response icon1
Ayush Joshi·Sep 7Databricks MERGE INTO Explained: Upserts, CDC, SCD, Deletes & Schema EvolutionIf you work with Databricks and Delta Lake, you will eventually run into MERGE INTO. It looks like a normal SQL statement, but it's…A response icon1A response icon1
Ayush Joshi·Sep 2Databricks Lakeflow Declarative Pipelines: What, Why & How ExplainedModern data engineering rarely becomes difficult because writing a SQL query is difficult.A response icon1A response icon1
Ayush Joshi·Sep 1Databricks Schema Evolution Explained: mergeSchema, overwriteSchema & Parquet RewritesSchema changes in Databricks often sound simple: add a column, rename one, drop one, or change a data type. But there’s a question that…
Ayush Joshi·Aug 25Deletion Vectors in Databricks: What They Are, Why They Exist, and How to See Them WorkIf you’ve ever run a DELETE on a massive Delta table and wondered why a single-row deletion touches gigabytes of data, this article is for…
Ayush Joshi·Aug 19Databricks Data Layout Explained: Partitioning, OPTIMIZE, Z-ORDER & Liquid ClusteringIf you’ve worked with Databricks, you’ve probably heard these four terms: Partitioning. OPTIMIZE. Z-ORDER. Liquid Clustering.
Ayush Joshi·Aug 12Azure Data Factory Copy Activity Performance Optimization: Complete GuideYour Copy Activity is taking 3 hours. You increase DIU. Nothing happens. You increase parallelism. It gets even slower.