InTowards Data EngineeringbyYeedu·Nov 14Trino federates your queries, not your source’s connection limitsA handful of ad-hoc Trino queries against a Postgres or MySQL OLTP catalog can exhaust that database’s connection limit or force a…A response icon1A response icon1
InTowards Data EngineeringbyYeedu·Nov 7Why Spark Shuffle Breaks Down at Petabyte ScalePast a few hundred terabytes of daily shuffle, tuning spark.sql.shuffle.partitions or flipping on adaptive query execution stops moving the…A response icon1A response icon1
InTowards Data EngineeringbyYeedu·Oct 26The Spark Small-File Problem: Why Compaction Is a Recurring TaxSpark chokes on small files because every file carries a fixed metadata cost that doesn’t shrink with the data inside it, and running…
Yeedu·Oct 7Why Kafka-to-Flink pipelines still deliver duplicates even with exactly-once enabledExactly-once checkpointing in Flink guarantees that Flink’s own state and any sink that fully implements two-phase commit agree with the…A response icon1A response icon1
InTowards Data EngineeringbyYeedu·Sep 18ClickHouse’s ‘Too Many Parts’ Wall: Why Streaming Small Inserts Breaks a Batch-Oriented Merge…ClickHouse throws DB::Exception: Too many parts (600). Merges are processing significantly slower than inserts. when the background merge…A response icon1A response icon1
Yeedu·Sep 14Idempotent backfills: why the fast partition-overwrite patterns in Spark, Delta and Iceberg are the…Every backfill path we reach for because it’s fast, Spark’s dynamic partition overwrite, Delta’s replaceWhere, Iceberg’s INSERT OVERWRITE…
Yeedu·Sep 10Why Airflow’s Scheduler Buckles When Dynamic Task Mapping Fans Out WideDynamic task mapping lets a DAG decide how many tasks it needs at runtime instead of at parse time, but the scheduler underneath still…
InTowards Data EngineeringbyYeedu·Sep 4What is TPC Benchmarks? Are They Still Relevant in 2026?Why the data industry’s oldest performance standard still matters when every vendor’s whitepaper tells a different story, and where it…A response icon1A response icon1
Yeedu·May 14Spark Performance Optimization: 10 Techniques That Actually Move the NeedleMost Spark performance advice sounds right but rarely helps in production. This blog is written for engineers who already run Spark at…
Yeedu·May 5Why Modern Query Engines Avoid the JVM — And What That Means for SparkA few years ago, most data teams didn’t question the execution layer of their analytics stack. If you were using Apache Spark, you were…