Shahzeb Ali·6h agoThe Merge That Failed on Rows It Was About to Throw AwayA delete marker for a key the table never stored, plus an insert in the same batch, and delta-rs refused the merge. The data was fine. The…
InTowards Data EngineeringbySaurav Kumar·10h agoWhat Is Data Skipping in Delta Lake, and Why Your Filter Still Reads EverythingDelta keeps a small note about every file it writes. Your query can only skip what those notes can rule out, and a lot of tables are laid…A response icon1A response icon1
Shree Kavya·1d agoChange Data Capture in Apache Spark, Hands-On: From Raw Change Events to a Synced TableA working demo with inserts, updates, deletes, duplicates and out-of-order events, plus the pitfall that resurrects deleted rows
Pranav Sadagopan·1d agoBuilding a Local Lakehouse with Spark Declarative Pipelines, Unity Catalog OSS, Delta Lake and…A hands-on exploration of building a local lakehouse using Apache Spark Declarative Pipelines, Unity Catalog Open Source, Delta Lake and…A response icon1A response icon1
Saleheen Faiyaz·1d agoDatabricks Gold Layer: Choosing the Right PatternMaterialized Views, Streaming Tables, REPLACE WHERE Flows, and Delta Tables — when to use each.A response icon1A response icon1
Pandraju Isaac·1d agoUpserts Without a MERGE String: DataFrame.mergeInto in PySpark 4Spark 4 gave the DataFrame API its own MERGE builder. I ran it against a local Delta table to see where it helps and where it still bites.
InITNEXTbySebastian Daberdaku·2d agoStreaming hundreds of Postgres tables into Delta Lake with one Spark query per databaseDebezium, Kafka and kafka2delta: one Structured Streaming query that fans out to a Delta table per topic.A response icon1A response icon1
Md Zakir Ahmed·2d agoIf Parquet files are immutable, how does Delta Lake support UPDATE and DELETE operations?Let’s understand what happens behind the scenes.A response icon2A response icon2
Greg Deckler·3d agoMicrosoft Fabric Change Data Feeds: What’s REALLY Going On Under the Hood?So in my YouTube video equivalent of my article Microsoft Fabric Materialized Lake Views Have One Big CATCH!, I change data feeds (CDF)…A response icon1A response icon1
Kamal Thapa·3d agoLiquid Clustering in Plain EnglishWhy I stopped arguing about partition columns, and what Databricks does insteadA response icon1A response icon1