PinnedInCutting Through the Data SpaghettibyWilliam·Mar 7Three Days Fixing the Same DWS Pipeline: A Data Engineer’s Debugging with AgentA data engineer’s day is full of manual workflows: aligning source tables, checking key distributions, comparing output to expectations…
InCutting Through the Data SpaghettibyWilliam·Apr 16The Case for a Declarative Shift: Fixing the Foundation Before We Chase AI1. The Maintenance Trap
InCutting Through the Data SpaghettibyWilliam·Mar 11Drawing a Map in the Commit Maze: Maintaining a 7-Year-Old Report Pipeline with Agents and Leaving…Ops asked for a dimension change. I got lost in a repo built from 295 commits — along with the AI — then tried to make it a little less…
InCutting Through the Data SpaghettibyWilliam·Mar 8Splitting a Spark Dataset Randomly into Two Parts: Same Group Key Must Stay in One PartI told the Agent “split by group key so each group stays together” — then data validation kept failing. A data engineer’s implementation…