Dor Refael Avrahamof·Sep 29Your Flink Cluster’s Bill Isn’t Spiking. It’s CreepingNobody gets paged for a cost problem. That’s the issue.
Dor Refael Avrahamof·Sep 25dbt, Spark, or Flink? The Question Isn’t Which Is Best, But What You’re Actually DoingThree tools that get lumped together in every “modern data stack” diagram, and almost never compete for the same job.
Dor Refael Avrahamof·Sep 16Stop Guessing at Spark Configs: A Pipeline Optimizer With Rules You Can Read and an AI That…Why explainable rules drive the recommendations — and where Claude fits in
Dor Refael Avrahamof·Sep 7Everyone’s Building an AI Agent for Your Data Pipeline. I’m Not.Three companies I’ve looked at closely this year are all making the same bet, independently, with real money behind it. I want to walk…
Dor Refael Avrahamof·Aug 14Spark Executor Memory Tuning: When to Trust Rules, When to Ask AITwo different tools for two different kinds of uncertainty.
Dor Refael Avrahamof·Aug 11Spark support just became real, not a promiseA few weeks back I published a piece here saying, plainly, that opti-pipe’s Spark rules were fully built and fully tested but ran on sample…A response icon1A response icon1
Dor Refael Avrahamof·Aug 8Why I stopped guessing at Spark and dbt config valuesA response icon1A response icon1
Dor Refael Avrahamof·Jun 3, 2023Exploring Graph Processing with GraphX in Apache Spark 3.3: Unleashing the Power of Connected DataIntroduction: