Amortising a Graph of Demands: Attach, Don’t Rebuild
Chain enough demands together and what forms is a graph. Walking it doesn’t have to mean rebuilding it.
A kitchen doesn’t build béarnaise from raw ingredients every time a dish calls for it. It builds hollandaise once, as a stable base. Béarnaise is hollandaise plus one small, specific addition, tarragon, shallot, a reduction. The base isn’t remade for every dish depending on it. Only the last step, the specific attachment, changes.
That’s the whole idea this piece is built on.
From Chain to DAG
The last piece in this series left an image. Demands chaining together, each one depending on the one before it. Formalise that and what it actually is becomes clear. It’s a DAG. Nodes are outcomes. Edges are dependency.
Once it’s a graph, two questions become available that weren’t before. Walk it forward, and it answers what’s affected if something breaks. Walk it backward, and it answers what’s actually required to make something happen.
Walking Forward: Impact
A real pipeline, ending in a parquet file written to an S3 location, needed a new inline data quality check added. Not a rewrite. One new step, inserted before delivery.
Walking forward from that new step answers the obvious question. Who’s downstream of it. At the time, the answer was simple, one consumer, reading the parquet file from its S3 location. The new check could go in with confidence, because the graph made the blast radius visible before the change ever shipped.
This is the same question project management has always asked with critical path analysis. What happens to everything downstream if this node changes or fails. The DAG just makes it something you can actually compute, on a live system, instead of something you reason about from memory.
The change can be assessed before it ships, evidence gathered to prove it’s safe. That evidence is what lets it be added to a running system autonomically, without waiting on a person to sign off. If the change affects several customers, each one can accept it, defer it, or reject it on their own terms. Their DAG stays exactly as it was until they choose otherwise.
Walking Backward: Effort
The other direction starts from the outcome and works back. Not what’s affected, but what does it actually take to get there, and by when.
Get Stephen James Price’s stories in your inbox
Join Medium for free to get updates from this writer.
This is where Manufacturing Resource Planning, MRP II, comes from originally, tracing a finished product back through everything required to produce it. This piece only needs the shape of that idea. The fuller version, accounting for efficiency and yield rather than just raw effort, belongs to the next piece in this series.
A demand might require a service to be started, or scaled out to meet it. That decision can be autonomic too. The system decides whether to act now or later, and business value becomes an active input into that decision, not just the fact that a demand exists. The system optimises for value, not merely because something was asked for.
The Expensive Part, Named Plainly
Walking a live, evolving DAG from scratch, in either direction, every time a question gets asked of it, doesn’t scale. Demands change. Sources change. New steps get inserted, the way the quality check was. Recomputing the whole graph on every question is the actual problem worth solving here, not a hypothetical one.
Amortising the Walk
The same pipeline, later, got asked for something new. The parquet to S3 delivery had been working fine. Now a customer wanted the same data delivered to Snowflake as well.
Nothing about extraction changed. Nothing about the transformation logic changed. The data quality check added earlier stayed exactly where it was, still doing its job before delivery. The only new work was the delivery path itself, a new terminal node attached to the graph at the exact point where the two outputs actually diverge.
The DAG didn’t get redrawn from the top. It grew one branch, at one point, and everything upstream of that point kept running unmodified and unaware anything had changed.
A restaurant’s seasonal menu makes the same point from a different angle. A few dishes on the menu can be adjusted each season, with changes based on produce that’s gone out of season. Everything else on the menu keeps running exactly as it was.
Not every restaurant makes that choice, either. Some keep a dish on the menu all year round, out of season or not, paying whatever it costs to source the produce anyway, because stability for the customer matters to them more than the saving. That’s a legitimate choice too. Amortisation isn’t a rule a system is forced into. It’s an option, weighed against the cost of change versus the cost of staying still.
This is amortisation, in the plain sense of the word. The cost of building the graph gets spread across every demand that reuses it, rather than paid in full again for each new one. Most of the structure is already there. Only the point of attachment, and a thin slice around it, is ever new.
Close
Walking forward and walking backward both stay cheap once the DAG stops being rebuilt from scratch for every question. What’s actually required to trust that walk, evidence it ran correctly, against the right version of the graph, is the subject of a later piece in this series, Providence.
