Reliability isn’t free: what each check costs in a multi-agent pipeline
Where the cost of trusting a multi-agent pipeline lands, and the check I still can’t audit.
My Closet & Wardrobe Recommender agent suggests what to wear from a cataloged closet and a weather forecast. Three agents do the work: a cataloguing agent that turns garment photos into structured records, a weather agent that turns a forecast into constraints, and a stylist that proposes outfits. There is a check at every handoff between them, and some of those checks cost something. Most are code and cost nothing at runtime. The rest cost a model call or my own time, and part of my eval re-checks the guards.
Most of the checks are code
Every boundary has a deterministic guard. A model response has to validate against a Pydantic schema or the call fails. The stylist’s picks are dropped if they name an id that was never sent, if they aren’t a complete outfit (one top and one bottom, or one dress, with at most one outer layer and one pair of shoes), or if they repeat an outfit already accepted. The weather agent’s model can’t switch off a rain or wind flag where the rules were already decisive. A truncated or out-of-range model response used to be able to crash the weather agent; since an October 5 commit it falls back to the rules.
None of these makes a model call, so none of them adds API spend. I have five saved recommend reports, all scripted scenarios on the seed closet with fixed forecasts. No guard dropped an outfit in any of them, and one run raised two warnings. Five scripted scenarios say very little about how often the model is wrong.
The paid checks are conditional
The checks that do cost a model call are set up to run only when needed. The weather agent calls the model only when the day is ambiguous, for example rain probability between 20 and 70 percent, wind of 15 to 35 kph, or a swing of 10 degrees or more between low and high. A dry 26-degree day makes no model call. The stylist gets one more call only when a guard drops an outfit and leaves fewer than requested. If the model just returns fewer outfits, there’s no retry.
Cataloguing runs when I add a garment, not on each recommendation. It sends each photo to a vision LLM and then holds the draft until I accept or edit it. In the web UI, uncertain fields are highlighted. That confirmation is a check paid once per garment, by me, and an error that gets past it would show up in every outfit the garment is in.
Get Anastasiia Bakhtoiarova’s stories in your inbox
Join Medium for free to get updates from this writer.
What decides whether a check can trigger a paid call is its severity. A guard that drops an outfit can cause a retry. However, a guard that warns can’t. A wet forecast calls for a waterproof outer layer, but an outfit without one is kept and flagged, because an umbrella is a fair answer, so that check never costs a retry. In the one run where the waterproof and windproof warnings fired, the model wasn’t at fault. The seed closet has no outer item tagged either way, so no outfit could have met the requirement, and the flag said so.
What I can count and what I can’t price
Here is what I can count. Each test case in the eval makes three model calls: one for the stylist to pick outfits, one for a judge that scores how well they suit the occasion and the weather, and one for a judge that checks whether the written explanations are true of the items. The suite has six cases and runs each once, so a run is 18 calls, or more if the stylist has to retry.
I can’t say how long a run takes or what it costs in dollars. The run reports record the model, the constraints, the outfits and the warnings, but not how long a call took or how many tokens it used. Effort defaults to high and the timeout to 180 seconds, so I know the limits, not how long a typical call takes or what it costs.
The eval re-checks the guards
Nine of the eleven scorers on the stylist are deterministic, and at least four of those restate a guard that already runs in code: complete outfits, one outer layer and one pair of shoes, no repeated outfits, and warmth within one step of the forecast window. The warmth scorer uses the same tolerance as the stylist’s own pre-filter, as the config’s comment says. These scorers pass when the guard works. They say nothing about whether the stylist chose well.
None of the eleven reads the run’s warnings, which is where dropped picks are recorded. The eval grades the output after the guards have cleaned it, and can’t see how often they had to. The rainy-errands case shows what that hides. It requires a waterproof outer layer, the seed closet it runs on has none, and the warning that says so sits in the run where no scorer looks.
That leaves two scorers that look at judgment, both LLM judges: whether the outfits suit the occasion and the weather, and whether the written rationale is true of the items. Each runs one sample against a 0.7 threshold, on a suite set to one trial, and the config itself notes that the stylist’s answer varies from call to call. With one sample on one output, a low score could be the stylist or the judge, and I can’t tell which. I haven’t compared either judge against my own ratings of the same outfits.
The cheapest way to find out is to rate the outfits myself, and that costs my time, as the cataloguing confirmation does: once to rate the outfits and again whenever a judge’s rubric changes. The guard drop rate is already in the reports as warnings. I haven’t counted it across enough runs to say anything about it.
Note: I used AI to help with proofreading — the production experience and opinions are mine.
