Agentic Workflow decomposition
Matching each step to the cheapest thing that can do it
Three recent engineering posts describe the same move from three directions: stop asking one large model to do an entire job. This post compares them, checks their numbers, and lists what a practitioner can reuse.
tldr;
- Amazon replaced Claude Sonnet 4.6 with a small model for ticket labeling. Per-ticket cost fell from about $0.0222 to about $0.0005, and latency fell 78%. Fewer tokens explain only part of that saving; the rest comes from how the small model is billed.
- TypeSafe argues that a “decision” is a mix of fuzzy judgments (give to a model) and exact logic (give to code), and that keeping them apart makes systems observable.
- Google’s Elephant-Goldfish plugin splits work by context: one long-lived session that holds the design, and stateless sessions that review it. This is decomposition, but not about cheaper models.
- None of the three posts publishes accuracy numbers. Treat the results as strong leads, not settled results.
1. Labeling tickets at Amazon
Amazon and AWS teams use an internal ticketing system that is one of the largest in the company. Tickets arrive as customer escalations, capacity requests, automated alarms and security findings. Each must be given one of 20 category labels.
The original setup sent every ticket to Claude Sonnet 4.6 with a long system prompt. The model reasoned against the label list and returned JSON. Two engineers replaced it with Strands Decider, a model that cannot write text. It is given a state (the ticket) and a question with options (the 20 labels, plus “none”), and returns a probability for each option and a confidence score. They used the E4B Gemma 4 build, after it beat the original 2B Decider on a test set, and tuned how confident it had to be before a label was applied.
Measured on the same tickets:
The Lambda function around the call also got shorter, from 3,312 ms to 748 ms on average (74%).
The term for this class of model is “decision model”: given a state and a set of options, it returns a probability for each option and nothing else. TypeSafe, covered below, calls the same kind of model a “System One” model, after the fast, intuitive mode of thinking in Kahneman’s Thinking, Fast and Slow. The label describes the role: a quick, focused judgment, with no long reasoning chain.
2. Where the saving comes from
The headline is “98% less cost”, but the post’s own numbers split it into two separate effects.
Effect 1: fewer tokens. Sonnet’s cost per ticket is $3 per million input tokens and $15 per million output tokens: 6,554 input tokens cost about $0.0197 and 170 output tokens about $0.0026, which gives $0.0222. Input is 88% of that cost, so the shorter prompt matters far more than the one-token output. If Decider’s token counts (2,009 in, 1 out) were billed at Sonnet’s prices, a ticket would cost about $0.0060. That is a 73% saving, from tokens alone.
Effect 2: a different billing model. The remaining 25 points come from paying per instance-hour for a small self-hosted model instead of per token. Decider ran on an `ml.g5.xlarge` SageMaker instance (one A10G GPU, $1.408 per hour), using about 19 GiB of its 24 GB, at about 640 ms per request. That is about 5,600 tickets per hour, or $250 per million tickets if the instance is never idle. The authors budget headroom for bursts and land on about $500 per million.
This has a consequence the post does not spell out. Per-instance pricing only wins if the instance is busy. At the same $1.408 per hour (our calculation, using the post’s 640 ms per request):
If one instance stays on all day, it costs $33.79 a day. Sonnet costs the same at about 1,520 tickets a day, roughly 63 an hour. Below that volume, the API is cheaper. (The post says the endpoint autoscales but does not give a minimum, so this assumes one instance is always up.)
The post says the change “will save millions of dollars of token costs in a year”. At $21,700 saved per million tickets, a $1 million saving needs roughly 46 million tickets a year. The post does not state ticket volume, so that figure is only an inference. It does suggest the ticket volume is very large, which is also why this case works.
3. The same idea, cut three ways
The Amazon case splits the workflow by the kind of model. Two other recent posts split by other criteria.
Cuts 1 and 2 replace some of the work with a cheaper kind of computation. Cut 3 does not. The Google post never says its stateless sessions use smaller models; what it changes is what each session is allowed to know. It is included here because it is the same engineering habit (split the work, then give each piece the right setup), but it should not be filed under “System One”.
4. Cut 1: inside a decision (TypeSafe)
TypeSafe builds a model of this kind called Jev. Its post, “Don’t call it a decision model”, argues that the usual mental picture is wrong. People imagine handing a model all their state and asking, “what should support do with this conversation?”, then stuffing the criteria field with caveats until edge cases work. TypeSafe calls the result prompt gymnastics: each new input the system fails on becomes another paragraph of clarification.
Its alternative starts from the logic the system already knows. For a customer-support workflow the exits look like this:
Some of these conditions are arithmetic or lookups (“claim eligible”, “$25 or less”, “subscription active”). Code answers those exactly. Others need judgment (“does the user want money back?”, “is this a legal threat?”). Only those go to the model, as small, separate questions. The rules then combine the answers.
The practical gain is the trace. With the single big question, a misrouted case leaves one probability and nothing to read. With the split, the log shows each judgment and each code-computed fact, and which rule fired (abridged from the post’s example):
exit fraud_team
g_fraud reports an unauthorized charge (p > 0.6) true <- decided
judgments (model)
wants_money_back 0.99
reports_unauthorized 0.94 <-
requests_human 0.02
threat_legal 0.01
wants_to_cancel 0.02
open_to_offer 0.22
facts (code)
refund_eligible false, small_refund false, subscription_active falseHere the customer clearly wants money back (0.99), but the unauthorized-charge judgment (0.94) fires first and sends the case to the fraud team. If that was the wrong exit, the trace shows whether the judgment was wrong or the rule order was. TypeSafe lists two more benefits: clearer design, because every atomic judgment and logic path is written down; and extendability, because judgments are evaluated in isolation, so adding a “waive the fee” path means one new judgment and one new code fact, with the validated ones untouched.
Get Anukriti Ranjan’s stories in your inbox
Join Medium for free to get updates from this writer.
On the name. TypeSafe prefers “System One model” over “decision model” because the model does not decide. “Jev makes judgments, code makes decisions.” This post follows that usage: “System One” for the judging role, whatever the vendor calls the product.
How this relates to the Amazon design. The two posts look like they disagree. Amazon’s Decider makes the whole decision in one call: one ticket, one choice from 20 labels. TypeSafe calls that naive. The difference is the task. Labeling is itself one judgment (“which category is this?”) with no branching business rules behind it. The support workflow has rules ($25 thresholds, subscription state) that code handles better than a prompt. A useful signal that a workflow belongs in TypeSafe’s case, not Amazon’s, is when the option descriptions keep growing caveats. That is the point to move the rules into code. This is an interpretation of the two posts; neither states it.
5. Cut 3: Elephant and Goldfish (Google)
Google’s Elephant-Goldfish Model (EGM) plugin targets AI-assisted software development. Its starting point is that AI coding tools raise the speed of producing code, and also of producing bugs and “context collapse” in long sessions. The authors’ phrase is that without guardrails, teams are “auto-scaling our mistakes”.
EGM splits the work into two kinds of session:
- The Elephant is a long-lived session that holds the project’s architecture and memory. It writes no code. It interviews the developer, debates trade-offs and drafts a markdown design document with the problem, the plan, alternatives considered, and the specific files to change.
- Goldfish are stateless sessions with zero context, used only to validate. Because they share none of the designer’s assumptions, they act as independent critics.
The plugin’s review step runs three checks:
1.Comprehension check. A Goldfish reads the design and explains the architecture and intent back. If it cannot, the document is ambiguous and needs more drafting.
2. Antagonistic review. Several Goldfish act as skeptical reviewers, looking for edge cases, hidden dependencies and logical gaps. Their critiques go back to the Elephant.
3. Implementation readiness. A final Goldfish does a go/no-go check that file paths, data structures and function signatures are all present and unambiguous.
The plugin is open source (Apache 2.0), was built for Google engineers, and the post says thousands use it daily. The part most easily reused is the comprehension check: any specification, prompt or runbook can be handed to a fresh reader with no context, and asked to explain it back. Gaps in the explanation show where the document relies on knowledge it never wrote down.
6. What is new, and what is not
The idea of using cheaper models where they suffice predates these posts. Three papers (none of them in the source set) show the lineage:
-FrugalGPT (Chen, Zaharia, Zou, 2023) described three ways to cut LLM cost: prompt adaptation, LLM approximation and LLM cascades. Its cascade learns which models to try, in what order, and reports matching the best single model with up to 98% less cost on its benchmarks.
- RouteLLM (Ong et al., 2024) trains routers that choose per query between a stronger and a weaker model, reporting cost cuts of more than 2x in some cases without a quality loss.
- “Small Language Models are the Future of Agentic AI” (Belcak et al., NVIDIA, 2025, revised September 2026) argues that agents make many repetitive, narrow model calls, which small models can handle more cheaply, and that where general conversation is still needed, systems calling several different models are the natural design.
Compared with these, the three posts here differ in what they split and what they change:
-Cascades and routers decide which generative model answers a whole query. Decider replaces generation itself with a closed choice that returns probabilities, so the output type changes, not just the model.
- TypeSafe changes where the boundary sits: the judgment goes to a model, and everything exact stays in code, with the logic kept inspectable.
- EGM uses separate sessions for a different reason: independence from the designer’s context, not cost.
Offloading minor tasks to sub-agents” is the nearest familiar pattern, and EGM’s Goldfish are a form of it. But a sub-agent is usually another full LLM with its own context; the Decider and Jev cases remove the generative step altogether. Trained text classifiers have also labeled tickets for years (general background, not from the sources). The posts do not discuss why small models are now good enough to take a plain-language list of criteria and choose between them, so these sources cannot say why this became practical now. The one related data point is Amazon’s: the E4B build clearly beat the 2B original on their tickets, which suggests model capability still mattered.
7. What a practitioner can reuse
1. Sketch one decision before changing anything. TypeSafe’s exercise is simple: write down the rules in your system, mark which are lookups or arithmetic, and write a question next to each place that needs judgment. If the list of judgments is short and each one can be answered from a bounded piece of context, the step is a candidate.
2. Prefer a closed choice with short options. To fit Decider’s 4,096-token window, Amazon rewrote each of its 20 label descriptions as one or two sentences, added a “none” option, and dropped the oldest comments on long tickets. The task changed along with the model, which matters when comparing results.
3. Test for input-format sensitivity. Sending the same ticket with JSON keys in a different order moved Decider’s output probabilities by up to 0.19. Amazon now sends every request with sorted keys. If you serve such a model behind an API, check this before comparing runs.
4. Run both models on real traffic before switching. Amazon’s labeling Lambda reads a three-way setting from AWS AppConfig on every message:
In `DUAL` mode both models run and Sonnet’s result is applied while the two are compared. The authors say judging every disagreement on real traffic told them more than an offline evaluation would have, and going back was a 45-second config change. This part transfers to any replacement of one model by another, whatever the model.
5. Log judgments and facts separately. Whichever model you use, store each small judgment with its probability and each code-computed fact next to the final exit. This is what makes a wrong exit debuggable.
6. Do the volume arithmetic. Work out cost per ticket at your real utilisation and your real volume, not at full load. With the post’s prices, the small model is the cheaper option above roughly 1,500 tickets a day on one always-on instance, and the gap widens as the instance fills.
7. Use stateless reviewers for specs. If an artifact will be handed off (a design document, a prompt, an agent instruction file), run the comprehension check from section 5 before the handoff.
Where Amazon says this does not apply. The authors report that the approach worked because the task was a closed choice from a known catalog, the volume was high, and latency on the old path mattered. If labels need explanation text, the catalog changes weekly, or the useful context cannot fit in a few thousand tokens, they expect a decision model probably cannot replace the LLM’s job.
8. What the sources do not tell us
-No accuracy numbers from Amazon. The post says Decider “performed just as well” and that E4B beat production Sonnet on a separate set of tickets, but gives no agreement rate, no per-label results and no results from the `DUAL` comparison.
- The baseline is not the cheapest alternative. Sonnet 4.6 with a 6,500-token prompt is a heavy baseline. A smaller hosted LLM with a short prompt and structured output was not tested in the post, and neither was a conventional trained classifier. Part of the saving may be reachable without a decision model at all.
- The comparison is not like-for-like. Labels were rewritten, a “none” option was added, and old comments were dropped on long tickets.
- Costs are list prices only. They exclude the Lambda, engineering time and evaluation work. The 4,096-token window also limits how much of a long ticket the model can see.
- TypeSafe’s post is a vendor argument with no benchmarks. The observability claim is a sound design argument, and an easy one to test in your own system, but the post does not measure it.
- Google gives no numbers. The only evidence for EGM’s effect is that thousands of engineers use it.
The pattern across all three is consistent, and each part is cheap to try in a small way: write the rules down, find the real judgments, run the replacement beside the original, and compare the disagreements. The sources support the idea; your own data will show whether it holds for your workflow.
References:
- Onstott, Jaiswal, Strands Agents Team. “Amazon saves millions by replacing Sonnet 4.6 with Strands Decider.” Strands Agents, 9 Nov 2026. strandsagents.com/blog/amazon-saves-millions-replacing-sonnet-with-strands-decider/
- Laabs. “Don’t Call It a Decision Model.” TypeSafe AI blog, 3 Nov 2026. typesafe.ai/blog/dont-call-it-a-decision-model
- Suchy, Quinn, Ewing, Newth, Rensin. “Introducing the Elephant-Goldfish Model plugin: scaling design-driven AI software development.” Google Open Source Blog, 22 Oct 2026. opensource.googleblog.com/2026/10/introducing-elephant-goldfish-model-plugin.html - Chen, Zaharia, Zou. “FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance.” arXiv:2305.05176, 2023.
- Ong et al. “RouteLLM: Learning to Route LLMs with Preference Data.” arXiv:2406.18665, 2024 (v4, Feb 2025).
- Belcak et al. “Small Language Models are the Future of Agentic AI.” arXiv:2506.02153, 2025 (v3, Sept 2026).
Calculations that are not in the sources (marked in the text): the 73% token-only saving, the utilisation table, the break-even volume and the volume implied by “millions”. They use only the prices and token counts printed in the Amazon post.
