Typed LLM Routing in Rust: Cut Cost Without Cutting Quality

The most expensive way to save money

Teams decide to “route requests to cheaper models” and then — almost reflexively — implement the router with another LLM. Now every request pays for a frontier call to decide which frontier call to avoid. The savings evaporate into the decision itself. I have watched a routing layer become a new cost center for exactly this reason.

Get André Claudino’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Routing is not a reasoning task. It is a classification task: given a request, predict which model tier will satisfy it. Done right, routing is one small forward pass — and if that router is written in Rust, it is one of the cheapest, most predictable components in your entire stack. That is the idea behind typed-lm.

Routing is a classification problem

Reframe the problem and the engineering becomes obvious:

• Input: the request (prompt, features, or an embedding).
• Output: a typed decision — a model tier, a routing label, a confidence.
• Constraint: the router must cost a rounding error compared to the models it selects.

The moment you write the output as an LLM-generated string, you have added latency, tokens, a parse step, and a new failure mode. A typed output is a plain enum: matchable, testable, and cheap.

Typed routing: one forward pass, no LLM call

The design that works:

1. Encode the request into a compact feature or embedding.
2. Run a single forward pass through a small classifier — no autoregressive decoding, no prompt, no token tax.
3. Emit a typed label (e.g. Small, Medium, Frontier) plus a confidence.
4. Fall through to escalation when confidence is low. Cheap is the default; frontier is the exception.

Because it is a forward pass, latency is bounded and roughly constant. Because the label is typed, the routing decision is impossible to mis-parse — the type system rejects nonsense at compile time.

Why Rust for the router

The router sits on the critical path of every request, so its properties matter more, not less:

• No GC pauses — a routing decision that occasionally spikes defeats its purpose.
• Small and embeddable — one static binary next to the gateway or the serving tier.
• Zero-copy between stages — borrow the embedding instead of copying it per request.
• Composable with the inference stack — the same Rust process can run the router and, where relevant, the small model itself, using Candle (typed-lm is built on it).

Train in Python, serve in Rust

This is the same split that works everywhere else in ML:

• Training stays in Python. Frameworks, notebooks, the fast research loop.
• Export the router through ONNX/Safetensors.
• Serve in Rust — decode with serde-onnx, run the forward pass, emit the typed label.
• Version the router like a model, because it is one. Its decisions gate user-visible quality.

The economics, with real numbers

Routing only counts if the numbers are honest. The public evidence:

• RouteLLM (2024–2026) reports up to ~85% cost reduction while retaining ~95% of GPT-4-level quality on selected model pairs — the strongest argument that routing is a real lever, not a theory.
• Inworld (2026) estimates a routing-vs-non-routing cost difference on the order of $200k–$400k per month (report estimate at their scale).
• SLM industry reports (2026) put self-hosted small models at roughly ~90% cheaper than frontier models for high-volume, repetitive work.
• My own platform: domain SLMs plus intent-based routing cut analytics/LLM cost by ~77%.

The pattern is consistent: route by difficulty, default to cheap, escalate deliberately.

Failure modes and guardrails

• Silent quality drift. A router optimized for cost will happily send too much to the small model. Gate promotions on evals, and monitor quality per route — not only in aggregate.
• Confidence collapse. If every request comes back low-confidence, you have rebuilt the frontier tax with extra steps. Tune the threshold against a held-out set.
• Underestimating the tail. A router that is fast on average but slow at p99 is a liability. Watch the tail.
• Treating routing as static. Traffic shifts. Retrain and re-version the router like any other model.

Takeaways

• Routing is classification, not reasoning — never spend an LLM call to decide an LLM call.
• A typed, single-forward-pass router is cheap, bounded, and parse-proof by construction.
• Rust makes the decision-path properties explicit: no GC, small binary, zero-copy stages.
• Train the router in Python, serve it in Rust, and prove the win with evals and your own numbers.

Sources

• RouteLLM / model-routing benchmarks, 2024–2026 (~85% cost reduction at ~95% of GPT-4-level quality on selected pairs).
• Inworld, 2026 (routing vs non-routing cost difference ~$200k–$400k/month — report estimate).
• SLM industry reports, 2026 (self-hosted small models ~90% cheaper at high volume for repetitive work).
• Own production experience: domain SLMs + intent routing cut analytics/LLM cost by ~77%.
• typed-lm — typed, single-forward-pass LLM routing in Rust (project documentation).

Originally published on LinkedIn · More at my blog · Open source at GitHub.