Member-only story
Tiered Inference Routing: Keeping Most AI Coding Requests Off the Cloud
Tiered inference routing sends AI coding requests to on-device, on-prem, or cloud models by complexity, cutting cloud tokens while keeping quality.
This article walks through a tiered inference routing architecture for AI coding assistants: a design that sends each request to the smallest model that can answer it well, instead of sending everything to a large cloud model by default. You will see the workload data that motivates it, the on-device hardware and quantization choices that make a local tier practical, the classifier that decides where each request runs, and a step-by-step path to build the same pattern on Windows with ONNX Runtime and Foundry Local.
By the end, you will know how to measure your own request mix, route by complexity, and cut cloud token usage without giving up answer quality.
Reference here.
The routing problem
Start with one request type and follow the cost. A prompt that adds a docstring runs about 180 tokens. A lint-error fix is around 210. A generated CRUD route is roughly 460. A security audit jumps to about 3,800, and a cross-repo refactor lands near 3,900. The two heavy tasks cost roughly twenty times what the light ones do. That spread is the whole story.
