Capacity Management: What to do when your best model runs out

Turn your evaluations into a capacity planning tool, and stop leaving throughput on the table

(Code examples in this repo, referenced throughout this post)

Capacity management is an evaluations problem

As a Principal Applied AI Architect for AWS, I specialize in helping companies scale generative AI workloads. Over the last few years I have seen one story play itself out again and again:

• We picked one model and it works. The evaluation was clean, accuracy is good, the team is happy. New models have come out that may be better but we don’t have time to evaluate them.
• We grow, and ask for a quota increase. Sometimes the request is approved, but often the answer is no or not yet. Now we’re scrambling to find alternative capacity which is limited everywhere.
• We’re stuck. We could send the overflow to another model, but we do not know which one. During development the team considered other LLM’s, but that was months ago and nobody has looked since.

The fix is not higher quotas. The fix is a different way of thinking about model selection. Rather than finding “the best model”, model selection becomes a portfolio problem. We want to find every model that works and add them to a portfolio. Adding a model means optimizing our workload uniquely for each model, giving each model its best…