What is Model Routing?
Operations (LLMOps & AgentOps)Sending each call to the model that fits it: cheap models for classification, expensive ones for judgment. A cost and latency decision made per call, and one of the few levers that cuts spend without touching quality.
Why It Matters
Most agent runs mix call types: classification steps, extraction steps, and the occasional judgment call that actually needs the expensive model. Sending everything to the frontier model is the default because it is simple, and it is also the largest single line item most teams can cut without losing anything. Routing each call to the cheapest model that clears the task’s bar is one of the few levers that reduces spend without touching answer quality, because the judgment calls still get the model they need.
What the Router Decides On
Task class. Classification, extraction, and formatting route to small models; multi-step reasoning and judgment route to the frontier tier.
Cost ceiling. The routing table lives inside the cost bounds, so a run that cannot afford the expensive tier degrades deliberately instead of spending silently.
Eval evidence. Route assignments are claims about quality, so each routing rule goes through the eval gate like any other change: a cheap model on a step it cannot handle is a regression, not a saving.
Where It Breaks
The naive failure is routing to the cheapest model that can attempt the call rather than the cheapest that can succeed at it, which converts a cost optimization into a quality regression nobody measured. The subtler failure is a stale routing table: provider pricing and model capabilities shift monthly, and a table tuned in January quietly over- or under-provisions by June. Routing is a manifest item, which makes it a versioning and review problem, not a set-once config.
How Flytebit Handles It
Routing tables in the systems we ship are versioned with the manifest and gated by evals, so a route change is a measured change rather than a cost experiment run on production traffic. The spend context is in Operating Agentic AI Systems, and the operating model is our LLMOps work.