Model Routing and Fallbacks
The first version of a document tool I shipped sent everything to the biggest model the provider sold. Every request. “Classify this ticket into one of five buckets” went to the same endpoint as “read these forty pages and reconcile the totals.” It worked, and working was the problem, because working is what let it stay that way for four months.
Then two bills came due. The first was money. The overwhelming majority of calls were tiny (a label, a yes-or-no, a two-line reply) and a model costing a fraction as much would have nailed every one of them. I was couriering postcards by chartered jet. The second bill was worse and it arrived all at once: the provider had a bad afternoon, started returning 529 overloaded, and the whole feature went dark. There was exactly one code path to exactly one model, and no plan B.
Routing is the fix for both, and they turn out to be the same fix wearing two coats.
Two questions, one requirement
Routing answers two different questions that people lump together.
The first is a cost-and-quality question: which model should handle this request? Send everything to the frontier model and you overpay on the easy majority. Send everything to the cheap one and you botch the hard minority. The right answer depends on the request.
The second is a reliability question: what do you do when the model you picked isn’t there? Providers rate-limit you, have outages, spike in latency, and retire models out from under you on a schedule. If your code can only speak to one model, any of those takes you down.
Both questions share one requirement, and it is the thing to get right before any of the clever parts: your code must not be welded to a single model. The moment a model choice is a value you can change instead of a rewrite you have to schedule, both kinds of routing become easy. Weld the model in and neither is possible.
Most requests are easy
Here is the observation the whole cost side rests on. Production traffic is lopsided. A big fat head of easy work (classify, extract, format, answer a short question) and a thin tail of genuinely hard work (multi-step reasoning, long-document analysis, tricky code). If you have never looked, go look, because the split is usually more extreme than you would guess. Eighty percent easy is common. Ninety-five is not rare.
An easy request does not get a better answer from a bigger model. Both the tiny model and the frontier model return “billing” for “which department owns this ticket,” so the premium you paid on that call bought nothing. The bigger model earns its price only on the hard tail. So route: cheap by default, expensive by exception.
How you decide which branch a request takes, from the dullest and most reliable to the fanciest:
- By task type, with explicit tiers per feature. You already know the extraction endpoint is easy and the planning endpoint is hard. You do not need to predict anything. You just declare it. This is boring and it is right most of the time. Start here.
- By a cheap input signal. Length, presence of code, number of steps requested, which tenant. A crude heuristic (“over 8k tokens or contains a table, go big”) catches a lot.
- By a fast classifier. A small model or an embedding-based classifier reads the request and predicts difficulty in well under a tenth of a second. Worth it only when a single endpoint genuinely mixes easy and hard and you cannot tell them apart up front.
- By a learned router. Trained on real preference data (which model actually satisfied which query), it beats hand-written rules. It is also the most machinery to build, run, and keep honest.
The temptation is to jump straight to the classifier because it feels like the “real” solution. Resist it. A static tier map keyed by feature is a dictionary lookup with no extra latency and no new failure mode, and for most apps it captures the majority of the savings.
// Each feature declares the smallest model that does its job. That is the routing.
const tiers = {
classifyTicket: "small",
extractFields: "small",
chatReply: "small",
planRefund: "reasoning", // the one request type that actually needs to think
};
function pickModel(feature) {
return tiers[feature] ?? "small"; // default down, not up
}The one rule inside that snippet worth saying out loud: default down, not up. A missing or unknown case routes to the cheap model, not the expensive one. A bug in your routing should cost you a slightly worse answer, never a surprise bill.
How much does this save? Published routing results in 2026 land in a wide but consistent band: cost cut by half to three-quarters while keeping something like 95 percent of the frontier model’s quality, because the frontier model was only pulling its weight on a small slice of traffic to begin with. Your mileage depends entirely on how lopsided your traffic is, so measure yours rather than trusting the headline. The reasoning-vs-standard tradeoff behind the “hard tail” is its own topic in reasoning models and thinking budgets, and where these tokens turn into an actual invoice is in cost and tokens.
Cascades: try cheap first, escalate on failure
Up-front routing has to predict difficulty before it has seen the answer. Sometimes it guesses wrong and sends a hard request to the weak model, which then ships a confident bad answer. A cascade sidesteps the prediction. Run the cheap model first, check its output, and escalate to the stronger model only when the check fails.
In code the shape is small:
async function cascade(input) {
const draft = await call("small", input);
if (passesCheck(draft)) return draft; // the easy majority stops right here
return call("strong", input); // escalate only when the check fails
}The whole thing lives or dies on passesCheck. And the trap is to ask the cheap model how confident it is, because models are cheerfully overconfident and a self-reported 0.95 is not a probability, it is a vibe. Prefer checks that are cheap and external to the model’s opinion of itself.
Now the catch, and it is a real one. A cascade adds latency on the hard path. When the check fails you have paid for the cheap call, the check, and the strong call, and the user sat through all three. You roughly double the latency on every escalated request. So the arithmetic only works when the escalation rate is genuinely low:
expected cost = pass_rate × cheap + fail_rate × (cheap + strong)If ninety percent of requests pass the cheap check, the cascade is close to free and you almost never touch the expensive model. If half of them escalate, you have built a slower, more expensive machine than just calling the strong model directly, plus a check in the middle that can be wrong. Look at your real pass rate before you commit.
The other half: staying up when the model isn’t there
Everything so far assumed the model you picked answers. Often it does not, and this is the half people skip until an incident teaches them.
Start with the failure that is not an accident: models get retired. As of 2026 a given model version tends to live somewhere in the 12-to-18-month range before the provider deprecates it, gives you a sunset date a couple of months out, and eventually turns it off whether or not you were ready. This is not a rare event you can ignore. It is a scheduled certainty for every model you ship on. If swapping models is a code rewrite, that email starts a fire drill. If it is a config change, it is a Tuesday.
The unscheduled failures are the ordinary ones: rate limits, capacity, blips. The first move is not to fail over. It is to retry, because most of these clear on their own in seconds. You already know this machinery from retries and backoff, so none of it is repeated here. What matters for routing is which errors deserve a retry, because retrying the wrong one is anywhere from useless to actively expensive.
A 429 is not one thing. Most of the time it means you tripped a rate limit (requests or tokens per minute) and the fix is to wait and retry, honoring the Retry-After header if the provider sends one, since that value is calibrated to the real reset window. But some 429s mean your account is out of credit, and that one is a wolf in sheep’s clothing: it looks like backpressure, so a naive retry loop hammers it and a naive failover happily runs up a second provider’s bill for a request that was never going to land. Read the body.
function classify(err) {
const s = err.status;
// A 429 that mentions money is not a rate limit. Don't retry it, don't fail over it.
if (s === 429 && /quota|billing|credit|insufficient/i.test(err.message)) return "billing";
if (s === 429 || s === 503 || s === 529) return "transient";
if (s >= 400 && s < 500) return "permanent"; // bad request, auth, validation
return "transient"; // network blips and other 5xx: worth a bounded retry
}Only once retries are exhausted on a transient error do you fall over to a different model or provider. And this is where the “not welded to one model” requirement pays off. If every model lives behind the same interface, the fallback is just the next item in a list.
The fallback loop is unremarkable, which is the point:
// Each attempt is just config: a provider and a model. Order is preference.
const chain = [
{ provider: providerA, model: "big-1" },
{ provider: providerB, model: "big-2" }, // a different vendor, on purpose
{ provider: providerA, model: "small-1" }, // last resort: cheaper, but still up
];
async function generate(request) {
let lastErr;
for (const attempt of chain) {
try {
// withRetries handles transient errors before we ever fall through
return await withRetries(() => attempt.provider.run(attempt.model, request));
} catch (err) {
lastErr = err;
if (classify(err) !== "transient") throw err; // don't fall over a bad request
}
}
throw lastErr; // everything transient-failed: time to degrade (next section)
}Two design choices in there earn their keep. The second entry is a different vendor on purpose, because a fallback to another model on the same provider does nothing when it is the provider having the outage. And a permanent error breaks the loop immediately instead of falling through, because a request the primary rejected as malformed will be just as malformed at the backup, and trying it three times wastes three round trips to arrive at the same 400.
Graceful degradation: never a hard error
Fallover buys you a lot, but it does not cover the case where every model is unreachable, or the case where the user has blown their budget, or the case where you simply decide the good answer is not worth its price right now. For those, degrade instead of failing. The same instinct as a graceful shutdown: when you cannot do the best thing, do the best available thing, and never hand the user a spinner that never resolves or a naked 500.
Each rung is a real, shippable behavior:
- Premium model. The answer you want, at full price.
- Cheap model. A visibly-fine answer when the premium one is unreachable or too pricey for this request right now.
- Cached answer. If you have a recent good answer for a similar request, serve it. Instant and free, with the caveat that a cached model response can be confidently, invisibly wrong, which is a real hazard covered in caching model responses.
- Honest wait. No answer, but a truthful “we’re a bit overloaded, try again in a moment” and a clean status. This still beats a hung request or a stack trace.
Budget is a first-class input to that ladder, not an afterthought. If a tenant or a feature has burned through its spend cap for the window, that is a routing decision made of money rather than difficulty: step them down to the cheap tier or a cached answer instead of returning an error or, worse, quietly letting them run up an unbounded bill. Per-tenant caps like this fall naturally out of multi-tenancy.
function chooseTier(feature, tenant) {
if (overBudget(tenant)) return "small"; // out of money → downgrade, don't 500 and don't overspend
return tiers[feature] ?? "small";
}The path that only runs during an incident
Here is the sentence to tattoo somewhere. Your fallback path is, by definition, the code that only runs when the primary is down. That means it runs during incidents, which is the single worst time to discover it never worked.
The reason it might not work is that different models behave differently, and not only in quality. They differ in format. The backup might wrap its JSON in prose, use a different key casing, structure its tool calls differently, or be chattier in a way your parser was never built for. So the fallover succeeds at the API level, returns a 200, and then your downstream code chokes on the shape, and now you have two incidents instead of one: the original outage plus a parsing failure in the exact code path you were relying on to save you.
And do not forget that the router itself is new machinery with its own failure modes. A classifier can be slow or down. A rule can misfire and send a hard request to the weak model. Two guardrails keep the router from becoming the thing that takes you down: put a tight timeout on any routing decision, and fail safe when the router errors, meaning default to a known-good tier rather than throwing. Routing that adds a single point of failure in the name of removing one is a bad trade.
Did routing actually save money?
Routing is a bet: cost goes down and quality stays flat. You have to prove both halves, because it is entirely possible to cut the bill and quietly cut quality with it, for instance by setting a cascade’s check too loose so easy answers degrade and nobody notices until support tickets climb.
So instrument the decision. For every request, log which model handled it, why (the routing decision and the reason), the tokens and cost, and a quality signal. That last one is the whole ballgame and it is what evals and tracing are for. With those fields you can answer the only two questions that matter: what did routing actually save, and did quality hold per tier. Without them you are flying blind and “it feels cheaper” is the best report you can file.
One measurement discipline is worth stating plainly: score the routed system as a whole, not each model in isolation. A cascade that uses a great cheap model and a great strong model can still be worse than either alone if its escalation logic sends the wrong requests to the wrong tier. The router is part of the system under test.
See it move
Below is a fully simulated router. No model is called and nothing hits the network. There is a fixed batch of forty requests, mostly easy with a hard minority, and three strategies to run them under. Flip the strategy and watch cost and quality move. Then hit Inject provider outage and watch what happens to the naive all-premium strategy compared to the two that can degrade.
Run the numbers the sim is showing and the shape of the whole article falls out. All-premium with no outage is the expensive default I started with: forty requests at premium price for quality that barely moves versus routing. Route-by-difficulty pays premium on only the hard tenth. And when the outage lands, the two strategies with somewhere to fall back stay up at a lower quality, while all-premium goes to zero, because it never had a second option to reach for.
Summary
- Sending every request to one big model quietly overpays on the easy majority and stakes the whole feature on a single provider. Routing fixes both, and both need the same thing: model choice as config, not code.
- Traffic is lopsided. Route the easy fat head to a small cheap model and reserve the big or reasoning model for the hard tail. Reported savings run to half or more of the bill at near-frontier quality.
- Start with explicit per-feature tiers (a dictionary lookup, no new latency). Reach for input signals, then a fast classifier, then a learned router only when an endpoint genuinely mixes difficulty. Default down, not up.
- A cascade runs the cheap model first and escalates only when a check fails. Use cheap external checks (schema, rules, a tiny verifier), not the model’s self-reported confidence. It only pays off while the escalation rate stays low, because escalation roughly doubles latency and cost.
- Reliability routing is the other half. Retry transient errors (429 rate-limit, 503, 529) with backoff first, honor
Retry-After, and only then fall over to a different vendor behind one interface. Do not retry or fail over permanent errors or a billing 429. - Models get retired on a cadence of roughly 12 to 18 months. Provider-agnostic code turns that sunset email from a rewrite into a config change.
- Degrade instead of erroring: premium, then cheap, then cached, then an honest wait. Never a hard error or an endless spinner. Budget is a routing input too: over budget means downgrade.
- The fallback path only runs during incidents, so test it on purpose, run evals against the backup model, and keep it warm with a canary. Fail the router itself safe.
- Prove routing worked. Log which model ran, why, the cost, and a quality signal, then measure the routed system as a whole, not each model alone.