Routing Requests Between Models

What model routing is

Model routing is the practice of choosing, per request, which of several models will serve it, rather than sending every request to one fixed model. The routing decision is usually made on cost, latency, or expected task difficulty, and it sits in the request path — between the caller and the provider.

The router is a component, not a config value

A router adds a decision before every call, and that decision has its own latency, its own failure mode, and its own maintenance burden. Whatever the router does — read a field, count tokens, call a classifier — happens on every request, so its cost is multiplied by traffic volume rather than amortised. Its failure mode is separate from the model’s: a router that returns the wrong model produces a plausible-looking response from the wrong tier, which is harder to detect than a timeout. Its maintenance burden is ongoing, because the routing table encodes assumptions about model capability and price that change whenever a provider ships a new model or repositions an existing one. Anthropic’s model documentation is the reference for which model identifiers and capability tiers currently exist; a routing table that names models is a table that goes stale.

If you route, the routing decision belongs in the same trace as the call it precedes, so that per-request cost and latency can be attributed to the tier that actually served it. This is the same requirement described in cost and latency attribution, applied to a decision rather than a call.

Cheap heuristics avoid paying to decide

Routing on a cheap heuristic — input length, task type declared by the caller — avoids paying for a model call to decide which model call to make. Input length is available before any network call and correlates with the work the model has to do. A task type supplied by the caller is even cheaper, because the caller usually knows what it is asking for: a code completion, a summarisation, a multi-step plan. Both are deterministic, which means they can be unit-tested and reasoned about without sampling.

The cost of a heuristic is precision. Input length does not distinguish a long, easy document from a short, hard question. A caller-declared task type is only as accurate as the caller, and drifts as calling code changes. Heuristics are the wrong answer when the routing decision genuinely requires understanding the request’s content, and misrouting is expensive enough to justify measuring it.

Classifier routing has to clear its own cost

A router that classifies with a model first can cost more than it saves whenever the saving per routed request is smaller than the classification call. This is arithmetic, not a judgement call: if classifying costs c per request and the average saving from routing a request to a cheaper model is s, routing is only worth it when s > c. Two things push s down — a high share of requests that would have gone to the cheap model anyway, and a small price gap between the tiers in the table. Both are common.

The classifier also adds a serial round trip to every request, so the latency of the classification call is added to the latency of the request it routes. That cost is paid by every request, including the ones the classifier sends to the expensive model, which save nothing. Research on LLM routing, including RouterBench, a benchmark for evaluating routing systems, frames routing explicitly as a cost-quality trade-off to be measured rather than assumed, and the router’s own overhead belongs in that measurement.

Every model in the table multiplies the evaluation surface

Every additional model in the routing table multiplies the evaluation surface, because each one needs its own regression coverage per task class. A table with three models and four task classes is twelve model-task combinations, each of which can regress independently when a provider updates a model or you change a prompt. Adding a fourth model is not a 33% increase in evaluation work; it is four more combinations to cover, and the coverage has to be maintained, not written once.

This is the practical ceiling on routing tables. The evaluation cost is the reason to keep the table to the smallest set of models that produces a measurable difference, and it is why the frontier versus mid-tier model choice is worth settling per task class before it becomes a routing decision. Evaluating any of these combinations means evaluating a non-deterministic system in CI, with all the sampling and threshold problems that implies — multiplied by the size of the table.

Fallback routing is a different mechanism, and comes first

Fallback routing on error is a different mechanism from cost routing and is worth building first, since it addresses availability rather than spend. Cost routing decides which model to call. Fallback routing decides what to do after a call has already failed — a 5xx, a timeout, a rate-limit response. The trigger is an error, not a prediction, so it has no classification cost and no accuracy question: either the call failed or it did not.

Fallback shares its failure modes with any retry mechanism, which means it needs the same discipline: bounded attempts, a timeout per attempt, and backoff with jitter so that a provider incident does not turn into a synchronised retry storm from your own fleet. AWS’s description of timeouts, retries and backoff with jitter covers why unjittered backoff clusters retries and how to spread them. When the failure being handled is a 429 rather than a 5xx, the retry policy interacts with provider rate limits and concurrency caps, and retrying immediately against the same limit makes things worse.

Building fallback first also produces the machinery cost routing needs later: a way to name more than one model, a way to select between them at call time, and a way to record which one served a request.

What to check next

• The current model identifiers and tiers in your provider’s documentation, since a routing table hard-codes them.
• Your measured price gap and misroute rate between the tiers in the table — both are inputs to whether classifier routing clears its cost.
• Retry and timeout policy for the fallback path, including per-attempt timeouts and jitter.
• Which task classes your evaluation suite covers, and whether that coverage exists for every model in the table.