The most popular advice about LLM routing is also the least useful: send simple prompts to a cheap model and difficult prompts to a frontier model. That binary rule ignores the variables that decide whether a gateway works in production, including quality by task, live latency, provider health, rate limits, token cost, and router overhead.
A production router isn't a switch between “small” and “large.” It's an observable control loop. For every request, it should make a decision, record the outcome, compare that outcome with a quality and cost target, and adjust the next decision when traffic or provider conditions change. The objective is not to minimize model spend at any cost. It's to find an acceptable point on the quality × cost × latency frontier.
Table of Contents
- Why Single-Model Setups Stop Scaling
- What LLM Routing Actually Means
- Priority Ordering, Rate Limits, and Mid-Flight Retries
- Choosing the Right Provider Heuristic
- Cascade Routing and Two-Stage Policies
- Where Routing Quietly Fails
- Metrics That Tell You the Router Is Working
- Bringing It All Together
Why Single-Model Setups Stop Scaling
A single-model integration is attractive because it keeps the application simple. One SDK, one credential, one request shape, and one set of failure behaviors are easy to operate while a product is still proving demand. The limitations appear when different request types begin competing for the same model budget.
A coding task, a long-form summary, and a structured extraction request don't place the same demands on an LLM. One model may handle code reliably but produce inconsistent JSON. Another may be strong at summarization but introduce unnecessary latency on short classification requests. A premium model can deliver excellent answers, yet using it for every low-risk request turns quality headroom into avoidable spend.
Research on learned routing makes this trade-off measurable. A 2025 ICLR study evaluated routing with benchmarks including MMLU and MT-Bench and reported cost reductions of more than two times without substantially reducing quality. Its comparison estimated average inference expense at approximately $24.70 per million tokens for GPT-4 and $0.24 per million tokens for Mixtral 8x7B, a roughly 103x price difference. Those figures come from the study's comparison, not from a universal pricing rule, but they show why sending every request to the same premium tier is a poor default. The ICLR routing study also reported maximum savings of 3.66x from learned routing policies.
The three-way constraint
Quality, cost, and latency pull the gateway in different directions.
- Quality: Different models have different strengths, and aggregate benchmark quality can conceal task-specific weaknesses.
- Cost: Token prices vary substantially, and unnecessary reasoning or output tokens can make a seemingly small request expensive.
- Latency: Providers differ in queueing, time to first token, streaming behavior, and tail performance. A model that looks fast in a quiet test can still violate a user-facing SLO under load.
Provider outages add a fourth operational concern. A single upstream failure shouldn't take down an application that could have served the same capability elsewhere. Failover and model selection therefore belong in the platform layer, where the gateway can see provider health, quotas, and historical outcomes.
The broader benchmark picture reinforces this point. RouterEval assembled more than 200 million performance records across 12 evaluations and over 8,500 LLMs, showing how narrow averages can obscure task-specific strengths. The RouterEval benchmark discussion frames routing as a performance-cost-latency optimization problem rather than a simple model-size decision.
Practical rule: Don't ask which model is best in the abstract. Ask which route satisfies the quality floor for this intent, within the latency budget and spend limit available right now.
What LLM Routing Actually Means
Think of a gateway as an air-traffic controller for inference. Requests arrive at one interface, and the controller decides which provider, model, and service tier should handle each one. It then returns a response through the same interface while recording the route, timing, token usage, errors, and final outcome.
That decision can be simple or learned. A keyword rule might identify a request that requires a tool. An embedding model might compare the prompt with examples from several task classes. A classifier might predict whether a small model will meet the quality floor. A cascade might let a first model answer and escalate only when an evaluator finds the answer insufficient.
The word “routing” also has an older meaning inside model architecture. Sparse mixture-of-experts systems route tokens to specialized computational experts rather than activating the entire network for every token. The Switch Transformer used a top-1 strategy, sending each token to a single expert. Its reported experiments found pretraining speeds of up to 7x over a dense T5-Base model at comparable floating-point operations per token, while capacity factors between 1.0 and 1.25 kept token loss below 1% when routing was balanced. This overview of Switch Transformer and mixture-of-experts routing describes the capacity and load-balancing constraints behind that design.
Internal and external routing
Internal expert routing answers, “Which subnetwork should process this token?” External application routing answers, “Which complete model or provider should process this request?”
The two problems share a dispatch principle, but external routing has more operational context:
- Billing: The gateway must attribute input, output, and reasoning tokens to a route.
- Provider health: A model can be available in theory while its upstream is timing out or rate-limiting.
- Quota state: Requests may need to move between providers serving the same capability.
- Latency: The router must account for both its own decision time and the selected model's response time.
- Policy: Some intents require a stronger model, a specific region, a tool-capable endpoint, or a fallback path.
This article focuses on application-layer request routing across multiple upstream LLM APIs. It isn't about selecting individual experts inside a model. The practical unit is a request, and the gateway's job is to dispatch that request while making the decision auditable.
Priority Ordering, Rate Limits, and Mid-Flight Retries
A useful provider policy starts with preference order, then changes that order when reality changes. A team might prefer a local or small model for routine requests, a mid-tier model for ambiguous work, and a frontier model for high-risk reasoning. Those tiers are policy categories, not fixed traffic quotas. A request should move upward when its intent, confidence score, history, or output validation says the lower tier isn't adequate.
Under load, the router should combine three inputs:
- Preference order, which expresses product policy and model fit.
- Quota budget, including current requests-per-minute and tokens-per-minute headroom.
- Observed health, including recent errors, timeouts, queueing, and latency tails.
A provider that normally ranks first may temporarily rank below a sibling because its quota is nearly exhausted or its error rate has risen. This is why rate limits should be treated as pooled capacity where the same model is available through multiple upstreams, rather than as isolated counters that leave usable capacity stranded.
Openbase rate-limit documentation is a practical reference for designing this kind of provider-aware gateway behavior.
Signals that change the route
| Signal | Source | Effect on Decision |
|---|---|---|
| Model preference | Product policy and evaluation history | Establishes the normal provider order |
| Token headroom | Provider quota telemetry | Avoids sending a request to a route likely to be throttled |
| Recent error rate | Gateway response logs | Temporarily reduces route priority or opens a fallback |
| Time to first token | Client-edge measurements | Protects interactive requests from slow upstreams |
| Request intent | Classifier, embeddings, rules, or metadata | Selects an appropriate model tier |
| Budget state | Per-request and aggregate accounting | Enables escalation or cost-aware downgrade policies |
Retries become harder once a response has started streaming. A cold-start retry can resend the request after a connection failure before any output reaches the client. A mid-flight failure is different. The client may already have rendered partial text, consumed tool-call fragments, or committed to a response state.
The gateway needs an explicit policy for that case:
- Resume, when the protocol and upstream support safe continuation.
- Replay, when the request is idempotent and the application can tolerate regenerated output.
- Fallback, when a second provider can finish the request without confusing the client.
- Terminate cleanly, when replay could duplicate an irreversible tool action.
A retry policy must also distinguish transient failures from client errors. Rate limits, timeouts, and upstream server failures can justify another route. Invalid request parameters and unsupported resources generally require correction, not repetition.
A provider router should make a worse provider available before the first provider fails, not discover the fallback only after the product has already timed out.
Choosing the Right Provider Heuristic
The right heuristic depends on the gateway's SLO. A router that adds substantial decision latency can erase the savings from selecting a cheaper model. A router that decides instantly but misclassifies difficult requests can reduce cost while increasing retries, escalations, and user-visible errors.
The main options form a progression from predictable rules to increasingly expensive inference.
| Heuristic | Latency Overhead | Accuracy Recovery | Best Fit | Failure Mode |
|---|---|---|---|---|
| Keyword or metadata rules | Very low when implemented locally | Strong for explicit, stable intents | Tool selection, known formats, hard policy constraints | Paraphrases and unfamiliar phrasing bypass the rule |
| Embedding router | An analysis of routing systems estimates roughly 5–15 ms for embedding routers, depending on implementation and environment | Better semantic coverage than literal matching | Repeated intent classes with representative examples | Embeddings can drift from live traffic and hide uncertainty |
| Classifier router | The same analysis estimates roughly 10–50 ms for classifier-based routing | Often more consistent when trained on labeled task data | Stable production taxonomies and clear quality labels | Distribution shift degrades decisions without retraining |
| LLM-based router | The analysis estimates roughly 200–800 ms for LLM-based routing | Can reason about nuanced task differences | High-value, low-volume requests where decision quality dominates latency | Decision cost and delay can cancel the downstream savings |
The RouterEval analysis supplies the cited routing-overhead ranges and emphasizes that workload-specific capability matters more than a universal model ranking.
Match the heuristic to the request path
Rules work well when the signal is explicit. If a request carries a known operation type, requires a particular tool, or has a strict schema, a deterministic policy is easier to test than a semantic router. Rules fail when users express the same intent in many ways, so they should not be stretched into a general-purpose classifier.
Embeddings offer a useful middle ground. They can compare a new request with curated examples without requiring a large supervised dataset. The cost is operational: teams need to monitor confidence, refresh examples, and test new prompt styles against the old route.
Classifiers are appropriate when the organization has stable labels and evaluation data. They provide a clear place to set thresholds, but they aren't self-correcting. A change in user behavior, product features, or prompt templates can invalidate the training distribution.
LLM-based routing is the most tempting and the easiest to misuse. Asking a strong model to select another model can improve nuanced decisions, but it adds a complete inference step before the actual response begins. Use it selectively for expensive or high-risk requests, not as the default gate for every interactive call.
Your model catalog also needs explicit capability metadata. Track context limits, tool support, structured-output behavior, reasoning characteristics, provider availability, and measured latency alongside price. Openbase model documentation illustrates the kind of model-level information a gateway needs to expose consistently.
Cascade Routing and Two-Stage Policies
A cascade begins with an initial model and escalates only when the first answer appears insufficient. The first model may be selected by a rule, classifier, or embedding router. A judge then checks signals such as schema validity, confidence, retrieval support, tool-call correctness, or task-specific evaluation results.
The important difference from pre-generation routing is evidence. A selector must predict difficulty from the request alone. A cascade can inspect the first response before deciding whether additional compute is justified.
A RouterBench study evaluated routing with more than 405,000 precomputed inference outcomes from 11 models across seven tasks, including MMLU, MT-Bench, MBPP, HellaSwag, WinoGrande, GSM8K, and ARC. It found that comparable performance could carry serving costs varying by roughly 2–5x across models. KNN and MLP routers reached performance comparable to strong individual models at lower or similar cost, while cascade routers improved as the permitted budget increased. RouterBench's evaluation supports treating routing as a measurable cost-quality frontier.
Build two decisions, not one
A practical two-stage policy separates where to start from whether to continue.
- Stage one, pre-generation selection: Choose a model family based on intent, required capabilities, current provider health, and the request's latency budget.
- Stage two, post-generation escalation: Validate the response and escalate if the confidence score, schema check, judge result, or task-specific signal falls below the required threshold.
- Budget control: Tighten or relax escalation based on remaining spend, queue pressure, and the error cost of the request. A customer-facing answer and an internal draft don't need identical boundaries.
- Outcome recording: Store the first route, escalation reason, final route, total tokens, latency, and final quality result. Without this record, threshold tuning becomes guesswork.
The cited cascade study reported consistent quality gains of 1–4 percentage points over its baseline across settings, representing approximately 13–80% relative improvement over a naive routing strategy. The cascade-routing research attributes the advantage to budget adaptivity, because the system spends extra compute only after seeing evidence from the first response.
A cascade judge can still become a bottleneck. If it escalates too readily, spend and latency rise. If it accepts weak outputs, the system preserves savings by lowering quality. Calibrate thresholds against the cost of each failure, not against an abstract global average.
The cascade architecture is useful because it gives the gateway a fast path and an escalation path. It doesn't eliminate the need for measurement. It makes measurement more actionable.
Where Routing Quietly Fails
Routing can make a system worse while the cost dashboard looks healthier. The most common reason is that teams measure the selected model's nominal price but not the complete decision path.
The router itself consumes compute and adds time. A classifier may look inexpensive in isolation, yet its overhead matters on a latency-sensitive endpoint. An LLM-based decision can be especially damaging when the downstream request is short, because the gateway spends a large amount of time deciding how to save a small amount of inference cost.
Three failure patterns
Distribution shift is the quiet failure. A router trained on support tickets can misroute a new product's agentic requests, even when the model's aggregate validation score remains stable. Monitor route composition and quality by intent, not just overall quality.
Adversarial prompts create a different problem. A user who understands the routing features may deliberately phrase a request to reach a weaker or cheaper route. Treat routing signals as an attack surface, especially when the selected model can access sensitive tools or data.
Unobserved counterfactuals hide the opportunity cost. If every request is sent to the route chosen by the router, the team doesn't know whether a different provider would have produced a better answer at acceptable cost. A control sample, shadow evaluation, or frozen comparison set provides that missing reference.
Openbase error documentation is relevant when designing the error taxonomy behind these safeguards. The gateway should distinguish provider failure, router failure, validation failure, and application failure rather than collapsing them into one retryable error.
Run policy changes in shadow mode before making them authoritative. Log the proposed route, the production route, and a control-model result where the evaluation budget permits. Watch for sudden changes in escalation rate, route distribution, judge disagreement, validation failures, and latency tails.
A router that isn't tested against traffic it didn't train on is making a confident guess, not operating a control loop.
Metrics That Tell You the Router Is Working
A dashboard without targets is decoration. Before deploying a routing policy, define what acceptable quality, cost, and latency look like for each request class, then attach an alert to a sustained breach rather than a single noisy observation.
Start with per-call accounting. Record input tokens, output tokens, reasoning tokens when available, selected model, provider, cache status, route reason, and exact cost. Aggregate spend by intent and route, because a stable total can conceal one expensive class growing rapidly while another class declines.
Measure latency from the client edge, not only inside the provider adapter. Time to first token captures the user's first wait, while end-to-end p95 captures the tail that determines whether an interactive feature feels reliable. Include router decision time, queue time, upstream time, retries, and escalation in the total.
The operating scorecard
| Metric | Target Band | Alert Trigger |
|---|---|---|
| Per-call cost | Within the approved budget for each intent | Sustained cost growth, route mix change, or unexpected token increase |
| Time to first token | Inside the interaction SLO | Persistent breach after separating router and provider time |
| End-to-end p95 latency | Within the endpoint's latency budget | Sustained p95 over budget, especially on the fast path |
| Escalation rate | Within the control band for each task class | Drift beyond the baseline band or an abrupt drop suggesting silent acceptance |
| Fallback frequency | Near the established provider-health baseline | Climbing frequency over the weekly operating baseline |
| Quality score | Above the task-specific floor | Frozen-eval or sampled-production quality falls below the floor |
| Judge disagreement | Stable against a frozen evaluation set | Disagreement rises after a policy, prompt, or provider change |
Escalation rate deserves special attention. A rise can mean the first model is failing, the judge has become stricter, or traffic has changed. A sudden fall can be equally concerning, because it may mean the judge is no longer detecting weak responses.
Fallback frequency should be segmented by provider and error class. A rise in 429 responses points to quota pressure. A rise in timeouts points to latency or upstream saturation. A rise in validation failures may indicate that the route is selecting a model with the wrong structured-output behavior.
Keep an independent quality signal. A judge model can help, but it can also share blind spots with the models being evaluated. Use a frozen eval set, task-specific validators, and sampled human review where the failure cost is high. The router should optimize against observed outcomes, not only against its own confidence score.
Bringing It All Together
LLM routing works when the gateway closes a feedback loop. It observes the request, selects a route, measures quality, cost, latency, and failure behavior, then uses those observations to refine the next decision. That is fundamentally different from hard-coding “cheap for easy, frontier for hard.”
Three failures repeatedly undermine production systems:
- Unmeasured router overhead, which can erase downstream savings.
- Silent distribution shift, which causes old routing assumptions to misclassify new traffic.
- Missing budget-adaptive cascade boundaries, which either waste spend through excessive escalation or allow weak answers through.
The most valuable instrument is a frozen evaluation set scored independently, run regularly against every provider and model the gateway can select. Keep the prompts stable, preserve historical results, and compare quality, cost, latency, escalation, and fallback behavior after every routing-policy change.
A multi-provider gateway such as Openbase can provide a consistent OpenAI-compatible interface across providers, automatic failover, and per-call records for models, providers, tokens, latency, time to first token, and cost. Use those records to build the control loop, then validate every routing change against your quality floor before expanding it to production traffic.