StackLift
All terms

Glossary

Model routing

Sending each AI request to the model best suited to it, by difficulty, cost, speed or data rules.

Model routing is the technique of choosing, for each request, which AI model should handle it. A router looks at the task, such as its difficulty, topic, length or data sensitivity, and sends it to a small, fast model or a larger, stronger one. Teams use it to control cost and latency, add fallbacks when a provider fails and keep sensitive data on approved models.

A simple router is a set of rules: classification tasks go to a small model, long reasoning goes to a frontier model. More advanced routers use a classifier or a small model to score difficulty. Many gateways also add automatic fallback to a second provider.

Routing adds a moving part, so it should earn its place. It pays off when traffic is high and tasks vary in difficulty. Test the router on your own data, since a bad routing choice quietly lowers quality.

Start in 60 seconds

Put this to work on your project.

Get an AI-assisted, architect-reviewed scope, cost range, and timeline. Free, no login.

The complete guide

Model routing: the complete guide

Updated November 18, 2026 ยท Published by , Founder, StackLift AI. Written with AI assistance; how we make content.

Short answer

Model routing is choosing, for each request, which AI model should handle it. A router looks at the task, such as its difficulty, topic, length or data sensitivity, and sends it to a small, fast model or a larger, stronger one. Teams use it to control cost and latency, add fallbacks when a provider fails and keep sensitive data on approved models.

What is model routing?

Not every request needs your strongest model. Asking "what are your opening hours?" and asking "compare these two contracts and flag conflicting clauses" are very different jobs, yet many applications send both to the same large, expensive model. Model routing fixes that by deciding, request by request, which model should do the work.

A router sits between your application and the models. It reads the request, applies a rule or a small classifier, and forwards the request to the chosen model. The application gets one answer back and does not need to know which model produced it.

Routing is also called LLM routing or model selection. It often lives in an AI gateway, a service that gives every model one common interface, plus logging, limits and fallbacks.

Why would you use model routing?

Reasons to route
GoalHow routing helps
Lower costEasy tasks go to cheaper models, hard ones to stronger models. See AI cost optimization.
Lower latencySmall models answer faster, which matters in chat and voice
Higher reliabilityIf one provider is slow or down, the request goes to another
Data rulesRequests with sensitive data stay on approved or self-hosted models
Fit for purposeCode, images, long documents or other languages go to the model best at them
Freedom to changeSwap a model without rewriting the application

Model prices and abilities move quickly. A routing layer lets you adopt a cheaper or better model for one task type without a rebuild. Compare current prices on the LLM pricing page.

What kinds of routers are there?

Routing approaches, from simple to advanced
ApproachHow it decidesGood forWatch out for
Rule-basedFixed rules on task type, user, feature, length or keywordsClear, predictable splitsRules go stale as traffic changes
Classifier-basedA small model or classifier labels difficulty or topicMixed traffic with varied difficultyThe classifier needs its own testing
CascadeTry a cheap model first; escalate if confidence or a check failsHigh volume where most tasks are easyExtra latency when it escalates
Fallback chainTry the main model; on error or timeout use the nextReliability across providersFallback models may answer differently
Policy-basedData class, region or contract decides the allowed modelsRegulated or sensitive dataNeeds accurate data labeling
Learned or hybridA trained router predicts the best model per requestLarge, stable workloads with good dataComplex to build and maintain

A cascade is worth a closer look. The system sends a request to a small model, checks the answer with a cheap test, such as a format check, a confidence signal or a second model, and escalates to a larger model only when the check fails. When most traffic is easy, most requests never touch the expensive model.

Routing is related to, but different from, fallback. Routing chooses the best model for a request up front. Fallback reacts to a failure after it happens. Many systems use both.

How does a model router work, step by step?

  1. The application sends a request to the gateway with metadata such as feature name, user tier and data class.
  2. The router checks policy first: which models may see this data in this region?
  3. It then scores the task, by rule, classifier or both.
  4. It picks the model and sends the request, with a timeout.
  5. A quality check runs where needed, for example schema validation or a confidence test.
  6. On failure or low confidence, it escalates or falls back to the next model.
  7. It records the choice, tokens used, cost and latency for analysis.

The last step matters most for improvement. Logging which model handled which request, and how it went, is how you tune the router later. See inference and tokens for what is being measured.

What are real-world examples of model routing?

  • A support assistant sends simple status questions to a small model and complicated billing disputes to a stronger one.
  • A document pipeline uses a small model to classify each page, and a larger model only for pages with unusual layouts.
  • A coding assistant uses a fast model for autocomplete and a reasoning model for multi-file changes.
  • A healthcare or finance application routes requests that contain personal data to an approved private deployment and everything else to a hosted API.
  • A multilingual product sends each language to the model that handles it best.
  • An agent uses a strong planner model and cheaper models for tool-result summaries (multi-agent systems).

In each case, the routing rule matches a real difference in the work, not a hunch about which model is smarter.

How do you add routing safely?

  1. Start with one model and a test set drawn from real requests.
  2. Log the cost, latency and quality for each task type.
  3. Find the task types where a cheaper model passes your test set.
  4. Add a simple rule for those types and compare quality and cost before and after.
  5. Add fallback to a second provider for reliability.
  6. Add more advanced routing only if the simple rules leave real savings or gaps.
  7. Re-run the tests whenever a model or provider changes.

Resist building a clever router first. Most of the benefit comes from two or three clear rules and a fallback.

How do you measure whether routing is working?

A router is working if cost or latency improves while quality holds. Measure all three together, by task type, before and after each change.

  • Quality: score on your test set for each route, and sampled review of live answers.
  • Cost per completed task, not just per call, so escalations and retries are counted.
  • Share of traffic on each model, which shows whether the router is doing anything.
  • Latency at the typical and slow ends, including the router step itself.
  • Escalation and fallback rates; a rising rate means the first-choice model is struggling.
  • User signals such as corrections, complaints and abandoned sessions.

Rerun the same test set after every model or rule change. A model update from a provider can change which route is best, which ties routing to model drift monitoring.

When is routing not worth it?

Routing adds a moving part, so it should earn its place. Skip it when traffic is low, when all tasks are of similar difficulty, when the savings are smaller than the effort to maintain the router, or when a single model clearly meets cost and quality needs.

Quick decision guide
SituationRoute?
High volume, varied difficultyYes, savings are likely
Need provider resilienceYes, at least a fallback
Sensitive data with approved-model rulesYes, policy routing
Low volume, one simple taskNo, keep one model
Quality is hard to measureNot yet; build evaluation first

What risks come with routing?

  • Silent quality loss: a bad routing choice sends hard tasks to a weak model and nobody notices.
  • Inconsistency: different models answer in different styles, so users see the product change from one message to the next.
  • Hidden cost: classifiers, retries and escalations add their own calls.
  • Added latency from the routing step or from cascades.
  • Compliance gaps if data classification is wrong.
  • Harder debugging, since one feature may run on several models.

Controls are straightforward. Test the router on your own data, sample routed answers for review, keep a log of every decision and keep a single switch to send everything to the main model. See LLMOps and AI observability.

What tools are used for model routing?

You can write routing in a few dozen lines of your own code. Many teams use an AI gateway, either a hosted one or an open-source proxy, that provides one interface to many providers, along with fallbacks, rate limits, caching and logging. Cloud platforms such as AWS Bedrock, Azure OpenAI and Google Vertex AI also offer access to several models under one account. Pick a tool by what you need to control, and keep your own code behind a thin interface so the tool can change.

How does StackLift approach it, and what does it do to cost and timeline?

Routing changes the running cost more than the build cost. Build effort depends on the number of models and providers, the quality checks and the data rules. Running savings depend on traffic volume and how varied the tasks are. See the LLM cost optimization playbook and LLM integration cost.

StackLift starts with one model, measures, and adds routing where the data shows a gain. Evaluation, CI/CD, observability, security review and rollback are in the plan from sprint one, and a senior architect reviews scope. See LLM integration and run the free estimate.

Questions people ask

What is an LLM router?

A component that decides which language model handles each request. It can follow fixed rules, use a small classifier or combine both, and it usually lives in an AI gateway.

Does model routing reduce cost?

It can, when traffic is high and tasks vary in difficulty, because easy requests go to cheaper models. It also adds its own cost and upkeep, so measure before and after.

What is the difference between routing and fallback?

Routing picks the best model for a request before it runs. Fallback switches to another model after a failure or timeout. Most production systems use both.

Can routing hurt quality?

Yes. If hard tasks reach a weak model, answers get worse and nobody notices. Test the router on your own data, sample the results and keep a way to send all traffic to the main model.

Do I need an AI gateway?

Not at first. A gateway helps when you use several models or providers and want fallbacks, limits and logs in one place. A thin interface in your own code is enough to begin.

What is a cascade in model routing?

A pattern that tries a cheap model first and escalates to a stronger one only when a check fails or confidence is low. It saves cost when most requests are easy.

Can I route between providers?

Yes. A gateway or your own adapter layer can send requests to OpenAI, Anthropic Claude, Google Gemini or a self-hosted open-weight model. Test each model, since prompts that work on one may need changes on another.

How do I decide which tasks go to a small model?

Run your test set on both a small and a large model, grouped by task type. Where the small model passes your quality bar, route that type to it. Where it falls short, keep the larger model. Repeat the check after any model or prompt change.

Does routing work with agents?

Yes. An agent makes many calls, and they differ in difficulty: planning needs a strong model, while summarizing a tool result often does not. Routing by step type is a common way to control agent cost. See AI agents.

Is model routing the same as using a small language model?

No. A small language model is one kind of model. Routing is the decision about when to use it and when to use a larger one.

Sources

Sources and further reading

The standards, rules, research and official documentation this page draws on.