What Is an LLM Router? The Model-Picking Layer, the Real Math, and When to Skip It

An LLM router is a software layer that sits between your application and the model providers and decides, for every request, which model should answer it — a cheap, fast model like DeepSeek V4 Flash when the task is easy, a frontier model like Claude Opus 5 when it is genuinely hard. The point is money: DeepSeek V4 Flash costs $0.09 per million input tokens and Claude Opus 5 costs $5.00, provider-direct rates from the OrcaRouter model catalogue read on 2026-08-10 — a 55× difference on the same call. Send the easy 80% of your traffic down and keep the hard 20% up, and you are pulling the single biggest cost lever most teams never touch.

What an LLM router actually is

Strip away the marketing and a model router has three jobs, all of them boring and all of them valuable. It is a single endpoint: your code calls one OpenAI-compatible URL instead of one SDK per provider. It grades the request, estimating how hard it is, and routes it: easy work to a cheap model, genuinely hard reasoning to a frontier model. And it fails over: if the chosen provider rate-limits you or returns a 5xx, the router retries against a healthy model without your application ever seeing the error.

The decision step is where routers differ, and the spectrum is a familiar one. Rule-based routers match keywords or prompt length to a model — under a millisecond, predictable, brittle when pricing or models change. Semantic routers embed the prompt and route by meaning, so "what's my balance?" and "how much money do I have" land on the same path. Learned routers watch real traffic and shift toward whichever model is winning on quality per dollar. The mechanics of each — including the accuracy bands of the different classifier types and the auto mode that grades every prompt — get their full treatment in our guide, Auto Router for LLMs; here the point is that routing intelligence exists on a spectrum, and which point you need is a product decision, not a feature checklist.

If you have also seen "AI router" used for Wi-Fi hardware with an NPU on it — that is a different product that happens to share the name, and we untangle that collision in our AI router guide. Everything in this article is about the software category.

The price spread is what makes it pay

A router only earns its keep when models differ a lot in price, and right now they differ enormously. All rates below are provider-direct prices per 1M tokens from the OrcaRouter model catalogue, read on 2026-08-10:

Read the spread and the argument writes itself. DeepSeek V4 Flash at $0.09 against Claude Opus 5 at $5.00 is about a 55× difference on input tokens alone, and the gap between the cheap tier and the frontier tier is now a permanent feature of the market rather than a one-off discount. If your traffic is a typical mix — a majority of short, well-specified requests and a minority of genuinely hard reasoning — running everything on the frontier means paying top price for the easy 80%.

This is where the current page-one results for "llm router" let you down. Decagon's glossary entry, for example, cites "GPT-4o, Claude 3.5 Sonnet and Gemini 1.5 Pro" at "$5–15 per million input tokens" — models that were current two years ago at prices roughly double today's. The market moved; the definition didn't. That is the single biggest reason to distrust an LLM router explainer with no dates on it.

What the savings research actually says

The arithmetic above is real, and so is the research — but the honest picture is a range, not a single number.

Here is the worked arithmetic, with assumptions stated so you can adjust them. A support bot handling 100,000 conversations a month, each sending 1,000 input tokens and generating 200 output tokens: running everything on Claude Sonnet 5 costs about $400 a month. Route the easy 80% to DeepSeek V4 Flash and keep the hard 20% on Claude Sonnet 5, and the same traffic costs about $90 — roughly 78% cheaper, before any prompt caching, which would widen the gap further.

The shape to take away: aggressive routing saves 60–85% when your traffic is mostly easy, balanced routing saves 35–45%, and even conservative routing saves 10–15%. The exact figure for you is a property of your traffic, measured on your own prompts — not a constant you can copy from a blog post.

The three costs routing hides

The two costs nobody puts on the glossary entry are latency and accuracy, and there is a third that is subtler.

Latency. Every routed request pays a classification tax before the model even starts. A rule-based classifier is under a millisecond; a small embedding or classifier model adds roughly 50–200ms; a trained domain classifier more. A good router hides most of this by classifying while it prepares the upstream request, and one production routing endpoint measured end-to-end overhead at roughly 55ms median — under 1% of a normal response time. But if your traffic is real-time — a voice agent, a UI that must answer inside 300ms — a routing hop in the hundreds of milliseconds can be the difference between usable and not. That single constraint is the most common reason a team with a legitimate routing use case decides not to route.

Accuracy. A router is only as good as the judgment it routes on. A classifier trained on general benchmarks can be confidently wrong about your domain: your "easy" summarization task might be easy for the frontier and impossible for the cheap model. The failure mode is silent — the user just gets worse output and nobody logs it — which is why every credible router ships a quality floor or a balanced mode.

Cache invalidation. Both OpenAI and Anthropic discount repeated prompt prefixes, typically around 90% off input tokens on cache reads. If your routing layer rewrites, reorders, or injects a rotating timestamp into the prompt, it invalidates the prefix and throws that discount away. A good router passes the prompt through byte-for-byte and lets the provider's own caching work.

The recommendation: a managed router with a quality floor

If you run more than one model in production, your traffic is a mix of easy and hard, and your volume is big enough that a percentage is real money, the recommendation is a managed router that keeps a quality floor and charges no markup on tokens. Our own product is OrcaRouter, and it is the one we can speak to in detail; the checklist that follows applies to any router you evaluate.

OrcaRouter's auto mode — request the model name orcarouter/auto on the standard OpenAI-compatible endpoint — grades each prompt and routes it across 200+ models. It ships four routing policies with Balanced as the default: Cheapest sends to the lowest-cost model that can answer; Balanced sends to the cheapest model that clears the quality bar; Quality sends to the highest-scoring model regardless of price; Adaptive learns from your live traffic and shifts routing as it goes. Grading is measured in under a millisecond, total added latency stays under 50ms, and if the chosen provider rate-limits or errors, the router fails over mid-stream to a healthy model in under 50ms. There is no markup at any layer: you pay each provider its exact published price — the $0.09 or $5.00 figures above are what you pay — and routing itself is free.

On accuracy, the June 2026 RouterArena leaderboard scores OrcaRouter at 75.5% against the closest tracked alternative at 74.0% (per orcarouter.ai). One leaderboard is not proof of anything — treat it as a single data point and run your own eval on your own prompts before you trust any router with production traffic, including ours. And if you are trying to decide whether you need router features or gateway features at all, the router-versus-gateway distinction is covered in our guide, AI API Gateway in 2026.

When this recommendation is wrong

The honest skip list, stated plainly:

The short version

An LLM router is the layer that picks which model answers each request — cheap and fast for the easy majority, frontier for the hard minority, with failover when a provider drops. It pays when your models differ a lot in price (DeepSeek V4 Flash at $0.09 against Claude Opus 5 at $5.00 per 1M input tokens is a 55× spread) and your traffic is a mix of easy and hard. The research puts the ceiling around 85% savings behind a quality floor and the floor at whatever your eval says it is. It costs you latency and some accuracy control, and it is the wrong answer for single-model shops, sub-300ms latency budgets, tiny spend, and teams that cannot measure output quality. When it is right, run it behind a quality floor with no token markup, route a slice first, and read the routing logs before you believe the savings.