OrcaRouter session-aware routing figure 1
Engineering & Research

OrcaRouter Routing Infra: Session-Aware Routing and Frontier Escalation

Author

Alistair Wren

Date Published

Latest models · 20View all models
Benchmarks: Artificial Analysis · updated daily
Back to all posts

ORCAROUTER · ROUTING ARCHITECTURE

Every LLM gateway that caches prompts must pin a conversation to one model. Every gateway that pins a conversation makes its routing decision on the least informative turn of that conversation. This is a report on that trade-off, and on the tiered-stickiness mechanism OrcaRouter ships to escape it.

Subject: OrcaRouter LLM gateway (Go / Gin / Redis) · Component: session affinity + Frontier Escalation engine · Method: 400-session replay against the production decision code · Date: 14 August 2026

ABSTRACT — Request-level LLM routing — scoring each request independently and dispatching it to the cheapest adequate model — is the regime almost all published router work addresses. It is also the wrong regime for the traffic that now dominates gateway volume: multi-turn agent sessions, where the prompt is 90 % carried-over context and the provider's prompt cache pays for continuity. Switching models mid-conversation forfeits a 10× discount on the shared prefix, so gateways pin sessions. But a pin made on turn 1 is a pin made on the turn with the least evidence, and it persists for the life of the conversation.

100 / 100 — latent-hard sessions whose turn-1 score is indistinguishable from a trivial one

+16% — difficulty-score drift from transcript length alone, on identical task difficulty

45% — of always-frontier cost, for 67 % of its hard-turn coverage

0.019 — margin between the shipped gate and the ceiling of realistic scores

1 Two routing regimes

An LLM gateway that fronts many providers has to answer one question per request: which model serves this? There are two structurally different ways to answer it, and the literature and the production reality have drifted apart on which one matters.

Request-level routing treats each request as independent. A scorer estimates query difficulty or predicted response quality, and the request is dispatched to the cheapest model expected to handle it. This is the regime of essentially all published router work: RouteLLM trains preference-data routers that hit 95 % of GPT-4 quality with 14 % strong-model callssup>[1]/sup>; FrugalGPT cascades cheap-to-expensive with an accept/reject check and reports up to 98 % cost reductionsup>[2]/sup>; RouterArena builds an 8,400-query benchmark to compare routers on exactly this axissup>[3]/sup>. The unit of analysis is the query.

Session-aware routing

The reason session-aware routing exists is not elegance. It is arithmetic.

2 The cache economics that make stickiness mandatory

In a multi-turn agent session, turn n's prompt is turn n−1's prompt plus a delta. By turn 10 the carried-over prefix is the overwhelming majority of the input tokens. Every major provider now prices that prefix differently depending on whether it is a cache hit:

Table 1. Prompt-cache semantics by provider. The cache is keyed on an exact prefix and on the serving key — a model switch or a key rotation is a full-price cold read.

OrcaRouter encodes exactly these lifetimes as pin TTLs: a per-channel-type map of provider cache windows — 5 minutes for OpenAI, Anthropic and Gemini, 60 minutes for DeepSeek — with an unmapped-provider default of 5 minutes. The channel+key pin expires with that window, because a stale key index has no cache value and only distorts load balancing. The model pin, on a Redis-backed deployment and for a long-pin-eligible session id, persists for 30 days — not for cache value, which is long gone, but for request-format continuity. A mid-conversation model switch forces a request-format conversion that can be data-incompatible: thinking blocks and tool-call ids do not necessarily survive translation between provider schemas.

THE UNDER-APPRECIATED DETAIL

Prompt caches are keyed per API key, not per model. A gateway that pins the model but load-balances across three keys on the same channel still cold-reads two turns in three. This is why OrcaRouter's channel pin stores {ChannelID, KeyIndex} rather than a channel id, and why the pin is dropped when the recorded key index no longer resolves to an enabled key — a boosted-but-rotated key would bias toward a cold cache while bypassing balancing, which is the worst of both.

The pins are soft throughout: no resolvable session id is a no-op, a disabled or unhealthy pinned channel degrades to normal balanced selection, and a pin to a weight-zero channel in a mixed pool is dropped so an admin draining a channel is not defeated by stickiness. They never fail a request.

3 The trap: stickiness disables the router

Here is the failure mode. In OrcaRouter's pre-escalation code path, for a session-aware router on any non-DSL strategy, the session→model pin returned

That would be tolerable if turn 1 were representative. It is systematically not, for two compounding reasons.

3.1 Turn 1 is the least informative turn

The difficulty scalar (service/model_router_difficulty.go) is a weighted linear combination over six lexical features:

LogPromptTokens × 0.20 cap log(8001) ≈ 8.99

ReasoningCueCount × 0.15 cap 5

SystemPromptLogLen × 0.10 cap log(2001) ≈ 7.60

CodeKeywordDensity × 0.20 cap 5.0 (matches per 100 chars)

HasTools × 0.15 already 0/1

MathMarkerCount × 0.20 cap 5

A short opener with no history scores low almost by construction: the 0.20-weighted token term is near its floor, and the reasoning/math terms fire on vocabulary the user has not yet had a reason to use. Sessions therefore commit to a weak-pool model at the moment of least information — and with a Redis-backed 30-day model pin, that commitment is long.

Figure 1. Mean latest-turn difficulty by conversation turn, over 100 latent-hard sessions and 200 genuinely easy ones, scored by the production scorer. At turn 1 — the turn on which the sticky pin is written — the two populations are indistinguishable (0.210 vs 0.208). The hard population crosses the gate at turn 5. Under a pin-only policy, all 100 latent-hard sessions are committed to the cheap pool before any of that evidence exists.

3.2 Length masquerades as difficulty

The second problem is subtler and it undermines the obvious fix. If you simply re-run the difficulty gate every turn, you are re-running it on a score computed over the whole concatenated transcript. That score has a built-in upward drift: the 0.20-weighted LogPromptTokens term rises monotonically with conversation length, and for any agent session the 0.15 HasTools and 0.10 SystemPromptLogLen terms are effectively constant floors. A long, boring session looks progressively harder.

OrcaRouter session-aware routing figure 2

Figure 2. The length-bias artefact, measured on 60 sessions consisting entirely of trivial edits (“rename this variable”, “add a nil check”). The full-transcript score drifts +16 % across 25 turns on constant task difficulty; the latest-turn (delta) score is flat. A naive re-evaluation of the full-transcript score every turn would escalate sessions for the crime of being long.

The fix OrcaRouter ships is a separate delta extractor (service/model_router_delta.go) that scores only the latest turn — the new user text plus any tool results attached after the last assistant message — reusing the same weights and caps but deliberately zeroing SystemPromptLogLen, which is not part of the delta. Figure 2's flat blue line is that extractor.

4 Design: tiered stickiness

The naive escape from turn-1 lock-in is to re-route every turn — which is just request-level routing, and forfeits the cache. The naive fix in the other direction is to make the pin the memory of “this session got hard” — which cannot express de-escalation and cannot be capped. OrcaRouter's design refuses both.

The reframe: a session is pinned to a model within a tier, and a small Redis tier state is the only escalation memory. The model pin is never the memory.

Tier pools. The strong tier is the resolved escalation pool (escalation_pool, defaulting to the router's strong_pool). The base tier is AllowedModels \ strong-tier pool; a model in both belongs to the strong tier. Inside the base tier, gated_adaptive's weak/mid/strong difficulty banding keeps operating exactly as before.

Tier-scoped pins. The strong tier's model-pin key gets a :t:strong suffix; the base tier keeps the legacy key unchanged. Escalation therefore preserves the base pin, so a de-escalated session — or one resumed after the tier state expires — lands back on the exact model it started on, not an arbitrary re-pick. Strong pins are written with the short provider-window TTL only: a 30-day strong pin would outlive the 4-hour tier state that justified it.

The gate runs first. In selectByStrategy (service/model_router.go:1374) the tier is resolved up front, the candidate set is narrowed to the tier's pool, and only then is the sticky pin consulted — within that tier. This is the structural fix for §3: the difficulty computation and the escalation triggers run every turn, before the pin can short-circuit them.

4.1 Three trigger classes, ranked by trust

Table 2. Escalation triggers. No fuzzy signal ever ratchets alone; only an explicit client ask commits at n=1, and even it obeys the caps.

Three hygiene invariants are load-bearing. Strikes are deduplicated by request id through a ring buffer, so interleaved client retries cannot double-count. Infrastructure failure is never capability failure — 429s, 5xxs and channel fallbacks never strike; only post-success quality signals count. And a “turn” is defined as a completed, billed-success request that ran strike evaluation, so failed requests advance neither strike decay nor the clean-turn counter.

4.2 Resolve is pure; commit is deferred

The engine's most consequential structural property is that ResolveEscalation writes nothing. It returns a decision plus a list of pending intents. The distributor applies those intents in its post-success block, onto a fresh read inside a Redis WATCH transaction. This matters because the resolver runs on paths that must never mutate state: speculative fallback-chain resolutions, the read-only diagnostic endpoints, and requests that later 403 or fail upstream. Re-applying intents onto fresh state also means a stale concurrent writer cannot clobber a committed escalation, and two racing identical escalations merge idempotently.

4.3 Caps, and why they bind everything

A false-positive escalation costs (strong − base) price × remaining warm-episode tokens, and it costs it silently — nothing fails. The blast radius is bounded by caps that apply to every class:

escalation_max_per_session (default 1). De-escalation and client resets do not refund it, which closes the reset-loop gaming path.

A per-router escalated-share cap (default 20 %) over a trailing 24–48 h window of Redis day-buckets, plus a workspace-wide cross-router cap. At the cap, all escalation routing is suppressed — including explicit asks and once boosts.

De-escalation only at cache-cold boundaries, so a false positive is bounded to one warm episode.

The reason Class A obeys the caps is a threat-model conclusion, not a policy preference: on an API gateway, whoever holds the workspace token controls the headers. A cap-exempt “the client asked for it” path is an unmetered spend channel. §7 measures what happens when every client abuses it.

4.4 De-escalation is asymmetric by design

Escalate on corroborated evidence; de-escalate only when it is free. A strong session returns to base only when all of: the session is cache-cold (idle past the provider window recorded at escalation), it has accumulated ≥3 strike-free evaluated turns, and the latest delta difficulty is below T1. Inside the warm window, a switch pays a full-price cold re-read — flapping is the one guaranteed way to make escalation cost-negative.

5 Method

We measured the mechanism by replaying a synthetic session corpus through the actual production decision code. The harness is a Go test in the service package that calls ResolveEscalation and CommitEscalationDecision per turn against a miniredis-backed tier store, with the real difficulty scorers, the real request-side strike producers, and the real share-cap machinery. Nothing about the decision path is reimplemented or mocked except the audit-event sink.

WHAT IS REAL AND WHAT IS NOT

Real: every routing decision, difficulty score, strike detection, streak rule, cap evaluation and Redis state transition — these are the shipped functions. Synthetic: the traffic. The corpus is generated, not sampled from production logs. Its archetype mix (50 % hard) is a stress mix chosen to exercise the mechanism, not an estimate of real traffic; §6.4 reports the sensitivity to that choice, and it is large. The clean precision numbers below reflect a corpus whose classes are separable by construction, and should be read as “the mechanism fires where it was designed to”, not as a production precision estimate.

5.1 Corpus

400 sessions, 3,968 turns, seeded and deterministic. Each turn is a full chat-completions request body carrying the cumulative history, a two-tool definition array, and a realistic system prompt — the shape a coding agent actually sends. Five archetypes, each carrying a ground-truth label:

Table 3. Corpus composition. “Needs strong” is the ground truth used for the precision and coverage scores.

Hard turns carry a pasted goroutine dump or source excerpt of 3–8 KB in addition to the prose, because that is what a real hard debugging turn contains. This detail turned out to matter enormously — see §6.2.

5.2 Cost model

Costs are computed from published list prices with per-provider cache semantics; the model is stated in full so it can be disagreed with.

Table 4. Cost model parameters. Prices are $ per 1M tokens, August 2026 list.

A warm turn costs 0.1·p_in·prefix + write·p_in·delta; a cold turn costs write·p_in·prompt. Turn 1 is always a full cache write. The tier-switch turn under the escalation policy is explicitly charged as cold, so the mechanism pays for its own cache invalidation.

Quality is reported as hard-turn coverage — the fraction of ground-truth-hard turns actually served by the strong model — rather than as an accuracy figure. We did not run upstream inference, so we decline to invent accuracy numbers.

6 Results

6.1 The mechanism fires where it was designed to

Table 5. Escalation outcomes by archetype, auto mode, canary 100 %, T2 = 0.70 (shipped default).

Zero false positives on the 200 easy sessions, including the 60 long ones that a full-transcript scorer would have drifted into the hard band. The trigger classes specialise cleanly and without overlap: difficulty catches reasoning-heavy work, strikes catch failure loops. Note that failure_loop's peak difficulty score is 0.262 — the difficulty gate never sees those sessions at all. An agent stuck in a compile-error loop is not producing reasoning-cue-dense prose; it is producing the same short prompt with a different stack trace. Without Class C strikes, every one of those 60 sessions would grind on the cheap model indefinitely.

OrcaRouter session-aware routing figure 3

Figure 3. When sessions escalate, split by trigger. Strike-driven escalations are sharply concentrated (turn 4, the first turn at which two strikes can have accumulated inside the decay window); difficulty-driven escalations spread across turns 2–11 following the corpus's onset distribution. The two-consecutive-turn streak rule means the earliest possible difficulty escalation is turn 2.

6.2 Finding: the shipped gate sits on a cliff edge

Our first corpus produced zero difficulty-driven escalations. The hard turns — loaded with race conditions, invariants, complexity analysis and proof vocabulary — peaked at 0.658 against a 0.70 gate. Adding the pasted stack traces that real debugging turns actually carry pushed them to 0.719. The gate is passed by a margin of 0.019.

OrcaRouter session-aware routing figure 4

Figure 4. Where the difficulty budget actually goes, averaged over 855 hard and 3,113 easy turns. A realistic hard turn reaches 0.719 of a theoretical 0.90 delta maximum. The CodeKeywordDensity term contributes 0.069 of its 0.20 budget — measured density is 1.72 matches per 100 characters against a saturation cap of 5.0 — and SystemPromptLogLen's 0.10 is structurally zero in the delta extractor. Roughly a third of the score's nominal range is unreachable by realistic text.

The threshold sweep confirms this is a cliff, not a slope. Across T2 from 0.35 to 0.65 the outcome is identical — 200 of 400 sessions escalate, with zero misses. At the shipped 0.70 the classifier starts losing sessions; at 0.75 difficulty-driven escalation collapses from 122 sessions to 23.

OrcaRouter session-aware routing figure 5

Figure 5. Threshold sensitivity. The whole 0.35–0.65 range is behaviourally identical because no realistic delta text lands in it — the score distribution is bimodal, with easy turns clustered near 0.23 and hard turns near 0.72, and nothing in between. The shipped default sits at the top edge of the upper mode.

ENGINEERING IMPLICATION

T2 is calibrated for the full-transcript distribution the gated_adaptive bands were tuned on, and it is being reused as the delta extractor's threshold. The design document flags that the delta extractor “needs its own tuning”; this measurement quantifies how much. Either the delta gate needs a lower T2 of its own — anywhere in 0.45–0.60 buys identical behaviour with real margin — or the percentile-based threshold already scheduled for Phase 3 (“top X % of this router's recent traffic”) should land, which makes the escalation rate the operator's knob and sidesteps absolute calibration entirely.

6.3 Cost and coverage

OrcaRouter session-aware routing figure 6

Figure 6. Five policies over the same 400 sessions. Left: cost per 1,000 sessions (log scale). Right: fraction of genuinely hard turns served by the strong model.

Table 6. Policy comparison. Cost per 1,000 sessions under the Table 4 model.

Two results are worth separating. First, session affinity alone saves 24 % at identical model choice (16.64 → 12.63) and 35 % on the frontier pair (290.93 → 188.30). That is pure cache economics — same models, same everything, only the key stickiness differs. The saving is larger on the frontier pair because Anthropic's 1.25× write premium makes cold turns disproportionately expensive.

Second, escalation lands where a rescue mechanism should: 45 % of always-frontier cost for 67 % of its hard-turn coverage, serving the strong model on only 21.4 % of turns.

The missing third of coverage is not a defect; it is the ratchet's price. The corroboration rules that give zero false positives also mean the mechanism cannot act on turn one of a problem:

Table 7. Escalation latency — hard turns served on the cheap model before the ratchet fires.

Two turns is exactly what the two-consecutive-turn streak rule specifies, and one turn is exactly what two-strikes-to-ratchet specifies. The latency is the design, and it is the same property that produced zero false positives. Anyone who wants faster rescue has the Class A header, which acts at n=1 — that is precisely why the manual escape hatch shipped first.

6.4 The headline ratio depends entirely on your traffic

The corpus is 50 % hard by construction. Real router traffic is not, and the cost comparison is extremely sensitive to that. Re-weighting the measured per-archetype costs across a range of hard-session prevalences:

OrcaRouter session-aware routing figure 7

Figure 7. Cost per 1,000 sessions as a function of how much of your traffic genuinely needs the strong model. Within-class behaviour is held at the measured values; only the mix changes.

Table 8. Prevalence sensitivity, $ per 1,000 sessions.

At the design document's own target escalation rate of ≤5 % of sessions, escalation costs 1.6× the cheap-pool bill and 12 % of the frontier bill. At the stress-mix 50 % it costs 6.7× the cheap-pool bill. Both are true; they answer different questions. The operationally relevant one is the first, and it is why the share cap defaults to 20 % rather than to “off” — the cap, not the trigger precision, is what actually bounds the bill.

6.5 The caps hold under adversarial abuse

We re-ran the corpus with the real share-cap machinery — no stub, real Redis day-buckets — under the §8 threat model: every client sends X-OrcaRouter-Tier: strong on every single turn.

OrcaRouter session-aware routing figure 8

Figure 8. Adversarial header abuse against the 20 % escalated-share cap. The first 20 requests are unconstrained by design — the warm-up floor prevents “1 escalation out of 2” reading as 50 % and locking the feature on a fresh router — after which the share converges and holds. Final state: 296 of 1,439 requests served strong (20.6 %), with 1,143 explicit asks denied and audited as denied_cap events.

The residual 0.6 % overshoot is the intended behaviour of a strictly-greater comparison on an approximate trailing counter, and the per-session cap of 1 keeps individual sessions from consuming the budget. Every denial is visible to the client in the X-Orca-Session-Tier: base; reason=denied:share_cap response header and to the operator in the audit table — a suppressed escalation is never silent.

7 What we would change

Give the delta extractor its own threshold. Reusing the full-transcript T2 leaves a 0.019 margin (§6.2). A delta-specific T2 in 0.45–0.60 is behaviourally identical on this corpus with two orders of magnitude more headroom. The percentile-threshold work already scheduled subsumes this and is the better fix.

Do not let the code-density term stay decorative. It contributes 0.069 of its 0.20 budget on the densest realistic text we could construct, because its saturation cap of 5 matches per 100 characters implies roughly one code keyword every twenty characters. Either re-cap it against a measured production distribution or reallocate its weight.

Class C is the workhorse for agent traffic, and it is the least developed. The failure_loop population is invisible to the difficulty gate (peak 0.262) and is caught entirely by strikes. Agent sessions fail by looping, not by getting lexically harder. The remaining response-side producers — and the native-Gemini streaming capture hook that is still missing — are worth more than further difficulty tuning.

Publish the escalation latency. Two turns of hard work served on the cheap model is the honest cost of a corroborating ratchet, and operators should see it in the analytics panel next to precision, not discover it.

8 Limitations

The corpus is synthetic. It was constructed to separate cleanly, so the zero-false-positive result characterises the mechanism's specificity on separable input, not its precision on production traffic. The real precision number can only come from the shadow-mode labelling job the design specifies — full trigger pipeline running, routing nothing, decisions labelled retroactively — with a go-live gate at ≥70 % labelled precision.

The cost model assumes a fixed 500 output tokens per turn, which suppresses a real effect: frontier models emit more reasoning tokens, so the true frontier premium is understated. It also models request-level cache warmth as a uniform 1/N over key slots; a weighted pool would use the Herfindahl index Σw², and a single-key channel would show no cache advantage for session affinity at all at the channel layer — though the model-layer pin still matters for adaptive strategies.

We did not run upstream inference, so no accuracy or task-success claim is made. Hard-turn coverage is a proxy for quality, and it assumes the strong model is actually better on those turns — plausible for the archetypes constructed, unverified here.

Finally, this measures one gateway's implementation. The turn-1 lock-in failure mode should generalise to any cache-aware router that pins sessions, but the specific numbers are properties of these thresholds, these weights and these prices.

Request-level routing is well covered. FrugalGPTsup>[2]/sup> introduced the LLM cascade — query the cheap model, score the answer, escalate on low confidence — reporting up to 98 % cost reduction at matched accuracy. RouteLLMsup>[1]/sup> trains routers on Chatbot Arena preference data and reports 95 % of GPT-4 quality with 14 % strong-model calls, with routers that transfer across model pairs without retraining. RouterArenasup>[3]/sup> supplies the missing evaluation foundation: 8,400 queries across domains and difficulty levels, scored on accuracy, cost, routing optimality, robustness and router overhead.

What none of these address is the conversation as the routing unit. A cascade escalates a request and forgets; the next turn re-runs the same cheap model on the same now-known-hard task. A preference-trained router scores a query, not a trajectory. The gap this report addresses is what a router should remember between turns, how long, and what should be allowed to change its mind — a question that only becomes urgent once prompt caching makes forgetting expensive.

OrcaRouter ships an in-tree RouterArena harness (eval/) that benchmarks its five request-level strategies — cheapest, quality, balanced, linucb, gated_adaptive — against the open dataset without modifying the upstream repository. The session-level mechanism described here is orthogonal to and composes with all five.

10 Conclusion

Prompt caching changed the economics of LLM routing in a way the routing literature has not caught up with. Once continuity is worth a 10× discount on the majority of your input tokens, a router must pin — and the moment it pins, it makes its decision on the turn where it knows least, and lives with that decision for the length of the conversation. Request-level routing does not have this problem and pays for it in cache misses; naive per-turn re-evaluation reintroduces the misses and adds a length-bias artefact on top.

Tiered stickiness resolves it by separating two things that look like one: which model serves this session (the pin, stable within a tier) and which tier this session belongs to (a small, capped, corroborated, expiring piece of state). In our replay that separation recovers 87 % of sessions whose difficulty is undetectable at turn 1, with zero false positives on 200 easy sessions, at 45 % of always-frontier cost — and holds a 20 % spend cap against clients actively trying to defeat it.

The mechanism's honest weaknesses are calibration, not architecture: a difficulty gate reused from a distribution it was not tuned for, a feature term that cannot reach its budget, and two turns of unavoidable rescue latency. Those are tractable. The architectural claim — that the escalation memory must be separate from the pin, that no fuzzy signal may ratchet alone, and that caps must bind the client's own explicit request because the client holds the token — is the part we would keep.

11 Sources

1. LMSYS Org. RouteLLM: An Open-Source Framework for Cost-Effective LLM Routing. a href="https://www.lmsys.org/blog/2024-07-01-routellm/">u>lmsys.org/blog/2024-07-01-routellm//u>/a> · code: a href="https://github.com/lm-sys/RouteLLM">u>github.com/lm-sys/RouteLLM/u>/a>

2. Chen, Zaharia & Zou. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. arXiv:2305.05176. a href="https://arxiv.org/abs/2305.05176">u>arxiv.org/abs/2305.05176/u>/a>

3. Lu, Liu, Yuan, Cui, Zhang, Liu & Xing. RouterArena: An Open Platform for Comprehensive Comparison of LLM Routers. arXiv:2510.00202. a href="https://arxiv.org/abs/2510.00202">u>arxiv.org/abs/2510.00202/u>/a>

4. OpenAI. Prompt Caching in the API. a href="https://openai.com/index/api-prompt-caching/">u>openai.com/index/api-prompt-caching//u>/a> — automatic caching, ≥1,024-token prefix in 128-token increments, 5–10 minute idle eviction, ≤1 hour; cached-input discount by model tier. Pricing: a href="https://openai.com/api/pricing/">u>openai.com/api/pricing//u>/a>

5. Anthropic. Prompt caching. a href="https://platform.claude.com/docs/en/build-with-claude/prompt-caching">u>platform.claude.com/docs/en/build-with-claude/prompt-caching/u>/a> — cache reads 0.1× base input, writes 1.25× (5-minute TTL) or 2× (1-hour TTL), refreshed on use. Pricing: a href="https://www.anthropic.com/pricing">u>anthropic.com/pricing/u>/a>

6. DeepSeek. DeepSeek API introduces Context Caching on Disk. a href="https://api-docs.deepseek.com/news/news0802/">u>api-docs.deepseek.com/news/news0802//u>/a> — automatic, billed on actual cache hits, order-of-magnitude reduction on hit.

7. Google. Gemini API context caching. a href="https://ai.google.dev/gemini-api/docs/caching">u>ai.google.dev/gemini-api/docs/caching/u>/a> — implicit and explicit caching with storage-priced TTL.

8. OrcaRouter source, this repository: service/session_affinity.go (pins, TTLs, tier-scoped keys) · service/session_escalation.go (the engine) · service/model_router.go:1374 (selectByStrategy: tier narrowing before the pin read) · service/model_router_difficulty.go (weights and caps) · service/model_router_delta.go (delta extractor) · service/escalation_strikes.go (request-side producers) · service/escalation_caps.go (share caps) · docs/features/frontier-escalation.md (design, review rounds 1–4).

Reproducibility. The measurement harness is a Go test in the service package driving ResolveEscalation / CommitEscalationDecision against miniredis, plus a Python analysis and figure pipeline. Corpus generation is seeded (rand.NewSource(20260814)) and the full run is deterministic: 400 sessions, 3,968 turns, three experiments (main replay, adversarial cap run, 9-point threshold sweep). Figures use a CVD-validated categorical palette; every figure is paired with its underlying table. No production data was accessed, and no part of this analysis was committed to the repository.

Compared in this article1

Detected from this article · Benchmarks: Artificial Analysis · updated daily

© 2026 OrcaRouter

For Providers

Run an inference platform? Get your models on OrcaRouter.

Contact us

Join our community

DiscordEmailXGitHubYouTube