Model Routing

27.1 The Premise #

Model tiers differ by roughly twenty to one on output price, and most traffic does not require the top tier. Routing is the practice of matching each request to the cheapest model that will handle it acceptably.

Verified September 8, 2026, per million tokens:^26,52

The ratio between the cheapest small model and the frontier tier on output is just under twenty-one to one. That spread, rather than the routing algorithm, is where the saving comes from.

The premise is sound and the savings arrive. The difficulty is deciding which requests the cheap tier handles well enough, and the published numbers are measured on workloads where that question answers itself more easily than yours will.

27.2 Four Strategies #

Static routing is a lookup table from task class to model. It requires no infrastructure, and it captures most of the available saving in agentic systems because the task class is usually known before the work begins. Start here.

Classifier routing trains a small model to predict difficulty and picks a tier from that prediction. It adds a few hundred milliseconds before the main call and requires labeled data, though less of it hand-made than expected: the published router learned from roughly 65,000 public preference comparisons, and the golden-labeled set added on top to lift benchmark performance was about 1,500 examples, under two percent of the training data.⁶⁰

Cascade routing sends everything to the cheap model first and escalates when the result fails a quality check. It needs no difficulty prediction because the cheap model’s own output supplies the signal. It also pays for the cheap call on every request, including the ones that were always going to escalate.

Semantic routing embeds the request and routes by proximity to labeled clusters. It works where the domain is bounded and the mapping from topic to difficulty is stable.

27.3 What the Numbers Actually Say #

RouteLLM learned routing policies from human preference data and reported over 85 percent cost reduction on MT-Bench while retaining 95 percent of the strong model’s performance, sending only 14 percent of queries to the frontier tier.⁶⁰ Cascade approaches have reported reductions up to 98 percent on some benchmarks.⁶¹ A useful secondary finding is that routers transfer: policies trained on one model pair generalize to a different pair without retraining, which suggests the classifier is learning something about query difficulty rather than about specific models.⁶⁰

Three significant qualifications apply, and they have considerable implications..

27.4 Why Agentic Workloads Are the Hard Case #

This is the part most routing material omits, and it is precisely the part that matters for anyone reading this document.

A routing failure in a chat product produces a worse answer, which the user re-asks. A routing failure inside an agent loop produces a malformed tool call: the downstream step parses invalid input, throws or produces a wrong result, and the error propagates. The retry then costs a frontier call anyway, on top of the cheap call, the failed tool execution, and the wasted turns.⁶³

The published routing benchmarks make no claim about structured agent task completion.⁶³ Extrapolating an MT-Bench conversation result to a fifty-turn agentic loop is not supported by the evidence, and the failure mode is asymmetric: a wrong route costs far more than a right one saves.

That asymmetry has three consequences.

27.5 Routing You Did Not Build #

Everything above assumes you are the one routing. On several platforms you need not be, and on at least one you are highly encouraged away from it.

Automatic model selection is the recommended path on GitHub Copilot rather than an advanced option. GitHub’s own material for the feature establishes the orchestration mechanism and its vendor-measured results; what it does not carry is adoption data. Neither the orchestration announcement nor its research-preview discussion supplies a request count or a share of paying users,⁶⁴ so treat figures of that shape as unsourced until someone names the window and the denominator.

That inverts the decision this chapter has been describing. The question is no longer merely which strategy to implement, but whether to delegate the choice at all, and on what terms.

What delegation gains and costs. It gains a routing policy maintained by someone with visibility you do not have: per-model capability measurements across a large traffic population, updated as models change. No individual team can replicate that. It costs the two things §27.7 says to instrument. You do not choose the tiers, so escalation rate is the platform’s to observe rather than yours. And unless the platform exposes its decisions, cost per accepted outcome cannot be attributed to a tier because you do not know which tier ran.

The category is broader than tier selection. Platform routing now extends past “which model” into “which workflow,” and the patterns being productized are the ones this chapter already named. One system currently in research preview composes three: a single model solving directly; a cascade in which an efficient model drafts and a quality gate decides whether to escalate; and a critique pattern in which one model drafts, an independent read-only critic from a different model family reviews, and the drafting model revises once.⁶⁴

Two observations about that set, both bearing on material established earlier.

The critique pattern is not the self-critique §10.3 warns against. The measurements there concern a model reviewing its own output, where iterative self-review raised recall while collapsing signal-to-noise below 1.0. Using an independent critic from a different model family is the mitigation §34.3 prescribes for model-as-judge, and the distinction is the whole difference between the two.

The same vendor states that first-turn, single-prompt tasks are the best place to begin and that multi-turn performance is the next target.⁶⁴ That is an independent arrival at §27.4’s argument, from a team building the system rather than critiquing it, and it leaves compound workflows inside long loops as the unsolved case.

Evaluating a routing product. Four questions, and the first is the one vendors answer least clearly.

On the published numbers. Routing products are evaluated by their vendors, against baselines their vendors select, and the resulting figures deserve the treatment §18.3 gives multi-agent claims. The preview referenced above reports improved verified task quality at substantially reduced estimated cost against a frontier baseline on one public benchmark, and the same publication notes that benchmark’s relative saturation and includes an internal benchmark nobody outside the company can reproduce.⁶⁴ Those disclosures are creditable and they do not make the numbers independent. Treat them as a reason to run your own comparison rather than as a substitute for one.

A research preview is also not an architecture decision. Availability, pricing, execution patterns, and the models composed are all subject to change before general availability, and some previews do not reach it.

Where that leaves you. Evaluate platform routing before building anything, since it replaces engineering you would otherwise do, under a policy maintained by someone measuring more traffic than you will. Evaluate it the way you would evaluate any routing change: on cost per accepted outcome for your own task classes, over your own traffic, with the eval suite running (§34). If the platform will not tell you what it decided, that is not a minor gap in reporting. It is §27.7’s second metric made uncomputable, and it should weigh accordingly.

27.6 A Reference Implementation #

from dataclasses import dataclass
from enum import Enum

class Tier(str, Enum):
    SMALL    = "claude-haiku-4-5"
    MID      = "claude-sonnet-5"
    FRONTIER = "claude-opus-5"

@dataclass(frozen=True)
class RoutePolicy:
    tier: Tier
    max_cost_usd: float
    max_turns: int
    escalate_to: Tier | None = None

# Static, by task class. No classifier, no latency, no routing failure mode.
POLICY = {
    "extract":         RoutePolicy(Tier.SMALL,    0.05,  4),
    "classify":        RoutePolicy(Tier.SMALL,    0.05,  4),
    "format":          RoutePolicy(Tier.SMALL,    0.05,  4),
    "lint-fix":        RoutePolicy(Tier.SMALL,    0.10,  8,  escalate_to=Tier.MID),
    "test-repair":     RoutePolicy(Tier.MID,      0.75, 20,  escalate_to=Tier.FRONTIER),
    "feature":         RoutePolicy(Tier.MID,      3.00, 40,  escalate_to=Tier.FRONTIER),
    "architecture":    RoutePolicy(Tier.FRONTIER, 8.00, 60),
    "incident-debug":  RoutePolicy(Tier.FRONTIER,15.00, 80),
}

def run_routed(task_class: str, task: str, oracle, run_session) -> dict:
    """Escalate only on an oracle verdict, and only once. Never mid-session."""
    policy = POLICY[task_class]
    first = run_session(policy.tier, task, policy.max_cost_usd, policy.max_turns)

    ok, feedback = oracle(first)                  # tests, typecheck, schema — §10.4
    if ok or policy.escalate_to is None:
        return {**first, "tier": policy.tier, "escalated": False,
                "accepted": ok, "total_cost_usd": first["cost_usd"]}

    # Fresh session on the stronger tier, and a continuation of nothing: the
    # failed attempt would contaminate the context (§4.3) and the model change
    # invalidates the cache regardless (§23.1).
    stronger = run_session(policy.escalate_to, task, policy.max_cost_usd * 3,
                           policy.max_turns)
    accepted, second_feedback = oracle(stronger)  # the escalated run is a
                                                  # candidate, not a verdict
    return {**stronger, "tier": policy.escalate_to, "escalated": True,
            "accepted": accepted,
            # Both attempts were billed. Reporting only the second one makes
            # cost per accepted outcome (§27.7) impossible to compute.
            "total_cost_usd": first["cost_usd"] + stronger["cost_usd"],
            "first_attempt_feedback": feedback,
            "final_feedback": second_feedback}

Three design decisions carry the weight here. Escalation starts a fresh session rather than continuing the failed one because carrying the failure forward is the self-conditioning problem in §4.3 and the model change invalidates the cache anyway. Escalation happens once because a two-step escalation that fails twice has spent more than going straight to the frontier tier would have. And the return value carries the summed cost of both attempts along with a verdict on the escalated one because a router that reports only what the winning tier spent, and never checks whether the winner was right, has removed the two inputs the next subsection needs.

27.7 Instrumenting It #

Routing without measurement is merely a cost increase with extra steps. Four metrics apply.

The first row is the alarm that matters. A cascade escalating most of its traffic has handed back most of its saving, and past the crossing in the figure below it costs more than no routing at all. The failure is silent—the outputs are fine, the bill is higher, and nobody looks.

The second row supplies the denominator. Routing that cuts per-attempt cost 40 percent while dropping first-pass acceptance from 85 to 45 percent has raised cost per accepted outcome by 13 percent even though the invoice fell. Denominate the cut in money, not tokens: a routing change moves the rate as well as the count, so a 40 percent reduction in tokens and a 40 percent reduction in spend are different quantities and only the second supports that conclusion (§25.6).

The arithmetic, worked on a month of a single task class:

1,000 tasks/month, previously all on the frontier tier
  cost per task              $2.40
  first-pass acceptance      85%
  accepted outcomes          850
  cost per accepted outcome  $2,400 / 850 = $2.82

After routing 70% to the mid tier
  700 mid-tier tasks   x $0.95 =   $665
  300 frontier tasks   x $2.40 =   $720
  invoice                          $1,385   <- down 42%, and this is the
                                               number that gets reported

  mid-tier acceptance        62%   ->  434 accepted
  frontier acceptance        85%   ->  255 accepted
  accepted outcomes                    689
  cost per accepted outcome  $1,385 / 689 = $2.01   <- genuinely better

That routing worked, and it worked by less than the invoice suggests: 29 percent on the metric that matters against 42 percent on the one that gets quoted. Now run the same arithmetic with mid-tier acceptance at 30 percent instead of 62:

  accepted outcomes          210 + 255 = 465
  cost per accepted outcome  $1,385 / 465 = $2.98   <- worse than the baseline

Same 42 percent cut to the invoice, and the change is now a loss. The line between those two outcomes sits at 33.7 percent mid-tier acceptance for these costs, and it is further down than intuition puts it: even at 45 percent, which looks like a rout, routing is still 14 percent ahead of sending everything to the frontier tier. Nothing in the token metrics distinguishes any of those cases. Only acceptance does, which is why it is the first thing to instrument and the last thing most teams add.

27.8 Where to Start #

27.9 Failure Modes #

• Routing per turn inside a session. Cache invalidation plus cold start on every switch.
• Escalating on an uncalibrated confidence, self-reported or logprob. Calibrate it or use an oracle (§31.4).
• Quoting a benchmark percentage to finance. MT-Bench is not your traffic.⁶⁰
• Continuing the failed session on the stronger model. Carries the failure into the new context.
• No escalation-rate alarm. The most common way a cascade quietly costs more than it saves.
• Routing structured agent work by classifier. A malformed tool call cascades; the retry costs the frontier call anyway.
• Delegating routing to a platform that will not report its decisions. Cost per accepted outcome cannot be attributed to a tier you cannot see (§27.5).
• Treating a vendor’s routing benchmark as a prediction. Vendor-selected baselines on vendor-selected benchmarks, with at least one that nobody outside the company can reproduce (§27.5).⁶⁴