BLUF: Production model routing is not a price switch. It is a policy decision that assigns each request class a permitted capability set, a quality floor, a latency budget and a cost budget—then produces evidence when the router uses a fallback. If those constraints are not explicit and testable, an outage rule can quietly turn a critical workflow into a lower-quality experiment.
Why a single “best model” is the wrong production abstraction
A production portfolio normally serves requests with materially different failure costs. A document-extraction workflow may need schema-valid output and a low p95 response time. A bilingual technical-support answer may need a stronger quality floor and a larger context window. A low-risk internal summary may be deliberately cost-capped. Treating all three as one alias wastes budget in the easy path and under-specifies the hard path.
LiteLLM documents routing strategies including weighted, rate-limit-aware, latency-based and cost-based selection, as well as queues, cooldowns, retries and fallbacks across deployments. Those mechanisms are useful transport controls. They do not decide whether a specific fallback is acceptable for a particular business outcome. Your control plane must supply that policy.
The decision plane: policy before provider selection
Make a request class the unit of routing. A class is a stable, named promise such as extraction.standard, support.de-technical or incident.advisory—not a model name. The application sends the class plus an allowed set of non-sensitive operational attributes. The router resolves a candidate only if it satisfies all hard constraints.
A practical policy record contains: the class owner; approved model/deployment IDs; required capability flags; a frozen evaluation-suite version; per-segment quality floors; p95 latency and per-request cost budgets; data-residency and tool-use restrictions; fallback order; and an expiry date. Version this record, review it like a release artefact, and put its policy ID in the inference event.
Routing control-plane diagram
Request class + permitted signals → policy lookup → hard gates (capability, quality, residency, budget) → eligible deployment set → selection → inference event + cost record → reconciliation and review.
Hard gates reject candidates. Selection chooses among candidates that have already passed. This distinction prevents a low-cost or low-latency heuristic from overriding a quality floor. A router should return an explicit controlled-degradation response when no candidate is eligible; it should not silently relax the class.
Request-class policy table
Structured extraction — Required: JSON/schema support; frozen field-level evaluation floor. Latency: p95 3 s. Cost: €0.015. Fallback: same-capability deployment; otherwise queue or typed failure.
German technical support — Required: German and retrieval-grounded segment floors. Latency: p95 8 s. Cost: €0.08. Fallback: approved equivalent; otherwise human hand-off.
Internal summarisation — Required: basic summary floor; no external tools. Latency: p95 5 s. Cost: €0.02. Fallback: lower-cost approved model permitted.
Incident advisory — Required: incident-suite floor; evidence links required. Latency: p95 10 s. Cost: €0.15. Fallback: no quality-lowering fallback; controlled unavailable state.
Which signals may influence a route
Use signals that are observable, authorised and auditable: request class, declared language, approved region, estimated prompt size, queue depth, deployment health, remaining rate-limit capacity, measured latency percentile and policy-approved token/cost estimate. Do not route on raw customer text, hidden inferred sensitivity, user identity attributes or a provider’s opaque “quality” label unless the data-protection, fairness and audit consequences have been reviewed.
OpenTelemetry’s Generative AI semantic conventions are a useful starting point for consistent telemetry, but naming spans is not governance. Preserve a minimal routing event with request class, policy version, candidate set hash, selected deployment, fallback reason, measured latency, token categories, estimated cost and response outcome. Redact or separate prompts and outputs; do not turn routing logs into an uncontrolled copy of customer data.
Outage drills: test the decision, not only the retry
LiteLLM documents ordered fallbacks after retries and distinguishes general, context-window and content-policy fallbacks. A production drill should induce representative failure modes in a non-production environment: provider 5xx or timeout, rate limit, context-window rejection, malformed response, missing region, stale pricing map and exhausted monthly budget. For each, assert the selected fallback, the policy ID, the user-visible degradation, the trace and the financial record.
A drill passes only when the result is safe for the request class. For incident.advisory, a cheap model that produces a fluent but unqualified recommendation is a failed drill even if availability is restored. The correct result may be an unavailable state, queued request or human escalation.
Shadow traffic and invoice reconciliation
Use redacted shadow traffic only after data owners approve the processing path. Replay a representative sample through candidate routes without affecting users, preserving the request class and recording comparability limits. Compare outputs against the frozen evaluation suite and measure policy decisions, not just mean latency. Shadow experiments can expose routing regressions before a cost rule reaches customer traffic.
Spend telemetry is an estimate until reconciled. LiteLLM documents calculated response cost and spend logs, and explicitly recommends aligning time ranges and comparing token categories including cache when provider bills differ. Reconcile provider invoices against deployment, model, region, token category and time window. Investigate divergence as a data-quality incident: it may be a pricing-map change, cache-accounting gap, ingestion delay or an unapproved route.
Failure modes that price-only routing hides
1. A provider outage sends extraction traffic to a model without verified structured-output behaviour. Mitigation: capability and field-level quality floors are hard gates, not routing weights.
2. A latency heuristic shifts German technical queries to a deployment evaluated mainly in English. Mitigation: retain language and document-type segments in the frozen suite; require each segment floor.
3. A fallback looks successful in application logs but doubles cached-token charges. Mitigation: join routing events to token-category spend records and reconcile monthly against invoices.
4. A health signal is stale, so the router selects a nominally healthy but failing deployment. Mitigation: time-bound health evidence, circuit breaking, cooldowns and an explicit no-eligible-candidate state.
5. Shadow traffic leaks prompt content into a wider observability estate. Mitigation: use approved redaction, separate restricted payload storage from routing metadata and enforce retention.
Connect routing to release discipline
Routing policy is a released production artefact. The LLM release-bundle pattern provides the right operational model: bind the policy version, approved deployment list, evaluation evidence and rollback condition together. The self-hosted inference SLO approach provides the operational measurements: queueing, latency and error budgets should affect capacity and eligibility only within an approved class. For workflows that must integrate downstream systems, retain the structured-output API-contract gate after the model response; a qualified route does not guarantee a valid business object.
What this does not solve
A routing plane does not establish that a model is accurate, secure, legally compliant or suitable for a high-impact decision. It cannot compensate for weak source data, missing evaluation coverage, an unsafe tool path or an unclear human decision owner. Data-residency, contractual and regulatory obligations vary by role, sector and use case; obtain legal counsel for those questions. The engineering recommendation is narrower: make every route and fallback constrained, observable, reproducible and reversible.
Start with four classes, not a universal router
Inventory today’s high-volume and high-consequence request types. Define four classes, one owner per class, one frozen evaluation slice per class, explicit budgets and a no-eligible-candidate response. Then run an outage drill and reconcile one billing period before adding dynamic optimisation. I can facilitate a routing-control-plane workshop that turns model choices, SLOs and spend evidence into an implementable policy and test plan.
Sources
LiteLLM Router and load balancing documentation
LiteLLM provider-failover documentation
