Model routing, fallbacks and LLM gateways

How it works: model routing

1. Classify requests by difficulty or type: simple FAQ, extraction, long-document analysis, code, agent tasks.
2. Map classes to models: cheap and fast for easy, strong (or reasoning) for hard (small vs large).
3. Decide how to route:
   • Rules: length, keywords, feature, customer tier.
   • A classifier trained on labelled examples.
   • A cheap LLM call that picks the route (query routing).
   • Cascade: try the cheap model first and escalate when a check fails (low confidence, failed validation).
4. Evaluate each route: does the cheap model pass the evals for its traffic? Measure cost saved vs quality lost.
5. Monitor: share of traffic per route, quality per route, escalation rate.

How it works: model fallbacks

1. Define a chain: primary model → backup model (maybe another provider) → last resort (a simpler response or a queue).
2. Trigger on: timeouts, 5xx errors, overloads, rate limits (429), and sometimes refusals or invalid output.
3. Retry wisely: retry the same model once or twice with backoff for transient errors, then fall back (retries); keep a total time budget.
4. Add a circuit breaker: after repeated failures, skip the failing model for a cool-down period instead of trying it every time.
5. Keep prompts and features compatible: the backup must support the tools, structured output and context length you use; test it with your evals.
6. Log which model served each request for debugging and cost tracking.

How it works: LLM gateways

1. An LLM gateway sits between your apps and all providers and exposes one API (often OpenAI-compatible).
2. It centralises: routing rules, fallbacks and retries, provider keys (apps never hold them), per-team budgets and rate limits, caching (semantic caching), logging and tracing, and sometimes guardrails.
3. Options (as of November 2026): open-source LiteLLM (a Python library with a Router, and a proxy server), cloud AI gateways (e.g. Cloudflare AI Gateway, cloud providers' model hubs), and commercial gateways.
4. Apps change one URL to switch or add models; policy changes happen in one place.
5. Run it reliably: it's now on the critical path, so make it highly available and fast.

import os
os.environ["LITELLM_LOCAL_MODEL_COST_MAP"] = "True"   # don't download the price list at import
os.environ["LITELLM_LOG"] = "ERROR"
from litellm import Router
 
# 1) ROUTING: send each request to the cheapest model that can handle it.
def pick_model(question: str) -> str:
    hard = len(question) > 200 or any(w in question.lower() for w in ("analyse", "compare", "code"))
    return "strong" if hard else "fast"
for q in ["What are your opening hours?", "Compare these two contracts and list the risks"]:
    print(f"{q[:40]!r:45} -> {pick_model(q)}")
 
# 2) FALLBACKS through a gateway library (LiteLLM). mock_response simulates an outage offline.
router = Router(
    model_list=[
        {"model_name": "strong", "litellm_params": {"model": "anthropic/claude-sonnet-5-5", "api_key": "x",
                                                     "mock_response": Exception("529 overloaded")}},
        {"model_name": "strong-backup", "litellm_params": {"model": "openai/gpt-6.1-sol", "api_key": "x",
                                                            "mock_response": "Answer from the backup."}},
    ],
    fallbacks=[{"strong": ["strong-backup"]}],    # if 'strong' fails, try 'strong-backup'
    num_retries=0,
)
r = router.completion(model="strong", messages=[{"role": "user", "content": "Compare these contracts"}])
print("served:", r.choices[0].message.content, "| by", r.model)

'What are your opening hours?'                -> fast
'Compare these two contracts and list the'    -> strong
served: Answer from the backup. | by gpt-6.1-sol