Routing¶
Routing decides which upstream serves a request after the gateway has accepted it. One LLM proxy exposes a single OpenAI-compatible endpoint, and a routing policy chooses the provider, the model, or both. The calling application changes neither its endpoint nor its request format.
Routing matters for three reasons:
- A single model is a single quota.
- A single provider is a single point of failure.
- Models differ enough in cost and capability that sending every request to one of them is rarely the right default.
In this section¶
The pages in this section cover the two routing dimensions and the policy-level reference for each mechanism:
| Page | What it covers |
|---|---|
| Multi-provider routing | One OpenAI-compatible proxy in front of several providers, with the transformer each provider needs |
| Multi model routing | Distributing traffic across models, and what happens when one starts failing |
| Load balancing and failover | The policy reference for the round robin and weighted round robin policies |
| LLM header routing | The policy reference for selecting a provider from a request header |
Combine provider and model routing¶
The two dimensions are related rather than exclusive. Each entry in a model-round-robin or model-weighted-round-robin pool accepts an optional provider, and a model that omits it uses the primary provider. One pool can therefore mix models from several providers, which makes multi model routing a way to route across providers as well.
Read Multi-provider routing when the endpoint shape is the problem — an application speaks OpenAI and the upstream does not. Read Multi model routing when the distribution is the problem — one model cannot absorb the traffic, or must not be the only one that can.
Related topics¶
- Timeouts and resilience — how long the gateway waits before it treats an upstream as failed.
- Transform requests and responses — the transformer policies that convert between provider request shapes.
- LLM proxy — the artifact these policies attach to.