Quick answer
An AI gateway (or LLM gateway) is a proxy that your applications call instead of calling OpenAI, Anthropic, or Google directly. It translates one request format to every provider, tracks who spent what, retries and fails over when a provider errors, caches repeated requests, and can apply guardrails. Examples are LiteLLM (open source, self-hosted), Portkey (hosted platform), and OpenRouter (hosted marketplace).
The first app a company builds calls one model with one key. The fifth app, built by a different team, calls a different provider with a key in a config file nobody remembers. A gateway is the point where that stops.
What a gateway does
- One API: your code speaks one format (usually OpenAI's) and the gateway translates to each provider
- Keys and budgets: the gateway holds the real provider keys and issues virtual keys to teams with spending limits
- Reliability: retries, timeouts, and fallback to another model or provider when one fails
- Cost control: caching identical requests, routing cheap tasks to cheap models
- Observability: every request logged with tokens, cost, latency, and errors
- Safety: guardrails that block prompt injection, redact personal data, or filter outputs
When you need one
A solo developer with one app does not. A team with several apps, several providers, or a finance department asking for the AI bill by project does. So does anyone who has been paged because a provider had an outage and there was no fallback. The gateway is also where model swaps happen: change a route, not a deployment.
The trade-off
A gateway is one more thing in the request path. Hosted gateways add a vendor that sees your traffic; self-hosted ones add software to run. Latency overhead is small but not zero. Most teams accept it because the alternative, five apps with five uncontrolled keys, is worse.
Bottom line
An AI gateway turns model access from a scattered set of keys into a managed service. If more than one team at your company calls a model, you either have one or you need one.
