API Platform AI Gateway¶
A gateway for managing and securing AI traffic, including Large Language Model (LLM) APIs and Model Context Protocol (MCP) servers.
Quick Start¶
- LLM Quick Start Guide—Install the gateway, confirm the Gateway Controller admin health endpoint reports it healthy, and route traffic to an LLM provider such as OpenAI.
- MCP Quick Start Guide—Install the gateway, confirm the Gateway Controller admin health endpoint reports it healthy, and route traffic to an MCP server.
Key Concepts¶
LLM Provider Template¶
An LLM Provider Template defines the characteristics and behaviors specific to an AI service provider, such as OpenAI, Azure OpenAI, or other LLM platforms. It describes how the gateway should interpret and extract usage and operational metadata, including prompt, completion, total, and remaining token information, as well as request and response model metadata.
Following templates are shipped out-of-the-box
- OpenAI
- Azure OpenAI
- Anthropic
- AWS Bedrock
- Azure AI Foundry
- Gemini
LLM Provider¶
An LLM Provider represents a connection to an AI backend service such as OpenAI, Azure OpenAI, or other LLM APIs. Platform administrators configure LLM Providers to define:
- The LLM Provider Template
- The upstream LLM service URL
- Authentication credentials (API keys, tokens)
- Access control rules for which endpoints are exposed
- Budget control policies, such as token-based rate limiting
- Organization-wide policies such as guardrails
Once configured, the LLM Provider allows traffic to flow through the gateway to the AI backend.
To connect the gateway to AWS Bedrock, see Configure an AWS Bedrock LLM Provider. The guide covers both Bedrock bearer API keys and AWS Signature Version 4 (SigV4) authentication.
LLM Proxy¶
An LLM Proxy allows developers to create custom API endpoints that consume an LLM Provider, while inheriting administrator-enforced access control, budgeting and organization-wide policies defined at the provider level. Each proxy gets its own URL context (e.g., /assistant) and can have its own policies applied. This enables:
- Multiple AI applications to share a single LLM Provider
- A single OpenAI-compatible endpoint to route requests to multiple LLM providers. See Multi-Provider Routing for LLM Proxies.
- Per-application policies such as prompt management and guardrails
- Separation between platform administration and application development
MCP Proxy¶
An MCP Proxy routes Model Context Protocol traffic to MCP servers. MCP is a protocol that enables AI assistants to interact with external tools and data sources. With MCP Proxies, you can:
- Expose MCP servers through a centralized gateway
- Apply authentication and access control to MCP traffic
- Manage multiple MCP servers from a single control plane
Streaming¶
When an upstream service streams its response, the gateway relays it to the client chunk by chunk instead of buffering the whole response. This holds for LLM providers and LLM proxies, and needs no configuration. On MCP proxies, request bodies stream, but response bodies stay buffered. See Real-time AI streaming.
Default Ports¶
| Port | Service | Description |
|---|---|---|
| 8080 | Router | HTTP traffic |
| 8443 | Router | HTTPS traffic |
| 9090 | Gateway-Controller | REST API |
| 9094 | Gateway-Controller Admin | Health and admin endpoints |
Architecture¶
┌─────────────────┐
│ LLM Providers │
│ (OpenAI, etc.) │
└────────▲────────┘
│
┌──────────┐ ┌──────────────┐ │
│ AI Apps │───▶│ AI Gateway │────┤
└──────────┘ └──────────────┘ │
│
┌────────▼────────┐
│ MCP Servers │
└─────────────────┘
How it works:
- Administrators verify the Gateway-Controller admin health endpoint and configure LLM Providers and MCP Proxies via the Gateway-Controller API
- Developers create LLM Proxies to build AI applications on top of available providers
- The gateway routes traffic, applies policies, and manages authentication
AI Guardrails¶
AI Guardrails allow you to enforce safety, content, and compliance policies on AI traffic flowing through the AI Gateway. They can be applied at the LLM Provider level (organization-wide), at the LLM Proxy level (per-application), or on MCP Proxies.
The complete and up-to-date guardrail catalogue — with configuration references and examples — is maintained in the gateway-controllers repository: gateway-controllers documentation
You can extend the AI Gateway with custom guardrail policies by building a custom gateway image using the ap CLI. See Customizing the Gateway by Adding and Removing Policies.
Documentation¶
The following table lists the AI Gateway documentation sections and what each one covers:
| Section | Description |
|---|---|
| LLM | LLM provider configuration, guardrails, prompt management, and semantic caching |
| MCP | MCP proxy setup and policies |
| Real-time AI streaming | Streamed responses across providers and proxies, and how policies and analytics behave |
| Observability | Logging and tracing configuration |
| Analytics | Analytics integrations (Moesif) |
| Policies and Guardrails | Gateway policies and guardrails for AI traffic control |
| Management API | REST API reference for managing LLM providers, LLM proxies, MCP proxies, certificates, and secrets |
| Production deployment | High-availability Kubernetes deployment with Helm, an external database, replicated workloads, and AI workload tuning |
| AI Workspace | The control plane for governing LLM providers, proxies, and policies across every gateway you run |