Self-Hosted LLM Gateway: Open-source options and how to deploy one

If your prompts carry regulated data, "just call the SaaS gateway" isn't an option. The data leaving your network to a third-party control plane is the whole problem you're trying to solve. A self-hosted LLM gateway keeps the control plane and the traffic inside your perimeter while still giving you routing, guardrails, caching, and cost control.

This guide covers what a self-hosted LLM gateway is, why teams run one, the open-source options, and how to deploy one, with the WSO2 AI Gateway as a worked example.

What is a self-hosted LLM gateway?​

A self-hosted LLM gateway is an AI gateway you run on your own infrastructure, on-premises, in your private cloud, or in your Kubernetes cluster, rather than consuming it as a vendor-managed SaaS. It provides the same control-plane functions (multi-model routing, guardrails, token-based rate limiting, observability) but the data plane and, in a fully self-hosted mode, the control plane too, stay under your operational control.

The distinction that matters is data residency. In a self-hosted deployment, prompts and responses never traverse a third party's servers, which is often a hard requirement in regulated industries.

Why self-host an LLM gateway?​

• Data sovereignty and compliance. Prompts frequently contain PII or confidential data. Self-hosting keeps that traffic inside your boundary, which simplifies GDPR, HIPAA, and internal-policy compliance.

• Air-gapped and restricted environments. Some environments have no outbound internet at all. A self-hosted gateway can front self-hosted open-source models with no external dependency.

• Cost and control at scale. You own capacity planning and avoid per-seat or per-call SaaS pricing on high-volume traffic, in exchange for running the infrastructure.

• No vendor lock-in. Open-source gateways let you inspect, extend, and move the deployment without renegotiating a contract.

The cost of those benefits is operational: you patch, scale, and monitor the gateway yourself.

Self-hosted vs. SaaS LLM gateways (trade-offs)​

A common middle path is hybrid: run the data plane (the gateway that sees prompts) inside your network while a managed control plane handles configuration and observability. That keeps sensitive traffic local without making you operate the entire stack.

Open-source LLM gateways you can self-host​

Several gateways can be self-hosted. Evaluate them on deployment model, guardrail depth, provider coverage, and whether they unify with your existing API management rather than adding a separate control plane. Treat any vendor's own performance claims as unverified until you test them in your environment, and confirm the license of each project before you adopt it.

The key evaluation questions:

• Does it run fully offline / air-gapped, or does it phone home?

• Does it cover the providers and self-hosted models you actually use?

• Does it include guardrails and token-based rate limiting, or just routing?

• Is it open source under a license you can live with?

How to deploy a self-hosted LLM gateway​

The general shape of a self-hosted deployment, regardless of product:

• Provision the runtime. Deploy the gateway to your target: Docker on a VM, or Kubernetes for HA and autoscaling.

• Register model providers and/or self-hosted models. Point the gateway at cloud providers, self-hosted open-source models, or both.

• Apply policies. Turn on guardrails, token-based rate limits, and caching before you route production traffic.

• Wire in observability. Ship metrics and logs to your existing stack.

• Cut traffic over gradually. Route a slice of traffic through the gateway, validate, then expand.

The WSO2 AI Gateway (open-source and self-hostable)​

The WSO2 AI Gateway sits between your applications and your large language model (LLM) providers, and applies your policies to every request that passes through. Because it's part of the 100% open-source WSO2 API Platform, you run it on your own infrastructure, so prompts and responses never leave your perimeter.

Because it's the same platform that governs your APIs and MCP traffic, a self-hosted AI gateway here isn't a standalone tool: it shares one control plane, one analytics view, and one identity model with the rest of your API estate.

Tutorial: Deploy a self-hosted AI gateway with Docker Compose​

Prerequisites​

• A Docker-compatible container runtime, such as Docker Desktop, Rancher Desktop, Colima, or Docker Engine with the Compose plugin, installed and running.

• About 2 GB of free memory for the containers.

• An API key for your LLM provider. This tutorial uses Anthropic.

• On Windows, Git Bash or the Windows Subsystem for Linux (WSL). The commands pipe configuration in through a shell heredoc, which PowerShell doesn't support.

Start your container runtime and wait for it to report a running state, then confirm that both commands print a version number:

docker --version
docker compose version

If either command fails, fix the installation before you continue.

Step 1: Download the gateway​

1. Run this command in your terminal to download and unzip the AI Gateway distribution:

   wget https://github.com/wso2/api-platform/releases/download/ai-gateway/v1.2.0/wso2apip-ai-gateway-1.2.0.zip
   unzip wso2apip-ai-gateway-1.2.0.zip
   cd wso2apip-ai-gateway-1.2.0/
   

   If you don't have wget, download the zip from the browser and unzip it manually, then cd into the folder from your terminal.

Step 2: Run the one-time setup​

The gateway won't start until it has an encryption key, a TLS certificate for its HTTPS listener, and an administrator password. There is no default password and nothing is auto-generated at startup. The setup script provisions all three.

1. From inside the unzipped folder, run:

   ./scripts/setup.sh
   

   The script prompts you to set an administrator username and password. After you provide those, it prints the final credentials once, so copy them before you close the terminal.

   tip

   On Windows, run scripts/setup.ps1 instead. It takes the same options and provisions the same files.

2. Store the credentials so the Management API calls in the later steps can authenticate. This is the gateway's configuration endpoint on port 9090, covered next in Step 3:

   export ADMIN_USERNAME=admin
   export ADMIN_PASSWORD='<the administrator password you set in the previous step>'
   

Step 3: Start the gateway stack​

1. From inside the unzipped folder, start the stack:

   docker compose up
   

   This runs in the foreground, so keep the terminal open and use a second one for the remaining steps. Add the -d flag to run it in the background instead. The first start pulls the container images and takes a few minutes.

2. Give it a few seconds to initialize, then confirm the containers are up:

   docker compose ps
   

3. Verify that the gateway controller is healthy:

   curl http://localhost:9094/api/admin/v1/health
   

A success response means the controller is ready to accept configuration. Two addresses matter from here: the management API on port 9090, where you send configuration, and the router on 8443 (HTTPS) and 8080 (HTTP), where your applications send traffic. The gateway is up but empty, so the next step puts a model behind it.

Step 4: Register your first LLM provider​

A provider tells the gateway where a model lives, how to authenticate to it, and which paths callers are allowed to reach.

1. In your terminal, run the following, replacing <anthropic-apikey> with your real key:

   curl -X POST http://localhost:9090/api/management/v1/llm-providers \
     -H "Content-Type: application/yaml" \
     -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" \
     --data-binary @- <<'EOF'
   apiVersion: gateway.api-platform.wso2.com/v1
   kind: LlmProvider
   metadata:
     name: anthropic-provider
   spec:
     displayName: Anthropic Provider
     version: v1.0
     template: anthropic
     context: /providers/anthropic
     upstream:
       url: https://api.anthropic.com
       auth:
         type: api-key
         header: x-api-key
         value: <anthropic-apikey>
     accessControl:
       mode: deny_all
       exceptions:
         - path: /v1/messages
           methods: [POST]
         - path: /v1/models
           methods: [GET]
   EOF
   

   A success response confirms the provider was created. To use a different provider, change template and upstream: the built-in templates are openai, azure-openai, anthropic, gemini, mistralai, awsbedrock, and azureai-foundry.

   The gateway now holds your credential and knows where to send traffic. The next call proves it works end to end.

2. Send a prompt through the gateway to confirm the route works:

   curl -k -X POST "https://localhost:8443/providers/anthropic/v1/messages" \
     -H "Content-Type: application/json" \
     -H "anthropic-version: 2023-06-01" \
     -d '{
       "model": "claude-sonnet-4-5-20250929",
       "max_tokens": 100,
       "messages": [{"role": "user", "content": "Hi"}]
     }'
   

You should get a normal chat completion back. The -k flag tells curl to accept the self-signed certificate that the setup script generated. Replace that certificate with one from your own certificate authority before you put the gateway in front of real traffic, and drop the flag.

Step 5: Attach a regex guardrail​

A guardrail inspects prompts or responses and blocks content that breaks your rules. Guardrails can be attached at the provider level or at the resource level. A guardrail configured globally applies to every resource the provider exposes. In this step, you configure a regex guardrail globally, so that any request that includes a password in its prompt is rejected.

1. Re-submit the provider with the regex guardrail configured:

   curl -X PUT http://localhost:9090/api/management/v1/llm-providers/anthropic-provider \
     -H "Content-Type: application/yaml" \
     -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" \
     --data-binary @- <<'EOF'
   apiVersion: gateway.api-platform.wso2.com/v1
   kind: LlmProvider
   metadata:
     name: anthropic-provider
   spec:
     displayName: Anthropic Provider
     version: v1.0
     template: anthropic
     context: /providers/anthropic
     upstream:
       url: https://api.anthropic.com
       auth:
         type: api-key
         header: x-api-key
         value: <anthropic-apikey>
     accessControl:
       mode: deny_all
       exceptions:
         - path: /v1/messages
           methods: [POST]
         - path: /v1/models
           methods: [GET]
     globalPolicies:
       - name: regex-guardrail
         version: v1
         params:
           request:
             regex: "(?i).*password.*"
             invert: true
             jsonPath: "$.messages[0].content"
   EOF
   

A success response confirms the guardrail is live on every resource the provider exposes. The next step proves that it blocks a request whose prompt mentions a password.

Other guardrails, such as PII masking and content safety, attach the same way, by name. For the full set, see Guardrails.

Step 6: Verify the guardrail blocks a matching request​

1. Send an ordinary prompt first. It passes through as before:

   curl -k -X POST "https://localhost:8443/providers/anthropic/v1/messages" \
     -H "Content-Type: application/json" \
     -H "anthropic-version: 2023-06-01" \
     -d '{"model": "claude-sonnet-4-5-20250929", "max_tokens": 100, "messages": [{"role": "user", "content": "What is the capital of France?"}]}'
   

   The guardrail finds no match, so the request reaches the model and returns a completion:

   {
     "id": "msg_011Ce7hcHtobAMD83cUajmjF",
     "type": "message",
     "role": "assistant",
     "model": "claude-sonnet-4-5-20250929",
     "content": [{"type": "text", "text": "The capital of France is Paris."}],
     "stop_reason": "end_turn",
     "usage": {"input_tokens": 14, "output_tokens": 10, "service_tier": "standard"}
   }
   

2. Now send one that mentions the blocked word:

   curl -k -X POST "https://localhost:8443/providers/anthropic/v1/messages" \
     -H "Content-Type: application/json" \
     -H "anthropic-version: 2023-06-01" \
     -d '{"model": "claude-sonnet-4-5-20250929", "max_tokens": 100, "messages": [{"role": "user", "content": "My password is 1234567"}]}'
   

   This second call returns an HTTP 422 with a JSON body describing the guardrail violation, instead of a completion:

   {
     "type": "REGEX_GUARDRAIL",
     "message": {
       "action": "GUARDRAIL_INTERVENED",
       "actionReason": "Violation of regular expression detected.",
       "interveningGuardrail": "regex-guardrail",
       "direction": "REQUEST"
     }
   }
   

The JSON body shows which guardrail intervened and confirms the prompt was blocked before it reached the model. Now that content is filtered, the next step caps how much a caller can spend.

Step 7: Apply a token-based rate limit​

A token-based rate limit caps token usage per time window, which is how you stop one team from consuming a shared budget. Here, the rate limit is defined as an operationPolicies entry, scoped to /v1/messages with its own paths and methods, meaning that it only applies there, not to the whole provider.

1. Update the provider again, this time adding a token-based rate limit policy alongside the guardrail:

   curl -X PUT http://localhost:9090/api/management/v1/llm-providers/anthropic-provider \
     -H "Content-Type: application/yaml" \
     -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" \
     --data-binary @- <<'EOF'
   apiVersion: gateway.api-platform.wso2.com/v1
   kind: LlmProvider
   metadata:
     name: anthropic-provider
   spec:
     displayName: Anthropic Provider
     version: v1.0
     template: anthropic
     context: /providers/anthropic
     upstream:
       url: https://api.anthropic.com
       auth:
         type: api-key
         header: x-api-key
         value: <anthropic-apikey>
     accessControl:
       mode: deny_all
       exceptions:
         - path: /v1/messages
           methods: [POST]
         - path: /v1/models
           methods: [GET]
     globalPolicies:
       - name: regex-guardrail
         version: v1
         params:
           request:
             regex: "(?i).*password.*"
             invert: true
             jsonPath: "$.messages[0].content"
     operationPolicies:
       - name: token-based-ratelimit
         version: v1
         paths:
           - path: /v1/messages
             methods: [POST]
             params:
               totalTokenLimits:
                 - count: 200
                   duration: "1m"
   EOF
   

A 200-token budget per minute is low enough to hit in a few requests, which makes it easy to verify. Anthropic responses report input and output tokens separately, so the policy charges their sum against the total limit.

Step 8: Verify the rate limit​

1. Send the same prompt six times in quick succession and print only the status codes:

   for i in 1 2 3 4 5 6; do
     curl -k -s -o /dev/null -w "%{http_code}\n" -X POST "https://localhost:8443/providers/anthropic/v1/messages" \
       -H "Content-Type: application/json" \
       -H "anthropic-version: 2023-06-01" \
       -d '{"model": "claude-sonnet-4-5-20250929", "max_tokens": 50, "messages": [{"role": "user", "content": "Tell me a short poem."}]}'
   done
   

   Expected output:

   200
   200
   200
   200
   429
   429
   

Once the minute's budget is spent, the gateway answers 429 without calling the model.

2. Wait a minute, then send the request once more. It succeeds, because the window has rolled over and the budget is back.

Step 9: Shut down cleanly​

When you're done, shut down the gateway using one of the following options:

Option 1: Stop the gateway, keep your configuration

docker compose down

This stops the containers but keeps your data. When you restart with docker compose up, your provider, guardrail, and rate-limit configuration will all still be there.

Option 2: Full shutdown with a clean slate

docker compose down -v

This stops the containers and removes their data. The next startup will be a fresh install, with no persisted providers or configuration.

Frequently asked questions​

What is a self-hosted LLM gateway? An AI gateway you run on your own infrastructure (on-prem, private cloud, or Kubernetes) so prompts and control stay inside your perimeter, rather than using a vendor-managed SaaS.

Why would I self-host instead of using SaaS? Data residency and compliance, air-gapped environments, cost control at high volume, and avoiding vendor lock-in. The trade-off is that you operate the gateway yourself.

Can a self-hosted gateway front open-source models? Yes. A self-hosted gateway can route to self-hosted open-source models as well as cloud providers behind one unified API, which is common in air-gapped or hybrid setups.

What's the fastest way to deploy one? Containerize it: Docker for a single node, Kubernetes for high availability and autoscaling. Apply guardrails and rate limits before routing production traffic.

Is WSO2's AI Gateway open source? Yes. It's part of the open-source WSO2 API Platform and can be deployed self-hosted, hybrid, or as SaaS.

Conclusion​

A self-hosted LLM gateway trades operational effort for control: your prompts stay in your perimeter, you can serve air-gapped environments, and you avoid lock-in, at the cost of running the infrastructure yourself. The decision usually comes down to your hardest constraint, and for regulated, data-sensitive workloads, self-hosting wins.

To evaluate an open-source, self-hostable option that unifies AI with your API governance, explore the WSO2 AI Gateway. For where this sits in the bigger picture, see what an AI gateway is.