AI Gateway
next
Deployment
Production Deployment

Resources and scaling¶

Resource limits¶

Set requests and limits on both components in production. Without them, a traffic spike on the gateway can starve other workloads on the node.

The allocations below match the ones used in the AI Gateway performance tests, so the published throughput and latency figures describe what these settings deliver.

Gateway Controller:

The controller handles artifact deployment and configuration distribution, not request traffic, so its load doesn't scale with the number of large language model (LLM) calls.

gateway:
  controller:
    deployment:
      resources:
        requests:
          cpu: 500m
          memory: 1Gi
        limits:
          cpu: 1000m
          memory: 2Gi

Gateway Runtime:

The runtime carries every request. It runs Envoy and the policy engine in one container. Each guardrail, personally identifiable information (PII) masking rule, and rate-limit policy adds per-request work in that container.

gateway:
  gatewayRuntime:
    deployment:
      resources:
        requests:
          cpu: 2000m
          memory: 2Gi
        limits:
          cpu: 4000m
          memory: 2Gi

Two things drive runtime CPU on an AI Gateway more than raw request rate:

Guardrails. PII masking, JSON schema validation, and URL guardrails inspect the full request and response bodies. The performance tests show measurably lower throughput with PII masking enabled, and lower again with PII masking plus URL and JSON schema guardrails, at the same concurrency.
Payload size. LLM request and response bodies are far larger than typical representational state transfer (REST) payloads, and body-inspecting policies buffer them. See Tune the gateway for AI traffic for the buffer limits that govern this.

Benchmark with your own proxies and policy set before settling on final numbers. A proxy with several guardrails needs noticeably more CPU per request than one with authentication alone.

Pod anti-affinity¶

Anti-affinity rules control how replicas spread across the cluster. Two topology keys matter in production:

`topologyKey`	Spread scope	What it protects against
`kubernetes.io/hostname`	Across nodes	Single-node failure
`topology.kubernetes.io/zone`	Across availability zones	Full availability zone outage

Combine both for maximum resilience. Spread across zones at the higher weight, then across nodes within a zone:

gateway:
  controller:
    deployment:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app.kubernetes.io/component: controller
                topologyKey: topology.kubernetes.io/zone
            - weight: 50
              podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app.kubernetes.io/component: controller
                topologyKey: kubernetes.io/hostname

If your cluster spans a single availability zone, use kubernetes.io/hostname on its own.

Apply the equivalent block to gateway.gatewayRuntime.deployment.affinity, matching on app.kubernetes.io/component: gateway-runtime.

Note

preferredDuringSchedulingIgnoredDuringExecution is a soft rule. The scheduler honors it when it can and still places the pod when it can't. Use requiredDuringSchedulingIgnoredDuringExecution for a hard guarantee, and accept that pods stay Pending when the constraint can't be met.

The chart also accepts topologySpreadConstraints on both components if you prefer even distribution over the weighted preference model.

Horizontal Pod Autoscaler¶

The horizontal pod autoscaler (HPA) adjusts replica counts from CPU utilization. The configuration below scales both components on CPU alone. To scale on memory as well, add a memory metric.

Configure the HPA on both components:

gateway:
  controller:
    hpa:
      enabled: true
      minReplicas: 2
      maxReplicas: 5
      targetCPUUtilizationPercentage: 70
  gatewayRuntime:
    hpa:
      enabled: true
      minReplicas: 2
      maxReplicas: 5
      targetCPUUtilizationPercentage: 70

When the HPA is enabled the chart omits the Deployment's replicas field, so replicaCount no longer applies.

The controller HPA needs an external database

Scaling the controller past one replica requires storage.type set to postgres or sqlserver. SQLite doesn't support concurrent replicas. See Database configuration.

Note

Keep minReplicas at 2 or higher on both components. At 1, a single pod restart is a full outage.

Autoscaling on CPU reacts more slowly to LLM traffic than to typical REST traffic, because an LLM request occupies a connection for seconds while contributing little CPU. If your load is dominated by long streaming responses, scale on a custom metric such as active connections through customMetrics rather than on CPU alone.

Pod Disruption Budget¶

A pod disruption budget (PDB) keeps a minimum number of replicas available during voluntary disruptions:

gateway:
  controller:
    podDisruptionBudget:
      enabled: true
      minAvailable: 50%
  gatewayRuntime:
    podDisruptionBudget:
      enabled: true
      minAvailable: 50%

This protects against voluntary disruptions such as node drains during cluster upgrades, autoscaler scale-down, and planned maintenance.

Set minAvailable or maxUnavailable, not both.

Drain in-flight requests¶

A streaming LLM response can stay open for minutes. The Kubernetes default grace period of 30 seconds cuts those connections off mid-response during a rolling update. Give the runtime longer to drain:

gateway:
  gatewayRuntime:
    deployment:
      terminationGracePeriodSeconds: 300
      strategy:
        type: RollingUpdate
        rollingUpdate:
          maxSurge: 1
          maxUnavailable: 0

maxUnavailable: 0 stops the rollout controller from removing an old pod before its replacement is ready.

← Database configuration | Tune the gateway for AI traffic →