Skip to main content
Copy page
Version: Next

Configure Guardrail System Parameters

Some guardrails call an external service to do their work, such as a content safety API, an embedding provider, or a vector database. The endpoint and credentials those guardrails need are system parameters: deployment-wide settings supplied as Helm values on the gateway extension release, not per-policy settings you fill in when applying a guardrail.

Enabling one of these guardrails takes changes to two Helm releases:

ReleaseWhat it does
Gateway extension (api-platform-<org>-<env>)Supplies the system parameters, so the guardrail works
Agent Manager (amp)Raises a capability flag, so the guardrail is visible in the console

Do only the first and the guardrail works but nobody can select it. Do only the second and the guardrail appears in the Console and deploys without an error, but the gateway cannot build the policy chain for that LLM Service Provider. Every request through the provider then returns HTTP 500, including requests the guardrail would allow.

See Guardrails and Policies for how these layers relate.

Which Guardrails Need This​

Most guardrails need nothing here. These run entirely inside the gateway and work as soon as the image contains them:

regex-guardrail, pii-masking-regex, url-guardrail, json-schema-guardrail, word-count-guardrail, sentence-count-guardrail, content-length-guardrail

These call an external service and need system parameters:

GuardrailCapability flagConfiguration keys
aws-bedrock-guardrailawsBedrockawsbedrock_guardrail_region, awsbedrock_guardrail_id, awsbedrock_guardrail_version, awsbedrock_access_key_id, awsbedrock_secret_access_key, awsbedrock_session_token, awsbedrock_role_arn, awsbedrock_role_region, awsbedrock_role_external_id
azure-content-safety-content-moderationazureContentSafetyazurecontentsafety_endpoint, azurecontentsafety_key
granite-guardian-prompt-injectiongraniteGuardiangranite_guardian_endpoint, granite_guardian_model, granite_guardian_timeout, granite_guardian_api_key
nvidia-nemoguard-content-safetynemoGuardnemoguard_endpoint, nemoguard_model, nemoguard_timeout, nemoguard_api_key
semantic-prompt-guardsemanticGuardrailsembedding_provider, embedding_provider_endpoint, embedding_provider_model, embedding_provider_api_key
semantic-cachesemanticGuardrailsThe embedding keys above, plus embedding_provider_dimension, vector_db_provider, vector_db_provider_host, vector_db_provider_port, vector_db_provider_username, vector_db_provider_password, vector_db_provider_database, vector_db_provider_ttl

Rate limiting and JWT authentication policies also take system parameters, but through a different block. See Rate Limiting and Authentication.

How the Configuration Reaches the Gateway​

You do not configure the gateway pods directly. The gateway extension release renders its values into a ConfigMap named <extension-release>-config, which the environment's APIGateway resource references. When the content of that ConfigMap changes, the gateway operator redeploys the gateway stack with the new values. That redeployment restarts the gateway controller and rolls the gateway runtime.

Editing the rendered ConfigMap by hand is not a shortcut. The next helm upgrade of the extension fails with a field-manager conflict. Always change the values on the extension release.

Prerequisites​

  • A deployed environment with its API Platform Gateway. See Environment Management.
  • helm and kubectl configured against the cluster.
  • The external service the guardrail calls, set up and tested. See Prepare the External Service.
  • The VERSION and HELM_CHART_REGISTRY variables you installed Agent Manager with. On a production install these are set in On Your Environment.

Prepare the External Service​

Set up the service and test it directly before you change the cluster. A problem found here is a problem with the service or its credentials, not with the gateway.

  1. In the Azure portal, create a Content Safety resource.

  2. Open the resource's Keys and Endpoint page and copy the endpoint and one of the two keys.

  3. Test the key against the endpoint:

    curl -s -X POST "<endpoint>/contentsafety/text:analyze?api-version=2023-10-01" \
      -H "Ocp-Apim-Subscription-Key: <key>" \
      -H "Content-Type: application/json" \
      -d '{"text": "I am going to hurt you"}'

    The response lists each category with a severity. A 401 means the key does not belong to this endpoint.

Step 1: Find the Gateway Extension Release​

Each environment has its own gateway extension release, named api-platform-<org>-<env>. Its namespace depends on how the environment was created:

EnvironmentNamespace
Default environment, installed by following On Your Environmentopenchoreo-data-plane
Default environment, installed by the quick-start or k3d setup<org>-<env>, for example default-default
Environment added with add-environment.sh<org>-<env>

Rather than assuming, list the APIGateway resources. Each one carries the name of its extension release and lives in the same namespace:

kubectl get apigateway -A
NAMESPACE               NAME                           PROGRAMMED
openchoreo-data-plane   api-platform-default-default   True

Set the two values for the environment you are configuring:

export GATEWAY_NS="openchoreo-data-plane"
export GATEWAY_RELEASE="api-platform-default-default"

Under a split gateway topology, the environment also has an egress release, named api-platform-<org>-<env>-egress in namespace <org>-<env>-egress. Egress gateways carry the traffic to LLM providers, so they need the same configuration. Repeat Steps 2 to 4 for it.

Step 2: Store Credentials in a Kubernetes Secret​

The gateway configuration renders into a ConfigMap, not a Secret. Never put an API key directly in a Helm value. Supply it as an environment variable backed by a Secret, and reference that variable from the configuration.

Create the Secret in the gateway's namespace. The gateway pods mount it from there, so a Secret in any other namespace is not found. Gateways that share a namespace can share one Secret.

kubectl create secret generic azure-content-safety \
  --namespace ${GATEWAY_NS} \
  --from-literal=subscription-key='<your-key>'

Step 3: Write the Gateway Values Overlay​

Put the configuration in a file rather than passing --set. The value is multi-line TOML containing =, ,, and quotes, all of which --set parses as syntax.

Create guardrails.yaml:

apiGateway:
  config:
    config_toml: |
      azurecontentsafety_endpoint = "https://my-resource.cognitiveservices.azure.com"
      azurecontentsafety_key = '{{ env "APIP_GW_AZURECONTENTSAFETY_KEY" }}'
    systemExtraEnv:
      - name: APIP_GW_AZURECONTENTSAFETY_KEY
        valueFrom:
          secretKeyRef:
            name: azure-content-safety
            key: subscription-key

config_toml is raw TOML injected at the top of the gateway's generated config.toml. systemExtraEnv provides the environment variables its {{ env "..." }} tokens resolve to, and is applied to both the gateway controller and the gateway runtime, because both load that file.

Three rules govern this block:

  • Single-quote the {{ env }} tokens. The policy engine resolves them when it loads the configuration, not Helm.
  • Bare keys must precede any [table] header. TOML requires it, which is why this content is injected at the top of the file.
  • Never redefine a table the chart already emits, namely [policy_configurations.*], [controller.*], [router], and [policy_engine]. TOML rejects duplicate tables and the gateway runtime will crash-loop.
Keep every guardrail in one file

To configure several guardrails, put all their keys in the same config_toml block and all their variables in the same systemExtraEnv list, as the Both tab shows. A later upgrade with a different file replaces both values; it does not add to them. Applying a Bedrock-only file to a gateway that already has Azure configured removes the Azure settings.

Step 4: Apply the Values to the Gateway Extension Release​

helm upgrade ${GATEWAY_RELEASE} \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-api-platform-gateway-extension \
  --version ${VERSION} \
  --namespace ${GATEWAY_NS} \
  --reuse-values \
  --no-hooks \
  -f guardrails.yaml
  • --version keeps the chart at the version the environment was installed with. Without it, Helm pulls the latest published chart, and --reuse-values then applies your old values to a chart they may not fit.
  • --reuse-values keeps the settings the environment was created with, including its control plane wiring and key managers. Without it they revert to chart defaults and the gateway loses its connection to Agent Manager.
  • --no-hooks skips the bootstrap job, which runs as a pre-upgrade hook and is not needed for a configuration change. If you leave it in and Agent Manager is unreachable, the upgrade fails and the release is left in pending-upgrade.

The upgrade restarts the gateway controller. On a multi-node cluster the replacement pod can block on Multi-Attach error for volume. See the controller restart note for the fix.

Re-apply the overlay after re-running add-environment.sh

add-environment.sh installs the extension release with its own fixed set of values and does not reuse the existing ones. Re-running it, whether to repair an environment or to move it to a new chart version, removes config_toml and systemExtraEnv, and the guardrails stop working. Run Step 4 again afterwards.

Step 5: Make the Guardrail Visible in the Console​

Raise the matching capability flag on the Agent Manager release. These are plain booleans, so --set is fine:

helm upgrade amp \
  oci://${HELM_CHART_REGISTRY}/wso2-agent-manager \
  --version ${VERSION} \
  --namespace wso2-amp \
  --reuse-values \
  --set console.config.guardrailCapabilities.azureContentSafety=true

The available flags are awsBedrock, azureContentSafety, graniteGuardian, nemoGuard, and semanticGuardrails. All default to false. The flag is deployment-wide: it shows the guardrail for every environment, so configure Steps 2 to 4 on every environment where it will be used. In an environment you skip, an LLM Service Provider that uses the guardrail returns HTTP 500 for every request.

The console reads these flags at container start, so restart it:

kubectl rollout restart deployment -n wso2-amp -l app.kubernetes.io/component=console

Then hard-refresh the browser, since the configuration is served to the client and may be cached.

Verify the Configuration​

Confirm the values reached the rendered gateway configuration:

kubectl get configmap ${GATEWAY_RELEASE}-config -n ${GATEWAY_NS} \
  -o jsonpath='{.data.values\.yaml}' | grep -A 3 config_toml

Confirm the operator redeployed the gateway with them. PROGRAMMED turns False with reason ConfigChanged while the redeployment runs, then returns to True:

kubectl get apigateway ${GATEWAY_RELEASE} -n ${GATEWAY_NS}

Confirm the capability flag reached the console:

kubectl get configmap -n wso2-amp -o yaml | grep GUARDRAIL_CAP_

Then open an LLM Service Provider in the console, add a guardrail, and confirm the guardrail appears in the picker. Send a request through the provider to confirm it runs.

Rate Limiting and Authentication​

Rate limiting and JWT authentication policies take their system parameters from the structured apiGateway.config.policyConfigurations block instead of config_toml. The chart renders each section under [policy_configurations.<section>], which is why config_toml must never redefine those tables:

SectionPolicies
ratelimit_v1basic-ratelimit, advanced-ratelimit, token-based-ratelimit, llm-cost-based-ratelimit
llm_cost_ratelimit_v1llm-cost-based-ratelimit
jwtauth_v1jwt-auth, mcp-auth
mcpauth_v1mcp-auth

These are applied to the same extension release, the same way. Two cautions:

  • jwtauth_v1.keymanagers is a list. Setting one entry with --set replaces the whole list, so restate every key manager, as On Your Environment does.
  • Rate limiting counters are held in memory by default and are not shared between gateway replicas. See Gateway Hardening for switching to Redis.

For the full list of values, see the gateway extension chart reference.

Troubleshooting​

The guardrail is not in the console picker. The capability flag is not set, or the console was not restarted. Check GUARDRAIL_CAP_* in the console ConfigMap and restart the deployment.

Every request through the LLM Service Provider returns HTTP 500 after you add the guardrail. The gateway in that environment is missing a system parameter the guardrail needs, so it skipped building the provider's policy chain. The Console still shows the provider as deployed. The gateway runtime logs show the missing key when the provider is deployed, then one error per request:

Failed to build policy chain for route, skipping ... failed to resolve config for key "azureContentSafetyEndpoint"
Policy chain not found for route, returning 500

The capability flag only controls visibility; it does not supply credentials. Run Steps 2 to 4 for that environment's gateway extension release. Then check the rendered ConfigMap for the keys, and confirm the Secret named in systemExtraEnv exists in ${GATEWAY_NS}.

The guardrail worked, then stopped after environment maintenance. add-environment.sh was re-run and replaced the extension values. Run Step 4 again.

The gateway runtime is crash-looping after an upgrade. The injected TOML is invalid, most often a duplicate [table] the chart already emits, or a bare key placed after a table header. Check the runtime pod logs and helm get values ${GATEWAY_RELEASE} -n ${GATEWAY_NS}.

The APIGateway stays Programmed=False after the upgrade. Check the controller pods for a Multi-Attach error for volume and delete the older pod, as described in On Your Environment.

The upgrade fails and the release is stuck in pending-upgrade. The bootstrap job ran as a pre-upgrade hook and could not reach Agent Manager. Re-run with --no-hooks.

The gateway loses its connection to Agent Manager after an upgrade. --reuse-values was omitted, so the control plane settings reverted to chart defaults. Restore them with the command that created the release: add-environment.sh with the same ENV_NAME, or for the default environment of a production install, the gateway extension step in On Your Environment. Then run Step 4 again.

Next Steps​