# On Your Environment

Install Agent Manager on an existing Kubernetes cluster with production settings: a real domain, trusted TLS, external databases, and sealed secret storage. Phase 1 installs OpenChoreo and Thunder. Phase 2 installs Agent Manager.

Evaluating?

Use the [k3d guide](/agent-platform/docs/v2026.11.0/guides/on-k3d/.md) instead.

Short-term support (STS) release

2026.11.0 is a short-term support (STS) release available for evaluation. Agent Manager's charts and images, including the API Platform gateway, are hosted on `registry.wso2.com` and require registry credentials. To obtain these credentials, download Agent Manager from the [WSO2 website](https://wso2.com/products/downloads/?product=wso2agentmanager\&package=quickstart) to receive them by email, or use the credentials provided with your WSO2 subscription. Other components, such as OpenChoreo and the Gateway Operator chart, are hosted on public registries such as `ghcr.io`. See [About the STS Release](/agent-platform/docs/v2026.11.0/guides/install-with-a-wso2-subscription/.md) for more information.

Isolation tiers

Agents run under the standard **runc** runtime. For stronger isolation, see [gVisor](/agent-platform/docs/v2026.11.0/guides/isolation-tiers/gvisor/.md) and [Kata Containers](/agent-platform/docs/v2026.11.0/guides/isolation-tiers/kata/.md). Both need dedicated nodes.

## Before You Start[​](#before-you-start "Direct link to Before You Start")

### Cluster Requirements[​](#cluster-requirements "Direct link to Cluster Requirements")

* Kubernetes 1.32+ with 3 nodes, each with 4 CPU cores and 8 GB RAM
* LoadBalancer support
* A default StorageClass that provisions volumes. See [Cluster Requirements](#cluster-requirements).
* Linux 6.3+ on build nodes. See [Troubleshooting](#build-pods-fail-with-mount_attr_idmap--idmap-mounts-errors) for older kernels.
* A CNI that enforces `NetworkPolicy` (recommended)

On Red Hat OpenShift, including Azure Red Hat OpenShift, also follow [Appendix: Red Hat OpenShift](#appendix-red-hat-openshift).

### Tools[​](#tools "Direct link to Tools")

| Tool                                               | Version            |
| -------------------------------------------------- | ------------------ |
| [kubectl](https://kubernetes.io/docs/tasks/tools/) | v1.32+             |
| [Helm](https://helm.sh/docs/intro/install/)        | v3.12+ or Helm 4   |
| `curl`, `dig`, `openssl`, `python3`                | Any recent version |

Confirm that LoadBalancers provision before you start:

```
kubectl create deploy lbtest --image=nginx && kubectl expose deploy lbtest --port=80 --type=LoadBalancer

kubectl wait --for=jsonpath='{.status.loadBalancer.ingress}' svc/lbtest --timeout=5m && echo "LoadBalancer OK"

kubectl delete svc/lbtest deploy/lbtest
```

You need permission to create namespaces, CRDs, ClusterRoles, Helm releases, and LoadBalancer Services.

***

## Plan Your Deployment[​](#plan-your-deployment "Direct link to Plan Your Deployment")

Make these decisions first. The variables below drive every later step.

### 1. Hostnames[​](#1-hostnames "Direct link to 1. Hostnames")

Pick a base domain you control, such as `amp.yourdomain.com`. Every management hostname sits under it.

| Hostname                              | Serves                                                | DNS target                  |
| ------------------------------------- | ----------------------------------------------------- | --------------------------- |
| `console.<base>`                      | Console                                               | control-plane gateway       |
| `api-amp.<base>`                      | API (browser, `amctl`, MCP)                           | control-plane gateway       |
| `thunder.<base>`                      | Thunder login, plus `<handle>.<base>` per environment | control-plane gateway       |
| `cp.<base>`                           | Gateway control plane (optional)                      | control-plane gateway       |
| `traces.<base>`                       | Observer                                              | observability-plane gateway |
| `*.agents.<base>` and `agents.<base>` | Deployed agents                                       | data-plane gateway          |

A single `*.<base>` wildcard record covers the control-plane hostnames and the per-environment Thunder hostnames, which are created after install. [Step 10](#step-10-publish-dns-records) lists the exact records.

Download the release bundle. Every manifest, values file, and script this guide uses comes from it:

```
export VERSION="2026.11.0"

curl -fsSL -O "https://github.com/wso2/agent-manager/releases/download/amp/v${VERSION}/wso2-agent-manager-${VERSION}.tar.gz"

tar xzf "wso2-agent-manager-${VERSION}.tar.gz"

export AMP_DIST="$PWD"
```

Set the remaining variables. Replace `<version>` and `<project>` with your values from [About the STS Release](/agent-platform/docs/v2026.11.0/guides/install-with-a-wso2-subscription/.md). The version is `2026.11.0` for the release, or `2026.11.0-u<N>` for update N. The release bundle above serves both. `GATEWAY_REGISTRY` is where Phase 2 pulls the API Platform gateway from.

```
export VERSION="<version>"

export HELM_CHART_REGISTRY="registry.wso2.com/<project>/sts/helm-charts"

export GATEWAY_REGISTRY="registry.wso2.com/<project>/sts"

export AMP_NS="wso2-amp"

export BUILD_CI_NS="openchoreo-workflow-plane"

export OBSERVABILITY_NS="openchoreo-observability-plane"

export DEFAULT_NS="default"

export DATA_PLANE_NS="openchoreo-data-plane"

export THUNDER_NS="amp-thunder"



export BASE_DOMAIN="amp.yourdomain.com"



export CONSOLE_PUBLIC_HOST="console.${BASE_DOMAIN}"

export API_PUBLIC_HOST="api-amp.${BASE_DOMAIN}"

export THUNDER_PUBLIC_HOST="thunder.${BASE_DOMAIN}"

export CP_GW_PUBLIC_HOST="cp.${BASE_DOMAIN}"

export OBS_API_PUBLIC_HOST="traces.${BASE_DOMAIN}"

export AGENTS_DOMAIN="agents.${BASE_DOMAIN}"



export CONSOLE_PUBLIC_URL="https://${CONSOLE_PUBLIC_HOST}"

export API_PUBLIC_URL="https://${API_PUBLIC_HOST}"

export THUNDER_PUBLIC_URL="https://${THUNDER_PUBLIC_HOST}"

export OBS_API_PUBLIC_URL="https://${OBS_API_PUBLIC_HOST}"



export THUNDER_INTERNAL_URL="http://amp-thunder-extension-service.${THUNDER_NS}.svc.cluster.local:8090"



export INSTRUMENTATION_URL="http://default-default.gateway.localhost:19080/otel"
```

Sign Helm in to `registry.wso2.com` with your registry credentials. Phase 2 uses the same username and token to create the image pull secrets:

```
printf 'Registry username: ' && read -r AMP_REGISTRY_USERNAME

printf 'Registry token: ' && stty -echo && read -r AMP_REGISTRY_PASSWORD; stty echo; echo

printf '%s' "$AMP_REGISTRY_PASSWORD" | helm registry login registry.wso2.com --username "$AMP_REGISTRY_USERNAME" --password-stdin
```

note

Certificates are single-level wildcards (`*.<base>`). Every management hostname must sit directly under the base domain, with no second-level names such as `api.amp.<base>`.

### 2. TLS[​](#2-tls "Direct link to 2. TLS")

The main flow uses cert-manager with Let's Encrypt DNS-01. You need API credentials for the DNS provider that hosts your zone. [Step 3](#step-3-setup-tls-issuer) also covers a corporate CA. Certificate issuance needs only the zone, so it works before the gateway DNS records exist.

### 3. Decisions you cannot change later[​](#3-decisions-you-cannot-change-later "Direct link to 3. Decisions you cannot change later")

These settings are written to a database or certificate on first boot. Changing one afterwards means uninstalling and reinstalling that component.

| Decision                                                                           | Set in                                                           | Why it is frozen                                                                                                                                                                        |
| ---------------------------------------------------------------------------------- | ---------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Thunder's database (SQLite or PostgreSQL)                                          | [Step 4](#step-4-install-thunder-extension-identity-provider)    | Clients are seeded on first boot. `helm upgrade` does not re-seed.                                                                                                                      |
| Thunder's platform client secrets                                                  | [Step 4](#step-4-install-thunder-extension-identity-provider)    | Same seeding job                                                                                                                                                                        |
| `THUNDER_PUBLIC_URL`, `CONSOLE_PUBLIC_URL`, `API_PUBLIC_URL`, `OBS_API_PUBLIC_URL` | [Step 4](#step-4-install-thunder-extension-identity-provider)    | Stored as the issuer, redirect URIs, and MCP resource identifiers                                                                                                                       |
| Thunder replicas > 1                                                               | [Step 4](#step-4-install-thunder-extension-identity-provider)    | Requires external PostgreSQL and a shared Redis cache                                                                                                                                   |
| Each environment's Thunder database (SQLite or PostgreSQL)                         | [Prepare Storage](#prepare-storage-for-the-environments-thunder) | The provisioning script refuses to switch an installed environment to an empty database. Changing it means migrating the data yourself and setting `THUNDER_FORCE_STORAGE_CHANGE=true`. |
| Agent Manager's database                                                           | [Phase 2](#phase-2-agent-manager-installation)                   | Switching later is a data migration                                                                                                                                                     |
| Environment gateway topology                                                       | [Phase 2](#phase-2-agent-manager-installation)                   | Role is written at first registration. Changing it means recreating the environment.                                                                                                    |

Everything else, including OpenSearch sizing, the registry, replica counts, and isolation tiers, can change on a running platform. See the [production recommendations](/agent-platform/docs/v2026.11.0/guides/production-recommendations/.md).

### 4. Platform secrets[​](#4-platform-secrets "Direct link to 4. Platform secrets")

Generate real values for the platform's internal OAuth clients and the OpenSearch admin:

```
export AMP_API_CLIENT_SECRET="$(openssl rand -hex 32)"

export AMP_SYSTEM_CLIENT_SECRET="$(openssl rand -hex 32)"

export AMP_PUBLISHER_CLIENT_SECRET="$(openssl rand -hex 32)"

export AM_OBSERVER_CLIENT_SECRET="$(openssl rand -hex 32)"

export WORKFLOW_PUBLISHER_SECRET="$(openssl rand -hex 32)"

export OBSERVER_READER_SECRET="$(openssl rand -hex 32)"

export OPENSEARCH_USERNAME="admin"

export OPENSEARCH_PASSWORD="$(openssl rand -base64 24)"
```

Keep this shell session for the whole install. Store the values in your secret manager.

OTLP ingest

Agents inside the cluster can use the default `INSTRUMENTATION_URL`. For agents outside the cluster, install the API Platform Gateway extension with a public hostname (`--set gateway.vhost=https://otel.${BASE_DOMAIN}` and `--set gateway.hostname=otel.${BASE_DOMAIN}`), then set `INSTRUMENTATION_URL=https://otel.${BASE_DOMAIN}/otel`.

***

## Phase 1: OpenChoreo Platform[​](#phase-1-openchoreo-platform "Direct link to Phase 1: OpenChoreo Platform")

Phase 1 installs four OpenChoreo planes (Control, Data, Workflow, and Observability) and Thunder, the identity provider. Estimated time: 20 to 30 minutes.

### Step 1: Install Cluster Prerequisites[​](#step-1-install-cluster-prerequisites "Direct link to Step 1: Install Cluster Prerequisites")

Install the Gateway API CRDs:

```
kubectl apply --server-side --force-conflicts \

  -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.1/experimental-install.yaml
```

Add `--force-conflicts` if another controller, such as Traefik on k3s, already manages these CRDs.

Install cert-manager (v1.19.2):

* Helm 4
* Helm 3

```
helm upgrade --install --server-side=false cert-manager oci://quay.io/jetstack/charts/cert-manager \

  --namespace cert-manager \

  --create-namespace \

  --version v1.19.2 \

  --set crds.enabled=true \

  --set startupapicheck.timeout=5m \

  --wait --timeout 360s
```

```
helm upgrade --install cert-manager oci://quay.io/jetstack/charts/cert-manager \

  --namespace cert-manager \

  --create-namespace \

  --version v1.19.2 \

  --set crds.enabled=true \

  --set startupapicheck.timeout=5m \

  --wait --timeout 360s
```

Install External Secrets Operator (v1.3.2):

* Helm 4
* Helm 3

```
helm upgrade --install --server-side=false external-secrets oci://ghcr.io/external-secrets/charts/external-secrets \

  --namespace external-secrets \

  --create-namespace \

  --version 1.3.2 \

  --set installCRDs=true \

  --wait --timeout 180s
```

```
helm upgrade --install external-secrets oci://ghcr.io/external-secrets/charts/external-secrets \

  --namespace external-secrets \

  --create-namespace \

  --version 1.3.2 \

  --set installCRDs=true \

  --wait --timeout 180s
```

Install kgateway (v2.2.1), the gateway for the OpenChoreo planes:

* Helm 4
* Helm 3

```
helm upgrade --install --server-side=false kgateway-crds oci://cr.kgateway.dev/kgateway-dev/charts/kgateway-crds \

  --create-namespace \

  --namespace openchoreo-control-plane \

  --version v2.2.1



helm upgrade --install --server-side=false kgateway oci://cr.kgateway.dev/kgateway-dev/charts/kgateway \

  --namespace openchoreo-control-plane \

  --create-namespace \

  --version v2.2.1 \

  --set controller.extraEnv.KGW_ENABLE_GATEWAY_API_EXPERIMENTAL_FEATURES=true
```

```
helm upgrade --install kgateway-crds oci://cr.kgateway.dev/kgateway-dev/charts/kgateway-crds \

  --create-namespace \

  --namespace openchoreo-control-plane \

  --version v2.2.1



helm upgrade --install kgateway oci://cr.kgateway.dev/kgateway-dev/charts/kgateway \

  --namespace openchoreo-control-plane \

  --create-namespace \

  --version v2.2.1 \

  --set controller.extraEnv.KGW_ENABLE_GATEWAY_API_EXPERIMENTAL_FEATURES=true
```

Rancher Desktop or k3s

Remove Traefik first. See the [Rancher Desktop / k3s appendix](#appendix-rancher-desktop--k3s).

### Step 2: Set Up the Secret Stores[​](#step-2-set-up-the-secret-stores "Direct link to Step 2: Set Up the Secret Stores")

Every platform secret goes through External Secrets. OpenBao is required, because Agent Manager reads Git credentials for private repositories from it directly over the Vault API. Deployed agents' environment variables can use OpenBao or another External Secrets provider. See [a different store for deployed agents](#using-a-different-secret-store-for-deployed-agents).

* Production
* Evaluation only

Install OpenBao with persistent storage, sealed at rest:

* Helm 4
* Helm 3

```
helm upgrade --install --server-side=false openbao oci://ghcr.io/openbao/charts/openbao \

  --namespace openbao \

  --create-namespace \

  --version 0.25.6 \

  --set server.dataStorage.size=10Gi \

  --timeout 180s



# Wait until bao answers. Exit 2 means sealed but responding, which is the expected state.

until kubectl exec -n openbao openbao-0 -- bao status >/dev/null 2>&1; do

  [ $? -eq 2 ] && break

  sleep 5

done
```

```
helm upgrade --install openbao oci://ghcr.io/openbao/charts/openbao \

  --namespace openbao \

  --create-namespace \

  --version 0.25.6 \

  --set server.dataStorage.size=10Gi \

  --timeout 180s



# Wait until bao answers. Exit 2 means sealed but responding, which is the expected state.

until kubectl exec -n openbao openbao-0 -- bao status >/dev/null 2>&1; do

  [ $? -eq 2 ] && break

  sleep 5

done
```

Initialize and unseal. Store the keys and root token in your secret manager now. They are shown once.

```
kubectl exec -n openbao openbao-0 -- bao operator init -key-shares=5 -key-threshold=3

kubectl exec -n openbao openbao-0 -- bao operator unseal <unseal-key-1>

kubectl exec -n openbao openbao-0 -- bao operator unseal <unseal-key-2>

kubectl exec -n openbao openbao-0 -- bao operator unseal <unseal-key-3>
```

A sealed OpenBao stays sealed after every pod restart. For unattended restarts, enable auto-unseal against a cloud KMS. Choose your platform:

* Generic
* EKS
* GKE
* AKS

Set the `seal` stanza in `server.standalone.config`, and give the pod credentials for your KMS. You still run `bao operator init` once, but with `-recovery-shares` and `-recovery-threshold` in place of `-key-shares` and `-key-threshold`, which a KMS seal rejects. It returns recovery keys and the root token. Skip the `unseal` commands; the KMS unseals OpenBao.

Use AWS KMS with IAM Roles for Service Accounts:

1. Create a KMS key with an alias such as `alias/<cluster-name>-openbao-unseal`.
2. Create an IAM policy that allows `kms:Encrypt`, `kms:Decrypt`, and `kms:DescribeKey` on that key's ARN.
3. Bind the policy to the `openbao` ServiceAccount, which the chart creates. Use `--role-only` so eksctl does not create a second one.

```
eksctl create iamserviceaccount \

  --cluster <cluster-name> --region <region> \

  --namespace openbao --name openbao \

  --role-name <cluster-name>-openbao-kms --role-only \

  --attach-policy-arn <policy-arn> \

  --approve
```

4. Install OpenBao with this values file in place of the `--set` flags above. Then initialize it with recovery-key flags, and skip the `unseal` commands. The `-key-shares` init above fails with `parameters secret_shares,secret_threshold not applicable to seal type awskms`:

```
server:

  dataStorage:

    size: 10Gi

  serviceAccount:

    annotations:

      eks.amazonaws.com/role-arn: <role-arn>

  standalone:

    config: |

      ui = true



      listener "tcp" {

        tls_disable = 1

        address = "[::]:8200"

        cluster_address = "[::]:8201"

      }

      storage "file" {

        path = "/openbao/data"

      }

      seal "awskms" {

        region     = "<region>"

        kms_key_id = "<key-arn>"

      }
```

```
kubectl exec -n openbao openbao-0 -- bao operator init -recovery-shares=5 -recovery-threshold=3
```

Store the recovery keys and root token in your secret manager. `bao status` should show `Sealed false` once the pod is Ready.

Full GKE guidance is coming soon.

Full AKS guidance is coming soon.

Create the KV v2 mount, the Kubernetes auth roles, and the seeded secrets:

```
kubectl exec -n openbao openbao-0 -- sh -c "

export BAO_ADDR=http://127.0.0.1:8200

export BAO_TOKEN='<root-token>'



bao secrets enable -path=secret -version=2 kv

bao auth enable kubernetes

bao write auth/kubernetes/config kubernetes_host=\"https://\$KUBERNETES_PORT_443_TCP_ADDR:443\"



bao policy write openchoreo-secret-reader-policy - <<POLICY

path \"secret/data/*\" { capabilities = [\"read\"] }

path \"secret/metadata/*\" { capabilities = [\"list\", \"read\"] }

POLICY



bao policy write openchoreo-secret-writer-policy - <<POLICY

path \"secret/data/*\" { capabilities = [\"create\", \"read\", \"update\", \"delete\"] }

path \"secret/metadata/*\" { capabilities = [\"create\", \"read\", \"update\", \"delete\", \"list\"] }

POLICY



bao write auth/kubernetes/role/openchoreo-secret-reader-role \

  bound_service_account_names=default \

  bound_service_account_namespaces='dp*' \

  policies=openchoreo-secret-reader-policy ttl=20m



bao write auth/kubernetes/role/openchoreo-secret-writer-role \

  bound_service_account_names='*' \

  bound_service_account_namespaces='openbao,openchoreo-workflow-plane,wso2-amp' \

  policies=openchoreo-secret-writer-policy ttl=20m



bao kv put secret/workflow-plane-oauth-client-secret value='${WORKFLOW_PUBLISHER_SECRET}'

bao kv put secret/amp-publisher-client-secret value='${AMP_PUBLISHER_CLIENT_SECRET}'

bao kv put secret/amp-system-client-secret value='${AMP_SYSTEM_CLIENT_SECRET}'

bao kv put secret/observer-oauth-client-secret value='${OBSERVER_READER_SECRET}'

bao kv put secret/opensearch-username value='${OPENSEARCH_USERNAME}'

bao kv put secret/opensearch-password value='${OPENSEARCH_PASSWORD}'

"
```

Mint the token Agent Manager uses to write agent secrets:

```
AMP_BAO_TOKEN=$(kubectl exec -n openbao openbao-0 -- sh -c "

export BAO_ADDR=http://127.0.0.1:8200

export BAO_TOKEN='<root-token>'

bao token create -policy=openchoreo-secret-writer-policy -period=768h -format=json" \

  | python3 -c "import json,sys; print(json.load(sys.stdin)['auth']['client_token'])")



kubectl create namespace ${AMP_NS} --dry-run=client -o yaml | kubectl apply -f -

kubectl create secret generic amp-openbao-token -n ${AMP_NS} \

  --from-literal=openbao-token="${AMP_BAO_TOKEN}" \

  --from-literal=workflow-plane-openbao-token="${AMP_BAO_TOKEN}"
```

Renew the token

Phase 2 must reference this Secret. The chart defaults to the dev-mode token `root`, which a sealed OpenBao rejects. The platform still installs and passes health checks, but Git-backed repository pickers fail. The periodic token must be renewed before its 768-hour period lapses.

The values file runs OpenBao in dev mode, with in-memory storage and a well-known root token. It seeds the placeholder secrets for you.

* Helm 4
* Helm 3

```
helm upgrade --install --server-side=false openbao oci://ghcr.io/openbao/charts/openbao \

  --namespace openbao \

  --create-namespace \

  --version 0.25.6 \

  --values "${AMP_DIST}/deployments/single-cluster/values-openbao.yaml" \

  --timeout 180s



kubectl wait --for=condition=Ready pod -l app.kubernetes.io/name=openbao -n openbao --timeout=120s
```

```
helm upgrade --install openbao oci://ghcr.io/openbao/charts/openbao \

  --namespace openbao \

  --create-namespace \

  --version 0.25.6 \

  --values "${AMP_DIST}/deployments/single-cluster/values-openbao.yaml" \

  --timeout 180s



kubectl wait --for=condition=Ready pod -l app.kubernetes.io/name=openbao -n openbao --timeout=120s
```

Secrets are lost on every pod restart, including node drains. Do not use this beyond evaluation.

Configure the ClusterSecretStore. It must be named `default`, because the build templates reference that name:

```
kubectl apply -f - <<'EOF'

apiVersion: v1

kind: ServiceAccount

metadata:

  name: external-secrets-openbao

  namespace: openbao

---

apiVersion: external-secrets.io/v1

kind: ClusterSecretStore

metadata:

  name: default

spec:

  provider:

    vault:

      server: "http://openbao.openbao.svc:8200"

      path: "secret"

      version: "v2"

      auth:

        kubernetes:

          mountPath: "kubernetes"

          role: "openchoreo-secret-writer-role"

          serviceAccountRef:

            name: "external-secrets-openbao"

            namespace: "openbao"

EOF
```

#### Using a different secret store for deployed agents[​](#using-a-different-secret-store-for-deployed-agents "Direct link to Using a different secret store for deployed agents")

**Optional.** Skip this if OpenBao holds everything.

Deployed agents' environment variables can live in any [External Secrets provider](https://external-secrets.io/latest/provider/aws-secrets-manager/). Add a second store next to `default`. This example uses Google Cloud Secret Manager:

```
kubectl apply -f - <<'EOF'

apiVersion: external-secrets.io/v1

kind: ClusterSecretStore

metadata:

  name: dataplane-secrets

spec:

  provider:

    # Replace with your provider's stanza. See the External Secrets provider docs.

    gcpsm:

      projectID: "your-project"

      auth:

        workloadIdentity:

          clusterLocation: us-central1

          clusterName: your-cluster

          serviceAccountRef:

            name: external-secrets

            namespace: external-secrets

EOF
```

Then set `secretStoreRef.name: dataplane-secrets` on the ClusterDataPlane in [Step 6](#step-6-setup-data-plane). Leave the workflow plane on `default`.

Keep `default` on OpenBao

Repointing `default` breaks two things. Git credentials are read over the Vault API, so private-repository pickers stop listing anything. The platform's seeded secrets also use KV v2 remote keys that other providers do not match, and a mismatch shows as `SecretSyncedError`.

### Step 3: Setup TLS Issuer[​](#step-3-setup-tls-issuer "Direct link to Step 3: Setup TLS Issuer")

Every gateway certificate references one ClusterIssuer named `openchoreo-ca`. Choose the tab for your certificate authority:

* Let's Encrypt (DNS-01)
* Corporate CA
* Self-signed (evaluation only)

Confirm the zone is delegated. The output must list your DNS provider's nameservers:

```
dig +short NS <your-zone> @1.1.1.1
```

Create the DNS credentials and the issuer. The example uses Route 53. Cloudflare, Google Cloud DNS, Azure DNS, and [others](https://cert-manager.io/docs/configuration/acme/dns01/) use the same structure.

```
kubectl create secret generic route53-credentials \

  --namespace cert-manager \

  --from-literal=secret-access-key='<AWS_SECRET_ACCESS_KEY>'



kubectl apply -f - <<EOF

apiVersion: cert-manager.io/v1

kind: ClusterIssuer

metadata:

  name: openchoreo-ca

spec:

  acme:

    server: https://acme-v02.api.letsencrypt.org/directory

    email: ops@yourdomain.com

    privateKeySecretRef:

      name: letsencrypt-account-key

    solvers:

      - dns01:

          route53:

            region: us-east-1

            accessKeyID: <AWS_ACCESS_KEY_ID>

            secretAccessKeySecretRef:

              name: route53-credentials

              key: secret-access-key

        selector:

          dnsZones:

            - "${BASE_DOMAIN}"

EOF
```

tip

Use the staging server (`https://acme-staging-v02.api.letsencrypt.org/directory`) while you iterate. Production allows 5 duplicate certificates per week.

On EKS, cert-manager can use IAM Roles for Service Accounts or Pod Identity instead of a static key. Bind a role scoped to your hosted zone and reduce the solver to `region` and `hostedZoneID`.

Store your intermediate CA certificate and key, then create a CA issuer:

```
kubectl create secret tls corporate-ca-secret \

  --namespace cert-manager \

  --cert=/path/to/intermediate-ca.crt \

  --key=/path/to/intermediate-ca.key



kubectl apply -f - <<'EOF'

apiVersion: cert-manager.io/v1

kind: ClusterIssuer

metadata:

  name: openchoreo-ca

spec:

  ca:

    secretName: corporate-ca-secret

EOF
```

Use the [nip.io appendix](#appendix-installing-without-a-domain-nipio). It creates a self-signed chain under the same `openchoreo-ca` name.

### Step 4: Install Thunder Extension (Identity Provider)[​](#step-4-install-thunder-extension-identity-provider "Direct link to Step 4: Install Thunder Extension (Identity Provider)")

Thunder handles login, API keys, and token exchange for the platform. The browser reaches it at `https://${THUNDER_PUBLIC_HOST}`.

These settings are permanent

Thunder writes its issuer, redirect URIs, and OAuth clients to its database on first boot. Confirm the values in [Plan Your Deployment](#plan-your-deployment) before you run the install. Getting one wrong means uninstalling, discarding Thunder's data, and reinstalling.

* External PostgreSQL
* In-pod SQLite (evaluation only)

Create the four databases and a role that owns them. Thunder does not create its own tables on PostgreSQL. On managed services, grant the admin role membership first, or `CREATE DATABASE ... OWNER` fails:

```
CREATE ROLE thunder LOGIN PASSWORD '...';

GRANT thunder TO CURRENT_USER;  -- required on managed PostgreSQL; harmless elsewhere

CREATE DATABASE configdb OWNER thunder;  -- likewise entitydb, runtime_transient, runtime_persistent
```

Load the schema into each database from the Thunder image:

```
THUNDER_IMAGE="ghcr.io/thunder-id/thunderid:1.0.0"

docker create --name thunder-schema "${THUNDER_IMAGE}"

for db in configdb entitydb runtime_transient runtime_persistent; do

  docker cp "thunder-schema:/opt/thunderid/dbscripts/${db}/postgres.sql" "./thunder-${db}.sql"

  psql "postgresql://thunder@your-db.example.com:5432/${db}" -v ON_ERROR_STOP=1 -f "./thunder-${db}.sql"

done

docker rm thunder-schema
```

You should see 20 tables in `configdb`, 5 in `entitydb`, 13 in `runtime_transient`, and 7 in `runtime_persistent`.

Database on a private network

If `psql` on your machine cannot reach the database, which is the case for a private RDS, Cloud SQL, or Azure instance, keep the `docker create` and `docker cp` lines and run `psql` from a pod in the cluster instead:

```
kubectl run thunder-psql --image=postgres:17 --restart=Never \

  --env=PGHOST=your-db.example.com --env=PGSSLMODE=require -- sleep 3600

kubectl wait --for=condition=Ready pod/thunder-psql --timeout=120s

for db in configdb entitydb runtime_transient runtime_persistent; do

  kubectl cp "./thunder-${db}.sql" "thunder-psql:/tmp/thunder-${db}.sql"

  kubectl exec thunder-psql -- env PGUSER=thunder PGPASSWORD='<THUNDER_DB_PASSWORD>' \

    psql -d "${db}" -v ON_ERROR_STOP=1 -f "/tmp/thunder-${db}.sql"

done

kubectl delete pod thunder-psql
```

The same pod works for the `CREATE ROLE` and `CREATE DATABASE` statements: run `kubectl exec -it thunder-psql -- env PGUSER=<admin-user> psql -d postgres` before you delete it.

Store the password in a Secret, so it stays out of Helm history, then write the values file:

```
kubectl create namespace ${THUNDER_NS} --dry-run=client -o yaml | kubectl apply -f -

kubectl create secret generic thunder-db-credentials \

  --namespace ${THUNDER_NS} \

  --from-literal=password='<THUNDER_DB_PASSWORD>'



export THUNDER_DB_HOST="your-db.example.com"



cat > thunder-db-values.yaml <<EOF

thunder:

  configuration:

    database:

      config:

        type: postgres

        postgres:

          hostname: "${THUNDER_DB_HOST}"

          port: "5432"

          name: configdb

          username: thunder

          sslmode: require

          passwordRef:

            name: thunder-db-credentials

            key: password

      entity:

        type: postgres

        postgres:

          hostname: "${THUNDER_DB_HOST}"

          port: "5432"

          name: entitydb

          username: thunder

          sslmode: require

          passwordRef:

            name: thunder-db-credentials

            key: password

      runtime_transient:

        type: postgres

        postgres:

          hostname: "${THUNDER_DB_HOST}"

          port: "5432"

          name: runtime_transient

          username: thunder

          sslmode: require

          passwordRef:

            name: thunder-db-credentials

            key: password

      runtime_persistent:

        type: postgres

        postgres:

          hostname: "${THUNDER_DB_HOST}"

          port: "5432"

          name: runtime_persistent

          username: thunder

          sslmode: require

          passwordRef:

            name: thunder-db-credentials

            key: password

EOF
```

Use `sslmode: verify-full` if your provider gives you a CA to pin.

The default keeps all four databases in SQLite on a 1Gi volume. Thunder is limited to one replica. Write an empty overrides file so the install command stays the same:

```
echo '{}' > thunder-db-values.yaml
```

Install Thunder:

* Helm 4
* Helm 3

```
helm install --server-side=false amp-thunder-extension \

  oci://${HELM_CHART_REGISTRY}/wso2-amp-thunder-extension \

  --version ${VERSION} \

  --namespace ${THUNDER_NS} \

  --create-namespace \

  --set thunder.ocIngress.hostname="${THUNDER_PUBLIC_HOST}" \

  --set thunder.configuration.server.publicUrl="${THUNDER_PUBLIC_URL}" \

  --set thunder.configuration.jwt.issuer="${THUNDER_PUBLIC_URL}" \

  --set thunder.configuration.gateClient.hostname="${THUNDER_PUBLIC_HOST}" \

  --set thunder.configuration.gateClient.scheme=https \

  --set thunder.configuration.gateClient.port=443 \

  --set "thunder.configuration.cors.allowedOrigins={${CONSOLE_PUBLIC_URL}}" \

  --set "thunder.bootstrap.ampConsoleClient.redirectUris={${CONSOLE_PUBLIC_URL}/login}" \

  --set thunder.bootstrap.agentManagerMcpBaseUrl="${API_PUBLIC_URL}" \

  --set thunder.bootstrap.observerMcpBaseUrl="${OBS_API_PUBLIC_URL}" \

  --set thunder.bootstrap.ampApiClient.clientSecret="${AMP_API_CLIENT_SECRET}" \

  --set thunder.bootstrap.ampSystemClient.clientSecret="${AMP_SYSTEM_CLIENT_SECRET}" \

  --set thunder.bootstrap.ampPublisherClient.clientSecret="${AMP_PUBLISHER_CLIENT_SECRET}" \

  --set thunder.bootstrap.amObserverClient.clientSecret="${AM_OBSERVER_CLIENT_SECRET}" \

  --set thunder.bootstrap.workloadPublisherClient.clientSecret="${WORKFLOW_PUBLISHER_SECRET}" \

  --set thunder.bootstrap.observerResourceReaderClient.clientSecret="${OBSERVER_READER_SECRET}" \

  --set thunder.ocIngress.https.enabled=false \

  --values thunder-db-values.yaml \

  --timeout 1800s



kubectl wait --for=condition=Available \

  deployment -l app.kubernetes.io/instance=amp-thunder-extension \

  -n ${THUNDER_NS} --timeout=300s
```

```
helm install amp-thunder-extension \

  oci://${HELM_CHART_REGISTRY}/wso2-amp-thunder-extension \

  --version ${VERSION} \

  --namespace ${THUNDER_NS} \

  --create-namespace \

  --set thunder.ocIngress.hostname="${THUNDER_PUBLIC_HOST}" \

  --set thunder.configuration.server.publicUrl="${THUNDER_PUBLIC_URL}" \

  --set thunder.configuration.jwt.issuer="${THUNDER_PUBLIC_URL}" \

  --set thunder.configuration.gateClient.hostname="${THUNDER_PUBLIC_HOST}" \

  --set thunder.configuration.gateClient.scheme=https \

  --set thunder.configuration.gateClient.port=443 \

  --set "thunder.configuration.cors.allowedOrigins={${CONSOLE_PUBLIC_URL}}" \

  --set "thunder.bootstrap.ampConsoleClient.redirectUris={${CONSOLE_PUBLIC_URL}/login}" \

  --set thunder.bootstrap.agentManagerMcpBaseUrl="${API_PUBLIC_URL}" \

  --set thunder.bootstrap.observerMcpBaseUrl="${OBS_API_PUBLIC_URL}" \

  --set thunder.bootstrap.ampApiClient.clientSecret="${AMP_API_CLIENT_SECRET}" \

  --set thunder.bootstrap.ampSystemClient.clientSecret="${AMP_SYSTEM_CLIENT_SECRET}" \

  --set thunder.bootstrap.ampPublisherClient.clientSecret="${AMP_PUBLISHER_CLIENT_SECRET}" \

  --set thunder.bootstrap.amObserverClient.clientSecret="${AM_OBSERVER_CLIENT_SECRET}" \

  --set thunder.bootstrap.workloadPublisherClient.clientSecret="${WORKFLOW_PUBLISHER_SECRET}" \

  --set thunder.bootstrap.observerResourceReaderClient.clientSecret="${OBSERVER_READER_SECRET}" \

  --set thunder.ocIngress.https.enabled=false \

  --values thunder-db-values.yaml \

  --timeout 1800s



kubectl wait --for=condition=Available \

  deployment -l app.kubernetes.io/instance=amp-thunder-extension \

  -n ${THUNDER_NS} --timeout=300s
```

Pass all six client secrets

The chart bootstraps six OAuth clients. A client you do not override keeps its shipped default, and the install still succeeds. The two `workloadPublisherClient` and `observerResourceReaderClient` lines are easy to miss. Without them, every agent build fails at the publish step with `401 invalid_client`.

Disable Thunder's local-dev gateway

`thunder.ocIngress.https.enabled` defaults to `true`, which adds an unused self-signed gateway for local k3d. The `--set` above turns it off.

MCP resource identifiers

`agentManagerMcpBaseUrl` and `observerMcpBaseUrl` must equal the public origins of the API and observer. Thunder rejects a `resource` that does not match exactly, with `invalid_target`.

Verify the issuer:

```
kubectl exec -n ${THUNDER_NS} deploy/amp-thunder-extension-deployment -- \

  wget -qO- http://localhost:8090/.well-known/openid-configuration 2>/dev/null \

  | grep -o '"issuer":"[^"]*"'

# Expected: "issuer":"${THUNDER_PUBLIC_URL}"
```

The install also generates the console's `admin` password. Retrieve it:

```
kubectl get secret amp-admin-credentials -n ${THUNDER_NS} -o jsonpath='{.data.password}' | base64 -d
```

Reinstalling Thunder resets platform data

Discarding Thunder's data also recreates the organization. Agent Manager keys its tenant data on that organization, so the old rows become invisible without an error. On anything but a fresh install, also discard Agent Manager's database and re-run [environment Thunder provisioning](#provision-thunder-identity-provider-for-the-default-environment).

### Step 5: Install OpenChoreo Control Plane[​](#step-5-install-openchoreo-control-plane "Direct link to Step 5: Install OpenChoreo Control Plane")

Create the wildcard certificate first. Issuance needs only the zone:

```
kubectl create namespace openchoreo-control-plane --dry-run=client -o yaml | kubectl apply -f -



kubectl apply -f - <<EOF

apiVersion: cert-manager.io/v1

kind: Certificate

metadata:

  name: cp-gateway-tls

  namespace: openchoreo-control-plane

spec:

  secretName: cp-gateway-tls

  issuerRef:

    name: openchoreo-ca

    kind: ClusterIssuer

  dnsNames:

    - "*.${BASE_DOMAIN}"

    - "${BASE_DOMAIN}"

  privateKey:

    rotationPolicy: Always

EOF



kubectl wait --for=condition=Ready certificate/cp-gateway-tls \

  -n openchoreo-control-plane --timeout=300s
```

Install the Control Plane. The OpenChoreo API stays inside the cluster. Agent Manager and the observer reach it over cluster DNS.

* Helm 4
* Helm 3

```
helm upgrade --install --server-side=false openchoreo-control-plane \

  oci://ghcr.io/openchoreo/helm-charts/openchoreo-control-plane \

  --version 1.2.8 \

  --namespace openchoreo-control-plane \

  --create-namespace \

  --values - <<EOF

features:

  secretManagement:

    enabled: true

openchoreoApi:

  config:

    server:

      publicUrl: "http://openchoreo-api.openchoreo-control-plane.svc.cluster.local:8080"

  http:

    enabled: false

    hostnames:

      - "api.${BASE_DOMAIN}"

backstage:

  enabled: false

  baseUrl: ""

  http:

    hostnames:

      - ""

security:

  oidc:

    issuer: "${THUNDER_PUBLIC_URL}"

    wellKnownEndpoint: "${THUNDER_INTERNAL_URL}/.well-known/openid-configuration"

    jwksUrl: "${THUNDER_INTERNAL_URL}/oauth2/jwks"

    authorizationUrl: "${THUNDER_PUBLIC_URL}/oauth2/authorize"

    tokenUrl: "${THUNDER_INTERNAL_URL}/oauth2/token"

gateway:

  tls:

    enabled: true

    hostname: "*.${BASE_DOMAIN}"

    certificateRefs:

      - name: cp-gateway-tls

EOF



kubectl wait --for=condition=Available \

  deployment --all -n openchoreo-control-plane --timeout=300s
```

```
helm upgrade --install openchoreo-control-plane \

  oci://ghcr.io/openchoreo/helm-charts/openchoreo-control-plane \

  --version 1.2.8 \

  --namespace openchoreo-control-plane \

  --create-namespace \

  --values - <<EOF

features:

  secretManagement:

    enabled: true

openchoreoApi:

  config:

    server:

      publicUrl: "http://openchoreo-api.openchoreo-control-plane.svc.cluster.local:8080"

  http:

    enabled: false

    hostnames:

      - "api.${BASE_DOMAIN}"

backstage:

  enabled: false

  baseUrl: ""

  http:

    hostnames:

      - ""

security:

  oidc:

    issuer: "${THUNDER_PUBLIC_URL}"

    wellKnownEndpoint: "${THUNDER_INTERNAL_URL}/.well-known/openid-configuration"

    jwksUrl: "${THUNDER_INTERNAL_URL}/oauth2/jwks"

    authorizationUrl: "${THUNDER_PUBLIC_URL}/oauth2/authorize"

    tokenUrl: "${THUNDER_INTERNAL_URL}/oauth2/token"

gateway:

  tls:

    enabled: true

    hostname: "*.${BASE_DOMAIN}"

    certificateRefs:

      - name: cp-gateway-tls

EOF



kubectl wait --for=condition=Available \

  deployment --all -n openchoreo-control-plane --timeout=300s
```

Secret management is required

`features.secretManagement.enabled=true` is off by default. Without it, every agent creation fails with a `500`.

Webhook race

If the install fails with `no endpoints available for service "controller-manager-webhook-service"`, wait for the Control Plane deployments to become ready, then rerun the install.

#### Patch the service-account entitlement claim (required with Thunder ≥ 0.45)[​](#patch-the-service-account-entitlement-claim-required-with-thunder--045 "Direct link to Patch the service-account entitlement claim (required with Thunder ≥ 0.45)")

Thunder 0.45+ puts the client name in `client_id`, but OpenChoreo 1.2.8 reads entitlements from `sub`. Without this patch, service calls return `200` with empty lists or `403`, and the gateway bootstrap fails with `Environment 'default' not found`.

```
patched_yaml=$(kubectl get configmap openchoreo-api-config -n openchoreo-control-plane -o yaml   | sed -E "s/claim:[[:space:]]*['\"]?sub['\"]?/claim: client_id/g")

echo "$patched_yaml" | kubectl apply --server-side --field-manager=helm --force-conflicts -f -



kubectl rollout restart deployment/openchoreo-api -n openchoreo-control-plane

kubectl rollout status deployment/openchoreo-api -n openchoreo-control-plane --timeout=120s



for binding in $(kubectl get clusterauthzrolebindings.openchoreo.dev -o jsonpath='{.items[*].metadata.name}'); do

  claim=$(kubectl get clusterauthzrolebinding.openchoreo.dev "$binding" -o jsonpath='{.spec.entitlement.claim}')

  if [ "$claim" = "sub" ]; then

    kubectl patch clusterauthzrolebinding.openchoreo.dev "$binding" --type=merge       -p '{"spec":{"entitlement":{"claim":"client_id"}}}'

  fi

done
```

Re-apply this patch after every `helm upgrade` of the Control Plane. See [After a chart upgrade](/agent-platform/docs/v2026.11.0/guides/production-recommendations/.md#after-a-chart-upgrade).

### Step 6: Setup Data Plane[​](#step-6-setup-data-plane "Direct link to Step 6: Setup Data Plane")

Copy the cluster-gateway CA into the data-plane namespace:

```
kubectl create namespace openchoreo-data-plane --dry-run=client -o yaml | kubectl apply -f -



CA_CRT=$(kubectl get secret cluster-gateway-ca \

  -n openchoreo-control-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)

kubectl create configmap cluster-gateway-ca \

  --from-literal=ca.crt="$CA_CRT" \

  -n openchoreo-data-plane --dry-run=client -o yaml | kubectl apply -f -



TLS_CRT=$(kubectl get secret cluster-gateway-ca \

  -n openchoreo-control-plane -o jsonpath='{.data.tls\.crt}' | base64 -d)

TLS_KEY=$(kubectl get secret cluster-gateway-ca \

  -n openchoreo-control-plane -o jsonpath='{.data.tls\.key}' | base64 -d)

kubectl create secret generic cluster-gateway-ca \

  --from-literal=tls.crt="$TLS_CRT" \

  --from-literal=tls.key="$TLS_KEY" \

  --from-literal=ca.crt="$CA_CRT" \

  -n openchoreo-data-plane --dry-run=client -o yaml | kubectl apply -f -
```

Create the agents wildcard certificate and install the Data Plane:

* Helm 4
* Helm 3

```
kubectl apply -f - <<EOF

apiVersion: cert-manager.io/v1

kind: Certificate

metadata:

  name: dp-gateway-tls

  namespace: openchoreo-data-plane

spec:

  secretName: dp-gateway-tls

  issuerRef:

    name: openchoreo-ca

    kind: ClusterIssuer

  dnsNames:

    - "*.${AGENTS_DOMAIN}"

    - "${AGENTS_DOMAIN}"

  privateKey:

    rotationPolicy: Always

EOF



kubectl wait --for=condition=Ready certificate/dp-gateway-tls \

  -n openchoreo-data-plane --timeout=300s



helm install --server-side=false openchoreo-data-plane \

  oci://ghcr.io/openchoreo/helm-charts/openchoreo-data-plane \

  --version 1.2.8 \

  --namespace openchoreo-data-plane \

  --create-namespace \

  --set clusterAgent.tls.generateCerts=true \

  --set gateway.tls.enabled=true \

  --set "gateway.tls.hostname=*.${AGENTS_DOMAIN}" \

  --set "gateway.tls.certificateRefs[0].name=dp-gateway-tls" \

  --set gateway.httpPort=80 \

  --set gateway.httpsPort=443 \

  --values "${AMP_DIST}/deployments/single-cluster/values-dp.yaml"



kubectl wait --for=condition=Available \

  deployment --all -n openchoreo-data-plane --timeout=600s
```

```
kubectl apply -f - <<EOF

apiVersion: cert-manager.io/v1

kind: Certificate

metadata:

  name: dp-gateway-tls

  namespace: openchoreo-data-plane

spec:

  secretName: dp-gateway-tls

  issuerRef:

    name: openchoreo-ca

    kind: ClusterIssuer

  dnsNames:

    - "*.${AGENTS_DOMAIN}"

    - "${AGENTS_DOMAIN}"

  privateKey:

    rotationPolicy: Always

EOF



kubectl wait --for=condition=Ready certificate/dp-gateway-tls \

  -n openchoreo-data-plane --timeout=300s



helm install openchoreo-data-plane \

  oci://ghcr.io/openchoreo/helm-charts/openchoreo-data-plane \

  --version 1.2.8 \

  --namespace openchoreo-data-plane \

  --create-namespace \

  --set clusterAgent.tls.generateCerts=true \

  --set gateway.tls.enabled=true \

  --set "gateway.tls.hostname=*.${AGENTS_DOMAIN}" \

  --set "gateway.tls.certificateRefs[0].name=dp-gateway-tls" \

  --set gateway.httpPort=80 \

  --set gateway.httpsPort=443 \

  --values "${AMP_DIST}/deployments/single-cluster/values-dp.yaml"



kubectl wait --for=condition=Available \

  deployment --all -n openchoreo-data-plane --timeout=600s
```

Set the gateway ports

The values files come from the k3d layout, where all planes share one load balancer. The `httpPort` and `httpsPort` flags above put each gateway on 80 and 443. Without them, the platform looks healthy but every published URL points at a closed port. Verify:

```
kubectl get svc gateway-default -n <plane-namespace> -o jsonpath='{.spec.ports[*].port}{"\n"}'

# Expected: 80 443
```

Register the Data Plane:

```
CA_CERT=$(kubectl get secret cluster-agent-tls \

  -n openchoreo-data-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)



kubectl apply -f - <<EOF

apiVersion: openchoreo.dev/v1alpha1

kind: ClusterDataPlane

metadata:

  name: default

  namespace: default

spec:

  planeID: default

  clusterAgent:

    clientCA:

      value: |

$(echo "$CA_CERT" | sed 's/^/        /')

  gateway:

    ingress:

      external:

        name: gateway-default

        namespace: openchoreo-data-plane

        http:

          host: "${AGENTS_DOMAIN}"

          listenerName: http

          port: 80

        https:

          host: "${AGENTS_DOMAIN}"

          listenerName: https

          port: 443

  secretStoreRef:

    name: default

EOF
```

`secretStoreRef` decides where deployed agents' environment variables are stored. Keep `default` for OpenBao, or name the separate store from [Step 2](#using-a-different-secret-store-for-deployed-agents).

### Step 7: Setup Workflow Plane[​](#step-7-setup-workflow-plane "Direct link to Step 7: Setup Workflow Plane")

Copy the cluster-gateway CA:

```
kubectl create namespace openchoreo-workflow-plane --dry-run=client -o yaml | kubectl apply -f -



CA_CRT=$(kubectl get secret cluster-gateway-ca \

  -n openchoreo-control-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)

kubectl create configmap cluster-gateway-ca \

  --from-literal=ca.crt="$CA_CRT" \

  -n openchoreo-workflow-plane --dry-run=client -o yaml | kubectl apply -f -



TLS_CRT=$(kubectl get secret cluster-gateway-ca \

  -n openchoreo-control-plane -o jsonpath='{.data.tls\.crt}' | base64 -d)

TLS_KEY=$(kubectl get secret cluster-gateway-ca \

  -n openchoreo-control-plane -o jsonpath='{.data.tls\.key}' | base64 -d)

kubectl create secret generic cluster-gateway-ca \

  --from-literal=tls.crt="$TLS_CRT" \

  --from-literal=tls.key="$TLS_KEY" \

  --from-literal=ca.crt="$CA_CRT" \

  -n openchoreo-workflow-plane --dry-run=client -o yaml | kubectl apply -f -
```

Install the Workflow Plane:

* Helm 4
* Helm 3

```
helm install --server-side=false openchoreo-workflow-plane \

  oci://ghcr.io/openchoreo/helm-charts/openchoreo-workflow-plane \

  --version 1.2.8 \

  --namespace openchoreo-workflow-plane \

  --create-namespace \

  --set clusterAgent.tls.generateCerts=true \

  --set argo-workflows.controller.resources.requests.cpu=50m \

  --set argo-workflows.controller.resources.requests.memory=64Mi \

  --set argo-workflows.controller.resources.limits.cpu=200m \

  --set argo-workflows.controller.resources.limits.memory=128Mi \

  --timeout 600s



kubectl wait --for=condition=Available \

  deployment --all -n openchoreo-workflow-plane --timeout=600s
```

```
helm install openchoreo-workflow-plane \

  oci://ghcr.io/openchoreo/helm-charts/openchoreo-workflow-plane \

  --version 1.2.8 \

  --namespace openchoreo-workflow-plane \

  --create-namespace \

  --set clusterAgent.tls.generateCerts=true \

  --set argo-workflows.controller.resources.requests.cpu=50m \

  --set argo-workflows.controller.resources.requests.memory=64Mi \

  --set argo-workflows.controller.resources.limits.cpu=200m \

  --set argo-workflows.controller.resources.limits.memory=128Mi \

  --timeout 600s



kubectl wait --for=condition=Available \

  deployment --all -n openchoreo-workflow-plane --timeout=600s
```

Register the Workflow Plane:

```
BP_CA_CERT=$(kubectl get secret cluster-agent-tls \

  -n openchoreo-workflow-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)



kubectl apply -f - <<EOF

apiVersion: openchoreo.dev/v1alpha1

kind: ClusterWorkflowPlane

metadata:

  name: default

  namespace: default

spec:

  planeID: default

  clusterAgent:

    clientCA:

      value: |

$(echo "$BP_CA_CERT" | sed 's/^/        /')

  secretStoreRef:

    name: default

EOF
```

Keep `secretStoreRef` on `default`. Git credentials and build secrets live in OpenBao, and Agent Manager reads Git credentials from it directly.

Container registry

The workflow plane pushes built images to a registry set by `global.registry.endpoint` in [Phase 2](#phase-2-agent-manager-installation). Use one with access control that creates repositories on push and accepts static credentials, such as Harbor (with auto-create on), ACR, Artifact Registry, or CNCF Distribution, and set `global.defaultResources.registry.tlsVerify=true`. Amazon ECR does not work; see the registry requirements in Phase 2. On EKS without another registry, see [Set up a build registry on EKS](#set-up-a-build-registry-on-eks) in Troubleshooting. See [OpenChoreo's registry guide](https://openchoreo.dev/docs/platform-engineer-guide/container-registry-configuration/#registry-providers).

### Step 8: Setup Observability Plane[​](#step-8-setup-observability-plane "Direct link to Step 8: Setup Observability Plane")

Copy the cluster-gateway CA:

```
kubectl create namespace openchoreo-observability-plane --dry-run=client -o yaml | kubectl apply -f -



CA_CRT=$(kubectl get secret cluster-gateway-ca \

  -n openchoreo-control-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)

kubectl create configmap cluster-gateway-ca \

  --from-literal=ca.crt="$CA_CRT" \

  -n openchoreo-observability-plane --dry-run=client -o yaml | kubectl apply -f -



TLS_CRT=$(kubectl get secret cluster-gateway-ca \

  -n openchoreo-control-plane -o jsonpath='{.data.tls\.crt}' | base64 -d)

TLS_KEY=$(kubectl get secret cluster-gateway-ca \

  -n openchoreo-control-plane -o jsonpath='{.data.tls\.key}' | base64 -d)

kubectl create secret generic cluster-gateway-ca \

  --from-literal=tls.crt="$TLS_CRT" \

  --from-literal=tls.key="$TLS_KEY" \

  --from-literal=ca.crt="$CA_CRT" \

  -n openchoreo-observability-plane --dry-run=client -o yaml | kubectl apply -f -
```

Create the ExternalSecrets for OpenSearch and the observer:

```
kubectl apply -f - <<'EOF'

apiVersion: external-secrets.io/v1

kind: ExternalSecret

metadata:

  name: opensearch-admin-credentials

  namespace: openchoreo-observability-plane

spec:

  refreshInterval: 1h

  secretStoreRef:

    kind: ClusterSecretStore

    name: default

  target:

    name: opensearch-admin-credentials

  data:

  - secretKey: username

    remoteRef:

      key: opensearch-username

      property: value

  - secretKey: password

    remoteRef:

      key: opensearch-password

      property: value

---

apiVersion: external-secrets.io/v1

kind: ExternalSecret

metadata:

  name: observer-secret

  namespace: openchoreo-observability-plane

spec:

  refreshInterval: 1h

  secretStoreRef:

    kind: ClusterSecretStore

    name: default

  target:

    name: observer-secret

  data:

  - secretKey: OPENSEARCH_USERNAME

    remoteRef:

      key: opensearch-username

      property: value

  - secretKey: OPENSEARCH_PASSWORD

    remoteRef:

      key: opensearch-password

      property: value

  - secretKey: UID_RESOLVER_OAUTH_CLIENT_SECRET

    remoteRef:

      key: observer-oauth-client-secret

      property: value

EOF



kubectl wait -n openchoreo-observability-plane \

  --for=condition=Ready externalsecret/opensearch-admin-credentials \

  externalsecret/observer-secret --timeout=60s
```

Apply the custom OpenTelemetry Collector config, which trace ingest requires:

```
kubectl apply -f "${AMP_DIST}/deployments/values/oc-collector-configmap.yaml" \

  -n openchoreo-observability-plane
```

Create the gateway certificate and install the Observability Plane:

* Helm 4
* Helm 3

```
kubectl apply -f - <<EOF

apiVersion: cert-manager.io/v1

kind: Certificate

metadata:

  name: obs-gateway-tls

  namespace: openchoreo-observability-plane

spec:

  secretName: obs-gateway-tls

  issuerRef:

    name: openchoreo-ca

    kind: ClusterIssuer

  dnsNames:

    - "*.${BASE_DOMAIN}"

    - "${BASE_DOMAIN}"

  privateKey:

    rotationPolicy: Always

EOF



kubectl wait --for=condition=Ready certificate/obs-gateway-tls \

  -n openchoreo-observability-plane --timeout=300s



helm install --server-side=false openchoreo-observability-plane \

  oci://ghcr.io/openchoreo/helm-charts/openchoreo-observability-plane \

  --version 1.2.8 \

  --namespace openchoreo-observability-plane \

  --create-namespace \

  --set gateway.tls.enabled=true \

  --set "gateway.tls.hostname=*.${BASE_DOMAIN}" \

  --set "gateway.tls.certificateRefs[0].name=obs-gateway-tls" \

  --set clusterAgent.tls.generateCerts=true \

  --set gateway.httpPort=80 \

  --set gateway.httpsPort=443 \

  --set observer.controlPlaneApiUrl="http://openchoreo-api.openchoreo-control-plane.svc.cluster.local:8080" \

  --set observer.extraEnv.AUTH_SERVER_BASE_URL="${THUNDER_PUBLIC_URL}" \

  --set security.oidc.jwksUrl="${THUNDER_INTERNAL_URL}/oauth2/jwks" \

  --set security.oidc.tokenUrl="${THUNDER_INTERNAL_URL}/oauth2/token" \

  --values "${AMP_DIST}/deployments/single-cluster/values-op.yaml" \

  --timeout 25m



kubectl wait --for=condition=Available \

  deployment --all -n openchoreo-observability-plane --timeout=900s



for sts in $(kubectl get statefulset -n openchoreo-observability-plane -o name 2>/dev/null); do

  kubectl rollout status "${sts}" -n openchoreo-observability-plane --timeout=900s

done
```

```
kubectl apply -f - <<EOF

apiVersion: cert-manager.io/v1

kind: Certificate

metadata:

  name: obs-gateway-tls

  namespace: openchoreo-observability-plane

spec:

  secretName: obs-gateway-tls

  issuerRef:

    name: openchoreo-ca

    kind: ClusterIssuer

  dnsNames:

    - "*.${BASE_DOMAIN}"

    - "${BASE_DOMAIN}"

  privateKey:

    rotationPolicy: Always

EOF



kubectl wait --for=condition=Ready certificate/obs-gateway-tls \

  -n openchoreo-observability-plane --timeout=300s



helm install openchoreo-observability-plane \

  oci://ghcr.io/openchoreo/helm-charts/openchoreo-observability-plane \

  --version 1.2.8 \

  --namespace openchoreo-observability-plane \

  --create-namespace \

  --set gateway.tls.enabled=true \

  --set "gateway.tls.hostname=*.${BASE_DOMAIN}" \

  --set "gateway.tls.certificateRefs[0].name=obs-gateway-tls" \

  --set clusterAgent.tls.generateCerts=true \

  --set gateway.httpPort=80 \

  --set gateway.httpsPort=443 \

  --set observer.controlPlaneApiUrl="http://openchoreo-api.openchoreo-control-plane.svc.cluster.local:8080" \

  --set observer.extraEnv.AUTH_SERVER_BASE_URL="${THUNDER_PUBLIC_URL}" \

  --set security.oidc.jwksUrl="${THUNDER_INTERNAL_URL}/oauth2/jwks" \

  --set security.oidc.tokenUrl="${THUNDER_INTERNAL_URL}/oauth2/token" \

  --values "${AMP_DIST}/deployments/single-cluster/values-op.yaml" \

  --timeout 25m



kubectl wait --for=condition=Available \

  deployment --all -n openchoreo-observability-plane --timeout=900s



for sts in $(kubectl get statefulset -n openchoreo-observability-plane -o name 2>/dev/null); do

  kubectl rollout status "${sts}" -n openchoreo-observability-plane --timeout=900s

done
```

Apply the same entitlement-claim patch to the observer. Without it, build-log queries return `403`:

```
patched_yaml=$(kubectl get configmap observer-auth-config -n openchoreo-observability-plane -o yaml   | sed -E "s/claim:[[:space:]]*['\"]?sub['\"]?/claim: client_id/g")

echo "$patched_yaml" | kubectl apply --server-side --field-manager=helm --force-conflicts -f -

kubectl rollout restart deployment/observer -n openchoreo-observability-plane

kubectl rollout status deployment/observer -n openchoreo-observability-plane --timeout=120s
```

Install the observability modules. The logs module deploys OpenSearch.

* Helm 4
* Helm 3

```
# Logs module, which deploys OpenSearch

helm upgrade --install --server-side=false observability-logs-opensearch \

  oci://ghcr.io/openchoreo/helm-charts/observability-logs-opensearch \

  --create-namespace \

  --namespace openchoreo-observability-plane \

  --version 0.5.5 \

  --set openSearchSetup.openSearchSecretName="opensearch-admin-credentials" \

  --set adapter.openSearchSecretName="opensearch-admin-credentials" \

  --set openSearch.persistence.size=100Gi \

  --set-string "openSearch.extraEnvs[0].name=OPENSEARCH_INITIAL_ADMIN_PASSWORD" \

  --set-string "openSearch.extraEnvs[0].value=${OPENSEARCH_PASSWORD}" \

  --timeout 10m



# Fluent Bit log collection. On GKE, disable the managed logging agent first.

helm upgrade --server-side=false observability-logs-opensearch \

  oci://ghcr.io/openchoreo/helm-charts/observability-logs-opensearch \

  --namespace openchoreo-observability-plane \

  --version 0.5.5 \

  --reuse-values \

  --set fluent-bit.enabled=true \

  --timeout 10m



# Metrics module. The values file widens the kube-state-metrics label allowlist,

# which usage attribution needs. It also pins Prometheus to a ClusterIP Service,

# so its unauthenticated API is never put on a public IP.

helm upgrade --install --server-side=false observability-metrics-prometheus \

  oci://ghcr.io/openchoreo/helm-charts/observability-metrics-prometheus \

  --create-namespace \

  --namespace openchoreo-observability-plane \

  --version 0.8.1 \

  --values "${AMP_DIST}/deployments/values/observability-metrics-prometheus.yaml" \

  --timeout 10m



# Tracing module, which uses the custom OTel Collector ConfigMap

helm upgrade --install --server-side=false observability-traces-opensearch \

  oci://ghcr.io/openchoreo/helm-charts/observability-tracing-opensearch \

  --create-namespace \

  --namespace openchoreo-observability-plane \

  --version 0.8.1 \

  --set openSearch.enabled=false \

  --set openSearchSetup.openSearchSecretName="opensearch-admin-credentials" \

  --set opentelemetry-collector.configMap.existingName="amp-opentelemetry-collector-config" \

  --timeout 10m
```

```
# Logs module, which deploys OpenSearch

helm upgrade --install observability-logs-opensearch \

  oci://ghcr.io/openchoreo/helm-charts/observability-logs-opensearch \

  --create-namespace \

  --namespace openchoreo-observability-plane \

  --version 0.5.5 \

  --set openSearchSetup.openSearchSecretName="opensearch-admin-credentials" \

  --set adapter.openSearchSecretName="opensearch-admin-credentials" \

  --set openSearch.persistence.size=100Gi \

  --set-string "openSearch.extraEnvs[0].name=OPENSEARCH_INITIAL_ADMIN_PASSWORD" \

  --set-string "openSearch.extraEnvs[0].value=${OPENSEARCH_PASSWORD}" \

  --timeout 10m



# Fluent Bit log collection. On GKE, disable the managed logging agent first.

helm upgrade observability-logs-opensearch \

  oci://ghcr.io/openchoreo/helm-charts/observability-logs-opensearch \

  --namespace openchoreo-observability-plane \

  --version 0.5.5 \

  --reuse-values \

  --set fluent-bit.enabled=true \

  --timeout 10m



# Metrics module. The values file widens the kube-state-metrics label allowlist,

# which usage attribution needs. It also pins Prometheus to a ClusterIP Service,

# so its unauthenticated API is never put on a public IP.

helm upgrade --install observability-metrics-prometheus \

  oci://ghcr.io/openchoreo/helm-charts/observability-metrics-prometheus \

  --create-namespace \

  --namespace openchoreo-observability-plane \

  --version 0.8.1 \

  --values "${AMP_DIST}/deployments/values/observability-metrics-prometheus.yaml" \

  --timeout 10m



# Tracing module, which uses the custom OTel Collector ConfigMap

helm upgrade --install observability-traces-opensearch \

  oci://ghcr.io/openchoreo/helm-charts/observability-tracing-opensearch \

  --create-namespace \

  --namespace openchoreo-observability-plane \

  --version 0.8.1 \

  --set openSearch.enabled=false \

  --set openSearchSetup.openSearchSecretName="opensearch-admin-credentials" \

  --set opentelemetry-collector.configMap.existingName="amp-opentelemetry-collector-config" \

  --timeout 10m
```

GKE: free port 2020

Fluent Bit binds port 2020 on every node. GKE's `fluentbit-gke` agent already uses it, so Fluent Bit crash-loops with `Cannot listen on 0.0.0.0:2020`. Disable GKE's agent before the upgrade above:

```
gcloud container clusters update <cluster> --zone <zone> --logging=NONE
```

Do not turn off `hostNetwork` instead. The filter then cannot reach the kubelet.

OpenSearch sizing and password

`openSearch.persistence.size` raises the volume from its 8Gi default. The single node holds all traces, logs, and metrics, so size it to your retention. `OPENSEARCH_INITIAL_ADMIN_PASSWORD` must match the value in OpenBao. The subchart alias is `openSearch`, with a capital **S**. Lowercase values are silently ignored.

Register the Observability Plane and link the other planes to it:

```
OP_CA_CERT=$(kubectl get secret cluster-agent-tls \

  -n openchoreo-observability-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)



kubectl apply -f - <<EOF

apiVersion: openchoreo.dev/v1alpha1

kind: ClusterObservabilityPlane

metadata:

  name: default

spec:

  planeID: default

  clusterAgent:

    clientCA:

      value: |

$(echo "$OP_CA_CERT" | sed 's/^/        /')

  observerURL: http://observer.openchoreo-observability-plane.svc.cluster.local:8080

EOF



kubectl patch clusterdataplane default -n default --type merge \

  -p '{"spec":{"observabilityPlaneRef":{"kind":"ClusterObservabilityPlane","name":"default"}}}'



kubectl patch clusterworkflowplane default -n default --type merge \

  -p '{"spec":{"observabilityPlaneRef":{"kind":"ClusterObservabilityPlane","name":"default"}}}'
```

### Step 9: Verify OpenChoreo Installation[​](#step-9-verify-openchoreo-installation "Direct link to Step 9: Verify OpenChoreo Installation")

```
kubectl get pods -n openchoreo-control-plane

kubectl get pods -n openchoreo-data-plane

kubectl get pods -n openchoreo-workflow-plane

kubectl get pods -n openchoreo-observability-plane

kubectl get pods -n amp-thunder

kubectl get clusterdataplane,clusterworkflowplane,clusterobservabilityplane
```

All pods should be `Running` or `Completed`.

### Step 10: Publish DNS Records[​](#step-10-publish-dns-records "Direct link to Step 10: Publish DNS Records")

Read the gateway addresses:

```
for ns in openchoreo-control-plane openchoreo-observability-plane openchoreo-data-plane; do

  printf '%-34s %s\n' "$ns:" \

    "$(kubectl get svc gateway-default -n $ns \

        -o jsonpath='{.status.loadBalancer.ingress[0].ip}{.status.loadBalancer.ingress[0].hostname}')"

done
```

| Record                                                | Points to                   |
| ----------------------------------------------------- | --------------------------- |
| `*.${BASE_DOMAIN}`                                    | control-plane gateway       |
| `traces.${BASE_DOMAIN}`                               | observability-plane gateway |
| `*.agents.${BASE_DOMAIN}` and `agents.${BASE_DOMAIN}` | data-plane gateway          |

The `*.${BASE_DOMAIN}` wildcard is required. Per-environment Thunder hostnames are not known until after install, and only the wildcard resolves them. Use A records for IPs (GKE, AKS) and CNAME records for hostnames (EKS).

```
dig +short thunder.${BASE_DOMAIN} traces.${BASE_DOMAIN} test.${AGENTS_DOMAIN}
```

***

## Phase 2: Agent Manager Installation[​](#phase-2-agent-manager-installation "Direct link to Phase 2: Agent Manager Installation")

Install the Agent Manager components with production settings: an external database, the platform secrets from [Plan Your Deployment](#4-platform-secrets), and two replicas of each stateless service. Create the Agent Manager database and role before you start. Switching later is a migration.

<!-- -->

<!-- -->

The Agent Manager installs as a set of Helm charts on top of OpenChoreo. The components fall into two groups based on install order:

1. **Agent Manager Core :** Gateway Operator, Agent Manager, Agent Sandbox Module and Platform Resources (agent component types, workflow templates etc). Each depends on the one before it.
2. **Extensions :** Secret Management, Observability, Evaluation extensions and the API Platform Gateway Extension.

Prerequisites

Thunder (identity provider) must be installed before proceeding — see the Thunder installation step in Phase 1. The variables `THUNDER_PUBLIC_URL`, `THUNDER_INTERNAL_URL`, `CONSOLE_PUBLIC_URL`, `API_PUBLIC_URL`, `OBS_API_PUBLIC_URL`, `CONSOLE_PUBLIC_HOST`, `API_PUBLIC_HOST`, `OBS_API_PUBLIC_HOST`, and `INSTRUMENTATION_URL` must be set from the Configuration Variables section. The production steps below additionally use `AGENTS_DOMAIN` — the domain deployed agents are served on, `agents.<your base domain>` — to build the gateway's hostname and virtual host. The `*_PUBLIC_HOST` variables carry the bare hostname (no scheme or port) and become the HTTPRoute hostnames on the plane gateways.

Create the `wso2-registry-credentials` image pull secret in the namespaces that run Agent Manager images, with the registry username and token from the Configuration Variables step:

```
for ns in "${AMP_NS}" "${OBSERVABILITY_NS}" workflows-default; do

  kubectl create namespace "$ns" --dry-run=client -o yaml | kubectl apply -f -

  kubectl create secret docker-registry wso2-registry-credentials \

    --namespace "$ns" \

    --docker-server=registry.wso2.com \

    --docker-username="$AMP_REGISTRY_USERNAME" \

    --docker-password="$AMP_REGISTRY_PASSWORD" \

    --dry-run=client -o yaml | kubectl apply -f -

done
```

Evaluation monitors run in `workflows-default`, the workflow namespace of the `default` organization.

Create the gateway's own pull secret, `wso2-gateway-registry-credentials`, in `${DATA_PLANE_NS}`. The gateway operator reads it to pull the gateway chart, and every gateway pod lists it to pull the gateway images. Besides the usual `.dockerconfigjson`, it carries the credentials as plain `username` and `password` keys, which is the form the operator reads:

```
kubectl create namespace "${DATA_PLANE_NS}" --dry-run=client -o yaml | kubectl apply -f -

kubectl create secret generic wso2-gateway-registry-credentials \

  --namespace "${DATA_PLANE_NS}" \

  --type=kubernetes.io/dockerconfigjson \

  --from-literal=.dockerconfigjson="$(printf '{"auths":{"registry.wso2.com":{"auth":"%s"}}}' "$(printf '%s:%s' "$AMP_REGISTRY_USERNAME" "$AMP_REGISTRY_PASSWORD" | base64 | tr -d '\n')")" \

  --from-literal=username="$AMP_REGISTRY_USERNAME" \

  --from-literal=password="$AMP_REGISTRY_PASSWORD" \

  --dry-run=client -o yaml | kubectl apply -f -
```

***

### Core Components[​](#core-components "Direct link to Core Components")

Install these in order — each depends on the one before it.

Upgrading an existing installation? Reverse Steps 2 and 4

The order above is for a fresh install, where no agents exist yet and the order is harmless. When upgrading an already-running platform, upgrade `wso2-amp-platform-resources-extension` (Step 4) at or before `wso2-agent-manager` (Step 2), not after: the service writes component parameters that the chart must already know how to read, so an agent created in the window between the two upgrades can have its live invoke URL change on the later chart upgrade.

#### Step 1: Gateway Operator[​](#step-1-gateway-operator "Direct link to Step 1: Gateway Operator")

Manages API Gateway resources and enables secure, authenticated trace ingestion into the Observability Plane.

The gateway controller encrypts stored credentials at rest and will not start without a key, so create one before installing the operator:

```
kubectl create namespace ${DATA_PLANE_NS} --dry-run=client -o yaml | kubectl apply -f -



openssl rand 32 > gateway-aesgcm.key

kubectl create secret generic gateway-encryption-keys \

  --namespace ${DATA_PLANE_NS} \

  --from-file=default-aesgcm256-v1.bin=gateway-aesgcm.key

rm -f gateway-aesgcm.key
```

The key is required, and the file name is fixed

The controller looks for a specific path, `/app/data/aesgcm-keys/default-aesgcm256-v1.bin`, so the Secret key must be exactly `default-aesgcm256-v1.bin`. Without it the controller crash-loops on `failed to initialize key manager: encryption key file not found for version aesgcm256-v1`, and the gateway never programs — the rest of the platform stays healthy, so the only symptom is that agents cannot be invoked.

This applies whatever `developmentMode` is set to. Earlier gateway releases auto-generated a key in development mode; from `1.2.0-beta` they do not.

Store the key with the rest of your platform secrets. It encrypts credentials the gateway holds, and losing it means those entries cannot be decrypted.

The gateway for this release, `2026.10.0`, is published only to `registry.wso2.com`. The operator itself still comes from `ghcr.io`, and pulls the gateway chart and images from `GATEWAY_REGISTRY`, set in the Configuration Variables step, with the `wso2-gateway-registry-credentials` secret. The controller in this gateway release serves its health check on the `admin` port under `/api/admin/v1/health`, so the probes are set as well.

<!-- -->

* Helm 4
* Helm 3

```
helm install --server-side=false gateway-operator \

  oci://ghcr.io/wso2/api-platform/helm-charts/gateway-operator \

  --version 0.11.3 \

  --namespace ${DATA_PLANE_NS} \

  --set logging.level=info \

  --set gatewayApi.installStandardCRDs=false \

  --set gateway.helm.chartName=oci://${GATEWAY_REGISTRY}/helm-charts/gateway \

  --set gateway.helm.chartVersion=2026.10.0 \

  --set gateway.helm.registryCredentialsSecret.name=wso2-gateway-registry-credentials \

  --set "gateway.values.imagePullSecrets[0]=wso2-gateway-registry-credentials" \

  --set gateway.values.gateway.controller.image.repository=${GATEWAY_REGISTRY}/gateway-controller \

  --set gateway.values.gateway.controller.image.tag=2026.10.0 \

  --set gateway.values.gateway.gatewayRuntime.image.repository=${GATEWAY_REGISTRY}/gateway-runtime \

  --set gateway.values.gateway.gatewayRuntime.image.tag=2026.10.0 \

  --set gateway.values.gateway.controller.encryptionKeys.enabled=true \

  --set gateway.values.gateway.controller.encryptionKeys.secretName=gateway-encryption-keys \

  --set gateway.values.gateway.controller.deployment.livenessProbe.httpGet.path=/api/admin/v1/health \

  --set gateway.values.gateway.controller.deployment.livenessProbe.httpGet.port=admin \

  --set gateway.values.gateway.controller.deployment.readinessProbe.httpGet.path=/api/admin/v1/health \

  --set gateway.values.gateway.controller.deployment.readinessProbe.httpGet.port=admin \

  --timeout 600s
```

```
helm install gateway-operator \

  oci://ghcr.io/wso2/api-platform/helm-charts/gateway-operator \

  --version 0.11.3 \

  --namespace ${DATA_PLANE_NS} \

  --set logging.level=info \

  --set gatewayApi.installStandardCRDs=false \

  --set gateway.helm.chartName=oci://${GATEWAY_REGISTRY}/helm-charts/gateway \

  --set gateway.helm.chartVersion=2026.10.0 \

  --set gateway.helm.registryCredentialsSecret.name=wso2-gateway-registry-credentials \

  --set "gateway.values.imagePullSecrets[0]=wso2-gateway-registry-credentials" \

  --set gateway.values.gateway.controller.image.repository=${GATEWAY_REGISTRY}/gateway-controller \

  --set gateway.values.gateway.controller.image.tag=2026.10.0 \

  --set gateway.values.gateway.gatewayRuntime.image.repository=${GATEWAY_REGISTRY}/gateway-runtime \

  --set gateway.values.gateway.gatewayRuntime.image.tag=2026.10.0 \

  --set gateway.values.gateway.controller.encryptionKeys.enabled=true \

  --set gateway.values.gateway.controller.encryptionKeys.secretName=gateway-encryption-keys \

  --set gateway.values.gateway.controller.deployment.livenessProbe.httpGet.path=/api/admin/v1/health \

  --set gateway.values.gateway.controller.deployment.livenessProbe.httpGet.port=admin \

  --set gateway.values.gateway.controller.deployment.readinessProbe.httpGet.path=/api/admin/v1/health \

  --set gateway.values.gateway.controller.deployment.readinessProbe.httpGet.port=admin \

  --timeout 600s
```

Pin the gateway chart, do not inherit it

`gateway.helm.chartVersion` decides which gateway chart the operator deploys, and it is the only thing that decides it — the `APIGateway` resource carries no chart version. Inheriting the operator's default has historically produced a chart whose templates do not render the controller's control-plane address, giving a gateway that installs, programs, and serves traffic while never registering with Agent Manager — so it never appears in the gateway list. Pin the chart explicitly, as above.

Pin the container images too. The chart version and the image tags are set separately, so `chartVersion` alone can leave the gateway on the chart's default images, which point at a public registry that does not carry this release. The four `image.repository`/`image.tag` values above pull the controller and runtime at `2026.10.0` from the registry. Check the pod images after installing rather than trusting the chart version alone.

These are the versions the gateway extension chart of your version is tested with. The chart records them in its `gatewayRelease` values, which the one-command installer reads, and an update can move them. To see them for your version:

```
helm show values "oci://${HELM_CHART_REGISTRY}/wso2-amp-api-platform-gateway-extension" --version "${VERSION}" | grep -A3 '^gatewayRelease:'
```

`gateway.values` is the operator's passthrough into that chart: anything set under it is merged into the values the gateway is deployed with, which is how the encryption keys above are wired.

Wait for the operator to be ready:

```
kubectl wait --for=condition=Available \

  deployment -l app.kubernetes.io/name=gateway-operator \

  -n ${DATA_PLANE_NS} --timeout=300s
```

Grant RBAC for WSO2 API Platform CRDs to the Data Plane cluster-agent:

```
kubectl apply -f - <<EOF

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRole

metadata:

  name: wso2-api-platform-gateway-module

rules:

  - apiGroups: ["gateway.api-platform.wso2.com"]

    resources: ["restapis", "apigateways"]

    verbs: ["*"]

  - apiGroups: ["gateway.kgateway.dev"]

    resources: ["backends"]

    verbs: ["*"]

---

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRoleBinding

metadata:

  name: wso2-api-platform-gateway-module

roleRef:

  apiGroup: rbac.authorization.k8s.io

  kind: ClusterRole

  name: wso2-api-platform-gateway-module

subjects:

  - kind: ServiceAccount

    name: cluster-agent-dataplane

    namespace: ${DATA_PLANE_NS}

EOF
```

info

The API Platform Gateway is deployed as an extension after Agent Manager is running — see Step 7 below.

#### Step 2: Agent Manager (API + Console + PostgreSQL)[​](#step-2-agent-manager-api--console--postgresql "Direct link to Step 2: Agent Manager (API + Console + PostgreSQL)")

The core platform: a Go API server, a React web console, and a database.

Tell the Console where your registry is

The charts and images for this release come from the private registry `registry.wso2.com`. Add `--set console.config.registry="registry.wso2.com/<project>/sts"` to the install command below. The Console then generates registry-aware commands for adding environments: they run the scripts from the release bundle and pull from the registry, with your registry credentials. Without it, the Console generates the public command, which fetches from GitHub and `ghcr.io` and fails for this release.

<!-- -->

The chart deploys an in-cluster PostgreSQL with a default password unless you point it at an external database. Use a managed service — sizing, backups, failover, and patching then come from your database platform. Create the database and a credentials Secret first:

```
kubectl create namespace ${AMP_NS} --dry-run=client -o yaml | kubectl apply -f -

kubectl create secret generic amp-db-credentials \

  --namespace ${AMP_NS} \

  --from-literal=password='<AMP_DB_PASSWORD>'
```

Then install, wiring the external database and the platform client secrets generated earlier:

* Helm 4
* Helm 3

```
helm install --server-side=false amp \

  oci://${HELM_CHART_REGISTRY}/wso2-agent-manager \

  --version ${VERSION} \

  --namespace ${AMP_NS} \

  --create-namespace \

  --set console.config.instrumentationUrl="${INSTRUMENTATION_URL}" \

  --set console.config.auth.baseUrl="${THUNDER_PUBLIC_URL}" \

  --set console.config.auth.signInRedirectURL="${CONSOLE_PUBLIC_URL}/login" \

  --set console.config.auth.signOutRedirectURL="${CONSOLE_PUBLIC_URL}/login" \

  --set console.config.apiBaseUrl="${API_PUBLIC_URL}" \

  --set agentManagerService.config.amObserverPublicURL="${OBS_API_PUBLIC_URL}" \

  --set console.ocIngress.hostname="${CONSOLE_PUBLIC_HOST}" \

  --set agentManagerService.ocIngress.hostname="${API_PUBLIC_HOST}" \

  --set agentManagerService.config.serverPublicURL="${API_PUBLIC_URL}" \

  --set agentManagerService.config.keyManager.issuer="${THUNDER_PUBLIC_URL}" \

  --set agentManagerService.config.keyManager.jwksUrl="${THUNDER_INTERNAL_URL}/oauth2/jwks" \

  --set agentManagerService.config.oidc.tokenUrl="${THUNDER_INTERNAL_URL}/oauth2/token" \

  --set agentManagerService.config.oidc.clientSecret="${AMP_API_CLIENT_SECRET}" \

  --set agentManagerService.config.thunder.baseURL="${THUNDER_PUBLIC_URL}" \

  --set agentManagerService.config.thunder.resolveToHost="${THUNDER_INTERNAL_URL#http://}" \

  --set agentManagerService.config.thunder.clientSecret="${AMP_SYSTEM_CLIENT_SECRET}" \

  --set agentManagerService.config.openChoreo.baseURL="http://openchoreo-api.openchoreo-control-plane.svc.cluster.local:8080" \

  --set agentManagerService.config.tlsEnabled=true \

  --set agentManagerService.replicaCount=2 \

  --set agentManagerService.autoscaling.minReplicas=2 \

  --set console.replicaCount=2 \

  --set agentManagerService.config.openbao.existingSecret=amp-openbao-token \

  --set agentManagerService.config.workflowPlaneOpenbao.existingSecret=amp-openbao-token \

  --set agentManagerService.config.workflowPlaneOpenbao.existingSecretKey=workflow-plane-openbao-token \

  --set postgresql.enabled=false \

  --set postgresql.external.host="your-db.example.com" \

  --set postgresql.external.port=5432 \

  --set postgresql.external.database=agentmanager \

  --set postgresql.external.username=agentmanager \

  --set postgresql.external.existingSecret=amp-db-credentials \

  --set postgresql.external.existingSecretPasswordKey=password \

  --set postgresql.external.sslMode=require \

  --timeout 1800s
```

```
helm install amp \

  oci://${HELM_CHART_REGISTRY}/wso2-agent-manager \

  --version ${VERSION} \

  --namespace ${AMP_NS} \

  --create-namespace \

  --set console.config.instrumentationUrl="${INSTRUMENTATION_URL}" \

  --set console.config.auth.baseUrl="${THUNDER_PUBLIC_URL}" \

  --set console.config.auth.signInRedirectURL="${CONSOLE_PUBLIC_URL}/login" \

  --set console.config.auth.signOutRedirectURL="${CONSOLE_PUBLIC_URL}/login" \

  --set console.config.apiBaseUrl="${API_PUBLIC_URL}" \

  --set agentManagerService.config.amObserverPublicURL="${OBS_API_PUBLIC_URL}" \

  --set console.ocIngress.hostname="${CONSOLE_PUBLIC_HOST}" \

  --set agentManagerService.ocIngress.hostname="${API_PUBLIC_HOST}" \

  --set agentManagerService.config.serverPublicURL="${API_PUBLIC_URL}" \

  --set agentManagerService.config.keyManager.issuer="${THUNDER_PUBLIC_URL}" \

  --set agentManagerService.config.keyManager.jwksUrl="${THUNDER_INTERNAL_URL}/oauth2/jwks" \

  --set agentManagerService.config.oidc.tokenUrl="${THUNDER_INTERNAL_URL}/oauth2/token" \

  --set agentManagerService.config.oidc.clientSecret="${AMP_API_CLIENT_SECRET}" \

  --set agentManagerService.config.thunder.baseURL="${THUNDER_PUBLIC_URL}" \

  --set agentManagerService.config.thunder.resolveToHost="${THUNDER_INTERNAL_URL#http://}" \

  --set agentManagerService.config.thunder.clientSecret="${AMP_SYSTEM_CLIENT_SECRET}" \

  --set agentManagerService.config.openChoreo.baseURL="http://openchoreo-api.openchoreo-control-plane.svc.cluster.local:8080" \

  --set agentManagerService.config.tlsEnabled=true \

  --set agentManagerService.replicaCount=2 \

  --set agentManagerService.autoscaling.minReplicas=2 \

  --set console.replicaCount=2 \

  --set agentManagerService.config.openbao.existingSecret=amp-openbao-token \

  --set agentManagerService.config.workflowPlaneOpenbao.existingSecret=amp-openbao-token \

  --set agentManagerService.config.workflowPlaneOpenbao.existingSecretKey=workflow-plane-openbao-token \

  --set postgresql.enabled=false \

  --set postgresql.external.host="your-db.example.com" \

  --set postgresql.external.port=5432 \

  --set postgresql.external.database=agentmanager \

  --set postgresql.external.username=agentmanager \

  --set postgresql.external.existingSecret=amp-db-credentials \

  --set postgresql.external.existingSecretPasswordKey=password \

  --set postgresql.external.sslMode=require \

  --timeout 1800s
```

Wait for all components — with an external database no PostgreSQL StatefulSet is deployed, so there is nothing to wait for there:

```
# API server

kubectl wait --for=condition=Available \

  deployment/amp-api -n ${AMP_NS} --timeout=600s



# Console

kubectl wait --for=condition=Available \

  deployment/amp-console -n ${AMP_NS} --timeout=600s
```

`agentManagerService.config.thunder.clientSecret` can instead be supplied through `agentManagerService.config.thunder.existingSecret` (key `thunder-client-secret`) if you prefer to keep it out of Helm release history.

`agentManagerService.config.apiKey` (the `X-API-Key` header value some internal callers use) is left unset above deliberately: the chart generates a random 32-character value itself when empty, the same way `encryptionKey` already does. Set `agentManagerService.config.apiKey.value` or `.existingSecret` explicitly only if something outside the platform needs to know this value in advance.

`tlsEnabled` decides which agent invoke URL the console publishes

`agentManagerService.config.tlsEnabled` defaults to `false`, and it is what selects between an Environment's `http` and `https` endpoint variants when the API builds an agent's invoke URL. Left at the default on an HTTPS platform, the console publishes an `http://` URL — which the browser then refuses to call from an HTTPS page, showing the request as `blocked:mixed-content` with no error anywhere in the platform. The agent itself is healthy and reachable; only the published URL is wrong.

Two values that look redundant but are not

`agentManagerService.autoscaling.minReplicas=2` is what actually gives the API two replicas. Autoscaling is **on by default** for this component with `minReplicas: 1`, so the HPA takes ownership of the replica count and scales `replicaCount=2` straight back down to one — silently, within seconds. The console has autoscaling off by default, which is why `replicaCount` alone appears to work there.

The two `openbao.existingSecret` values point at the Secret minted when you set up the secret stores. Without them the chart falls back to the dev-mode token `root`, which a sealed OpenBao rejects. `workflowPlaneOpenbao` is the one that bites: it is what Agent Manager uses to read Git credentials, so the repository and branch pickers fail and no agent can be created from a private repository, while the platform otherwise looks healthy.

Require TLS on the database connection

`postgresql.external.sslMode=require` above makes the connection **fail closed**. Left unset, the driver's default of `prefer` applies: it negotiates TLS when the server offers it but silently falls back to an unencrypted connection when it does not, so a misconfigured or replaced instance downgrades without any error.

`require` encrypts but does **not** verify the server's identity. To also authenticate the database, use `sslMode=verify-full` with `postgresql.external.sslRootCert`. Set it to `system` only when the certificate chains to a **public** CA; otherwise mount the provider's CA PEM through the chart's `volumes`/`volumeMounts` values and point `sslRootCert` at that path. Managed services differ here — Cloud SQL, for example, issues its server certificate from a Google-managed private CA in either CA mode, so `system` fails with `certificate verify failed` and the CA must be mounted.

Check the hostname too: `verify-full` matches the certificate's SAN exactly, so `postgresql.external.host` has to be a name the certificate actually covers rather than an IP. Cloud SQL's SAN carries a trailing dot, which means the host value needs one as well.

Enforce it on the server as well, where the client cannot downgrade it: *Require SSL* on Cloud SQL, `rds.force_ssl=1` on RDS, or *require secure transport* on Azure Database for PostgreSQL. Keep the database on a private network so it is never reachable from outside the VPC.

info

`keyManager.issuer` must be the public Thunder URL — the chart default is the k3d hostname. The authorization server the API advertises in its RFC 9728 protected resource metadata is derived from it, so `agentManagerService.config.oauthAuthorizationServers` only needs setting if it must differ. Likewise the MCP audience entry is derived from `serverPublicURL` above, so `keyManager.audience` only needs setting to change the accepted client IDs.

`thunder.baseURL` must also be the public Thunder URL, for a related but separate reason: every admin identity call (users, roles, groups) requests a token scoped to Thunder's System resource server via an RFC 8707 `resource` parameter derived from this value, and Thunder only recognizes the public URL as that resource server's registered identifier — anything else fails with `invalid_target`. Since the API runs in-cluster and the public hostname often doesn't resolve there, `thunder.resolveToHost` tells it which address to actually connect to while still presenting `thunder.baseURL` on the wire; set it to the in-cluster Thunder service (as above) whenever the public hostname isn't directly dialable from this cluster. Leaving `thunder.baseURL` at the chart default (or omitting `resolveToHost` when the public hostname can't resolve in-cluster) breaks every admin identity call with `invalid_target` or a DNS lookup failure, while the rest of the platform looks healthy.

Verify

```
kubectl get pods -n ${AMP_NS}

# Expected: amp-api-xxx (Running), amp-console-xxx (Running), plus

# amp-postgresql-0 (Running) only when using the in-cluster database
```

#### Step 3: Agent Sandbox Module[​](#step-3-agent-sandbox-module "Direct link to Step 3: Agent Sandbox Module")

Agents run as **sandboxed pods** managed by the [Agent Sandbox](https://agent-sandbox.sigs.k8s.io/) controller (`SandboxTemplate` / `SandboxWarmPool` resources) instead of plain Deployments. This module is **required** — without it, agent deployments cannot be rendered.

* Helm 4
* Helm 3

```
helm upgrade --install --server-side=false agent-sandbox \

  oci://ghcr.io/openchoreo/helm-charts/agent-sandbox \

  --version 0.1.1 \

  --namespace ${DATA_PLANE_NS} \

  --create-namespace \

  --wait \

  --timeout 10m \

  --set namespace=openchoreo-control-plane \

  --set dataPlaneNamespace=${DATA_PLANE_NS} \

  --set dataPlaneServiceAccount=cluster-agent-dataplane \

  --set upstream.version=v0.4.6
```

```
helm upgrade --install agent-sandbox \

  oci://ghcr.io/openchoreo/helm-charts/agent-sandbox \

  --version 0.1.1 \

  --namespace ${DATA_PLANE_NS} \

  --create-namespace \

  --wait \

  --timeout 10m \

  --set namespace=openchoreo-control-plane \

  --set dataPlaneNamespace=${DATA_PLANE_NS} \

  --set dataPlaneServiceAccount=cluster-agent-dataplane \

  --set upstream.version=v0.4.6
```

Wait for the controller:

```
kubectl wait -n agent-sandbox-system \

  --for=condition=available \

  --timeout=180s \

  deployment/agent-sandbox-controller
```

Verify

```
kubectl get crd \

  sandboxtemplates.extensions.agents.x-k8s.io \

  sandboxwarmpools.extensions.agents.x-k8s.io \

  sandboxclaims.extensions.agents.x-k8s.io

# All three CRDs should be listed



kubectl get clusterrole openchoreo-agent-sandbox-access

# RBAC granting the data-plane agent access to sandbox resources
```

Stronger isolation tiers

With this module, agents run sandboxed under the standard **runc** runtime. Optionally, individual environments can run agents under **gVisor** or **Kata Containers** for stronger isolation — these have hardware/OS requirements and need a dedicated node. See the [gVisor](/agent-platform/docs/v2026.11.0/guides/isolation-tiers/gvisor/.md) and [Kata Containers](/agent-platform/docs/v2026.11.0/guides/isolation-tiers/kata/.md) isolation tier guides.

#### Step 4: Platform Resources[​](#step-4-platform-resources "Direct link to Step 4: Platform Resources")

Creates the default Organization, Project, Environment, DeploymentPipeline, and workflow template resources that the console needs on first login. This chart also configures the **container registry endpoint** used by build workflows to push agent images.

<!-- -->

* Helm 4
* Helm 3

```
helm install --server-side=false amp-platform-resources \

  oci://${HELM_CHART_REGISTRY}/wso2-amp-platform-resources-extension \

  --version ${VERSION} \

  --namespace ${DEFAULT_NS} \

  --set global.oauth.tokenUrl="${THUNDER_INTERNAL_URL}/oauth2/token" \

  --set global.oauth.hostHeader="amp-thunder-extension-service.${THUNDER_NS}.svc.cluster.local" \

  --set global.apiServer.url="http://openchoreo-api.openchoreo-control-plane.svc.cluster.local:8080" \

  --set global.apiServer.hostHeader="openchoreo-api.openchoreo-control-plane.svc.cluster.local" \

  --set apiPlatformGateway.namespace=${DATA_PLANE_NS} \

  --set global.registry.endpoint="<your-registry-host>" \

  --set global.defaultResources.registry.tlsVerify=true \

  --timeout 1800s
```

```
helm install amp-platform-resources \

  oci://${HELM_CHART_REGISTRY}/wso2-amp-platform-resources-extension \

  --version ${VERSION} \

  --namespace ${DEFAULT_NS} \

  --set global.oauth.tokenUrl="${THUNDER_INTERNAL_URL}/oauth2/token" \

  --set global.oauth.hostHeader="amp-thunder-extension-service.${THUNDER_NS}.svc.cluster.local" \

  --set global.apiServer.url="http://openchoreo-api.openchoreo-control-plane.svc.cluster.local:8080" \

  --set global.apiServer.hostHeader="openchoreo-api.openchoreo-control-plane.svc.cluster.local" \

  --set apiPlatformGateway.namespace=${DATA_PLANE_NS} \

  --set global.registry.endpoint="<your-registry-host>" \

  --set global.defaultResources.registry.tlsVerify=true \

  --timeout 1800s
```

Set `global.registry.endpoint` to the registry the workflow plane pushes agent images to. Left unset, it defaults to the k3d registry `host.k3d.internal:10082`, and the first agent build fails to push. The registry requirements are below.

These four values default to k3d addresses

`global.oauth.tokenUrl` and `global.apiServer.url` default to `http://host.k3d.internal:8080`, which the single-cluster k3d layout reaches through the Docker host and routes by `Host` header. On any other cluster that name does not resolve, and the failure surfaces only when an agent is built: the workflow's `generate-workload` step prints `Failed to get access token:` with an empty body — a connection failure, not an authentication one — and the build ends with `cannot save parameter /mnt/vol/workload-cr.yaml`.

Nothing earlier in the install touches these values, so a platform that installs and verifies cleanly still cannot complete a single build until they are set. Point them at the in-cluster services as above.

Point the traits at the namespace the gateway is actually in

Deployed agents reach the gateway runtime by DNS name, and both the trace-export endpoint and the route agent traffic is forwarded through are built from `apiPlatformGateway.namespace`. Leaving it empty derives the per-org-env convention `<org>-<env>`, which is what `add-environment.sh` creates — but [Step 7](#step-7-api-platform-gateway-extension) installs the gateway extension into `${DATA_PLANE_NS}`, matching that chart's own `apiGateway.namespace` default. Set this to the same namespace, as above.

Get it wrong and nothing reports an error. The agent starts and serves requests while every span batch fails inside it with `Failed to resolve '…-gw-gateway-gateway-runtime.default-default' ([Errno -2] Name or service not known)`, so the Traces view stays empty; and the backend agent routes forward to has no reachable host, so invocation never reaches the gateway. The only evidence is a `Transient error` warning in the agent's own log.

If you install the gateway extension into `<org>-<env>` instead, leave this unset.

<!-- -->

##### Container registry configuration

The chart defaults are configured for a local k3d cluster with an in-cluster registry at `host.k3d.internal:10082`. For other environments, override the registry settings:

* Helm 4
* Helm 3

```bash
# Example: external registry with a base domain
helm install --server-side=false amp-platform-resources \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-platform-resources-extension \
  --version ${VERSION} \
  --namespace ${DEFAULT_NS} \
  --set global.baseDomain="yourdomain.com" \
  --set global.defaultResources.registry.tlsVerify=true \
  --timeout 1800s
# Registry endpoint will be: registry.yourdomain.com

# Example: explicit registry endpoint
helm install --server-side=false amp-platform-resources \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-platform-resources-extension \
  --version ${VERSION} \
  --namespace ${DEFAULT_NS} \
  --set global.registry.endpoint="your-registry.example.com:5000" \
  --set global.defaultResources.registry.tlsVerify=true \
  --timeout 1800s
```

```bash
# Example: external registry with a base domain
helm install amp-platform-resources \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-platform-resources-extension \
  --version ${VERSION} \
  --namespace ${DEFAULT_NS} \
  --set global.baseDomain="yourdomain.com" \
  --set global.defaultResources.registry.tlsVerify=true \
  --timeout 1800s
# Registry endpoint will be: registry.yourdomain.com

# Example: explicit registry endpoint
helm install amp-platform-resources \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-platform-resources-extension \
  --version ${VERSION} \
  --namespace ${DEFAULT_NS} \
  --set global.registry.endpoint="your-registry.example.com:5000" \
  --set global.defaultResources.registry.tlsVerify=true \
  --timeout 1800s
```

| Value                                        | Default                   | Description                                                 |
| -------------------------------------------- | ------------------------- | ----------------------------------------------------------- |
| `global.registry.endpoint`                   | `host.k3d.internal:10082` | Registry endpoint for pushing images                        |
| `global.baseDomain`                          | `""`                      | When set, registry endpoint becomes `registry.<baseDomain>` |
| `global.defaultResources.registry.tlsVerify` | `false`                   | Enable TLS verification for registry connections            |

Not every registry can back build workflows

Two properties are required, and the failure surfaces only on the first agent build, long after the platform installs and verifies cleanly:

* **Push-to-create.** Each build pushes its image as `<workflow-run-name>-image` — a repository name that is different on every run, so repositories cannot be pre-created. The registry must create them on push. **Amazon ECR does not** and fails every build with `repository ... not found`; the same applies to any registry that requires repositories to be provisioned first.
* **Static credentials.** The push step reads a fixed `.dockerconfigjson` from the `registry-push-secret` key in OpenBao (or pushes unauthenticated if the key is absent). Registries whose credentials expire — ECR again, with its 12-hour tokens — cannot be refreshed by anything in the build pipeline.

[CNCF Distribution](https://distribution.github.io/distribution/), Harbor (with a project set to auto-create), and GitLab's registry satisfy both. If you have no registry yet, running Distribution in-cluster behind an internal LoadBalancer with a cert-manager certificate is a few resources — but it ships with **no authentication**; put htpasswd in front of it before it carries anything real.

To push with credentials, store them in OpenBao before the first build. Write the registry's `.dockerconfigjson` to `registry-dockerconfig.json`, in the form `{"auths":{"<registry-host>":{"auth":"<base64 of user:token>"}}}`, then:

```
kubectl exec -i -n openbao openbao-0 -- env BAO_ADDR=http://127.0.0.1:8200 BAO_TOKEN='<root-token>' \

  bao kv put secret/registry-push-secret - < registry-dockerconfig.json
```

***

### Extensions[​](#extensions "Direct link to Extensions")

These can be installed in any order after Core is ready.

#### Step 5: Observability Extension (Agent Manager Observer)[​](#step-5-observability-extension-agent-manager-observer "Direct link to Step 5: Observability Extension (Agent Manager Observer)")

Deploys the observer service that queries and serves trace, log, and metrics data to the console and CLI.

<!-- -->

`amObserver.observer.idpClientSecret` is the observer's own outbound identity for calls to the OpenChoreo observer, and must match the `am-observer-client` secret seeded into Thunder. It can instead be supplied through `amObserver.observer.existingSecret` (keys `existingSecretClientIdKey`/`existingSecretClientSecretKey`, default `idpClientId`/`idpClientSecret`) if you prefer to keep it out of Helm release history, the same option `thunder.clientSecret` has above:

* Helm 4
* Helm 3

```
helm install --server-side=false amp-observability-traces \

  oci://${HELM_CHART_REGISTRY}/wso2-amp-observability-extension \

  --version ${VERSION} \

  --namespace ${OBSERVABILITY_NS} \

  --set amObserver.ocIngress.hostname="${OBS_API_PUBLIC_HOST}" \

  --set amObserver.publicUrl="${OBS_API_PUBLIC_URL}" \

  --set amObserver.auth.issuer="${THUNDER_PUBLIC_URL}" \

  --set amObserver.observer.idpClientSecret="${AM_OBSERVER_CLIENT_SECRET}" \

  --set amObserver.replicaCount=2 \

  --timeout 1800s



kubectl wait --for=condition=Available \

  deployment/amp-observer -n ${OBSERVABILITY_NS} --timeout=600s
```

```
helm install amp-observability-traces \

  oci://${HELM_CHART_REGISTRY}/wso2-amp-observability-extension \

  --version ${VERSION} \

  --namespace ${OBSERVABILITY_NS} \

  --set amObserver.ocIngress.hostname="${OBS_API_PUBLIC_HOST}" \

  --set amObserver.publicUrl="${OBS_API_PUBLIC_URL}" \

  --set amObserver.auth.issuer="${THUNDER_PUBLIC_URL}" \

  --set amObserver.observer.idpClientSecret="${AM_OBSERVER_CLIENT_SECRET}" \

  --set amObserver.replicaCount=2 \

  --timeout 1800s



kubectl wait --for=condition=Available \

  deployment/amp-observer -n ${OBSERVABILITY_NS} --timeout=600s
```

warning

`amObserver.auth.issuer` must be the **public** Thunder URL, not the in-cluster service URL. The observer validates the same user token the console and `amctl` send to the Agent Manager API, so its issuer has to match `agentManagerService.config.keyManager.issuer` from Step 2. Leave it at the chart default on a custom-domain install and the traces page stays empty while the observer logs `JWT validation failed ... invalid issuer`.

The chart derives `amObserver.oauth.authorizationServers` and the MCP audience entry from the values above; set them explicitly only if they must differ.

#### Step 6: Evaluation Extension[​](#step-6-evaluation-extension "Direct link to Step 6: Evaluation Extension")

Installs workflow templates for running automated evaluations (accuracy, safety, reasoning, tool usage) against agent traces.

* Helm 4
* Helm 3

```
helm install --server-side=false amp-evaluation-extension \

  oci://${HELM_CHART_REGISTRY}/wso2-amp-evaluation-extension \

  --version ${VERSION} \

  --namespace ${BUILD_CI_NS} \

  --timeout 1800s
```

```
helm install amp-evaluation-extension \

  oci://${HELM_CHART_REGISTRY}/wso2-amp-evaluation-extension \

  --version ${VERSION} \

  --namespace ${BUILD_CI_NS} \

  --timeout 1800s
```

info

Evaluation jobs publish scores using the `amp-publisher-client` OAuth2 credentials bootstrapped by the Thunder extension. The client secret is fetched from OpenBao at workflow runtime (`secret/amp-publisher-client-secret`) and must match the value set on `thunder.bootstrap.ampPublisherClient.clientSecret`.

caution

The eval job needs egress to your API server, and the NetworkPolicy matches it by address range after kube-proxy DNAT — the endpoint address, never the `kubernetes` Service ClusterIP. Check what that address is:

```
kubectl -n default get endpoints kubernetes
```

`networkPolicy.evaluationJob.apiServer.cidrs` defaults to all of RFC1918, which covers that address on most private clusters but also spans your pod and service CIDRs. Add this to the command above to narrow it to your control-plane subnet:

* Helm 4
* Helm 3

```
helm install --server-side=false amp-evaluation-extension \

  oci://${HELM_CHART_REGISTRY}/wso2-amp-evaluation-extension \

  --version ${VERSION} \

  --namespace ${BUILD_CI_NS} \

  --timeout 1800s \

  --set "networkPolicy.evaluationJob.apiServer.cidrs[0]=<control-plane-subnet>"
```

```
helm install amp-evaluation-extension \

  oci://${HELM_CHART_REGISTRY}/wso2-amp-evaluation-extension \

  --version ${VERSION} \

  --namespace ${BUILD_CI_NS} \

  --timeout 1800s \

  --set "networkPolicy.evaluationJob.apiServer.cidrs[0]=<control-plane-subnet>"
```

Setting it is required if `kubernetes` resolves to a **public** endpoint (public GKE/AKS control planes) or to `100.64.0.0/10`. The default will not match those, and every evaluation will publish its scores and then report FAILED.

#### Step 7: API Platform Gateway Extension[​](#step-7-api-platform-gateway-extension "Direct link to Step 7: API Platform Gateway Extension")

Registers the API Platform Gateway with the Agent Manager and deploys the gateway stack. **Install this last** — it requires the Agent Manager API to be healthy and Thunder to be ready for token exchange.

<!-- -->

The bootstrap job authenticates to the Agent Manager API as `amp-api-client`, so it needs the secret you seeded into Thunder. Keep it out of Helm release history with a Secret reference:

* Helm 4
* Helm 3

```
kubectl create secret generic gateway-idp-credentials \

  --namespace ${DATA_PLANE_NS} \

  --from-literal=client-id=amp-api-client \

  --from-literal=client-secret="${AMP_API_CLIENT_SECRET}"



helm install --server-side=false api-platform-default-default \

  oci://${HELM_CHART_REGISTRY}/wso2-amp-api-platform-gateway-extension \

  --version ${VERSION} \

  --namespace ${DATA_PLANE_NS} \

  --set agentManager.orgName=default \

  --set gateway.environment=default \

  --set gateway.type=BOTH \

  --set developmentMode=false \

  --set gateway.vhost="https://default-default.${AGENTS_DOMAIN}" \

  --set gateway.hostname="default-default.${AGENTS_DOMAIN}" \

  --set agentManager.idp.existingSecret=gateway-idp-credentials \

  --timeout 1800s



kubectl wait --for=condition=complete job/api-platform-default-default-bootstrap \

  -n ${DATA_PLANE_NS} --timeout=300s
```

```
kubectl create secret generic gateway-idp-credentials \

  --namespace ${DATA_PLANE_NS} \

  --from-literal=client-id=amp-api-client \

  --from-literal=client-secret="${AMP_API_CLIENT_SECRET}"



helm install api-platform-default-default \

  oci://${HELM_CHART_REGISTRY}/wso2-amp-api-platform-gateway-extension \

  --version ${VERSION} \

  --namespace ${DATA_PLANE_NS} \

  --set agentManager.orgName=default \

  --set gateway.environment=default \

  --set gateway.type=BOTH \

  --set developmentMode=false \

  --set gateway.vhost="https://default-default.${AGENTS_DOMAIN}" \

  --set gateway.hostname="default-default.${AGENTS_DOMAIN}" \

  --set agentManager.idp.existingSecret=gateway-idp-credentials \

  --timeout 1800s



kubectl wait --for=condition=complete job/api-platform-default-default-bootstrap \

  -n ${DATA_PLANE_NS} --timeout=300s
```

Set the vhost now — it cannot be changed later

`gateway.vhost` and `gateway.hostname` default to the local k3d addresses (`http://default-default.gateway.localhost:19080`). The bootstrap job writes them into Agent Manager at **first registration only**: on every later run it finds the gateway already present, logs `already exists`, and exits without reconciling the record. A `helm upgrade` with corrected values changes nothing, and there is no error to notice — the console simply keeps showing the `.localhost:19080` virtual host, which is what an operator would copy when wiring an external caller.

Correcting it after the fact means deleting the gateway registration and re-registering, so get these right on the first install.

`gateway.type=BOTH` registers this gateway to serve both inbound and outbound traffic, which is what a single-gateway environment needs. It matches the chart default and is set explicitly here because the role is written once at registration and never rewritten — see [Gateway roles and topology](/agent-platform/docs/v2026.11.0/guides/production-recommendations/.md#gateway-roles-and-topology) for the split alternative and its constraints.

`developmentMode=false` turns off the relaxed security checks the chart enables by default. It depends on the encryption key from [Step 1](#step-1-gateway-operator): without one the install fails immediately with `encryptionKeys must be enabled: at-rest encryption is mandatory`. Set it here rather than hardening afterwards, so the gateway is never registered in a relaxed state.

The extension already renders an OTLP ingest route for its own gateway (`<release>-otel-restapi` in the data-plane namespace), so nothing further is needed for the default environment. The standalone manifest below targets the per-environment namespace `<org>-<env>`, which does not exist until the first agent is deployed there — apply it only if you need that variant, and only after a deployment has created the namespace:

```
kubectl apply -f "${AMP_DIST}/deployments/values/otel-collector-rest-api.yaml"
```

Verify

```
kubectl get apigateway api-platform-default-default -n ${DATA_PLANE_NS}

# STATUS should show "Programmed"



kubectl get jobs -n ${DATA_PLANE_NS} | grep api-platform-default-default-bootstrap

# STATUS should show "Complete"
```

### Wire the Remaining Public Endpoints[​](#wire-the-remaining-public-endpoints "Direct link to Wire the Remaining Public Endpoints")

Apply these after all Phase 2 charts are installed.

**External AI gateway endpoint.** Register the `cp.` hostname and show it in the console:

* Helm 4
* Helm 3

```
helm upgrade --server-side=false amp oci://${HELM_CHART_REGISTRY}/wso2-agent-manager \

  --version ${VERSION} \

  --namespace ${AMP_NS} \

  --reuse-values \

  --set "agentManagerService.ocIngress.gatewayMgmt.hostnames={${CP_GW_PUBLIC_HOST}}" \

  --set console.config.gatewayControlPlaneUrl="https://${CP_GW_PUBLIC_HOST}"
```

```
helm upgrade amp oci://${HELM_CHART_REGISTRY}/wso2-agent-manager \

  --version ${VERSION} \

  --namespace ${AMP_NS} \

  --reuse-values \

  --set "agentManagerService.ocIngress.gatewayMgmt.hostnames={${CP_GW_PUBLIC_HOST}}" \

  --set console.config.gatewayControlPlaneUrl="https://${CP_GW_PUBLIC_HOST}"
```

**Deployed-agent endpoints.** Point the default environment's gateway at the agents domain, and route the workload publisher's token calls in-cluster:

* Helm 4
* Helm 3

```
helm upgrade --server-side=false amp-platform-resources oci://${HELM_CHART_REGISTRY}/wso2-amp-platform-resources-extension \

  --version ${VERSION} \

  --namespace ${DEFAULT_NS} \

  --reuse-values \

  --set global.oauth.tokenUrl="${THUNDER_INTERNAL_URL}/oauth2/token" \

  --set environment.gateway.http.host="${AGENTS_DOMAIN}" \

  --set environment.gateway.http.port=80 \

  --set environment.gateway.https.host="${AGENTS_DOMAIN}" \

  --set environment.gateway.https.port=443
```

```
helm upgrade amp-platform-resources oci://${HELM_CHART_REGISTRY}/wso2-amp-platform-resources-extension \

  --version ${VERSION} \

  --namespace ${DEFAULT_NS} \

  --reuse-values \

  --set global.oauth.tokenUrl="${THUNDER_INTERNAL_URL}/oauth2/token" \

  --set environment.gateway.http.host="${AGENTS_DOMAIN}" \

  --set environment.gateway.http.port=80 \

  --set environment.gateway.https.host="${AGENTS_DOMAIN}" \

  --set environment.gateway.https.port=443
```

Set both listeners, on their own ports

The environment's gateway binding replaces the data plane's entirely. Set both `http` and `https`, and give each its own port: `80` for `http`, `443` for `https`. A `443` on `http` makes the console publish `http://<host>:443`, which browsers block as mixed content.

**Later environments.** The override above covers only the default environment. Record the same values in Agent Manager's config so that environments created with [`add-environment.sh`](/agent-platform/docs/v2026.11.0/guides/environment-management/.md) resolve the same way:

* Helm 4
* Helm 3

```
helm upgrade --server-side=false amp oci://${HELM_CHART_REGISTRY}/wso2-agent-manager \

  --version ${VERSION} \

  --namespace ${AMP_NS} \

  --reuse-values \

  --set agentManagerService.config.agentsBaseDomain="${AGENTS_DOMAIN}" \

  --set-string agentManagerService.config.agentsHttpPort=80 \

  --set-string agentManagerService.config.agentsHttpsPort=443 \

  --set agentManagerService.config.idpHostBaseDomain="${BASE_DOMAIN}" \

  --set agentManagerService.config.tlsEnabled=true \

  --set console.config.idpHostBaseDomain="${BASE_DOMAIN}" \

  --set-string console.config.tlsEnabled=true
```

```
helm upgrade amp oci://${HELM_CHART_REGISTRY}/wso2-agent-manager \

  --version ${VERSION} \

  --namespace ${AMP_NS} \

  --reuse-values \

  --set agentManagerService.config.agentsBaseDomain="${AGENTS_DOMAIN}" \

  --set-string agentManagerService.config.agentsHttpPort=80 \

  --set-string agentManagerService.config.agentsHttpsPort=443 \

  --set agentManagerService.config.idpHostBaseDomain="${BASE_DOMAIN}" \

  --set agentManagerService.config.tlsEnabled=true \

  --set console.config.idpHostBaseDomain="${BASE_DOMAIN}" \

  --set-string console.config.tlsEnabled=true
```

Console-generated commands

The Console's generated command for later environments omits the platform Thunder issuer and the CA-trust mode, so on a TLS install the new environment's Thunder trusts the wrong issuer or stops with `Platform Thunder CA cert is not available`. Add these variables to the generated command before you run it:

* `PLATFORM_THUNDER_ISSUER="${THUNDER_PUBLIC_URL}"`
* `SKIP_CA_BUNDLE_TRUST=true` when your certificates come from a public CA such as Let's Encrypt, or `PLATFORM_THUNDER_CA_PEM="$(cat <ca-file>.pem)"` when they come from a private CA.

If the Thunder step fails, the script falls back to the platform Thunder and still reports that setup is complete. Check its output for a warning before you continue. The command is safe to run again with the missing variables.

Publishing added environments' gateways

Each added environment gets its own API Platform Gateway at `<env>-<org>.<gatewayBaseDomain>`. It stays in-cluster unless you give it a public hostname. To publish it, run the command above with these extra settings in place of the last one. Then add `*.otel.${BASE_DOMAIN}` DNS and certificate coverage, and keep an explicit record for `otel.${BASE_DOMAIN}`, because a wildcard makes that name an empty non-terminal ([RFC 4592](https://www.rfc-editor.org/rfc/rfc4592)).

* Helm 4
* Helm 3

```
helm upgrade --server-side=false amp oci://${HELM_CHART_REGISTRY}/wso2-agent-manager \

  --version ${VERSION} \

  --namespace ${AMP_NS} \

  --reuse-values \

  --set agentManagerService.config.agentsBaseDomain="${AGENTS_DOMAIN}" \

  --set-string agentManagerService.config.agentsHttpPort=80 \

  --set-string agentManagerService.config.agentsHttpsPort=443 \

  --set agentManagerService.config.gatewayBaseDomain="otel.${BASE_DOMAIN}" \

  --set agentManagerService.config.gatewayVhostScheme=https \

  --set agentManagerService.config.gatewayVhostPort=443 \

  --set agentManagerService.config.idpHostBaseDomain="${BASE_DOMAIN}" \

  --set agentManagerService.config.tlsEnabled=true \

  --set console.config.idpHostBaseDomain="${BASE_DOMAIN}" \

  --set-string console.config.tlsEnabled=true
```

```
helm upgrade amp oci://${HELM_CHART_REGISTRY}/wso2-agent-manager \

  --version ${VERSION} \

  --namespace ${AMP_NS} \

  --reuse-values \

  --set agentManagerService.config.agentsBaseDomain="${AGENTS_DOMAIN}" \

  --set-string agentManagerService.config.agentsHttpPort=80 \

  --set-string agentManagerService.config.agentsHttpsPort=443 \

  --set agentManagerService.config.gatewayBaseDomain="otel.${BASE_DOMAIN}" \

  --set agentManagerService.config.gatewayVhostScheme=https \

  --set agentManagerService.config.gatewayVhostPort=443 \

  --set agentManagerService.config.idpHostBaseDomain="${BASE_DOMAIN}" \

  --set agentManagerService.config.tlsEnabled=true \

  --set console.config.idpHostBaseDomain="${BASE_DOMAIN}" \

  --set-string console.config.tlsEnabled=true
```

### Prepare Storage for the Environment's Thunder[​](#prepare-storage-for-the-environments-thunder "Direct link to Prepare Storage for the Environment's Thunder")

By default, each Environment's Thunder keeps its data in SQLite on a 1Gi volume, uses an in-memory cache, and runs one replica. That setup is fine for evaluation, but the data lives on one volume behind one pod. For production, point it at your own PostgreSQL, and add Redis if you want more than one replica. The next step reads these settings from environment variables, so set them in the same shell before you run it. Skip this section to keep the defaults.

| Setting                                                     | Meaning                                                                                                                                                        | Default                           |
| ----------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------- |
| `THUNDER_DB_TYPE`                                           | `sqlite` or `postgres`                                                                                                                                         | `sqlite`                          |
| `THUNDER_DB_HOST`, `THUNDER_DB_USER`, `THUNDER_DB_PASSWORD` | Database connection details. Required with `postgres`                                                                                                          | None                              |
| `THUNDER_DB_NAME_PREFIX`                                    | Prefix of the four database names. Required with `postgres`                                                                                                    | None                              |
| `THUNDER_DB_PORT`, `THUNDER_DB_SSLMODE`                     | Database port and SSL mode                                                                                                                                     | `5432`, `require`                 |
| `THUNDER_CACHE_TYPE`                                        | `inmemory` or `redis`                                                                                                                                          | `inmemory`                        |
| `THUNDER_REDIS_ADDRESS`                                     | Redis `host:port`. Required with `redis`                                                                                                                       | None                              |
| `THUNDER_REDIS_USERNAME`, `THUNDER_REDIS_PASSWORD`          | Redis credentials, if your Redis needs them                                                                                                                    | None                              |
| `THUNDER_REDIS_KEY_PREFIX`                                  | Prefix for every Redis key                                                                                                                                     | The Thunder release name plus `:` |
| `THUNDER_REPLICAS`                                          | Number of Thunder pods                                                                                                                                         | `1`                               |
| `THUNDER_STATE_ACCESS_MODE`                                 | Access mode for the shared certificate and secret volumes. Use `ReadWriteMany` only when replicas may run on different nodes and the storage class supports it | Chart default (`ReadWriteOnce`)   |
| `THUNDER_FORCE_STORAGE_CHANGE`                              | Bypass the database-change guard after you have migrated the data yourself                                                                                     | `false`                           |
| `PERSISTENCE_SIZE`, `STORAGE_CLASS`                         | Size and storage class of the SQLite database volume                                                                                                           | `1Gi`, cluster default            |

The script checks these settings before it creates anything and stops with a message if they do not fit together, for example if you set database details without `THUNDER_DB_TYPE=postgres`.

* External PostgreSQL
* In-pod SQLite (default)

1. Create a role and four databases for this Environment on your PostgreSQL server. Start every database name with a prefix that is unique to the Environment, so that two Environments on one server never share data:

   ```
   CREATE ROLE thunder_default LOGIN PASSWORD '...';

   GRANT thunder_default TO CURRENT_USER;  -- required on managed PostgreSQL; harmless elsewhere

   CREATE DATABASE th_default_configdb OWNER thunder_default;

   CREATE DATABASE th_default_entitydb OWNER thunder_default;

   CREATE DATABASE th_default_runtime_transient OWNER thunder_default;

   CREATE DATABASE th_default_runtime_persistent OWNER thunder_default;
   ```

   The role must own the databases, or hold `SELECT`, `INSERT`, `UPDATE`, and `DELETE` on every table in them. The `GRANT` line matters on managed services such as RDS, Cloud SQL, and Azure Database; [Step 4](#step-4-install-thunder-extension-identity-provider) explains why. The prefix can use up to 41 lowercase letters, digits, and underscores, and must start with a letter.

2. Load Thunder's schema into each database. The script does not create tables, and an empty database makes the install fail. Take the schema from the same ThunderID version the script installs:

   ```
   THUNDER_VERSION="$(grep -m1 -o 'CHART_VERSION:-[0-9][0-9.]*' "${AMP_DIST}/deployments/scripts/add-environment-thunder.sh" | cut -d- -f2)"

   echo "${THUNDER_VERSION}"

   docker create --name thunder-schema "ghcr.io/thunder-id/thunderid:${THUNDER_VERSION}"

   for db in configdb entitydb runtime_transient runtime_persistent; do

     docker cp "thunder-schema:/opt/thunderid/dbscripts/${db}/postgres.sql" "./thunder-${db}.sql"

     psql "postgresql://thunder_default@your-db.example.com:5432/th_default_${db}" -v ON_ERROR_STOP=1 -f "./thunder-${db}.sql"

   done

   docker rm thunder-schema
   ```

   Expect 20 tables in `th_default_configdb`, 5 in `th_default_entitydb`, 13 in `th_default_runtime_transient`, and 7 in `th_default_runtime_persistent`.

   If the database is on a private network, run `psql` from a pod in the cluster as shown in [Step 4](#step-4-install-thunder-extension-identity-provider), with `PGUSER=thunder_default` and the `th_default_` database names.

3. Set the database variables in the shell you will run the next step from:

   ```
   export THUNDER_DB_TYPE=postgres

   export THUNDER_DB_HOST=your-db.example.com

   export THUNDER_DB_USER=thunder_default

   export THUNDER_DB_PASSWORD='<password>'

   export THUNDER_DB_NAME_PREFIX=th_default_
   ```

   `THUNDER_DB_SSLMODE` accepts `disable`, `allow`, `prefer`, `require` (the default), `verify-ca`, and `verify-full`. The chart has no setting for a database CA certificate. Use `disable` only for a database inside the cluster.

   The script stores the password in a Secret and gives it to the chart by reference, so it never appears in the Helm release values.

There is nothing to prepare. The script creates a 1Gi volume for the database. Set `PERSISTENCE_SIZE` or `STORAGE_CLASS` to change its size or storage class. SQLite cannot be shared across pods, so an Environment on SQLite always runs one replica.

#### Add Redis for the cache (optional)[​](#add-redis-for-the-cache-optional "Direct link to Add Redis for the cache (optional)")

The in-memory cache is local to each pod, so it only suits one replica. Use Redis when you run more than one Thunder replica. Any Redis that the cluster can reach works, whether a managed service or one you run yourself. Thunder only needs its address:

```
export THUNDER_CACHE_TYPE=redis

export THUNDER_REDIS_ADDRESS=your-redis.example.com:6379

export THUNDER_REDIS_PASSWORD='<password>'  # only if your Redis requires one
```

Add `THUNDER_REDIS_USERNAME` if your Redis uses ACL users. Every key starts with `THUNDER_REDIS_KEY_PREFIX`, which defaults to the Thunder release name followed by `:` (for example `amp-thunder-default-default:`), so several Environments can share one Redis. The chart has no TLS setting for Redis, so keep it on a private network.

#### Run more than one replica (optional)[​](#run-more-than-one-replica-optional "Direct link to Run more than one replica (optional)")

```
export THUNDER_REPLICAS=2
```

More than one replica needs external PostgreSQL and Redis. The script refuses any other combination before it creates anything. All replicas mount the same certificate and secret volumes, which are `ReadWriteOnce` by default. If the replicas can land on different nodes, also set `THUNDER_STATE_ACCESS_MODE=ReadWriteMany`, which needs a storage class that supports it. The volumes are created once, so changing this setting later has no effect.

Choose the database before the first run

Thunder seeds its database once, at first install, and `helm upgrade` never repeats it. The script therefore refuses to point an installed Environment at a different database, and refuses to retry in place a first install that failed before its setup job completed. If the first run fails that way (the script's message says so), remove the Environment's Thunder with `bash "${AMP_DIST}/deployments/scripts/remove-environment-thunder.sh"` (use the same `ENV_NAME`, `AMP_API_URL`, and `IDP_*` values as the provisioning command), fix the cause, and run the script again. If the first install failed after the setup job had finished, the database is already seeded and running the script again is accepted.

Removing an Environment does not drop its PostgreSQL databases. To add the Environment again, drop and recreate its databases (and reload the schema), or use a new `THUNDER_DB_NAME_PREFIX`.

Running the script again applies exactly the settings you pass, and a setting you leave out goes back to its default. Pass the same values every time, or the Environment drops, for example, from PostgreSQL with Redis and three replicas to the in-memory cache and one replica. When a run would change the cache type or the replica count of an installed Environment, the script prints the change before it applies it. It does not refuse, because scaling or switching the cache loses no data.

Environments you add later from the Console use the defaults. To give one its own PostgreSQL or Redis, add the same variables in front of `bash` in the command the Console prints, as shown in [Manage Environments](/agent-platform/docs/v2026.11.0/guides/environment-management/.md#thunder-storage).

### Provision Thunder Identity Provider for the Default Environment[​](#provision-thunder-identity-provider-for-the-default-environment "Direct link to Provision Thunder Identity Provider for the Default Environment")

Each environment needs its own Thunder instance. It issues each agent its OAuth2 credential (AgentID). Run this once for the default environment after Agent Manager is reachable at `API_PUBLIC_URL`. Without it, agents are created but never get an AgentID, and Agent Manager keeps retrying in the background.

If you prepared PostgreSQL or Redis above, keep those variables set in this shell. The command picks them up. Without them, the script uses SQLite, an in-memory cache, and one replica.

Choose the tab that matches your [Step 3](#step-3-setup-tls-issuer) issuer:

Pass the platform client secret

`IDP_CLIENT_SECRET` must be `${AMP_API_CLIENT_SECRET}`. Its default is the shipped secret, so the script fails with `Could not obtain an access token` on this install. If it fails, rerun it once the credentials are correct.

* Let's Encrypt (DNS-01)
* Corporate CA

Platform Thunder's certificate is publicly trusted, so no custom CA is needed:

```
ENV_NAME=default \

DISPLAY_NAME="Default" \

ORG_NAME=default \

THUNDER_HANDLE=default-idp \

WAIT_TIMEOUT=300s \

AMP_API_URL="${API_PUBLIC_URL}/api/v1" \

IDP_TOKEN_URL="${THUNDER_PUBLIC_URL}/oauth2/token" \

IDP_CLIENT_ID=amp-api-client \

IDP_CLIENT_SECRET="${AMP_API_CLIENT_SECRET}" \

PLATFORM_THUNDER_ISSUER="${THUNDER_PUBLIC_URL}" \

PLATFORM_THUNDER_JWKS_URL="${THUNDER_PUBLIC_URL}/oauth2/jwks" \

THUNDER_HOST_BASE_DOMAIN="${BASE_DOMAIN}" \

TLS_ENABLED=true \

SKIP_CA_BUNDLE_TRUST=true \

bash "${AMP_DIST}/deployments/scripts/add-environment-thunder.sh"
```

`SKIP_CA_BUNDLE_TRUST` does not apply to a corporate CA. Extract the CA that the running certificate presents:

```
PLATFORM_THUNDER_CA_PEM="$(echo | openssl s_client -connect "${THUNDER_PUBLIC_HOST}:443" \

  -servername "${THUNDER_PUBLIC_HOST}" -showcerts 2>/dev/null \

  | awk '/BEGIN CERTIFICATE/,/END CERTIFICATE/{print}')"
```

The script's own `curl` calls run on your machine, so trust the CA locally too. Keep your system bundle in the file, because the script also downloads a public CA bundle:

```
CURL_CA_BUNDLE="$(mktemp)"

if [ -f /etc/ssl/cert.pem ]; then

  SYSTEM_CA_BUNDLE=/etc/ssl/cert.pem                        # macOS

elif [ -f /etc/ssl/certs/ca-certificates.crt ]; then

  SYSTEM_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt       # Debian/Ubuntu

else

  echo "Could not find a system CA bundle. Set SYSTEM_CA_BUNDLE to yours." >&2

  exit 1

fi

cat "${SYSTEM_CA_BUNDLE}" > "${CURL_CA_BUNDLE}"

echo "${PLATFORM_THUNDER_CA_PEM}" >> "${CURL_CA_BUNDLE}"

export CURL_CA_BUNDLE
```

```
ENV_NAME=default \

DISPLAY_NAME="Default" \

ORG_NAME=default \

THUNDER_HANDLE=default-idp \

WAIT_TIMEOUT=300s \

AMP_API_URL="${API_PUBLIC_URL}/api/v1" \

IDP_TOKEN_URL="${THUNDER_PUBLIC_URL}/oauth2/token" \

IDP_CLIENT_ID=amp-api-client \

IDP_CLIENT_SECRET="${AMP_API_CLIENT_SECRET}" \

PLATFORM_THUNDER_ISSUER="${THUNDER_PUBLIC_URL}" \

PLATFORM_THUNDER_JWKS_URL="${THUNDER_PUBLIC_URL}/oauth2/jwks" \

PLATFORM_THUNDER_CA_PEM="${PLATFORM_THUNDER_CA_PEM}" \

THUNDER_HOST_BASE_DOMAIN="${BASE_DOMAIN}" \

TLS_ENABLED=true \

bash "${AMP_DIST}/deployments/scripts/add-environment-thunder.sh"
```

The script prints the issuer as `https://<handle>.${BASE_DOMAIN}`. `THUNDER_HANDLE` is optional (3 to 63 characters). Omit it to have Agent Manager generate one.

Verify

```
kubectl get pods -n amp-thunder-default-default

kubectl get secret amp-thunder-default-default-admin-credentials \

  -n amp-thunder-default-default -o jsonpath='{.data.password}' | base64 -d
```

Verify the storage you chose

The script prints a `Storage:` line at the start and again at the end. With PostgreSQL it reads `db=postgres`, and the Environment's Thunder has no database volume:

```
kubectl get pvc -n amp-thunder-default-default

# With external PostgreSQL there is no "...-database-pvc", only the certs and secrets volumes



psql "postgresql://thunder_default@your-db.example.com:5432/th_default_configdb" \

  -tAc 'select count(*) from "INBOUND_CLIENT"'

# 1 or more: Thunder seeded its own database
```

The script reuses the system-client secret and admin password on re-runs. It never rotates them.

### Point the Gateway at the Environment's Thunder[​](#point-the-gateway-at-the-environments-thunder "Direct link to Point the Gateway at the Environment's Thunder")

Register Thunder as a gateway key manager and as an identity provider in the console:

* Helm 4
* Helm 3

```
export ENV_THUNDER_RELEASE="amp-thunder-default-default"

export ENV_THUNDER_ISSUER="<the Issuer URL printed by add-environment-thunder.sh above>"

export ENV_THUNDER_JWKS="http://${ENV_THUNDER_RELEASE}-service.${ENV_THUNDER_RELEASE}.svc.cluster.local:8090/oauth2/jwks"



helm upgrade --server-side=false api-platform-default-default \

  oci://${HELM_CHART_REGISTRY}/wso2-amp-api-platform-gateway-extension \

  --version ${VERSION} \

  --namespace ${DATA_PLANE_NS} \

  --reuse-values \

  --set "apiGateway.config.policyConfigurations.jwtauth_v1.keymanagers[0].name=agent-manager-service" \

  --set "apiGateway.config.policyConfigurations.jwtauth_v1.keymanagers[0].issuer=agent-manager-service" \

  --set "apiGateway.config.policyConfigurations.jwtauth_v1.keymanagers[0].jwks.remote.uri=http://amp-api.${AMP_NS}.svc.cluster.local:9000/auth/external/jwks.json" \

  --set "apiGateway.config.policyConfigurations.jwtauth_v1.keymanagers[0].jwks.remote.skipTlsVerify=true" \

  --set "apiGateway.config.policyConfigurations.jwtauth_v1.keymanagers[1].name=ThunderKeyManager" \

  --set "apiGateway.config.policyConfigurations.jwtauth_v1.keymanagers[1].issuer=${ENV_THUNDER_ISSUER}" \

  --set "apiGateway.config.policyConfigurations.jwtauth_v1.keymanagers[1].jwks.remote.uri=${ENV_THUNDER_JWKS}" \

  --set "apiGateway.config.policyConfigurations.jwtauth_v1.keymanagers[1].jwks.remote.skipTlsVerify=false" \

  --set "bootstrap.identityProviders[0].name=ThunderKeyManager" \

  --set "bootstrap.identityProviders[0].issuer=${ENV_THUNDER_ISSUER}" \

  --set "bootstrap.identityProviders[0].jwksUri=${ENV_THUNDER_JWKS}" \

  --set "bootstrap.identityProviders[0].skipTlsVerify=false" \

  --timeout 900s
```

```
export ENV_THUNDER_RELEASE="amp-thunder-default-default"

export ENV_THUNDER_ISSUER="<the Issuer URL printed by add-environment-thunder.sh above>"

export ENV_THUNDER_JWKS="http://${ENV_THUNDER_RELEASE}-service.${ENV_THUNDER_RELEASE}.svc.cluster.local:8090/oauth2/jwks"



helm upgrade api-platform-default-default \

  oci://${HELM_CHART_REGISTRY}/wso2-amp-api-platform-gateway-extension \

  --version ${VERSION} \

  --namespace ${DATA_PLANE_NS} \

  --reuse-values \

  --set "apiGateway.config.policyConfigurations.jwtauth_v1.keymanagers[0].name=agent-manager-service" \

  --set "apiGateway.config.policyConfigurations.jwtauth_v1.keymanagers[0].issuer=agent-manager-service" \

  --set "apiGateway.config.policyConfigurations.jwtauth_v1.keymanagers[0].jwks.remote.uri=http://amp-api.${AMP_NS}.svc.cluster.local:9000/auth/external/jwks.json" \

  --set "apiGateway.config.policyConfigurations.jwtauth_v1.keymanagers[0].jwks.remote.skipTlsVerify=true" \

  --set "apiGateway.config.policyConfigurations.jwtauth_v1.keymanagers[1].name=ThunderKeyManager" \

  --set "apiGateway.config.policyConfigurations.jwtauth_v1.keymanagers[1].issuer=${ENV_THUNDER_ISSUER}" \

  --set "apiGateway.config.policyConfigurations.jwtauth_v1.keymanagers[1].jwks.remote.uri=${ENV_THUNDER_JWKS}" \

  --set "apiGateway.config.policyConfigurations.jwtauth_v1.keymanagers[1].jwks.remote.skipTlsVerify=false" \

  --set "bootstrap.identityProviders[0].name=ThunderKeyManager" \

  --set "bootstrap.identityProviders[0].issuer=${ENV_THUNDER_ISSUER}" \

  --set "bootstrap.identityProviders[0].jwksUri=${ENV_THUNDER_JWKS}" \

  --set "bootstrap.identityProviders[0].skipTlsVerify=false" \

  --timeout 900s
```

Set both key managers. `--set` on an indexed array replaces the whole list, so re-stating `keymanagers[0]` keeps the internal entry that API-key authentication uses.

Without this, agents reject OAuth tokens

The gateway then has only its internal key manager. API keys still work, so the platform looks complete, but no agent endpoint validates a Thunder-issued token.

The controller restarts

The gateway controller's volume is `ReadWriteOnce`. On a multi-node cluster the new pod can block with `Multi-Attach error`. Delete the old controller pod to release the volume:

```
kubectl get pods -n ${DATA_PLANE_NS} | grep gateway-controller

kubectl delete pod <old-gateway-controller-pod> -n ${DATA_PLANE_NS}
```

***

## Verify and Access the Platform[​](#verify-and-access-the-platform "Direct link to Verify and Access the Platform")

```
kubectl get pods -n openchoreo-control-plane

kubectl get pods -n openchoreo-data-plane

kubectl get pods -n openchoreo-workflow-plane

kubectl get pods -n openchoreo-observability-plane

kubectl get pods -n wso2-amp

kubectl get pods -n amp-thunder

helm list -A | grep -E 'openchoreo|amp|gateway'
```

Everything routes through the three plane gateways over TLS:

| Service               | URL                              |
| --------------------- | -------------------------------- |
| Console               | `https://${CONSOLE_PUBLIC_HOST}` |
| API                   | `https://${API_PUBLIC_HOST}`     |
| Thunder               | `https://${THUNDER_PUBLIC_HOST}` |
| Observer              | `https://${OBS_API_PUBLIC_HOST}` |
| Gateway control plane | `https://${CP_GW_PUBLIC_HOST}`   |

Log in as `admin` with the password from `amp-admin-credentials`, then change it in the profile menu. Deployed agents' invoke URLs are published under `https://<org>-<project>.${AGENTS_DOMAIN}`.

For debugging, port-forward any ClusterIP service, for example `kubectl port-forward -n wso2-amp svc/amp-console 3000:3000`.

***

## Cloud Provider Notes[​](#cloud-provider-notes "Direct link to Cloud Provider Notes")

Pick your platform. The **Generic** tab applies to any cluster.

* Generic
* EKS
* GKE
* AKS

- LoadBalancers may return an IP address or a hostname. Resolve whichever you get before creating DNS records.
- Allow HTTP and HTTPS through your cloud firewall or security groups to the plane gateway LoadBalancers.
- Confirm the default StorageClass provisions volumes. See [Cluster Requirements](#cluster-requirements).

* LoadBalancers return a **hostname** instead of an IP. Resolve it with `dig`.
* The default StorageClass must be CSI-backed. Install the EBS CSI driver and make a `gp3` class the default. Clusters created at 1.30 or later ship no default class, and the stock `gp2` class does not provision volumes without the EBS CSI driver.
* Cilium can replace the VPC CNI and kube-proxy, which enforces `NetworkPolicy` and gives pods VPC addresses in ENI IPAM mode. If you keep the VPC CNI, enable its network policy support.
* cert-manager, External Secrets, and OpenBao can use IAM Roles for Service Accounts instead of static keys. The Route 53 solver in Step 3 and the KMS auto-unseal in Step 2 both use this approach.
* Amazon ECR cannot back build workflows. See [Set up a build registry on EKS](#set-up-a-build-registry-on-eks).
* Ensure security groups allow HTTP and HTTPS to the LoadBalancers.

Full GKE guidance is coming soon.

* LoadBalancers return IPs directly. No special handling is needed.
* Ensure firewall rules allow HTTP and HTTPS to the LoadBalancers.

Full AKS guidance is coming soon.

* LoadBalancers return IPs directly. No special handling is needed.
* Ensure Network Security Groups allow HTTP and HTTPS.

Red Hat OpenShift (including ARO)

* OpenShift's Security Context Constraints, built-in monitoring and DNS need extra settings at several steps — follow [Appendix: Red Hat OpenShift](#appendix-red-hat-openshift) alongside the main flow
* On ARO, LoadBalancers return public IPs directly and `managed-csi` is the default StorageClass

***

## Before You Go Live[​](#before-you-go-live "Direct link to Before You Go Live")

Work through the [production recommendations](/agent-platform/docs/v2026.11.0/guides/production-recommendations/.md) before you expose the platform. They cover the checklist, identity and access, gateway hardening, build limits, scaling, and cluster security.

## Appendix: Installing Without a Domain (nip.io)[​](#appendix-installing-without-a-domain-nipio "Direct link to Appendix: Installing Without a Domain (nip.io)")

For evaluation only. Browsers warn on every hostname, and the certificates are self-signed.

Differences from the main flow

**1. Self-signed issuer.** In Step 3, create a self-signed chain under the `openchoreo-ca` name:

```
kubectl apply -f - <<'EOF'

apiVersion: cert-manager.io/v1

kind: ClusterIssuer

metadata:

  name: selfsigned-bootstrap

spec:

  selfSigned: {}

---

apiVersion: cert-manager.io/v1

kind: Certificate

metadata:

  name: openchoreo-ca

  namespace: cert-manager

spec:

  isCA: true

  commonName: openchoreo-ca

  secretName: openchoreo-ca-secret

  privateKey:

    algorithm: ECDSA

    size: 256

  issuerRef:

    name: selfsigned-bootstrap

    kind: ClusterIssuer

---

apiVersion: cert-manager.io/v1

kind: ClusterIssuer

metadata:

  name: openchoreo-ca

spec:

  ca:

    secretName: openchoreo-ca-secret

EOF
```

The Control Plane install in Step 5 also needs `--set-string openchoreoApi.config.security.authentication.jwt.jwks.skip_tls_verify=true`, and the Observability install in Step 8 needs `--set-string security.oidc.jwksUrlTlsInsecureSkipVerify=true`. Both are needed because Thunder's JWKS cannot be verified against an untrusted chain.

**2. Hostnames come from the first LoadBalancer.** Install the Control Plane once with placeholder hostnames. Then derive the base domain:

```
CP_LB_IP=$(kubectl get svc gateway-default -n openchoreo-control-plane \

  -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

export BASE_DOMAIN="${CP_LB_IP//./-}.nip.io"
```

Export the variables from [Plan Your Deployment](#plan-your-deployment) with this base domain, then rerun the Step 5 install as `helm upgrade`.

**3. Per-gateway domains.** All hostnames resolve to the control-plane IP, but `traces.` and `*.agents.` must reach the other gateways. Derive `OBS_API_PUBLIC_HOST` and `AGENTS_DOMAIN` from those gateways' LoadBalancer IPs, for example `traces.<ip-with-dashes>.nip.io`.

**4. Skip Step 10.** nip.io resolves automatically.

**5. Trust the self-signed CA.** `SKIP_CA_BUNDLE_TRUST` and the script's auto-detect do not match this CA. Use the Corporate CA steps in [Provision Thunder](#provision-thunder-identity-provider-for-the-default-environment) with `PLATFORM_THUNDER_CA_PEM` extracted the same way. The `openchoreo-ca-secret` name is the one the script can't find.

If you recreate the cluster, the LoadBalancer IP changes, so every hostname changes too. Reinstall Thunder with the new values.

***

## Appendix: Red Hat OpenShift[​](#appendix-red-hat-openshift "Direct link to Appendix: Red Hat OpenShift")

Agent Manager runs on Red Hat OpenShift, including managed offerings such as Azure Red Hat OpenShift (ARO). The flow is the same as the main guide; OpenShift needs extra settings at a few steps, mostly because of its stricter pod security defaults, its built-in platform components (monitoring, DNS, image registry), and the Kubernetes version a given OpenShift release ships.

This appendix was validated on **ARO, OpenShift 4.18 (Kubernetes 1.31)**.

OpenShift-specific setup and changes

Each change is tagged with where it applies:

| Tag                                  | Applies to                                                                                                 |
| ------------------------------------ | ---------------------------------------------------------------------------------------------------------- |
| **All OpenShift**                    | Every OpenShift cluster, any version or provider                                                           |
| **OpenShift 4.18 (Kubernetes 1.31)** | OpenShift releases that ship Kubernetes 1.31 or earlier, which is below the guide's 1.32 minimum           |
| **ARO**                              | Azure Red Hat OpenShift only                                                                               |
| **Private registry**                 | Clusters that push agent images to a registry requiring credentials (for example Azure Container Registry) |

Use `oc` or `kubectl` interchangeably for the commands below; the SCC commands need `oc`.

### How OpenShift security affects the install[​](#how-openshift-security-affects-the-install "Direct link to How OpenShift security affects the install")

**All OpenShift.** OpenShift admits pods through Security Context Constraints (SCCs). The default, `restricted-v2`, runs every pod as a random, namespace-assigned user ID and rejects pods that request a specific user ID, root, host access, or privilege. Most charts in this guide pin fixed non-root user IDs, and a few components need root or privilege, so you grant SCCs per namespace or service account **before** installing the charts that need them. Three built-in SCCs are used:

| SCC          | Allows                                                | Used for                                        |
| ------------ | ----------------------------------------------------- | ----------------------------------------------- |
| `nonroot-v2` | Any fixed user ID except root (0)                     | Most platform namespaces                        |
| `anyuid`     | Any user ID, including root                           | Two setup jobs that run as root                 |
| `privileged` | Root, privileged containers, host paths, host network | Agent build steps, the Fluent Bit log collector |

Images that expect to run as their own user (rather than OpenShift's random one) are pinned to that user through chart values in the steps below.

### Prerequisites[​](#prerequisites "Direct link to Prerequisites")

* **Kernel (All OpenShift).** RHCOS nodes run a 5.14 kernel, below the 6.3 needed for user-namespaced builds. Set `buildWorkflows.userNamespaces=false` on the Platform Resources chart (Phase 2, Step 4).

* **Kubernetes version (OpenShift 4.18 (Kubernetes 1.31)).** The OpenChoreo control-plane CRDs need a patch before installing; see Step 5 under Phase 1 changes below.

* **LoadBalancer check (All OpenShift).** The `nginx` image in the LoadBalancer test runs as root and is rejected by `restricted-v2`. Use the unprivileged image instead:

  ```
  kubectl create deploy lbtest --image=nginxinc/nginx-unprivileged:alpine --port=8080

  kubectl expose deploy lbtest --port=80 --target-port=8080 --type=LoadBalancer
  ```

* **NetworkPolicy (All OpenShift).** OpenShift's default network plugin (OVN-Kubernetes) enforces NetworkPolicy, so the platform's policies take effect. Its DNS service listens on a different port from upstream Kubernetes, which two of those policies need adjusting for (Phase 2, Steps 4 and 6).

* **Default StorageClass (ARO).** ARO ships `managed-csi` (Azure Disk) as the default class, which works as-is. Azure disks are zonal: a pod with persistent volumes can only run on a worker in its volume's zone, so keep at least one worker per zone.

* **Node count (ARO).** ARO requires 3 control-plane nodes and at least 3 workers, and does not schedule workloads on control-plane nodes. All platform components run on the workers.

### SCC grants[​](#scc-grants "Direct link to SCC grants")

**All OpenShift.** Grant these before the corresponding steps. Some namespaces do not exist until a later step creates them; `oc adm policy` refuses a namespace that does not exist yet, so those are granted with a ClusterRoleBinding instead, which takes effect when the namespace is created.

```
for ns in openchoreo-control-plane openchoreo-data-plane openchoreo-workflow-plane openchoreo-observability-plane amp-thunder wso2-amp; do

  kubectl create namespace "$ns" --dry-run=client -o yaml | kubectl apply -f -

  oc adm policy add-scc-to-group nonroot-v2 "system:serviceaccounts:$ns"

done



oc create clusterrolebinding agent-sandbox-system-nonroot-v2 \

  --clusterrole=system:openshift:scc:nonroot-v2 \

  --group=system:serviceaccounts:agent-sandbox-system



oc create clusterrolebinding amp-thunder-default-default-nonroot-v2 \

  --clusterrole=system:openshift:scc:nonroot-v2 \

  --group=system:serviceaccounts:amp-thunder-default-default



oc adm policy add-scc-to-user anyuid -z amp-hooks -n wso2-amp

oc adm policy add-scc-to-user anyuid -z api-platform-default-default-bootstrap-sa -n openchoreo-data-plane

oc adm policy add-scc-to-user privileged -z fluent-bit -n openchoreo-observability-plane



oc create clusterrolebinding workflows-default-workflow-sa-privileged \

  --clusterrole=system:openshift:scc:privileged \

  --serviceaccount=workflows-default:workflow-sa
```

| Namespace / service account                                       | SCC          | Why                                                                                                           |
| ----------------------------------------------------------------- | ------------ | ------------------------------------------------------------------------------------------------------------- |
| The four OpenChoreo plane namespaces, `amp-thunder`, `wso2-amp`   | `nonroot-v2` | Pods pin fixed non-root user IDs (1000, 1001, 10001, 65532 and others)                                        |
| `agent-sandbox-system`                                            | `nonroot-v2` | Sandbox controller runs as user ID 1000                                                                       |
| `amp-thunder-<org>-<env>`                                         | `nonroot-v2` | Per-environment Thunder runs as user ID 10001. Repeat for every environment you add                           |
| `wso2-amp/amp-hooks`                                              | `anyuid`     | The JWT key generation job runs as root                                                                       |
| `openchoreo-data-plane/api-platform-default-default-bootstrap-sa` | `anyuid`     | The gateway bootstrap job runs as root                                                                        |
| `openchoreo-observability-plane/fluent-bit`                       | `privileged` | Reads node log files through host paths                                                                       |
| `workflows-default/workflow-sa`                                   | `privileged` | Agent build steps run privileged podman containers. Repeat for `workflows-<env>` of every environment you add |

### Phase 1 changes[​](#phase-1-changes "Direct link to Phase 1 changes")

#### Step 2: OpenBao[​](#step-2-openbao "Direct link to Step 2: OpenBao")

**All OpenShift.** The OpenBao chart pins user ID 100 by default. Add `--set global.openshift=true` to the OpenBao `helm upgrade --install` command; the chart then leaves the user ID to OpenShift and needs no SCC grant.

#### Step 5: OpenChoreo Control Plane[​](#step-5-openchoreo-control-plane "Direct link to Step 5: OpenChoreo Control Plane")

**OpenShift 4.18 (Kubernetes 1.31).** The control-plane chart's CRDs fail to install with `CustomResourceDefinition ... is invalid ... undefined field 'namespace'`. Some of their validation rules refer to fields named `namespace` and `var`, which Kubernetes 1.31 and earlier treat as reserved words and only accept in the escaped form `__namespace__` / `__var__`. The escaped form is also valid on newer Kubernetes.

Apply patched CRDs yourself, then install the chart with `--skip-crds`:

```
helm pull oci://ghcr.io/openchoreo/helm-charts/openchoreo-control-plane --version 1.2.8 --untar -d oc-cp-chart



sed -i.bak -E 's/has\(self\.var\)/has(self.__var__)/g; s/self\.scope\.namespace\)/self.scope.__namespace__)/g' \

  oc-cp-chart/openchoreo-control-plane/crds/*.yaml

rm -f oc-cp-chart/openchoreo-control-plane/crds/*.bak



kubectl apply --server-side -f oc-cp-chart/openchoreo-control-plane/crds/
```

Then add `--skip-crds` to every control-plane `helm upgrade --install` in this guide. Helm never upgrades CRDs, so on a later control-plane chart upgrade repeat these commands with the new chart version.

#### Step 8: Observability Plane[​](#step-8-observability-plane "Direct link to Step 8: Observability Plane")

**All OpenShift.** Three changes to the observability modules:

1. **OpenSearch.** Its init container changes the data volume's owner as root, which `nonroot-v2` does not allow. The pod's `fsGroup` already makes the volume writable, so turn the init container off. Add to the logs module install:

   ```
   --set openSearch.persistence.enableInitChown=false
   ```

2. **Fluent Bit.** OpenShift's container runtime writes log files readable only by root, with an SELinux label unprivileged containers cannot read. The `privileged` SCC grant only permits privilege; the container must also request it. Add to the command that enables Fluent Bit:

   ```
   --set fluent-bit.securityContext.runAsUser=0 \

   --set fluent-bit.securityContext.runAsNonRoot=false \

   --set fluent-bit.securityContext.privileged=true \

   --set fluent-bit.securityContext.allowPrivilegeEscalation=true
   ```

   Without this, every log query in the console stays empty while all pods look healthy.

3. **Metrics module.** OpenShift ships its own Prometheus Operator and kube-state-metrics, whose cluster roles use the same names as this chart's, so the install fails with `ClusterRole "kube-state-metrics" ... exists and cannot be imported into the current release`. The chart's operator would also manage OpenShift's own `Prometheus` and `Alertmanager` resources. Rename the chart's components, keep its operator out of OpenShift's monitoring namespaces, and keep Prometheus off a public LoadBalancer. Add to the metrics module install:

   ```
   --set kube-prometheus-stack.prometheusOperator.fullnameOverride=openchoreo-prometheus-operator \

   --set kube-prometheus-stack.kube-state-metrics.fullnameOverride=openchoreo-kube-state-metrics \

   --set "kube-prometheus-stack.prometheusOperator.denyNamespaces={openshift-monitoring,openshift-user-workload-monitoring}" \

   --set kube-prometheus-stack.prometheus.service.type=ClusterIP
   ```

   OpenShift owns the `monitoring.coreos.com` CRDs, so the chart's operator runs against OpenShift's versions; a warning such as `unknown field "spec.enableOTLPReceiver"` during install is expected and harmless.

### Phase 2 changes[​](#phase-2-changes "Direct link to Phase 2 changes")

#### Step 1: Gateway Operator[​](#step-1-gateway-operator "Direct link to Step 1: Gateway Operator")

**All OpenShift.** The gateway runtime image keeps its working directories owned by user ID 10001 and crash-loops (`Failed to bind to address unix:/var/run/api-platform/python-executor.sock ... Permission denied`) under OpenShift's random user ID. Pin it through the operator's gateway values. Add to the gateway operator install:

```
--set gateway.values.gateway.gatewayRuntime.deployment.podSecurityContext.runAsUser=10001 \

--set gateway.values.gateway.gatewayRuntime.deployment.podSecurityContext.runAsGroup=10001 \

--set gateway.values.gateway.gatewayRuntime.deployment.podSecurityContext.fsGroup=10001 \

--set gateway.values.gateway.gatewayRuntime.deployment.podSecurityContext.runAsNonRoot=true
```

Set these on the **initial** install. If the gateway extension (Step 7) is installed while the runtime is crash-looping, the operator records the gateway as permanently failed and does not retry on its own.

#### Step 2: Agent Manager[​](#step-2-agent-manager "Direct link to Step 2: Agent Manager")

**All OpenShift.** Two components need their user ID pinned. Add to the Agent Manager install:

```
--set agentManagerService.podSecurityContext.runAsUser=1000 \

--set agentManagerService.podSecurityContext.runAsNonRoot=true \

--set agentManagerService.podSecurityContext.fsGroup=1000 \

--set console.securityContext.runAsUser=1001 \

--set console.securityContext.runAsNonRoot=true \

--set console.securityContext.allowPrivilegeEscalation=false \

--set "console.securityContext.capabilities.drop={ALL}"
```

* The API pod's `wait-for-db` init container (used with the in-cluster database) runs an image that defaults to root and has no user ID of its own, so the pod-level user ID applies to it.
* The console's web server writes its runtime configuration into a directory owned by user ID 1001.

#### Step 4: Platform Resources[​](#step-4-platform-resources "Direct link to Step 4: Platform Resources")

**All OpenShift.** Add `--set buildWorkflows.userNamespaces=false` to the Platform Resources install (RHCOS kernel 5.14).

After installing, apply two patches the chart has no values for:

1. **Build checkout step.** It is the only non-privileged build step, so it runs under a random user ID with no home directory and git fails with `could not lock config file //.gitconfig`. Give it a writable `HOME`:

   ```
   kubectl patch clusterworkflowtemplate checkout-source --type=json \

     -p '[{"op":"add","path":"/spec/templates/0/container/env/-","value":{"name":"HOME","value":"/tmp"}}]'
   ```

2. **Agent DNS egress.** OpenShift's DNS (`openshift-dns`) listens on port **5353** behind a Service on port 53. NetworkPolicies match the pod's real port, and the agent egress policy only allows 53, so agents cannot resolve any hostname (`Temporary failure in name resolution`). Allow 5353 as well:

   ```
   kubectl patch componenttype.openchoreo.dev agent-api -n default --type=json \

     -p '[{"op":"add","path":"/spec/resources/0/template/spec/networkPolicy/egress/0/ports/-","value":{"port":5353,"protocol":"UDP"}},{"op":"add","path":"/spec/resources/0/template/spec/networkPolicy/egress/0/ports/-","value":{"port":5353,"protocol":"TCP"}}]'
   ```

   Agents deployed before this patch pick it up on their next deploy.

**Private registry.** Agent component types do not set `imagePullSecrets`, so agent pods cannot authenticate to a private registry themselves. On OpenShift, add the registry's credentials to the cluster-wide pull secret, which every node uses for every image:

```
oc get secret/pull-secret -n openshift-config -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d > cluster-pull-secret.json

jq -s '.[0] * .[1]' cluster-pull-secret.json registry-dockerconfig.json > cluster-pull-secret.merged.json

oc set data secret/pull-secret -n openshift-config --from-file=.dockerconfigjson=cluster-pull-secret.merged.json
```

`registry-dockerconfig.json` is your registry's `.dockerconfigjson` (for ACR, built from the registry's admin credentials). Keep every existing entry; on ARO the `arosvc.azurecr.io` entry must not be removed. OpenShift rolls the change out to all nodes without a reboot.

#### Step 6: Evaluation Extension[​](#step-6-evaluation-extension "Direct link to Step 6: Evaluation Extension")

**All OpenShift.** The evaluation job's egress policy expects DNS in `kube-system` on port 53. Point it at OpenShift's DNS through chart values, adding to the evaluation extension install:

```
--set networkPolicy.evaluationJob.dns.namespace=openshift-dns \

--set networkPolicy.evaluationJob.dns.podLabel.key=dns.operator.openshift.io/daemonset-dns \

--set networkPolicy.evaluationJob.dns.podLabel.value=default
```

The DNS port is fixed at 53 in the chart, so add 5353 right after installing (the chart creates `workflows-default` and the policy itself):

```
kubectl patch networkpolicy amp-evaluation-job-egress -n workflows-default --type=json \

  -p '[{"op":"add","path":"/spec/egress/0/ports/-","value":{"port":5353,"protocol":"UDP"}},{"op":"add","path":"/spec/egress/0/ports/-","value":{"port":5353,"protocol":"TCP"}}]'
```

#### After the first agent build[​](#after-the-first-agent-build "Direct link to After the first agent build")

**All OpenShift.** OpenShift adds an image-registry pull secret (`workflow-sa-dockercfg-...`) as the first entry in every service account's `secrets` list. Evaluation monitor workflows mount the service account's first secret as their API token, so they fail at start-up with `invalid configuration: no configuration has been provided`. Put the token Secret first:

```
DOCKERCFG=$(kubectl get sa workflow-sa -n workflows-default \

  -o jsonpath='{.secrets[?(@.name!="workflow-sa.service-account-token")].name}')

kubectl patch sa workflow-sa -n workflows-default --type=merge \

  -p "{\"secrets\":[{\"name\":\"workflow-sa.service-account-token\"},{\"name\":\"${DOCKERCFG}\"}]}"
```

Repeat for `workflows-<env>` of every environment you add.

#### Provision Thunder Identity Provider[​](#provision-thunder-identity-provider "Direct link to Provision Thunder Identity Provider")

**ARO.** Thunder's volumes are created in a single availability zone, and attaching them can take a few minutes on Azure. Pass `WAIT_TIMEOUT=600s` to `add-environment-thunder.sh` so the script does not stop waiting before Thunder is ready. Remember the `amp-thunder-<org>-<env>` SCC grant for every environment you add.

### After a chart upgrade[​](#after-a-chart-upgrade "Direct link to After a chart upgrade")

Some OpenShift changes are applied with `kubectl` on top of what Helm renders, and an upgrade of the owning chart reverts them. Re-apply after upgrading:

| Change                                                 | Owning chart             | Applies to                       |
| ------------------------------------------------------ | ------------------------ | -------------------------------- |
| Patched control-plane CRDs, install with `--skip-crds` | OpenChoreo Control Plane | OpenShift 4.18 (Kubernetes 1.31) |
| `checkout-source` `HOME=/tmp`                          | Platform Resources       | All OpenShift                    |
| `agent-api` DNS port 5353                              | Platform Resources       | All OpenShift                    |
| `amp-evaluation-job-egress` DNS port 5353              | Evaluation Extension     | All OpenShift                    |

The changes made through `--set` values persist across upgrades that use `--reuse-values`.

***

## Appendix: Rancher Desktop / k3s[​](#appendix-rancher-desktop--k3s "Direct link to Appendix: Rancher Desktop / k3s")

For evaluation only. Single-node k3s works, but it may not have enough resources for every observability module.

k3s ships Traefik on ports 80 and 443, which conflicts with kgateway. Remove it before [Step 1](#step-1-install-cluster-prerequisites):

```
helm uninstall traefik -n kube-system

helm uninstall traefik-crd -n kube-system
```

Traefik's CRD removal can also remove the Gateway API CRDs. Reapply them:

```
kubectl apply --server-side --force-conflicts \

  -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.1/experimental-install.yaml
```

***

## Cleanup[​](#cleanup "Direct link to Cleanup")

```
# 1. Delete plane registrations

kubectl delete clusterdataplane default -n default

kubectl delete clusterworkflowplane default -n default

kubectl delete clusterobservabilityplane default



# 2. Uninstall all Helm releases

helm uninstall amp -n wso2-amp

helm uninstall api-platform-default-default -n openchoreo-data-plane

helm uninstall amp-thunder-extension -n amp-thunder

helm uninstall amp-observability-traces -n openchoreo-observability-plane

helm uninstall amp-evaluation-extension -n openchoreo-workflow-plane

helm uninstall amp-platform-resources -n default

helm uninstall gateway-operator -n openchoreo-data-plane

helm uninstall openchoreo-observability-plane -n openchoreo-observability-plane

helm uninstall openchoreo-workflow-plane -n openchoreo-workflow-plane

helm uninstall openchoreo-data-plane -n openchoreo-data-plane

helm uninstall openchoreo-control-plane -n openchoreo-control-plane

helm uninstall openbao -n openbao

helm uninstall external-secrets -n external-secrets

helm uninstall cert-manager -n cert-manager



# 3. Delete namespaces

kubectl delete namespace wso2-amp amp-thunder \

  openchoreo-observability-plane openchoreo-workflow-plane \

  openchoreo-data-plane openchoreo-control-plane \

  openbao external-secrets cert-manager
```

If an Environment's Thunder uses your own PostgreSQL, uninstalling does not drop its databases. Drop `<prefix>configdb`, `<prefix>entitydb`, `<prefix>runtime_transient`, and `<prefix>runtime_persistent` yourself once you no longer need the data, where `<prefix>` is the `THUNDER_DB_NAME_PREFIX` you provisioned the Environment with.

***

## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting")

Environment's Thunder: "already installed with sqlite; this run asks for postgres"

The script refuses to point an installed Environment at a different database. Thunder seeds its database once, at first install, so a different database would be empty: no OAuth clients and no logins.

To move the Environment, remove its Thunder with `remove-environment-thunder.sh` and provision it again with the new settings. If you have already migrated the data yourself, set `THUNDER_FORCE_STORAGE_CHANGE=true`.

Related messages:

* `... is a failed first install`: the chart's setup job runs only on first install, and the script cannot confirm that it finished. If the job is still running (`kubectl get job <release>-setup -n <namespace>`), wait for it and run the script again. Otherwise run `remove-environment-thunder.sh`, fix the cause, and run the script again.
* `... runs with cache=redis, replicas=3; this run sets cache=inmemory, replicas=1`: a run applies exactly the settings it is given, and the ones you left out went back to their defaults. If that was not intended, run the script again with the same `THUNDER_CACHE_TYPE` and `THUNDER_REPLICAS` as the first run.
* `THUNDER_DB_HOST is set but THUNDER_DB_TYPE is 'sqlite'`: add `THUNDER_DB_TYPE=postgres`, or unset the database variables.

Environment's Thunder does not start on PostgreSQL

Check these in order:

* The schema is not loaded. The setup job fails and its pod is deleted, so the logs show nothing. [Step 4](#step-4-install-thunder-extension-identity-provider) shows the underlying error. Load the schema as described in [Prepare Storage](#prepare-storage-for-the-environments-thunder).

* The database names do not match your prefix. The script prints the four names it expects at the start of the run.

* The database is not reachable from the cluster, or the password is wrong. Read the Thunder pod's log:

  ```
  kubectl logs -n amp-thunder-<org>-<env> deploy/amp-thunder-<org>-<env>-deployment
  ```

With PostgreSQL or Redis selected, the script stops with `Thunder did not become ready in time` instead of printing the console details. Run `remove-environment-thunder.sh` for the Environment before you try again.

LoadBalancer has no external address

```
kubectl describe svc <service-name> -n <namespace>

kubectl get svc -A --field-selector spec.type=LoadBalancer
```

On EKS, confirm the AWS Load Balancer Controller is installed. On k3s, check whether Traefik holds the ports.

Certificate not issued

```
kubectl describe certificate <cert-name> -n <namespace>

kubectl get clusterissuers

kubectl get certificaterequests -n <namespace>
```

Plane registration fails

```
kubectl get clusterdataplane default -n default -o yaml

kubectl logs -n openchoreo-control-plane -l app.kubernetes.io/name=openchoreo-control-plane
```

API returns 401 for environment or gateway calls

The Control Plane's OIDC issuer does not match the `iss` claim in Thunder tokens. Compare them:

```
kubectl exec -n amp-thunder deploy/amp-thunder-extension-deployment -- \

  wget -qO- http://localhost:8090/.well-known/openid-configuration 2>/dev/null \

  | grep -o '"issuer":"[^"]*"'



kubectl get configmap openchoreo-api-config -n openchoreo-control-plane -o yaml | grep issuer
```

Both must match exactly. If they differ, set the Control Plane's `security.oidc.issuer` to Thunder's issuer.

Console shows "refused to connect" on login

```
dig +short thunder.${BASE_DOMAIN}

kubectl get httproute -n openchoreo-control-plane amp-thunder-extension

curl -s -o /dev/null -w '%{http_code}\n' https://thunder.${BASE_DOMAIN}/.well-known/openid-configuration
```

To change Thunder's public URL after install, uninstall, delete the PVC, and reinstall:

```
helm uninstall amp-thunder-extension -n amp-thunder

kubectl delete pvc -n amp-thunder --all
```

Evaluations fail with connection errors

The evaluation-job NetworkPolicy allows DNS egress only to `kube-system` with the `k8s-app=kube-dns` label. If your DNS runs elsewhere, such as NodeLocal DNSCache, every egress rule fails, because nothing resolves. Set `networkPolicy.evaluationJob.dns.namespace` and `podLabel` to match your DNS workload.

OpenSearch connectivity

```
kubectl get pods -n openchoreo-observability-plane -l app=opensearch

kubectl run -it --rm debug --image=curlimages/curl --restart=Never -- \

  curl -v http://opensearch.openchoreo-observability-plane.svc.cluster.local:9200
```

Build pods fail with MOUNT\_ATTR\_IDMAP or idmap errors

Build pods request a Linux user namespace (`hostUsers: false`). That needs idmapped-mount support on every mounted filesystem, and tmpfs gained it in Linux 6.3. Ubuntu 22.04 (kernel 5.15) and Amazon Linux 2 (kernel 5.10) do not have it. Errors look like this:

```
failed to create containerd container: snapshotter "overlayfs" doesn't support idmap mounts

OCI runtime create failed: ... failed to set MOUNT_ATTR_IDMAP ... invalid argument
```

Use one of these fixes:

1. **Preferred:** use build nodes with kernel 6.3 or newer.
2. Disable user namespaces with the Platform Resources chart value:

* Helm 4
* Helm 3

```
helm upgrade --server-side=false amp-platform-resources \

  oci://${HELM_CHART_REGISTRY}/wso2-amp-platform-resources-extension \

  --version ${VERSION} \

  --namespace ${DEFAULT_NS} \

  --reuse-values \

  --set buildWorkflows.userNamespaces=false
```

```
helm upgrade amp-platform-resources \

  oci://${HELM_CHART_REGISTRY}/wso2-amp-platform-resources-extension \

  --version ${VERSION} \

  --namespace ${DEFAULT_NS} \

  --reuse-values \

  --set buildWorkflows.userNamespaces=false
```

Changing containerd's `slow_chown` alone does not fix this. The build still fails on the tmpfs mount.

Agent deploys fail with "Secret does not exist" after an OpenBao restart

The guide's OpenBao dev-mode install keeps secrets in memory. Restarting `openbao-0` wipes them. Running agents keep working, but new deployments fail with `SecretSyncedError`. Re-save the agent's environment variables in the console. For production, use the persistent install from [Step 2](#step-2-set-up-the-secret-stores).

Build fails with a cgroup pids error (Rancher Desktop)

```
Error: OCI runtime error: crun: the requested cgroup controller `pids` is not available
```

Rancher Desktop's Lima VM does not delegate the `pids` controller to containers. Patch the three Podman-based templates to disable cgroup management. The patches require `python3` on your machine:

```
# gcp-buildpacks-build

kubectl get clusterworkflowtemplate gcp-buildpacks-build -o json | \

  python3 -c "

import json, sys

data = json.load(sys.stdin)

script = data['spec']['templates'][0]['container']['args'][0]

fix = '''set -e



# Fix: disable cgroup management for Podman (Rancher Desktop cgroup pids workaround)

cat > /tmp/containers.conf <<CCONF

[engine]

cgroup_manager = \"cgroupfs\"

events_logger = \"file\"

[containers]

pids_limit = 0

CCONF

export CONTAINERS_CONF=/tmp/containers.conf



'''

data['spec']['templates'][0]['container']['args'][0] = script.replace('set -e\n', fix, 1)

json.dump(data, sys.stdout)

" | kubectl apply -f -



# publish-image

kubectl get clusterworkflowtemplate publish-image -o json | \

  python3 -c "

import json, sys

data = json.load(sys.stdin)

script = data['spec']['templates'][0]['container']['args'][0]

fix = '''set -e



# Fix: disable cgroup management for Podman (Rancher Desktop cgroup pids workaround)

cat > /tmp/containers.conf <<CCONF

[engine]

cgroup_manager = \"cgroupfs\"

events_logger = \"file\"

[containers]

pids_limit = 0

CCONF

export CONTAINERS_CONF=/tmp/containers.conf



'''

data['spec']['templates'][0]['container']['args'][0] = script.replace('set -e\n', fix, 1)

json.dump(data, sys.stdout)

" | kubectl apply -f -



# amp-generate-workload

kubectl get clusterworkflowtemplate amp-generate-workload -o json | \

  python3 -c "

import json, sys

data = json.load(sys.stdin)

script = data['spec']['templates'][0]['container']['args'][0]

fix = '''# Fix: disable cgroup management for Podman (Rancher Desktop cgroup pids workaround)

cat > /tmp/containers.conf <<CCONF

[engine]

cgroup_manager = \"cgroupfs\"

events_logger = \"file\"

[containers]

pids_limit = 0

CCONF

export CONTAINERS_CONF=/tmp/containers.conf



'''

data['spec']['templates'][0]['container']['args'][0] = fix + script

json.dump(data, sys.stdout)

" | kubectl apply -f -
```

Re-trigger the build. These patches are in-cluster, so reinstalling `amp-platform-resources` overwrites them. This affects Rancher Desktop only. EKS, GKE, and AKS are not affected.

Agents cannot resolve hostnames on EKS (Try It returns 504)

On EKS with the VPC CNI's network policy support on, a deployed agent cannot reach DNS or its Environment's Thunder. Try It returns `504 upstream request timeout`, and the agent's log shows `Temporary failure in name resolution`.

The agent's egress policy allows DNS (port 53) and Thunder (port 8090) to any address, and allows ports 80 and 443 to the internet but not to private address ranges. The VPC CNI's policy agent applies that private-range exception to the DNS and Thunder rules as well. Cluster pods have private VPC addresses, so DNS and Thunder traffic is dropped. Other network plugins, such as Calico and Cilium, are not affected.

Allow DNS and Thunder explicitly in the agent's namespace. Agent namespaces start with `dp-`; list them with `kubectl get ns | grep '^dp-'`. Set the namespace and the Thunder namespace of the agent's Environment, `amp-thunder-<org>-<env>`:

```
export AGENT_NS="<agent-namespace>"

export ENV_THUNDER_NS="amp-thunder-<org>-<env>"

kubectl apply -n "${AGENT_NS}" -f - <<EOF

apiVersion: networking.k8s.io/v1

kind: NetworkPolicy

metadata:

  name: allow-agent-dns-and-thunder

spec:

  podSelector:

    matchExpressions:

    - key: agents.x-k8s.io/sandbox-template-ref-hash

      operator: Exists

  policyTypes:

  - Egress

  egress:

  - to:

    - namespaceSelector:

        matchLabels:

          kubernetes.io/metadata.name: kube-system

      podSelector:

        matchLabels:

          k8s-app: kube-dns

    ports:

    - port: 53

      protocol: UDP

    - port: 53

      protocol: TCP

  - to:

    - namespaceSelector:

        matchLabels:

          kubernetes.io/metadata.name: ${ENV_THUNDER_NS}

    ports:

    - port: 8090

      protocol: TCP

EOF
```

Naming the DNS pods and the Thunder namespace makes the policy agent allow those addresses ahead of the private-range exception. Agents recover within a few seconds, with no redeploy. Each project and Environment has its own agent namespace, so apply this to each new one.

Set up a build registry on EKS (Amazon ECR cannot back builds)

Amazon ECR cannot back build workflows. If you have no other registry, run [CNCF Distribution](https://distribution.github.io/distribution/) in the cluster behind an internal LoadBalancer. It creates repositories on push and needs no `registry-push-secret`. It has **no authentication**, so keep it internal: anything inside the VPC can push to it.

Issue a certificate for `registry.${BASE_DOMAIN}` from the issuer created in [Step 3](#step-3-setup-tls-issuer):

```
kubectl create namespace registry --dry-run=client -o yaml | kubectl apply -f -
```

```
kubectl apply -f - <<EOF

apiVersion: cert-manager.io/v1

kind: Certificate

metadata:

  name: registry-tls

  namespace: registry

spec:

  secretName: registry-tls

  issuerRef:

    name: openchoreo-ca

    kind: ClusterIssuer

  dnsNames:

    - "registry.${BASE_DOMAIN}"

EOF
```

```
kubectl wait --for=condition=Ready certificate/registry-tls -n registry --timeout=300s
```

Run Distribution with a 50 GiB volume and an internal LoadBalancer on port 443:

```
kubectl apply -f - <<'EOF'

apiVersion: v1

kind: PersistentVolumeClaim

metadata:

  name: registry-data

  namespace: registry

spec:

  accessModes: ["ReadWriteOnce"]

  resources:

    requests:

      storage: 50Gi

---

apiVersion: apps/v1

kind: Deployment

metadata:

  name: registry

  namespace: registry

spec:

  replicas: 1

  strategy:

    type: Recreate

  selector:

    matchLabels:

      app: registry

  template:

    metadata:

      labels:

        app: registry

    spec:

      containers:

        - name: registry

          image: registry:3.0.0

          ports:

            - containerPort: 5000

          env:

            - name: REGISTRY_HTTP_TLS_CERTIFICATE

              value: /certs/tls.crt

            - name: REGISTRY_HTTP_TLS_KEY

              value: /certs/tls.key

          volumeMounts:

            - name: data

              mountPath: /var/lib/registry

            - name: certs

              mountPath: /certs

              readOnly: true

      volumes:

        - name: data

          persistentVolumeClaim:

            claimName: registry-data

        - name: certs

          secret:

            secretName: registry-tls

---

apiVersion: v1

kind: Service

metadata:

  name: registry

  namespace: registry

  annotations:

    service.beta.kubernetes.io/aws-load-balancer-internal: "true"

spec:

  type: LoadBalancer

  selector:

    app: registry

  ports:

    - name: https

      port: 443

      targetPort: 5000

EOF
```

```
kubectl rollout status deploy/registry -n registry --timeout=300s
```

```
kubectl get svc registry -n registry -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'
```

The hostname starts with `internal-`, which confirms the LoadBalancer is not reachable from the internet. Create a CNAME record from `registry.${BASE_DOMAIN}` to it. The specific record takes precedence over the `*.${BASE_DOMAIN}` wildcard, but until the wildcard's cached answer expires (about 5 minutes), the name still resolves to the control-plane gateway.

Check the registry from inside the cluster, the way builds reach it. Expect `HTTP/2 200`:

```
kubectl run regtest --rm -i --restart=Never --image=curlimages/curl -- curl -sI https://registry.${BASE_DOMAIN}/v2/
```

In Phase 2, set `global.registry.endpoint` to `registry.${BASE_DOMAIN}` and `global.defaultResources.registry.tlsVerify=true`.