Skip to main content
Copy page
Version: STS 2026.11.0

On Your Environment

Install Agent Manager on an existing Kubernetes cluster with production settings: a real domain, trusted TLS, external databases, and sealed secret storage. Phase 1 installs OpenChoreo and Thunder. Phase 2 installs Agent Manager.

Evaluating?

Use the k3d guide instead.

Short-term support (STS) release

2026.11.0 is a short-term support (STS) release available for evaluation. Agent Manager's charts and images, including the API Platform gateway, are hosted on registry.wso2.com and require registry credentials. To obtain these credentials, download Agent Manager from the WSO2 website to receive them by email, or use the credentials provided with your WSO2 subscription. Other components, such as OpenChoreo and the Gateway Operator chart, are hosted on public registries such as ghcr.io. See About the STS Release for more information.

Isolation tiers

Agents run under the standard runc runtime. For stronger isolation, see gVisor and Kata Containers. Both need dedicated nodes.

Before You Start​

Cluster Requirements​

  • Kubernetes 1.32+ with 3 nodes, each with 4 CPU cores and 8 GB RAM
  • LoadBalancer support
  • A default StorageClass that provisions volumes. See Cluster Requirements.
  • Linux 6.3+ on build nodes. See Troubleshooting for older kernels.
  • A CNI that enforces NetworkPolicy (recommended)

On Red Hat OpenShift, including Azure Red Hat OpenShift, also follow Appendix: Red Hat OpenShift.

Tools​

ToolVersion
kubectlv1.32+
Helmv3.12+ or Helm 4
curl, dig, openssl, python3Any recent version

Confirm that LoadBalancers provision before you start:

kubectl create deploy lbtest --image=nginx && kubectl expose deploy lbtest --port=80 --type=LoadBalancer
kubectl wait --for=jsonpath='{.status.loadBalancer.ingress}' svc/lbtest --timeout=5m && echo "LoadBalancer OK"
kubectl delete svc/lbtest deploy/lbtest

You need permission to create namespaces, CRDs, ClusterRoles, Helm releases, and LoadBalancer Services.

Plan Your Deployment​

Make these decisions first. The variables below drive every later step.

1. Hostnames​

Pick a base domain you control, such as amp.yourdomain.com. Every management hostname sits under it.

HostnameServesDNS target
console.<base>Consolecontrol-plane gateway
api-amp.<base>API (browser, amctl, MCP)control-plane gateway
thunder.<base>Thunder login, plus <handle>.<base> per environmentcontrol-plane gateway
cp.<base>Gateway control plane (optional)control-plane gateway
traces.<base>Observerobservability-plane gateway
*.agents.<base> and agents.<base>Deployed agentsdata-plane gateway

A single *.<base> wildcard record covers the control-plane hostnames and the per-environment Thunder hostnames, which are created after install. Step 10 lists the exact records.

Download the release bundle. Every manifest, values file, and script this guide uses comes from it:

export VERSION="2026.11.0"
curl -fsSL -O "https://github.com/wso2/agent-manager/releases/download/amp/v${VERSION}/wso2-agent-manager-${VERSION}.tar.gz"
tar xzf "wso2-agent-manager-${VERSION}.tar.gz"
export AMP_DIST="$PWD"

Set the remaining variables. Replace <version> and <project> with your values from About the STS Release. The version is 2026.11.0 for the release, or 2026.11.0-u<N> for update N. The release bundle above serves both. GATEWAY_REGISTRY is where Phase 2 pulls the API Platform gateway from.

export VERSION="<version>"
export HELM_CHART_REGISTRY="registry.wso2.com/<project>/sts/helm-charts"
export GATEWAY_REGISTRY="registry.wso2.com/<project>/sts"
export AMP_NS="wso2-amp"
export BUILD_CI_NS="openchoreo-workflow-plane"
export OBSERVABILITY_NS="openchoreo-observability-plane"
export DEFAULT_NS="default"
export DATA_PLANE_NS="openchoreo-data-plane"
export THUNDER_NS="amp-thunder"

export BASE_DOMAIN="amp.yourdomain.com"

export CONSOLE_PUBLIC_HOST="console.${BASE_DOMAIN}"
export API_PUBLIC_HOST="api-amp.${BASE_DOMAIN}"
export THUNDER_PUBLIC_HOST="thunder.${BASE_DOMAIN}"
export CP_GW_PUBLIC_HOST="cp.${BASE_DOMAIN}"
export OBS_API_PUBLIC_HOST="traces.${BASE_DOMAIN}"
export AGENTS_DOMAIN="agents.${BASE_DOMAIN}"

export CONSOLE_PUBLIC_URL="https://${CONSOLE_PUBLIC_HOST}"
export API_PUBLIC_URL="https://${API_PUBLIC_HOST}"
export THUNDER_PUBLIC_URL="https://${THUNDER_PUBLIC_HOST}"
export OBS_API_PUBLIC_URL="https://${OBS_API_PUBLIC_HOST}"

export THUNDER_INTERNAL_URL="http://amp-thunder-extension-service.${THUNDER_NS}.svc.cluster.local:8090"

export INSTRUMENTATION_URL="http://default-default.gateway.localhost:19080/otel"

Sign Helm in to registry.wso2.com with your registry credentials. Phase 2 uses the same username and token to create the image pull secrets:

printf 'Registry username: ' && read -r AMP_REGISTRY_USERNAME
printf 'Registry token: ' && stty -echo && read -r AMP_REGISTRY_PASSWORD; stty echo; echo
printf '%s' "$AMP_REGISTRY_PASSWORD" | helm registry login registry.wso2.com --username "$AMP_REGISTRY_USERNAME" --password-stdin
note

Certificates are single-level wildcards (*.<base>). Every management hostname must sit directly under the base domain, with no second-level names such as api.amp.<base>.

2. TLS​

The main flow uses cert-manager with Let's Encrypt DNS-01. You need API credentials for the DNS provider that hosts your zone. Step 3 also covers a corporate CA. Certificate issuance needs only the zone, so it works before the gateway DNS records exist.

3. Decisions you cannot change later​

These settings are written to a database or certificate on first boot. Changing one afterwards means uninstalling and reinstalling that component.

DecisionSet inWhy it is frozen
Thunder's database (SQLite or PostgreSQL)Step 4Clients are seeded on first boot. helm upgrade does not re-seed.
Thunder's platform client secretsStep 4Same seeding job
THUNDER_PUBLIC_URL, CONSOLE_PUBLIC_URL, API_PUBLIC_URL, OBS_API_PUBLIC_URLStep 4Stored as the issuer, redirect URIs, and MCP resource identifiers
Thunder replicas > 1Step 4Requires external PostgreSQL and a shared Redis cache
Each environment's Thunder database (SQLite or PostgreSQL)Prepare StorageThe provisioning script refuses to switch an installed environment to an empty database. Changing it means migrating the data yourself and setting THUNDER_FORCE_STORAGE_CHANGE=true.
Agent Manager's databasePhase 2Switching later is a data migration
Environment gateway topologyPhase 2Role is written at first registration. Changing it means recreating the environment.

Everything else, including OpenSearch sizing, the registry, replica counts, and isolation tiers, can change on a running platform. See the production recommendations.

4. Platform secrets​

Generate real values for the platform's internal OAuth clients and the OpenSearch admin:

export AMP_API_CLIENT_SECRET="$(openssl rand -hex 32)"
export AMP_SYSTEM_CLIENT_SECRET="$(openssl rand -hex 32)"
export AMP_PUBLISHER_CLIENT_SECRET="$(openssl rand -hex 32)"
export AM_OBSERVER_CLIENT_SECRET="$(openssl rand -hex 32)"
export WORKFLOW_PUBLISHER_SECRET="$(openssl rand -hex 32)"
export OBSERVER_READER_SECRET="$(openssl rand -hex 32)"
export OPENSEARCH_USERNAME="admin"
export OPENSEARCH_PASSWORD="$(openssl rand -base64 24)"

Keep this shell session for the whole install. Store the values in your secret manager.

OTLP ingest

Agents inside the cluster can use the default INSTRUMENTATION_URL. For agents outside the cluster, install the API Platform Gateway extension with a public hostname (--set gateway.vhost=https://otel.${BASE_DOMAIN} and --set gateway.hostname=otel.${BASE_DOMAIN}), then set INSTRUMENTATION_URL=https://otel.${BASE_DOMAIN}/otel.

Phase 1: OpenChoreo Platform​

Phase 1 installs four OpenChoreo planes (Control, Data, Workflow, and Observability) and Thunder, the identity provider. Estimated time: 20 to 30 minutes.

Step 1: Install Cluster Prerequisites​

Install the Gateway API CRDs:

kubectl apply --server-side --force-conflicts \
  -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.1/experimental-install.yaml

Add --force-conflicts if another controller, such as Traefik on k3s, already manages these CRDs.

Install cert-manager (v1.19.2):

helm upgrade --install --server-side=false cert-manager oci://quay.io/jetstack/charts/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.19.2 \
  --set crds.enabled=true \
  --set startupapicheck.timeout=5m \
  --wait --timeout 360s

Install External Secrets Operator (v1.3.2):

helm upgrade --install --server-side=false external-secrets oci://ghcr.io/external-secrets/charts/external-secrets \
  --namespace external-secrets \
  --create-namespace \
  --version 1.3.2 \
  --set installCRDs=true \
  --wait --timeout 180s

Install kgateway (v2.2.1), the gateway for the OpenChoreo planes:

helm upgrade --install --server-side=false kgateway-crds oci://cr.kgateway.dev/kgateway-dev/charts/kgateway-crds \
  --create-namespace \
  --namespace openchoreo-control-plane \
  --version v2.2.1

helm upgrade --install --server-side=false kgateway oci://cr.kgateway.dev/kgateway-dev/charts/kgateway \
  --namespace openchoreo-control-plane \
  --create-namespace \
  --version v2.2.1 \
  --set controller.extraEnv.KGW_ENABLE_GATEWAY_API_EXPERIMENTAL_FEATURES=true
Rancher Desktop or k3s

Remove Traefik first. See the Rancher Desktop / k3s appendix.

Step 2: Set Up the Secret Stores​

Every platform secret goes through External Secrets. OpenBao is required, because Agent Manager reads Git credentials for private repositories from it directly over the Vault API. Deployed agents' environment variables can use OpenBao or another External Secrets provider. See a different store for deployed agents.

Install OpenBao with persistent storage, sealed at rest:

helm upgrade --install --server-side=false openbao oci://ghcr.io/openbao/charts/openbao \
  --namespace openbao \
  --create-namespace \
  --version 0.25.6 \
  --set server.dataStorage.size=10Gi \
  --timeout 180s

# Wait until bao answers. Exit 2 means sealed but responding, which is the expected state.
until kubectl exec -n openbao openbao-0 -- bao status >/dev/null 2>&1; do
  [ $? -eq 2 ] && break
  sleep 5
done

Initialize and unseal. Store the keys and root token in your secret manager now. They are shown once.

kubectl exec -n openbao openbao-0 -- bao operator init -key-shares=5 -key-threshold=3
kubectl exec -n openbao openbao-0 -- bao operator unseal <unseal-key-1>
kubectl exec -n openbao openbao-0 -- bao operator unseal <unseal-key-2>
kubectl exec -n openbao openbao-0 -- bao operator unseal <unseal-key-3>

A sealed OpenBao stays sealed after every pod restart. For unattended restarts, enable auto-unseal against a cloud KMS. Choose your platform:

Set the seal stanza in server.standalone.config, and give the pod credentials for your KMS. You still run bao operator init once, but with -recovery-shares and -recovery-threshold in place of -key-shares and -key-threshold, which a KMS seal rejects. It returns recovery keys and the root token. Skip the unseal commands; the KMS unseals OpenBao.

Create the KV v2 mount, the Kubernetes auth roles, and the seeded secrets:

kubectl exec -n openbao openbao-0 -- sh -c "
export BAO_ADDR=http://127.0.0.1:8200
export BAO_TOKEN='<root-token>'

bao secrets enable -path=secret -version=2 kv
bao auth enable kubernetes
bao write auth/kubernetes/config kubernetes_host=\"https://\$KUBERNETES_PORT_443_TCP_ADDR:443\"

bao policy write openchoreo-secret-reader-policy - <<POLICY
path \"secret/data/*\" { capabilities = [\"read\"] }
path \"secret/metadata/*\" { capabilities = [\"list\", \"read\"] }
POLICY

bao policy write openchoreo-secret-writer-policy - <<POLICY
path \"secret/data/*\" { capabilities = [\"create\", \"read\", \"update\", \"delete\"] }
path \"secret/metadata/*\" { capabilities = [\"create\", \"read\", \"update\", \"delete\", \"list\"] }
POLICY

bao write auth/kubernetes/role/openchoreo-secret-reader-role \
  bound_service_account_names=default \
  bound_service_account_namespaces='dp*' \
  policies=openchoreo-secret-reader-policy ttl=20m

bao write auth/kubernetes/role/openchoreo-secret-writer-role \
  bound_service_account_names='*' \
  bound_service_account_namespaces='openbao,openchoreo-workflow-plane,wso2-amp' \
  policies=openchoreo-secret-writer-policy ttl=20m

bao kv put secret/workflow-plane-oauth-client-secret value='${WORKFLOW_PUBLISHER_SECRET}'
bao kv put secret/amp-publisher-client-secret value='${AMP_PUBLISHER_CLIENT_SECRET}'
bao kv put secret/amp-system-client-secret value='${AMP_SYSTEM_CLIENT_SECRET}'
bao kv put secret/observer-oauth-client-secret value='${OBSERVER_READER_SECRET}'
bao kv put secret/opensearch-username value='${OPENSEARCH_USERNAME}'
bao kv put secret/opensearch-password value='${OPENSEARCH_PASSWORD}'
"

Mint the token Agent Manager uses to write agent secrets:

AMP_BAO_TOKEN=$(kubectl exec -n openbao openbao-0 -- sh -c "
export BAO_ADDR=http://127.0.0.1:8200
export BAO_TOKEN='<root-token>'
bao token create -policy=openchoreo-secret-writer-policy -period=768h -format=json" \
  | python3 -c "import json,sys; print(json.load(sys.stdin)['auth']['client_token'])")

kubectl create namespace ${AMP_NS} --dry-run=client -o yaml | kubectl apply -f -
kubectl create secret generic amp-openbao-token -n ${AMP_NS} \
  --from-literal=openbao-token="${AMP_BAO_TOKEN}" \
  --from-literal=workflow-plane-openbao-token="${AMP_BAO_TOKEN}"
Renew the token

Phase 2 must reference this Secret. The chart defaults to the dev-mode token root, which a sealed OpenBao rejects. The platform still installs and passes health checks, but Git-backed repository pickers fail. The periodic token must be renewed before its 768-hour period lapses.

Configure the ClusterSecretStore. It must be named default, because the build templates reference that name:

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-secrets-openbao
  namespace: openbao
---
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: default
spec:
  provider:
    vault:
      server: "http://openbao.openbao.svc:8200"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "openchoreo-secret-writer-role"
          serviceAccountRef:
            name: "external-secrets-openbao"
            namespace: "openbao"
EOF

Using a different secret store for deployed agents​

Optional. Skip this if OpenBao holds everything.

Deployed agents' environment variables can live in any External Secrets provider. Add a second store next to default. This example uses Google Cloud Secret Manager:

kubectl apply -f - <<'EOF'
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: dataplane-secrets
spec:
  provider:
    # Replace with your provider's stanza. See the External Secrets provider docs.
    gcpsm:
      projectID: "your-project"
      auth:
        workloadIdentity:
          clusterLocation: us-central1
          clusterName: your-cluster
          serviceAccountRef:
            name: external-secrets
            namespace: external-secrets
EOF

Then set secretStoreRef.name: dataplane-secrets on the ClusterDataPlane in Step 6. Leave the workflow plane on default.

Keep default on OpenBao

Repointing default breaks two things. Git credentials are read over the Vault API, so private-repository pickers stop listing anything. The platform's seeded secrets also use KV v2 remote keys that other providers do not match, and a mismatch shows as SecretSyncedError.

Step 3: Setup TLS Issuer​

Every gateway certificate references one ClusterIssuer named openchoreo-ca. Choose the tab for your certificate authority:

Confirm the zone is delegated. The output must list your DNS provider's nameservers:

dig +short NS <your-zone> @1.1.1.1

Create the DNS credentials and the issuer. The example uses Route 53. Cloudflare, Google Cloud DNS, Azure DNS, and others use the same structure.

kubectl create secret generic route53-credentials \
  --namespace cert-manager \
  --from-literal=secret-access-key='<AWS_SECRET_ACCESS_KEY>'

kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: openchoreo-ca
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: [email protected]
    privateKeySecretRef:
      name: letsencrypt-account-key
    solvers:
      - dns01:
          route53:
            region: us-east-1
            accessKeyID: <AWS_ACCESS_KEY_ID>
            secretAccessKeySecretRef:
              name: route53-credentials
              key: secret-access-key
        selector:
          dnsZones:
            - "${BASE_DOMAIN}"
EOF
tip

Use the staging server (https://acme-staging-v02.api.letsencrypt.org/directory) while you iterate. Production allows 5 duplicate certificates per week.

On EKS, cert-manager can use IAM Roles for Service Accounts or Pod Identity instead of a static key. Bind a role scoped to your hosted zone and reduce the solver to region and hostedZoneID.

Step 4: Install Thunder Extension (Identity Provider)​

Thunder handles login, API keys, and token exchange for the platform. The browser reaches it at https://${THUNDER_PUBLIC_HOST}.

These settings are permanent

Thunder writes its issuer, redirect URIs, and OAuth clients to its database on first boot. Confirm the values in Plan Your Deployment before you run the install. Getting one wrong means uninstalling, discarding Thunder's data, and reinstalling.

Create the four databases and a role that owns them. Thunder does not create its own tables on PostgreSQL. On managed services, grant the admin role membership first, or CREATE DATABASE ... OWNER fails:

CREATE ROLE thunder LOGIN PASSWORD '...';
GRANT thunder TO CURRENT_USER;  -- required on managed PostgreSQL; harmless elsewhere
CREATE DATABASE configdb OWNER thunder;  -- likewise entitydb, runtime_transient, runtime_persistent

Load the schema into each database from the Thunder image:

THUNDER_IMAGE="ghcr.io/thunder-id/thunderid:1.0.0"
docker create --name thunder-schema "${THUNDER_IMAGE}"
for db in configdb entitydb runtime_transient runtime_persistent; do
  docker cp "thunder-schema:/opt/thunderid/dbscripts/${db}/postgres.sql" "./thunder-${db}.sql"
  psql "postgresql://[email protected]:5432/${db}" -v ON_ERROR_STOP=1 -f "./thunder-${db}.sql"
done
docker rm thunder-schema

You should see 20 tables in configdb, 5 in entitydb, 13 in runtime_transient, and 7 in runtime_persistent.

Database on a private network

If psql on your machine cannot reach the database, which is the case for a private RDS, Cloud SQL, or Azure instance, keep the docker create and docker cp lines and run psql from a pod in the cluster instead:

kubectl run thunder-psql --image=postgres:17 --restart=Never \
  --env=PGHOST=your-db.example.com --env=PGSSLMODE=require -- sleep 3600
kubectl wait --for=condition=Ready pod/thunder-psql --timeout=120s
for db in configdb entitydb runtime_transient runtime_persistent; do
  kubectl cp "./thunder-${db}.sql" "thunder-psql:/tmp/thunder-${db}.sql"
  kubectl exec thunder-psql -- env PGUSER=thunder PGPASSWORD='<THUNDER_DB_PASSWORD>' \
    psql -d "${db}" -v ON_ERROR_STOP=1 -f "/tmp/thunder-${db}.sql"
done
kubectl delete pod thunder-psql

The same pod works for the CREATE ROLE and CREATE DATABASE statements: run kubectl exec -it thunder-psql -- env PGUSER=<admin-user> psql -d postgres before you delete it.

Store the password in a Secret, so it stays out of Helm history, then write the values file:

kubectl create namespace ${THUNDER_NS} --dry-run=client -o yaml | kubectl apply -f -
kubectl create secret generic thunder-db-credentials \
  --namespace ${THUNDER_NS} \
  --from-literal=password='<THUNDER_DB_PASSWORD>'

export THUNDER_DB_HOST="your-db.example.com"

cat > thunder-db-values.yaml <<EOF
thunder:
  configuration:
    database:
      config:
        type: postgres
        postgres:
          hostname: "${THUNDER_DB_HOST}"
          port: "5432"
          name: configdb
          username: thunder
          sslmode: require
          passwordRef:
            name: thunder-db-credentials
            key: password
      entity:
        type: postgres
        postgres:
          hostname: "${THUNDER_DB_HOST}"
          port: "5432"
          name: entitydb
          username: thunder
          sslmode: require
          passwordRef:
            name: thunder-db-credentials
            key: password
      runtime_transient:
        type: postgres
        postgres:
          hostname: "${THUNDER_DB_HOST}"
          port: "5432"
          name: runtime_transient
          username: thunder
          sslmode: require
          passwordRef:
            name: thunder-db-credentials
            key: password
      runtime_persistent:
        type: postgres
        postgres:
          hostname: "${THUNDER_DB_HOST}"
          port: "5432"
          name: runtime_persistent
          username: thunder
          sslmode: require
          passwordRef:
            name: thunder-db-credentials
            key: password
EOF

Use sslmode: verify-full if your provider gives you a CA to pin.

Install Thunder:

helm install --server-side=false amp-thunder-extension \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-thunder-extension \
  --version ${VERSION} \
  --namespace ${THUNDER_NS} \
  --create-namespace \
  --set thunder.ocIngress.hostname="${THUNDER_PUBLIC_HOST}" \
  --set thunder.configuration.server.publicUrl="${THUNDER_PUBLIC_URL}" \
  --set thunder.configuration.jwt.issuer="${THUNDER_PUBLIC_URL}" \
  --set thunder.configuration.gateClient.hostname="${THUNDER_PUBLIC_HOST}" \
  --set thunder.configuration.gateClient.scheme=https \
  --set thunder.configuration.gateClient.port=443 \
  --set "thunder.configuration.cors.allowedOrigins={${CONSOLE_PUBLIC_URL}}" \
  --set "thunder.bootstrap.ampConsoleClient.redirectUris={${CONSOLE_PUBLIC_URL}/login}" \
  --set thunder.bootstrap.agentManagerMcpBaseUrl="${API_PUBLIC_URL}" \
  --set thunder.bootstrap.observerMcpBaseUrl="${OBS_API_PUBLIC_URL}" \
  --set thunder.bootstrap.ampApiClient.clientSecret="${AMP_API_CLIENT_SECRET}" \
  --set thunder.bootstrap.ampSystemClient.clientSecret="${AMP_SYSTEM_CLIENT_SECRET}" \
  --set thunder.bootstrap.ampPublisherClient.clientSecret="${AMP_PUBLISHER_CLIENT_SECRET}" \
  --set thunder.bootstrap.amObserverClient.clientSecret="${AM_OBSERVER_CLIENT_SECRET}" \
  --set thunder.bootstrap.workloadPublisherClient.clientSecret="${WORKFLOW_PUBLISHER_SECRET}" \
  --set thunder.bootstrap.observerResourceReaderClient.clientSecret="${OBSERVER_READER_SECRET}" \
  --set thunder.ocIngress.https.enabled=false \
  --values thunder-db-values.yaml \
  --timeout 1800s

kubectl wait --for=condition=Available \
  deployment -l app.kubernetes.io/instance=amp-thunder-extension \
  -n ${THUNDER_NS} --timeout=300s
Pass all six client secrets

The chart bootstraps six OAuth clients. A client you do not override keeps its shipped default, and the install still succeeds. The two workloadPublisherClient and observerResourceReaderClient lines are easy to miss. Without them, every agent build fails at the publish step with 401 invalid_client.

Disable Thunder's local-dev gateway

thunder.ocIngress.https.enabled defaults to true, which adds an unused self-signed gateway for local k3d. The --set above turns it off.

MCP resource identifiers

agentManagerMcpBaseUrl and observerMcpBaseUrl must equal the public origins of the API and observer. Thunder rejects a resource that does not match exactly, with invalid_target.

Verify the issuer:

kubectl exec -n ${THUNDER_NS} deploy/amp-thunder-extension-deployment -- \
  wget -qO- http://localhost:8090/.well-known/openid-configuration 2>/dev/null \
  | grep -o '"issuer":"[^"]*"'
# Expected: "issuer":"${THUNDER_PUBLIC_URL}"

The install also generates the console's admin password. Retrieve it:

kubectl get secret amp-admin-credentials -n ${THUNDER_NS} -o jsonpath='{.data.password}' | base64 -d
Reinstalling Thunder resets platform data

Discarding Thunder's data also recreates the organization. Agent Manager keys its tenant data on that organization, so the old rows become invisible without an error. On anything but a fresh install, also discard Agent Manager's database and re-run environment Thunder provisioning.

Step 5: Install OpenChoreo Control Plane​

Create the wildcard certificate first. Issuance needs only the zone:

kubectl create namespace openchoreo-control-plane --dry-run=client -o yaml | kubectl apply -f -

kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: cp-gateway-tls
  namespace: openchoreo-control-plane
spec:
  secretName: cp-gateway-tls
  issuerRef:
    name: openchoreo-ca
    kind: ClusterIssuer
  dnsNames:
    - "*.${BASE_DOMAIN}"
    - "${BASE_DOMAIN}"
  privateKey:
    rotationPolicy: Always
EOF

kubectl wait --for=condition=Ready certificate/cp-gateway-tls \
  -n openchoreo-control-plane --timeout=300s

Install the Control Plane. The OpenChoreo API stays inside the cluster. Agent Manager and the observer reach it over cluster DNS.

helm upgrade --install --server-side=false openchoreo-control-plane \
  oci://ghcr.io/openchoreo/helm-charts/openchoreo-control-plane \
  --version 1.2.8 \
  --namespace openchoreo-control-plane \
  --create-namespace \
  --values - <<EOF
features:
  secretManagement:
    enabled: true
openchoreoApi:
  config:
    server:
      publicUrl: "http://openchoreo-api.openchoreo-control-plane.svc.cluster.local:8080"
  http:
    enabled: false
    hostnames:
      - "api.${BASE_DOMAIN}"
backstage:
  enabled: false
  baseUrl: ""
  http:
    hostnames:
      - ""
security:
  oidc:
    issuer: "${THUNDER_PUBLIC_URL}"
    wellKnownEndpoint: "${THUNDER_INTERNAL_URL}/.well-known/openid-configuration"
    jwksUrl: "${THUNDER_INTERNAL_URL}/oauth2/jwks"
    authorizationUrl: "${THUNDER_PUBLIC_URL}/oauth2/authorize"
    tokenUrl: "${THUNDER_INTERNAL_URL}/oauth2/token"
gateway:
  tls:
    enabled: true
    hostname: "*.${BASE_DOMAIN}"
    certificateRefs:
      - name: cp-gateway-tls
EOF

kubectl wait --for=condition=Available \
  deployment --all -n openchoreo-control-plane --timeout=300s
Secret management is required

features.secretManagement.enabled=true is off by default. Without it, every agent creation fails with a 500.

Webhook race

If the install fails with no endpoints available for service "controller-manager-webhook-service", wait for the Control Plane deployments to become ready, then rerun the install.

Patch the service-account entitlement claim (required with Thunder ≥ 0.45)​

Thunder 0.45+ puts the client name in client_id, but OpenChoreo 1.2.8 reads entitlements from sub. Without this patch, service calls return 200 with empty lists or 403, and the gateway bootstrap fails with Environment 'default' not found.

patched_yaml=$(kubectl get configmap openchoreo-api-config -n openchoreo-control-plane -o yaml   | sed -E "s/claim:[[:space:]]*['\"]?sub['\"]?/claim: client_id/g")
echo "$patched_yaml" | kubectl apply --server-side --field-manager=helm --force-conflicts -f -

kubectl rollout restart deployment/openchoreo-api -n openchoreo-control-plane
kubectl rollout status deployment/openchoreo-api -n openchoreo-control-plane --timeout=120s

for binding in $(kubectl get clusterauthzrolebindings.openchoreo.dev -o jsonpath='{.items[*].metadata.name}'); do
  claim=$(kubectl get clusterauthzrolebinding.openchoreo.dev "$binding" -o jsonpath='{.spec.entitlement.claim}')
  if [ "$claim" = "sub" ]; then
    kubectl patch clusterauthzrolebinding.openchoreo.dev "$binding" --type=merge       -p '{"spec":{"entitlement":{"claim":"client_id"}}}'
  fi
done

Re-apply this patch after every helm upgrade of the Control Plane. See After a chart upgrade.

Step 6: Setup Data Plane​

Copy the cluster-gateway CA into the data-plane namespace:

kubectl create namespace openchoreo-data-plane --dry-run=client -o yaml | kubectl apply -f -

CA_CRT=$(kubectl get secret cluster-gateway-ca \
  -n openchoreo-control-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)
kubectl create configmap cluster-gateway-ca \
  --from-literal=ca.crt="$CA_CRT" \
  -n openchoreo-data-plane --dry-run=client -o yaml | kubectl apply -f -

TLS_CRT=$(kubectl get secret cluster-gateway-ca \
  -n openchoreo-control-plane -o jsonpath='{.data.tls\.crt}' | base64 -d)
TLS_KEY=$(kubectl get secret cluster-gateway-ca \
  -n openchoreo-control-plane -o jsonpath='{.data.tls\.key}' | base64 -d)
kubectl create secret generic cluster-gateway-ca \
  --from-literal=tls.crt="$TLS_CRT" \
  --from-literal=tls.key="$TLS_KEY" \
  --from-literal=ca.crt="$CA_CRT" \
  -n openchoreo-data-plane --dry-run=client -o yaml | kubectl apply -f -

Create the agents wildcard certificate and install the Data Plane:

kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: dp-gateway-tls
  namespace: openchoreo-data-plane
spec:
  secretName: dp-gateway-tls
  issuerRef:
    name: openchoreo-ca
    kind: ClusterIssuer
  dnsNames:
    - "*.${AGENTS_DOMAIN}"
    - "${AGENTS_DOMAIN}"
  privateKey:
    rotationPolicy: Always
EOF

kubectl wait --for=condition=Ready certificate/dp-gateway-tls \
  -n openchoreo-data-plane --timeout=300s

helm install --server-side=false openchoreo-data-plane \
  oci://ghcr.io/openchoreo/helm-charts/openchoreo-data-plane \
  --version 1.2.8 \
  --namespace openchoreo-data-plane \
  --create-namespace \
  --set clusterAgent.tls.generateCerts=true \
  --set gateway.tls.enabled=true \
  --set "gateway.tls.hostname=*.${AGENTS_DOMAIN}" \
  --set "gateway.tls.certificateRefs[0].name=dp-gateway-tls" \
  --set gateway.httpPort=80 \
  --set gateway.httpsPort=443 \
  --values "${AMP_DIST}/deployments/single-cluster/values-dp.yaml"

kubectl wait --for=condition=Available \
  deployment --all -n openchoreo-data-plane --timeout=600s
Set the gateway ports

The values files come from the k3d layout, where all planes share one load balancer. The httpPort and httpsPort flags above put each gateway on 80 and 443. Without them, the platform looks healthy but every published URL points at a closed port. Verify:

kubectl get svc gateway-default -n <plane-namespace> -o jsonpath='{.spec.ports[*].port}{"\n"}'
# Expected: 80 443

Register the Data Plane:

CA_CERT=$(kubectl get secret cluster-agent-tls \
  -n openchoreo-data-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)

kubectl apply -f - <<EOF
apiVersion: openchoreo.dev/v1alpha1
kind: ClusterDataPlane
metadata:
  name: default
  namespace: default
spec:
  planeID: default
  clusterAgent:
    clientCA:
      value: |
$(echo "$CA_CERT" | sed 's/^/        /')
  gateway:
    ingress:
      external:
        name: gateway-default
        namespace: openchoreo-data-plane
        http:
          host: "${AGENTS_DOMAIN}"
          listenerName: http
          port: 80
        https:
          host: "${AGENTS_DOMAIN}"
          listenerName: https
          port: 443
  secretStoreRef:
    name: default
EOF

secretStoreRef decides where deployed agents' environment variables are stored. Keep default for OpenBao, or name the separate store from Step 2.

Step 7: Setup Workflow Plane​

Copy the cluster-gateway CA:

kubectl create namespace openchoreo-workflow-plane --dry-run=client -o yaml | kubectl apply -f -

CA_CRT=$(kubectl get secret cluster-gateway-ca \
  -n openchoreo-control-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)
kubectl create configmap cluster-gateway-ca \
  --from-literal=ca.crt="$CA_CRT" \
  -n openchoreo-workflow-plane --dry-run=client -o yaml | kubectl apply -f -

TLS_CRT=$(kubectl get secret cluster-gateway-ca \
  -n openchoreo-control-plane -o jsonpath='{.data.tls\.crt}' | base64 -d)
TLS_KEY=$(kubectl get secret cluster-gateway-ca \
  -n openchoreo-control-plane -o jsonpath='{.data.tls\.key}' | base64 -d)
kubectl create secret generic cluster-gateway-ca \
  --from-literal=tls.crt="$TLS_CRT" \
  --from-literal=tls.key="$TLS_KEY" \
  --from-literal=ca.crt="$CA_CRT" \
  -n openchoreo-workflow-plane --dry-run=client -o yaml | kubectl apply -f -

Install the Workflow Plane:

helm install --server-side=false openchoreo-workflow-plane \
  oci://ghcr.io/openchoreo/helm-charts/openchoreo-workflow-plane \
  --version 1.2.8 \
  --namespace openchoreo-workflow-plane \
  --create-namespace \
  --set clusterAgent.tls.generateCerts=true \
  --set argo-workflows.controller.resources.requests.cpu=50m \
  --set argo-workflows.controller.resources.requests.memory=64Mi \
  --set argo-workflows.controller.resources.limits.cpu=200m \
  --set argo-workflows.controller.resources.limits.memory=128Mi \
  --timeout 600s

kubectl wait --for=condition=Available \
  deployment --all -n openchoreo-workflow-plane --timeout=600s

Register the Workflow Plane:

BP_CA_CERT=$(kubectl get secret cluster-agent-tls \
  -n openchoreo-workflow-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)

kubectl apply -f - <<EOF
apiVersion: openchoreo.dev/v1alpha1
kind: ClusterWorkflowPlane
metadata:
  name: default
  namespace: default
spec:
  planeID: default
  clusterAgent:
    clientCA:
      value: |
$(echo "$BP_CA_CERT" | sed 's/^/        /')
  secretStoreRef:
    name: default
EOF

Keep secretStoreRef on default. Git credentials and build secrets live in OpenBao, and Agent Manager reads Git credentials from it directly.

Container registry

The workflow plane pushes built images to a registry set by global.registry.endpoint in Phase 2. Use one with access control that creates repositories on push and accepts static credentials, such as Harbor (with auto-create on), ACR, Artifact Registry, or CNCF Distribution, and set global.defaultResources.registry.tlsVerify=true. Amazon ECR does not work; see the registry requirements in Phase 2. On EKS without another registry, see Set up a build registry on EKS in Troubleshooting. See OpenChoreo's registry guide.

Step 8: Setup Observability Plane​

Copy the cluster-gateway CA:

kubectl create namespace openchoreo-observability-plane --dry-run=client -o yaml | kubectl apply -f -

CA_CRT=$(kubectl get secret cluster-gateway-ca \
  -n openchoreo-control-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)
kubectl create configmap cluster-gateway-ca \
  --from-literal=ca.crt="$CA_CRT" \
  -n openchoreo-observability-plane --dry-run=client -o yaml | kubectl apply -f -

TLS_CRT=$(kubectl get secret cluster-gateway-ca \
  -n openchoreo-control-plane -o jsonpath='{.data.tls\.crt}' | base64 -d)
TLS_KEY=$(kubectl get secret cluster-gateway-ca \
  -n openchoreo-control-plane -o jsonpath='{.data.tls\.key}' | base64 -d)
kubectl create secret generic cluster-gateway-ca \
  --from-literal=tls.crt="$TLS_CRT" \
  --from-literal=tls.key="$TLS_KEY" \
  --from-literal=ca.crt="$CA_CRT" \
  -n openchoreo-observability-plane --dry-run=client -o yaml | kubectl apply -f -

Create the ExternalSecrets for OpenSearch and the observer:

kubectl apply -f - <<'EOF'
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: opensearch-admin-credentials
  namespace: openchoreo-observability-plane
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: default
  target:
    name: opensearch-admin-credentials
  data:
  - secretKey: username
    remoteRef:
      key: opensearch-username
      property: value
  - secretKey: password
    remoteRef:
      key: opensearch-password
      property: value
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: observer-secret
  namespace: openchoreo-observability-plane
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: default
  target:
    name: observer-secret
  data:
  - secretKey: OPENSEARCH_USERNAME
    remoteRef:
      key: opensearch-username
      property: value
  - secretKey: OPENSEARCH_PASSWORD
    remoteRef:
      key: opensearch-password
      property: value
  - secretKey: UID_RESOLVER_OAUTH_CLIENT_SECRET
    remoteRef:
      key: observer-oauth-client-secret
      property: value
EOF

kubectl wait -n openchoreo-observability-plane \
  --for=condition=Ready externalsecret/opensearch-admin-credentials \
  externalsecret/observer-secret --timeout=60s

Apply the custom OpenTelemetry Collector config, which trace ingest requires:

kubectl apply -f "${AMP_DIST}/deployments/values/oc-collector-configmap.yaml" \
  -n openchoreo-observability-plane

Create the gateway certificate and install the Observability Plane:

kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: obs-gateway-tls
  namespace: openchoreo-observability-plane
spec:
  secretName: obs-gateway-tls
  issuerRef:
    name: openchoreo-ca
    kind: ClusterIssuer
  dnsNames:
    - "*.${BASE_DOMAIN}"
    - "${BASE_DOMAIN}"
  privateKey:
    rotationPolicy: Always
EOF

kubectl wait --for=condition=Ready certificate/obs-gateway-tls \
  -n openchoreo-observability-plane --timeout=300s

helm install --server-side=false openchoreo-observability-plane \
  oci://ghcr.io/openchoreo/helm-charts/openchoreo-observability-plane \
  --version 1.2.8 \
  --namespace openchoreo-observability-plane \
  --create-namespace \
  --set gateway.tls.enabled=true \
  --set "gateway.tls.hostname=*.${BASE_DOMAIN}" \
  --set "gateway.tls.certificateRefs[0].name=obs-gateway-tls" \
  --set clusterAgent.tls.generateCerts=true \
  --set gateway.httpPort=80 \
  --set gateway.httpsPort=443 \
  --set observer.controlPlaneApiUrl="http://openchoreo-api.openchoreo-control-plane.svc.cluster.local:8080" \
  --set observer.extraEnv.AUTH_SERVER_BASE_URL="${THUNDER_PUBLIC_URL}" \
  --set security.oidc.jwksUrl="${THUNDER_INTERNAL_URL}/oauth2/jwks" \
  --set security.oidc.tokenUrl="${THUNDER_INTERNAL_URL}/oauth2/token" \
  --values "${AMP_DIST}/deployments/single-cluster/values-op.yaml" \
  --timeout 25m

kubectl wait --for=condition=Available \
  deployment --all -n openchoreo-observability-plane --timeout=900s

for sts in $(kubectl get statefulset -n openchoreo-observability-plane -o name 2>/dev/null); do
  kubectl rollout status "${sts}" -n openchoreo-observability-plane --timeout=900s
done

Apply the same entitlement-claim patch to the observer. Without it, build-log queries return 403:

patched_yaml=$(kubectl get configmap observer-auth-config -n openchoreo-observability-plane -o yaml   | sed -E "s/claim:[[:space:]]*['\"]?sub['\"]?/claim: client_id/g")
echo "$patched_yaml" | kubectl apply --server-side --field-manager=helm --force-conflicts -f -
kubectl rollout restart deployment/observer -n openchoreo-observability-plane
kubectl rollout status deployment/observer -n openchoreo-observability-plane --timeout=120s

Install the observability modules. The logs module deploys OpenSearch.

# Logs module, which deploys OpenSearch
helm upgrade --install --server-side=false observability-logs-opensearch \
  oci://ghcr.io/openchoreo/helm-charts/observability-logs-opensearch \
  --create-namespace \
  --namespace openchoreo-observability-plane \
  --version 0.5.5 \
  --set openSearchSetup.openSearchSecretName="opensearch-admin-credentials" \
  --set adapter.openSearchSecretName="opensearch-admin-credentials" \
  --set openSearch.persistence.size=100Gi \
  --set-string "openSearch.extraEnvs[0].name=OPENSEARCH_INITIAL_ADMIN_PASSWORD" \
  --set-string "openSearch.extraEnvs[0].value=${OPENSEARCH_PASSWORD}" \
  --timeout 10m

# Fluent Bit log collection. On GKE, disable the managed logging agent first.
helm upgrade --server-side=false observability-logs-opensearch \
  oci://ghcr.io/openchoreo/helm-charts/observability-logs-opensearch \
  --namespace openchoreo-observability-plane \
  --version 0.5.5 \
  --reuse-values \
  --set fluent-bit.enabled=true \
  --timeout 10m

# Metrics module. The values file widens the kube-state-metrics label allowlist,
# which usage attribution needs. It also pins Prometheus to a ClusterIP Service,
# so its unauthenticated API is never put on a public IP.
helm upgrade --install --server-side=false observability-metrics-prometheus \
  oci://ghcr.io/openchoreo/helm-charts/observability-metrics-prometheus \
  --create-namespace \
  --namespace openchoreo-observability-plane \
  --version 0.8.1 \
  --values "${AMP_DIST}/deployments/values/observability-metrics-prometheus.yaml" \
  --timeout 10m

# Tracing module, which uses the custom OTel Collector ConfigMap
helm upgrade --install --server-side=false observability-traces-opensearch \
  oci://ghcr.io/openchoreo/helm-charts/observability-tracing-opensearch \
  --create-namespace \
  --namespace openchoreo-observability-plane \
  --version 0.8.1 \
  --set openSearch.enabled=false \
  --set openSearchSetup.openSearchSecretName="opensearch-admin-credentials" \
  --set opentelemetry-collector.configMap.existingName="amp-opentelemetry-collector-config" \
  --timeout 10m
GKE: free port 2020

Fluent Bit binds port 2020 on every node. GKE's fluentbit-gke agent already uses it, so Fluent Bit crash-loops with Cannot listen on 0.0.0.0:2020. Disable GKE's agent before the upgrade above:

gcloud container clusters update <cluster> --zone <zone> --logging=NONE

Do not turn off hostNetwork instead. The filter then cannot reach the kubelet.

OpenSearch sizing and password

openSearch.persistence.size raises the volume from its 8Gi default. The single node holds all traces, logs, and metrics, so size it to your retention. OPENSEARCH_INITIAL_ADMIN_PASSWORD must match the value in OpenBao. The subchart alias is openSearch, with a capital S. Lowercase values are silently ignored.

Register the Observability Plane and link the other planes to it:

OP_CA_CERT=$(kubectl get secret cluster-agent-tls \
  -n openchoreo-observability-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)

kubectl apply -f - <<EOF
apiVersion: openchoreo.dev/v1alpha1
kind: ClusterObservabilityPlane
metadata:
  name: default
spec:
  planeID: default
  clusterAgent:
    clientCA:
      value: |
$(echo "$OP_CA_CERT" | sed 's/^/        /')
  observerURL: http://observer.openchoreo-observability-plane.svc.cluster.local:8080
EOF

kubectl patch clusterdataplane default -n default --type merge \
  -p '{"spec":{"observabilityPlaneRef":{"kind":"ClusterObservabilityPlane","name":"default"}}}'

kubectl patch clusterworkflowplane default -n default --type merge \
  -p '{"spec":{"observabilityPlaneRef":{"kind":"ClusterObservabilityPlane","name":"default"}}}'

Step 9: Verify OpenChoreo Installation​

kubectl get pods -n openchoreo-control-plane
kubectl get pods -n openchoreo-data-plane
kubectl get pods -n openchoreo-workflow-plane
kubectl get pods -n openchoreo-observability-plane
kubectl get pods -n amp-thunder
kubectl get clusterdataplane,clusterworkflowplane,clusterobservabilityplane

All pods should be Running or Completed.

Step 10: Publish DNS Records​

Read the gateway addresses:

for ns in openchoreo-control-plane openchoreo-observability-plane openchoreo-data-plane; do
  printf '%-34s %s\n' "$ns:" \
    "$(kubectl get svc gateway-default -n $ns \
        -o jsonpath='{.status.loadBalancer.ingress[0].ip}{.status.loadBalancer.ingress[0].hostname}')"
done
RecordPoints to
*.${BASE_DOMAIN}control-plane gateway
traces.${BASE_DOMAIN}observability-plane gateway
*.agents.${BASE_DOMAIN} and agents.${BASE_DOMAIN}data-plane gateway

The *.${BASE_DOMAIN} wildcard is required. Per-environment Thunder hostnames are not known until after install, and only the wildcard resolves them. Use A records for IPs (GKE, AKS) and CNAME records for hostnames (EKS).

dig +short thunder.${BASE_DOMAIN} traces.${BASE_DOMAIN} test.${AGENTS_DOMAIN}

Phase 2: Agent Manager Installation​

Install the Agent Manager components with production settings: an external database, the platform secrets from Plan Your Deployment, and two replicas of each stateless service. Create the Agent Manager database and role before you start. Switching later is a migration.

The Agent Manager installs as a set of Helm charts on top of OpenChoreo. The components fall into two groups based on install order:

  1. Agent Manager Core : Gateway Operator, Agent Manager, Agent Sandbox Module and Platform Resources (agent component types, workflow templates etc). Each depends on the one before it.
  2. Extensions : Secret Management, Observability, Evaluation extensions and the API Platform Gateway Extension.
Prerequisites

Thunder (identity provider) must be installed before proceeding — see the Thunder installation step in Phase 1. The variables THUNDER_PUBLIC_URL, THUNDER_INTERNAL_URL, CONSOLE_PUBLIC_URL, API_PUBLIC_URL, OBS_API_PUBLIC_URL, CONSOLE_PUBLIC_HOST, API_PUBLIC_HOST, OBS_API_PUBLIC_HOST, and INSTRUMENTATION_URL must be set from the Configuration Variables section. The production steps below additionally use AGENTS_DOMAIN — the domain deployed agents are served on, agents.<your base domain> — to build the gateway's hostname and virtual host. The *_PUBLIC_HOST variables carry the bare hostname (no scheme or port) and become the HTTPRoute hostnames on the plane gateways.

Create the wso2-registry-credentials image pull secret in the namespaces that run Agent Manager images, with the registry username and token from the Configuration Variables step:

for ns in "${AMP_NS}" "${OBSERVABILITY_NS}" workflows-default; do
  kubectl create namespace "$ns" --dry-run=client -o yaml | kubectl apply -f -
  kubectl create secret docker-registry wso2-registry-credentials \
    --namespace "$ns" \
    --docker-server=registry.wso2.com \
    --docker-username="$AMP_REGISTRY_USERNAME" \
    --docker-password="$AMP_REGISTRY_PASSWORD" \
    --dry-run=client -o yaml | kubectl apply -f -
done

Evaluation monitors run in workflows-default, the workflow namespace of the default organization.

Create the gateway's own pull secret, wso2-gateway-registry-credentials, in ${DATA_PLANE_NS}. The gateway operator reads it to pull the gateway chart, and every gateway pod lists it to pull the gateway images. Besides the usual .dockerconfigjson, it carries the credentials as plain username and password keys, which is the form the operator reads:

kubectl create namespace "${DATA_PLANE_NS}" --dry-run=client -o yaml | kubectl apply -f -
kubectl create secret generic wso2-gateway-registry-credentials \
  --namespace "${DATA_PLANE_NS}" \
  --type=kubernetes.io/dockerconfigjson \
  --from-literal=.dockerconfigjson="$(printf '{"auths":{"registry.wso2.com":{"auth":"%s"}}}' "$(printf '%s:%s' "$AMP_REGISTRY_USERNAME" "$AMP_REGISTRY_PASSWORD" | base64 | tr -d '\n')")" \
  --from-literal=username="$AMP_REGISTRY_USERNAME" \
  --from-literal=password="$AMP_REGISTRY_PASSWORD" \
  --dry-run=client -o yaml | kubectl apply -f -

Core Components​

Install these in order — each depends on the one before it.

Upgrading an existing installation? Reverse Steps 2 and 4

The order above is for a fresh install, where no agents exist yet and the order is harmless. When upgrading an already-running platform, upgrade wso2-amp-platform-resources-extension (Step 4) at or before wso2-agent-manager (Step 2), not after: the service writes component parameters that the chart must already know how to read, so an agent created in the window between the two upgrades can have its live invoke URL change on the later chart upgrade.

Step 1: Gateway Operator​

Manages API Gateway resources and enables secure, authenticated trace ingestion into the Observability Plane.

The gateway controller encrypts stored credentials at rest and will not start without a key, so create one before installing the operator:

kubectl create namespace ${DATA_PLANE_NS} --dry-run=client -o yaml | kubectl apply -f -

openssl rand 32 > gateway-aesgcm.key
kubectl create secret generic gateway-encryption-keys \
  --namespace ${DATA_PLANE_NS} \
  --from-file=default-aesgcm256-v1.bin=gateway-aesgcm.key
rm -f gateway-aesgcm.key
The key is required, and the file name is fixed

The controller looks for a specific path, /app/data/aesgcm-keys/default-aesgcm256-v1.bin, so the Secret key must be exactly default-aesgcm256-v1.bin. Without it the controller crash-loops on failed to initialize key manager: encryption key file not found for version aesgcm256-v1, and the gateway never programs — the rest of the platform stays healthy, so the only symptom is that agents cannot be invoked.

This applies whatever developmentMode is set to. Earlier gateway releases auto-generated a key in development mode; from 1.2.0-beta they do not.

Store the key with the rest of your platform secrets. It encrypts credentials the gateway holds, and losing it means those entries cannot be decrypted.

The gateway for this release, 2026.10.0, is published only to registry.wso2.com. The operator itself still comes from ghcr.io, and pulls the gateway chart and images from GATEWAY_REGISTRY, set in the Configuration Variables step, with the wso2-gateway-registry-credentials secret. The controller in this gateway release serves its health check on the admin port under /api/admin/v1/health, so the probes are set as well.

helm install --server-side=false gateway-operator \
  oci://ghcr.io/wso2/api-platform/helm-charts/gateway-operator \
  --version 0.11.3 \
  --namespace ${DATA_PLANE_NS} \
  --set logging.level=info \
  --set gatewayApi.installStandardCRDs=false \
  --set gateway.helm.chartName=oci://${GATEWAY_REGISTRY}/helm-charts/gateway \
  --set gateway.helm.chartVersion=2026.10.0 \
  --set gateway.helm.registryCredentialsSecret.name=wso2-gateway-registry-credentials \
  --set "gateway.values.imagePullSecrets[0]=wso2-gateway-registry-credentials" \
  --set gateway.values.gateway.controller.image.repository=${GATEWAY_REGISTRY}/gateway-controller \
  --set gateway.values.gateway.controller.image.tag=2026.10.0 \
  --set gateway.values.gateway.gatewayRuntime.image.repository=${GATEWAY_REGISTRY}/gateway-runtime \
  --set gateway.values.gateway.gatewayRuntime.image.tag=2026.10.0 \
  --set gateway.values.gateway.controller.encryptionKeys.enabled=true \
  --set gateway.values.gateway.controller.encryptionKeys.secretName=gateway-encryption-keys \
  --set gateway.values.gateway.controller.deployment.livenessProbe.httpGet.path=/api/admin/v1/health \
  --set gateway.values.gateway.controller.deployment.livenessProbe.httpGet.port=admin \
  --set gateway.values.gateway.controller.deployment.readinessProbe.httpGet.path=/api/admin/v1/health \
  --set gateway.values.gateway.controller.deployment.readinessProbe.httpGet.port=admin \
  --timeout 600s
Pin the gateway chart, do not inherit it

gateway.helm.chartVersion decides which gateway chart the operator deploys, and it is the only thing that decides it — the APIGateway resource carries no chart version. Inheriting the operator's default has historically produced a chart whose templates do not render the controller's control-plane address, giving a gateway that installs, programs, and serves traffic while never registering with Agent Manager — so it never appears in the gateway list. Pin the chart explicitly, as above.

Pin the container images too. The chart version and the image tags are set separately, so chartVersion alone can leave the gateway on the chart's default images, which point at a public registry that does not carry this release. The four image.repository/image.tag values above pull the controller and runtime at 2026.10.0 from the registry. Check the pod images after installing rather than trusting the chart version alone.

These are the versions the gateway extension chart of your version is tested with. The chart records them in its gatewayRelease values, which the one-command installer reads, and an update can move them. To see them for your version:

helm show values "oci://${HELM_CHART_REGISTRY}/wso2-amp-api-platform-gateway-extension" --version "${VERSION}" | grep -A3 '^gatewayRelease:'

gateway.values is the operator's passthrough into that chart: anything set under it is merged into the values the gateway is deployed with, which is how the encryption keys above are wired.

Wait for the operator to be ready:

kubectl wait --for=condition=Available \
  deployment -l app.kubernetes.io/name=gateway-operator \
  -n ${DATA_PLANE_NS} --timeout=300s

Grant RBAC for WSO2 API Platform CRDs to the Data Plane cluster-agent:

kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: wso2-api-platform-gateway-module
rules:
  - apiGroups: ["gateway.api-platform.wso2.com"]
    resources: ["restapis", "apigateways"]
    verbs: ["*"]
  - apiGroups: ["gateway.kgateway.dev"]
    resources: ["backends"]
    verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: wso2-api-platform-gateway-module
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: wso2-api-platform-gateway-module
subjects:
  - kind: ServiceAccount
    name: cluster-agent-dataplane
    namespace: ${DATA_PLANE_NS}
EOF
info

The API Platform Gateway is deployed as an extension after Agent Manager is running — see Step 7 below.

Step 2: Agent Manager (API + Console + PostgreSQL)​

The core platform: a Go API server, a React web console, and a database.

Tell the Console where your registry is

The charts and images for this release come from the private registry registry.wso2.com. Add --set console.config.registry="registry.wso2.com/<project>/sts" to the install command below. The Console then generates registry-aware commands for adding environments: they run the scripts from the release bundle and pull from the registry, with your registry credentials. Without it, the Console generates the public command, which fetches from GitHub and ghcr.io and fails for this release.

The chart deploys an in-cluster PostgreSQL with a default password unless you point it at an external database. Use a managed service — sizing, backups, failover, and patching then come from your database platform. Create the database and a credentials Secret first:

kubectl create namespace ${AMP_NS} --dry-run=client -o yaml | kubectl apply -f -
kubectl create secret generic amp-db-credentials \
  --namespace ${AMP_NS} \
  --from-literal=password='<AMP_DB_PASSWORD>'

Then install, wiring the external database and the platform client secrets generated earlier:

helm install --server-side=false amp \
  oci://${HELM_CHART_REGISTRY}/wso2-agent-manager \
  --version ${VERSION} \
  --namespace ${AMP_NS} \
  --create-namespace \
  --set console.config.instrumentationUrl="${INSTRUMENTATION_URL}" \
  --set console.config.auth.baseUrl="${THUNDER_PUBLIC_URL}" \
  --set console.config.auth.signInRedirectURL="${CONSOLE_PUBLIC_URL}/login" \
  --set console.config.auth.signOutRedirectURL="${CONSOLE_PUBLIC_URL}/login" \
  --set console.config.apiBaseUrl="${API_PUBLIC_URL}" \
  --set agentManagerService.config.amObserverPublicURL="${OBS_API_PUBLIC_URL}" \
  --set console.ocIngress.hostname="${CONSOLE_PUBLIC_HOST}" \
  --set agentManagerService.ocIngress.hostname="${API_PUBLIC_HOST}" \
  --set agentManagerService.config.serverPublicURL="${API_PUBLIC_URL}" \
  --set agentManagerService.config.keyManager.issuer="${THUNDER_PUBLIC_URL}" \
  --set agentManagerService.config.keyManager.jwksUrl="${THUNDER_INTERNAL_URL}/oauth2/jwks" \
  --set agentManagerService.config.oidc.tokenUrl="${THUNDER_INTERNAL_URL}/oauth2/token" \
  --set agentManagerService.config.oidc.clientSecret="${AMP_API_CLIENT_SECRET}" \
  --set agentManagerService.config.thunder.baseURL="${THUNDER_PUBLIC_URL}" \
  --set agentManagerService.config.thunder.resolveToHost="${THUNDER_INTERNAL_URL#http://}" \
  --set agentManagerService.config.thunder.clientSecret="${AMP_SYSTEM_CLIENT_SECRET}" \
  --set agentManagerService.config.openChoreo.baseURL="http://openchoreo-api.openchoreo-control-plane.svc.cluster.local:8080" \
  --set agentManagerService.config.tlsEnabled=true \
  --set agentManagerService.replicaCount=2 \
  --set agentManagerService.autoscaling.minReplicas=2 \
  --set console.replicaCount=2 \
  --set agentManagerService.config.openbao.existingSecret=amp-openbao-token \
  --set agentManagerService.config.workflowPlaneOpenbao.existingSecret=amp-openbao-token \
  --set agentManagerService.config.workflowPlaneOpenbao.existingSecretKey=workflow-plane-openbao-token \
  --set postgresql.enabled=false \
  --set postgresql.external.host="your-db.example.com" \
  --set postgresql.external.port=5432 \
  --set postgresql.external.database=agentmanager \
  --set postgresql.external.username=agentmanager \
  --set postgresql.external.existingSecret=amp-db-credentials \
  --set postgresql.external.existingSecretPasswordKey=password \
  --set postgresql.external.sslMode=require \
  --timeout 1800s

Wait for all components — with an external database no PostgreSQL StatefulSet is deployed, so there is nothing to wait for there:

# API server
kubectl wait --for=condition=Available \
  deployment/amp-api -n ${AMP_NS} --timeout=600s

# Console
kubectl wait --for=condition=Available \
  deployment/amp-console -n ${AMP_NS} --timeout=600s

agentManagerService.config.thunder.clientSecret can instead be supplied through agentManagerService.config.thunder.existingSecret (key thunder-client-secret) if you prefer to keep it out of Helm release history.

agentManagerService.config.apiKey (the X-API-Key header value some internal callers use) is left unset above deliberately: the chart generates a random 32-character value itself when empty, the same way encryptionKey already does. Set agentManagerService.config.apiKey.value or .existingSecret explicitly only if something outside the platform needs to know this value in advance.

tlsEnabled decides which agent invoke URL the console publishes

agentManagerService.config.tlsEnabled defaults to false, and it is what selects between an Environment's http and https endpoint variants when the API builds an agent's invoke URL. Left at the default on an HTTPS platform, the console publishes an http:// URL — which the browser then refuses to call from an HTTPS page, showing the request as blocked:mixed-content with no error anywhere in the platform. The agent itself is healthy and reachable; only the published URL is wrong.

Two values that look redundant but are not

agentManagerService.autoscaling.minReplicas=2 is what actually gives the API two replicas. Autoscaling is on by default for this component with minReplicas: 1, so the HPA takes ownership of the replica count and scales replicaCount=2 straight back down to one — silently, within seconds. The console has autoscaling off by default, which is why replicaCount alone appears to work there.

The two openbao.existingSecret values point at the Secret minted when you set up the secret stores. Without them the chart falls back to the dev-mode token root, which a sealed OpenBao rejects. workflowPlaneOpenbao is the one that bites: it is what Agent Manager uses to read Git credentials, so the repository and branch pickers fail and no agent can be created from a private repository, while the platform otherwise looks healthy.

Require TLS on the database connection

postgresql.external.sslMode=require above makes the connection fail closed. Left unset, the driver's default of prefer applies: it negotiates TLS when the server offers it but silently falls back to an unencrypted connection when it does not, so a misconfigured or replaced instance downgrades without any error.

require encrypts but does not verify the server's identity. To also authenticate the database, use sslMode=verify-full with postgresql.external.sslRootCert. Set it to system only when the certificate chains to a public CA; otherwise mount the provider's CA PEM through the chart's volumes/volumeMounts values and point sslRootCert at that path. Managed services differ here — Cloud SQL, for example, issues its server certificate from a Google-managed private CA in either CA mode, so system fails with certificate verify failed and the CA must be mounted.

Check the hostname too: verify-full matches the certificate's SAN exactly, so postgresql.external.host has to be a name the certificate actually covers rather than an IP. Cloud SQL's SAN carries a trailing dot, which means the host value needs one as well.

Enforce it on the server as well, where the client cannot downgrade it: Require SSL on Cloud SQL, rds.force_ssl=1 on RDS, or require secure transport on Azure Database for PostgreSQL. Keep the database on a private network so it is never reachable from outside the VPC.

info

keyManager.issuer must be the public Thunder URL — the chart default is the k3d hostname. The authorization server the API advertises in its RFC 9728 protected resource metadata is derived from it, so agentManagerService.config.oauthAuthorizationServers only needs setting if it must differ. Likewise the MCP audience entry is derived from serverPublicURL above, so keyManager.audience only needs setting to change the accepted client IDs.

thunder.baseURL must also be the public Thunder URL, for a related but separate reason: every admin identity call (users, roles, groups) requests a token scoped to Thunder's System resource server via an RFC 8707 resource parameter derived from this value, and Thunder only recognizes the public URL as that resource server's registered identifier — anything else fails with invalid_target. Since the API runs in-cluster and the public hostname often doesn't resolve there, thunder.resolveToHost tells it which address to actually connect to while still presenting thunder.baseURL on the wire; set it to the in-cluster Thunder service (as above) whenever the public hostname isn't directly dialable from this cluster. Leaving thunder.baseURL at the chart default (or omitting resolveToHost when the public hostname can't resolve in-cluster) breaks every admin identity call with invalid_target or a DNS lookup failure, while the rest of the platform looks healthy.

Verify
kubectl get pods -n ${AMP_NS}
# Expected: amp-api-xxx (Running), amp-console-xxx (Running), plus
# amp-postgresql-0 (Running) only when using the in-cluster database

Step 3: Agent Sandbox Module​

Agents run as sandboxed pods managed by the Agent Sandbox controller (SandboxTemplate / SandboxWarmPool resources) instead of plain Deployments. This module is required — without it, agent deployments cannot be rendered.

helm upgrade --install --server-side=false agent-sandbox \
  oci://ghcr.io/openchoreo/helm-charts/agent-sandbox \
  --version 0.1.1 \
  --namespace ${DATA_PLANE_NS} \
  --create-namespace \
  --wait \
  --timeout 10m \
  --set namespace=openchoreo-control-plane \
  --set dataPlaneNamespace=${DATA_PLANE_NS} \
  --set dataPlaneServiceAccount=cluster-agent-dataplane \
  --set upstream.version=v0.4.6

Wait for the controller:

kubectl wait -n agent-sandbox-system \
  --for=condition=available \
  --timeout=180s \
  deployment/agent-sandbox-controller
Verify
kubectl get crd \
  sandboxtemplates.extensions.agents.x-k8s.io \
  sandboxwarmpools.extensions.agents.x-k8s.io \
  sandboxclaims.extensions.agents.x-k8s.io
# All three CRDs should be listed

kubectl get clusterrole openchoreo-agent-sandbox-access
# RBAC granting the data-plane agent access to sandbox resources
Stronger isolation tiers

With this module, agents run sandboxed under the standard runc runtime. Optionally, individual environments can run agents under gVisor or Kata Containers for stronger isolation — these have hardware/OS requirements and need a dedicated node. See the gVisor and Kata Containers isolation tier guides.

Step 4: Platform Resources​

Creates the default Organization, Project, Environment, DeploymentPipeline, and workflow template resources that the console needs on first login. This chart also configures the container registry endpoint used by build workflows to push agent images.

helm install --server-side=false amp-platform-resources \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-platform-resources-extension \
  --version ${VERSION} \
  --namespace ${DEFAULT_NS} \
  --set global.oauth.tokenUrl="${THUNDER_INTERNAL_URL}/oauth2/token" \
  --set global.oauth.hostHeader="amp-thunder-extension-service.${THUNDER_NS}.svc.cluster.local" \
  --set global.apiServer.url="http://openchoreo-api.openchoreo-control-plane.svc.cluster.local:8080" \
  --set global.apiServer.hostHeader="openchoreo-api.openchoreo-control-plane.svc.cluster.local" \
  --set apiPlatformGateway.namespace=${DATA_PLANE_NS} \
  --set global.registry.endpoint="<your-registry-host>" \
  --set global.defaultResources.registry.tlsVerify=true \
  --timeout 1800s

Set global.registry.endpoint to the registry the workflow plane pushes agent images to. Left unset, it defaults to the k3d registry host.k3d.internal:10082, and the first agent build fails to push. The registry requirements are below.

These four values default to k3d addresses

global.oauth.tokenUrl and global.apiServer.url default to http://host.k3d.internal:8080, which the single-cluster k3d layout reaches through the Docker host and routes by Host header. On any other cluster that name does not resolve, and the failure surfaces only when an agent is built: the workflow's generate-workload step prints Failed to get access token: with an empty body — a connection failure, not an authentication one — and the build ends with cannot save parameter /mnt/vol/workload-cr.yaml.

Nothing earlier in the install touches these values, so a platform that installs and verifies cleanly still cannot complete a single build until they are set. Point them at the in-cluster services as above.

Point the traits at the namespace the gateway is actually in

Deployed agents reach the gateway runtime by DNS name, and both the trace-export endpoint and the route agent traffic is forwarded through are built from apiPlatformGateway.namespace. Leaving it empty derives the per-org-env convention <org>-<env>, which is what add-environment.sh creates — but Step 7 installs the gateway extension into ${DATA_PLANE_NS}, matching that chart's own apiGateway.namespace default. Set this to the same namespace, as above.

Get it wrong and nothing reports an error. The agent starts and serves requests while every span batch fails inside it with Failed to resolve '…-gw-gateway-gateway-runtime.default-default' ([Errno -2] Name or service not known), so the Traces view stays empty; and the backend agent routes forward to has no reachable host, so invocation never reaches the gateway. The only evidence is a Transient error warning in the agent's own log.

If you install the gateway extension into <org>-<env> instead, leave this unset.

Container registry configuration

The chart defaults are configured for a local k3d cluster with an in-cluster registry at host.k3d.internal:10082. For other environments, override the registry settings:

# Example: external registry with a base domain
helm install --server-side=false amp-platform-resources \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-platform-resources-extension \
  --version ${VERSION} \
  --namespace ${DEFAULT_NS} \
  --set global.baseDomain="yourdomain.com" \
  --set global.defaultResources.registry.tlsVerify=true \
  --timeout 1800s
# Registry endpoint will be: registry.yourdomain.com

# Example: explicit registry endpoint
helm install --server-side=false amp-platform-resources \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-platform-resources-extension \
  --version ${VERSION} \
  --namespace ${DEFAULT_NS} \
  --set global.registry.endpoint="your-registry.example.com:5000" \
  --set global.defaultResources.registry.tlsVerify=true \
  --timeout 1800s
ValueDefaultDescription
global.registry.endpointhost.k3d.internal:10082Registry endpoint for pushing images
global.baseDomain""When set, registry endpoint becomes registry.<baseDomain>
global.defaultResources.registry.tlsVerifyfalseEnable TLS verification for registry connections
Not every registry can back build workflows

Two properties are required, and the failure surfaces only on the first agent build, long after the platform installs and verifies cleanly:

  • Push-to-create. Each build pushes its image as <workflow-run-name>-image — a repository name that is different on every run, so repositories cannot be pre-created. The registry must create them on push. Amazon ECR does not and fails every build with repository ... not found; the same applies to any registry that requires repositories to be provisioned first.
  • Static credentials. The push step reads a fixed .dockerconfigjson from the registry-push-secret key in OpenBao (or pushes unauthenticated if the key is absent). Registries whose credentials expire — ECR again, with its 12-hour tokens — cannot be refreshed by anything in the build pipeline.

CNCF Distribution, Harbor (with a project set to auto-create), and GitLab's registry satisfy both. If you have no registry yet, running Distribution in-cluster behind an internal LoadBalancer with a cert-manager certificate is a few resources — but it ships with no authentication; put htpasswd in front of it before it carries anything real.

To push with credentials, store them in OpenBao before the first build. Write the registry's .dockerconfigjson to registry-dockerconfig.json, in the form {"auths":{"<registry-host>":{"auth":"<base64 of user:token>"}}}, then:

kubectl exec -i -n openbao openbao-0 -- env BAO_ADDR=http://127.0.0.1:8200 BAO_TOKEN='<root-token>' \
  bao kv put secret/registry-push-secret - < registry-dockerconfig.json

Extensions​

These can be installed in any order after Core is ready.

Step 5: Observability Extension (Agent Manager Observer)​

Deploys the observer service that queries and serves trace, log, and metrics data to the console and CLI.

amObserver.observer.idpClientSecret is the observer's own outbound identity for calls to the OpenChoreo observer, and must match the am-observer-client secret seeded into Thunder. It can instead be supplied through amObserver.observer.existingSecret (keys existingSecretClientIdKey/existingSecretClientSecretKey, default idpClientId/idpClientSecret) if you prefer to keep it out of Helm release history, the same option thunder.clientSecret has above:

helm install --server-side=false amp-observability-traces \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-observability-extension \
  --version ${VERSION} \
  --namespace ${OBSERVABILITY_NS} \
  --set amObserver.ocIngress.hostname="${OBS_API_PUBLIC_HOST}" \
  --set amObserver.publicUrl="${OBS_API_PUBLIC_URL}" \
  --set amObserver.auth.issuer="${THUNDER_PUBLIC_URL}" \
  --set amObserver.observer.idpClientSecret="${AM_OBSERVER_CLIENT_SECRET}" \
  --set amObserver.replicaCount=2 \
  --timeout 1800s

kubectl wait --for=condition=Available \
  deployment/amp-observer -n ${OBSERVABILITY_NS} --timeout=600s
warning

amObserver.auth.issuer must be the public Thunder URL, not the in-cluster service URL. The observer validates the same user token the console and amctl send to the Agent Manager API, so its issuer has to match agentManagerService.config.keyManager.issuer from Step 2. Leave it at the chart default on a custom-domain install and the traces page stays empty while the observer logs JWT validation failed ... invalid issuer.

The chart derives amObserver.oauth.authorizationServers and the MCP audience entry from the values above; set them explicitly only if they must differ.

Step 6: Evaluation Extension​

Installs workflow templates for running automated evaluations (accuracy, safety, reasoning, tool usage) against agent traces.

helm install --server-side=false amp-evaluation-extension \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-evaluation-extension \
  --version ${VERSION} \
  --namespace ${BUILD_CI_NS} \
  --timeout 1800s
info

Evaluation jobs publish scores using the amp-publisher-client OAuth2 credentials bootstrapped by the Thunder extension. The client secret is fetched from OpenBao at workflow runtime (secret/amp-publisher-client-secret) and must match the value set on thunder.bootstrap.ampPublisherClient.clientSecret.

caution

The eval job needs egress to your API server, and the NetworkPolicy matches it by address range after kube-proxy DNAT — the endpoint address, never the kubernetes Service ClusterIP. Check what that address is:

kubectl -n default get endpoints kubernetes

networkPolicy.evaluationJob.apiServer.cidrs defaults to all of RFC1918, which covers that address on most private clusters but also spans your pod and service CIDRs. Add this to the command above to narrow it to your control-plane subnet:

helm install --server-side=false amp-evaluation-extension \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-evaluation-extension \
  --version ${VERSION} \
  --namespace ${BUILD_CI_NS} \
  --timeout 1800s \
  --set "networkPolicy.evaluationJob.apiServer.cidrs[0]=<control-plane-subnet>"

Setting it is required if kubernetes resolves to a public endpoint (public GKE/AKS control planes) or to 100.64.0.0/10. The default will not match those, and every evaluation will publish its scores and then report FAILED.

Step 7: API Platform Gateway Extension​

Registers the API Platform Gateway with the Agent Manager and deploys the gateway stack. Install this last — it requires the Agent Manager API to be healthy and Thunder to be ready for token exchange.

The bootstrap job authenticates to the Agent Manager API as amp-api-client, so it needs the secret you seeded into Thunder. Keep it out of Helm release history with a Secret reference:

kubectl create secret generic gateway-idp-credentials \
  --namespace ${DATA_PLANE_NS} \
  --from-literal=client-id=amp-api-client \
  --from-literal=client-secret="${AMP_API_CLIENT_SECRET}"

helm install --server-side=false api-platform-default-default \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-api-platform-gateway-extension \
  --version ${VERSION} \
  --namespace ${DATA_PLANE_NS} \
  --set agentManager.orgName=default \
  --set gateway.environment=default \
  --set gateway.type=BOTH \
  --set developmentMode=false \
  --set gateway.vhost="https://default-default.${AGENTS_DOMAIN}" \
  --set gateway.hostname="default-default.${AGENTS_DOMAIN}" \
  --set agentManager.idp.existingSecret=gateway-idp-credentials \
  --timeout 1800s

kubectl wait --for=condition=complete job/api-platform-default-default-bootstrap \
  -n ${DATA_PLANE_NS} --timeout=300s
Set the vhost now — it cannot be changed later

gateway.vhost and gateway.hostname default to the local k3d addresses (http://default-default.gateway.localhost:19080). The bootstrap job writes them into Agent Manager at first registration only: on every later run it finds the gateway already present, logs already exists, and exits without reconciling the record. A helm upgrade with corrected values changes nothing, and there is no error to notice — the console simply keeps showing the .localhost:19080 virtual host, which is what an operator would copy when wiring an external caller.

Correcting it after the fact means deleting the gateway registration and re-registering, so get these right on the first install.

gateway.type=BOTH registers this gateway to serve both inbound and outbound traffic, which is what a single-gateway environment needs. It matches the chart default and is set explicitly here because the role is written once at registration and never rewritten — see Gateway roles and topology for the split alternative and its constraints.

developmentMode=false turns off the relaxed security checks the chart enables by default. It depends on the encryption key from Step 1: without one the install fails immediately with encryptionKeys must be enabled: at-rest encryption is mandatory. Set it here rather than hardening afterwards, so the gateway is never registered in a relaxed state.

The extension already renders an OTLP ingest route for its own gateway (<release>-otel-restapi in the data-plane namespace), so nothing further is needed for the default environment. The standalone manifest below targets the per-environment namespace <org>-<env>, which does not exist until the first agent is deployed there — apply it only if you need that variant, and only after a deployment has created the namespace:

kubectl apply -f "${AMP_DIST}/deployments/values/otel-collector-rest-api.yaml"
Verify
kubectl get apigateway api-platform-default-default -n ${DATA_PLANE_NS}
# STATUS should show "Programmed"

kubectl get jobs -n ${DATA_PLANE_NS} | grep api-platform-default-default-bootstrap
# STATUS should show "Complete"

Wire the Remaining Public Endpoints​

Apply these after all Phase 2 charts are installed.

External AI gateway endpoint. Register the cp. hostname and show it in the console:

helm upgrade --server-side=false amp oci://${HELM_CHART_REGISTRY}/wso2-agent-manager \
  --version ${VERSION} \
  --namespace ${AMP_NS} \
  --reuse-values \
  --set "agentManagerService.ocIngress.gatewayMgmt.hostnames={${CP_GW_PUBLIC_HOST}}" \
  --set console.config.gatewayControlPlaneUrl="https://${CP_GW_PUBLIC_HOST}"

Deployed-agent endpoints. Point the default environment's gateway at the agents domain, and route the workload publisher's token calls in-cluster:

helm upgrade --server-side=false amp-platform-resources oci://${HELM_CHART_REGISTRY}/wso2-amp-platform-resources-extension \
  --version ${VERSION} \
  --namespace ${DEFAULT_NS} \
  --reuse-values \
  --set global.oauth.tokenUrl="${THUNDER_INTERNAL_URL}/oauth2/token" \
  --set environment.gateway.http.host="${AGENTS_DOMAIN}" \
  --set environment.gateway.http.port=80 \
  --set environment.gateway.https.host="${AGENTS_DOMAIN}" \
  --set environment.gateway.https.port=443
Set both listeners, on their own ports

The environment's gateway binding replaces the data plane's entirely. Set both http and https, and give each its own port: 80 for http, 443 for https. A 443 on http makes the console publish http://<host>:443, which browsers block as mixed content.

Later environments. The override above covers only the default environment. Record the same values in Agent Manager's config so that environments created with add-environment.sh resolve the same way:

helm upgrade --server-side=false amp oci://${HELM_CHART_REGISTRY}/wso2-agent-manager \
  --version ${VERSION} \
  --namespace ${AMP_NS} \
  --reuse-values \
  --set agentManagerService.config.agentsBaseDomain="${AGENTS_DOMAIN}" \
  --set-string agentManagerService.config.agentsHttpPort=80 \
  --set-string agentManagerService.config.agentsHttpsPort=443 \
  --set agentManagerService.config.idpHostBaseDomain="${BASE_DOMAIN}" \
  --set agentManagerService.config.tlsEnabled=true \
  --set console.config.idpHostBaseDomain="${BASE_DOMAIN}" \
  --set-string console.config.tlsEnabled=true
Console-generated commands

The Console's generated command for later environments omits the platform Thunder issuer and the CA-trust mode, so on a TLS install the new environment's Thunder trusts the wrong issuer or stops with Platform Thunder CA cert is not available. Add these variables to the generated command before you run it:

  • PLATFORM_THUNDER_ISSUER="${THUNDER_PUBLIC_URL}"
  • SKIP_CA_BUNDLE_TRUST=true when your certificates come from a public CA such as Let's Encrypt, or PLATFORM_THUNDER_CA_PEM="$(cat <ca-file>.pem)" when they come from a private CA.

If the Thunder step fails, the script falls back to the platform Thunder and still reports that setup is complete. Check its output for a warning before you continue. The command is safe to run again with the missing variables.

Publishing added environments' gateways

Each added environment gets its own API Platform Gateway at <env>-<org>.<gatewayBaseDomain>. It stays in-cluster unless you give it a public hostname. To publish it, run the command above with these extra settings in place of the last one. Then add *.otel.${BASE_DOMAIN} DNS and certificate coverage, and keep an explicit record for otel.${BASE_DOMAIN}, because a wildcard makes that name an empty non-terminal (RFC 4592).

helm upgrade --server-side=false amp oci://${HELM_CHART_REGISTRY}/wso2-agent-manager \
  --version ${VERSION} \
  --namespace ${AMP_NS} \
  --reuse-values \
  --set agentManagerService.config.agentsBaseDomain="${AGENTS_DOMAIN}" \
  --set-string agentManagerService.config.agentsHttpPort=80 \
  --set-string agentManagerService.config.agentsHttpsPort=443 \
  --set agentManagerService.config.gatewayBaseDomain="otel.${BASE_DOMAIN}" \
  --set agentManagerService.config.gatewayVhostScheme=https \
  --set agentManagerService.config.gatewayVhostPort=443 \
  --set agentManagerService.config.idpHostBaseDomain="${BASE_DOMAIN}" \
  --set agentManagerService.config.tlsEnabled=true \
  --set console.config.idpHostBaseDomain="${BASE_DOMAIN}" \
  --set-string console.config.tlsEnabled=true

Prepare Storage for the Environment's Thunder​

By default, each Environment's Thunder keeps its data in SQLite on a 1Gi volume, uses an in-memory cache, and runs one replica. That setup is fine for evaluation, but the data lives on one volume behind one pod. For production, point it at your own PostgreSQL, and add Redis if you want more than one replica. The next step reads these settings from environment variables, so set them in the same shell before you run it. Skip this section to keep the defaults.

SettingMeaningDefault
THUNDER_DB_TYPEsqlite or postgressqlite
THUNDER_DB_HOST, THUNDER_DB_USER, THUNDER_DB_PASSWORDDatabase connection details. Required with postgresNone
THUNDER_DB_NAME_PREFIXPrefix of the four database names. Required with postgresNone
THUNDER_DB_PORT, THUNDER_DB_SSLMODEDatabase port and SSL mode5432, require
THUNDER_CACHE_TYPEinmemory or redisinmemory
THUNDER_REDIS_ADDRESSRedis host:port. Required with redisNone
THUNDER_REDIS_USERNAME, THUNDER_REDIS_PASSWORDRedis credentials, if your Redis needs themNone
THUNDER_REDIS_KEY_PREFIXPrefix for every Redis keyThe Thunder release name plus :
THUNDER_REPLICASNumber of Thunder pods1
THUNDER_STATE_ACCESS_MODEAccess mode for the shared certificate and secret volumes. Use ReadWriteMany only when replicas may run on different nodes and the storage class supports itChart default (ReadWriteOnce)
THUNDER_FORCE_STORAGE_CHANGEBypass the database-change guard after you have migrated the data yourselffalse
PERSISTENCE_SIZE, STORAGE_CLASSSize and storage class of the SQLite database volume1Gi, cluster default

The script checks these settings before it creates anything and stops with a message if they do not fit together, for example if you set database details without THUNDER_DB_TYPE=postgres.

  1. Create a role and four databases for this Environment on your PostgreSQL server. Start every database name with a prefix that is unique to the Environment, so that two Environments on one server never share data:

    CREATE ROLE thunder_default LOGIN PASSWORD '...';
    GRANT thunder_default TO CURRENT_USER;  -- required on managed PostgreSQL; harmless elsewhere
    CREATE DATABASE th_default_configdb OWNER thunder_default;
    CREATE DATABASE th_default_entitydb OWNER thunder_default;
    CREATE DATABASE th_default_runtime_transient OWNER thunder_default;
    CREATE DATABASE th_default_runtime_persistent OWNER thunder_default;

    The role must own the databases, or hold SELECT, INSERT, UPDATE, and DELETE on every table in them. The GRANT line matters on managed services such as RDS, Cloud SQL, and Azure Database; Step 4 explains why. The prefix can use up to 41 lowercase letters, digits, and underscores, and must start with a letter.

  2. Load Thunder's schema into each database. The script does not create tables, and an empty database makes the install fail. Take the schema from the same ThunderID version the script installs:

    THUNDER_VERSION="$(grep -m1 -o 'CHART_VERSION:-[0-9][0-9.]*' "${AMP_DIST}/deployments/scripts/add-environment-thunder.sh" | cut -d- -f2)"
    echo "${THUNDER_VERSION}"
    docker create --name thunder-schema "ghcr.io/thunder-id/thunderid:${THUNDER_VERSION}"
    for db in configdb entitydb runtime_transient runtime_persistent; do
      docker cp "thunder-schema:/opt/thunderid/dbscripts/${db}/postgres.sql" "./thunder-${db}.sql"
      psql "postgresql://[email protected]:5432/th_default_${db}" -v ON_ERROR_STOP=1 -f "./thunder-${db}.sql"
    done
    docker rm thunder-schema

    Expect 20 tables in th_default_configdb, 5 in th_default_entitydb, 13 in th_default_runtime_transient, and 7 in th_default_runtime_persistent.

    If the database is on a private network, run psql from a pod in the cluster as shown in Step 4, with PGUSER=thunder_default and the th_default_ database names.

  3. Set the database variables in the shell you will run the next step from:

    export THUNDER_DB_TYPE=postgres
    export THUNDER_DB_HOST=your-db.example.com
    export THUNDER_DB_USER=thunder_default
    export THUNDER_DB_PASSWORD='<password>'
    export THUNDER_DB_NAME_PREFIX=th_default_

    THUNDER_DB_SSLMODE accepts disable, allow, prefer, require (the default), verify-ca, and verify-full. The chart has no setting for a database CA certificate. Use disable only for a database inside the cluster.

    The script stores the password in a Secret and gives it to the chart by reference, so it never appears in the Helm release values.

Add Redis for the cache (optional)​

The in-memory cache is local to each pod, so it only suits one replica. Use Redis when you run more than one Thunder replica. Any Redis that the cluster can reach works, whether a managed service or one you run yourself. Thunder only needs its address:

export THUNDER_CACHE_TYPE=redis
export THUNDER_REDIS_ADDRESS=your-redis.example.com:6379
export THUNDER_REDIS_PASSWORD='<password>'  # only if your Redis requires one

Add THUNDER_REDIS_USERNAME if your Redis uses ACL users. Every key starts with THUNDER_REDIS_KEY_PREFIX, which defaults to the Thunder release name followed by : (for example amp-thunder-default-default:), so several Environments can share one Redis. The chart has no TLS setting for Redis, so keep it on a private network.

Run more than one replica (optional)​

export THUNDER_REPLICAS=2

More than one replica needs external PostgreSQL and Redis. The script refuses any other combination before it creates anything. All replicas mount the same certificate and secret volumes, which are ReadWriteOnce by default. If the replicas can land on different nodes, also set THUNDER_STATE_ACCESS_MODE=ReadWriteMany, which needs a storage class that supports it. The volumes are created once, so changing this setting later has no effect.

Choose the database before the first run

Thunder seeds its database once, at first install, and helm upgrade never repeats it. The script therefore refuses to point an installed Environment at a different database, and refuses to retry in place a first install that failed before its setup job completed. If the first run fails that way (the script's message says so), remove the Environment's Thunder with bash "${AMP_DIST}/deployments/scripts/remove-environment-thunder.sh" (use the same ENV_NAME, AMP_API_URL, and IDP_* values as the provisioning command), fix the cause, and run the script again. If the first install failed after the setup job had finished, the database is already seeded and running the script again is accepted.

Removing an Environment does not drop its PostgreSQL databases. To add the Environment again, drop and recreate its databases (and reload the schema), or use a new THUNDER_DB_NAME_PREFIX.

Running the script again applies exactly the settings you pass, and a setting you leave out goes back to its default. Pass the same values every time, or the Environment drops, for example, from PostgreSQL with Redis and three replicas to the in-memory cache and one replica. When a run would change the cache type or the replica count of an installed Environment, the script prints the change before it applies it. It does not refuse, because scaling or switching the cache loses no data.

Environments you add later from the Console use the defaults. To give one its own PostgreSQL or Redis, add the same variables in front of bash in the command the Console prints, as shown in Manage Environments.

Provision Thunder Identity Provider for the Default Environment​

Each environment needs its own Thunder instance. It issues each agent its OAuth2 credential (AgentID). Run this once for the default environment after Agent Manager is reachable at API_PUBLIC_URL. Without it, agents are created but never get an AgentID, and Agent Manager keeps retrying in the background.

If you prepared PostgreSQL or Redis above, keep those variables set in this shell. The command picks them up. Without them, the script uses SQLite, an in-memory cache, and one replica.

Choose the tab that matches your Step 3 issuer:

Pass the platform client secret

IDP_CLIENT_SECRET must be ${AMP_API_CLIENT_SECRET}. Its default is the shipped secret, so the script fails with Could not obtain an access token on this install. If it fails, rerun it once the credentials are correct.

Platform Thunder's certificate is publicly trusted, so no custom CA is needed:

ENV_NAME=default \
DISPLAY_NAME="Default" \
ORG_NAME=default \
THUNDER_HANDLE=default-idp \
WAIT_TIMEOUT=300s \
AMP_API_URL="${API_PUBLIC_URL}/api/v1" \
IDP_TOKEN_URL="${THUNDER_PUBLIC_URL}/oauth2/token" \
IDP_CLIENT_ID=amp-api-client \
IDP_CLIENT_SECRET="${AMP_API_CLIENT_SECRET}" \
PLATFORM_THUNDER_ISSUER="${THUNDER_PUBLIC_URL}" \
PLATFORM_THUNDER_JWKS_URL="${THUNDER_PUBLIC_URL}/oauth2/jwks" \
THUNDER_HOST_BASE_DOMAIN="${BASE_DOMAIN}" \
TLS_ENABLED=true \
SKIP_CA_BUNDLE_TRUST=true \
bash "${AMP_DIST}/deployments/scripts/add-environment-thunder.sh"

The script prints the issuer as https://<handle>.${BASE_DOMAIN}. THUNDER_HANDLE is optional (3 to 63 characters). Omit it to have Agent Manager generate one.

Verify
kubectl get pods -n amp-thunder-default-default
kubectl get secret amp-thunder-default-default-admin-credentials \
  -n amp-thunder-default-default -o jsonpath='{.data.password}' | base64 -d
Verify the storage you chose

The script prints a Storage: line at the start and again at the end. With PostgreSQL it reads db=postgres, and the Environment's Thunder has no database volume:

kubectl get pvc -n amp-thunder-default-default
# With external PostgreSQL there is no "...-database-pvc", only the certs and secrets volumes

psql "postgresql://[email protected]:5432/th_default_configdb" \
  -tAc 'select count(*) from "INBOUND_CLIENT"'
# 1 or more: Thunder seeded its own database

The script reuses the system-client secret and admin password on re-runs. It never rotates them.

Point the Gateway at the Environment's Thunder​

Register Thunder as a gateway key manager and as an identity provider in the console:

export ENV_THUNDER_RELEASE="amp-thunder-default-default"
export ENV_THUNDER_ISSUER="<the Issuer URL printed by add-environment-thunder.sh above>"
export ENV_THUNDER_JWKS="http://${ENV_THUNDER_RELEASE}-service.${ENV_THUNDER_RELEASE}.svc.cluster.local:8090/oauth2/jwks"

helm upgrade --server-side=false api-platform-default-default \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-api-platform-gateway-extension \
  --version ${VERSION} \
  --namespace ${DATA_PLANE_NS} \
  --reuse-values \
  --set "apiGateway.config.policyConfigurations.jwtauth_v1.keymanagers[0].name=agent-manager-service" \
  --set "apiGateway.config.policyConfigurations.jwtauth_v1.keymanagers[0].issuer=agent-manager-service" \
  --set "apiGateway.config.policyConfigurations.jwtauth_v1.keymanagers[0].jwks.remote.uri=http://amp-api.${AMP_NS}.svc.cluster.local:9000/auth/external/jwks.json" \
  --set "apiGateway.config.policyConfigurations.jwtauth_v1.keymanagers[0].jwks.remote.skipTlsVerify=true" \
  --set "apiGateway.config.policyConfigurations.jwtauth_v1.keymanagers[1].name=ThunderKeyManager" \
  --set "apiGateway.config.policyConfigurations.jwtauth_v1.keymanagers[1].issuer=${ENV_THUNDER_ISSUER}" \
  --set "apiGateway.config.policyConfigurations.jwtauth_v1.keymanagers[1].jwks.remote.uri=${ENV_THUNDER_JWKS}" \
  --set "apiGateway.config.policyConfigurations.jwtauth_v1.keymanagers[1].jwks.remote.skipTlsVerify=false" \
  --set "bootstrap.identityProviders[0].name=ThunderKeyManager" \
  --set "bootstrap.identityProviders[0].issuer=${ENV_THUNDER_ISSUER}" \
  --set "bootstrap.identityProviders[0].jwksUri=${ENV_THUNDER_JWKS}" \
  --set "bootstrap.identityProviders[0].skipTlsVerify=false" \
  --timeout 900s

Set both key managers. --set on an indexed array replaces the whole list, so re-stating keymanagers[0] keeps the internal entry that API-key authentication uses.

Without this, agents reject OAuth tokens

The gateway then has only its internal key manager. API keys still work, so the platform looks complete, but no agent endpoint validates a Thunder-issued token.

The controller restarts

The gateway controller's volume is ReadWriteOnce. On a multi-node cluster the new pod can block with Multi-Attach error. Delete the old controller pod to release the volume:

kubectl get pods -n ${DATA_PLANE_NS} | grep gateway-controller
kubectl delete pod <old-gateway-controller-pod> -n ${DATA_PLANE_NS}

Verify and Access the Platform​

kubectl get pods -n openchoreo-control-plane
kubectl get pods -n openchoreo-data-plane
kubectl get pods -n openchoreo-workflow-plane
kubectl get pods -n openchoreo-observability-plane
kubectl get pods -n wso2-amp
kubectl get pods -n amp-thunder
helm list -A | grep -E 'openchoreo|amp|gateway'

Everything routes through the three plane gateways over TLS:

ServiceURL
Consolehttps://${CONSOLE_PUBLIC_HOST}
APIhttps://${API_PUBLIC_HOST}
Thunderhttps://${THUNDER_PUBLIC_HOST}
Observerhttps://${OBS_API_PUBLIC_HOST}
Gateway control planehttps://${CP_GW_PUBLIC_HOST}

Log in as admin with the password from amp-admin-credentials, then change it in the profile menu. Deployed agents' invoke URLs are published under https://<org>-<project>.${AGENTS_DOMAIN}.

For debugging, port-forward any ClusterIP service, for example kubectl port-forward -n wso2-amp svc/amp-console 3000:3000.

Cloud Provider Notes​

Pick your platform. The Generic tab applies to any cluster.

  • LoadBalancers may return an IP address or a hostname. Resolve whichever you get before creating DNS records.
  • Allow HTTP and HTTPS through your cloud firewall or security groups to the plane gateway LoadBalancers.
  • Confirm the default StorageClass provisions volumes. See Cluster Requirements.
Red Hat OpenShift (including ARO)
  • OpenShift's Security Context Constraints, built-in monitoring and DNS need extra settings at several steps — follow Appendix: Red Hat OpenShift alongside the main flow
  • On ARO, LoadBalancers return public IPs directly and managed-csi is the default StorageClass

Before You Go Live​

Work through the production recommendations before you expose the platform. They cover the checklist, identity and access, gateway hardening, build limits, scaling, and cluster security.

Appendix: Installing Without a Domain (nip.io)​

For evaluation only. Browsers warn on every hostname, and the certificates are self-signed.

Differences from the main flow

1. Self-signed issuer. In Step 3, create a self-signed chain under the openchoreo-ca name:

kubectl apply -f - <<'EOF'
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned-bootstrap
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: openchoreo-ca
  namespace: cert-manager
spec:
  isCA: true
  commonName: openchoreo-ca
  secretName: openchoreo-ca-secret
  privateKey:
    algorithm: ECDSA
    size: 256
  issuerRef:
    name: selfsigned-bootstrap
    kind: ClusterIssuer
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: openchoreo-ca
spec:
  ca:
    secretName: openchoreo-ca-secret
EOF

The Control Plane install in Step 5 also needs --set-string openchoreoApi.config.security.authentication.jwt.jwks.skip_tls_verify=true, and the Observability install in Step 8 needs --set-string security.oidc.jwksUrlTlsInsecureSkipVerify=true. Both are needed because Thunder's JWKS cannot be verified against an untrusted chain.

2. Hostnames come from the first LoadBalancer. Install the Control Plane once with placeholder hostnames. Then derive the base domain:

CP_LB_IP=$(kubectl get svc gateway-default -n openchoreo-control-plane \
  -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
export BASE_DOMAIN="${CP_LB_IP//./-}.nip.io"

Export the variables from Plan Your Deployment with this base domain, then rerun the Step 5 install as helm upgrade.

3. Per-gateway domains. All hostnames resolve to the control-plane IP, but traces. and *.agents. must reach the other gateways. Derive OBS_API_PUBLIC_HOST and AGENTS_DOMAIN from those gateways' LoadBalancer IPs, for example traces.<ip-with-dashes>.nip.io.

4. Skip Step 10. nip.io resolves automatically.

5. Trust the self-signed CA. SKIP_CA_BUNDLE_TRUST and the script's auto-detect do not match this CA. Use the Corporate CA steps in Provision Thunder with PLATFORM_THUNDER_CA_PEM extracted the same way. The openchoreo-ca-secret name is the one the script can't find.

If you recreate the cluster, the LoadBalancer IP changes, so every hostname changes too. Reinstall Thunder with the new values.

Appendix: Red Hat OpenShift​

Agent Manager runs on Red Hat OpenShift, including managed offerings such as Azure Red Hat OpenShift (ARO). The flow is the same as the main guide; OpenShift needs extra settings at a few steps, mostly because of its stricter pod security defaults, its built-in platform components (monitoring, DNS, image registry), and the Kubernetes version a given OpenShift release ships.

This appendix was validated on ARO, OpenShift 4.18 (Kubernetes 1.31).

OpenShift-specific setup and changes

Each change is tagged with where it applies:

TagApplies to
All OpenShiftEvery OpenShift cluster, any version or provider
OpenShift 4.18 (Kubernetes 1.31)OpenShift releases that ship Kubernetes 1.31 or earlier, which is below the guide's 1.32 minimum
AROAzure Red Hat OpenShift only
Private registryClusters that push agent images to a registry requiring credentials (for example Azure Container Registry)

Use oc or kubectl interchangeably for the commands below; the SCC commands need oc.

How OpenShift security affects the install​

All OpenShift. OpenShift admits pods through Security Context Constraints (SCCs). The default, restricted-v2, runs every pod as a random, namespace-assigned user ID and rejects pods that request a specific user ID, root, host access, or privilege. Most charts in this guide pin fixed non-root user IDs, and a few components need root or privilege, so you grant SCCs per namespace or service account before installing the charts that need them. Three built-in SCCs are used:

SCCAllowsUsed for
nonroot-v2Any fixed user ID except root (0)Most platform namespaces
anyuidAny user ID, including rootTwo setup jobs that run as root
privilegedRoot, privileged containers, host paths, host networkAgent build steps, the Fluent Bit log collector

Images that expect to run as their own user (rather than OpenShift's random one) are pinned to that user through chart values in the steps below.

Prerequisites​

  • Kernel (All OpenShift). RHCOS nodes run a 5.14 kernel, below the 6.3 needed for user-namespaced builds. Set buildWorkflows.userNamespaces=false on the Platform Resources chart (Phase 2, Step 4).

  • Kubernetes version (OpenShift 4.18 (Kubernetes 1.31)). The OpenChoreo control-plane CRDs need a patch before installing; see Step 5 under Phase 1 changes below.

  • LoadBalancer check (All OpenShift). The nginx image in the LoadBalancer test runs as root and is rejected by restricted-v2. Use the unprivileged image instead:

    kubectl create deploy lbtest --image=nginxinc/nginx-unprivileged:alpine --port=8080
    kubectl expose deploy lbtest --port=80 --target-port=8080 --type=LoadBalancer

  • NetworkPolicy (All OpenShift). OpenShift's default network plugin (OVN-Kubernetes) enforces NetworkPolicy, so the platform's policies take effect. Its DNS service listens on a different port from upstream Kubernetes, which two of those policies need adjusting for (Phase 2, Steps 4 and 6).

  • Default StorageClass (ARO). ARO ships managed-csi (Azure Disk) as the default class, which works as-is. Azure disks are zonal: a pod with persistent volumes can only run on a worker in its volume's zone, so keep at least one worker per zone.

  • Node count (ARO). ARO requires 3 control-plane nodes and at least 3 workers, and does not schedule workloads on control-plane nodes. All platform components run on the workers.

SCC grants​

All OpenShift. Grant these before the corresponding steps. Some namespaces do not exist until a later step creates them; oc adm policy refuses a namespace that does not exist yet, so those are granted with a ClusterRoleBinding instead, which takes effect when the namespace is created.

for ns in openchoreo-control-plane openchoreo-data-plane openchoreo-workflow-plane openchoreo-observability-plane amp-thunder wso2-amp; do
  kubectl create namespace "$ns" --dry-run=client -o yaml | kubectl apply -f -
  oc adm policy add-scc-to-group nonroot-v2 "system:serviceaccounts:$ns"
done

oc create clusterrolebinding agent-sandbox-system-nonroot-v2 \
  --clusterrole=system:openshift:scc:nonroot-v2 \
  --group=system:serviceaccounts:agent-sandbox-system

oc create clusterrolebinding amp-thunder-default-default-nonroot-v2 \
  --clusterrole=system:openshift:scc:nonroot-v2 \
  --group=system:serviceaccounts:amp-thunder-default-default

oc adm policy add-scc-to-user anyuid -z amp-hooks -n wso2-amp
oc adm policy add-scc-to-user anyuid -z api-platform-default-default-bootstrap-sa -n openchoreo-data-plane
oc adm policy add-scc-to-user privileged -z fluent-bit -n openchoreo-observability-plane

oc create clusterrolebinding workflows-default-workflow-sa-privileged \
  --clusterrole=system:openshift:scc:privileged \
  --serviceaccount=workflows-default:workflow-sa
Namespace / service accountSCCWhy
The four OpenChoreo plane namespaces, amp-thunder, wso2-ampnonroot-v2Pods pin fixed non-root user IDs (1000, 1001, 10001, 65532 and others)
agent-sandbox-systemnonroot-v2Sandbox controller runs as user ID 1000
amp-thunder-<org>-<env>nonroot-v2Per-environment Thunder runs as user ID 10001. Repeat for every environment you add
wso2-amp/amp-hooksanyuidThe JWT key generation job runs as root
openchoreo-data-plane/api-platform-default-default-bootstrap-saanyuidThe gateway bootstrap job runs as root
openchoreo-observability-plane/fluent-bitprivilegedReads node log files through host paths
workflows-default/workflow-saprivilegedAgent build steps run privileged podman containers. Repeat for workflows-<env> of every environment you add

Phase 1 changes​

Step 2: OpenBao​

All OpenShift. The OpenBao chart pins user ID 100 by default. Add --set global.openshift=true to the OpenBao helm upgrade --install command; the chart then leaves the user ID to OpenShift and needs no SCC grant.

Step 5: OpenChoreo Control Plane​

OpenShift 4.18 (Kubernetes 1.31). The control-plane chart's CRDs fail to install with CustomResourceDefinition ... is invalid ... undefined field 'namespace'. Some of their validation rules refer to fields named namespace and var, which Kubernetes 1.31 and earlier treat as reserved words and only accept in the escaped form __namespace__ / __var__. The escaped form is also valid on newer Kubernetes.

Apply patched CRDs yourself, then install the chart with --skip-crds:

helm pull oci://ghcr.io/openchoreo/helm-charts/openchoreo-control-plane --version 1.2.8 --untar -d oc-cp-chart

sed -i.bak -E 's/has\(self\.var\)/has(self.__var__)/g; s/self\.scope\.namespace\)/self.scope.__namespace__)/g' \
  oc-cp-chart/openchoreo-control-plane/crds/*.yaml
rm -f oc-cp-chart/openchoreo-control-plane/crds/*.bak

kubectl apply --server-side -f oc-cp-chart/openchoreo-control-plane/crds/

Then add --skip-crds to every control-plane helm upgrade --install in this guide. Helm never upgrades CRDs, so on a later control-plane chart upgrade repeat these commands with the new chart version.

Step 8: Observability Plane​

All OpenShift. Three changes to the observability modules:

  1. OpenSearch. Its init container changes the data volume's owner as root, which nonroot-v2 does not allow. The pod's fsGroup already makes the volume writable, so turn the init container off. Add to the logs module install:

    --set openSearch.persistence.enableInitChown=false

  2. Fluent Bit. OpenShift's container runtime writes log files readable only by root, with an SELinux label unprivileged containers cannot read. The privileged SCC grant only permits privilege; the container must also request it. Add to the command that enables Fluent Bit:

    --set fluent-bit.securityContext.runAsUser=0 \
    --set fluent-bit.securityContext.runAsNonRoot=false \
    --set fluent-bit.securityContext.privileged=true \
    --set fluent-bit.securityContext.allowPrivilegeEscalation=true

    Without this, every log query in the console stays empty while all pods look healthy.

  3. Metrics module. OpenShift ships its own Prometheus Operator and kube-state-metrics, whose cluster roles use the same names as this chart's, so the install fails with ClusterRole "kube-state-metrics" ... exists and cannot be imported into the current release. The chart's operator would also manage OpenShift's own Prometheus and Alertmanager resources. Rename the chart's components, keep its operator out of OpenShift's monitoring namespaces, and keep Prometheus off a public LoadBalancer. Add to the metrics module install:

    --set kube-prometheus-stack.prometheusOperator.fullnameOverride=openchoreo-prometheus-operator \
    --set kube-prometheus-stack.kube-state-metrics.fullnameOverride=openchoreo-kube-state-metrics \
    --set "kube-prometheus-stack.prometheusOperator.denyNamespaces={openshift-monitoring,openshift-user-workload-monitoring}" \
    --set kube-prometheus-stack.prometheus.service.type=ClusterIP

    OpenShift owns the monitoring.coreos.com CRDs, so the chart's operator runs against OpenShift's versions; a warning such as unknown field "spec.enableOTLPReceiver" during install is expected and harmless.

Phase 2 changes​

Step 1: Gateway Operator​

All OpenShift. The gateway runtime image keeps its working directories owned by user ID 10001 and crash-loops (Failed to bind to address unix:/var/run/api-platform/python-executor.sock ... Permission denied) under OpenShift's random user ID. Pin it through the operator's gateway values. Add to the gateway operator install:

--set gateway.values.gateway.gatewayRuntime.deployment.podSecurityContext.runAsUser=10001 \
--set gateway.values.gateway.gatewayRuntime.deployment.podSecurityContext.runAsGroup=10001 \
--set gateway.values.gateway.gatewayRuntime.deployment.podSecurityContext.fsGroup=10001 \
--set gateway.values.gateway.gatewayRuntime.deployment.podSecurityContext.runAsNonRoot=true

Set these on the initial install. If the gateway extension (Step 7) is installed while the runtime is crash-looping, the operator records the gateway as permanently failed and does not retry on its own.

Step 2: Agent Manager​

All OpenShift. Two components need their user ID pinned. Add to the Agent Manager install:

--set agentManagerService.podSecurityContext.runAsUser=1000 \
--set agentManagerService.podSecurityContext.runAsNonRoot=true \
--set agentManagerService.podSecurityContext.fsGroup=1000 \
--set console.securityContext.runAsUser=1001 \
--set console.securityContext.runAsNonRoot=true \
--set console.securityContext.allowPrivilegeEscalation=false \
--set "console.securityContext.capabilities.drop={ALL}"
  • The API pod's wait-for-db init container (used with the in-cluster database) runs an image that defaults to root and has no user ID of its own, so the pod-level user ID applies to it.
  • The console's web server writes its runtime configuration into a directory owned by user ID 1001.

Step 4: Platform Resources​

All OpenShift. Add --set buildWorkflows.userNamespaces=false to the Platform Resources install (RHCOS kernel 5.14).

After installing, apply two patches the chart has no values for:

  1. Build checkout step. It is the only non-privileged build step, so it runs under a random user ID with no home directory and git fails with could not lock config file //.gitconfig. Give it a writable HOME:

    kubectl patch clusterworkflowtemplate checkout-source --type=json \
      -p '[{"op":"add","path":"/spec/templates/0/container/env/-","value":{"name":"HOME","value":"/tmp"}}]'

  2. Agent DNS egress. OpenShift's DNS (openshift-dns) listens on port 5353 behind a Service on port 53. NetworkPolicies match the pod's real port, and the agent egress policy only allows 53, so agents cannot resolve any hostname (Temporary failure in name resolution). Allow 5353 as well:

    kubectl patch componenttype.openchoreo.dev agent-api -n default --type=json \
      -p '[{"op":"add","path":"/spec/resources/0/template/spec/networkPolicy/egress/0/ports/-","value":{"port":5353,"protocol":"UDP"}},{"op":"add","path":"/spec/resources/0/template/spec/networkPolicy/egress/0/ports/-","value":{"port":5353,"protocol":"TCP"}}]'

    Agents deployed before this patch pick it up on their next deploy.

Private registry. Agent component types do not set imagePullSecrets, so agent pods cannot authenticate to a private registry themselves. On OpenShift, add the registry's credentials to the cluster-wide pull secret, which every node uses for every image:

oc get secret/pull-secret -n openshift-config -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d > cluster-pull-secret.json
jq -s '.[0] * .[1]' cluster-pull-secret.json registry-dockerconfig.json > cluster-pull-secret.merged.json
oc set data secret/pull-secret -n openshift-config --from-file=.dockerconfigjson=cluster-pull-secret.merged.json

registry-dockerconfig.json is your registry's .dockerconfigjson (for ACR, built from the registry's admin credentials). Keep every existing entry; on ARO the arosvc.azurecr.io entry must not be removed. OpenShift rolls the change out to all nodes without a reboot.

Step 6: Evaluation Extension​

All OpenShift. The evaluation job's egress policy expects DNS in kube-system on port 53. Point it at OpenShift's DNS through chart values, adding to the evaluation extension install:

--set networkPolicy.evaluationJob.dns.namespace=openshift-dns \
--set networkPolicy.evaluationJob.dns.podLabel.key=dns.operator.openshift.io/daemonset-dns \
--set networkPolicy.evaluationJob.dns.podLabel.value=default

The DNS port is fixed at 53 in the chart, so add 5353 right after installing (the chart creates workflows-default and the policy itself):

kubectl patch networkpolicy amp-evaluation-job-egress -n workflows-default --type=json \
  -p '[{"op":"add","path":"/spec/egress/0/ports/-","value":{"port":5353,"protocol":"UDP"}},{"op":"add","path":"/spec/egress/0/ports/-","value":{"port":5353,"protocol":"TCP"}}]'

After the first agent build​

All OpenShift. OpenShift adds an image-registry pull secret (workflow-sa-dockercfg-...) as the first entry in every service account's secrets list. Evaluation monitor workflows mount the service account's first secret as their API token, so they fail at start-up with invalid configuration: no configuration has been provided. Put the token Secret first:

DOCKERCFG=$(kubectl get sa workflow-sa -n workflows-default \
  -o jsonpath='{.secrets[?(@.name!="workflow-sa.service-account-token")].name}')
kubectl patch sa workflow-sa -n workflows-default --type=merge \
  -p "{\"secrets\":[{\"name\":\"workflow-sa.service-account-token\"},{\"name\":\"${DOCKERCFG}\"}]}"

Repeat for workflows-<env> of every environment you add.

Provision Thunder Identity Provider​

ARO. Thunder's volumes are created in a single availability zone, and attaching them can take a few minutes on Azure. Pass WAIT_TIMEOUT=600s to add-environment-thunder.sh so the script does not stop waiting before Thunder is ready. Remember the amp-thunder-<org>-<env> SCC grant for every environment you add.

After a chart upgrade​

Some OpenShift changes are applied with kubectl on top of what Helm renders, and an upgrade of the owning chart reverts them. Re-apply after upgrading:

ChangeOwning chartApplies to
Patched control-plane CRDs, install with --skip-crdsOpenChoreo Control PlaneOpenShift 4.18 (Kubernetes 1.31)
checkout-source HOME=/tmpPlatform ResourcesAll OpenShift
agent-api DNS port 5353Platform ResourcesAll OpenShift
amp-evaluation-job-egress DNS port 5353Evaluation ExtensionAll OpenShift

The changes made through --set values persist across upgrades that use --reuse-values.

Appendix: Rancher Desktop / k3s​

For evaluation only. Single-node k3s works, but it may not have enough resources for every observability module.

k3s ships Traefik on ports 80 and 443, which conflicts with kgateway. Remove it before Step 1:

helm uninstall traefik -n kube-system
helm uninstall traefik-crd -n kube-system

Traefik's CRD removal can also remove the Gateway API CRDs. Reapply them:

kubectl apply --server-side --force-conflicts \
  -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.1/experimental-install.yaml

Cleanup​

# 1. Delete plane registrations
kubectl delete clusterdataplane default -n default
kubectl delete clusterworkflowplane default -n default
kubectl delete clusterobservabilityplane default

# 2. Uninstall all Helm releases
helm uninstall amp -n wso2-amp
helm uninstall api-platform-default-default -n openchoreo-data-plane
helm uninstall amp-thunder-extension -n amp-thunder
helm uninstall amp-observability-traces -n openchoreo-observability-plane
helm uninstall amp-evaluation-extension -n openchoreo-workflow-plane
helm uninstall amp-platform-resources -n default
helm uninstall gateway-operator -n openchoreo-data-plane
helm uninstall openchoreo-observability-plane -n openchoreo-observability-plane
helm uninstall openchoreo-workflow-plane -n openchoreo-workflow-plane
helm uninstall openchoreo-data-plane -n openchoreo-data-plane
helm uninstall openchoreo-control-plane -n openchoreo-control-plane
helm uninstall openbao -n openbao
helm uninstall external-secrets -n external-secrets
helm uninstall cert-manager -n cert-manager

# 3. Delete namespaces
kubectl delete namespace wso2-amp amp-thunder \
  openchoreo-observability-plane openchoreo-workflow-plane \
  openchoreo-data-plane openchoreo-control-plane \
  openbao external-secrets cert-manager

If an Environment's Thunder uses your own PostgreSQL, uninstalling does not drop its databases. Drop <prefix>configdb, <prefix>entitydb, <prefix>runtime_transient, and <prefix>runtime_persistent yourself once you no longer need the data, where <prefix> is the THUNDER_DB_NAME_PREFIX you provisioned the Environment with.

Troubleshooting​

Environment's Thunder: "already installed with sqlite; this run asks for postgres"

The script refuses to point an installed Environment at a different database. Thunder seeds its database once, at first install, so a different database would be empty: no OAuth clients and no logins.

To move the Environment, remove its Thunder with remove-environment-thunder.sh and provision it again with the new settings. If you have already migrated the data yourself, set THUNDER_FORCE_STORAGE_CHANGE=true.

Related messages:

  • ... is a failed first install: the chart's setup job runs only on first install, and the script cannot confirm that it finished. If the job is still running (kubectl get job <release>-setup -n <namespace>), wait for it and run the script again. Otherwise run remove-environment-thunder.sh, fix the cause, and run the script again.
  • ... runs with cache=redis, replicas=3; this run sets cache=inmemory, replicas=1: a run applies exactly the settings it is given, and the ones you left out went back to their defaults. If that was not intended, run the script again with the same THUNDER_CACHE_TYPE and THUNDER_REPLICAS as the first run.
  • THUNDER_DB_HOST is set but THUNDER_DB_TYPE is 'sqlite': add THUNDER_DB_TYPE=postgres, or unset the database variables.
Environment's Thunder does not start on PostgreSQL

Check these in order:

  • The schema is not loaded. The setup job fails and its pod is deleted, so the logs show nothing. Step 4 shows the underlying error. Load the schema as described in Prepare Storage.

  • The database names do not match your prefix. The script prints the four names it expects at the start of the run.

  • The database is not reachable from the cluster, or the password is wrong. Read the Thunder pod's log:

    kubectl logs -n amp-thunder-<org>-<env> deploy/amp-thunder-<org>-<env>-deployment

With PostgreSQL or Redis selected, the script stops with Thunder did not become ready in time instead of printing the console details. Run remove-environment-thunder.sh for the Environment before you try again.

LoadBalancer has no external address
kubectl describe svc <service-name> -n <namespace>
kubectl get svc -A --field-selector spec.type=LoadBalancer

On EKS, confirm the AWS Load Balancer Controller is installed. On k3s, check whether Traefik holds the ports.

Certificate not issued
kubectl describe certificate <cert-name> -n <namespace>
kubectl get clusterissuers
kubectl get certificaterequests -n <namespace>
Plane registration fails
kubectl get clusterdataplane default -n default -o yaml
kubectl logs -n openchoreo-control-plane -l app.kubernetes.io/name=openchoreo-control-plane
API returns 401 for environment or gateway calls

The Control Plane's OIDC issuer does not match the iss claim in Thunder tokens. Compare them:

kubectl exec -n amp-thunder deploy/amp-thunder-extension-deployment -- \
  wget -qO- http://localhost:8090/.well-known/openid-configuration 2>/dev/null \
  | grep -o '"issuer":"[^"]*"'

kubectl get configmap openchoreo-api-config -n openchoreo-control-plane -o yaml | grep issuer

Both must match exactly. If they differ, set the Control Plane's security.oidc.issuer to Thunder's issuer.

Console shows "refused to connect" on login
dig +short thunder.${BASE_DOMAIN}
kubectl get httproute -n openchoreo-control-plane amp-thunder-extension
curl -s -o /dev/null -w '%{http_code}\n' https://thunder.${BASE_DOMAIN}/.well-known/openid-configuration

To change Thunder's public URL after install, uninstall, delete the PVC, and reinstall:

helm uninstall amp-thunder-extension -n amp-thunder
kubectl delete pvc -n amp-thunder --all
Evaluations fail with connection errors

The evaluation-job NetworkPolicy allows DNS egress only to kube-system with the k8s-app=kube-dns label. If your DNS runs elsewhere, such as NodeLocal DNSCache, every egress rule fails, because nothing resolves. Set networkPolicy.evaluationJob.dns.namespace and podLabel to match your DNS workload.

OpenSearch connectivity
kubectl get pods -n openchoreo-observability-plane -l app=opensearch
kubectl run -it --rm debug --image=curlimages/curl --restart=Never -- \
  curl -v http://opensearch.openchoreo-observability-plane.svc.cluster.local:9200
Build pods fail with MOUNT_ATTR_IDMAP or idmap errors

Build pods request a Linux user namespace (hostUsers: false). That needs idmapped-mount support on every mounted filesystem, and tmpfs gained it in Linux 6.3. Ubuntu 22.04 (kernel 5.15) and Amazon Linux 2 (kernel 5.10) do not have it. Errors look like this:

failed to create containerd container: snapshotter "overlayfs" doesn't support idmap mounts
OCI runtime create failed: ... failed to set MOUNT_ATTR_IDMAP ... invalid argument

Use one of these fixes:

  1. Preferred: use build nodes with kernel 6.3 or newer.
  2. Disable user namespaces with the Platform Resources chart value:
helm upgrade --server-side=false amp-platform-resources \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-platform-resources-extension \
  --version ${VERSION} \
  --namespace ${DEFAULT_NS} \
  --reuse-values \
  --set buildWorkflows.userNamespaces=false

Changing containerd's slow_chown alone does not fix this. The build still fails on the tmpfs mount.

Agent deploys fail with "Secret does not exist" after an OpenBao restart

The guide's OpenBao dev-mode install keeps secrets in memory. Restarting openbao-0 wipes them. Running agents keep working, but new deployments fail with SecretSyncedError. Re-save the agent's environment variables in the console. For production, use the persistent install from Step 2.

Build fails with a cgroup pids error (Rancher Desktop)
Error: OCI runtime error: crun: the requested cgroup controller `pids` is not available

Rancher Desktop's Lima VM does not delegate the pids controller to containers. Patch the three Podman-based templates to disable cgroup management. The patches require python3 on your machine:

# gcp-buildpacks-build
kubectl get clusterworkflowtemplate gcp-buildpacks-build -o json | \
  python3 -c "
import json, sys
data = json.load(sys.stdin)
script = data['spec']['templates'][0]['container']['args'][0]
fix = '''set -e

# Fix: disable cgroup management for Podman (Rancher Desktop cgroup pids workaround)
cat > /tmp/containers.conf <<CCONF
[engine]
cgroup_manager = \"cgroupfs\"
events_logger = \"file\"
[containers]
pids_limit = 0
CCONF
export CONTAINERS_CONF=/tmp/containers.conf

'''
data['spec']['templates'][0]['container']['args'][0] = script.replace('set -e\n', fix, 1)
json.dump(data, sys.stdout)
" | kubectl apply -f -

# publish-image
kubectl get clusterworkflowtemplate publish-image -o json | \
  python3 -c "
import json, sys
data = json.load(sys.stdin)
script = data['spec']['templates'][0]['container']['args'][0]
fix = '''set -e

# Fix: disable cgroup management for Podman (Rancher Desktop cgroup pids workaround)
cat > /tmp/containers.conf <<CCONF
[engine]
cgroup_manager = \"cgroupfs\"
events_logger = \"file\"
[containers]
pids_limit = 0
CCONF
export CONTAINERS_CONF=/tmp/containers.conf

'''
data['spec']['templates'][0]['container']['args'][0] = script.replace('set -e\n', fix, 1)
json.dump(data, sys.stdout)
" | kubectl apply -f -

# amp-generate-workload
kubectl get clusterworkflowtemplate amp-generate-workload -o json | \
  python3 -c "
import json, sys
data = json.load(sys.stdin)
script = data['spec']['templates'][0]['container']['args'][0]
fix = '''# Fix: disable cgroup management for Podman (Rancher Desktop cgroup pids workaround)
cat > /tmp/containers.conf <<CCONF
[engine]
cgroup_manager = \"cgroupfs\"
events_logger = \"file\"
[containers]
pids_limit = 0
CCONF
export CONTAINERS_CONF=/tmp/containers.conf

'''
data['spec']['templates'][0]['container']['args'][0] = fix + script
json.dump(data, sys.stdout)
" | kubectl apply -f -

Re-trigger the build. These patches are in-cluster, so reinstalling amp-platform-resources overwrites them. This affects Rancher Desktop only. EKS, GKE, and AKS are not affected.

Agents cannot resolve hostnames on EKS (Try It returns 504)

On EKS with the VPC CNI's network policy support on, a deployed agent cannot reach DNS or its Environment's Thunder. Try It returns 504 upstream request timeout, and the agent's log shows Temporary failure in name resolution.

The agent's egress policy allows DNS (port 53) and Thunder (port 8090) to any address, and allows ports 80 and 443 to the internet but not to private address ranges. The VPC CNI's policy agent applies that private-range exception to the DNS and Thunder rules as well. Cluster pods have private VPC addresses, so DNS and Thunder traffic is dropped. Other network plugins, such as Calico and Cilium, are not affected.

Allow DNS and Thunder explicitly in the agent's namespace. Agent namespaces start with dp-; list them with kubectl get ns | grep '^dp-'. Set the namespace and the Thunder namespace of the agent's Environment, amp-thunder-<org>-<env>:

export AGENT_NS="<agent-namespace>"
export ENV_THUNDER_NS="amp-thunder-<org>-<env>"
kubectl apply -n "${AGENT_NS}" -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-agent-dns-and-thunder
spec:
  podSelector:
    matchExpressions:
    - key: agents.x-k8s.io/sandbox-template-ref-hash
      operator: Exists
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
      podSelector:
        matchLabels:
          k8s-app: kube-dns
    ports:
    - port: 53
      protocol: UDP
    - port: 53
      protocol: TCP
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: ${ENV_THUNDER_NS}
    ports:
    - port: 8090
      protocol: TCP
EOF

Naming the DNS pods and the Thunder namespace makes the policy agent allow those addresses ahead of the private-range exception. Agents recover within a few seconds, with no redeploy. Each project and Environment has its own agent namespace, so apply this to each new one.

Set up a build registry on EKS (Amazon ECR cannot back builds)

Amazon ECR cannot back build workflows. If you have no other registry, run CNCF Distribution in the cluster behind an internal LoadBalancer. It creates repositories on push and needs no registry-push-secret. It has no authentication, so keep it internal: anything inside the VPC can push to it.

Issue a certificate for registry.${BASE_DOMAIN} from the issuer created in Step 3:

kubectl create namespace registry --dry-run=client -o yaml | kubectl apply -f -
kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: registry-tls
  namespace: registry
spec:
  secretName: registry-tls
  issuerRef:
    name: openchoreo-ca
    kind: ClusterIssuer
  dnsNames:
    - "registry.${BASE_DOMAIN}"
EOF
kubectl wait --for=condition=Ready certificate/registry-tls -n registry --timeout=300s

Run Distribution with a 50 GiB volume and an internal LoadBalancer on port 443:

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: registry-data
  namespace: registry
spec:
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 50Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: registry
  namespace: registry
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: registry
  template:
    metadata:
      labels:
        app: registry
    spec:
      containers:
        - name: registry
          image: registry:3.0.0
          ports:
            - containerPort: 5000
          env:
            - name: REGISTRY_HTTP_TLS_CERTIFICATE
              value: /certs/tls.crt
            - name: REGISTRY_HTTP_TLS_KEY
              value: /certs/tls.key
          volumeMounts:
            - name: data
              mountPath: /var/lib/registry
            - name: certs
              mountPath: /certs
              readOnly: true
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: registry-data
        - name: certs
          secret:
            secretName: registry-tls
---
apiVersion: v1
kind: Service
metadata:
  name: registry
  namespace: registry
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-internal: "true"
spec:
  type: LoadBalancer
  selector:
    app: registry
  ports:
    - name: https
      port: 443
      targetPort: 5000
EOF
kubectl rollout status deploy/registry -n registry --timeout=300s
kubectl get svc registry -n registry -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'

The hostname starts with internal-, which confirms the LoadBalancer is not reachable from the internet. Create a CNAME record from registry.${BASE_DOMAIN} to it. The specific record takes precedence over the *.${BASE_DOMAIN} wildcard, but until the wildcard's cached answer expires (about 5 minutes), the name still resolves to the control-plane gateway.

Check the registry from inside the cluster, the way builds reach it. Expect HTTP/2 200:

kubectl run regtest --rm -i --restart=Never --image=curlimages/curl -- curl -sI https://registry.${BASE_DOMAIN}/v2/

In Phase 2, set global.registry.endpoint to registry.${BASE_DOMAIN} and global.defaultResources.registry.tlsVerify=true.