On k3d (Locally)

Install the Agent Manager on a local Kubernetes cluster using k3d — a lightweight tool that runs Kubernetes inside Docker.

Just want it running?

Use the Run One Command tab under Install: one script creates the cluster and installs everything. Use the Guided Install tab when you want to understand the full setup or need to customize it.

Short-term support (STS) release

2026.11.0 is a short-term support (STS) release available for evaluation. Agent Manager's charts and images, including the API Platform gateway, are hosted on registry.wso2.com and require registry credentials. To obtain these credentials, download Agent Manager from the WSO2 website to receive them by email, or use the credentials provided with your WSO2 subscription. Other components, such as OpenChoreo and the Gateway Operator chart, are hosted on public registries such as ghcr.io. See About the STS Release for more information.

Stronger agent isolation tiers

Agents run sandboxed under the standard runc runtime by default. Agent Manager also supports stronger per-environment isolation tiers — gVisor (userspace kernel) and Kata Containers (per-agent VM) — but they have hardware/OS requirements and need a dedicated node. For more information, see the gVisor and Kata Containers setup guides.

What You Will Get

Agent Manager is a two-layer system installed in two phases:

• Phase 1 — OpenChoreo (base layer): OpenChoreo is an open-source platform that provides the Kubernetes infrastructure Agent Manager runs on. It consists of four planes: a Control Plane for API and configuration, a Data Plane for running workloads and gateways, a Workflow Plane for builds and CI pipelines, and an Observability Plane for traces, logs, and metrics via OpenSearch.

• Phase 2 — Agent Manager : The AI agent management platform installed on top of OpenChoreo. It includes Thunder (identity provider — installed with OpenChoreo since it provides JWT validation for all planes), the Console (web UI), AMP API (backend), AI Gateway, PostgreSQL (database), Secrets Extension (OpenBao for runtime secret injection), Agent Manager Observer (traces, logs, and metrics), and Evaluation Engine (automated agent evaluations).

This guide installs both layers on a single-node k3d cluster on your local machine.

Not for production

This k3d setup is for local development and evaluation only, and is not supported for production. For a production deployment, follow On Your Environment, then check it against the production recommendations.

Its secret store, OpenBao, runs in dev mode and keeps secrets in memory only, so restarting Docker, Colima or your machine loses them. Troubleshooting covers what fails and how to recover.

Prerequisites

Hardware

Resource	Minimum
RAM	8 GB
CPU	4 cores
Disk	~10 GB free

Required Tools

Tool	Version	Purpose
Docker	v26.0+	Container runtime
kubectl	v1.32+	Kubernetes CLI
Helm	v3.12+	Kubernetes package manager
k3d	v5.8+	Local Kubernetes clusters

Verify all tools:

docker --version && kubectl version --client && helm version && k3d version

macOS with Colima

If you use Colima instead of Docker Desktop, start a dedicated agent-manager profile so the install does not interfere with your default Colima instance:

colima start --profile agent-manager \
  --vm-type=vz --vz-rosetta --network-address \
  --cpu 4 --memory 8

--network-address assigns the VM a stable IP so the cluster keeps working after network changes.

Choose --cpu and --memory now rather than adjusting them later. Changing either requires restarting the VM, which can leave an already-created k3d cluster unable to start. Troubleshooting covers the recovery if you hit this.

Whenever you restart Colima, start the cluster through k3d afterwards (k3d cluster stop amp-local && k3d cluster start amp-local) rather than relying on Docker to bring its containers back. Troubleshooting explains why.

On Apple silicon, install Rosetta 2 first (softwareupdate --install-rosetta). Without it, Colima runs amd64 images, including the images built for platform-hosted agents, under QEMU emulation, and a Python agent can take longer to start than its startup probe allows.

For the guided install, prefix the cluster creation command in Step 1 with K3D_FIX_DNS=0. The one-command installer sets it for you.

Required Ports

The following host ports must be free before installation. The Agent Manager services do not need host ports of their own: the Console and API are served through the Control Plane gateway (8080), the Agent Manager Observer through the Observability Plane gateway (11080), and trace ingestion through the Data Plane gateway (19080).

Port	Purpose
6550	Kubernetes API
8080	Control Plane Gateway (HTTP) — Console, API, Thunder, OpenChoreo API, external AI gateway control plane
8443	Control Plane Gateway (HTTPS)
19080	Data Plane Gateway (HTTP) — deployed agents, OTLP /otel ingest
19443	Data Plane Gateway (HTTPS)
10082	Container Registry (Workflow Plane)
11080	Observability Plane Gateway (HTTP) — Agent Manager Observer, Observer API
11082	OpenSearch API
11085	OpenSearch HTTPS

---

Install

Pick how you want to install it:

Run One Command

One script creates the cluster and installs every plane, plus Agent Manager:

curl -fsSL https://github.com/wso2/agent-manager/releases/download/amp/v2026.11.0/k3d-install-subscription.sh | bash -s -- --project <project>

Replace <project> with wso2-agent-manager for a subscription, or wso2-agent-manager-evaluations for an evaluation. The script asks for your registry credentials and installs the newest published version. To install a particular version, add --version, for example --version 2026.11.0-u2. See About the STS Release for its options.

The script:

1. Checks that Docker is running and that k3d, kubectl, Helm, curl, tar, and lsof are installed.
2. Signs in to registry.wso2.com with your registry credentials and finds your version: the one you named with --version, or the newest one published.
3. Downloads the 2026.11.0 release bundle into ~/.wso2-agent-manager/subscription/<project>/<image tag>/.
4. Creates the amp-local k3d cluster and installs the four OpenChoreo planes and Agent Manager, pulling the Agent Manager charts and images and the API Platform gateway from the registry. These are the same steps as the Guided Install tab.
5. Prints the console URL, the admin login, and the version it installed when it finishes.

The install takes about 15-20 minutes, and it is safe to rerun if it stops part way. On Colima, the script sets K3D_FIX_DNS=0 for you.

When it completes, continue with Verify and Access the Platform.

Guided Install

Install each plane step by step.

Phase 1: OpenChoreo Platform

OpenChoreo organises its infrastructure into four planes, each handling a different concern:

• Control Plane — API server and configuration management for the platform
• Data Plane — runs deployed workloads and API gateways
• Workflow Plane — builds and CI pipelines for agent deployments
• Observability Plane — trace, log, and metrics collection via OpenSearch

This phase installs all four with Agent Manager-specific configuration. Estimated time: ~15-20 minutes.

Step 0: Download the Release Bundle

Download the release bundle once. Every manifest, values file and script this guide applies comes from it, so the install needs no further network access to GitHub — which also sidesteps the per-IP rate limiting that makes helm --values <url> fail with HTTP 429 behind shared or corporate NAT.

export VERSION="2026.11.0"
curl -fsSL -O "https://github.com/wso2/agent-manager/releases/download/amp/v${VERSION}/wso2-agent-manager-${VERSION}.tar.gz"
tar xzf "wso2-agent-manager-${VERSION}.tar.gz"
export AMP_DIST="$PWD"

Step 1: Create k3d Cluster

Create the cluster using the Agent Manager cluster configuration, which includes all required port mappings:

k3d cluster create --config="${AMP_DIST}/deployments/quick-start/k3d-config.yaml"

Set the kubectl context:

k3d kubeconfig merge amp-local --kubeconfig-merge-default
kubectl config use-context k3d-amp-local

Generate machine IDs for Fluent Bit log collection:

for NODE in $(k3d node list -o json | grep -o '"name":"[^"]*"' | sed 's/"name":"//;s/"//' | grep "^k3d-amp-local-"); do
  docker exec ${NODE} sh -c "cat /proc/sys/kernel/random/uuid | tr -d '-' > /etc/machine-id"
done

Verify

kubectl cluster-info --context k3d-amp-local
# Should show cluster running at https://0.0.0.0:6550

Step 2: Apply CoreDNS Configuration

Enables *.openchoreo.localhost, *.amp.localhost, *.agentmanager.localhost, *.am-gateway.localhost, and *.gateway.localhost DNS resolution inside the cluster — required for in-cluster lookups of the Console/API/Thunder hostnames as well as agent-to-agent and AI gateway traffic:

kubectl apply -f "${AMP_DIST}/deployments/k8s/coredns-amp-custom.yaml"

CoreDNS's reload plugin can miss the override if the ConfigMap mounts after the pod's initial parse, so restart it to make sure the rewrite rules are active before anything depends on them:

kubectl rollout restart deployment/coredns -n kube-system
kubectl rollout status deployment/coredns -n kube-system --timeout=60s

Step 3: Install Cluster Prerequisites

These are infrastructure components that OpenChoreo depends on. You only install them once per cluster.

Gateway API CRDs — standard Kubernetes resources for managing network gateways and routing:

kubectl apply --server-side \
  -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.1/experimental-install.yaml

cert-manager (v1.19.2) — automates TLS certificate issuance and renewal:

helm upgrade --install cert-manager oci://quay.io/jetstack/charts/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.19.2 \
  --set crds.enabled=true \
  --wait --timeout 180s

External Secrets Operator (v1.3.2) — syncs secrets from external stores (like OpenBao) into Kubernetes:

helm upgrade --install external-secrets oci://ghcr.io/external-secrets/charts/external-secrets \
  --namespace external-secrets \
  --create-namespace \
  --version 1.3.2 \
  --set installCRDs=true \
  --wait --timeout 180s

kgateway (v2.2.1) — the network gateway for OpenChoreo planes:

helm upgrade --install kgateway-crds oci://cr.kgateway.dev/kgateway-dev/charts/kgateway-crds \
  --create-namespace \
  --namespace openchoreo-control-plane \
  --version v2.2.1

helm upgrade --install kgateway oci://cr.kgateway.dev/kgateway-dev/charts/kgateway \
  --namespace openchoreo-control-plane \
  --create-namespace \
  --version v2.2.1 \
  --set controller.extraEnv.KGW_ENABLE_GATEWAY_API_EXPERIMENTAL_FEATURES=true

Step 4: Setup Secrets Store (OpenBao)

OpenBao provides secret management for the Workflow Plane and deployed agents:

helm upgrade --install openbao oci://ghcr.io/openbao/charts/openbao \
  --namespace openbao \
  --create-namespace \
  --version 0.25.6 \
  --values "${AMP_DIST}/deployments/single-cluster/values-openbao.yaml" \
  --timeout 180s

kubectl wait --for=condition=Ready pod -l app.kubernetes.io/name=openbao -n openbao --timeout=120s

Configure the External Secrets ClusterSecretStore:

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-secrets-openbao
  namespace: openbao
---
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: default
spec:
  provider:
    vault:
      server: "http://openbao.openbao.svc:8200"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "openchoreo-secret-writer-role"
          serviceAccountRef:
            name: "external-secrets-openbao"
            namespace: "openbao"
EOF

Step 5: Configuration Variables

Set these variables before continuing. They are used by the remaining Phase 1 steps and all of Phase 2.

Replace <version> and <project> with your values from About the STS Release. The version is 2026.11.0 for the release, or 2026.11.0-u<N> for update N; the release bundle from Step 0 serves both. GATEWAY_REGISTRY is where Phase 2 pulls the API Platform gateway from.

export VERSION="<version>"
export HELM_CHART_REGISTRY="registry.wso2.com/<project>/sts/helm-charts"
export GATEWAY_REGISTRY="registry.wso2.com/<project>/sts"
export AMP_NS="wso2-amp"
export BUILD_CI_NS="openchoreo-workflow-plane"
export OBSERVABILITY_NS="openchoreo-observability-plane"
export DEFAULT_NS="default"
export DATA_PLANE_NS="openchoreo-data-plane"
export THUNDER_NS="amp-thunder"
export OPENCHOREO_INTERNAL_URL="http://openchoreo-api.openchoreo-control-plane.svc.cluster.local:8080"
export THUNDER_PUBLIC_URL="http://thunder.amp.localhost:8080"
export THUNDER_INTERNAL_URL="http://amp-thunder-extension-service.${THUNDER_NS}.svc.cluster.local:8090"
export CONSOLE_PUBLIC_URL="http://console.amp.localhost:8080"
export API_PUBLIC_URL="http://api.amp.localhost:8080"
export OBS_API_PUBLIC_URL="http://traces.amp.localhost:11080"
# Bare hostnames (no scheme/port) — used as the HTTPRoute hostnames on the plane gateways
export CONSOLE_PUBLIC_HOST="console.amp.localhost"
export API_PUBLIC_HOST="api.amp.localhost"
export OBS_API_PUBLIC_HOST="traces.amp.localhost"
export INSTRUMENTATION_URL="http://default-default.gateway.localhost:19080/otel"

info

In k3d, every management endpoint is accessed through a plane gateway using a hostname: Thunder, the Console, and the API via the Control Plane kgateway on port 8080 (thunder.amp.localhost, console.amp.localhost, api.amp.localhost), and the Agent Manager Observer via the Observability Plane kgateway on port 11080 (traces.amp.localhost). These match the chart defaults and the Control Plane / Observability Plane OIDC configuration. No port-forwarding is needed; *.localhost names resolve to the loopback address in browsers and curl.

Sign Helm in to registry.wso2.com with your registry credentials. Phase 2 uses the same username and token to create the image pull secrets:

printf 'Registry username: ' && read -r AMP_REGISTRY_USERNAME
printf 'Registry token: ' && stty -echo && read -r AMP_REGISTRY_PASSWORD; stty echo; echo
printf '%s' "$AMP_REGISTRY_PASSWORD" | helm registry login registry.wso2.com --username "$AMP_REGISTRY_USERNAME" --password-stdin

Step 6: Install Thunder Extension (Identity Provider)

Thunder provides authentication and user management for the entire platform — login, API keys, and OAuth token exchange. It must be installed before the Control Plane because the Control Plane, Observability Plane, and Agent Manager all validate JWTs issued by Thunder.

helm install amp-thunder-extension \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-thunder-extension \
  --version ${VERSION} \
  --namespace ${THUNDER_NS} \
  --create-namespace \
  --timeout 1800s

kubectl wait --for=condition=Available \
  deployment -l app.kubernetes.io/instance=amp-thunder-extension \
  -n ${THUNDER_NS} --timeout=300s

Console admin password

This install also generated the console's admin password. Retrieve it with:

kubectl get secret amp-admin-credentials -n ${THUNDER_NS} -o jsonpath='{.data.password}' | base64 -d

info

No MCP overrides are needed here: thunder.bootstrap.agentManagerMcpBaseUrl and observerMcpBaseUrl become the OAuth resource-server identifiers Thunder registers for the /mcp endpoints, and their chart defaults are already the k3d gateway URLs used throughout this guide. If you deviate from those hostnames, override both to match agentManagerService.config.serverPublicURL and amObserver.publicUrl — Thunder answers invalid_target unless an MCP client's requested resource matches a registered identifier exactly. Both are seeded by the pre-install bootstrap job only, so a helm upgrade will not re-seed them.

warning

Thunder persists its configuration (including the issuer URL) in a database on first boot. If you need to change THUNDER_PUBLIC_URL after installation, you must uninstall the chart, delete its PVC, and reinstall — a helm upgrade alone will not change the issuer in issued tokens.

Verify

kubectl exec -n ${THUNDER_NS} deploy/amp-thunder-extension-deployment -- \
  wget -qO- http://localhost:8090/.well-known/openid-configuration 2>/dev/null \
  | grep -o '"issuer":"[^"]*"'
# Expected: "issuer":"http://thunder.amp.localhost:8080"

Step 7: Install OpenChoreo Control Plane

The Control Plane is configured with Backstage disabled (Agent Manager provides its own console) and OIDC pointing to the AMP Thunder Extension:

helm install openchoreo-control-plane \
  oci://ghcr.io/openchoreo/helm-charts/openchoreo-control-plane \
  --version 1.2.8 \
  --namespace openchoreo-control-plane \
  --create-namespace \
  --timeout 600s \
  --values "${AMP_DIST}/deployments/single-cluster/values-cp.yaml"

kubectl wait --for=condition=Available \
  deployment --all -n openchoreo-control-plane --timeout=600s

Webhook race condition

If the install fails with no endpoints available for service "controller-manager-webhook-service", this is a known transient race in the OpenChoreo control-plane install. Wait for the control plane deployments to become ready, then rerun the helm install command.

kubectl wait --for=condition=Available deployment --all \
  -n openchoreo-control-plane --timeout=300s

What the values file configures

• Backstage disabled (AMP provides its own console)
• OIDC pointing to AMP Thunder Extension (thunder.amp.localhost:8080)
• OpenChoreo API hostname: api.openchoreo.localhost
• Gateway on HTTP port 8080 / HTTPS port 8443, TLS disabled

Patch the service-account entitlement claim (required with Thunder ≥ 0.45)

Thunder 0.45+ puts the client name in the client_id claim (its sub is an opaque UUID), but OpenChoreo 1.2.8 extracts service-account entitlements from sub, and the chart does not expose the setting. Without this patch, every service-to-service call is silently unauthorized — the API returns 200 with empty lists and 403s while all pods look healthy, and the gateway bootstrap later fails with Environment 'default' not found.

patched_yaml=$(kubectl get configmap openchoreo-api-config -n openchoreo-control-plane -o yaml \
  | sed -E "s/claim:[[:space:]]*['\"]?sub['\"]?/claim: client_id/g")
echo "$patched_yaml" | kubectl apply --server-side --field-manager=helm --force-conflicts -f -

kubectl rollout restart deployment/openchoreo-api -n openchoreo-control-plane
kubectl rollout status deployment/openchoreo-api -n openchoreo-control-plane --timeout=120s

for binding in $(kubectl get clusterauthzrolebindings.openchoreo.dev -o jsonpath='{.items[*].metadata.name}'); do
  claim=$(kubectl get clusterauthzrolebinding.openchoreo.dev "$binding" -o jsonpath='{.spec.entitlement.claim}')
  if [ "$claim" = "sub" ]; then
    kubectl patch clusterauthzrolebinding.openchoreo.dev "$binding" --type=merge \
      -p '{"spec":{"entitlement":{"claim":"client_id"}}}'
  fi
done

note

Re-apply this patch after any helm upgrade of the Control Plane chart — the upgrade reverts both the ConfigMap and the bindings. The same patch is needed a second time for the Observability Plane's observer-auth-config in Step 10.

Step 8: Setup Data Plane

Each plane needs the Control Plane's CA certificate to establish trusted communication. Copy it to the Data Plane namespace:

kubectl create namespace openchoreo-data-plane --dry-run=client -o yaml | kubectl apply -f -

CA_CRT=$(kubectl get secret cluster-gateway-ca \
  -n openchoreo-control-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)
kubectl create configmap cluster-gateway-ca \
  --from-literal=ca.crt="$CA_CRT" \
  -n openchoreo-data-plane --dry-run=client -o yaml | kubectl apply -f -

helm install openchoreo-data-plane \
  oci://ghcr.io/openchoreo/helm-charts/openchoreo-data-plane \
  --version 1.2.8 \
  --namespace openchoreo-data-plane \
  --create-namespace \
  --timeout 600s \
  --values "${AMP_DIST}/deployments/single-cluster/values-dp.yaml"

kubectl wait --for=condition=Available \
  deployment --all -n openchoreo-data-plane --timeout=600s

Register the Data Plane with the Control Plane. The command below automatically extracts the CA certificate and inserts it into the YAML — run the entire block as-is:

CA_CERT=$(kubectl get secret cluster-agent-tls \
  -n openchoreo-data-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)

kubectl apply -f - <<EOF
apiVersion: openchoreo.dev/v1alpha1
kind: ClusterDataPlane
metadata:
  name: default
  namespace: default
spec:
  planeID: "default"
  clusterAgent:
    clientCA:
      value: |
$(echo "$CA_CERT" | sed 's/^/        /')
  gateway:
    ingress:
      external:
        name: gateway-default
        namespace: openchoreo-data-plane
        http:
          host: "openchoreoapis.localhost"
          listenerName: http
          port: 19080
        https:
          host: "openchoreoapis.localhost"
          listenerName: https
          port: 19443
  secretStoreRef:
    name: default
EOF

Step 9: Setup Workflow Plane

Copy the CA certificate:

kubectl create namespace openchoreo-workflow-plane --dry-run=client -o yaml | kubectl apply -f -

CA_CRT=$(kubectl get secret cluster-gateway-ca \
  -n openchoreo-control-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)
kubectl create configmap cluster-gateway-ca \
  --from-literal=ca.crt="$CA_CRT" \
  -n openchoreo-workflow-plane --dry-run=client -o yaml | kubectl apply -f -

Install the container registry and the Workflow Plane:

helm upgrade --install registry docker-registry \
  --repo https://twuni.github.io/docker-registry.helm \
  --namespace openchoreo-workflow-plane \
  --create-namespace \
  --values https://raw.githubusercontent.com/openchoreo/openchoreo/v1.2.8/install/k3d/single-cluster/values-registry.yaml \
  --timeout 120s

helm install openchoreo-workflow-plane \
  oci://ghcr.io/openchoreo/helm-charts/openchoreo-workflow-plane \
  --version 1.2.8 \
  --namespace openchoreo-workflow-plane \
  --create-namespace \
  --set argo-workflows.controller.resources.requests.cpu=50m \
  --set argo-workflows.controller.resources.requests.memory=64Mi \
  --set argo-workflows.controller.resources.limits.cpu=200m \
  --set argo-workflows.controller.resources.limits.memory=128Mi \
  --timeout 600s

Register the Workflow Plane:

BP_CA_CERT=$(kubectl get secret cluster-agent-tls \
  -n openchoreo-workflow-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)

kubectl apply -f - <<EOF
apiVersion: openchoreo.dev/v1alpha1
kind: ClusterWorkflowPlane
metadata:
  name: default
  namespace: default
spec:
  planeID: "default"
  secretStoreRef:
    name: default
  clusterAgent:
    clientCA:
      value: |
$(echo "$BP_CA_CERT" | sed 's/^/        /')
EOF

kubectl wait --for=condition=Available \
  deployment --all -n openchoreo-workflow-plane --timeout=600s

Step 10: Setup Observability Plane

Copy the CA certificate:

kubectl create namespace openchoreo-observability-plane --dry-run=client -o yaml | kubectl apply -f -

CA_CRT=$(kubectl get secret cluster-gateway-ca \
  -n openchoreo-control-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)
kubectl create configmap cluster-gateway-ca \
  --from-literal=ca.crt="$CA_CRT" \
  -n openchoreo-observability-plane --dry-run=client -o yaml | kubectl apply -f -

Create the ExternalSecrets for OpenSearch and Observer credentials:

kubectl apply -f - <<'EOF'
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: opensearch-admin-credentials
  namespace: openchoreo-observability-plane
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: default
  target:
    name: opensearch-admin-credentials
  data:
  - secretKey: username
    remoteRef:
      key: opensearch-username
      property: value
  - secretKey: password
    remoteRef:
      key: opensearch-password
      property: value
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: observer-secret
  namespace: openchoreo-observability-plane
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: default
  target:
    name: observer-secret
  data:
  - secretKey: OPENSEARCH_USERNAME
    remoteRef:
      key: opensearch-username
      property: value
  - secretKey: OPENSEARCH_PASSWORD
    remoteRef:
      key: opensearch-password
      property: value
  - secretKey: UID_RESOLVER_OAUTH_CLIENT_SECRET
    remoteRef:
      key: observer-oauth-client-secret
      property: value
EOF

Wait for the ExternalSecrets to sync:

kubectl wait -n openchoreo-observability-plane \
  --for=condition=Ready externalsecret/opensearch-admin-credentials \
  externalsecret/observer-secret --timeout=60s

Apply the custom OpenTelemetry Collector ConfigMap (required for trace ingestion):

kubectl apply -f "${AMP_DIST}/deployments/values/oc-collector-configmap.yaml" \
  -n openchoreo-observability-plane

Install the Observability Plane:

helm install openchoreo-observability-plane \
  oci://ghcr.io/openchoreo/helm-charts/openchoreo-observability-plane \
  --version 1.2.8 \
  --namespace openchoreo-observability-plane \
  --create-namespace \
  --timeout 25m \
  --values "${AMP_DIST}/deployments/single-cluster/values-op.yaml"

kubectl wait --for=condition=Available \
  deployment --all -n openchoreo-observability-plane --timeout=300s

With Thunder ≥ 0.45, apply the same entitlement-claim patch used for the Control Plane to the observer (without it, log and trace queries return 403 Access denied):

patched_yaml=$(kubectl get configmap observer-auth-config -n openchoreo-observability-plane -o yaml \
  | sed -E "s/claim:[[:space:]]*['\"]?sub['\"]?/claim: client_id/g")
echo "$patched_yaml" | kubectl apply --server-side --field-manager=helm --force-conflicts -f -
kubectl rollout restart deployment/observer -n openchoreo-observability-plane
kubectl rollout status deployment/observer -n openchoreo-observability-plane --timeout=120s

Install observability modules (logs, metrics, tracing):

# Logs module
helm upgrade --install observability-logs-opensearch \
  oci://ghcr.io/openchoreo/helm-charts/observability-logs-opensearch \
  --create-namespace \
  --namespace openchoreo-observability-plane \
  --version 0.5.5 \
  --set openSearchSetup.openSearchSecretName="opensearch-admin-credentials" \
  --set adapter.openSearchSecretName="opensearch-admin-credentials" \
  --timeout 10m

# Enable Fluent Bit log collection
helm upgrade observability-logs-opensearch \
  oci://ghcr.io/openchoreo/helm-charts/observability-logs-opensearch \
  --namespace openchoreo-observability-plane \
  --version 0.5.5 \
  --reuse-values \
  --set fluent-bit.enabled=true \
  --timeout 10m

# Metrics module
helm upgrade --install observability-metrics-prometheus \
  oci://ghcr.io/openchoreo/helm-charts/observability-metrics-prometheus \
  --create-namespace \
  --namespace openchoreo-observability-plane \
  --version 0.8.1 \
  --timeout 10m

# Tracing module (uses the custom OTel Collector ConfigMap)
helm upgrade --install observability-traces-opensearch \
  oci://ghcr.io/openchoreo/helm-charts/observability-tracing-opensearch \
  --create-namespace \
  --namespace openchoreo-observability-plane \
  --version 0.8.1 \
  --set openSearch.enabled=false \
  --set openSearchSetup.openSearchSecretName="opensearch-admin-credentials" \
  --set opentelemetry-collector.configMap.existingName="amp-opentelemetry-collector-config" \
  --timeout 10m

Pin the tracing module to 0.6.0 or later

The OpenChoreo 1.2.8 observer returns a span's status as an object ({code, message}), but the tracing adapter at 0.5.0 and 0.5.1 still expects a string. With those versions the span-details endpoint returns 500 and traces are silently dropped. Upstream's own install script still pins 0.5.0; do not follow it here.

Register the Observability Plane and link it to other planes:

OP_CA_CERT=$(kubectl get secret cluster-agent-tls \
  -n openchoreo-observability-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)

kubectl apply -f - <<EOF
apiVersion: openchoreo.dev/v1alpha1
kind: ObservabilityPlane
metadata:
  name: default
  namespace: default
spec:
  planeID: "default"
  clusterAgent:
    clientCA:
      value: |
$(echo "$OP_CA_CERT" | sed 's/^/        /')
  observerURL: http://observer.openchoreo.localhost:11080
EOF

# Link Data Plane to Observability Plane
kubectl patch clusterdataplane default -n default --type merge \
  -p '{"spec":{"observabilityPlaneRef":{"kind":"ClusterObservabilityPlane","name":"default"}}}'

# Link Workflow Plane to Observability Plane
kubectl patch clusterworkflowplane default -n default --type merge \
  -p '{"spec":{"observabilityPlaneRef":{"kind":"ClusterObservabilityPlane","name":"default"}}}'

Step 11: Verify OpenChoreo Installation

Before proceeding to Phase 2, confirm all planes and Thunder are running:

echo "--- Thunder ---"
kubectl get pods -n amp-thunder
echo "--- Control Plane ---"
kubectl get pods -n openchoreo-control-plane
echo "--- Data Plane ---"
kubectl get pods -n openchoreo-data-plane
echo "--- Workflow Plane ---"
kubectl get pods -n openchoreo-workflow-plane
echo "--- Observability Plane ---"
kubectl get pods -n openchoreo-observability-plane
echo "--- Plane Registrations ---"
kubectl get clusterdataplane,clusterworkflowplane,observabilityplane -n default

All pods should be in Running or Completed state.

---

Phase 2: Agent Manager Installation

With OpenChoreo running, you can now install the Agent Manager components — the API, console, identity provider, and extensions that provide the AI agent management capabilities.

The Agent Manager installs as a set of Helm charts on top of OpenChoreo. The components fall into two groups based on install order:

1. Agent Manager Core : Gateway Operator, Agent Manager, Agent Sandbox Module and Platform Resources (agent component types, workflow templates etc). Each depends on the one before it.
2. Extensions : Secret Management, Observability, Evaluation extensions and the API Platform Gateway Extension.

Prerequisites

Thunder (identity provider) must be installed before proceeding — see the Thunder installation step in Phase 1. The variables THUNDER_PUBLIC_URL, THUNDER_INTERNAL_URL, CONSOLE_PUBLIC_URL, API_PUBLIC_URL, OBS_API_PUBLIC_URL, CONSOLE_PUBLIC_HOST, API_PUBLIC_HOST, OBS_API_PUBLIC_HOST, and INSTRUMENTATION_URL must be set from the Configuration Variables section. The production steps below additionally use AGENTS_DOMAIN — the domain deployed agents are served on, agents.<your base domain> — to build the gateway's hostname and virtual host. The *_PUBLIC_HOST variables carry the bare hostname (no scheme or port) and become the HTTPRoute hostnames on the plane gateways.

Create the wso2-registry-credentials image pull secret in the namespaces that run Agent Manager images, with the registry username and token from the Configuration Variables step:

for ns in "${AMP_NS}" "${OBSERVABILITY_NS}" workflows-default; do
  kubectl create namespace "$ns" --dry-run=client -o yaml | kubectl apply -f -
  kubectl create secret docker-registry wso2-registry-credentials \
    --namespace "$ns" \
    --docker-server=registry.wso2.com \
    --docker-username="$AMP_REGISTRY_USERNAME" \
    --docker-password="$AMP_REGISTRY_PASSWORD" \
    --dry-run=client -o yaml | kubectl apply -f -
done

Evaluation monitors run in workflows-default, the workflow namespace of the default organization.

Create the gateway's own pull secret, wso2-gateway-registry-credentials, in ${DATA_PLANE_NS}. The gateway operator reads it to pull the gateway chart, and every gateway pod lists it to pull the gateway images. Besides the usual .dockerconfigjson, it carries the credentials as plain username and password keys, which is the form the operator reads:

kubectl create namespace "${DATA_PLANE_NS}" --dry-run=client -o yaml | kubectl apply -f -
kubectl create secret generic wso2-gateway-registry-credentials \
  --namespace "${DATA_PLANE_NS}" \
  --type=kubernetes.io/dockerconfigjson \
  --from-literal=.dockerconfigjson="$(printf '{"auths":{"registry.wso2.com":{"auth":"%s"}}}' "$(printf '%s:%s' "$AMP_REGISTRY_USERNAME" "$AMP_REGISTRY_PASSWORD" | base64 | tr -d '\n')")" \
  --from-literal=username="$AMP_REGISTRY_USERNAME" \
  --from-literal=password="$AMP_REGISTRY_PASSWORD" \
  --dry-run=client -o yaml | kubectl apply -f -

---

Core Components

Install these in order — each depends on the one before it.

Upgrading an existing installation? Reverse Steps 2 and 4

The order above is for a fresh install, where no agents exist yet and the order is harmless. When upgrading an already-running platform, upgrade wso2-amp-platform-resources-extension (Step 4) at or before wso2-agent-manager (Step 2), not after: the service writes component parameters that the chart must already know how to read, so an agent created in the window between the two upgrades can have its live invoke URL change on the later chart upgrade.

Step 1: Gateway Operator

Manages API Gateway resources and enables secure, authenticated trace ingestion into the Observability Plane.

The gateway controller encrypts stored credentials at rest and will not start without a key, so create one before installing the operator:

kubectl create namespace ${DATA_PLANE_NS} --dry-run=client -o yaml | kubectl apply -f -

openssl rand 32 > gateway-aesgcm.key
kubectl create secret generic gateway-encryption-keys \
  --namespace ${DATA_PLANE_NS} \
  --from-file=default-aesgcm256-v1.bin=gateway-aesgcm.key
rm -f gateway-aesgcm.key

The key is required, and the file name is fixed

The controller looks for a specific path, /app/data/aesgcm-keys/default-aesgcm256-v1.bin, so the Secret key must be exactly default-aesgcm256-v1.bin. Without it the controller crash-loops on failed to initialize key manager: encryption key file not found for version aesgcm256-v1, and the gateway never programs — the rest of the platform stays healthy, so the only symptom is that agents cannot be invoked.

This applies whatever developmentMode is set to. Earlier gateway releases auto-generated a key in development mode; from 1.2.0-beta they do not.

Store the key with the rest of your platform secrets. It encrypts credentials the gateway holds, and losing it means those entries cannot be decrypted.

The gateway for this release, 2026.10.0, is published only to registry.wso2.com. The operator itself still comes from ghcr.io, and pulls the gateway chart and images from GATEWAY_REGISTRY, set in the Configuration Variables step, with the wso2-gateway-registry-credentials secret. The controller in this gateway release serves its health check on the admin port under /api/admin/v1/health, so the probes are set as well.

Helm 4

helm install --server-side=false gateway-operator \
  oci://ghcr.io/wso2/api-platform/helm-charts/gateway-operator \
  --version 0.11.3 \
  --namespace ${DATA_PLANE_NS} \
  --set logging.level=debug \
  --set gatewayApi.installStandardCRDs=false \
  --set gateway.helm.chartName=oci://${GATEWAY_REGISTRY}/helm-charts/gateway \
  --set gateway.helm.chartVersion=2026.10.0 \
  --set gateway.helm.registryCredentialsSecret.name=wso2-gateway-registry-credentials \
  --set "gateway.values.imagePullSecrets[0]=wso2-gateway-registry-credentials" \
  --set gateway.values.gateway.controller.image.repository=${GATEWAY_REGISTRY}/gateway-controller \
  --set gateway.values.gateway.controller.image.tag=2026.10.0 \
  --set gateway.values.gateway.gatewayRuntime.image.repository=${GATEWAY_REGISTRY}/gateway-runtime \
  --set gateway.values.gateway.gatewayRuntime.image.tag=2026.10.0 \
  --set gateway.values.gateway.controller.encryptionKeys.enabled=true \
  --set gateway.values.gateway.controller.encryptionKeys.secretName=gateway-encryption-keys \
  --set gateway.values.gateway.controller.deployment.livenessProbe.httpGet.path=/api/admin/v1/health \
  --set gateway.values.gateway.controller.deployment.livenessProbe.httpGet.port=admin \
  --set gateway.values.gateway.controller.deployment.readinessProbe.httpGet.path=/api/admin/v1/health \
  --set gateway.values.gateway.controller.deployment.readinessProbe.httpGet.port=admin \
  --timeout 600s

Helm 3

helm install gateway-operator \
  oci://ghcr.io/wso2/api-platform/helm-charts/gateway-operator \
  --version 0.11.3 \
  --namespace ${DATA_PLANE_NS} \
  --set logging.level=debug \
  --set gatewayApi.installStandardCRDs=false \
  --set gateway.helm.chartName=oci://${GATEWAY_REGISTRY}/helm-charts/gateway \
  --set gateway.helm.chartVersion=2026.10.0 \
  --set gateway.helm.registryCredentialsSecret.name=wso2-gateway-registry-credentials \
  --set "gateway.values.imagePullSecrets[0]=wso2-gateway-registry-credentials" \
  --set gateway.values.gateway.controller.image.repository=${GATEWAY_REGISTRY}/gateway-controller \
  --set gateway.values.gateway.controller.image.tag=2026.10.0 \
  --set gateway.values.gateway.gatewayRuntime.image.repository=${GATEWAY_REGISTRY}/gateway-runtime \
  --set gateway.values.gateway.gatewayRuntime.image.tag=2026.10.0 \
  --set gateway.values.gateway.controller.encryptionKeys.enabled=true \
  --set gateway.values.gateway.controller.encryptionKeys.secretName=gateway-encryption-keys \
  --set gateway.values.gateway.controller.deployment.livenessProbe.httpGet.path=/api/admin/v1/health \
  --set gateway.values.gateway.controller.deployment.livenessProbe.httpGet.port=admin \
  --set gateway.values.gateway.controller.deployment.readinessProbe.httpGet.path=/api/admin/v1/health \
  --set gateway.values.gateway.controller.deployment.readinessProbe.httpGet.port=admin \
  --timeout 600s

Pin the gateway chart, do not inherit it

gateway.helm.chartVersion decides which gateway chart the operator deploys, and it is the only thing that decides it — the APIGateway resource carries no chart version. Inheriting the operator's default has historically produced a chart whose templates do not render the controller's control-plane address, giving a gateway that installs, programs, and serves traffic while never registering with Agent Manager — so it never appears in the gateway list. Pin the chart explicitly, as above.

Pin the container images too. The chart version and the image tags are set separately, so chartVersion alone can leave the gateway on the chart's default images, which point at a public registry that does not carry this release. The four image.repository/image.tag values above pull the controller and runtime at 2026.10.0 from the registry. Check the pod images after installing rather than trusting the chart version alone.

These are the versions the gateway extension chart of your version is tested with. The chart records them in its gatewayRelease values, which the one-command installer reads, and an update can move them. To see them for your version:

helm show values "oci://${HELM_CHART_REGISTRY}/wso2-amp-api-platform-gateway-extension" --version "${VERSION}" | grep -A3 '^gatewayRelease:'

gateway.values is the operator's passthrough into that chart: anything set under it is merged into the values the gateway is deployed with, which is how the encryption keys above are wired.

Wait for the operator to be ready:

kubectl wait --for=condition=Available \
  deployment -l app.kubernetes.io/name=gateway-operator \
  -n ${DATA_PLANE_NS} --timeout=300s

Grant RBAC for WSO2 API Platform CRDs to the Data Plane cluster-agent:

kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: wso2-api-platform-gateway-module
rules:
  - apiGroups: ["gateway.api-platform.wso2.com"]
    resources: ["restapis", "apigateways"]
    verbs: ["*"]
  - apiGroups: ["gateway.kgateway.dev"]
    resources: ["backends"]
    verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: wso2-api-platform-gateway-module
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: wso2-api-platform-gateway-module
subjects:
  - kind: ServiceAccount
    name: cluster-agent-dataplane
    namespace: ${DATA_PLANE_NS}
EOF

info

The API Platform Gateway is deployed as an extension after Agent Manager is running — see Step 7 below.

Step 2: Agent Manager (API + Console + PostgreSQL)

The core platform: a Go API server, a React web console, and a database.

Tell the Console where your registry is

The charts and images for this release come from the private registry registry.wso2.com. Add --set console.config.registry="registry.wso2.com/<project>/sts" to the install command below. The Console then generates registry-aware commands for adding environments: they run the scripts from the release bundle and pull from the registry, with your registry credentials. Without it, the Console generates the public command, which fetches from GitHub and ghcr.io and fails for this release.

Helm 4

helm install --server-side=false amp \
  oci://${HELM_CHART_REGISTRY}/wso2-agent-manager \
  --version ${VERSION} \
  --namespace ${AMP_NS} \
  --create-namespace \
  --set console.config.instrumentationUrl="${INSTRUMENTATION_URL}" \
  --set console.config.auth.baseUrl="${THUNDER_PUBLIC_URL}" \
  --set console.config.auth.signInRedirectURL="${CONSOLE_PUBLIC_URL}/login" \
  --set console.config.auth.signOutRedirectURL="${CONSOLE_PUBLIC_URL}/login" \
  --set console.config.apiBaseUrl="${API_PUBLIC_URL}" \
  --set agentManagerService.config.amObserverPublicURL="${OBS_API_PUBLIC_URL}" \
  --set console.ocIngress.hostname="${CONSOLE_PUBLIC_HOST}" \
  --set agentManagerService.ocIngress.hostname="${API_PUBLIC_HOST}" \
  --set agentManagerService.config.serverPublicURL="${API_PUBLIC_URL}" \
  --set agentManagerService.config.keyManager.issuer="${THUNDER_PUBLIC_URL}" \
  --set agentManagerService.config.keyManager.jwksUrl="${THUNDER_INTERNAL_URL}/oauth2/jwks" \
  --set agentManagerService.config.oidc.tokenUrl="${THUNDER_INTERNAL_URL}/oauth2/token" \
  --set agentManagerService.config.thunder.baseURL="${THUNDER_PUBLIC_URL}" \
  --set agentManagerService.config.thunder.resolveToHost="${THUNDER_INTERNAL_URL#http://}" \
  --set agentManagerService.config.openChoreo.baseURL="http://openchoreo-api.openchoreo-control-plane.svc.cluster.local:8080" \
  --timeout 1800s

Helm 3

helm install amp \
  oci://${HELM_CHART_REGISTRY}/wso2-agent-manager \
  --version ${VERSION} \
  --namespace ${AMP_NS} \
  --create-namespace \
  --set console.config.instrumentationUrl="${INSTRUMENTATION_URL}" \
  --set console.config.auth.baseUrl="${THUNDER_PUBLIC_URL}" \
  --set console.config.auth.signInRedirectURL="${CONSOLE_PUBLIC_URL}/login" \
  --set console.config.auth.signOutRedirectURL="${CONSOLE_PUBLIC_URL}/login" \
  --set console.config.apiBaseUrl="${API_PUBLIC_URL}" \
  --set agentManagerService.config.amObserverPublicURL="${OBS_API_PUBLIC_URL}" \
  --set console.ocIngress.hostname="${CONSOLE_PUBLIC_HOST}" \
  --set agentManagerService.ocIngress.hostname="${API_PUBLIC_HOST}" \
  --set agentManagerService.config.serverPublicURL="${API_PUBLIC_URL}" \
  --set agentManagerService.config.keyManager.issuer="${THUNDER_PUBLIC_URL}" \
  --set agentManagerService.config.keyManager.jwksUrl="${THUNDER_INTERNAL_URL}/oauth2/jwks" \
  --set agentManagerService.config.oidc.tokenUrl="${THUNDER_INTERNAL_URL}/oauth2/token" \
  --set agentManagerService.config.thunder.baseURL="${THUNDER_PUBLIC_URL}" \
  --set agentManagerService.config.thunder.resolveToHost="${THUNDER_INTERNAL_URL#http://}" \
  --set agentManagerService.config.openChoreo.baseURL="http://openchoreo-api.openchoreo-control-plane.svc.cluster.local:8080" \
  --timeout 1800s

Wait for all components:

# PostgreSQL
kubectl wait --for=jsonpath='{.status.readyReplicas}'=1 \
  statefulset/amp-postgresql -n ${AMP_NS} --timeout=600s

# API server
kubectl wait --for=condition=Available \
  deployment/amp-api -n ${AMP_NS} --timeout=600s

# Console
kubectl wait --for=condition=Available \
  deployment/amp-console -n ${AMP_NS} --timeout=600s

info

keyManager.issuer must be the public Thunder URL — the chart default is the k3d hostname. The authorization server the API advertises in its RFC 9728 protected resource metadata is derived from it, so agentManagerService.config.oauthAuthorizationServers only needs setting if it must differ. Likewise the MCP audience entry is derived from serverPublicURL above, so keyManager.audience only needs setting to change the accepted client IDs.

thunder.baseURL must also be the public Thunder URL, for a related but separate reason: every admin identity call (users, roles, groups) requests a token scoped to Thunder's System resource server via an RFC 8707 resource parameter derived from this value, and Thunder only recognizes the public URL as that resource server's registered identifier — anything else fails with invalid_target. Since the API runs in-cluster and the public hostname often doesn't resolve there, thunder.resolveToHost tells it which address to actually connect to while still presenting thunder.baseURL on the wire; set it to the in-cluster Thunder service (as above) whenever the public hostname isn't directly dialable from this cluster. Leaving thunder.baseURL at the chart default (or omitting resolveToHost when the public hostname can't resolve in-cluster) breaks every admin identity call with invalid_target or a DNS lookup failure, while the rest of the platform looks healthy.

Verify

kubectl get pods -n ${AMP_NS}
# Expected: amp-api-xxx (Running), amp-console-xxx (Running), plus
# amp-postgresql-0 (Running) only when using the in-cluster database

Step 3: Agent Sandbox Module

Agents run as sandboxed pods managed by the Agent Sandbox controller (SandboxTemplate / SandboxWarmPool resources) instead of plain Deployments. This module is required — without it, agent deployments cannot be rendered.

Helm 4

helm upgrade --install --server-side=false agent-sandbox \
  oci://ghcr.io/openchoreo/helm-charts/agent-sandbox \
  --version 0.1.1 \
  --namespace ${DATA_PLANE_NS} \
  --create-namespace \
  --wait \
  --timeout 10m \
  --set namespace=openchoreo-control-plane \
  --set dataPlaneNamespace=${DATA_PLANE_NS} \
  --set dataPlaneServiceAccount=cluster-agent-dataplane \
  --set upstream.version=v0.4.6

Helm 3

helm upgrade --install agent-sandbox \
  oci://ghcr.io/openchoreo/helm-charts/agent-sandbox \
  --version 0.1.1 \
  --namespace ${DATA_PLANE_NS} \
  --create-namespace \
  --wait \
  --timeout 10m \
  --set namespace=openchoreo-control-plane \
  --set dataPlaneNamespace=${DATA_PLANE_NS} \
  --set dataPlaneServiceAccount=cluster-agent-dataplane \
  --set upstream.version=v0.4.6

Wait for the controller:

kubectl wait -n agent-sandbox-system \
  --for=condition=available \
  --timeout=180s \
  deployment/agent-sandbox-controller

Verify

kubectl get crd \
  sandboxtemplates.extensions.agents.x-k8s.io \
  sandboxwarmpools.extensions.agents.x-k8s.io \
  sandboxclaims.extensions.agents.x-k8s.io
# All three CRDs should be listed

kubectl get clusterrole openchoreo-agent-sandbox-access
# RBAC granting the data-plane agent access to sandbox resources

Stronger isolation tiers

With this module, agents run sandboxed under the standard runc runtime. Optionally, individual environments can run agents under gVisor or Kata Containers for stronger isolation — these have hardware/OS requirements and need a dedicated node. See the gVisor and Kata Containers isolation tier guides.

Step 4: Platform Resources

Creates the default Organization, Project, Environment, DeploymentPipeline, and workflow template resources that the console needs on first login. This chart also configures the container registry endpoint used by build workflows to push agent images.

Helm 4

helm install --server-side=false amp-platform-resources \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-platform-resources-extension \
  --version ${VERSION} \
  --namespace ${DEFAULT_NS} \
  --set apiPlatformGateway.namespace=${DATA_PLANE_NS} \
  --timeout 1800s

Helm 3

helm install amp-platform-resources \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-platform-resources-extension \
  --version ${VERSION} \
  --namespace ${DEFAULT_NS} \
  --set apiPlatformGateway.namespace=${DATA_PLANE_NS} \
  --timeout 1800s

Point the traits at the namespace the gateway is actually in

Deployed agents reach the gateway runtime by DNS name, and both the trace-export endpoint and the route agent traffic is forwarded through are built from apiPlatformGateway.namespace. Leaving it empty derives the per-org-env convention <org>-<env>, which is what add-environment.sh creates — but Step 7 installs the gateway extension into ${DATA_PLANE_NS}, matching that chart's own apiGateway.namespace default. Set this to the same namespace, as above.

Get it wrong and nothing reports an error. The agent starts and serves requests while every span batch fails inside it with Failed to resolve '…-gw-gateway-gateway-runtime.default-default' ([Errno -2] Name or service not known), so the Traces view stays empty; and the backend agent routes forward to has no reachable host, so invocation never reaches the gateway. The only evidence is a Transient error warning in the agent's own log.

If you install the gateway extension into <org>-<env> instead, leave this unset.

Container registry configuration

The chart defaults are configured for a local k3d cluster with an in-cluster registry at host.k3d.internal:10082. For other environments, override the registry settings:

Helm 4

# Example: external registry with a base domain
helm install --server-side=false amp-platform-resources \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-platform-resources-extension \
  --version ${VERSION} \
  --namespace ${DEFAULT_NS} \
  --set global.baseDomain="yourdomain.com" \
  --set global.defaultResources.registry.tlsVerify=true \
  --timeout 1800s
# Registry endpoint will be: registry.yourdomain.com

# Example: explicit registry endpoint
helm install --server-side=false amp-platform-resources \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-platform-resources-extension \
  --version ${VERSION} \
  --namespace ${DEFAULT_NS} \
  --set global.registry.endpoint="your-registry.example.com:5000" \
  --set global.defaultResources.registry.tlsVerify=true \
  --timeout 1800s

Helm 3

# Example: external registry with a base domain
helm install amp-platform-resources \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-platform-resources-extension \
  --version ${VERSION} \
  --namespace ${DEFAULT_NS} \
  --set global.baseDomain="yourdomain.com" \
  --set global.defaultResources.registry.tlsVerify=true \
  --timeout 1800s
# Registry endpoint will be: registry.yourdomain.com

# Example: explicit registry endpoint
helm install amp-platform-resources \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-platform-resources-extension \
  --version ${VERSION} \
  --namespace ${DEFAULT_NS} \
  --set global.registry.endpoint="your-registry.example.com:5000" \
  --set global.defaultResources.registry.tlsVerify=true \
  --timeout 1800s

Value	Default	Description
global.registry.endpoint	host.k3d.internal:10082	Registry endpoint for pushing images
global.baseDomain	""	When set, registry endpoint becomes registry.<baseDomain>
global.defaultResources.registry.tlsVerify	false	Enable TLS verification for registry connections

Not every registry can back build workflows

Two properties are required, and the failure surfaces only on the first agent build, long after the platform installs and verifies cleanly:

• Push-to-create. Each build pushes its image as <workflow-run-name>-image — a repository name that is different on every run, so repositories cannot be pre-created. The registry must create them on push. Amazon ECR does not and fails every build with repository ... not found; the same applies to any registry that requires repositories to be provisioned first.
• Static credentials. The push step reads a fixed .dockerconfigjson from the registry-push-secret key in OpenBao (or pushes unauthenticated if the key is absent). Registries whose credentials expire — ECR again, with its 12-hour tokens — cannot be refreshed by anything in the build pipeline.

CNCF Distribution, Harbor (with a project set to auto-create), and GitLab's registry satisfy both. If you have no registry yet, running Distribution in-cluster behind an internal LoadBalancer with a cert-manager certificate is a few resources — but it ships with no authentication; put htpasswd in front of it before it carries anything real.

To push with credentials, store them in OpenBao before the first build. Write the registry's .dockerconfigjson to registry-dockerconfig.json, in the form {"auths":{"<registry-host>":{"auth":"<base64 of user:token>"}}}, then:

kubectl exec -i -n openbao openbao-0 -- env BAO_ADDR=http://127.0.0.1:8200 BAO_TOKEN='<root-token>' \
  bao kv put secret/registry-push-secret - < registry-dockerconfig.json

---

Extensions

These can be installed in any order after Core is ready.

Step 5: Observability Extension (Agent Manager Observer)

Deploys the observer service that queries and serves trace, log, and metrics data to the console and CLI.

Helm 4

helm install --server-side=false amp-observability-traces \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-observability-extension \
  --version ${VERSION} \
  --namespace ${OBSERVABILITY_NS} \
  --set amObserver.ocIngress.hostname="${OBS_API_PUBLIC_HOST}" \
  --set amObserver.publicUrl="${OBS_API_PUBLIC_URL}" \
  --set amObserver.auth.issuer="${THUNDER_PUBLIC_URL}" \
  --timeout 1800s

kubectl wait --for=condition=Available \
  deployment/amp-observer -n ${OBSERVABILITY_NS} --timeout=600s

Helm 3

helm install amp-observability-traces \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-observability-extension \
  --version ${VERSION} \
  --namespace ${OBSERVABILITY_NS} \
  --set amObserver.ocIngress.hostname="${OBS_API_PUBLIC_HOST}" \
  --set amObserver.publicUrl="${OBS_API_PUBLIC_URL}" \
  --set amObserver.auth.issuer="${THUNDER_PUBLIC_URL}" \
  --timeout 1800s

kubectl wait --for=condition=Available \
  deployment/amp-observer -n ${OBSERVABILITY_NS} --timeout=600s

warning

amObserver.auth.issuer must be the public Thunder URL, not the in-cluster service URL. The observer validates the same user token the console and amctl send to the Agent Manager API, so its issuer has to match agentManagerService.config.keyManager.issuer from Step 2. Leave it at the chart default on a custom-domain install and the traces page stays empty while the observer logs JWT validation failed ... invalid issuer.

The chart derives amObserver.oauth.authorizationServers and the MCP audience entry from the values above; set them explicitly only if they must differ.

Step 6: Evaluation Extension

Installs workflow templates for running automated evaluations (accuracy, safety, reasoning, tool usage) against agent traces.

Helm 4

helm install --server-side=false amp-evaluation-extension \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-evaluation-extension \
  --version ${VERSION} \
  --namespace ${BUILD_CI_NS} \
  --timeout 1800s

Helm 3

helm install amp-evaluation-extension \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-evaluation-extension \
  --version ${VERSION} \
  --namespace ${BUILD_CI_NS} \
  --timeout 1800s

info

Evaluation jobs publish scores using the amp-publisher-client OAuth2 credentials bootstrapped by the Thunder extension. The client secret is fetched from OpenBao at workflow runtime (secret/amp-publisher-client-secret) and must match the value set on thunder.bootstrap.ampPublisherClient.clientSecret.

caution

The eval job needs egress to your API server, and the NetworkPolicy matches it by address range after kube-proxy DNAT — the endpoint address, never the kubernetes Service ClusterIP. Check what that address is:

kubectl -n default get endpoints kubernetes

networkPolicy.evaluationJob.apiServer.cidrs defaults to all of RFC1918, which covers that address on most private clusters but also spans your pod and service CIDRs. Add this to the command above to narrow it to your control-plane subnet:

Helm 4

helm install --server-side=false amp-evaluation-extension \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-evaluation-extension \
  --version ${VERSION} \
  --namespace ${BUILD_CI_NS} \
  --timeout 1800s \
  --set "networkPolicy.evaluationJob.apiServer.cidrs[0]=<control-plane-subnet>"

Helm 3

helm install amp-evaluation-extension \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-evaluation-extension \
  --version ${VERSION} \
  --namespace ${BUILD_CI_NS} \
  --timeout 1800s \
  --set "networkPolicy.evaluationJob.apiServer.cidrs[0]=<control-plane-subnet>"

Setting it is required if kubernetes resolves to a public endpoint (public GKE/AKS control planes) or to 100.64.0.0/10. The default will not match those, and every evaluation will publish its scores and then report FAILED.

Step 7: API Platform Gateway Extension

Registers the API Platform Gateway with the Agent Manager and deploys the gateway stack. Install this last — it requires the Agent Manager API to be healthy and Thunder to be ready for token exchange.

Helm 4

helm install --server-side=false api-platform-default-default \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-api-platform-gateway-extension \
  --version ${VERSION} \
  --namespace ${DATA_PLANE_NS} \
  --set agentManager.orgName=default \
  --set gateway.environment=default \
  --timeout 1800s

kubectl wait --for=condition=complete job/api-platform-default-default-bootstrap \
  -n ${DATA_PLANE_NS} --timeout=300s

Helm 3

helm install api-platform-default-default \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-api-platform-gateway-extension \
  --version ${VERSION} \
  --namespace ${DATA_PLANE_NS} \
  --set agentManager.orgName=default \
  --set gateway.environment=default \
  --timeout 1800s

kubectl wait --for=condition=complete job/api-platform-default-default-bootstrap \
  -n ${DATA_PLANE_NS} --timeout=300s

The extension already renders an OTLP ingest route for its own gateway (<release>-otel-restapi in the data-plane namespace), so nothing further is needed for the default environment. The standalone manifest below targets the per-environment namespace <org>-<env>, which does not exist until the first agent is deployed there — apply it only if you need that variant, and only after a deployment has created the namespace:

kubectl apply -f "${AMP_DIST}/deployments/values/otel-collector-rest-api.yaml"

Verify

kubectl get apigateway api-platform-default-default -n ${DATA_PLANE_NS}
# STATUS should show "Programmed"

kubectl get jobs -n ${DATA_PLANE_NS} | grep api-platform-default-default-bootstrap
# STATUS should show "Complete"

Provision Thunder Identity Provider for the Default Environment

Every Environment needs its own dedicated Thunder ID instance. This is separate from the platform Thunder installed in Phase 1, which only handles console and API login. This instance is what issues each agent its own OAuth2 credential (AgentID) in that Environment. Agents can still be created without it, but they will never get an AgentID there — Agent Manager keeps retrying and failing in the background, and the API Platform Gateway Extension silently skips registering its ThunderKeyManager identity provider, so the Identity Providers page stays empty with no error anywhere in the platform. Run this once the Agent Manager API is reachable at ${API_PUBLIC_URL}:

ENV_NAME=default \
DISPLAY_NAME="Default" \
ORG_NAME=default \
THUNDER_HANDLE=default-idp \
AMP_API_URL="${API_PUBLIC_URL}/api/v1" \
IDP_TOKEN_URL="${THUNDER_PUBLIC_URL}/oauth2/token" \
THUNDER_HOST_BASE_DOMAIN="amp.localhost" \
bash "${AMP_DIST}/deployments/scripts/add-environment-thunder.sh"

The script registers an unguessable handle with Agent Manager before it creates the environment's ThunderID instance. THUNDER_HANDLE is optional (3-63 characters); omit it to have Agent Manager generate one instead. The script prints the resulting issuer as http://<handle>.amp.localhost:8080. Do not derive it from the organization and environment names. The existing *.amp.localhost routing covers each handle, generated or chosen.

Verify

kubectl get pods -n amp-thunder-default-default
# All pods should be Running

---

Verify and Access the Platform

Run a full status check to confirm everything is running:

# All pods across key namespaces
kubectl get pods -n openchoreo-control-plane
kubectl get pods -n openchoreo-data-plane
kubectl get pods -n openchoreo-workflow-plane
kubectl get pods -n openchoreo-observability-plane
kubectl get pods -n wso2-amp
kubectl get pods -n amp-thunder
# Helm releases
helm list -A | grep -E 'openchoreo|amp|gateway'

Service	URL
Agent Manager Console	http://console.amp.localhost:8080
Agent Manager API	http://api.amp.localhost:8080
Agent Manager Observer	http://traces.amp.localhost:11080
OTLP trace ingest (HTTP)	http://default-default.gateway.localhost:19080/otel

Console login: the username is admin; the password was generated at install time. Retrieve it with:

kubectl get secret amp-admin-credentials -n amp-thunder -o jsonpath='{.data.password}' | base64 -d

HTTPS ingest is disabled by default on this setup, and enabling it doesn't cover the documented hosts

values-dp.yaml, installed in Step 8 of the guided install (the one-command installer applies the same file), sets gateway.tls.enabled: false even though it already configures httpsPort: 19443 and a certificateRefs entry pointing at the gateway-tls Secret (created earlier by cert-manager). With TLS disabled, the Data Plane's gateway-default Gateway/Service only get an http listener — there is no 19443 listener to connect to, so an HTTPS OTLP endpoint is not reachable on a default install.

Turning gateway.tls.enabled on would not make the OTLP host above reachable over HTTPS anyway: gateway-tls's certificate only covers *.agentmanager.localhost and *.am-gateway.localhost (its configured hostname and SANs), not *.gateway.localhost — the domain default-default.gateway.localhost above actually uses. Reaching HTTPS on that specific host would additionally require reissuing the certificate with *.gateway.localhost in its dnsNames and pointing gateway.tls.hostname at it.

Given that, this setup keeps TLS disabled and uses the HTTP endpoint above for trace ingestion.

---

Cleanup

To remove the install, delete the entire k3d cluster and all resources. Do this before you install again:

k3d cluster delete amp-local

If you used the one-command installer, running the uninstall script it left behind with --delete-cluster does the same.

To remove Agent Manager and OpenChoreo but keep the cluster, run the uninstall script without that flag:

bash ~/.wso2-agent-manager/subscription/<project>/<image tag>/deployments/quick-start/uninstall.sh

warning

When it keeps the cluster, the uninstall script currently leaves some components installed, including the gateway, OpenBao, the identity provider's data, the observability modules and the CRDs. If you install again on that cluster, the gateway cannot connect to the new Agent Manager. Delete the cluster before you reinstall.

---

Troubleshooting

Pods stuck in Pending

Usually a resource constraint. Check node capacity:

kubectl describe pod <pod-name> -n <namespace>
kubectl top nodes

Increase Colima/Docker Desktop resources if needed.

On Colima, resizing is not a free action

Changing --cpu or --memory requires stopping and restarting the VM, which rebuilds Docker's networks underneath the cluster. An already-created k3d cluster may not come back. Read the next entry before you resize.

k3d server container crash-loops after restarting or resizing Colima

This does not present as a Kubernetes error. kubectl stops responding altogether, because the API server never starts:

kubectl cluster-info --context k3d-amp-local
# Hangs, or reports the connection was refused

docker ps -a --filter name=k3d-amp-local
# k3d-amp-local-server-0 shows STATUS "Restarting", or repeated "Exited"

docker logs k3d-amp-local-server-0

The server node's log ends with:

level=fatal msg="Failed to start networking: unable to initialize network policy controller: error getting node subnet: failed to find interface with specified node ip"

Stopping and starting the Colima VM recreates the k3d-amp-local bridge network, and the containers rejoin it in a different order than before. The load balancer can take the address the server node previously held. k3s still has the old address in its persisted node record, so its network policy controller finds no interface matching it and the process exits. Restarting again does not help, because the mismatch is stored in the node's own state rather than recalculated at boot.

Recreate the cluster. Nothing in this install is persistent across a cluster delete, so recovery is the same work as a fresh install:

k3d cluster delete amp-local
docker network rm k3d-amp-local 2>/dev/null || true

colima stop --profile agent-manager
colima start --profile agent-manager \
  --vm-type=vz --vz-rosetta --network-address \
  --cpu <cpus> --memory <gb>

Then rerun the one-command installer, or the guided install from Step 1: Create k3d Cluster onward.

Agent logs, traces and monitor-run logs do not load after restarting Docker or Colima

When Docker restarts the cluster's containers on its own, k3s rewrites the CoreDNS NodeHosts entry without host.k3d.internal. This install's CoreDNS rewrites point the platform's *.localhost names, such as api.openchoreo.localhost, at host.k3d.internal, so those names stop resolving inside the cluster. The console then fails to load agent logs, traces and monitor-run logs.

Check whether the entry is missing:

kubectl get configmap coredns -n kube-system -o jsonpath='{.data.NodeHosts}'
# No line ends with host.k3d.internal

Stop and start the cluster through k3d, which adds the entry back:

k3d cluster stop amp-local && k3d cluster start amp-local

The names resolve again about 1-2 minutes after the cluster starts. Do this after every Docker or Colima restart.

Monitor runs and agent builds fail after restarting Docker, Colima or your machine

OpenBao, the secret store in this install, runs in dev mode and keeps its data in memory, so a restart wipes the secrets stored in it. Agents that are already running keep working on the secrets they last synced, but anything that needs to read a secret again fails:

• Monitor runs do not start. Their pods stay in CreateContainerConfigError with a secret ... not found event.
• Agent builds cannot push their images.
• ExternalSecrets report Secret does not exist (kubectl get externalsecrets -A).

The lost secrets cannot be restored. Delete the cluster and install again:

bash ~/.wso2-agent-manager/subscription/<project>/<image tag>/deployments/quick-start/uninstall.sh --delete-cluster

If you used the guided install, run k3d cluster delete amp-local instead. Then rerun the one-command installer, or the guided install from Step 1: Create k3d Cluster onward.

For a deployment whose secrets survive restarts, follow On Your Environment.

Gateway not becoming Programmed

kubectl logs -n openchoreo-data-plane -l app.kubernetes.io/name=gateway-operator
kubectl describe apigateway api-platform-default-default -n openchoreo-data-plane

Plane registration issues

kubectl get clusterdataplane default -n default -o yaml
kubectl logs -n openchoreo-control-plane -l app.kubernetes.io/name=openchoreo-control-plane

OpenSearch connectivity issues

kubectl get pods -n openchoreo-observability-plane -l app=opensearch
kubectl run -it --rm debug --image=curlimages/curl --restart=Never -- \
  curl -v http://opensearch.openchoreo-observability-plane.svc.cluster.local:9200

Port already in use

Find the process occupying the port and stop it:

lsof -i :<port>

---

Reference: Configuration Files

All Agent Manager-specific configuration files used in this guide:

File	Purpose
k3d-config.yaml	k3d cluster with all required port mappings
coredns-amp-custom.yaml	CoreDNS rewrites for *.openchoreo.localhost, *.amp.localhost, *.agentmanager.localhost, *.am-gateway.localhost, *.gateway.localhost
add-environment-thunder.sh	Provisions the per-environment Thunder identity provider
values-cp.yaml	Control Plane — Backstage disabled, AMP Thunder OIDC
values-dp.yaml	Data Plane — gateway ports, Fluent Bit config
values-op.yaml	Observability Plane — standalone OpenSearch, AMP Thunder OIDC
values-openbao.yaml	OpenBao — dev mode, Kubernetes auth, pre-seeded secrets
oc-collector-configmap.yaml	Custom OTel Collector ConfigMap for trace ingestion
wso2-amp-api-platform-gateway-extension	API Platform Gateway Helm chart (includes gateway config, bootstrap, CR)
otel-collector-rest-api.yaml	OTel Collector REST API resource