# Upgrade platform dependencies This page is part of the [v1.0.0 upgrade guide](/agent-platform/docs/next/guides/upgrades/from-v1-0-0/.md). Run it from Step 3 of that guide, in the same shell. ## Target versions[​](#target-versions "Direct link to Target versions") The WSO2 Agent Manager version is not the version of every dependency. Set each installed component to the target below and keep your deployment-specific configuration. Check the inventory with `helm list -A` in every cluster. | Component or chart | v1.0.0 | Target | Notes | | ----------------------------------------- | ------------------------- | ----------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- | | `openchoreo-control-plane` | 1.2.0 | 1.2.8 | Upgrade first. Re-apply the `client_id` patch afterwards. | | `openchoreo-data-plane` | 1.2.0 | 1.2.8 | Upgrade each installation. | | `openchoreo-workflow-plane` | 1.2.0 | 1.2.8 | The default cluster agent CPU and memory resources double. | | `openchoreo-observability-plane` | 1.2.0 | 1.2.8 | Re-apply the `client_id` patch. Set the NetworkPolicy exemption and the OpenSearch passthrough listener. | | `observability-logs-opensearch` | 0.5.3 | 0.5.5 | Moves OpenSearch from 3.3.0 to 3.9.0, which cannot be rolled back without a snapshot. | | `observability-tracing-opensearch` | 0.6.0 | 0.8.1 | The default release name is `observability-traces-opensearch`. | | `observability-metrics-prometheus` | 0.6.1 | 0.8.1 | Apply the Prometheus Operator CRDs first. Prometheus and Alertmanager move to PersistentVolumeClaims. | | `wso2-amp-thunder-extension` | v1.0.0 | 0.0.0-dev | Its bundled ThunderID chart moves from 1.0.0 to 1.0.2. | | Environment `thunderid` releases | 1.0.0 | 1.0.2 | Upgrade every environment instance separately. | | `wso2-amp-platform-resources-extension` | v1.0.0 | 0.0.0-dev | Upgrade before `wso2-agent-manager`. | | `wso2-agent-manager` | v1.0.0 | 0.0.0-dev | Applies database migrations 0044 to 0050. | | `wso2-amp-observability-extension` | v1.0.0 | 0.0.0-dev | Upgrade after the Observability Plane. | | `wso2-amp-evaluation-extension` | v1.0.0 | 0.0.0-dev | Upgrade after the Workflow Plane. | | `wso2-amp-api-platform-gateway-extension` | v1.0.0 | 0.0.0-dev | Upgrade once for each gateway. | | `kgateway-crds`, `kgateway` | v2.2.1 | v2.3.11 | Upgrade `kgateway-crds` first. This is separate from the WSO2 Gateway Operator. | | `gateway-operator` | 0.11.0 | 0.11.3 | Chart 0.11.3 runs operator image 0.11.1. | | Operator-managed API Platform gateway | Chart 1.2.2, images 1.2.1 | `GW_VERSION`: 2026.09.24 from `ghcr.io`, or the subscription release's gateway from the WSO2 registry | Pinned in the operator values. Existing gateways need an explicit redeploy. | The OpenChoreo CRDs are identical in 1.2.0 and 1.2.8, so the OpenChoreo upgrade needs no CRD step. ### Unchanged components[​](#unchanged-components "Direct link to Unchanged components") The target installer keeps these pins from v1.0.0. Verify them and keep their configuration. Do not move them to a new version as part of this procedure. | Component | Pin | Notes | | --------------------------- | ---------------------------- | ----------------------------------------------------------------- | | `cert-manager` | v1.19.2 | Verify certificates and issuers. | | `external-secrets` | 1.3.2 | Verify secret synchronization. | | Kubernetes Gateway API CRDs | v1.4.1, experimental channel | Not a Helm release. | | `openbao` | 0.25.6 | Keep its storage, unseal state, and authentication configuration. | | Local `docker-registry` | Not pinned by the installer | Record the installed version and keep its storage. | | Agent sandbox module | 0.1.1, upstream v0.4.6 | Inventory it if installed. | ## Step 1: Back up dependency state[​](#step-1-back-up-dependency-state "Direct link to Step 1: Back up dependency state") Complete the backups in the main guide first. Also back up: * OpenChoreo custom resources in all namespaces, including projects, components, environments, deployment pipelines, release bindings, and plane registrations. * Each plane's Helm values, generated credentials, and cluster-agent certificates. * OpenSearch, Prometheus, registry, and workflow data, using each store's own backup procedure. Step 5 requires an OpenSearch snapshot. * Every Thunder instance's database volume, signing and encryption material, and client Secrets. Also record issuer URLs and CA bundle ConfigMaps. * Gateway Operator CRDs and APIGateway custom resources. ## Step 2: Confirm the OpenChoreo version[​](#step-2-confirm-the-openchoreo-version "Direct link to Step 2: Confirm the OpenChoreo version") ``` helm list -A | grep openchoreo ``` The standard v1.0.0 installation runs OpenChoreo 1.2.0, and this page covers the 1.2.0 to 1.2.8 patch upgrade. If your installation runs OpenChoreo 1.1.x or earlier, first complete the [OpenChoreo 1.1 to 1.2 migration](https://openchoreo.dev/docs/platform-engineer-guide/upgrades/v1.1-to-v1.2/). That guide describes Gateway API, kgateway, and observability module versions that differ from the Agent Manager pins above. Confirm the dependency versions with the platform maintainers before you run it. ## Step 3: Upgrade kgateway to v2.3.11[​](#step-3-upgrade-kgateway-to-v2311 "Direct link to Step 3: Upgrade kgateway to v2.3.11") kgateway is the network gateway for the OpenChoreo planes. The `kgateway-crds` chart ships its CRDs as templates, so `helm upgrade` updates them. Upgrade it before `kgateway`. Set `OC_CONTEXT` to the context of the cluster that runs kgateway, which is the Control Plane cluster in the default installation: ``` export OC_CONTEXT="" helm upgrade kgateway-crds oci://cr.kgateway.dev/kgateway-dev/charts/kgateway-crds \ --version v2.3.11 -n openchoreo-control-plane --kube-context "$OC_CONTEXT" \ --reset-then-reuse-values --timeout 5m --wait helm upgrade kgateway oci://cr.kgateway.dev/kgateway-dev/charts/kgateway \ --version v2.3.11 -n openchoreo-control-plane --kube-context "$OC_CONTEXT" \ --reset-then-reuse-values --timeout 5m --wait kubectl --context "$OC_CONTEXT" get pods -n openchoreo-control-plane | grep kgateway ``` ## Step 4: Upgrade the OpenChoreo planes to 1.2.8[​](#step-4-upgrade-the-openchoreo-planes-to-128 "Direct link to Step 4: Upgrade the OpenChoreo planes to 1.2.8") Upgrade one plane at a time, in this order, and verify each one before you continue: | Order | Chart and default release | Default namespace | | ----- | -------------------------------- | -------------------------------- | | 1 | `openchoreo-control-plane` | `openchoreo-control-plane` | | 2 | `openchoreo-data-plane` | `openchoreo-data-plane` | | 3 | `openchoreo-workflow-plane` | `openchoreo-workflow-plane` | | 4 | `openchoreo-observability-plane` | `openchoreo-observability-plane` | If a plane spans several clusters, repeat it for each cluster with that cluster's context. For the Control Plane: ``` export OC_VERSION="1.2.8" export OC_REG="oci://ghcr.io/openchoreo/helm-charts" export OC_RELEASE="openchoreo-control-plane" export OC_NS="openchoreo-control-plane" export OC_CONTEXT="" umask 077 mkdir -p upgrade-backup helm get values "$OC_RELEASE" -n "$OC_NS" --kube-context "$OC_CONTEXT" \ -o yaml > "upgrade-backup/${OC_RELEASE}-user-values.yaml" helm upgrade "$OC_RELEASE" "$OC_REG/$OC_RELEASE" \ --version "$OC_VERSION" -n "$OC_NS" --kube-context "$OC_CONTEXT" \ --reset-then-reuse-values --timeout 15m --wait kubectl --context "$OC_CONTEXT" get pods -n "$OC_NS" ``` Replace `` with the context of the cluster that runs the plane. Run the same commands for each plane in the table, changing the release, namespace, and context. ### Re-apply the `client_id` entitlement patch[​](#re-apply-the-client_id-entitlement-patch "Direct link to re-apply-the-client_id-entitlement-patch") Thunder puts the client name in the `client_id` claim, while OpenChoreo 1.2.8 still reads service-account entitlements from `sub` by default. The installer patches this after install. A Helm upgrade of the Control Plane or Observability Plane reverts the patch. Without it, service-to-service calls are unauthorized: the API returns empty lists and `403` responses while every pod looks healthy. After the Control Plane upgrade: ``` patched_yaml=$(kubectl get configmap openchoreo-api-config -n openchoreo-control-plane -o yaml \ | sed -E "s/claim:[[:space:]]*['\"]?sub['\"]?/claim: client_id/g") echo "$patched_yaml" | kubectl apply --server-side --field-manager=helm --force-conflicts -f - kubectl rollout restart deployment/openchoreo-api -n openchoreo-control-plane kubectl rollout status deployment/openchoreo-api -n openchoreo-control-plane --timeout=120s for binding in $(kubectl get clusterauthzrolebindings.openchoreo.dev -o jsonpath='{.items[*].metadata.name}'); do claim=$(kubectl get clusterauthzrolebinding.openchoreo.dev "$binding" -o jsonpath='{.spec.entitlement.claim}') if [ "$claim" = "sub" ]; then kubectl patch clusterauthzrolebinding.openchoreo.dev "$binding" --type=merge \ -p '{"spec":{"entitlement":{"claim":"client_id"}}}' fi done ``` After the Observability Plane upgrade: ``` patched_yaml=$(kubectl get configmap observer-auth-config -n openchoreo-observability-plane -o yaml \ | sed -E "s/claim:[[:space:]]*['\"]?sub['\"]?/claim: client_id/g") echo "$patched_yaml" | kubectl apply --server-side --field-manager=helm --force-conflicts -f - kubectl rollout restart deployment/observer -n openchoreo-observability-plane kubectl rollout status deployment/observer -n openchoreo-observability-plane --timeout=120s ``` ### Set the Observability Plane ingress values[​](#set-the-observability-plane-ingress-values "Direct link to Set the Observability Plane ingress values") OpenChoreo 1.2.8 enables an ingress NetworkPolicy in the Observability Plane chart. It accepts traffic only from pods in the plane's namespace, except pods whose `app.kubernetes.io/name` is listed in `networkPolicy.unrestrictedIngressAppNames`. The Agent Manager observer runs in that namespace and is called from the `wso2-amp` namespace and from evaluation jobs. Without an exemption, monitor runs and run logs fail with `Connection refused`. Upgrade the Observability Plane with these values in addition to your existing ones. They exempt the observer and add the TLS passthrough listener that the OpenSearch route in the next step uses. Replace the hostname if your installation does not use the quick-start domains. ``` cat > upgrade-backup/observability-plane-target-values.yaml <<'EOF' networkPolicy: unrestrictedIngressAppNames: - gateway-default - opentelemetry-collector - wso2-amp-observability-extension gateway: tlsPassthrough: enabled: true hostname: "opensearch.observability.openchoreo.localhost" port: 11443 EOF helm upgrade openchoreo-observability-plane "$OC_REG/openchoreo-observability-plane" \ --version "$OC_VERSION" -n openchoreo-observability-plane --kube-context "$OC_CONTEXT" \ --reset-then-reuse-values -f upgrade-backup/observability-plane-target-values.yaml \ --timeout 15m --wait ``` Use this command for the Observability Plane instead of the generic one above, then re-apply the observer `client_id` patch. The observer pods receive the matching `wso2-amp-observability-extension` label when you upgrade the observability extension in Step 5 of the main guide. ### Route log collection through the Observability Plane gateway[​](#route-log-collection-through-the-observability-plane-gateway "Direct link to Route log collection through the Observability Plane gateway") Fluent Bit now sends logs to OpenSearch through the Observability Plane gateway's TLS passthrough listener, which also serves Fluent Bit running in other clusters. Download the route and the logs module values: ``` curl -fsSL -o upgrade-backup/opensearch-tls-route.yaml \ https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.0.0-dev/deployments/values/opensearch-tls-route.yaml curl -fsSL -o upgrade-backup/observability-logs-opensearch.yaml \ https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.0.0-dev/deployments/values/observability-logs-opensearch.yaml ``` The values file sets the Fluent Bit OpenSearch host, port 11443, and the SNI hostname. If you changed the listener hostname in the previous step, edit both files now: set `spec.hostnames` in `opensearch-tls-route.yaml` and `fluent-bit.openSearchVHost` in `observability-logs-opensearch.yaml` to the same hostname. The values file points Fluent Bit at the gateway's in-cluster address, `gateway-default.openchoreo-observability-plane.svc.cluster.local`. Fluent Bit that runs in another cluster cannot resolve it. For each such cluster, set `fluent-bit.openSearchHost` to an address of the Observability Plane gateway that the cluster can reach on port 11443, and keep `fluent-bit.openSearchVHost` set to the listener hostname. Apply the route, then point the logs module at it. This upgrade keeps the installed chart version, so `--reuse-values` is safe here. Step 5 moves the module to 0.5.5: ``` kubectl --context "$OC_CONTEXT" apply -f upgrade-backup/opensearch-tls-route.yaml helm upgrade observability-logs-opensearch \ oci://ghcr.io/openchoreo/helm-charts/observability-logs-opensearch \ --version 0.5.3 -n openchoreo-observability-plane --kube-context "$OC_CONTEXT" \ --reuse-values -f upgrade-backup/observability-logs-opensearch.yaml \ --set fluent-bit.enabled=true --timeout 10m kubectl --context "$OC_CONTEXT" rollout restart daemonset/fluent-bit -n openchoreo-observability-plane ``` ### Check each plane[​](#check-each-plane "Direct link to Check each plane") * Control Plane: the API responds, controllers are healthy, and existing projects, components, and release bindings reconcile. * Data Plane: the cluster agent connects, and existing workloads and routes work. * Workflow Plane: the cluster agent and controller are healthy, and a representative build succeeds. * Observability Plane: the observer is healthy, and logs, metrics, and traces arrive. Monitor runs work again after the observability extension upgrade in Step 5 of the main guide, which adds the exempt label to the observer. ## Step 5: Upgrade the observability modules[​](#step-5-upgrade-the-observability-modules "Direct link to Step 5: Upgrade the observability modules") The logs, tracing, and metrics modules are OpenChoreo community modules installed as separate Helm releases in the Observability Plane namespace. Upgrade them after the Observability Plane and its log routing in Step 4, and before the Agent Manager observability extension. The logs module keeps the routing values from Step 4, because `--reset-then-reuse-values` keeps every value you supplied. Before you start: * Take an [OpenSearch snapshot](https://docs.opensearch.org/latest/tuning-your-cluster/availability-and-recovery/snapshots/index/). The logs module moves OpenSearch from 3.3.0 to 3.9.0. Once OpenSearch 3.9.0 has opened the data, 3.3.0 cannot start on it: `helm rollback` succeeds, but OpenSearch fails to start. Restoring the snapshot is the only way back. The tracing module stores traces in the same OpenSearch. * Confirm that the cluster has a default StorageClass with `kubectl get storageclass`. The metrics module now stores Prometheus and Alertmanager data on PersistentVolumeClaims of 10 Gi and 1 Gi. Without a default StorageClass, the claims stay `Pending` and the pods never start. Metrics collected before the upgrade are not kept, because 0.6.1 stored them on an `emptyDir` volume. * Upgrade with `--reset-then-reuse-values`, which needs Helm 3.14 or later. Do not use `--reuse-values`. It keeps the previous chart's defaults, so OpenSearch stays on 3.3.0, and the logs chart refuses to render because Fluent Bit 5.x no longer serves the old readiness probe path. Save each module's values: ``` export OC_REG="oci://ghcr.io/openchoreo/helm-charts" export OBS_NS="openchoreo-observability-plane" umask 077 mkdir -p upgrade-backup for release in observability-logs-opensearch observability-traces-opensearch observability-metrics-prometheus; do helm get values "$release" -n "$OBS_NS" -o yaml > "upgrade-backup/${release}-user-values.yaml" done ``` Upgrade the logs module. OpenSearch restarts once, and logs written while it is down are dropped. OpenSearch 3.9.0 uses about twice the CPU of 3.3.0 while it starts, so upgrade when the nodes have spare CPU: ``` helm upgrade observability-logs-opensearch "$OC_REG/observability-logs-opensearch" \ --version 0.5.5 -n "$OBS_NS" \ --reset-then-reuse-values --timeout 15m --wait kubectl get pods -n "$OBS_NS" -o jsonpath='{..image}' | tr ' ' '\n' | grep -E 'opensearch:|fluent-bit:' | sort -u ``` The images should be `opensearchproject/opensearch:3.9.0` and `cr.fluentbit.io/fluent/fluent-bit:5.1.5`. The Fluent Bit toleration for isolation-tier nodes is a user-supplied value, so the upgrade keeps it. If `logs-adapter-opensearch` is still restarting once OpenSearch is ready, run `kubectl rollout restart deployment/logs-adapter-opensearch -n "$OBS_NS"`. If `helm upgrade` already timed out, run it again. The metrics module moves the Prometheus Operator from v0.88.1 to v0.94.2, and Helm does not upgrade CRDs. Apply the CRDs that the target chart bundles, then upgrade the chart. `--force-conflicts` is needed because Helm owns the CRDs it installed: ``` helm show crds "$OC_REG/observability-metrics-prometheus" --version 0.8.1 \ | kubectl apply --server-side --force-conflicts -f - helm upgrade observability-metrics-prometheus "$OC_REG/observability-metrics-prometheus" \ --version 0.8.1 -n "$OBS_NS" \ --reset-then-reuse-values --timeout 15m --wait kubectl get pvc -n "$OBS_NS" ``` Prometheus, Alertmanager, the operator, and kube-state-metrics restart once. Both new claims should be `Bound`. Upgrade the tracing module. It keeps using the `amp-opentelemetry-collector-config` ConfigMap, and the OpenTelemetry Collector moves from 0.154.0 to 0.163.2: ``` helm upgrade observability-traces-opensearch "$OC_REG/observability-tracing-opensearch" \ --version 0.8.1 -n "$OBS_NS" \ --reset-then-reuse-values --timeout 15m --wait ``` If you mirror images to a private registry, mirror the new tags first. The metrics and tracing adapter and setup images move from `ghcr.io/openchoreo` to `cr.openchoreo.dev/openchoreo`, Prometheus uses the `-distroless` tag, and the logs module pins `busybox:1.37.0` for its Fluent Bit init container. After the three upgrades, open an agent in the Console and confirm that new logs, metrics, and traces appear. The Metrics tab can return `502` for up to an hour after kube-state-metrics restarts, and then recovers on its own. ## Step 6: Upgrade platform Thunder to 1.0.2[​](#step-6-upgrade-platform-thunder-to-102 "Direct link to Step 6: Upgrade platform Thunder to 1.0.2") The Agent Manager Thunder extension bundles the platform Thunder instance. The target chart sets the Thunder image to 1.0.2 by default: ``` helm upgrade amp-thunder-extension "$REG/wso2-amp-thunder-extension" \ --version "$VERSION" -n amp-thunder \ --reset-then-reuse-values --timeout 15m --wait kubectl get pods -n amp-thunder -o jsonpath='{..image}' | tr ' ' '\n' | sort -u ``` If your values override `thunder.deployment.image.tag`, update it to `1.0.2`. The extension's bootstrap job runs only at install time, so this upgrade does not change existing Thunder clients or resource servers. New installations name the Agent Manager MCP resource server `amp-agent-manager-mcp-resource-srv`; an upgraded installation keeps `amp-agent-manager-mcp-resource-server`. Verify existing users and clients, Console sign-in and sign-out, token refresh, discovery and JWKS, and the credentials used by OpenChoreo and Agent Manager. ## Step 7: Upgrade every environment Thunder instance[​](#step-7-upgrade-every-environment-thunder-instance "Direct link to Step 7: Upgrade every environment Thunder instance") Each environment has its own release of the upstream `thunderid` chart. The installer names the release and its namespace `amp-thunder--`. Upgrading `amp-thunder-extension` does not change them. The environment installer mounts the platform CA bundle with a `kubectl patch` after the Helm install. Save the full Deployment before the upgrade so you can compare it afterwards: ``` export ENV_THUNDER_RELEASE="amp-thunder-default-default" export ENV_THUNDER_NS="amp-thunder-default-default" export ENV_CONTEXT="" umask 077 mkdir -p upgrade-backup helm get values "$ENV_THUNDER_RELEASE" -n "$ENV_THUNDER_NS" \ --kube-context "$ENV_CONTEXT" -o yaml \ > "upgrade-backup/${ENV_THUNDER_RELEASE}-user-values.yaml" kubectl --context "$ENV_CONTEXT" get deployment "${ENV_THUNDER_RELEASE}-deployment" \ -n "$ENV_THUNDER_NS" -o yaml \ > "upgrade-backup/${ENV_THUNDER_RELEASE}-deployment.yaml" helm upgrade "$ENV_THUNDER_RELEASE" \ oci://ghcr.io/thunder-id/helm-charts/thunderid \ --version 1.0.2 -n "$ENV_THUNDER_NS" \ --kube-context "$ENV_CONTEXT" \ --reset-then-reuse-values \ --set-string deployment.image.tag=1.0.2 \ --timeout 15m --wait ``` Replace `` with the context of the environment's cluster. After the upgrade, confirm that the Deployment still mounts `/etc/ssl/amp/ca-bundle.crt` and sets `SSL_CERT_FILE`. If the mount is gone, re-apply it from the saved Deployment. Keep client Secrets, storage, issuer and public URLs, trusted issuers, and routing unchanged. Do not regenerate client credentials. Check token issuance, existing agent credentials, and MCP authorization before you move to the next instance. ## Gateway Operator CRDs[​](#gateway-operator-crds "Direct link to Gateway Operator CRDs") Apply these CRDs during Step 6 of the main guide, not earlier. Only the `llmprovidertemplates` CRD changes between 0.11.0 and 0.11.3. The APIGateway CRD is unchanged. The commands are in [Step 6 of the main guide](/agent-platform/docs/next/guides/upgrades/from-v1-0-0/.md#step-6-upgrade-the-gateway-extensions-and-the-gateway-operator). ## Recovery[​](#recovery "Direct link to Recovery") A Helm rollback does not restore CRDs or persistent data. Recover a failed stage using its backed-up data, chart revision, and credentials. For OpenChoreo, roll back the plane and re-apply the `client_id` patch. For the logs module, roll back the release and then restore the OpenSearch snapshot from Step 5, because OpenSearch 3.3.0 cannot open data that 3.9.0 has written. Never restore identity data without its signing and encryption material. Confirm dependency health and sign-in before you continue with Agent Manager recovery. ## References[​](#references "Direct link to References") * [Target Agent Manager installer](https://github.com/wso2/agent-manager/blob/amp/v0.0.0-dev/deployments/quick-start/install.sh) * [Agent Manager v1.0.0 installer](https://github.com/wso2/agent-manager/blob/amp/v1.0.0/deployments/quick-start/install.sh) * [Environment Thunder installer](https://github.com/wso2/agent-manager/blob/amp/v0.0.0-dev/deployments/scripts/add-environment-thunder.sh) * [OpenChoreo 1.2.8 release](https://github.com/openchoreo/openchoreo/releases/tag/v1.2.8) * [OpenChoreo upgrade overview](https://openchoreo.dev/docs/platform-engineer-guide/upgrades/overview/) * [Logs module 0.5.5 upgrade notes](https://github.com/openchoreo/community-modules/blob/observability-logs-opensearch-0.5.5/observability-logs-opensearch/README.md#upgrading-to-055) * [Metrics module 0.8.1 upgrade notes](https://github.com/openchoreo/community-modules/blob/observability-metrics-prometheus-0.8.1/observability-metrics-prometheus/README.md#upgrading) * [Tracing module 0.8.1](https://github.com/openchoreo/community-modules/blob/observability-tracing-opensearch-0.8.1/observability-tracing-opensearch/README.md)