Upgrade platform dependencies
This page is part of the v1.0.0 upgrade guide. Run it from Step 3 of that guide, in the same shell.
Target versions​
The WSO2 Agent Manager version is not the version of every dependency. Set each installed component to the target below and keep your deployment-specific configuration. Check the inventory with helm list -A in every cluster.
| Component or chart | v1.0.0 | Target | Notes |
|---|---|---|---|
openchoreo-control-plane | 1.2.0 | 1.2.8 | Upgrade first. Re-apply the client_id patch afterwards. |
openchoreo-data-plane | 1.2.0 | 1.2.8 | Upgrade each installation. |
openchoreo-workflow-plane | 1.2.0 | 1.2.8 | The default cluster agent CPU and memory resources double. |
openchoreo-observability-plane | 1.2.0 | 1.2.8 | Re-apply the client_id patch. Set the NetworkPolicy exemption and the OpenSearch passthrough listener. |
observability-logs-opensearch | 0.5.3 | 0.5.5 | Moves OpenSearch from 3.3.0 to 3.9.0, which cannot be rolled back without a snapshot. |
observability-tracing-opensearch | 0.6.0 | 0.8.1 | The default release name is observability-traces-opensearch. |
observability-metrics-prometheus | 0.6.1 | 0.8.1 | Apply the Prometheus Operator CRDs first. Prometheus and Alertmanager move to PersistentVolumeClaims. |
wso2-amp-thunder-extension | v1.0.0 | 0.0.0-dev | Its bundled ThunderID chart moves from 1.0.0 to 1.0.2. |
Environment thunderid releases | 1.0.0 | 1.0.2 | Upgrade every environment instance separately. |
wso2-amp-platform-resources-extension | v1.0.0 | 0.0.0-dev | Upgrade before wso2-agent-manager. |
wso2-agent-manager | v1.0.0 | 0.0.0-dev | Applies database migrations 0044 to 0050. |
wso2-amp-observability-extension | v1.0.0 | 0.0.0-dev | Upgrade after the Observability Plane. |
wso2-amp-evaluation-extension | v1.0.0 | 0.0.0-dev | Upgrade after the Workflow Plane. |
wso2-amp-api-platform-gateway-extension | v1.0.0 | 0.0.0-dev | Upgrade once for each gateway. |
kgateway-crds, kgateway | v2.2.1 | v2.3.11 | Upgrade kgateway-crds first. This is separate from the WSO2 Gateway Operator. |
gateway-operator | 0.11.0 | 0.11.3 | Chart 0.11.3 runs operator image 0.11.1. |
| Operator-managed API Platform gateway | Chart 1.2.2, images 1.2.1 | GW_VERSION: 2026.09.24 from ghcr.io, or the subscription release's gateway from the WSO2 registry | Pinned in the operator values. Existing gateways need an explicit redeploy. |
The OpenChoreo CRDs are identical in 1.2.0 and 1.2.8, so the OpenChoreo upgrade needs no CRD step.
Unchanged components​
The target installer keeps these pins from v1.0.0. Verify them and keep their configuration. Do not move them to a new version as part of this procedure.
| Component | Pin | Notes |
|---|---|---|
cert-manager | v1.19.2 | Verify certificates and issuers. |
external-secrets | 1.3.2 | Verify secret synchronization. |
| Kubernetes Gateway API CRDs | v1.4.1, experimental channel | Not a Helm release. |
openbao | 0.25.6 | Keep its storage, unseal state, and authentication configuration. |
Local docker-registry | Not pinned by the installer | Record the installed version and keep its storage. |
| Agent sandbox module | 0.1.1, upstream v0.4.6 | Inventory it if installed. |
Step 1: Back up dependency state​
Complete the backups in the main guide first. Also back up:
- OpenChoreo custom resources in all namespaces, including projects, components, environments, deployment pipelines, release bindings, and plane registrations.
- Each plane's Helm values, generated credentials, and cluster-agent certificates.
- OpenSearch, Prometheus, registry, and workflow data, using each store's own backup procedure. Step 5 requires an OpenSearch snapshot.
- Every Thunder instance's database volume, signing and encryption material, and client Secrets. Also record issuer URLs and CA bundle ConfigMaps.
- Gateway Operator CRDs and APIGateway custom resources.
Step 2: Confirm the OpenChoreo version​
helm list -A | grep openchoreo
The standard v1.0.0 installation runs OpenChoreo 1.2.0, and this page covers the 1.2.0 to 1.2.8 patch upgrade.
If your installation runs OpenChoreo 1.1.x or earlier, first complete the OpenChoreo 1.1 to 1.2 migration. That guide describes Gateway API, kgateway, and observability module versions that differ from the Agent Manager pins above. Confirm the dependency versions with the platform maintainers before you run it.
Step 3: Upgrade kgateway to v2.3.11​
kgateway is the network gateway for the OpenChoreo planes. The kgateway-crds chart ships its CRDs as templates, so helm upgrade updates them. Upgrade it before kgateway. Set OC_CONTEXT to the context of the cluster that runs kgateway, which is the Control Plane cluster in the default installation:
export OC_CONTEXT="<control-plane-context>"
helm upgrade kgateway-crds oci://cr.kgateway.dev/kgateway-dev/charts/kgateway-crds \
--version v2.3.11 -n openchoreo-control-plane --kube-context "$OC_CONTEXT" \
--reset-then-reuse-values --timeout 5m --wait
helm upgrade kgateway oci://cr.kgateway.dev/kgateway-dev/charts/kgateway \
--version v2.3.11 -n openchoreo-control-plane --kube-context "$OC_CONTEXT" \
--reset-then-reuse-values --timeout 5m --wait
kubectl --context "$OC_CONTEXT" get pods -n openchoreo-control-plane | grep kgateway
Step 4: Upgrade the OpenChoreo planes to 1.2.8​
Upgrade one plane at a time, in this order, and verify each one before you continue:
| Order | Chart and default release | Default namespace |
|---|---|---|
| 1 | openchoreo-control-plane | openchoreo-control-plane |
| 2 | openchoreo-data-plane | openchoreo-data-plane |
| 3 | openchoreo-workflow-plane | openchoreo-workflow-plane |
| 4 | openchoreo-observability-plane | openchoreo-observability-plane |
If a plane spans several clusters, repeat it for each cluster with that cluster's context. For the Control Plane:
export OC_VERSION="1.2.8"
export OC_REG="oci://ghcr.io/openchoreo/helm-charts"
export OC_RELEASE="openchoreo-control-plane"
export OC_NS="openchoreo-control-plane"
export OC_CONTEXT="<control-plane-context>"
umask 077
mkdir -p upgrade-backup
helm get values "$OC_RELEASE" -n "$OC_NS" --kube-context "$OC_CONTEXT" \
-o yaml > "upgrade-backup/${OC_RELEASE}-user-values.yaml"
helm upgrade "$OC_RELEASE" "$OC_REG/$OC_RELEASE" \
--version "$OC_VERSION" -n "$OC_NS" --kube-context "$OC_CONTEXT" \
--reset-then-reuse-values --timeout 15m --wait
kubectl --context "$OC_CONTEXT" get pods -n "$OC_NS"
Replace <control-plane-context> with the context of the cluster that runs the plane. Run the same commands for each plane in the table, changing the release, namespace, and context.
Re-apply the client_id entitlement patch​
Thunder puts the client name in the client_id claim, while OpenChoreo 1.2.8 still reads service-account entitlements from sub by default. The installer patches this after install. A Helm upgrade of the Control Plane or Observability Plane reverts the patch. Without it, service-to-service calls are unauthorized: the API returns empty lists and 403 responses while every pod looks healthy.
After the Control Plane upgrade:
patched_yaml=$(kubectl get configmap openchoreo-api-config -n openchoreo-control-plane -o yaml \
| sed -E "s/claim:[[:space:]]*['\"]?sub['\"]?/claim: client_id/g")
echo "$patched_yaml" | kubectl apply --server-side --field-manager=helm --force-conflicts -f -
kubectl rollout restart deployment/openchoreo-api -n openchoreo-control-plane
kubectl rollout status deployment/openchoreo-api -n openchoreo-control-plane --timeout=120s
for binding in $(kubectl get clusterauthzrolebindings.openchoreo.dev -o jsonpath='{.items[*].metadata.name}'); do
claim=$(kubectl get clusterauthzrolebinding.openchoreo.dev "$binding" -o jsonpath='{.spec.entitlement.claim}')
if [ "$claim" = "sub" ]; then
kubectl patch clusterauthzrolebinding.openchoreo.dev "$binding" --type=merge \
-p '{"spec":{"entitlement":{"claim":"client_id"}}}'
fi
done
After the Observability Plane upgrade:
patched_yaml=$(kubectl get configmap observer-auth-config -n openchoreo-observability-plane -o yaml \
| sed -E "s/claim:[[:space:]]*['\"]?sub['\"]?/claim: client_id/g")
echo "$patched_yaml" | kubectl apply --server-side --field-manager=helm --force-conflicts -f -
kubectl rollout restart deployment/observer -n openchoreo-observability-plane
kubectl rollout status deployment/observer -n openchoreo-observability-plane --timeout=120s
Set the Observability Plane ingress values​
OpenChoreo 1.2.8 enables an ingress NetworkPolicy in the Observability Plane chart. It accepts traffic only from pods in the plane's namespace, except pods whose app.kubernetes.io/name is listed in networkPolicy.unrestrictedIngressAppNames. The Agent Manager observer runs in that namespace and is called from the wso2-amp namespace and from evaluation jobs. Without an exemption, monitor runs and run logs fail with Connection refused.
Upgrade the Observability Plane with these values in addition to your existing ones. They exempt the observer and add the TLS passthrough listener that the OpenSearch route in the next step uses. Replace the hostname if your installation does not use the quick-start domains.
cat > upgrade-backup/observability-plane-target-values.yaml <<'EOF'
networkPolicy:
unrestrictedIngressAppNames:
- gateway-default
- opentelemetry-collector
- wso2-amp-observability-extension
gateway:
tlsPassthrough:
enabled: true
hostname: "opensearch.observability.openchoreo.localhost"
port: 11443
EOF
helm upgrade openchoreo-observability-plane "$OC_REG/openchoreo-observability-plane" \
--version "$OC_VERSION" -n openchoreo-observability-plane --kube-context "$OC_CONTEXT" \
--reset-then-reuse-values -f upgrade-backup/observability-plane-target-values.yaml \
--timeout 15m --wait
Use this command for the Observability Plane instead of the generic one above, then re-apply the observer client_id patch. The observer pods receive the matching wso2-amp-observability-extension label when you upgrade the observability extension in Step 5 of the main guide.
Route log collection through the Observability Plane gateway​
Fluent Bit now sends logs to OpenSearch through the Observability Plane gateway's TLS passthrough listener, which also serves Fluent Bit running in other clusters. Download the route and the logs module values:
curl -fsSL -o upgrade-backup/opensearch-tls-route.yaml \
https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.0.0-dev/deployments/values/opensearch-tls-route.yaml
curl -fsSL -o upgrade-backup/observability-logs-opensearch.yaml \
https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.0.0-dev/deployments/values/observability-logs-opensearch.yaml
The values file sets the Fluent Bit OpenSearch host, port 11443, and the SNI hostname. If you changed the listener hostname in the previous step, edit both files now: set spec.hostnames in opensearch-tls-route.yaml and fluent-bit.openSearchVHost in observability-logs-opensearch.yaml to the same hostname.
The values file points Fluent Bit at the gateway's in-cluster address, gateway-default.openchoreo-observability-plane.svc.cluster.local. Fluent Bit that runs in another cluster cannot resolve it. For each such cluster, set fluent-bit.openSearchHost to an address of the Observability Plane gateway that the cluster can reach on port 11443, and keep fluent-bit.openSearchVHost set to the listener hostname.
Apply the route, then point the logs module at it. This upgrade keeps the installed chart version, so --reuse-values is safe here. Step 5 moves the module to 0.5.5:
kubectl --context "$OC_CONTEXT" apply -f upgrade-backup/opensearch-tls-route.yaml
helm upgrade observability-logs-opensearch \
oci://ghcr.io/openchoreo/helm-charts/observability-logs-opensearch \
--version 0.5.3 -n openchoreo-observability-plane --kube-context "$OC_CONTEXT" \
--reuse-values -f upgrade-backup/observability-logs-opensearch.yaml \
--set fluent-bit.enabled=true --timeout 10m
kubectl --context "$OC_CONTEXT" rollout restart daemonset/fluent-bit -n openchoreo-observability-plane
Check each plane​
- Control Plane: the API responds, controllers are healthy, and existing projects, components, and release bindings reconcile.
- Data Plane: the cluster agent connects, and existing workloads and routes work.
- Workflow Plane: the cluster agent and controller are healthy, and a representative build succeeds.
- Observability Plane: the observer is healthy, and logs, metrics, and traces arrive. Monitor runs work again after the observability extension upgrade in Step 5 of the main guide, which adds the exempt label to the observer.
Step 5: Upgrade the observability modules​
The logs, tracing, and metrics modules are OpenChoreo community modules installed as separate Helm releases in the Observability Plane namespace. Upgrade them after the Observability Plane and its log routing in Step 4, and before the Agent Manager observability extension. The logs module keeps the routing values from Step 4, because --reset-then-reuse-values keeps every value you supplied.
Before you start:
- Take an OpenSearch snapshot. The logs module moves OpenSearch from 3.3.0 to 3.9.0. Once OpenSearch 3.9.0 has opened the data, 3.3.0 cannot start on it:
helm rollbacksucceeds, but OpenSearch fails to start. Restoring the snapshot is the only way back. The tracing module stores traces in the same OpenSearch. - Confirm that the cluster has a default StorageClass with
kubectl get storageclass. The metrics module now stores Prometheus and Alertmanager data on PersistentVolumeClaims of 10 Gi and 1 Gi. Without a default StorageClass, the claims stayPendingand the pods never start. Metrics collected before the upgrade are not kept, because 0.6.1 stored them on anemptyDirvolume. - Upgrade with
--reset-then-reuse-values, which needs Helm 3.14 or later. Do not use--reuse-values. It keeps the previous chart's defaults, so OpenSearch stays on 3.3.0, and the logs chart refuses to render because Fluent Bit 5.x no longer serves the old readiness probe path.
Save each module's values:
export OC_REG="oci://ghcr.io/openchoreo/helm-charts"
export OBS_NS="openchoreo-observability-plane"
umask 077
mkdir -p upgrade-backup
for release in observability-logs-opensearch observability-traces-opensearch observability-metrics-prometheus; do
helm get values "$release" -n "$OBS_NS" -o yaml > "upgrade-backup/${release}-user-values.yaml"
done
Upgrade the logs module. OpenSearch restarts once, and logs written while it is down are dropped. OpenSearch 3.9.0 uses about twice the CPU of 3.3.0 while it starts, so upgrade when the nodes have spare CPU:
helm upgrade observability-logs-opensearch "$OC_REG/observability-logs-opensearch" \
--version 0.5.5 -n "$OBS_NS" \
--reset-then-reuse-values --timeout 15m --wait
kubectl get pods -n "$OBS_NS" -o jsonpath='{..image}' | tr ' ' '\n' | grep -E 'opensearch:|fluent-bit:' | sort -u
The images should be opensearchproject/opensearch:3.9.0 and cr.fluentbit.io/fluent/fluent-bit:5.1.5. The Fluent Bit toleration for isolation-tier nodes is a user-supplied value, so the upgrade keeps it.
If logs-adapter-opensearch is still restarting once OpenSearch is ready, run kubectl rollout restart deployment/logs-adapter-opensearch -n "$OBS_NS". If helm upgrade already timed out, run it again.
The metrics module moves the Prometheus Operator from v0.88.1 to v0.94.2, and Helm does not upgrade CRDs. Apply the CRDs that the target chart bundles, then upgrade the chart. --force-conflicts is needed because Helm owns the CRDs it installed:
helm show crds "$OC_REG/observability-metrics-prometheus" --version 0.8.1 \
| kubectl apply --server-side --force-conflicts -f -
helm upgrade observability-metrics-prometheus "$OC_REG/observability-metrics-prometheus" \
--version 0.8.1 -n "$OBS_NS" \
--reset-then-reuse-values --timeout 15m --wait
kubectl get pvc -n "$OBS_NS"
Prometheus, Alertmanager, the operator, and kube-state-metrics restart once. Both new claims should be Bound.
Upgrade the tracing module. It keeps using the amp-opentelemetry-collector-config ConfigMap, and the OpenTelemetry Collector moves from 0.154.0 to 0.163.2:
helm upgrade observability-traces-opensearch "$OC_REG/observability-tracing-opensearch" \
--version 0.8.1 -n "$OBS_NS" \
--reset-then-reuse-values --timeout 15m --wait
If you mirror images to a private registry, mirror the new tags first. The metrics and tracing adapter and setup images move from ghcr.io/openchoreo to cr.openchoreo.dev/openchoreo, Prometheus uses the -distroless tag, and the logs module pins busybox:1.37.0 for its Fluent Bit init container.
After the three upgrades, open an agent in the Console and confirm that new logs, metrics, and traces appear. The Metrics tab can return 502 for up to an hour after kube-state-metrics restarts, and then recovers on its own.
Step 6: Upgrade platform Thunder to 1.0.2​
The Agent Manager Thunder extension bundles the platform Thunder instance. The target chart sets the Thunder image to 1.0.2 by default:
helm upgrade amp-thunder-extension "$REG/wso2-amp-thunder-extension" \
--version "$VERSION" -n amp-thunder \
--reset-then-reuse-values --timeout 15m --wait
kubectl get pods -n amp-thunder -o jsonpath='{..image}' | tr ' ' '\n' | sort -u
If your values override thunder.deployment.image.tag, update it to 1.0.2. The extension's bootstrap job runs only at install time, so this upgrade does not change existing Thunder clients or resource servers. New installations name the Agent Manager MCP resource server amp-agent-manager-mcp-resource-srv; an upgraded installation keeps amp-agent-manager-mcp-resource-server.
Verify existing users and clients, Console sign-in and sign-out, token refresh, discovery and JWKS, and the credentials used by OpenChoreo and Agent Manager.
Step 7: Upgrade every environment Thunder instance​
Each environment has its own release of the upstream thunderid chart. The installer names the release and its namespace amp-thunder-<org>-<env>. Upgrading amp-thunder-extension does not change them.
The environment installer mounts the platform CA bundle with a kubectl patch after the Helm install. Save the full Deployment before the upgrade so you can compare it afterwards:
export ENV_THUNDER_RELEASE="amp-thunder-default-default"
export ENV_THUNDER_NS="amp-thunder-default-default"
export ENV_CONTEXT="<environment-cluster-context>"
umask 077
mkdir -p upgrade-backup
helm get values "$ENV_THUNDER_RELEASE" -n "$ENV_THUNDER_NS" \
--kube-context "$ENV_CONTEXT" -o yaml \
> "upgrade-backup/${ENV_THUNDER_RELEASE}-user-values.yaml"
kubectl --context "$ENV_CONTEXT" get deployment "${ENV_THUNDER_RELEASE}-deployment" \
-n "$ENV_THUNDER_NS" -o yaml \
> "upgrade-backup/${ENV_THUNDER_RELEASE}-deployment.yaml"
helm upgrade "$ENV_THUNDER_RELEASE" \
oci://ghcr.io/thunder-id/helm-charts/thunderid \
--version 1.0.2 -n "$ENV_THUNDER_NS" \
--kube-context "$ENV_CONTEXT" \
--reset-then-reuse-values \
--set-string deployment.image.tag=1.0.2 \
--timeout 15m --wait
Replace <environment-cluster-context> with the context of the environment's cluster. After the upgrade, confirm that the Deployment still mounts /etc/ssl/amp/ca-bundle.crt and sets SSL_CERT_FILE. If the mount is gone, re-apply it from the saved Deployment.
Keep client Secrets, storage, issuer and public URLs, trusted issuers, and routing unchanged. Do not regenerate client credentials. Check token issuance, existing agent credentials, and MCP authorization before you move to the next instance.
Gateway Operator CRDs​
Apply these CRDs during Step 6 of the main guide, not earlier. Only the llmprovidertemplates CRD changes between 0.11.0 and 0.11.3. The APIGateway CRD is unchanged. The commands are in Step 6 of the main guide.
Recovery​
A Helm rollback does not restore CRDs or persistent data. Recover a failed stage using its backed-up data, chart revision, and credentials. For OpenChoreo, roll back the plane and re-apply the client_id patch. For the logs module, roll back the release and then restore the OpenSearch snapshot from Step 5, because OpenSearch 3.3.0 cannot open data that 3.9.0 has written. Never restore identity data without its signing and encryption material. Confirm dependency health and sign-in before you continue with Agent Manager recovery.