# Upgrade from v1.0.0 to v0.0.0-dev

This guide upgrades an existing WSO2 Agent Manager v1.0.0 Helm installation to the community release or to a WSO2 subscription release. The commands use the quick-start release names and namespaces. Substitute the names from your installation, and repeat the gateway steps for every gateway in every organization and environment, including the `<org>-<env>-egress` gateway of an environment with a separate egress gateway.

The two distributions differ only in where the charts and images come from, the gateway version, and the registry credentials. You select the distribution once, in Step 1. The steps marked **WSO2 Subscription** apply only to a subscription installation.

## Breaking changes[​](#breaking-changes "Direct link to Breaking changes")

| Change                                                                                                  | Impact                                                                                                                                                                                                                               | Required action                                                                                                                                                                                                                                                                        |
| ------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| OpenChoreo moves from 1.2.0 to 1.2.8                                                                    | The CRDs are unchanged. A Helm upgrade of the Control Plane and Observability Plane reverts the service-account `client_id` entitlement patch.                                                                                       | Upgrade the four planes and re-apply the patch, as described in [Upgrade platform dependencies](/agent-platform/docs/next/guides/upgrades/platform-dependencies/.md).                                                                                                                  |
| ThunderID moves from 1.0.0 to 1.0.2                                                                     | Platform Thunder and each environment Thunder are separate releases.                                                                                                                                                                 | Upgrade each instance separately.                                                                                                                                                                                                                                                      |
| kgateway moves from v2.2.1 to v2.3.11                                                                   | The `kgateway-crds` chart updates the kgateway CRDs.                                                                                                                                                                                 | Upgrade `kgateway-crds`, then `kgateway`.                                                                                                                                                                                                                                              |
| The Observability Plane enables an ingress NetworkPolicy                                                | Monitor runs and run logs fail with `Connection refused` unless the Agent Manager observer is exempt. Log collection moves to a TLS passthrough route.                                                                               | Upgrade the Observability Plane with the values in [Upgrade platform dependencies](/agent-platform/docs/next/guides/upgrades/platform-dependencies/.md#set-the-observability-plane-ingress-values).                                                                                    |
| The observability modules move to logs 0.5.5, tracing 0.8.1, and metrics 0.8.1                          | OpenSearch moves from 3.3.0 to 3.9.0 and cannot be rolled back without a snapshot. The Prometheus Operator CRDs change, and Prometheus and Alertmanager move to PersistentVolumeClaims, so earlier metrics are not kept.             | Take an OpenSearch snapshot, confirm a default StorageClass, and apply the Prometheus Operator CRDs before the upgrade, as described in [Upgrade platform dependencies](/agent-platform/docs/next/guides/upgrades/platform-dependencies/.md#step-5-upgrade-the-observability-modules). |
| Gateway Operator moves from chart 0.11.0 to 0.11.3                                                      | Only the `llmprovidertemplates` CRD changes. Upgrading the operator does not redeploy existing gateways.                                                                                                                             | Apply the CRDs, upgrade the operator, and then trigger each gateway explicitly (Step 7).                                                                                                                                                                                               |
| API Platform gateway moves from images 1.2.1 to 2026.09.24 (community) or 2026.10.0 (WSO2 subscription) | The gateway controller's SQLite schema moves from version 4 to version 5 (2026.09.24) or version 6 (2026.10.0). Both refuse a version 4 database. Agent Manager does not resend gateway configuration to an empty controller.        | Upgrade each controller database to version 5 in place before its gateway redeploys (Step 7). Gateway 2026.10.0 then migrates it to version 6 on its first start. Do not delete it.                                                                                                    |
| Gateway 2026.10.0 (WSO2 subscription) changes three defaults                                            | Gateways present a default client certificate to backends, return `x-request-id`, and serve `/.well-known/jwks` on every host.                                                                                                       | To opt out, set `present_default_identity`, `return_request_id_in_response`, or `jwks_endpoint_enabled` to `false`.                                                                                                                                                                    |
| A WSO2 subscription release comes from `registry.wso2.com`                                              | Charts, Agent Manager images, and the gateway chart and images need registry credentials and pull Secrets.                                                                                                                           | Complete the **WSO2 Subscription** part of Step 1.                                                                                                                                                                                                                                     |
| New LLM proxies take their API key header from the LLM Service Provider                                 | New proxies expect the provider's header, such as `X-API-Key`. Existing proxies keep `API-Key`. Agents that send `API-Key` to a new proxy get `401`.                                                                                 | Send the header shown in the agent's LLM configuration in the Console.                                                                                                                                                                                                                 |
| The Agent Manager database moves from migration 0043 to 0050                                            | Migrations 0044 to 0050 run in one transaction. v1.0.0 refuses to start against a database that contains them.                                                                                                                       | Back up PostgreSQL first. A chart rollback alone cannot restore v1.0.0.                                                                                                                                                                                                                |
| The gateway catch-all route request timeout moves from 30 seconds to unlimited                          | Long-running and streaming requests are no longer cut off at 30 seconds. The gateway's 5-minute idle timeout still applies.                                                                                                          | Review any client behavior that relied on the 30-second limit.                                                                                                                                                                                                                         |
| The platform Thunder bootstrap runs only at install time                                                | The target adds a refresh-token lifetime and login-screen branding to the Console, CLI, and MCP clients, and renames the Agent Manager MCP resource server to `amp-agent-manager-mcp-resource-srv`. An upgrade does not apply these. | Existing clients and resource servers keep their v1.0.0 settings and names.                                                                                                                                                                                                            |

## Prerequisites[​](#prerequisites "Direct link to Prerequisites")

* Helm 3.14 or later, for `--reset-then-reuse-values`. The target installer uses Helm 3.19.4.
* `kubectl` access to every cluster that runs an Agent Manager component.
* Access to the chart registry: `ghcr.io` for the community release, or `registry.wso2.com` with the [registry credentials](/agent-platform/docs/next/guides/install-with-a-wso2-subscription/.md#get-your-registry-credentials) for a WSO2 subscription.
* WSO2 Subscription only: every cluster node can pull from `registry.wso2.com`. Pull Secrets cover the namespaces this guide names, but not the namespaces OpenChoreo creates for agent workloads. Those pulls use the node's container runtime credentials. On k3d, the installer writes them to `/etc/rancher/k3s/registries.yaml` only when it creates the cluster, so a cluster created for the community release does not have them.
* `sqlite3` on the machine where you run the commands.
* A maintenance window. Each gateway is unavailable while its controller is stopped and redeployed.

## Step 1: Set variables and record the installation[​](#step-1-set-variables-and-record-the-installation "Direct link to Step 1: Set variables and record the installation")

Select your distribution and set its variables. The rest of the guide uses only these variables.

* Community
* WSO2 Subscription

```
export VERSION="0.0.0-dev"

export REG="oci://ghcr.io/wso2"

export GW_VERSION="2026.09.24"

GW_REGISTRY_ARGS=()

AMP_IMAGE_ARGS=()
```

```
export PROJECT="<project>"

export VERSION="<subscription-version>"

export REG="oci://registry.wso2.com/$PROJECT/sts/helm-charts"

export GW_VERSION="<gateway-version>"

export GW_REGISTRY="registry.wso2.com/$PROJECT/sts"

GW_REGISTRY_ARGS=(

  --set-string gateway.helm.chartName="oci://$GW_REGISTRY/helm-charts/gateway"

  --set-string gateway.helm.registryCredentialsSecret.name=wso2-gateway-registry-credentials

  --set-string "gateway.values.imagePullSecrets[0]=wso2-gateway-registry-credentials"

  --set-string gateway.values.gateway.controller.image.repository="$GW_REGISTRY/gateway-controller"

  --set-string gateway.values.gateway.gatewayRuntime.image.repository="$GW_REGISTRY/gateway-runtime"

)

AMP_IMAGE_ARGS=(--set-string global.ampImageRegistry="registry.wso2.com/$PROJECT/sts")
```

Replace `<project>` with `wso2-agent-manager` for a subscription, or `wso2-agent-manager-evaluations` for an evaluation. Replace `<subscription-version>` with the release or update to install, for example `2026.11.0` or `2026.11.0-u2`. See [List the Published Versions](/agent-platform/docs/next/guides/install-with-a-wso2-subscription/.md#list-the-published-versions). Replace `<gateway-version>` with the gateway version of that release, which is `2026.10.0` for the 2026.11.0 release. The gateway comes from the same registry and uses the same credentials.

Then set the shared variables and record the installation:

```
export AMP_NS="wso2-amp"

export DP_NS="openchoreo-data-plane"

export GATEWAY_CONTEXT="<gateway-cluster-context>"

export GW_OPERATOR_VERSION="0.11.3"



helm version

helm list -A

kubectl --context "$GATEWAY_CONTEXT" get apigateways -A
```

Replace `<gateway-cluster-context>` with the Kubernetes context of the cluster that runs the Gateway Operator and your gateways. On a single-cluster installation, this is the output of `kubectl config current-context`. For development testing, set `VERSION` to the exact snapshot under test.

Record each Helm release, namespace, chart version, and Kubernetes context. Record every APIGateway with its namespace. The quick-start installer names them `api-platform-<org>-<env>` in the `<org>-<env>` namespace, and an egress gateway `api-platform-<org>-<env>-egress` in the `<org>-<env>-egress` namespace.

### WSO2 Subscription: sign in and create the pull Secrets[​](#wso2-subscription-sign-in-and-create-the-pull-secrets "Direct link to WSO2 Subscription: sign in and create the pull Secrets")

Skip this part for the community release.

Read your registry credentials without storing them in the shell history, and sign Helm in to the registry. The commands work in both bash and zsh. Quote the username wherever you use it, because registry usernames can contain `$`:

```
printf 'Registry username: '; read -r AMP_REGISTRY_USERNAME

printf 'Registry token: '; read -rs AMP_REGISTRY_PASSWORD; echo

printf '%s' "$AMP_REGISTRY_PASSWORD" | helm registry login registry.wso2.com \

  --username "$AMP_REGISTRY_USERNAME" --password-stdin

helm show chart "$REG/wso2-agent-manager" --version "$VERSION"

helm show chart "oci://$GW_REGISTRY/helm-charts/gateway" --version "$GW_VERSION"
```

Both `helm show chart` commands print the chart. Write the credentials to files that only you can read, so the token never appears on a command line:

```
SECRET_DIR="$(mktemp -d)"

chmod 700 "$SECRET_DIR"

printf '{"auths":{"registry.wso2.com":{"auth":"%s"}}}' \

  "$(printf '%s:%s' "$AMP_REGISTRY_USERNAME" "$AMP_REGISTRY_PASSWORD" | base64 | tr -d '\n')" \

  > "$SECRET_DIR/config.json"

printf '%s' "$AMP_REGISTRY_USERNAME" > "$SECRET_DIR/username"

printf '%s' "$AMP_REGISTRY_PASSWORD" > "$SECRET_DIR/password"
```

Create the `wso2-registry-credentials` Secret in the namespaces that run Agent Manager images, including the `workflows-<org>` namespace of every organization. In a multi-cluster installation, run the loop against each cluster that has these namespaces, with `kubectl --context`:

```
for ns in "$AMP_NS" openchoreo-observability-plane \

    $(kubectl get namespaces -o name | sed -n 's|^namespace/\(workflows-.*\)|\1|p'); do

  kubectl create secret generic wso2-registry-credentials \

    --namespace "$ns" \

    --type=kubernetes.io/dockerconfigjson \

    --from-file=.dockerconfigjson="$SECRET_DIR/config.json" \

    --dry-run=client -o yaml | kubectl apply -f -

done
```

The gateway has its own Secret, `wso2-gateway-registry-credentials`. The Gateway Operator reads it from its namespace to pull the gateway chart, and each gateway reads it from its own namespace to pull its images. Besides the Docker configuration, the Secret holds plain `username` and `password` keys, which the operator uses for the chart. Create it in the operator namespace and in every namespace that contains an APIGateway, egress gateways included, then delete the files:

```
for ns in "$DP_NS" $(kubectl --context "$GATEWAY_CONTEXT" get apigateways -A \

    -o jsonpath='{range .items[*]}{.metadata.namespace}{"\n"}{end}' | sort -u); do

  kubectl --context "$GATEWAY_CONTEXT" create secret generic wso2-gateway-registry-credentials \

    --namespace "$ns" \

    --type=kubernetes.io/dockerconfigjson \

    --from-file=.dockerconfigjson="$SECRET_DIR/config.json" \

    --from-file=username="$SECRET_DIR/username" \

    --from-file=password="$SECRET_DIR/password" \

    --dry-run=client -o yaml | kubectl --context "$GATEWAY_CONTEXT" apply -f -

done

rm -rf "$SECRET_DIR"
```

## Step 2: Back up[​](#step-2-back-up "Direct link to Step 2: Back up")

Pause deployments, promotions, and configuration changes for the maintenance window.

Save each release's history and user-supplied values. These files can contain credentials, so keep them out of source control:

```
umask 077

mkdir -p upgrade-backup

helm history amp -n "$AMP_NS" > upgrade-backup/amp-history.txt

helm get values amp -n "$AMP_NS" -o yaml > upgrade-backup/amp-user-values.yaml
```

Repeat for the other releases recorded in Step 1.

Back up the Agent Manager database. This dump is the only way back to v1.0.0. The quick-start PostgreSQL instance requires a password, which is stored in the `amp-postgresql` Secret:

```
export PGPW="$(kubectl get secret amp-postgresql -n "$AMP_NS" -o jsonpath='{.data.password}' | base64 -d)"

kubectl exec -n "$AMP_NS" amp-postgresql-0 -- env PGPASSWORD="$PGPW" \

  pg_dump -U agentmanager -d agentmanager -Fc \

  > upgrade-backup/ams-1.0.0.dump
```

Check that the dump file is not empty. For an external database, use its own backup tooling. Test the restore in a separate database.

For each gateway, back up the controller data and its encryption key Secret. The key is mounted from the `gateway-encryption-keys` Secret, not stored on the volume. This backup is the recovery point for the 1.2.1 controller. The loop covers every namespace that contains an APIGateway, egress gateways included:

```
for GW_NS in $(kubectl --context "$GATEWAY_CONTEXT" get apigateways -A \

    -o jsonpath='{range .items[*]}{.metadata.namespace}{"\n"}{end}' | sort -u); do

  GWC=$(kubectl --context "$GATEWAY_CONTEXT" get deploy -n "$GW_NS" -o name | grep gateway-controller)

  kubectl --context "$GATEWAY_CONTEXT" exec -n "$GW_NS" "$GWC" -- tar cf - -C /app data \

    > "upgrade-backup/gw-data-$GW_NS.tar"

  kubectl --context "$GATEWAY_CONTEXT" get secret gateway-encryption-keys -n "$GW_NS" -o yaml \

    > "upgrade-backup/gw-encryption-keys-$GW_NS.yaml"

done

ls -l upgrade-backup/gw-*
```

Each `gw-data-*.tar` file must be larger than zero bytes.

Record a baseline: invoke representative agents, and note existing traces, monitor results, and API keys in use.

## Step 3: Upgrade OpenChoreo, the observability modules, and Thunder[​](#step-3-upgrade-openchoreo-the-observability-modules-and-thunder "Direct link to Step 3: Upgrade OpenChoreo, the observability modules, and Thunder")

Follow [Upgrade platform dependencies](/agent-platform/docs/next/guides/upgrades/platform-dependencies/.md) through its environment Thunder step. Leave the Gateway Operator at 0.11.0 until Step 6.

## Step 4: Upgrade platform resources, then Agent Manager[​](#step-4-upgrade-platform-resources-then-agent-manager "Direct link to Step 4: Upgrade platform resources, then Agent Manager")

Upgrade platform resources first. The target Agent Manager attaches a new `a2a-gateway-route` trait to A2A agents, and only the target platform resources chart defines it.

For a WSO2 subscription, `AMP_IMAGE_ARGS` points `global.ampImageRegistry` at the `sts` path. Without it, `--reset-then-reuse-values` keeps the v1.0.0 value and the pods fail with `ImagePullBackOff`. Step 5 uses it too.

```
helm upgrade amp-platform-resources \

  "$REG/wso2-amp-platform-resources-extension" \

  --version "$VERSION" -n default \

  --reset-then-reuse-values --timeout 15m --wait



helm upgrade amp "$REG/wso2-agent-manager" \

  --version "$VERSION" -n "$AMP_NS" \

  --reset-then-reuse-values "${AMP_IMAGE_ARGS[@]}" \

  --timeout 15m --wait
```

The `amp-db-migration` post-upgrade hook applies migrations 0044 to 0050. Kubernetes deletes the job 300 seconds after it finishes, so read its logs right away:

```
kubectl logs -n "$AMP_NS" job/amp-db-migration

kubectl exec -n "$AMP_NS" amp-postgresql-0 -- env PGPASSWORD="$PGPW" \

  psql -U agentmanager -d agentmanager -tc \

  "select max(id) from migration_history"
```

The query returns `0050`. If the job fails, the transaction leaves the database at `0043`. Fix the cause, usually database connectivity or credentials, and rerun the same `helm upgrade amp` command.

## Step 5: Upgrade the observability and evaluation extensions[​](#step-5-upgrade-the-observability-and-evaluation-extensions "Direct link to Step 5: Upgrade the observability and evaluation extensions")

Skip any extension that is not installed.

```
helm upgrade amp-observability-traces "$REG/wso2-amp-observability-extension" \

  --version "$VERSION" -n openchoreo-observability-plane \

  --reset-then-reuse-values "${AMP_IMAGE_ARGS[@]}" \

  --timeout 15m --wait



helm upgrade amp-evaluation-extension "$REG/wso2-amp-evaluation-extension" \

  --version "$VERSION" -n openchoreo-workflow-plane \

  --reset-then-reuse-values "${AMP_IMAGE_ARGS[@]}" \

  --timeout 15m --wait
```

## Step 6: Upgrade the gateway extensions and the Gateway Operator[​](#step-6-upgrade-the-gateway-extensions-and-the-gateway-operator "Direct link to Step 6: Upgrade the gateway extensions and the Gateway Operator")

Upgrade each gateway extension release, including the `api-platform-<org>-<env>-egress` release of an egress gateway. This does not redeploy the gateway. Its pre-upgrade bootstrap job calls Agent Manager, which must be running the target version.

Each gateway extension release has the same name and namespace as its APIGateway, so the loop upgrades one release per APIGateway:

```
for gw in $(kubectl --context "$GATEWAY_CONTEXT" get apigateways -A \

    -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{"\n"}{end}'); do

  helm upgrade "${gw#*/}" \

    "$REG/wso2-amp-api-platform-gateway-extension" \

    --version "$VERSION" -n "${gw%/*}" --kube-context "$GATEWAY_CONTEXT" \

    --reset-then-reuse-values --timeout 15m --wait

done
```

Apply the operator CRDs. Helm does not upgrade CRDs in a chart's `crds/` directory. The new CRDs keep serving the existing versions.

```
helm show crds \

  oci://ghcr.io/wso2/api-platform/helm-charts/gateway-operator \

  --version "$GW_OPERATOR_VERSION" \

  | kubectl --context "$GATEWAY_CONTEXT" apply --server-side --force-conflicts -f -
```

Upgrade the operator and pin the gateway chart and both gateway images. Setting only the chart version leaves the old images in place. For a WSO2 subscription, `GW_REGISTRY_ARGS` also points the operator at the registry, its credentials, and the gateway image repositories, which the existing release sets to `ghcr.io`.

```
helm upgrade gateway-operator \

  oci://ghcr.io/wso2/api-platform/helm-charts/gateway-operator \

  --version "$GW_OPERATOR_VERSION" -n "$DP_NS" \

  --kube-context "$GATEWAY_CONTEXT" --reset-then-reuse-values \

  "${GW_REGISTRY_ARGS[@]}" \

  --set-string gateway.helm.chartVersion="$GW_VERSION" \

  --set-string gateway.values.gateway.controller.image.tag="$GW_VERSION" \

  --set-string gateway.values.gateway.gatewayRuntime.image.tag="$GW_VERSION" \

  --timeout 15m --wait
```

Chart 0.11.3 runs operator image 0.11.1. Existing gateways keep running 1.2.1 after this step.

## Step 7: Upgrade each controller database in place and redeploy its gateway[​](#step-7-upgrade-each-controller-database-in-place-and-redeploy-its-gateway "Direct link to Step 7: Upgrade each controller database in place and redeploy its gateway")

The target controller refuses a version 4 database, and Agent Manager does not resend routes, LLM providers, LLM proxies, or MCP proxies to an empty controller. Upgrade each existing database in place instead. The only schema change between version 4 and version 5 is a new `agents` table, so the upgrade adds that table and sets the schema version. Gateway 2026.10.0 uses schema version 6, and migrates a version 5 database to version 6 on its first start, so the same version 5 upgrade applies to both distributions. Run this step for one gateway at a time, egress gateways included. You need `sqlite3` on the machine where you run the commands.

danger

Confirm the Step 2 backups and the Step 4 migration before you continue. Keep the `gateway-encryption-keys` Secret. Without it, the controller cannot decrypt the stored secrets.

Set the gateway to work on:

```
export GW_NS="default-default"

export GW_NAME="api-platform-default-default"

export GW_DIR="upgrade-backup/gw-$GW_NS"

mkdir -p "$GW_DIR"

GWC=$(kubectl --context "$GATEWAY_CONTEXT" get deploy -n "$GW_NS" -o name | grep gateway-controller)
```

Stop the controller process while keeping its pod and volume. The override replaces the controller command with `sleep`, so nothing writes to the database. First confirm that the container does not already set a command, because removing the override later deletes the field. The first command prints nothing:

```
kubectl --context "$GATEWAY_CONTEXT" get -n "$GW_NS" "$GWC" \

  -o jsonpath='{.spec.template.spec.containers[0].command}'

kubectl --context "$GATEWAY_CONTEXT" patch -n "$GW_NS" "$GWC" --type=json -p \

  '[{"op":"add","path":"/spec/template/spec/containers/0/command","value":["sh","-c","sleep 86400"]}]'

kubectl --context "$GATEWAY_CONTEXT" get pods -n "$GW_NS" -w | grep gateway-controller
```

Stop watching once a single controller pod is `Running`. It stays `0/1` ready because the controller is not running. Copy the database out of that pod:

```
POD=$(kubectl --context "$GATEWAY_CONTEXT" get pods -n "$GW_NS" --no-headers \

  | awk '/gateway-controller/ && $3=="Running"{print $1}')

kubectl --context "$GATEWAY_CONTEXT" exec -n "$GW_NS" "$POD" -c gateway-controller -- ps

kubectl --context "$GATEWAY_CONTEXT" exec -n "$GW_NS" "$POD" -c gateway-controller -- \

  sh -c 'cd /app/data && tar cf - gateway.db*' > "$GW_DIR/gateway-v4.tar"
```

The `ps` output lists `sleep` and no `gateway-controller` process. Upgrade the copy to version 5:

```
tar xf "$GW_DIR/gateway-v4.tar" -C "$GW_DIR"

sqlite3 "$GW_DIR/gateway.db" 'PRAGMA wal_checkpoint(TRUNCATE);'

sqlite3 "$GW_DIR/gateway.db" <<'SQL'

CREATE TABLE IF NOT EXISTS agents (

    uuid TEXT NOT NULL,

    gateway_id TEXT NOT NULL,

    configuration TEXT NOT NULL,

    signed_public_card TEXT,

    signed_protected_card TEXT,

    PRIMARY KEY (gateway_id, uuid),

    FOREIGN KEY(gateway_id, uuid) REFERENCES artifacts(gateway_id, uuid) ON DELETE CASCADE

);

PRAGMA user_version = 5;

SQL

sqlite3 "$GW_DIR/gateway.db" 'PRAGMA user_version; PRAGMA integrity_check; SELECT kind, handle FROM artifacts;'
```

The output shows `5`, `ok`, and the routes, LLM providers, LLM proxies, and MCP proxies on this gateway. Copy the upgraded database into the pod next to the old one, and compare checksums:

```
kubectl --context "$GATEWAY_CONTEXT" exec -i -n "$GW_NS" "$POD" -c gateway-controller -- \

  sh -c 'cat > /app/data/gateway.db.new' < "$GW_DIR/gateway.db"

shasum -a 256 "$GW_DIR/gateway.db"

kubectl --context "$GATEWAY_CONTEXT" exec -n "$GW_NS" "$POD" -c gateway-controller -- \

  sha256sum /app/data/gateway.db.new
```

If the two checksums differ, repeat the copy. When they match, replace the old database:

```
kubectl --context "$GATEWAY_CONTEXT" exec -n "$GW_NS" "$POD" -c gateway-controller -- \

  sh -c 'rm -f /app/data/gateway.db-wal /app/data/gateway.db-shm && mv /app/data/gateway.db.new /app/data/gateway.db'
```

Remove the override and redeploy the gateway, with no pause between the two commands. The operator redeploys an APIGateway only when its spec changes, so the second command changes `spec.infrastructure.annotations`. Until the redeploy replaces it, the 1.2.1 controller restarts and refuses the version 5 database without changing it.

```
kubectl --context "$GATEWAY_CONTEXT" patch -n "$GW_NS" "$GWC" --type=json -p \

  '[{"op":"remove","path":"/spec/template/spec/containers/0/command"}]'

kubectl --context "$GATEWAY_CONTEXT" patch apigateway "$GW_NAME" -n "$GW_NS" --type=merge -p \

  "{\"spec\":{\"infrastructure\":{\"annotations\":{\"upgrade.amp.wso2.com/gateway-version\":\"$GW_VERSION\"}}}}"
```

Wait until the gateway reports the new generation as programmed and the new controller is rolled out:

```
for i in $(seq 120); do

  s=$(kubectl --context "$GATEWAY_CONTEXT" get apigateway "$GW_NAME" -n "$GW_NS" -o jsonpath=\

'{.metadata.generation} {.status.conditions[?(@.type=="Programmed")].observedGeneration} {.status.conditions[?(@.type=="Programmed")].status}')

  echo "$s"

  echo "$s" | awk '$1==$2 && $3=="True"{ok=1} END{exit !ok}' && break

  sleep 5

done

kubectl --context "$GATEWAY_CONTEXT" rollout status -n "$GW_NS" "$GWC" --timeout=300s
```

The loop gives up after 10 minutes. If its last line is not two matching numbers and `True`, see Troubleshooting. Otherwise, check the controller:

```
kubectl --context "$GATEWAY_CONTEXT" get pods -n "$GW_NS" -o jsonpath='{..image}' | tr ' ' '\n' | sort -u

kubectl --context "$GATEWAY_CONTEXT" logs -n "$GW_NS" "$GWC" \

  | grep -E 'Migrating database schema|Database schema migrated successfully|Database schema up to date|Loaded configurations|Loaded API keys|Control plane connection established'
```

The images are `gateway-controller` and `gateway-runtime` at `$GW_VERSION`, from `$GW_REGISTRY` for a WSO2 subscription. On gateway 2026.09.24 the logs show schema version 5. On gateway 2026.10.0 they show `Migrating database schema` from version 5 to version 6, `Database schema migrated successfully`, and schema version 6. In both cases the configuration and API key counts match the counts before the upgrade. Invoke one agent through this gateway before you start the next gateway.

## Step 8: Verify existing workloads[​](#step-8-verify-existing-workloads "Direct link to Step 8: Verify existing workloads")

| Check                        | Command or action                                                                         | Expected result                                                                                |
| ---------------------------- | ----------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- |
| Agent Manager images         | `kubectl get pods -n "$AMP_NS" -o jsonpath='{..image}' \| tr ' ' '\n' \| sort -u`         | The `amp-*` images use the target version.                                                     |
| Migration history            | `select max(id) from migration_history`                                                   | `0050`                                                                                         |
| OpenChoreo planes            | `helm list -A`                                                                            | All four planes use chart 1.2.8.                                                               |
| OpenChoreo authorization     | List projects in the Console.                                                             | Existing projects appear. An empty list with no errors means the `client_id` patch is missing. |
| Gateways                     | `kubectl get apigateways -A`                                                              | Each gateway is `Programmed=True` at its current generation.                                   |
| Console gateway status       | In the Console left sidebar, under **INFRASTRUCTURE**, click **Gateways**.                | Each gateway is listed and active.                                                             |
| Invocation                   | Invoke an agent that uses an LLM Service Provider or MCP Proxy, with an existing API key. | The call succeeds through the gateway.                                                         |
| Deployment and promotion     | Redeploy an existing agent and promote it.                                                | Both succeed.                                                                                  |
| Observability and evaluation | Open a pre-upgrade trace and run a monitor.                                               | Historical traces load, new traces arrive, and the monitor completes.                          |

Do not mark the upgrade complete because Helm reported success.

## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting")

| Symptom                                                                                                                               | Cause and action                                                                                                                                                                                                                              |
| ------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| The controller crash-loops with `unsupported schema version 4, expected 5` (or `expected 6`)                                          | The gateway redeployed before its database was upgraded. Do not delete the database. Run Step 7 from the command override onwards: the override stops the crash loop, and the in-place upgrade makes the database readable.                   |
| Gateway images stay at 1.2.1                                                                                                          | The operator did not redeploy the gateway. Check `helm get values gateway-operator -n "$DP_NS"` for both image tags, then repeat the Step 7 patch with a new annotation value.                                                                |
| Agent invocations return `404 {"error":"Not Found"}`, which the Console reports as a CORS error, or an agent's LLM calls return `404` | The controller started with an empty database, so it has no routes, LLM providers, LLM proxies, or MCP proxies. Restore the gateway's Step 2 backup into its controller and upgrade it in place as in Step 7.                                 |
| Agent invocations return `401 Valid API key required` after a restore                                                                 | The API key was created or regenerated after the backup you restored. Regenerate the key in the Console.                                                                                                                                      |
| APIGateway reports `Multi-Attach error for volume`                                                                                    | The old and new controller pods are on different nodes. Check that the controller Deployment uses the `Recreate` strategy, and delete the old controller pod.                                                                                 |
| Migration job fails                                                                                                                   | Read the job logs before the 300-second cleanup. Fix the cause and rerun `helm upgrade amp`. The database stays at `0043`.                                                                                                                    |
| Agent Manager images remain at v1.0.0                                                                                                 | An explicit image tag override in your values pins the old version. Remove it and rerun the upgrade.                                                                                                                                          |
| WSO2 Subscription: Helm fails with `401 Unauthorized`                                                                                 | Helm is not signed in to `registry.wso2.com`. Run the `helm registry login` command from Step 1 again.                                                                                                                                        |
| WSO2 Subscription: Agent Manager pods report `ImagePullBackOff` for an image under `registry.wso2.com/<project>/` without `sts`       | The release kept the v1.0.0 `global.ampImageRegistry`. Once the failed upgrade finishes, rerun it with `"${AMP_IMAGE_ARGS[@]}"`.                                                                                                              |
| An agent's LLM calls return `401` through an LLM proxy created after the upgrade                                                      | The new LLM proxy expects the API key header set in its LLM Service Provider, and the agent sends `API-Key`. Send the header that the agent's LLM configuration shows in the Console.                                                         |
| WSO2 Subscription: pods report `ImagePullBackOff`                                                                                     | The pod's namespace lacks `wso2-registry-credentials` (Agent Manager images) or `wso2-gateway-registry-credentials` (gateway images). Create it as in Step 1. For agent workloads, check the node's registry credentials (see Prerequisites). |
| WSO2 Subscription: an APIGateway stays `Programmed=False` and the operator cannot pull the gateway chart                              | The operator namespace lacks `wso2-gateway-registry-credentials`, or the Secret has no `username` and `password` keys. Recreate it as in Step 1.                                                                                              |
| OpenChoreo calls return empty lists or `403`                                                                                          | The plane upgrade reverted the `client_id` entitlement patch. Re-apply it as described in [Upgrade platform dependencies](/agent-platform/docs/next/guides/upgrades/platform-dependencies/.md).                                               |

## Recovery to v1.0.0[​](#recovery-to-v100 "Direct link to Recovery to v1.0.0")

There is no chart-only downgrade after the database migration. Keep the installation in maintenance mode.

1. Stop the Agent Manager API by scaling its Deployment to zero.

2. Restore the database:

   ```
   export PGPW="$(kubectl get secret amp-postgresql -n "$AMP_NS" -o jsonpath='{.data.password}' | base64 -d)"

   kubectl exec -i -n "$AMP_NS" amp-postgresql-0 -- env PGPASSWORD="$PGPW" \

     pg_restore -U agentmanager -d agentmanager --clean --if-exists \

     < upgrade-backup/ams-1.0.0.dump
   ```

3. Roll back the Agent Manager and extension releases with `helm rollback`. The gateway extension's bootstrap job also runs on rollback and needs Agent Manager to be reachable. If it fails, rerun the rollback.

4. Roll back the `gateway-operator` release to its revision before Step 6 (`helm history gateway-operator -n "$DP_NS"`). This restores chart 0.11.0 and all its v1.0.0 values, including the gateway chart 1.2.2 and images 1.2.1 from `ghcr.io`, and removes any registry values set in Step 6. For each gateway, restore the controller data from its Step 2 backup and redeploy it. A controller database that gateway 2026.10.0 migrated to version 6 cannot be used with 1.2.1.

5. Restore OpenChoreo and Thunder as described in [Upgrade platform dependencies](/agent-platform/docs/next/guides/upgrades/platform-dependencies/.md).

Restoring the database discards writes made after the backup. Repeat the Step 8 checks before you resume service.