Kata Containers Isolation Tier

Run agents in an environment under Kata Containers — each agent boots in a lightweight QEMU/KVM virtual machine with its own guest kernel, so there is no shared kernel with the host at all. This is the strongest isolation tier, intended for the most untrusted or multi-tenant agent workloads.

Read the gVisor isolation tier guide first if you haven't — Kata uses the same environment-level mechanism and the same dedicated-node model; this page covers what is Kata-specific.

Hard requirement: KVM (nested virtualization)

Kata boots a real VM per agent, so the node must expose /dev/kvm:

• ✅ Bare-metal Linux with VT-x (Intel) or AMD-V
• ✅ A nested-virtualization-capable cloud VM — for example GCP Intel N2 with --enable-nested-virtualization, or AWS EKS bare-metal (*.metal) instances
• ❌ macOS / Colima — no nested virtualization; use gVisor locally instead
• ❌ GCP E2, AMD N2D, ARM T2A machine types, and Container-Optimized OS (read-only filesystem)
• ❌ k3d / kind node containers — even on a KVM-capable host, the Kata guest-agent handshake cannot complete inside a container-based node; Kata needs a real Linux node

Verify on the node itself, not on your workstation. From a machine with kubectl access, kubectl debug runs the check on the node:

kubectl debug node/<node-name> -it --profile=sysadmin --image=busybox:1.36 -- \
  chroot /host sh -c 'ls -l /dev/kvm; cat /sys/module/kvm_intel/parameters/nested'
# expect: a /dev/kvm character device, then Y (Intel)

This leaves a completed node-debugger-<node-name>-… pod in the current namespace; delete it afterwards. With SSH access to the node, run the two commands there instead.

How It Works

Identical to gVisor, with one naming detail: the isolation tier is called kata, but the RuntimeClass (and containerd handler) it maps to is kata-qemu — that is the handler kata-deploy registers.

Environment (isolation tier: kata)
  └─ deploy / promote → runtimeClassName: kata-qemu
       └─ RuntimeClass "kata-qemu" scheduling → pod on the Kata node, QEMU/KVM VM + guest kernel

Like gVisor, Kata uses a dedicated node labeled kata=true and tainted kata=true:NoSchedule, so existing runc (and gVisor) agents are never touched — no downtime.

Requirements

Requirement	Details
A new (empty) Linux node	Joined to the same Kubernetes cluster; dedicated to Kata agents
Virtualization	/dev/kvm present and functional (see warning above)
Architecture	x86_64
OS	Ubuntu (or similar) with systemd + containerd — not Container-Optimized OS
Access	kubectl and helm access to the cluster (the installer runs from your machine, not on the node)

Set Up a Kata Node

${AMP_DIST}

The commands below run scripts and manifests from the release bundle. If you do not already have it extracted, fetch it first and point AMP_DIST at it — the same bundle the install guides use:

export VERSION="0.0.0-dev"
curl -fsSL -O "https://github.com/wso2/agent-manager/releases/download/amp/v${VERSION}/wso2-agent-manager-${VERSION}.tar.gz"
tar xzf "wso2-agent-manager-${VERSION}.tar.gz"
export AMP_DIST="$PWD"

On a node-side step, extract it on that node: the installer script runs there, not on your workstation.

Step 1: Add a new node to the cluster

Provision a KVM-capable Linux node matching the requirements above and join it to your cluster using your platform's standard mechanism. Leave it empty.

Taint the node as early as possible

If your platform lets you set taints at node-pool creation, apply kata=true:NoSchedule there. An untainted node starts accepting regular platform pods immediately. The installer restarts the node's container runtime; running pods normally survive that, but if the node has already picked up workloads you can kubectl cordon + kubectl drain --ignore-daemonsets --delete-emptydir-data it first. To share the node with other pods instead, leave it untainted and run the installer with KATA_TAINT=false.

Step 2: Run the installer from a machine with kubectl and helm access

Unlike the gVisor installer (which runs on the node), the Kata installer runs from any machine with kubectl and helm access and does everything through the Kubernetes API:

KATA_NODES="<node-name>" bash "${AMP_DIST}/deployments/setup/install-kata.sh"

What it does:

1. Pre-flights /dev/kvm on every target node (via a short-lived privileged pod) and aborts if any node lacks it.
2. Labels the node(s) kata=true.
3. Installs upstream kata-deploy from its Helm chart (kata-containers KATA_VERSION, default 4.2.0), scoped to the labeled node(s) so your other nodes are never reconfigured. The chart wires the node's containerd itself on standard Kubernetes, k3s and RKE2; the installer detects which one from the node's kubelet version. Nodes set up by the earlier manifest-based installer are migrated on the first run.
4. Applies the kata-qemu RuntimeClass and boots a test pod under it on every target node. The installer reports success only once that pod has run in its own VM, with a kernel different from the node's.
5. Taints the node(s) kata=true:NoSchedule. Set KATA_TAINT=false to leave them schedulable for other pods.
6. Lets the Fluent Bit log DaemonSet tolerate the taint, so agent logs are collected. It checks the toleration stored in your observability-logs-opensearch Helm release and, if it is missing, sets the chart value fluent-bit.tolerations (a helm upgrade --reuse-values at the release's installed chart version). It patches the DaemonSet only when that release cannot be upgraded with Helm, and then prints the helm upgrade command to make it durable, because a patch is lost whenever the DaemonSet is recreated.

Upgrading from the earlier guide

Re-run the installer on a node set up with the earlier version of this guide; it replaces the manifest-based kata-deploy with the Helm release. On RKE2, the /etc/containerd/config.toml stub and the config.toml.tmpl that guide had you write can stay: kata-deploy 4.x writes its runtime entry where RKE2 reads it alongside that template.

Verify

kubectl get runtimeclass kata-qemu
kubectl -n kube-system rollout status daemonset/kata-deploy
kubectl get node <node-name> --show-labels   # kata=true

Keep the Fluent Bit toleration on later upgrades

The toleration is stored in the observability-logs-opensearch release's values, so it must be present on every upgrade that re-renders the Fluent Bit DaemonSet. An upgrade with --reset-values, or one that passes new --set/-f values without --reuse-values, drops it unless you pass --set "fluent-bit.tolerations[0].operator=Exists" again. Agents on the Kata node then stop producing logs with no error.

Local development

There is no macOS-local path for Kata (no nested virtualization, and k3d node containers cannot boot Kata VMs). On a KVM-capable Linux host running the repository's dev k3d cluster:

make setup-kata

This joins the host itself to the cluster as a real (non-Docker) k3s agent node and then runs install-kata.sh against it.

Create a Kata Environment

Isolation tier is set at environment creation time. Open the Create Environment drawer in the console (Environments → Create Environment), select Sandboxing T3 (Kata Containers) from the Isolation Tier dropdown, and run the generated command — it includes ISOLATION_TIER=kata automatically.

If you script environment creation directly, set the variable yourself:

ISOLATION_TIER=kata \
  ENV_NAME=secure-prod \
  DISPLAY_NAME="Secure Prod" \
  AGENT_MANAGER_TOKEN=<token> \
  bash "${AMP_DIST}/deployments/scripts/add-environment.sh"

The tier appears as a Sandboxing T3 shield chip in the environment list, and as a shield icon next to the environment name on the agent's Deploy and Overview pages — hover over it to confirm Sandboxing Tier 3 — Kata Containers.

A new environment is not part of any deployment pipeline yet. Add it to your project's pipeline (Managing Deployment Pipelines), then deploy or promote an agent to it as usual.

Verify VM Isolation

A Kata agent runs a different kernel from the node — that's the definitive check:

kubectl get pod <agent-pod> -n <namespace> -o jsonpath='{.spec.runtimeClassName}'   # kata-qemu

kubectl exec <agent-pod> -n <namespace> -- uname -r
# Differs from the node's own `uname -r`

kubectl exec <agent-pod> -n <namespace> -- grep hypervisor /proc/cpuinfo
# hypervisor flag present

If the pod reports the same kernel as the node, it is not running under Kata (the RuntimeClass wasn't applied or the pod fell back to runc).

Troubleshooting

Pod stuck in ContainerCreating

Almost always missing nested virtualization. Confirm on the node itself (with the kubectl debug command under the KVM requirement above, or over SSH):

ls -l /dev/kvm
cat /sys/module/kvm_intel/parameters/nested   # expect: Y
kubectl describe pod <pod> -n <ns>            # look for kvm / RuntimeClass errors

No /dev/kvm → move to a bare-metal or nested-virt-enabled host (e.g. GCP Intel N2 with --enable-nested-virtualization).

no runtime handler kata-qemu errors

The RuntimeClass exists but containerd on the node has no kata-qemu runtime — the install didn't complete or containerd wasn't reloaded:

kubectl -n kube-system rollout status daemonset/kata-deploy
kubectl -n kube-system logs -l name=kata-deploy
kubectl get node <node-name> -L katacontainers.io/kata-runtime   # true once kata-deploy has wired the node

Re-running install-kata.sh is safe: it re-applies the kata-deploy chart and boots a test pod on every target node again, failing with the reason if Kata still cannot run there.

Pod runs but no logs / metrics / traces / try-it

This is cross-node networking or log-collection scheduling, not Kata itself — the failure modes and fixes are identical to gVisor's. Work through the gVisor troubleshooting section, substituting runtimeClassName: kata-qemu in the test pod.

Performance notes

• Booting a VM adds startup latency compared to runc/gVisor; Agent Manager's warm pools absorb most of it.
• kubectl port-forward should be avoided for Kata pods — reach agents through their Service/gateway endpoint.
• Usage metrics are attributed to the sandbox VM as a whole (reported under a sandbox container label), including the guest kernel's footprint.

Next Steps

• Lighter-weight syscall isolation without KVM: gVisor isolation tier.
• Environment management — pipelines, promotion, and environment settings.