Sandboxing AI Agents on Kubernetes: gVisor, Kata, and the Sandbox CRD

An agent that writes and runs its own code is untrusted code holding your credentials. Most teams start it in an ordinary Pod on a shared node pool, next to the ingress controller, with a service account token mounted at /var/run/secrets. That holds until the first prompt injection or the first rm -rf in generated cleanup code. Sandboxing AI agents on Kubernetes is a runtime problem rather than a prompt problem, and the upstream API for it is now beta and shipping weekly. The useful parts are the Sandbox CRD itself, the gVisor or Kata decision, and two things almost nobody reads before rolling it out: the warm pool throughput numbers and the field that stops abandoned agents from billing forever.

We have written before about isolating MCP servers at the process level and about calibrating what agents are allowed to touch in CI/CD. This one picks up at the layer below both: the kernel boundary and the Kubernetes object that owns it.

Why a Deployment is the wrong shape for an agent

An agent session is a singleton with a name, a home directory it expects to find again, and an end. Deployments give you interchangeable replicas. StatefulSets give you numbered ordinals and a scaling model you do not want. The upstream project is blunt about the workaround most teams reach for, saying that approximating this with "StatefulSets (size 1), Services, and PersistentVolumeClaims" is "cumbersome and lacks specialized lifecycle management like hibernation" (Agent Sandbox overview).

The Sandbox CRD, developed under SIG Apps, is the replacement. The core API group is agents.x-k8s.io/v1beta1 and the spec is small enough to read in one sitting: podTemplate, volumeClaimTemplates, service, operatingMode (Running or Suspended, defaulting to Running), shutdownTime, and shutdownPolicy (Delete or Retain, defaulting to Retain). Status carries serviceFQDN, service, podIPs, nodeName, selector, and conditions (API reference). A second group, extensions.agents.x-k8s.io/v1beta1, adds SandboxTemplate, SandboxWarmPool, and SandboxClaim.

Release v1.0.0 removed the v1alpha1 API entirely, so every CRD serves v1beta1 only. Version v1.0.6 landed on 8 October 2026, a week after v1.0.5 (releases), which tells you the velocity: pin a version in your manifests rather than tracking latest.

If you currently run agents as single-replica StatefulSets with a sidecar that kills them on a timer, delete the sidecar. The timer is a spec field now.

Choosing an isolation runtime

The Sandbox controller does not isolate anything itself. It is an orchestrator that manages Pods configured to use a secure runtime through RuntimeClass, which means the real decision is which runtime you install.

| Option | runtimeClassName | Boundary | Requires | Choose it when | |---|---|---|---|---| | Standard container | none | Shared host kernel, namespaces and cgroups | Nothing | The agent only runs code you wrote and reviewed | | gVisor | gvisor | User-space kernel intercepting syscalls | gVisor installed on the nodes, or GKE Sandbox node pools | The agent executes model-generated code (the common case) | | Kata Containers | kata-qemu | A real VM with its own kernel per Pod | Nested virtualization, containerd | Tenant separation is a contractual boundary, or the workload needs the full syscall surface | | GKE microVM | microvm | Managed microVM | GKE Sandbox | You want Kata-grade isolation without running the hypervisor plumbing |

Kata has a hard prerequisite worth checking before you plan a migration. Nested virtualization must be enabled on the host, verified with cat /sys/module/kvm_intel/parameters/nested returning Y or 1, and the documentation states that "only the containerd runtime is supported without additional adjustments" (Kata Containers isolation). On a managed cluster that often decides the question for you.

Our default is gVisor for agents running generated code, and Kata only when a second party is paying for the isolation guarantee. Two runtimes in one cluster is two node pools, two sets of node-level debugging, and two upgrade paths, so start with one.

What gVisor takes away

The isolation is not free, and the subtractions catch teams mid-rollout. Google's GKE Sandbox documentation lists them for both backends: "Linux kernel security modules such as seccomp, AppArmor, SELinux...aren't supported in sandboxed Pods", and "Memory usage metrics at the container level aren't supported" (GKE Sandbox).

Read that first one twice. If your Pod Security Standards baseline requires a seccomp profile, your admission policy and your sandbox runtime now disagree, and the policy will reject the very Pods you hardened. Fix the policy, not the runtime.

Specific to gVisor, the same page records that privileged mode, raw block volumes, hostPath storage, and port forwarding are unsupported. For microVMs, hostNetwork: true is unsupported. Cloud Service Mesh does not work for sandboxes in Autopilot clusters. The missing port forwarding is the one that breaks developer workflow on day one, because kubectl port-forward into an agent is how everyone debugs. The project's answer is a Sandbox Router that routes on an X-Sandbox-ID header instead (gVisor isolation).

On performance, be careful with the numbers you read elsewhere. The official gVisor performance guide measures memory, CPU, system calls, start-up time, network, and file system separately, and it publishes no single multiplier because the answer depends on the workload and the platform. It is explicit that "gVisor does not perform emulation or otherwise interfere with the raw execution of CPU instructions" (gVisor performance guide). So a CPU-bound inference loop is not where you pay. A build step that opens ten thousand files is. Measure your own agent's syscall profile for an hour before you accept a blog post's percentage, including this one.

Enforce the runtime instead of requesting it

A runtimeClassName in a template is a request that any author can forget or remove. The upstream repository ships a ValidatingAdmissionPolicy example that closes this, and its test case is the interesting part: an otherwise clean Sandbox with a privileged: true initContainer. Policies that only inspect spec.containers wave that through. The example evaluates containers, initContainers, and ephemeralContainers together, and it enforces gVisor, no host network or PID or IPC namespaces, no hostPort, non-root execution, dropping all Linux capabilities, CPU and memory limits, no mounted service account token, and gVisor node pool targeting (secure sandbox policy).

The manifest shape it accepts looks like this:

apiVersion: agents.x-k8s.io/v1beta1
kind: Sandbox
metadata:
  name: secure-sandbox
spec:
  podTemplate:
    spec:
      runtimeClassName: gvisor
      hostNetwork: false
      automountServiceAccountToken: false
      nodeSelector:
        sandbox.gke.io/runtime: gvisor
      tolerations:
      - key: sandbox.gke.io/runtime
        operator: Equal
        value: gvisor
        effect: NoSchedule
      containers:
      - name: main-agent
        image: python:3.11-slim
        command: ["python3", "-c", "import time; time.sleep(3600)"]
        securityContext:
          runAsNonRoot: true
          runAsUser: 1000
          capabilities:
            drop: ["ALL"]
        resources:
          limits:
            cpu: "500m"
            memory: "512Mi"

Note automountServiceAccountToken: false. A kernel boundary around a Pod that still holds a cluster credential buys you very little, and this is the line most teams omit.

Writing this as a ValidatingAdmissionPolicy rather than a webhook is the right call on current clusters, for the reasons in our guide to migrating admission webhooks to admission policies. One less deployment in the request path, and no webhook outage taking your agent fleet with it.

Warm pools, and the numbers behind sharding

Starting a gVisor Pod, pulling an image, and mounting a volume is slow enough that an interactive agent feels broken. The extensions API answers this with a pre-warmed pool. A SandboxTemplate defines the shape, a SandboxWarmPool keeps replicas of them idle, and a SandboxClaim adopts one:

apiVersion: extensions.agents.x-k8s.io/v1beta1
kind: SandboxWarmPool
metadata:
  name: sandboxwarmpool-example
  namespace: default
spec:
  updateStrategy:
    type: Recreate
  replicas: 1
  sandboxTemplateRef:
    name: secure-datascience-template
---
apiVersion: extensions.agents.x-k8s.io/v1beta1
kind: SandboxClaim
metadata:
  name: my-secure-sandbox
  namespace: default
spec:
  warmPoolRef:
    name: sandboxwarmpool-example
  lifecycle:
    shutdownPolicy: Delete

The project publishes throughput figures that almost nobody writing about agent sandboxes seems to have read, and they change how you lay out a cluster. Draining an already-warm pool reaches roughly 300 claims per second per cluster at p90 latency of 200 ms or less, because binding is an optimistic lock and a status update with no Pod creation. Sustained operation, where the pool refills while claims arrive, settles at 70 to 85 sandboxes per second per SandboxWarmPool, limited by per-pool workqueue serialization, the expectations gate, and kube-scheduler's default QPS of 50 (performance tuning).

That ceiling is per pool, not per cluster, which is the actionable part. Above 30 claims per second, or pools beyond 500 to 1,000 replicas, the guidance is to shard: four pools of 450 replicas instead of one pool of 1,800, with --sandbox-warm-pool-concurrent-workers set to the number of active pools. Both layouts hold 1,800 warm sandboxes. Only the sharded one can refill them in parallel.

Two more defaults are worth changing deliberately. Setting --sandbox-warm-pool-max-refill-rate to 50 to 100 per pool paces refill as a token bucket instead of letting unpaced bursts of up to 300 sandboxes spike etcd MVCC growth and saturate the API server watch ring buffer. And leave --sandbox-write-behind-window alone unless you have evidence: at 250 ms it cuts 409 write conflicts by 44% and burst p50 by 13%, while raising sustained warm-adoption p50 by 58%. The same source notes the expectations gate can freeze replenishment for 10 to 30 seconds under heavy load, which is the symptom to look for before you start tuning worker counts.

shutdownTime is the cost control

Agents do not reliably exit. They wait on input that never arrives, or they finish and leave a Pod with a mounted volume running at full requests.

Three fields handle this, and the defaults are not the safe ones. operatingMode: Suspended terminates the Pod while keeping the Sandbox object, its identity, and its storage, so a resumed session finds its files. shutdownTime is an absolute RFC3339 timestamp after which the sandbox must terminate. shutdownPolicy defaults to Retain, which keeps the object with a SandboxExpired condition rather than deleting it; SandboxClaim extends the same field with Delete, DeleteForeground, and Retain, also defaulting to Retain.

So a claim created with no shutdownTime runs indefinitely, and the documented interaction with restartPolicy makes it worse: Kubernetes defaults an omitted restartPolicy to Always, and with Always the Pod never reaches a terminal state, so ttlSecondsAfterFinished never fires. You get a Pod that restarts forever inside a sandbox nobody is watching. Set shutdownTime on every claim your SDK creates, and set restartPolicy: Never for one-shot agent runs.

The bill here is node-hours in a sandbox-capable node pool plus the idle warm pool, and the warm pool is the line item people forget. A pool of 450 replicas reserves 450 Pods worth of CPU and memory requests whether or not a single agent claims one, and with container-level memory metrics unavailable under gVisor you cannot see inside them the way you normally would. Size the pool from measured claim rate, not from peak hope, and apply the same rigour we use in our Kubernetes cost optimization framework.

What we would ship first

A rollout that works, in order:

1. Install one isolation runtime, gVisor, on a dedicated tainted node pool. Do not mix Kata in until the first one is boring.
2. Install the controller at a pinned version, not latest, and move one non-critical agent workload to a Sandbox with automountServiceAccountToken: false.
3. Fix your Pod Security Standards policy for the missing seccomp support before it rejects the sandboxed Pods, and deploy the Sandbox Router so developers keep a debugging path.
4. Add the ValidatingAdmissionPolicy covering initContainers and ephemeralContainers, in dry-run first, then enforcing.
5. Only then add a SandboxWarmPool, sized from a week of measured claim rate, with shutdownTime set by the SDK on every claim.

Skipping step 3 is how most of these rollouts stall, because the platform team discovers the policy conflict after the agent team has already announced the migration.

When to Get Help

Agent isolation sits on the seam between platform engineering and security, which is exactly where ownership tends to be unclear. If you are deciding between gVisor and Kata, reconciling sandbox runtimes with an existing Pod Security Standards baseline, or sizing warm pools for an agent product you are about to put in front of customers, this is work we do with platform teams. Tell us what you are building and we will tell you which of these steps you can skip.