What a Kubernetes warm pool cannot prewarm for a stateful AI agent

TL;DR. Prewarm the runtime, not the tenant. A Kubernetes warm pool can pay for image pulls, container startup, runtime initialization, and even blank storage before a request arrives. It cannot safely guess which existing agent disk, tenant identity, or final credentials the next request will need. Those belong behind an atomic, claim-once transition. If that late binding fails, the correct fallback is the normal cold path, not a partly claimed sandbox.

A warm pool can prewarm a container, but it cannot prewarm a tenant. That is the line we found while cutting agent wake latency at plori, down to about 2.6 s median from wake to first streamed token. Each of our agents runs on its own computer with a persistent account disk, a per-agent identity, and credentials scoped to one account. Prewarming containers ahead of demand removed the largest share of cold-start time. Pre-assigning disks or credentials would have given the wrong user the right computer.

The design that worked splits activation into two phases: a tenant-agnostic phase that runs ahead of demand (image pull, process init, blank storage) and a tenant-bound phase that waits for a real claim (disk attachment, identity, credentials). The first phase is what a warm pool buys. The second is what it must never guess.

Prewarming is four different operations

Treating "the sandbox" as one object hides the security boundary. There are at least four classes of state:

State	Safe to create before a claim?	What the pool actually buys
Runtime and image	Yes	Image pull, container start, agent runtime initialization
Blank resources	Usually	Empty volumes, network allocation, generic capacity
Existing persistent state	No, not generically	The specific agent workspace must be selected and attached
Tenant identity and final credentials	No	Authority must be derived from the claimant and bound to one sandbox

The second row matters because "storage cannot be prewarmed" is too broad. Kubernetes can provision blank storage ahead of time. The current Agent Sandbox API even supports storage customization through claims, and recent releases added volume claim templates to that path. A blank volume is still tenant-agnostic. A returning agent's existing disk is not.

The distinction is provenance, not medium. An empty persistent volume can sit in a pool without knowing its future owner. A volume that already contains an agent's repository, artifacts, and history carries identity even if Kubernetes represents both resources as PVCs.

Prewarm everything that has no tenant meaning

The safe half of a warm pool should be deliberately boring. Pull the image. Start the container. Initialize language runtimes. Load immutable libraries. Establish generic network plumbing. Create an empty workspace if the product can assign fresh storage to a new agent. In plori's pool, that list is the container image, the agent runtime, and language toolchains.

This is where most cold-start time lives. None of these operations needs to know who will make the next request, and every result can be discarded if it becomes stale. That gives the pool a clean replacement rule: a sandbox with the wrong image version or failed health check is deleted, not repaired while a claimant waits.

Kubernetes SIG Apps now expresses this pattern directly through Agent Sandbox: a SandboxWarmPool maintains ready sandboxes and a SandboxClaim acquires one. As of July 2026, the project is moving quickly and its current releases use v1beta1 APIs. The abstraction is useful, but it does not remove the need to define what "ready" means for your application.

For a stateful agent, ready should mean "capable of being bound," not "already impersonating someone." A health check can prove the runtime accepts a claim. It should not require a real user's disk or credentials to do so.

Why is claiming a sandbox a one-way door?

At claim time, the platform finally knows the requesting tenant and the target agent. It can select the existing workspace, attach it, establish the sandbox's final identity, and deliver narrowly scoped authority. These operations belong in one logical transition:

tenant-agnostic prewarm -> atomic claim -> tenant-bound attach

The transition must be claim-once. After a sandbox accepts one tenant, it never returns to the generic pool. Deleting it is cheaper than proving that every process, file, socket, cache entry, and in-memory secret has been scrubbed. More importantly, destroy-on-release makes isolation review finite. One sandbox has zero tenants before claim and exactly one afterward.

Two of our control-plane replicas once selected the same ready sandbox simultaneously, a bug that briefly caused duplicate pod claims. The fix was not better caching; it was making the sandbox itself enforce a one-way state transition that accepts one claimant and rejects the rest. Relabeling or removing a sandbox from the pool is still useful for discovery, but discovery metadata should not be the final arbiter.

Identity deserves the same late-binding treatment. A pool can contain a trusted mechanism that obtains credentials, but not the next tenant's final credential. The distinction is easy to miss: prewarming the broker is safe; pre-assigning the principal is not. As of mid-2026, the Agent Sandbox roadmap lists dynamic sandbox or pod identity association at claim time as planned work. That is a useful signal: this boundary is an application concern today, not a solved side effect of having a WarmPool CRD.

How does existing state reach a pooled sandbox?

A returning agent should see the same files after its previous computer disappeared. That continuity comes from storage outliving the sandbox, not from keeping one warm replica per agent. The disk survives replacement of the machine underneath it, and our persistent-disk benchmark measures what that persistence costs.

At activation, a pooled runtime attaches the selected agent disk. This keeps the expensive generic startup work off the request path while leaving the user-specific mount where it belongs. It also makes the remaining latency honest. A warm pool can remove image and process startup, but it cannot make tenant-specific authentication and attachment take zero time. In our cold-start benchmark, that tenant-bound step is about a second at the median.

This model also prevents a misleading capacity calculation. A pool of ten blank sandboxes is ten units of ready compute. It is not ten warmed copies of every user's state. If your product requires memory snapshots or per-agent local caches, those are a different tier with different invalidation and isolation costs. Do not hide them inside the word "warm."

Why is cold boot the safety net, not the failure mode?

Late binding adds new failure modes: the chosen sandbox may die, lose a race, reject the claim, or fail while attaching state. The warm path is an optimization, so none of those failures should make an activation fail when the ordinary cold path would have worked.

The rule we use is:

1. Claim a ready, tenant-agnostic sandbox.
2. Bind identity and attach state.
3. Publish it as available only after both are ready.
4. If any step fails, discard that sandbox and start through the cold path.

Do not return a claimed sandbox to the pool after an ambiguous failure. The control plane may not know whether tenant data reached it. Also do not report success after only the claim acknowledgement. Availability means the agent can access the correct workspace under the correct authority.

Destroying the sandbox is not free either, and discarding one is only correct if the teardown actually completes. A mount that outlives the pod it belonged to is its own outage, which is what happened to us in we let 443 dead FUSE mounts pile up and it wedged kubelet.

This fallback has an operational benefit beyond reliability. It gives every warm-pool change a simple invariant: disabling the pool may increase latency, but it must not change which tenant gets which state or whether a valid activation succeeds.

Limitations

This design does not eliminate all cold starts. Pool depletion still exposes the normal boot path, and tenant-specific disk attachment and identity issuance remain on the claim path. It also assumes a disposable runtime whose durable truth lives elsewhere. If the only valid state is an in-memory process image, a generic claim-once pool is the wrong abstraction; snapshot restore or dedicated suspended sandboxes may fit better, with a larger security and invalidation surface.

FAQ

Can a Kubernetes warm pool prewarm persistent volumes?

It can pre-provision blank persistent volumes. That is different from attaching an existing tenant's agent disk. The former is generic capacity; the latter contains tenant-bound state and should be selected at claim time.

Why not scrub a sandbox and return it to the pool?

Because complete scrubbing is hard to prove across processes, memory, filesystem state, caches, sockets, and credentials. Claim-once plus destroy-on-release turns a broad sanitization problem into a much smaller lifecycle rule.

Should credentials ever exist in a warm sandbox?

A generic sandbox may contain a trusted credential acquisition mechanism. It should not contain the final credential or principal for an unknown future tenant. Bind that authority only after the claim identifies the tenant.

What should happen when late binding fails?

Discard the ambiguous sandbox and use the same cold activation path that works without a pool. A warm pool is allowed to improve latency, not to weaken isolation or availability.