Agent Substrate
Run pod Agents as Agent Substrate actors (preview).
Preview
Pods on Agent Substrate are a preview, from Runtime 0.45. Pods run on
agent-sandbox by default (Kubernetes), and nothing
changes for a Tenant that does not pass --backend substrate. Shell
credentials are not available on Agent Substrate yet.
npx nylorun local sandbox enable --context <ctx> --backend substrate
npx nylorun local sandbox status--backend substrate runs the Tenant's pod Agents as
Agent Substrate v1.0 actors
instead of agent-sandbox Sandboxes. An actor suspends to a snapshot of its
memory and resumes where it stopped, which suits sessions that are short and
mostly asleep. Agent manifests, Agents, sessions, and the APIs are the same on
either backend: see Pod Agents.
Requirements
- Kubernetes 1.37 with Agent Substrate v1.0 installed: its
ate-systemnamespace and itsapiService. A kind cluster on Docker Desktop, made with Substrate's own kind script, works.enablerefuses a context without Substrate. - Pods that reach the Docker host.
enablelooks uphost.docker.internalfrom a pod, three times, and refuses the context when it finds no address: pass--host-addresswith the Docker host's address as pods reach it (172.17.0.1for kind on Linux).
What enable does
It touches only the context you name:
- It creates the namespace
nylorun-sbx-<tenant>and a ServiceAccount allowed only what the sandboxes service uses: its state ConfigMaps, a token for itself, a port-forward to Substrate's API, and Substrate's trust bundles. - It adds a
hostsentry to CoreDNS, so that actors resolve the Tenant's edge,nylorun-sbx-<tenant>.edge.nylorun.internal, to the Docker host. When it cannot, it warns: add that name to the cluster's DNS yourself. - The edge is a TLS front in the
sandboxescontainer, through which actors reach the Harness API and the gates. It is published on two host ports, 8793 and 8794 by default. Its certificate is made for the edge's name, and each actor is given its CA. - As one of Substrate's owners, with a 10-minute token for
ate-system/ate-clientthat it deletes after use, it makes the Tenant's atespace, with the sandboxes service as its editor.
--tenant, --host-address, and --bind-address work as they do for
agent-sandbox. npx nylorun local sandbox status names the backend, the
atespace, and the edge. npx nylorun local sandbox disable works as it does
for agent-sandbox; the atespace stays until you delete it with Substrate's
tools.
Check the backend
npx nylorun doctor sandboxIts pods line names the backend, the context, the namespace, and whether the
sandboxes service is ready, such as
Agent Substrate (preview), atespace nylorun-sbx-<tenant>, on <ctx>, namespace nylorun-sbx-<tenant>, ready.
The Management API answers the same in cluster: backend is
agent-sandbox or substrate, and on Agent Substrate atespace names the
Tenant's atespace.
const { cluster } = await admin.settings.sandbox.get();
console.log(cluster?.backend, cluster?.atespace); // "substrate" "nylorun-sbx-<tenant>"In Studio, the Tenant's Compute tab shows it in its Sandbox section. See Get the Tenant's sandbox backend and Agent compute settings.
Network
Each Agent's network.allowedHosts, exact names and *.example.com patterns,
becomes its actors' egress policy: only the edge and the allowed hosts resolve
and connect, each host on ports 443 and 80.
Shell credentials, whose keys the egress gateway sets on each request, are not on Agent Substrate yet.
Wakes and suspends
A pod that is idle for lifecycle.idle suspends: the actor's memory snapshot
is in durable storage before the suspend returns. The next turn resumes it
where it stopped.
- One resume per wake. A resume is one call to Substrate that may take up to 5 minutes, the Runtime's ready deadline, so a pod's first start on a node that lacks its image can pull the image. It is tried again only while Substrate is busy, and stops as soon as a later request replaces it.
- A wake during a suspend waits for the suspend to finish.
- The engine rejoins at once. After a restore, a pod's engine notices the
jump in the clock, drops the connection attempt the snapshot caught in
flight, and dials again without a pause (
harness_clock_jumpedin its log). - No DNS lookup of the edge. At a fresh start, the sandboxes service pins
the edge's name in the actor's
/etc/hostswhen the image lets the engine's user write it. Substrate keeps that entry across restores, so a wake makes no lookup, which can be lost on a restored actor. On an image where the file is not writable, the actor resolves the edge through the cluster's DNS.
Each resume is logged as substrate resumed, with its time in ms:
npx nylorun local logs sandboxesCrashes
An actor that crashes loses what its Agent wrote since its last suspend.
- A running pod whose actor crashed is
failedat once, with anagent.failedevent on the Agent's events. A crash that an idle suspend finds is recorded the same way. - The event's reason gives Substrate's crash message and says what the Agent
lost:
Crashed: <message>. What the Agent wrote since its last suspend is lost; its next turn starts it from that suspend. - The next turn reverts the actor to its last suspend, once, and resumes it. A wake that finds the actor crashed, or that crashes it, does the same.
npx nylorun agents events <id> --types agent.failedFrom code, client.agents.events(id, { types: ["agent.failed"] }). See
Agents.