NylorunDocsBeta
Deploy

Agent Substrate

Run pod Agents as Agent Substrate actors (preview).

Preview

Pods on Agent Substrate are a preview, from Runtime 0.45. Pods run on agent-sandbox by default (Kubernetes), and nothing changes for a Tenant that does not pass --backend substrate. Shell credentials are not available on Agent Substrate yet.

npx nylorun local sandbox enable --context <ctx> --backend substrate
npx nylorun local sandbox status

--backend substrate runs the Tenant's pod Agents as Agent Substrate v1.0 actors instead of agent-sandbox Sandboxes. An actor suspends to a snapshot of its memory and resumes where it stopped, which suits sessions that are short and mostly asleep. Agent manifests, Agents, sessions, and the APIs are the same on either backend: see Pod Agents.

Requirements

  • Kubernetes 1.37 with Agent Substrate v1.0 installed: its ate-system namespace and its api Service. A kind cluster on Docker Desktop, made with Substrate's own kind script, works. enable refuses a context without Substrate.
  • Pods that reach the Docker host. enable looks up host.docker.internal from a pod, three times, and refuses the context when it finds no address: pass --host-address with the Docker host's address as pods reach it (172.17.0.1 for kind on Linux).

What enable does

It touches only the context you name:

  • It creates the namespace nylorun-sbx-<tenant> and a ServiceAccount allowed only what the sandboxes service uses: its state ConfigMaps, a token for itself, a port-forward to Substrate's API, and Substrate's trust bundles.
  • It adds a hosts entry to CoreDNS, so that actors resolve the Tenant's edge, nylorun-sbx-<tenant>.edge.nylorun.internal, to the Docker host. When it cannot, it warns: add that name to the cluster's DNS yourself.
  • The edge is a TLS front in the sandboxes container, through which actors reach the Harness API and the gates. It is published on two host ports, 8793 and 8794 by default. Its certificate is made for the edge's name, and each actor is given its CA.
  • As one of Substrate's owners, with a 10-minute token for ate-system/ate-client that it deletes after use, it makes the Tenant's atespace, with the sandboxes service as its editor.

--tenant, --host-address, and --bind-address work as they do for agent-sandbox. npx nylorun local sandbox status names the backend, the atespace, and the edge. npx nylorun local sandbox disable works as it does for agent-sandbox; the atespace stays until you delete it with Substrate's tools.

Check the backend

npx nylorun doctor sandbox

Its pods line names the backend, the context, the namespace, and whether the sandboxes service is ready, such as Agent Substrate (preview), atespace nylorun-sbx-<tenant>, on <ctx>, namespace nylorun-sbx-<tenant>, ready.

The Management API answers the same in cluster: backend is agent-sandbox or substrate, and on Agent Substrate atespace names the Tenant's atespace.

const { cluster } = await admin.settings.sandbox.get();
console.log(cluster?.backend, cluster?.atespace); // "substrate" "nylorun-sbx-<tenant>"

In Studio, the Tenant's Compute tab shows it in its Sandbox section. See Get the Tenant's sandbox backend and Agent compute settings.

Network

Each Agent's network.allowedHosts, exact names and *.example.com patterns, becomes its actors' egress policy: only the edge and the allowed hosts resolve and connect, each host on ports 443 and 80.

Shell credentials, whose keys the egress gateway sets on each request, are not on Agent Substrate yet.

Wakes and suspends

A pod that is idle for lifecycle.idle suspends: the actor's memory snapshot is in durable storage before the suspend returns. The next turn resumes it where it stopped.

  • One resume per wake. A resume is one call to Substrate that may take up to 5 minutes, the Runtime's ready deadline, so a pod's first start on a node that lacks its image can pull the image. It is tried again only while Substrate is busy, and stops as soon as a later request replaces it.
  • A wake during a suspend waits for the suspend to finish.
  • The engine rejoins at once. After a restore, a pod's engine notices the jump in the clock, drops the connection attempt the snapshot caught in flight, and dials again without a pause (harness_clock_jumped in its log).
  • No DNS lookup of the edge. At a fresh start, the sandboxes service pins the edge's name in the actor's /etc/hosts when the image lets the engine's user write it. Substrate keeps that entry across restores, so a wake makes no lookup, which can be lost on a restored actor. On an image where the file is not writable, the actor resolves the edge through the cluster's DNS.

Each resume is logged as substrate resumed, with its time in ms:

npx nylorun local logs sandboxes

Crashes

An actor that crashes loses what its Agent wrote since its last suspend.

  • A running pod whose actor crashed is failed at once, with an agent.failed event on the Agent's events. A crash that an idle suspend finds is recorded the same way.
  • The event's reason gives Substrate's crash message and says what the Agent lost: Crashed: <message>. What the Agent wrote since its last suspend is lost; its next turn starts it from that suspend.
  • The next turn reverts the actor to its last suspend, once, and resumes it. A wake that finds the actor crashed, or that crashes it, does the same.
npx nylorun agents events <id> --types agent.failed

From code, client.agents.events(id, { types: ["agent.failed"] }). See Agents.

Next step

On this page