# Agent Substrate (/docs/deploy/substrate)



<Callout title="Preview" type="warn">
  Pods on Agent Substrate are a preview, from Runtime 0.45. Pods run on
  agent-sandbox by default ([Kubernetes](/docs/deploy/kubernetes)), and nothing
  changes for a Tenant that does not pass `--backend substrate`. Shell
  credentials are not available on Agent Substrate yet.
</Callout>

```sh
npx nylorun local sandbox enable --context <ctx> --backend substrate
npx nylorun local sandbox status
```

`--backend substrate` runs the Tenant's pod Agents as
[Agent Substrate](https://github.com/agent-substrate/substrate) v1.0 actors
instead of agent-sandbox Sandboxes. An actor suspends to a snapshot of its
memory and resumes where it stopped, which suits sessions that are short and
mostly asleep. Agent manifests, Agents, sessions, and the APIs are the same on
either backend: see [Pod Agents](/docs/deploy/kubernetes#pod-agents).

## Requirements [#requirements]

* Kubernetes 1.37 with Agent Substrate v1.0 installed: its `ate-system`
  namespace and its `api` Service. A kind cluster on Docker Desktop, made with
  Substrate's own kind script, works. `enable` refuses a context without
  Substrate.
* Pods that reach the Docker host. `enable` looks up `host.docker.internal`
  from a pod, three times, and refuses the context when it finds no address:
  pass `--host-address` with the Docker host's address as pods reach it
  (`172.17.0.1` for kind on Linux).

## What `enable` does [#what-enable-does]

It touches only the context you name:

* It creates the namespace `nylorun-sbx-<tenant>` and a ServiceAccount allowed
  only what the sandboxes service uses: its state ConfigMaps, a token for
  itself, a port-forward to Substrate's API, and Substrate's trust bundles.
* It adds a `hosts` entry to CoreDNS, so that actors resolve the Tenant's edge,
  `nylorun-sbx-<tenant>.edge.nylorun.internal`, to the Docker host. When it
  cannot, it warns: add that name to the cluster's DNS yourself.
* The edge is a TLS front in the `sandboxes` container, through which actors
  reach the Harness API and the gates. It is published on two host ports, 8793
  and 8794 by default. Its certificate is made for the edge's name, and each
  actor is given its CA.
* As one of Substrate's owners, with a 10-minute token for
  `ate-system/ate-client` that it deletes after use, it makes the Tenant's
  atespace, with the sandboxes service as its editor.

`--tenant`, `--host-address`, and `--bind-address` work as they do for
agent-sandbox. `npx nylorun local sandbox status` names the backend, the
atespace, and the edge. `npx nylorun local sandbox disable` works as it does
for agent-sandbox; the atespace stays until you delete it with Substrate's
tools.

## Check the backend [#check-the-backend]

```sh
npx nylorun doctor sandbox
```

Its `pods` line names the backend, the context, the namespace, and whether the
sandboxes service is ready, such as
`Agent Substrate (preview), atespace nylorun-sbx-<tenant>, on <ctx>, namespace nylorun-sbx-<tenant>, ready`.

The Management API answers the same in `cluster`: `backend` is
`agent-sandbox` or `substrate`, and on Agent Substrate `atespace` names the
Tenant's atespace.

```ts
const { cluster } = await admin.settings.sandbox.get();
console.log(cluster?.backend, cluster?.atespace); // "substrate" "nylorun-sbx-<tenant>"
```

In Studio, the Tenant's **Compute** tab shows it in its Sandbox section. See
[Get the Tenant's sandbox backend and Agent compute settings](/reference/management/settings/get-the-tenants-sandbox-backend-and-agent-compute-settings).

## Network [#network]

Each Agent's `network.allowedHosts`, exact names and `*.example.com` patterns,
becomes its actors' egress policy: only the edge and the allowed hosts resolve
and connect, each host on ports 443 and 80.

Shell credentials, whose keys the egress gateway sets on each request, are
not on Agent Substrate yet.

## Wakes and suspends [#wakes-and-suspends]

A pod that is idle for `lifecycle.idle` suspends: the actor's memory snapshot
is in durable storage before the suspend returns. The next turn resumes it
where it stopped.

* **One resume per wake.** A resume is one call to Substrate that may take up
  to 5 minutes, the Runtime's ready deadline, so a pod's first start on a node
  that lacks its image can pull the image. It is tried again only while
  Substrate is busy, and stops as soon as a later request replaces it.
* **A wake during a suspend** waits for the suspend to finish.
* **The engine rejoins at once.** After a restore, a pod's engine notices the
  jump in the clock, drops the connection attempt the snapshot caught in
  flight, and dials again without a pause (`harness_clock_jumped` in its log).
* **No DNS lookup of the edge.** At a fresh start, the sandboxes service pins
  the edge's name in the actor's `/etc/hosts` when the image lets the engine's
  user write it. Substrate keeps that entry across restores, so a wake makes
  no lookup, which can be lost on a restored actor. On an image where the file
  is not writable, the actor resolves the edge through the cluster's DNS.

Each resume is logged as `substrate resumed`, with its time in `ms`:

```sh
npx nylorun local logs sandboxes
```

## Crashes [#crashes]

An actor that crashes loses what its Agent wrote since its last suspend.

* A running pod whose actor crashed is `failed` at once, with an
  `agent.failed` event on the Agent's events. A crash that an idle suspend
  finds is recorded the same way.
* The event's reason gives Substrate's crash message and says what the Agent
  lost:
  `Crashed: <message>. What the Agent wrote since its last suspend is lost; its next turn starts it from that suspend.`
* The next turn reverts the actor to its last suspend, once, and resumes it. A
  wake that finds the actor crashed, or that crashes it, does the same.

```sh
npx nylorun agents events <id> --types agent.failed
```

From code, `client.agents.events(id, { types: ["agent.failed"] })`. See
[Agents](/docs/run/agents).

## Next step [#next-step]

<Cards>
  <Card title="Kubernetes" description="Pod Agents, their versions, and the Tenant's limits." href="/docs/deploy/kubernetes" />

  <Card title="Agents" description="Make the Agent your sessions run on, and read its events." href="/docs/run/agents" />
</Cards>
