Kubernetes
Run pod Agents on a cluster.
Kubernetes runs pod Agents: each one an Agent whose agent manifest says
runtime.kind: "pod", with one pod and its own volume. The Runtime itself runs
with Docker Compose; see Docker Compose.
npx nylorun local sandbox enable --context <ctx>
npx nylorun local sandbox statusenable installs the sandboxes service (agent-sandbox) against that
kubeconfig context. Optional flags: --tenant, --host-address,
--bind-address, --no-pull. npx nylorun local sandbox disable stops it;
--delete-namespace removes the namespace and every pod in it.
--backend substrate runs the pods as Agent Substrate actors instead, on a
cluster with Substrate installed. It is a preview: see
Agent Substrate.
npx nylorun local sandbox status --json reports the cluster, readiness, and
what the service sees. The controller applies a NetworkPolicy so pod egress matches
the Agent's allowlist. On a slow cluster, give the CLI's waits more time with
its sandbox tuning group, such as
NYLORUN_CLI_TUNING='{"sandbox":{"rolloutWaitSeconds":600}}' (see
The CLI's own waits).
Pod Agents
Save an agent manifest whose runtime section is a pod, make an Agent from it, then open sessions on the Agent:
await client.agentManifests.save({
name: "analyst",
harness: { instructions: ["Analyse the project's data and explain what you find."] },
runtime: {
kind: "pod",
image: "node:24",
network: { allowedHosts: ["registry.npmjs.org", "github.com"] },
storage: "10GiB",
lifecycle: { idle: "15m", deleteAfter: "30d" },
},
});
await client.agents.put("projects/acme", { manifest: "analyst" });
const session = await client.createSession({
agent: "projects/acme",
ownerUserId,
});A built agent saves the same way:
client.agentManifests.save({ agent: analyst, runtime: { kind: "pod", … } }).
Egress is limited to the hosts you list; a pod also takes *.example.com
patterns. Private networks, loopback, the host, and cloud metadata are always
blocked.
The turns of every session on the Agent run in its one pod at the same time,
one turn per session, and their sandbox commands overlap; each session's
commands keep their order. Stop, reset, restart, snapshot, restore, and delete
answer 409 agent_busy while any turn runs.
client.agents.stop(id) stops the pod and keeps its volume; reset(id) gives
it a new, empty volume with its setup and input files. A pod that is idle for
lifecycle.idle stops, and the next turn starts it again.
npx nylorun agents list [--outdated] and npx nylorun agents rm <id> list and
delete Agents. See Agents.
A pod's disk can start prepared with
setup commands and input files,
and every Agent's disk keeps versions, within
the runtime section's limits.disk. A pod whose volume is lost starts again
from its latest version. A fork onto a version with
other setup commands runs that setup on the copied disk before its first turn.
New versions on a pod
An Agent made with manifest: "analyst" tracks the manifest: each save moves
it to the new version ("analyst@3" pins version 3). A harness change applies
at the next turn. A runtime change a pod applies only when it starts (image,
resources, network, setup, files) waits for the pod's next start, on the same
volume:
- While it waits, the Agent's view says
targetVersionandwaitingFor: "start", and its events sayagent.outdated, thenagent.version_changedonce the pod runs the new version. client.agents.restart(id)ornpx nylorun agents restart <id>applies it now: a running pod stops, and the next turn starts it.- An idle pod that waits stops after
tenant.upgradeIdleMs(1 minute, or its ownlifecycle.idlewhen shorter; Host tuning). - Setup runs again on the same volume when its commands or files changed.
- A start that fails (its setup, the pod, or slower than
tenant.upgradeStartMs, 10 minutes) goes back to the version the Agent ran, with its disk as it is: the view saysfailedVersion, the event isagent.start_failed, and the Agent's state isfailing. A tracking Agent does not move to that version again on its own:client.agents.restart(id, { retry: true })ornpx nylorun agents restart <id> --retrytries it again.
Plan a move first with client.agents.plan(id, { manifest: "analyst@4" }) or
npx nylorun agents put <id> --manifest analyst@4 --dry-run. The manifest's
name, kind, harness, and storage are fixed for the Agent's life: a move
that changes one is 409 agent_fixed_field, so fork it
(client.agents.fork()). See Versions.
Images by digest
The first time a pod of an Agent is ready, the Runtime records the image it
pulled by digest, compute.pod.imageDigest in the Agent's view. A stop, a
relaunch, or a reset runs that digest again, never another image pushed under
the same tag. A version that changes image pulls the new tag at the pod's
next start; a start that fails goes back to the old digest. A fork pulls the
tag afresh. Only a digest from the requested image's repository is recorded,
so an image loaded onto a node without a registry (such as with
kind load docker-image) pins nothing.
Tenant limits
What every Agent runs must fit the Tenant's settings. Set them with a
management key; a PUT replaces all of them:
import { createAdmin } from "@nylorun/admin";
const admin = createAdmin();
await admin.settings.sandbox.put({
limits: {
environments: 200,
idle: "30m",
network: ["registry.npmjs.org", "github.com", "*.githubusercontent.com"],
resources: { cpus: 8, memory: "16GiB" },
defaultResources: { cpus: 2, memory: "2GiB" },
},
lifecycle: { stopGrace: "10s" },
});| Setting | Meaning | Default |
|---|---|---|
limits.environments | The most Agents the Tenant holds. | 100 |
limits.idle | The longest lifecycle.idle, and the default. | 15m |
limits.network | Hosts any Agent may allow. | Package registries and code hosts |
limits.resources | The most one Agent may ask for. | 4 CPUs, 8192MiB |
limits.defaultResources | What an Agent gets when it asks for none. | 2 CPUs, 1024MiB |
lifecycle.stopGrace | A pod's grace period to stop. | 10s |
placement | Where each harness may run. See Harness placement. | Anywhere |
engine.images | Image repositories (no tag or digest) that already carry this Runtime's engine under /opt/nylorun, copied from the Runtime image when they were built. A pod of such an image skips copying the engine in at each start, while the engine it carries is this Runtime's version. | None |
A sandbox gets at least pods.memoryFloorMiB of memory (128 MiB by default,
Host tuning); a smaller memory rounds up.
Within the Tenant's limits, a manifest's runtime section may set lower ones for
its Agents: limits.artifacts.fileBytes and totalBytes,
limits.skills.totalBytes, and limits.disk.versionBytes and
versionEntries.
admin.settings.sandbox.get() reads them twice. config is the settings as
the Tenant set them, with no defaults, in the shape put takes, so you can put
it back changed. effective is the settings in force, with every default
filled in and sizes in MiB (limits.resources.memoryMiB). Runtime 0.47 and
later; before it, config was what effective is now.
const { config, effective } = await admin.settings.sandbox.get();
console.log(effective.limits.resources); // { cpus: 4, memoryMiB: 8192 }
await admin.settings.sandbox.put({ ...config, lifecycle: { stopGrace: "30s" } });See Set the Tenant's Agent compute settings.
Harness placement
A session on a pod Agent runs its harness in the pod; every other session
runs it in the Runtime's harness container. placement limits where each
harness may run, keyed by harness id, with "*" for the rest. A put replaces
every setting, so start from the stored config:
const { config } = await admin.settings.sandbox.get();
await admin.settings.sandbox.put({
...config,
placement: {
nylorun: { hosts: ["harness-container", "sandbox"] },
minimal: { hosts: ["sandbox"] },
"*": { hosts: [] },
},
});| Host | Where |
|---|---|
harness-container | The Runtime's harness container. |
sandbox | A pod. |
A session whose harness may not run where its Agent needs is refused at
open with 409 placement_refused. "*": { hosts: [] } refuses every harness
the map does not name.
Custom harnesses
A harness of your own runs as a pod Agent's image. Its agent manifest names it
in harness.id, with runtime.kind: "pod" and an image; a save without them
is 422 manifest_inconsistent.
await client.agentManifests.save({
name: "minimal-box",
harness: { id: "minimal", instructions: ["Answer briefly."] },
runtime: { kind: "pod", image: "harness-minimal:dev" },
});
await client.agents.put("demos/minimal", { manifest: "minimal-box" });- The image is the pod's entrypoint, with nothing copied in. It speaks Harness
API 5, says the manifest's
harness.idinhello, and joins with the pod's join token; it needs no harness token of its own. - Load the image where the cluster can pull it.
- An Agent's harness is fixed for its life: to run another, save a version that names it and fork the Agent onto it.
The harness-minimal example in
nylorun/agents
is a complete, working harness. To write one, see
Harness and Harness API.
Egress events
Each connection a pod opens through the egress gateway is one agent.egress
event on the Agent's events, allowed or refused. It carries the host, port,
outcome, bytes, and duration, runningTurns (the sessions and turns running
when it opened, at most 16), and for a refusal its reason: a host not in
network.allowedHosts, a port other than 443 or 80, a private address, too
many tunnels, or a name that does not resolve. It never carries a header
value, a URL path, or a body. Session logs and traces hold no egress: read a
turn's with
client.agents.events(id, { types: ["agent.egress"], since, until }) or
npx nylorun agents events <id> --types agent.egress. See
Egress records.
Shell credentials
A skill's CLI in a pod authenticates with a
shell credential whose value never
enters the pod. The skill's secrets in the harness section name the secrets
it reads (in an agent manifest, AGENT.md's skills block lists them), which
must be environment_secret or environment_variable slots of the manifest's
runtime section (422 manifest_inconsistent otherwise). In the pod, an
environment_secret variable reads nylorun_managed.
Pods trust the installation's egress CA: SSL_CERT_FILE,
NODE_EXTRA_CA_CERTS, REQUESTS_CA_BUNDLE, CURL_CA_BUNDLE,
GIT_SSL_CAINFO, and AWS_CA_BUNDLE point at a bundle of the public roots
plus that CA. For HTTPS on port 443 to a host in a secret's allowedHosts,
the egress gateway terminates TLS, sets the secret's header on each request,
and forwards it over verified TLS. The host must also be in the runtime
section's network.allowedHosts. Every other host stays an opaque tunnel.
A CLI that pins certificates or ignores those variables cannot use this path,
and HTTP/2-only and WebSocket endpoints are not served on it. When the secret
cannot be released, the CLI gets a 502 and npx nylorun local logs gateway shows
egress_credential_refused.
Telemetry
The runtime and the gateway each serve Prometheus metrics on /metrics, port
9464: the metrics port of the <name>-runtime-internal and
<name>-gateway Services. Point a scrape job, or a ServiceMonitor, at that
port. To export metrics and spans over OTLP as well, set the standard
variables on both:
extraEnv:
runtime: &otel
- { name: OTEL_EXPORTER_OTLP_ENDPOINT, value: http://otel-collector.monitoring:4318 }
- name: OTEL_EXPORTER_OTLP_HEADERS
valueFrom: { secretKeyRef: { name: otlp, key: headers } }
- { name: OTEL_TRACES_SAMPLER, value: parentbased_traceidratio }
- { name: OTEL_TRACES_SAMPLER_ARG, value: "0.25" }
gateway: *otelWith metrics.from set, only the pods it names may scrape the port; Studio
always may, for Tenant › Traffic. The metrics, the variables, and content
capture are on Monitor.
Upgrading the cluster
The upgrade from protocol 14 deletes the Tenant's old pod workloads from its database but not from the cluster. A pod Agent made with an old Environment's id would take over its pod and volume, so delete them, with their join Secrets, before you make pod Agents:
kubectl delete sandboxes.agents.x-k8s.io,secrets -l app.kubernetes.io/managed-by=nylorun-sandboxes -n nylorun-sbx-<tenant>npx nylorun local sandbox status names the namespace.
See Upgrades.