NylorunDocsBeta
Deploy

Docker Compose

Run one Tenant with nylorun start and Docker Compose.

One Tenant, started with nylorun start. Compose runs the Runtime and its stores. There are no replicas.

npx nylorun start --tenant app

Provision

Install Node 24 and Docker Compose v2. A running Tenant uses about 1.2 GB, most of it Restate.

npx nylorun start --tenant app
npx nylorun status

start opens no browser; it prints the Runtime and Studio URLs. Reach Studio through an SSH tunnel to its loopback port. Studio uploads a message's attachments through its own server, which sets no size cap of its own: each upload is held to the Tenant's artifact limits (100 MiB per file by default).

Reverse proxy

nylorun start publishes the Runtime on 127.0.0.1 only. Put a TLS reverse proxy on the same machine:

RuleWhy
Listen with TLS; forward to 127.0.0.1:<port>The application key travels on every request.
Rewrite Host to localhost:<port>The Runtime answers 421 to any other Host.
Forward only /health, /ready, /.well-known/oauth-protected-resource, and /v1/*Nothing else is the Runtime or Management API. The well-known document tells OAuth clients where to sign in.
Never proxy Studio or RestateThey stay on the machine.
Do not buffer responsesEvent streams stay open with a keepalive every 15 seconds.
Allow request bodies as large as the largest upload you allowAn artifact upload is up to the Tenant's artifact file limit (100 MiB by default). A skill or runtime file that nylorun apply or client.agentManifests.save() uploads is up to the Tenant's skill limit (2 GiB by default), and an agent manifest's files are up to its folderBytes limit (50 MiB by default) together. An AG-UI or A2A message carries files inline, about 28 MiB at the default limits.
Pass Authorization and Nylorun-* throughYour app sets those headers.
Pass 401 answers and WWW-Authenticate through unchangedA refused key or token is a 401 with a Bearer challenge; clients need both.
Start the Runtime with NYLORUN_PUBLIC_URL set to the proxy's URLWebhook triggers' webhookUrl and the OAuth metadata name it: NYLORUN_PUBLIC_URL=https://runtime.example.com npx nylorun start. Without it they name http://localhost:<port>.
Webhooks: publish only /v1/webhooks/* to the internet, and rate-limit itA webhook trigger's deliveries arrive at POST /v1/webhooks/{endpointId} with no Authorization and no Nylorun-Protocol; a Standard Webhooks signature authenticates each one. Keep every other path private.
runtime.example.com {
	@api path /health /ready /.well-known/oauth-protected-resource /v1/*
	handle @api {
		reverse_proxy 127.0.0.1:8787 {
			header_up Host localhost:8787
			flush_interval -1
		}
	}

	handle {
		respond 404
	}
}

When a webhook sender is on the internet and the Runtime is not, a second site can publish the webhook path alone:

hooks.example.com {
	handle /v1/webhooks/* {
		reverse_proxy 127.0.0.1:8787 {
			header_up Host localhost:8787
		}
	}

	handle {
		respond 404
	}
}

Give the sender https://hooks.example.com/v1/webhooks/<endpointId>. The Runtime's webhookUrl names NYLORUN_PUBLIC_URL, so when webhooks have a host of their own, keep only its path.

PlacementCertificate
Same LANTailscale, or a local CA (tls internal) whose root the app server trusts. Never plain HTTP.
InternetA public certificate. Prefer a private path (VPN or the same network) over a public endpoint.

Firewall so only the proxy port and your SSH port are reachable.

When the Runtime trusts your identity provider's tokens (an identity file), it publishes those issuers at /.well-known/oauth-protected-resource; without an identity file, that path is a 404. The document's resource is NYLORUN_PUBLIC_URL. Set it to the proxy URL clients use, such as https://runtime.example.com, when you run nylorun start; without it, nylorun start sets http://localhost:<port>.

An identity file grants Agents with each issuer's agents: list and the agents:write scope:

issuers:
  - name: keycloak
    issuer: https://sso.acme.dev/realms/eng
    audience: https://runtime.example.com
    jwks: https://sso.acme.dev/realms/eng/protocol/openid-connect/certs
    allowedScopes: [agents:read, sessions:own, agents:write]
    agents: ["users/{sub}/*", support]
  • agents lists the Agents an issuer's tokens reach: exact ids, prefixes ending in /*, templates such as {org_id}/*, or *. An issuer without agents reaches no Agent.
  • A file with an environments: or sandboxes: key, or the environments:write or sandboxes:write scope, stops the Runtime with a message naming the new spelling.
  • A token never saves agent manifests: agent-manifests:write is an application key's.
  • A token may be up to 16 KiB, the most a request's headers hold.

App server

Give the app server an application key of its own. On the Runtime's machine:

npx nylorun tenant keys put app-server

The key prints once; keep it in the app server's secret store. On the app machine, set:

VariableValue
NYLORUN_RUNTIME_URLThe proxy URL, https://runtime.example.com. On the Runtime's machine, the Runtime URL nylorun status prints.
NYLORUN_SERVER_KEYThe application key tenant keys put printed. Never a management key or the admin key.

In the project directory on the Runtime's machine, the SDK reads the project link that nylorun start wrote and needs neither variable.

Save the agent manifests and make their Agents from a deploy step: npm start in a generated project, or npx nylorun apply <dir> with the two variables set. See Saving manifests. The app server runs no code for the Runtime: the services your HTTP tools call must be reachable from the Runtime's machine. See Serve your users.

Private addresses

The gateway's requests to URLs that agents name (HTTP tools and remote MCP servers), to a credential's via, and those of MCP server previews follow these settings. Each is checked on the address actually connected to, and no redirect is followed.

SettingValuesDefault
NYLORUN_ENDPOINT_PRIVATEallow or refuse: private, loopback, and link-local addressesallow
NYLORUN_ENDPOINT_HTTPallow or refuse: plain http URLsallow
NYLORUN_ENDPOINT_LOOPBACKdocker-host: localhost means the machine that runs DockerUnset; a local Tenant sets it

On a server, set NYLORUN_ENDPOINT_PRIVATE=refuse and NYLORUN_ENDPOINT_HTTP=refuse on the gateway and the runtime, so a tool's URL or a via cannot point the gateway at your internal network. If your tools' services live on a private network, keep allow and limit the gateway's egress with your firewall instead.

The workload token

A platform that runs Runtimes can give each one a short-lived token and rotate it through a file, so its gateways trust the workload instead of a stored key. A workload credential sends the token to an MCP gateway, and a custom model endpoint with auth: { type: "workload" } sends it as the endpoint's bearer. Point the Runtime at the file:

VariableValue
NYLORUN_WORKLOAD_TOKEN_FILEThe token's file. The Runtime reads it at each use and keeps nothing, so a rotation applies to the next request with no restart. A missing, unreadable, or empty file fails the request with a message naming the variable; nothing is sent without the token.
NYLORUN_WORKLOAD_TOKEN_ORIGINSThe comma-separated origins a workload credential may send the token to, such as https://connections.example.com: each https, or http only to a loopback host. Checked at each request: a credential whose via has another origin is refused at discovery. The model endpoint needs no entry, since its baseUrl is yours.

Set both on the gateway container, which makes model and tool calls, and on the runtime container of a Runtime that runs without the gateway. Mount the file read-only.

nylorun start does not pass these variables to its containers yet, so a local Tenant cannot use the workload token: set them in a Compose file or deployment you run yourself.

The token stays in the process that makes the call. It never reaches a sandbox (a sandbox's environment has no NYLORUN_* variable), the model, an event, or a log, and it is scrubbed from what a server or the provider echoes, like any credential value. A gateway that refuses it answers credential_rejected, and the call's tool.completed names the gateway's origin in error.gateway.

Operate

ServiceRole
runtimeTenant API and scheduling.
gatewayModel and tool calls, with the files a model call names. Holds provider keys.
harnessAgent turns and workspaces.
studioOperator dashboard on loopback.
postgresSessions, agent manifests, Agents, their memories, and their triggers.
restateSession advancement and trigger timers.
s2-liteEvent history.
rustfsFile bytes.
npx nylorun status
npx nylorun local logs runtime -f
curl -sS http://127.0.0.1:8787/health
curl -sS http://127.0.0.1:8787/ready

GET /health reports version, protocol range, and features. GET /ready checks the listener, Tenant, Postgres, and Restate; it answers 503 until they are ready. It does not check S2: an S2 outage affects only history and event-stream reads, and shows as streams.reachable: false in GET /v1/tenant.

Triggers and wakes fire from the Runtime's durable timers, with Postgres holding when each one is due; there is no extra process to run. A stopped Tenant fires nothing. At the next start, each overdue trigger fires once, marked late, and the times it missed are not run.

A 500 internal_error answer is logged as request_failed (keys_request_failed for a key operation), with the method, the route's pattern, and the error's message and stack; never a request body, header, or argument.

Configure the Tenant through the Management API (/v1/tenant/*) with a management key and @nylorun/admin, or with npx nylorun on the machine. Sizes, counts, and deadlines that requests run into are the Tenant's Runtime limits, set the same way. Issue keys on the machine:

npx nylorun tenant keys put backend            # the app server's application key
npx nylorun tenant keys put ops --management   # a management key, for @nylorun/admin

A self-hosted Runtime container runs the same jobs with nylorun-operate keys put <id> [--role management] and nylorun-operate status.

Host tuning

NYLORUN_TUNING='{"tenant":{"upgradeStartMs":1200000},"pods":{"readyDeadlineMs":600000}}' \
npx nylorun start --tenant app

The Runtime's own timings, retries, poll intervals, and internal sizes are its Host tuning: one JSON object in NYLORUN_TUNING, read once when a process starts. Set only what you change; everything left out keeps its default.

  • nylorun start passes NYLORUN_TUNING from its shell to the runtime, gateway, and harness containers. Elsewhere, give every container of the deployment the same value: each reads the groups it uses. A pod's engine gets the harness and engine groups, as JSON of at most 4096 bytes.
  • An unknown key, a value of the wrong kind, or one past its ceiling stops the process with the key's path, such as NYLORUN_TUNING: tenant.openTimeoutMs: Too small: expected number to be >=1.
  • A value in milliseconds (…Ms) is a whole number from 1 to 2,147,483,647. A backoff is { "minMs", "maxMs" }: the first wait, doubled after each failure up to the longest.
GroupWhat it tunes
keysToken lifetimes, clock tolerance, the egress certificates, and the keys service.
harnessApiThe Harness API listener: joins, run offers, and segment rollover.
apiThe AG-UI heartbeat and the event-stream keepalive.
hostShutdown grace and readiness checks.
objectStoreMultipart part size and retries.
tenantOpening the Tenant, leases, Retry-After, and moving pod Agents to new versions.
sweepThe Tenant's sweep and outbox.
triggersLate fires and webhook timestamps.
executionAdvance deadlines and wake de-duplication.
restateSends, handler retries, and registration.
postgresPool sizes and timeouts, the control bus, and replication.
streamsS2 and the stream relay.
gatesGateway timeouts, MCP connections, and model-call reuse.
egressEgress gateway tunnels and timeouts.
podsSandbox pod readiness, polling, and the memory floor.
harnessHarness connections, MCP listing, sandbox helpers, and pod setup.
tracesTrace export batching and retries.
engineOur engine's options, such as compaction and tool results.

The settings an operator is most likely to change:

SettingDefaultWhat it sets
tenant.upgradeStartMs10 minHow long a pod may take to start on a new version before the Agent goes back to the one it ran.
tenant.upgradeIdleMs1 minHow long a pod Agent waiting to start on a new version stays idle before it stops (its own lifecycle.idle when shorter).
tenant.retryAfterSeconds5The Retry-After of a request refused while its Agent is busy or pods are not ready.
triggers.webhookToleranceMs5 minThe clock skew a webhook delivery's webhook-timestamp may have, either way.
triggers.lateGraceMs2 minHow long after it was due a trigger's fire counts as late.
pods.readyDeadlineMs5 minHow long a pod may take to be ready before its start fails.
pods.memoryFloorMiB128The least memory a sandbox gets; a smaller request rounds up.
keys.tokenTtlMaxSeconds900The longest life of a token the Runtime signs, such as a capability link.
postgres.poolSize10The Tenant pool's connections.
gates.clientTimeoutMs630 sHow long a model or tool call's connection to the gateway may stay silent.
traces.batchSpans, traces.batchDelayMs, traces.retries512, 2 s, 3Spans per export post, how long a span waits for its batch, and retries of a failed post.

Every setting, with its default and ceiling, is listed in DEPLOYMENT.md. nylorun's own waits are NYLORUN_CLI_TUNING; see The CLI's own waits.

Traces

Each turn is one OpenTelemetry trace: a root span for the turn, named for the agent manifest and carrying the version the turn ran (nylorun.manifest.version), and a span for each model call and tool call. A sandbox's network connections are not spans: each is an agent.egress event on its Agent's events (see Egress records). Export is off until an OTLP endpoint is set on the runtime container. nylorun start passes these variables to it, and to the gateway, from the shell it runs in. The same endpoint takes the processes' live spans and metrics; Monitor lists them, with /metrics and sampling:

OTEL_EXPORTER_OTLP_ENDPOINT=http://host.docker.internal:4318 \
OTEL_SERVICE_NAME=support-runtime \
npx nylorun start --tenant app
VariableMeaning
OTEL_EXPORTER_OTLP_ENDPOINTThe collector's OTLP/HTTP base URL. Spans go to <url>/v1/traces as JSON.
OTEL_EXPORTER_OTLP_TRACES_ENDPOINTThe full traces URL, instead of the base URL.
OTEL_EXPORTER_OTLP_HEADERSkey=value pairs separated by commas (values percent-encoded), sent with every post.
OTEL_SERVICE_NAMEservice.name on the resource. Default nylorun-runtime (the gateway's nylorun-gateway).
OTEL_RESOURCE_ATTRIBUTESMore resource attributes, as key=value pairs.
OTEL_INSTRUMENTATION_GENAI_CAPTURE_CONTENTThe content spans carry: NO_CONTENT (the default) or SPAN_ONLY, every kind. See Monitor, Content.
NYLORUN_TRACE_CONTENT, OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENTDeprecated: read for this release only when the switch above is unset.

A collector on the same machine is host.docker.internal from the container. By default a span carries references to the session log, never content; with SPAN_ONLY, each text is cut at the Tenant's traceContentBytes (Runtime limits, 8 KiB by default). A span never carries a credential value, a person's id, an identity header, or an idempotency key.

  • Off the hot path. The Runtime reads committed events from its own replication slot, nylorun_trace_export, created at start, and posts batches of up to 512 spans, or every 2 seconds (traces.batchSpans and traces.batchDelayMs in Host tuning). No turn waits for it. Without an endpoint there is no slot.
  • A collector that is down loses only its spans. A 429, a 5xx, or a lost connection is tried three times with backoff (traces.retries); then the batch is dropped, logged as trace_export_dropped, and counted on nylorun.telemetry.dropped (exporter="projector"), and the slot moves on, so Postgres keeps no WAL for the collector.
  • Backfill. Read the turns a collector missed from GET /v1/sessions/{id}/turns/{turnId}/trace (session.trace(turnId)) and post them to the collector's /v1/traces. The span ids are the same each time a turn is read or exported.
  • Lag is not reported: watch nylorun_trace_export in pg_replication_slots.

Reading a turn's trace and passing traceparent in from your app are on Traces.

Custom harnesses

A harness of your own runs only as a pod Agent's image: its agent manifest names it in harness.id, with runtime.kind: "pod" and an image. It needs sandbox pods; see Kubernetes → Custom harnesses.

Backups

Back up the Postgres and RustFS volumes together with keys/vault-kek. Postgres holds agent manifests (with the file list of each version saved from files), Agents, memories with every version, triggers and their runs, session queues, and webhook signing secrets. RustFS holds file bytes, among them every file those versions were saved from. Agents' secrets and webhook signing secrets are sealed with the vault key: a restore without keys/vault-kek loses them. Make a new webhook secret with npx nylorun agents triggers new-secret <id> <trigger>.

nylorun stop keeps volumes. nylorun local reset deletes this Tenant's data.

Upgrades

Upgrade nylorun and run start. The Runtime applies its database migrations at start. start refuses to run an older Runtime than the Tenant last ran (--allow-downgrade overrides).

The Runtime speaks protocol 16, 17, and 18, and the SDKs send 18: upgrade nylorun first, which brings the Runtime and Studio, then the SDKs. An older SDK keeps working against a newer Runtime; a newer SDK against an older Runtime throws IncompatibleRuntimeError. From nylorun 0.17, migrations 0037 (model call cost by token kind) and 0038 (agent manifest sources) run at the first start, and delete nothing; from 0.15, 0028 to 0038. Read Compatibility before you upgrade.

From protocol 14 or older, the upgrade deletes Environments

Migration 0032 deletes every Environment with its pod's state, its disk, its disk versions, and its secrets. It drops saved agents and environment manifests, and closes every open session; their history stays readable.

  1. Before upgrading, with the old release, export what you need. Snapshot each disk, find the folder artifact and version the snapshot made, and download it as a zip. Secret values cannot be exported: list the names and have the values at hand.

    npx nylorun environments snapshot <id>
    npx nylorun environments versions <id>
    npx nylorun artifacts zip <artifactId> --version <n> --output <id>.zip
    npx nylorun environments secrets list <id>
    npx nylorun environments manifests get <name>

    Keep each agent's v7 manifest JSON and each environment manifest's JSON for nylorun migrate v8.

  2. Before the first start, rename the identity file's environments: to agents: and the environments:write scope to agents:write.

  3. After the upgrade, on a cluster, delete the dropped Environments' pods and their join Secrets before you make pod Agents: kubectl delete sandboxes.agents.x-k8s.io,secrets -l app.kubernetes.io/managed-by=nylorun-sandboxes -n nylorun-sbx-<tenant>. A pod Agent made with an old Environment's id would otherwise take over its pod and volume. If a tool policy rule names a tool as server__tool, put the policy again with server and tool. Write agent manifests (npx nylorun migrate v8 <agent.json> --runtime <environment-manifest.json>, then npx nylorun apply <dir>), make the Agents, and write their secrets.

nylorun start runs the ghcr.io/nylorun/runtime and ghcr.io/nylorun/studio images its release pins. NYLORUN_RUNTIME_IMAGE and NYLORUN_STUDIO_IMAGE replace them, for example with a local build.

SettingMeaning
NYLORUN_WORKER_VERSIONThe Worker's version on the runtime container: 1 to 64 letters, digits, ., _, or -. Default: the Runtime's version. Set it on an image you build whose Worker code changed without a new Runtime version.

The image's entry point is /app/runtime/server/dist/host/main.js. A script that runs a file in the image directly uses that path.

Your own Postgres

A local Tenant starts Postgres with these settings already. A Postgres you run yourself needs them too:

SettingValueWhy
wal_levellogicalThe Runtime reads committed events from replication slots.
max_replication_slots, max_wal_sendersAt least 2 (the default 10 is enough)Two slots: nylorun_stream_relay, and nylorun_trace_export when traces are exported.
max_slot_wal_keep_sizeA few GB (nylorun start uses 4GB)A stuck slot cannot fill the disk.
The Runtime's roleREPLICATION, plus read access to nylorun_streamsThe relay reads the Tenant's events.

On a managed Postgres, turn on its logical replication option, for example rds.logical_replication on RDS.

Connect the Runtime to Postgres directly or through a pooler in session mode. Cancels and session resets reach every Runtime process through LISTEN and NOTIFY, which a pooler in transaction mode (PgBouncer's default) does not deliver; a cancel then reaches another process only through a 5-second poll.

Next step

On this page