Troubleshooting
Recover when the Tenant, an agent manifest, an Agent, a tool, a model, or Studio fails.
Start with the smallest failing layer: Tenant, connection, agent manifest, Agent, tool, then Studio.
npx nylorun doctor
npx nylorun statusTenant does not start
- Run
npx nylorun doctorandnpx nylorun status --json. - Follow
npx nylorun local logs runtime -f. /readyreturns503until the listener, the Tenant, Postgres, and Restate are ready. It does not check S2: an unreachable S2 shows asstreams.reachable: falsein the Tenant's status (GET /v1/tenant).- Ports are in the Tenant's
docker/.env. Another Tenant may already hold8787or4161. - An identity file with an
environments:orsandboxes:key, or theenvironments:writeorsandboxes:writescope, stops the Runtime. Rename themagents:andagents:write. See Docker Compose. NYLORUN_TUNING: <path>: …means a Host tuning key is unknown, of the wrong kind, or past its ceiling. Fix or remove it; see Host tuning.NYLORUN_CLI_TUNINGstops anyloruncommand the same way; see The CLI's own waits.
No Runtime connection
Clients need a complete { url, key }, both NYLORUN_RUNTIME_URL and
NYLORUN_SERVER_KEY, or .nylorun/link.json plus credentials.json. Partial
environment blocks the project-link fallback.
The project link is read from the nearest .nylorun/ directory, searching
upward from the working directory. The search stops below your home directory,
so a ~/.nylorun/ is never read as a project. A link without
credentials.json, or a link or credentials file that does not validate, is
connection_missing naming the file.
npx nylorun startConnectionError (connection_missing) names the fix: run npx nylorun start
in the project, or set NYLORUN_RUNTIME_URL and NYLORUN_SERVER_KEY. Outside
the project, give the process a key of its own
(npx nylorun tenant keys put <id>) and the URL npx nylorun status prints.
IncompatibleRuntimeError means the Runtime's protocol range or features do
not match this client; @nylorun/admin throws incompatible_host for the same
reason. The Runtime accepts protocol 16, 17, and 18 and answers an older client
426 protocol_unsupported. The SDKs send protocol 18 and need the Runtime's
usage-by-kind feature, so they need Runtime 0.29 or later: a project on
@nylorun/agents 0.25 or later (a new project gets 0.28), against a Tenant
last started by nylorun 0.17 or older, throws it at the first request.
Restart the Tenant with the current nylorun (upgrade it first in a project
that lists it), then run the app again:
npx nylorun start
npx nylorun statusUpgrade nylorun first, then the SDKs: an older SDK keeps working against a
newer Runtime, not the reverse. The error's remedy says which side to
upgrade, and GET /health reports the Runtime's version and protocol range.
See Compatibility.
A 401 means the Runtime refused the key: credential_required when none was
sent, credential_invalid when the Tenant does not know it (rotated, deleted,
or another Tenant's). Both carry a WWW-Authenticate: Bearer challenge. Run
npx nylorun tenant keys list to see the Tenant's keys.
Saving an agent manifest is refused
Check a manifest without saving it with client.agentManifests.plan(input) or
npx nylorun apply <dir> --dry-run. Both list each error and change, and save
nothing.
-
The agent manifest does not compile.
nylorun check <dir>prints each problem by file, line, and column, offline, andapplysends nothing until it compiles. The Runtime answers such a folder with422 folder_invalid, each problem indetails.diagnostics;planFolder()andsaveFolder()throwFolderInvalidErrorfirst. See Diagnostics.agents/support/AGENT.md:4:1 error: Unknown key 'skils'. Did you mean 'skills'? (fixable with --fix) agents/support: 1 errorcheckrefuses offline most of what only the save refused: sections that do not fit, cron schedules and time zones,files/without a pod, and schemas that are not valid JSON Schema.- Some errors hide later ones, such as a
runtime.jsontypo behind anAGENT.mderror: runcheckagain after each fix. - A problem marked
(fixable with --fix)is fixed bynpx nylorun check <dir> --fix, which prints each change. See Fix. - A
warning, such as a.envfile or a literal token in a header, does not block the save, and neither does anote. Read it: the folder is kept as the version's source, readable withagents:read. See Warnings.
-
An unresolved
extends. The Runtime refuses a folder whoseruntime.jsonnamesextends(not_supported: resolve it before uploading).nylorun check,apply, andpackFolder()merge the file it names; a client that uploads a folder itself merges it first. See extends. -
A folder past the Tenant's folder limits. More files or bytes than
folderFiles(2,000) orfolderBytes(50 MiB) is413 limit_exceeded, withdetails.limitnaming the limit; a Markdown or JSON file larger thanfolderTextFileBytes(1 MiB) is422 folder_invalid. They are Runtime limits. Raising them helps only a client that uploads a folder itself:nylorunandpackFolder()hold a folder to these defaults whatever the Tenant sets. -
Its sections do not fit.
422 manifest_inconsistentlists each gap indetails.gaps, such asharness.mcpServers.github.credential "GITHUB_TOKEN" is not a secret slot in runtime.secrets:- a custom harness (
harness.idother thannylorun) needsruntime.kind: "pod"and animage; - an MCP server's or HTTP tool's
credentialmust name abearerorheadersslot ofruntime.secretsbound to its exact URL; - a skill's
secretsmust nameenvironment_secretorenvironment_variableslots; - skills need a sandbox:
runtime.kindvirtualorpod.
See Sections that fit.
- a custom harness (
-
The SDK refuses before sending.
ManifestGapsError(manifest,gaps) means the manifest misses itsrequires, or its sections do not fit.composeManifest()(the deprecated three-file format) throwsComposeError, naming the file inpath, when a manifest folder's files cannot be read. -
A tool runs your code.
agentManifests.save()refuses, before sending anything, an agent with atool({ run })tool or a flow agent with a code tool stage. The message names the tool and agent. Make it an HTTP tool or serve it from a remote MCP server. -
The agent sets
approval.http({ approval })and.mcp({ … approval })throw,nylorun checkrefuses it in an agent manifest (approval_removed), and the Runtime refuses a manifest withapproval. Remove it and add anaskrule to the tool policy. -
An older manifest. The Runtime takes agent manifest
manifestSchemaVersion: 8only; a flow is aharnesswithkind: "workflow". A v7 manifest is refused with "Rebuild it with nylorun migrate v8": runnpx nylorun migrate v8 <agent.json>, thennpx nylorun apply <dir>. -
412 manifest_changed. The save'sIf-Matchnamed a version that is no longer the latest. Read the manifest again and save over the new version, or save withoutIf-Match. -
400 definition_files_missing. The manifest names skill or runtime files the Runtime does not hold; the hashes are indetails.missing.agentManifests.save()andnylorun applyupload them first. When you callPUT /v1/agent-manifests/{name}yourself, upload each file withPUT /v1/files/sha256:<hex>first. -
413 limit_exceededfor a skill. A skill holds more than the Tenant's skill limit (2 GiB by default,PUT /v1/tenant/skills) or the runtime section'slimits.skills.totalBytes. -
An MCP server is refused. Only remote servers (
streamable-http,sse) are accepted. See MCP.
An HTTP tool fails
- The model sees a tool error with the status and the start of the body. Check
your service's logs and
npx nylorun local logs gateway. - On a local Tenant,
localhostis your machine. A remote Runtime needs a URL its gateway can reach. - A refused address means
NYLORUN_ENDPOINT_PRIVATE=refuseorNYLORUN_ENDPOINT_HTTP=refuseis set. Redirects are not followed. See Docker Compose. http.credentialmeans the tool names acredentialslot that has no value in the Agent's secrets. Write it withclient.agents.secrets.put(id, name, { value })ornpx nylorun agents secrets put <id> <name>(the value on standard input); see Secrets and credentials.credential_rejectedmeans the service answered401to the credential sent. It is not retried: write the secret again under the same name.tool.unavailablemeans the Runtime cannot run the tool.http.runtime-onlymeans an HTTP tool ran in a localrun(). HTTP tools run only on the Runtime.
An MCP tool fails
- The model sees a failed tool result with a code such as
mcp.unreachable,mcp.forbidden, orcredential_rejected. See MCP errors for each code and whether it is retried. - List the tools a server offers with an Agent's credential:
npx nylorun tenant mcp inspect <url> --agent <id>. A server that answers401needs a credential; see Preview an MCP server. - Check which of the servers and HTTP tools of the manifest version an Agent
runs its secrets cover with
admin.coverage({ agentId })ornpx nylorun tenant coverage --agent <id>; see Coverage. 502 mcp_preview_failedmeans the preview could not list the tools;details.failureholds the code a tool call would fail with.- Through a gateway (a credential with
via),credential_rejectedand afaileddiscovery also mean the gateway reported that the server behind it answered401, such as LiteLLM'sauth_requiredfor a token the provider revoked. Sign the person in again, or rotate the key, at the gateway. - With a
workloadcredential, acredential_rejectedcall'stool.completednames inerror.gatewaythe origin that refused the Runtime's workload token: check that the token file is current. A serverrefusedat discovery meansvia's origin is not in the Runtime's allowed origins; see The workload token. - Tools listed with the gateway's prefix (
github-create_issuewhere the server's own name iscreate_issue) mean the credential has notoolPrefix. Write it again with the prefix, then name the server's own tools intoolssettings and tool policy rules. See Gateway tool prefix. - On a server, a refused private address is
mcp.unreachable. See Docker Compose.
A tool call is refused or waits
The Runtime ends some calls itself. The model sees a failed tool result with
one of these codes, and the call's tool.completed event says
status: "failed".
| Code | Cause |
|---|---|
tool.denied | A deny rule of the Tenant's tool policy refused the call, or a person denied a held call. It was not sent. If the message names a rule whose tool holds __, the stored policy fails closed: every MCP and HTTP call is denied and GET /v1/tenant/tool-policy answers 409 request_rejected. Put the policy again, naming MCP tools by server and tool. |
tool.interrupted | A crash or lost connection interrupted the call. An MCP or HTTP call was sent up to three times with the same key first; a sandbox command fails at once. It may have run. |
tool.disabled | The agent's MCP server settings disable the tool. |
mcp.lost | The call was sent and its answer lost: the tool may have run. The model decides whether to call it again. |
artifact.unsupported | read_artifact named a file of a type the Tenant's model does not read. See Files. |
tool.unavailable | The Runtime cannot run the tool. For schedule_wake, the manifest's runtime section does not set selfWake: true. See Triggers and wakes. |
memory.forbidden | The memory tool wrote from an agent whose memory is read_only, or from an agent used as a tool. See Agent memory. |
A call that an ask rule holds pauses the turn with one approval
interaction per call. session.toolCalls() lists it with status: "held";
resolve it with session.approve(). See
Approve a held call.
A pod's refused outbound connection is an agent.egress event on the Agent's
events, with outcome: "refused", a reason such as host_not_allowed, and
the turns that were running (runningTurns). Read them with
client.agents.events(id, { types: ["agent.egress"] }) or
npx nylorun agents events <id> --types agent.egress. To allow the host, add it
to network.allowedHosts in a new manifest version: a tracking Agent moves to
it, and a pod applies it at its next start (npx nylorun agents restart <id>
applies it now). See Egress records.
A session does not open
Opening a session or starting a turn checks the session's Agent.
| Code | Status | Fix |
|---|---|---|
agent_not_granted | 403 | The token's identity-file agents grants do not reach the Agent. Add a grant for its id or prefix. |
agent_owner_mismatch | 403 | The Agent's owner is another person. Use that person's Agent, or one without an owner. See Owner guard. A caller acting for a person gets 404. |
agent_unavailable | 409 | The Agent was deleted, or it is a pod Agent and this Runtime has no sandbox pods or the sandboxes service is not ready. Make it again, or enable pods. |
agent_lost | 409 | The pod's volume or node is gone. Reset it, or restore a disk version. |
placement_refused | 409 | The Tenant's placement keeps the session's harness off the host its Agent needs. |
harness_mismatch | 409 | The Agent's id was deleted and made again with another harness than the session pinned. Open a new session. |
session_closed | 409 | The session was opened before protocol 16 (closedReason such as protocol_15 or protocol_16). Its history stays readable; open a new session. |
credential_conflict | 409 | At the secret write, not at open: two credentials bound to one URL, or two secrets bound to one host, in one Agent. See Shell credentials. |
agent_busy
Sessions on one Agent run their turns at the same time, one turn per session.
409 agent_busy, with Retry-After, answers work on the Agent itself while
any of its sessions has a turn running: stop, reset, restart, delete, snapshot,
restore, or a fork that copies its disk. A turn that starts while a disk copy
runs is refused the same way. Wait for the turn or the copy to finish and try
again. See Shared disks and turns.
An Agent or manifest is refused
| Code | Status | Fix |
|---|---|---|
agent_fixed_field | 409 | A PUT /v1/agents/{id} would change what is fixed for the Agent's life: its manifest's name, its kind, its harness, or its storage, named in details.fields. Fork it with client.agents.fork() or npx nylorun agents fork. |
secret_not_in_manifest | 409 | A secret write named a secret the Agent's manifest version does not declare (while the Agent moves, the version it moves to). Add the slot to the runtime section and save; a tracking Agent moves to the new version. |
manifest_inconsistent | 422 | The manifest's harness and runtime sections do not fit. See Saving an agent manifest is refused. |
folder_invalid | 422 | An agent manifest does not compile; details.diagnostics lists every problem by file, line, and column. Run npx nylorun check <dir>. See Diagnostics. |
manifest_changed | 412 | The save's If-Match named a version that is no longer the latest. Read the manifest again, or save without If-Match. |
manifest_in_use | 409 | An Agent was made from one of the manifest's versions, so it cannot be deleted. Delete those Agents first. |
manifest_not_found | 404 | No agent manifest has this name, or no version this number. |
limit_exceeded | 409 | The Tenant already holds limits.environments Agents. See Tenant limits. |
See Agents and Saving manifests.
An Agent does not run its new version
Read the Agent with client.agents.get(id) or npx nylorun agents get <id>.
| What the view says | Why | Fix |
|---|---|---|
targetVersion, waitingFor: "start" | A pod applies a runtime change at its next start. | Wait for the next start, or run npx nylorun agents restart <id>. |
waitingFor: "secrets", state input_required | The version adds a required secret slot that has no value. | Write the secret: npx nylorun agents secrets put <id> <name>. |
State input_required, attention reason secret_unset: required secret ORDERS_TOKEN unset | A required secret of the version the Agent runs has no value, so a tool that needs it would fail. | Write the secret; the attention item clears once it is set. |
failedVersion, state failing, event agent.start_failed | The pod's start on that version failed (its setup, the pod, or a slow start), and the Agent went back to the version it ran. | Fix it and save a new version, or try it again with npx nylorun agents restart <id> --retry. |
A pinned version ("name@n") | The Agent does not track its manifest. | npx nylorun agents track <id>, or put it with another version. |
npx nylorun agents list --outdated and client.agents.list({ outdated: true })
list the Agents moving to another version; list({ state: "failing" }) lists
those that failed to start. See Versions.
A message is refused
| Code | Status | Fix |
|---|---|---|
session_busy | 409 | The message was sent with delivery: "reject" (--reject-if-busy) while a turn runs or waits. Without it, the message is queued and starts the next turn. See Messages while a turn runs. |
queue_full | 429 | The session already holds sessionQueuedMessages queued messages (100 by default). Wait for the running turn to end. |
session_closed | 409 | The session was opened before protocol 16. Open a new session. |
invalid_request | 400 | The message has more parts than messageParts, or cannot be read; the message names the limit. |
limit_exceeded | 413 | The body is larger than requestBodyBytes, or the record entry it writes is larger than recordEntryBytes (details.limit). |
Any file may be sent; the harness decides how the model sees it. A model call
that fails with unsupported_input means the model or its API does not read a
part the harness sent, such as a custom harness sending a text file as media.
One that fails with invalid_request before anything is sent means the gateway
cannot fetch a file the call names: not the session's, missing, larger than
modelFileBytes, or audio other than WAV or MP3 on OpenAI-compatible
completions. Neither is retried. See Files and
Runtime limits.
A trigger or webhook does not run
Read what each fire came to with client.agents.runs(id) or
npx nylorun agents runs <id>, newest first.
| Run | Reason | Fix |
|---|---|---|
skipped | overlap | A schedule already queued one run on its busy session; later fires are skipped until it starts. |
skipped | rate_limited | The Agent passed triggerRunsPerHour. |
skipped | paused | The trigger is paused. A trigger that a newer version adds arrives paused: npx nylorun agents triggers resume <id> <trigger>. |
failed | session_conflict | The session the trigger names belongs to another Agent. Name another session, or "new". |
Any, marked late | The Tenant was stopped when the trigger was due. It fires once at the next start; missed times are not run. |
Resuming an at trigger whose time has passed is 409 trigger_expired: save a
version with a new time.
A webhook delivery to POST /v1/webhooks/{endpointId} answers:
| Status | Why | Fix |
|---|---|---|
401 credential_invalid | No signing secret made yet, a missing webhook-* header, a webhook-timestamp more than 5 minutes off (triggers.webhookToleranceMs), or a signature that does not match. | Make a secret with npx nylorun agents triggers new-secret <id> <trigger> (it replaces the one before) and give it to the sender. |
404 not_found | No endpoint has this id. | Use the endpoint id from the trigger's webhookUrl. |
409 trigger_paused | The trigger is paused; nothing runs. | npx nylorun agents triggers resume <id> <trigger>. |
413 | The body is larger than webhookBodyBytes (256 KiB by default). | Send a smaller payload, or raise the limit. |
When a trigger's webhookUrl names http://localhost:<port>, the Runtime
has no public URL: start it with NYLORUN_PUBLIC_URL set to the URL your
proxy publishes. See Triggers and wakes and
Docker Compose.
Memory
409 memory_conflict answers a PUT /v1/agents/{id}/memories/{path} whose
contentSha256 no longer matches the memory, or whose path a folder or another
memory already holds. Read the memory again and write on top of it. See
Agent memory.
Custom harness gate codes
A harness of your own sends each MCP and HTTP tool call through the gateway under its run token.
| Code | Cause |
|---|---|
400 key_required | The call carries no Idempotency-Key. Send the call's effect id. |
409 effect_not_journaled | The effect journal holds no running call with that key for the run's turn, tool, and input. Journal the intent before you send the call. |
See Harness API.
A turn fails at its start
A turn that fails at its start with agent.build-failed, naming a tool, means
an agent's tool shares a name with one of our harness's own tools, such as an
agent tool named bash on an Agent with a sandbox. Rename the agent's tool.
A custom harness that says an older api in hello is refused: rebuild it for
Harness API 5. See Harness.
Traces do not arrive
A batch the collector refuses (429, 5xx, or a lost connection) is tried
three times (traces.retries), then dropped and logged as trace_export_dropped in
npx nylorun local logs runtime. Read the missed turns with session.trace(turnId)
and post them to the collector. See
Docker Compose.
Model errors and budgets
- Confirm the provider on the Models tab of Studio's
Tenant page or with
npx nylorun tenant models get; change it withnpx nylorun tenant models set. model.budget_exhaustedmeans a hard cap onPUT /v1/tenant/budgetswas reached. Amanifestbudget caps every Agent made from that agent manifest.- Provider keys stay in the Tenant's model vault. Check
npx nylorun local logs gateway.
Studio sign-in
On this machine, open Studio at its loopback URL (npx nylorun status prints
it); it needs no login there. Over an SSH tunnel, forward to the same port and
open http://localhost:<port>.
Error codes
ERROR_CODES is exported from @nylorun/agents:
| Code | When |
|---|---|
not_found | 404: missing route, session, Agent, agent manifest, trigger, or webhook endpoint. |
credential_required | 401: no key or token was sent. |
credential_invalid | 401: the Tenant does not know the key, the token fails a check, or a webhook delivery's signature does not verify. |
protocol_unsupported | 426: the request's Nylorun-Protocol is outside the Runtime's range. The body names the range. |
host_rejected | 421: the Host header is not an address the Runtime serves. |
origin_rejected | 403: a key was sent with an Origin header. |
unsupported_media_type | 415: the body's content type is not accepted. |
connection_missing | No complete Runtime URL and key, or a link or credentials file that does not validate. |
incompatible_host | Protocol or required-feature mismatch. |
subject_invalid | 400: the Nylorun-Subject or Nylorun-Scopes header is malformed, repeated, or Nylorun-Scopes came without a subject. 403: a management key was sent with them. |
scope_required | 403: the token or client.as() lacks the route's scope. |
key_role_mismatch | 403: an application key on the Management API, or a management key on the Runtime API. |
token_expired | 401: the token is past exp. |
limit_exceeded | A Tenant limit was hit: a Runtime limit (413, details.limit), the most Agents the Tenant holds, a skill over the skill limit, or an artifact limit. |
request_rejected | The Runtime refused the request for the reason its message gives. |
invalid_request | 400: the Runtime could not read the headers, body, or query. |
subject_required | 400: a route that acts for a person was called without Nylorun-Subject. |
agent_not_granted | 403: the token's agents grants do not reach the Agent. |
session_busy | 409: a message sent with delivery: "reject" while the session's turn runs or waits. |
queue_full | 429: the session already holds sessionQueuedMessages queued messages. |
trigger_expired | 409: resuming an at trigger whose time has passed. |
trigger_paused | 409: a signed delivery to a paused webhook trigger; nothing runs. |
memory_conflict | 409: a memory write whose contentSha256 no longer matches, or a path a folder or another memory holds. |
agent_busy | 409: stop, reset, restart, delete, snapshot, restore, or a disk-copying fork while a turn runs, or a turn while a disk copy runs. |
agent_unavailable | 409: the Agent is gone, or its kind cannot run here. |
placement_refused | 409: the Tenant's placement keeps the harness off the host. |
agent_lost | 409: a pod Agent's volume or node is gone. |
agent_owner_mismatch | 403: the Agent is for another owner's sessions. |
harness_mismatch | 409: the Agent now runs another harness than the session pinned. |
session_closed | 409: the session was opened before protocol 16. |
issuer_unavailable | 401: a token names a new signing key and the issuer's JWKS cannot be fetched. Retry later. |
definition_files_missing | 400: the manifest names files the Runtime does not hold. |
credential_rejected | Not an HTTP answer: the tool error the model sees when an MCP server or HTTP tool answers 401, or a gateway reports that the server behind it did. |
mcp_preview_failed | 502: an MCP server preview could not list its tools. |
credential_conflict | 409: a secret write would bind two credentials to one URL or host. |
agent_fixed_field | 409: a PUT would change an Agent's manifest name, kind, harness, or storage. |
secret_not_in_manifest | 409: a secret write named a secret the Agent's manifest version does not declare. |
manifest_inconsistent | 422: an agent manifest's harness and runtime sections do not fit. |
folder_invalid | 422: an agent manifest does not compile. |
manifest_not_found | 404: no agent manifest has this name or version. |
manifest_in_use | 409: an Agent was made from the manifest. |
manifest_changed | 412: If-Match named a version that is no longer the latest. |
internal_error | 5xx: the Runtime failed. Check npx nylorun local logs runtime. |
Use the code for control flow and the message for operator context.