NylorunDocsBeta
More

Troubleshooting

Recover when the Tenant, an agent manifest, an Agent, a tool, a model, or Studio fails.

Start with the smallest failing layer: Tenant, connection, agent manifest, Agent, tool, then Studio.

npx nylorun doctor
npx nylorun status

Tenant does not start

  • Run npx nylorun doctor and npx nylorun status --json.
  • Follow npx nylorun local logs runtime -f.
  • /ready returns 503 until the listener, the Tenant, Postgres, and Restate are ready. It does not check S2: an unreachable S2 shows as streams.reachable: false in the Tenant's status (GET /v1/tenant).
  • Ports are in the Tenant's docker/.env. Another Tenant may already hold 8787 or 4161.
  • An identity file with an environments: or sandboxes: key, or the environments:write or sandboxes:write scope, stops the Runtime. Rename them agents: and agents:write. See Docker Compose.
  • NYLORUN_TUNING: <path>: … means a Host tuning key is unknown, of the wrong kind, or past its ceiling. Fix or remove it; see Host tuning. NYLORUN_CLI_TUNING stops a nylorun command the same way; see The CLI's own waits.

No Runtime connection

Clients need a complete { url, key }, both NYLORUN_RUNTIME_URL and NYLORUN_SERVER_KEY, or .nylorun/link.json plus credentials.json. Partial environment blocks the project-link fallback.

The project link is read from the nearest .nylorun/ directory, searching upward from the working directory. The search stops below your home directory, so a ~/.nylorun/ is never read as a project. A link without credentials.json, or a link or credentials file that does not validate, is connection_missing naming the file.

npx nylorun start

ConnectionError (connection_missing) names the fix: run npx nylorun start in the project, or set NYLORUN_RUNTIME_URL and NYLORUN_SERVER_KEY. Outside the project, give the process a key of its own (npx nylorun tenant keys put <id>) and the URL npx nylorun status prints.

IncompatibleRuntimeError means the Runtime's protocol range or features do not match this client; @nylorun/admin throws incompatible_host for the same reason. The Runtime accepts protocol 16, 17, and 18 and answers an older client 426 protocol_unsupported. The SDKs send protocol 18 and need the Runtime's usage-by-kind feature, so they need Runtime 0.29 or later: a project on @nylorun/agents 0.25 or later (a new project gets 0.28), against a Tenant last started by nylorun 0.17 or older, throws it at the first request. Restart the Tenant with the current nylorun (upgrade it first in a project that lists it), then run the app again:

npx nylorun start
npx nylorun status

Upgrade nylorun first, then the SDKs: an older SDK keeps working against a newer Runtime, not the reverse. The error's remedy says which side to upgrade, and GET /health reports the Runtime's version and protocol range. See Compatibility.

A 401 means the Runtime refused the key: credential_required when none was sent, credential_invalid when the Tenant does not know it (rotated, deleted, or another Tenant's). Both carry a WWW-Authenticate: Bearer challenge. Run npx nylorun tenant keys list to see the Tenant's keys.

Saving an agent manifest is refused

Check a manifest without saving it with client.agentManifests.plan(input) or npx nylorun apply <dir> --dry-run. Both list each error and change, and save nothing.

  • The agent manifest does not compile. nylorun check <dir> prints each problem by file, line, and column, offline, and apply sends nothing until it compiles. The Runtime answers such a folder with 422 folder_invalid, each problem in details.diagnostics; planFolder() and saveFolder() throw FolderInvalidError first. See Diagnostics.

    agents/support/AGENT.md:4:1 error: Unknown key 'skils'. Did you mean 'skills'? (fixable with --fix)
    agents/support: 1 error
    • check refuses offline most of what only the save refused: sections that do not fit, cron schedules and time zones, files/ without a pod, and schemas that are not valid JSON Schema.
    • Some errors hide later ones, such as a runtime.json typo behind an AGENT.md error: run check again after each fix.
    • A problem marked (fixable with --fix) is fixed by npx nylorun check <dir> --fix, which prints each change. See Fix.
    • A warning, such as a .env file or a literal token in a header, does not block the save, and neither does a note. Read it: the folder is kept as the version's source, readable with agents:read. See Warnings.
  • An unresolved extends. The Runtime refuses a folder whose runtime.json names extends (not_supported: resolve it before uploading). nylorun check, apply, and packFolder() merge the file it names; a client that uploads a folder itself merges it first. See extends.

  • A folder past the Tenant's folder limits. More files or bytes than folderFiles (2,000) or folderBytes (50 MiB) is 413 limit_exceeded, with details.limit naming the limit; a Markdown or JSON file larger than folderTextFileBytes (1 MiB) is 422 folder_invalid. They are Runtime limits. Raising them helps only a client that uploads a folder itself: nylorun and packFolder() hold a folder to these defaults whatever the Tenant sets.

  • Its sections do not fit. 422 manifest_inconsistent lists each gap in details.gaps, such as harness.mcpServers.github.credential "GITHUB_TOKEN" is not a secret slot in runtime.secrets:

    • a custom harness (harness.id other than nylorun) needs runtime.kind: "pod" and an image;
    • an MCP server's or HTTP tool's credential must name a bearer or headers slot of runtime.secrets bound to its exact URL;
    • a skill's secrets must name environment_secret or environment_variable slots;
    • skills need a sandbox: runtime.kind virtual or pod.

    See Sections that fit.

  • The SDK refuses before sending. ManifestGapsError (manifest, gaps) means the manifest misses its requires, or its sections do not fit. composeManifest() (the deprecated three-file format) throws ComposeError, naming the file in path, when a manifest folder's files cannot be read.

  • A tool runs your code. agentManifests.save() refuses, before sending anything, an agent with a tool({ run }) tool or a flow agent with a code tool stage. The message names the tool and agent. Make it an HTTP tool or serve it from a remote MCP server.

  • The agent sets approval. http({ approval }) and .mcp({ … approval }) throw, nylorun check refuses it in an agent manifest (approval_removed), and the Runtime refuses a manifest with approval. Remove it and add an ask rule to the tool policy.

  • An older manifest. The Runtime takes agent manifest manifestSchemaVersion: 8 only; a flow is a harness with kind: "workflow". A v7 manifest is refused with "Rebuild it with nylorun migrate v8": run npx nylorun migrate v8 <agent.json>, then npx nylorun apply <dir>.

  • 412 manifest_changed. The save's If-Match named a version that is no longer the latest. Read the manifest again and save over the new version, or save without If-Match.

  • 400 definition_files_missing. The manifest names skill or runtime files the Runtime does not hold; the hashes are in details.missing. agentManifests.save() and nylorun apply upload them first. When you call PUT /v1/agent-manifests/{name} yourself, upload each file with PUT /v1/files/sha256:<hex> first.

  • 413 limit_exceeded for a skill. A skill holds more than the Tenant's skill limit (2 GiB by default, PUT /v1/tenant/skills) or the runtime section's limits.skills.totalBytes.

  • An MCP server is refused. Only remote servers (streamable-http, sse) are accepted. See MCP.

An HTTP tool fails

  • The model sees a tool error with the status and the start of the body. Check your service's logs and npx nylorun local logs gateway.
  • On a local Tenant, localhost is your machine. A remote Runtime needs a URL its gateway can reach.
  • A refused address means NYLORUN_ENDPOINT_PRIVATE=refuse or NYLORUN_ENDPOINT_HTTP=refuse is set. Redirects are not followed. See Docker Compose.
  • http.credential means the tool names a credential slot that has no value in the Agent's secrets. Write it with client.agents.secrets.put(id, name, { value }) or npx nylorun agents secrets put <id> <name> (the value on standard input); see Secrets and credentials.
  • credential_rejected means the service answered 401 to the credential sent. It is not retried: write the secret again under the same name.
  • tool.unavailable means the Runtime cannot run the tool.
  • http.runtime-only means an HTTP tool ran in a local run(). HTTP tools run only on the Runtime.

An MCP tool fails

  • The model sees a failed tool result with a code such as mcp.unreachable, mcp.forbidden, or credential_rejected. See MCP errors for each code and whether it is retried.
  • List the tools a server offers with an Agent's credential: npx nylorun tenant mcp inspect <url> --agent <id>. A server that answers 401 needs a credential; see Preview an MCP server.
  • Check which of the servers and HTTP tools of the manifest version an Agent runs its secrets cover with admin.coverage({ agentId }) or npx nylorun tenant coverage --agent <id>; see Coverage.
  • 502 mcp_preview_failed means the preview could not list the tools; details.failure holds the code a tool call would fail with.
  • Through a gateway (a credential with via), credential_rejected and a failed discovery also mean the gateway reported that the server behind it answered 401, such as LiteLLM's auth_required for a token the provider revoked. Sign the person in again, or rotate the key, at the gateway.
  • With a workload credential, a credential_rejected call's tool.completed names in error.gateway the origin that refused the Runtime's workload token: check that the token file is current. A server refused at discovery means via's origin is not in the Runtime's allowed origins; see The workload token.
  • Tools listed with the gateway's prefix (github-create_issue where the server's own name is create_issue) mean the credential has no toolPrefix. Write it again with the prefix, then name the server's own tools in tools settings and tool policy rules. See Gateway tool prefix.
  • On a server, a refused private address is mcp.unreachable. See Docker Compose.

A tool call is refused or waits

The Runtime ends some calls itself. The model sees a failed tool result with one of these codes, and the call's tool.completed event says status: "failed".

CodeCause
tool.deniedA deny rule of the Tenant's tool policy refused the call, or a person denied a held call. It was not sent. If the message names a rule whose tool holds __, the stored policy fails closed: every MCP and HTTP call is denied and GET /v1/tenant/tool-policy answers 409 request_rejected. Put the policy again, naming MCP tools by server and tool.
tool.interruptedA crash or lost connection interrupted the call. An MCP or HTTP call was sent up to three times with the same key first; a sandbox command fails at once. It may have run.
tool.disabledThe agent's MCP server settings disable the tool.
mcp.lostThe call was sent and its answer lost: the tool may have run. The model decides whether to call it again.
artifact.unsupportedread_artifact named a file of a type the Tenant's model does not read. See Files.
tool.unavailableThe Runtime cannot run the tool. For schedule_wake, the manifest's runtime section does not set selfWake: true. See Triggers and wakes.
memory.forbiddenThe memory tool wrote from an agent whose memory is read_only, or from an agent used as a tool. See Agent memory.

A call that an ask rule holds pauses the turn with one approval interaction per call. session.toolCalls() lists it with status: "held"; resolve it with session.approve(). See Approve a held call.

A pod's refused outbound connection is an agent.egress event on the Agent's events, with outcome: "refused", a reason such as host_not_allowed, and the turns that were running (runningTurns). Read them with client.agents.events(id, { types: ["agent.egress"] }) or npx nylorun agents events <id> --types agent.egress. To allow the host, add it to network.allowedHosts in a new manifest version: a tracking Agent moves to it, and a pod applies it at its next start (npx nylorun agents restart <id> applies it now). See Egress records.

A session does not open

Opening a session or starting a turn checks the session's Agent.

CodeStatusFix
agent_not_granted403The token's identity-file agents grants do not reach the Agent. Add a grant for its id or prefix.
agent_owner_mismatch403The Agent's owner is another person. Use that person's Agent, or one without an owner. See Owner guard. A caller acting for a person gets 404.
agent_unavailable409The Agent was deleted, or it is a pod Agent and this Runtime has no sandbox pods or the sandboxes service is not ready. Make it again, or enable pods.
agent_lost409The pod's volume or node is gone. Reset it, or restore a disk version.
placement_refused409The Tenant's placement keeps the session's harness off the host its Agent needs.
harness_mismatch409The Agent's id was deleted and made again with another harness than the session pinned. Open a new session.
session_closed409The session was opened before protocol 16 (closedReason such as protocol_15 or protocol_16). Its history stays readable; open a new session.
credential_conflict409At the secret write, not at open: two credentials bound to one URL, or two secrets bound to one host, in one Agent. See Shell credentials.

agent_busy

Sessions on one Agent run their turns at the same time, one turn per session. 409 agent_busy, with Retry-After, answers work on the Agent itself while any of its sessions has a turn running: stop, reset, restart, delete, snapshot, restore, or a fork that copies its disk. A turn that starts while a disk copy runs is refused the same way. Wait for the turn or the copy to finish and try again. See Shared disks and turns.

An Agent or manifest is refused

CodeStatusFix
agent_fixed_field409A PUT /v1/agents/{id} would change what is fixed for the Agent's life: its manifest's name, its kind, its harness, or its storage, named in details.fields. Fork it with client.agents.fork() or npx nylorun agents fork.
secret_not_in_manifest409A secret write named a secret the Agent's manifest version does not declare (while the Agent moves, the version it moves to). Add the slot to the runtime section and save; a tracking Agent moves to the new version.
manifest_inconsistent422The manifest's harness and runtime sections do not fit. See Saving an agent manifest is refused.
folder_invalid422An agent manifest does not compile; details.diagnostics lists every problem by file, line, and column. Run npx nylorun check <dir>. See Diagnostics.
manifest_changed412The save's If-Match named a version that is no longer the latest. Read the manifest again, or save without If-Match.
manifest_in_use409An Agent was made from one of the manifest's versions, so it cannot be deleted. Delete those Agents first.
manifest_not_found404No agent manifest has this name, or no version this number.
limit_exceeded409The Tenant already holds limits.environments Agents. See Tenant limits.

See Agents and Saving manifests.

An Agent does not run its new version

Read the Agent with client.agents.get(id) or npx nylorun agents get <id>.

What the view saysWhyFix
targetVersion, waitingFor: "start"A pod applies a runtime change at its next start.Wait for the next start, or run npx nylorun agents restart <id>.
waitingFor: "secrets", state input_requiredThe version adds a required secret slot that has no value.Write the secret: npx nylorun agents secrets put <id> <name>.
State input_required, attention reason secret_unset: required secret ORDERS_TOKEN unsetA required secret of the version the Agent runs has no value, so a tool that needs it would fail.Write the secret; the attention item clears once it is set.
failedVersion, state failing, event agent.start_failedThe pod's start on that version failed (its setup, the pod, or a slow start), and the Agent went back to the version it ran.Fix it and save a new version, or try it again with npx nylorun agents restart <id> --retry.
A pinned version ("name@n")The Agent does not track its manifest.npx nylorun agents track <id>, or put it with another version.

npx nylorun agents list --outdated and client.agents.list({ outdated: true }) list the Agents moving to another version; list({ state: "failing" }) lists those that failed to start. See Versions.

A message is refused

CodeStatusFix
session_busy409The message was sent with delivery: "reject" (--reject-if-busy) while a turn runs or waits. Without it, the message is queued and starts the next turn. See Messages while a turn runs.
queue_full429The session already holds sessionQueuedMessages queued messages (100 by default). Wait for the running turn to end.
session_closed409The session was opened before protocol 16. Open a new session.
invalid_request400The message has more parts than messageParts, or cannot be read; the message names the limit.
limit_exceeded413The body is larger than requestBodyBytes, or the record entry it writes is larger than recordEntryBytes (details.limit).

Any file may be sent; the harness decides how the model sees it. A model call that fails with unsupported_input means the model or its API does not read a part the harness sent, such as a custom harness sending a text file as media. One that fails with invalid_request before anything is sent means the gateway cannot fetch a file the call names: not the session's, missing, larger than modelFileBytes, or audio other than WAV or MP3 on OpenAI-compatible completions. Neither is retried. See Files and Runtime limits.

A trigger or webhook does not run

Read what each fire came to with client.agents.runs(id) or npx nylorun agents runs <id>, newest first.

RunReasonFix
skippedoverlapA schedule already queued one run on its busy session; later fires are skipped until it starts.
skippedrate_limitedThe Agent passed triggerRunsPerHour.
skippedpausedThe trigger is paused. A trigger that a newer version adds arrives paused: npx nylorun agents triggers resume <id> <trigger>.
failedsession_conflictThe session the trigger names belongs to another Agent. Name another session, or "new".
Any, marked lateThe Tenant was stopped when the trigger was due. It fires once at the next start; missed times are not run.

Resuming an at trigger whose time has passed is 409 trigger_expired: save a version with a new time.

A webhook delivery to POST /v1/webhooks/{endpointId} answers:

StatusWhyFix
401 credential_invalidNo signing secret made yet, a missing webhook-* header, a webhook-timestamp more than 5 minutes off (triggers.webhookToleranceMs), or a signature that does not match.Make a secret with npx nylorun agents triggers new-secret <id> <trigger> (it replaces the one before) and give it to the sender.
404 not_foundNo endpoint has this id.Use the endpoint id from the trigger's webhookUrl.
409 trigger_pausedThe trigger is paused; nothing runs.npx nylorun agents triggers resume <id> <trigger>.
413The body is larger than webhookBodyBytes (256 KiB by default).Send a smaller payload, or raise the limit.

When a trigger's webhookUrl names http://localhost:<port>, the Runtime has no public URL: start it with NYLORUN_PUBLIC_URL set to the URL your proxy publishes. See Triggers and wakes and Docker Compose.

Memory

409 memory_conflict answers a PUT /v1/agents/{id}/memories/{path} whose contentSha256 no longer matches the memory, or whose path a folder or another memory already holds. Read the memory again and write on top of it. See Agent memory.

Custom harness gate codes

A harness of your own sends each MCP and HTTP tool call through the gateway under its run token.

CodeCause
400 key_requiredThe call carries no Idempotency-Key. Send the call's effect id.
409 effect_not_journaledThe effect journal holds no running call with that key for the run's turn, tool, and input. Journal the intent before you send the call.

See Harness API.

A turn fails at its start

A turn that fails at its start with agent.build-failed, naming a tool, means an agent's tool shares a name with one of our harness's own tools, such as an agent tool named bash on an Agent with a sandbox. Rename the agent's tool.

A custom harness that says an older api in hello is refused: rebuild it for Harness API 5. See Harness.

Traces do not arrive

A batch the collector refuses (429, 5xx, or a lost connection) is tried three times (traces.retries), then dropped and logged as trace_export_dropped in npx nylorun local logs runtime. Read the missed turns with session.trace(turnId) and post them to the collector. See Docker Compose.

Model errors and budgets

  • Confirm the provider on the Models tab of Studio's Tenant page or with npx nylorun tenant models get; change it with npx nylorun tenant models set.
  • model.budget_exhausted means a hard cap on PUT /v1/tenant/budgets was reached. A manifest budget caps every Agent made from that agent manifest.
  • Provider keys stay in the Tenant's model vault. Check npx nylorun local logs gateway.

Studio sign-in

On this machine, open Studio at its loopback URL (npx nylorun status prints it); it needs no login there. Over an SSH tunnel, forward to the same port and open http://localhost:<port>.

Error codes

ERROR_CODES is exported from @nylorun/agents:

CodeWhen
not_found404: missing route, session, Agent, agent manifest, trigger, or webhook endpoint.
credential_required401: no key or token was sent.
credential_invalid401: the Tenant does not know the key, the token fails a check, or a webhook delivery's signature does not verify.
protocol_unsupported426: the request's Nylorun-Protocol is outside the Runtime's range. The body names the range.
host_rejected421: the Host header is not an address the Runtime serves.
origin_rejected403: a key was sent with an Origin header.
unsupported_media_type415: the body's content type is not accepted.
connection_missingNo complete Runtime URL and key, or a link or credentials file that does not validate.
incompatible_hostProtocol or required-feature mismatch.
subject_invalid400: the Nylorun-Subject or Nylorun-Scopes header is malformed, repeated, or Nylorun-Scopes came without a subject. 403: a management key was sent with them.
scope_required403: the token or client.as() lacks the route's scope.
key_role_mismatch403: an application key on the Management API, or a management key on the Runtime API.
token_expired401: the token is past exp.
limit_exceededA Tenant limit was hit: a Runtime limit (413, details.limit), the most Agents the Tenant holds, a skill over the skill limit, or an artifact limit.
request_rejectedThe Runtime refused the request for the reason its message gives.
invalid_request400: the Runtime could not read the headers, body, or query.
subject_required400: a route that acts for a person was called without Nylorun-Subject.
agent_not_granted403: the token's agents grants do not reach the Agent.
session_busy409: a message sent with delivery: "reject" while the session's turn runs or waits.
queue_full429: the session already holds sessionQueuedMessages queued messages.
trigger_expired409: resuming an at trigger whose time has passed.
trigger_paused409: a signed delivery to a paused webhook trigger; nothing runs.
memory_conflict409: a memory write whose contentSha256 no longer matches, or a path a folder or another memory holds.
agent_busy409: stop, reset, restart, delete, snapshot, restore, or a disk-copying fork while a turn runs, or a turn while a disk copy runs.
agent_unavailable409: the Agent is gone, or its kind cannot run here.
placement_refused409: the Tenant's placement keeps the harness off the host.
agent_lost409: a pod Agent's volume or node is gone.
agent_owner_mismatch403: the Agent is for another owner's sessions.
harness_mismatch409: the Agent now runs another harness than the session pinned.
session_closed409: the session was opened before protocol 16.
issuer_unavailable401: a token names a new signing key and the issuer's JWKS cannot be fetched. Retry later.
definition_files_missing400: the manifest names files the Runtime does not hold.
credential_rejectedNot an HTTP answer: the tool error the model sees when an MCP server or HTTP tool answers 401, or a gateway reports that the server behind it did.
mcp_preview_failed502: an MCP server preview could not list its tools.
credential_conflict409: a secret write would bind two credentials to one URL or host.
agent_fixed_field409: a PUT would change an Agent's manifest name, kind, harness, or storage.
secret_not_in_manifest409: a secret write named a secret the Agent's manifest version does not declare.
manifest_inconsistent422: an agent manifest's harness and runtime sections do not fit.
folder_invalid422: an agent manifest does not compile.
manifest_not_found404: no agent manifest has this name or version.
manifest_in_use409: an Agent was made from the manifest.
manifest_changed412: If-Match named a version that is no longer the latest.
internal_error5xx: the Runtime failed. Check npx nylorun local logs runtime.

Use the code for control flow and the message for operator context.

Next step

On this page