NylorunDocsBeta
Deploy

Monitor

Health, metrics, spans, logs, and the Tenant's audit, for the operator running the Runtime.

docker compose -f examples/observability/compose.yaml up -d
OTEL_EXPORTER_OTLP_ENDPOINT=http://host.docker.internal:4318 \
OTEL_METRICS_EXPORTER=prometheus,otlp \
npx nylorun start --tenant app

That sends the Runtime's metrics and spans to Grafana's all-in-one OpenTelemetry backend on your machine (compose.yaml is in examples/observability). Open Grafana at http://localhost:3001 and import grafana-dashboard.json.

The Runtime keeps two records for two readers. The event log is what the agent did and what the Runtime did to it: exact, never sampled, and read by developers through the Runtime API and Studio (see Events and Traces). Telemetry is how the Runtime runs: OpenTelemetry metrics, spans, and logs, best effort, for the operator and the tools they already run. The Runtime stores no telemetry: metrics live in each process's memory and go out over OTLP; spans go out over OTLP only. When the two disagree, for example on a model call's tokens, the log is right.

Health

ProbeAnswers
GET /health on the Runtime (port 4000) and the gateway (4100)The process is up, with its version and protocol.
GET /readyThe Runtime can serve: its listener, Tenant, Postgres, and Restate.
GET /v1/tenant (Management API)The Tenant's checks, counts, sandboxes, harness, and streams.
nylorun statusThe local stack's services and Studio's URL.

Metrics

Every runtime and gateway process serves Prometheus metrics on /metrics, port 9464, from the moment it starts. On a local Tenant the port is on the Compose network only (http://runtime:9464/metrics, http://gateway:9464/metrics); on Kubernetes it is the metrics port of the <name>-runtime-internal and <name>-gateway Services (see Kubernetes). Set an OTLP endpoint and the same metrics are pushed there every minute too.

MetricType, unitAttributesMade by
http.server.request.durationhistogram, shttp.request.method, http.route, http.response.status_code, nylorun.client, error.type (a 5xx)The runtime, for every API request.
gen_ai.client.inference.durationhistogram, sgen_ai.operation.name, gen_ai.provider.name, gen_ai.request.model, error.typeThe gateway, for every model call.
gen_ai.client.inference.time_to_first_chunkhistogram, sAs aboveThe same, when the provider streamed.
gen_ai.client.inference.usage.input_tokens, .output_tokenscounter, {token}As aboveThe same, for an answered call.
mcp.client.operation.durationhistogram, smcp.method.name, gen_ai.tool.name (a call), error.typeThe gateway, for every request to a remote MCP server.
http.client.request.durationhistogram, shttp.request.method, http.response.status_code, error.typeThe gateway, for every HTTP tool request.
gen_ai.invoke_agent.durationhistogram, sgen_ai.agent.name (the agent manifest's name), error.type (cancelled, or the failure's code)The runtime, as each turn ends, waits for a person included.
nylorun.queue.waithistogram, snylorun.queue.outcome (taken, failed, cancelled)The runtime, as each queued message leaves the queue.
nylorun.egress.connectionscounter, {connection}nylorun.egress.outcome (allowed, refused), nylorun.egress.reasonThe egress gate, once per sandbox connection.
nylorun.egress.bytescounter, Bynetwork.io.direction (transmit, receive)The egress gate, for an allowed connection as it closes.
nylorun.telemetry.droppedcounter, {item}signal (metrics, traces), exporter (otlp, or projector for turn traces)Every process, for what an export failed to deliver.
  • Names on /metrics. Dots become underscores and counters end in _total, with no unit suffix: http_server_request_duration_bucket, gen_ai_client_inference_usage_input_tokens_total, nylorun_egress_connections_total. Each point carries otel_scope_name="@nylorun/runtime"; service_name and service_version are on target_info.
  • Bounded labels. http.route is the route's pattern (/v1/sessions/{sessionId}/commands), and other for a path that names no route. A metric never carries a session, turn, request, or Tenant id, a host name, a URL, or a person.
  • nylorun.client says who sent a request: studio, sdk (@nylorun/agents and @nylorun/admin), cli, a2a and ag-ui (by route), else other. Clients send it as the Nylorun-Client header. A browser app that calls the Runtime may send it too: allow the header in your proxy's CORS.

The golden signals

PartRateErrorsDurationSaturation
Runtime APIhttp.server.request.duration count by http.route, nylorun.clientPoints with error.typeIts histogramnylorun.queue.wait
Turnsgen_ai.invoke_agent.duration countPoints with error.typeIts histogramnylorun.queue.wait by outcome
Model callsgen_ai.client.inference.duration count by modelerror.type (rate_limited, overloaded, timeout, …)Its histogram, time_to_first_chunkInput and output tokens
MCP servers and HTTP toolsmcp.client.operation.duration, http.client.request.duration countserror.typeTheir histograms
Egressnylorun.egress.connectionsoutcome="refused" by reasonnylorun.egress.bytes
Telemetry itselfnylorun.telemetry.dropped

Spans

Set an OTLP endpoint and the Runtime exports spans; without one it makes none.

  • Live spans from each process: a server span {method} {http.route} for every API request, with nylorun.client, and a client span tools/call {tool} for each remote MCP tool call. A request's span is the parent of the turn it starts, so a traceparent your app sends with a command makes your span, the request's, and the turn's one trace.
  • Turn traces from the log: invoke_agent, chat, and execute_tool, one trace per turn, the same spans session.trace(turnId) reads. How they leave is in Docker Compose, Traces.
  • Egress has no span: a connection belongs to the Agent's computer, not to a turn. Count it with nylorun.egress.connections, or read its agent.egress event (see Egress records).
  • Sampling. OTEL_TRACES_SAMPLER decides which live spans are kept: parentbased_always_on (the default), always_on, always_off, traceidratio, or parentbased_traceidratio with OTEL_TRACES_SAMPLER_ARG (0 to 1). Metrics are never sampled. Every turn's trace is exported; sample those in your collector.

A harness must not instrument its own model calls: the Runtime already makes their spans, and a second instrumentation counts every call twice. Spans for work the Runtime cannot see, such as a retrieval step inside the harness, are welcome.

Content

By default spans carry references, never content: what a person sent, the model's output, tool arguments and results, and error messages stay in the log, which a span points at with nylorun.event.cursor. One switch, from the OpenTelemetry GenAI semantic conventions, copies content onto the turn traces' spans. Set it on the runtime.

OTEL_INSTRUMENTATION_GENAI_CAPTURE_CONTENTSpans carry
NO_CONTENT (default)No content.
SPAN_ONLYgen_ai.input.messages, gen_ai.output.messages, gen_ai.tool.call.arguments, gen_ai.tool.call.result, and error messages.
EVENT_ONLYNo content, with a warning: the Runtime emits no GenAI events.
SPAN_AND_EVENTAs SPAN_ONLY, with a warning.

An unknown value means NO_CONTENT, with a warning. Content is recorded as structured values, and each text in it is cut at the Tenant's traceContentBytes (Runtime limits, 8 KiB by default). A span never carries a credential, a prompt, a person's id, or an idempotency key, whatever the switch says. Turn content on only where your backend may hold what people and models wrote.

NYLORUN_TRACE_CONTENT and OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT still work for this release, with a trace_content_deprecated warning at start, and are ignored once the switch is set. See Compatibility.

Variables

Set them on the runtime and the gateway. nylorun start passes them from its shell to both; on Kubernetes, put them in extraEnv.runtime and extraEnv.gateway.

VariableDefaultMeaning
OTEL_METRICS_EXPORTERprometheus, plus otlp with an endpointA comma list of prometheus, otlp, and none.
OTEL_EXPORTER_PROMETHEUS_HOST, OTEL_EXPORTER_PROMETHEUS_PORT0.0.0.0, 9464Where /metrics listens.
OTEL_EXPORTER_OTLP_ENDPOINTNoneThe collector's OTLP/HTTP base URL: metrics to /v1/metrics, spans to /v1/traces.
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT, OTEL_EXPORTER_OTLP_TRACES_ENDPOINTNoneOne signal's full URL instead.
OTEL_EXPORTER_OTLP_HEADERSNonekey=value pairs separated by commas, values percent-encoded.
OTEL_TRACES_SAMPLER, OTEL_TRACES_SAMPLER_ARGparentbased_always_onWhich live spans are kept.
OTEL_SERVICE_NAMEnylorun-runtime, nylorun-gatewayservice.name.
OTEL_RESOURCE_ATTRIBUTESNoneMore resource attributes.
OTEL_INSTRUMENTATION_GENAI_CAPTURE_CONTENTNO_CONTENTContent on spans (above). The runtime only.

An unknown exporter, sampler, or ratio is skipped or replaced by its default, with a warning. A /metrics port that is not a number stops the start.

Logs

Each service writes JSON lines to stdout: level, message, time, and fields. The runtime logs one request line per API request (method, redacted path, status, duration); the gateway logs model_call, model_retry, mcp_request, and http_tool_call, and the egress gate egress_refused. With span export on, a line written during a request carries trace_id and span_id, the ids of the request's span. On a local Tenant, nylorun local logs <service> reads them.

At start each process logs telemetry_config, then metrics_ready with its /metrics URL, or metrics_listen_failed when the port is taken (the process runs on without it). While an export fails, it logs telemetry_export_dropped at most once a minute.

Audit

Who changed what the Tenant allows, and which credentials its sessions used, is on the Tenant's own stream (Host feature tenant-audit). Like a session's events it is a record: written with the change, never sampled, and kept by a reset.

import { createAdmin } from "@nylorun/admin";

const admin = createAdmin();
let cursor: string | undefined;
do {
  const page = await admin.tenant.audit({ types: ["credential.used"], since: "2026-11-20T00:00:00Z", cursor });
  for (const event of page.events) console.log(event.time, event.principal, event.payload.target, event.payload.outcome);
  cursor = page.nextCursor ?? undefined;
} while (cursor);
EventWhenPayload
key.created, key.revokedAn application or management key is created, rotated, or deleted.keyId, role, rotated
signing_key.rotated, signing_key.revokedThe signing keys rotate, or one is revoked.The key ids in each state, force
policy.changedThe tool policy is replaced.rules (the count), updatedAt
settings.changedA Tenant setting is replaced, such as limits.config or model.key, never the value
credential.created, credential.rotated, credential.deletedAn Agent's secret is written, replaced, or deleted.vaultId, credentialId, target, outcome
credential.usedA gate asks for a credential for a session's MCP server or HTTP tool, approved or refused.vaultId, credentialId, sessionId, target, outcome
studio.signed_inA person signs in to Studio.method, subject when there is one

Every event names who acted (principal), taken from the request's credential: a key's name, <key>:<subject> for a key acting for a person, operate for nylorun-operate, and host when the Runtime acted itself. No payload holds a secret value. GET /v1/tenant/audit takes since, until, types (comma-separated), cursor, and limit (default 50); it needs a management key, and an application key gets 403 key_role_mismatch.

In Studio

  • Tenant › Traffic reads the runtime's and the gateway's /metrics every 10 seconds: API requests by route and client, model calls and tokens, MCP tool calls, turns, queue waits, egress, and dropped telemetry, with counts since each process started, the current rate, and p50 and p95. Studio stores none of it. It reads the sources in NYLORUN_STUDIO_METRICS_URLS (name=url pairs), which nylorun start and the Helm chart set.
  • Tenant › Audit shows the Tenant's audit, with a type filter.

For history, alerts, and dashboards, send the telemetry to your own backend. examples/observability has a Grafana dashboard of the golden signals, and collector.yaml, an OpenTelemetry Collector for production that keeps every trace with an error or a slow turn and a quarter of the rest, then sends them to any OTLP backend.

On this page