Monitor
Health, metrics, spans, logs, and the Tenant's audit, for the operator running the Runtime.
docker compose -f examples/observability/compose.yaml up -d
OTEL_EXPORTER_OTLP_ENDPOINT=http://host.docker.internal:4318 \
OTEL_METRICS_EXPORTER=prometheus,otlp \
npx nylorun start --tenant appThat sends the Runtime's metrics and spans to Grafana's all-in-one OpenTelemetry
backend on your machine (compose.yaml is in
examples/observability).
Open Grafana at http://localhost:3001 and import grafana-dashboard.json.
The Runtime keeps two records for two readers. The event log is what the agent did and what the Runtime did to it: exact, never sampled, and read by developers through the Runtime API and Studio (see Events and Traces). Telemetry is how the Runtime runs: OpenTelemetry metrics, spans, and logs, best effort, for the operator and the tools they already run. The Runtime stores no telemetry: metrics live in each process's memory and go out over OTLP; spans go out over OTLP only. When the two disagree, for example on a model call's tokens, the log is right.
Health
| Probe | Answers |
|---|---|
GET /health on the Runtime (port 4000) and the gateway (4100) | The process is up, with its version and protocol. |
GET /ready | The Runtime can serve: its listener, Tenant, Postgres, and Restate. |
GET /v1/tenant (Management API) | The Tenant's checks, counts, sandboxes, harness, and streams. |
nylorun status | The local stack's services and Studio's URL. |
Metrics
Every runtime and gateway process serves Prometheus metrics on /metrics, port
9464, from the moment it starts. On a local Tenant the port is on the Compose
network only (http://runtime:9464/metrics, http://gateway:9464/metrics); on
Kubernetes it is the metrics port of the <name>-runtime-internal and
<name>-gateway Services (see Kubernetes).
Set an OTLP endpoint and the same metrics are pushed there every minute too.
| Metric | Type, unit | Attributes | Made by |
|---|---|---|---|
http.server.request.duration | histogram, s | http.request.method, http.route, http.response.status_code, nylorun.client, error.type (a 5xx) | The runtime, for every API request. |
gen_ai.client.inference.duration | histogram, s | gen_ai.operation.name, gen_ai.provider.name, gen_ai.request.model, error.type | The gateway, for every model call. |
gen_ai.client.inference.time_to_first_chunk | histogram, s | As above | The same, when the provider streamed. |
gen_ai.client.inference.usage.input_tokens, .output_tokens | counter, {token} | As above | The same, for an answered call. |
mcp.client.operation.duration | histogram, s | mcp.method.name, gen_ai.tool.name (a call), error.type | The gateway, for every request to a remote MCP server. |
http.client.request.duration | histogram, s | http.request.method, http.response.status_code, error.type | The gateway, for every HTTP tool request. |
gen_ai.invoke_agent.duration | histogram, s | gen_ai.agent.name (the agent manifest's name), error.type (cancelled, or the failure's code) | The runtime, as each turn ends, waits for a person included. |
nylorun.queue.wait | histogram, s | nylorun.queue.outcome (taken, failed, cancelled) | The runtime, as each queued message leaves the queue. |
nylorun.egress.connections | counter, {connection} | nylorun.egress.outcome (allowed, refused), nylorun.egress.reason | The egress gate, once per sandbox connection. |
nylorun.egress.bytes | counter, By | network.io.direction (transmit, receive) | The egress gate, for an allowed connection as it closes. |
nylorun.telemetry.dropped | counter, {item} | signal (metrics, traces), exporter (otlp, or projector for turn traces) | Every process, for what an export failed to deliver. |
- Names on
/metrics. Dots become underscores and counters end in_total, with no unit suffix:http_server_request_duration_bucket,gen_ai_client_inference_usage_input_tokens_total,nylorun_egress_connections_total. Each point carriesotel_scope_name="@nylorun/runtime";service_nameandservice_versionare ontarget_info. - Bounded labels.
http.routeis the route's pattern (/v1/sessions/{sessionId}/commands), andotherfor a path that names no route. A metric never carries a session, turn, request, or Tenant id, a host name, a URL, or a person. nylorun.clientsays who sent a request:studio,sdk(@nylorun/agentsand@nylorun/admin),cli,a2aandag-ui(by route), elseother. Clients send it as theNylorun-Clientheader. A browser app that calls the Runtime may send it too: allow the header in your proxy's CORS.
The golden signals
| Part | Rate | Errors | Duration | Saturation |
|---|---|---|---|---|
| Runtime API | http.server.request.duration count by http.route, nylorun.client | Points with error.type | Its histogram | nylorun.queue.wait |
| Turns | gen_ai.invoke_agent.duration count | Points with error.type | Its histogram | nylorun.queue.wait by outcome |
| Model calls | gen_ai.client.inference.duration count by model | error.type (rate_limited, overloaded, timeout, …) | Its histogram, time_to_first_chunk | Input and output tokens |
| MCP servers and HTTP tools | mcp.client.operation.duration, http.client.request.duration counts | error.type | Their histograms | |
| Egress | nylorun.egress.connections | outcome="refused" by reason | nylorun.egress.bytes | |
| Telemetry itself | nylorun.telemetry.dropped |
Spans
Set an OTLP endpoint and the Runtime exports spans; without one it makes none.
- Live spans from each process: a server span
{method} {http.route}for every API request, withnylorun.client, and a client spantools/call {tool}for each remote MCP tool call. A request's span is the parent of the turn it starts, so atraceparentyour app sends with a command makes your span, the request's, and the turn's one trace. - Turn traces from the log:
invoke_agent,chat, andexecute_tool, one trace per turn, the same spanssession.trace(turnId)reads. How they leave is in Docker Compose, Traces. - Egress has no span: a connection belongs to the Agent's computer, not to a
turn. Count it with
nylorun.egress.connections, or read itsagent.egressevent (see Egress records). - Sampling.
OTEL_TRACES_SAMPLERdecides which live spans are kept:parentbased_always_on(the default),always_on,always_off,traceidratio, orparentbased_traceidratiowithOTEL_TRACES_SAMPLER_ARG(0to1). Metrics are never sampled. Every turn's trace is exported; sample those in your collector.
A harness must not instrument its own model calls: the Runtime already makes their spans, and a second instrumentation counts every call twice. Spans for work the Runtime cannot see, such as a retrieval step inside the harness, are welcome.
Content
By default spans carry references, never content: what a person sent, the
model's output, tool arguments and results, and error messages stay in the log,
which a span points at with nylorun.event.cursor. One switch, from the
OpenTelemetry GenAI semantic conventions, copies content onto the turn traces'
spans. Set it on the runtime.
OTEL_INSTRUMENTATION_GENAI_CAPTURE_CONTENT | Spans carry |
|---|---|
NO_CONTENT (default) | No content. |
SPAN_ONLY | gen_ai.input.messages, gen_ai.output.messages, gen_ai.tool.call.arguments, gen_ai.tool.call.result, and error messages. |
EVENT_ONLY | No content, with a warning: the Runtime emits no GenAI events. |
SPAN_AND_EVENT | As SPAN_ONLY, with a warning. |
An unknown value means NO_CONTENT, with a warning. Content is recorded as
structured values, and each text in it is cut at the Tenant's
traceContentBytes (Runtime limits, 8 KiB by default). A
span never carries a credential, a prompt, a person's id, or an idempotency key,
whatever the switch says. Turn content on only where your backend may hold what
people and models wrote.
NYLORUN_TRACE_CONTENT and OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT
still work for this release, with a trace_content_deprecated warning at start,
and are ignored once the switch is set. See Compatibility.
Variables
Set them on the runtime and the gateway. nylorun start passes them from its
shell to both; on Kubernetes, put them in extraEnv.runtime and
extraEnv.gateway.
| Variable | Default | Meaning |
|---|---|---|
OTEL_METRICS_EXPORTER | prometheus, plus otlp with an endpoint | A comma list of prometheus, otlp, and none. |
OTEL_EXPORTER_PROMETHEUS_HOST, OTEL_EXPORTER_PROMETHEUS_PORT | 0.0.0.0, 9464 | Where /metrics listens. |
OTEL_EXPORTER_OTLP_ENDPOINT | None | The collector's OTLP/HTTP base URL: metrics to /v1/metrics, spans to /v1/traces. |
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT, OTEL_EXPORTER_OTLP_TRACES_ENDPOINT | None | One signal's full URL instead. |
OTEL_EXPORTER_OTLP_HEADERS | None | key=value pairs separated by commas, values percent-encoded. |
OTEL_TRACES_SAMPLER, OTEL_TRACES_SAMPLER_ARG | parentbased_always_on | Which live spans are kept. |
OTEL_SERVICE_NAME | nylorun-runtime, nylorun-gateway | service.name. |
OTEL_RESOURCE_ATTRIBUTES | None | More resource attributes. |
OTEL_INSTRUMENTATION_GENAI_CAPTURE_CONTENT | NO_CONTENT | Content on spans (above). The runtime only. |
An unknown exporter, sampler, or ratio is skipped or replaced by its default,
with a warning. A /metrics port that is not a number stops the start.
Logs
Each service writes JSON lines to stdout: level, message, time, and
fields. The runtime logs one request line per API request (method, redacted
path, status, duration); the gateway logs model_call, model_retry,
mcp_request, and http_tool_call, and the egress gate egress_refused. With
span export on, a line written during a request carries trace_id and
span_id, the ids of the request's span. On a local Tenant,
nylorun local logs <service> reads them.
At start each process logs telemetry_config, then metrics_ready with its
/metrics URL, or metrics_listen_failed when the port is taken (the process
runs on without it). While an export fails, it logs telemetry_export_dropped
at most once a minute.
Audit
Who changed what the Tenant allows, and which credentials its sessions used, is
on the Tenant's own stream (Host feature tenant-audit). Like a session's
events it is a record: written with the change, never sampled, and kept by a
reset.
import { createAdmin } from "@nylorun/admin";
const admin = createAdmin();
let cursor: string | undefined;
do {
const page = await admin.tenant.audit({ types: ["credential.used"], since: "2026-11-20T00:00:00Z", cursor });
for (const event of page.events) console.log(event.time, event.principal, event.payload.target, event.payload.outcome);
cursor = page.nextCursor ?? undefined;
} while (cursor);| Event | When | Payload |
|---|---|---|
key.created, key.revoked | An application or management key is created, rotated, or deleted. | keyId, role, rotated |
signing_key.rotated, signing_key.revoked | The signing keys rotate, or one is revoked. | The key ids in each state, force |
policy.changed | The tool policy is replaced. | rules (the count), updatedAt |
settings.changed | A Tenant setting is replaced, such as limits.config or model. | key, never the value |
credential.created, credential.rotated, credential.deleted | An Agent's secret is written, replaced, or deleted. | vaultId, credentialId, target, outcome |
credential.used | A gate asks for a credential for a session's MCP server or HTTP tool, approved or refused. | vaultId, credentialId, sessionId, target, outcome |
studio.signed_in | A person signs in to Studio. | method, subject when there is one |
Every event names who acted (principal), taken from the request's credential:
a key's name, <key>:<subject> for a key acting for a person, operate for
nylorun-operate, and host when the Runtime acted itself. No payload holds a
secret value. GET /v1/tenant/audit takes since, until, types
(comma-separated), cursor, and limit (default 50); it needs a management
key, and an application key gets 403 key_role_mismatch.
In Studio
- Tenant › Traffic reads the runtime's and the gateway's
/metricsevery 10 seconds: API requests by route and client, model calls and tokens, MCP tool calls, turns, queue waits, egress, and dropped telemetry, with counts since each process started, the current rate, and p50 and p95. Studio stores none of it. It reads the sources inNYLORUN_STUDIO_METRICS_URLS(name=urlpairs), whichnylorun startand the Helm chart set. - Tenant › Audit shows the Tenant's audit, with a type filter.
For history, alerts, and dashboards, send the telemetry to your own backend.
examples/observability
has a Grafana dashboard of the golden signals, and collector.yaml, an
OpenTelemetry Collector for production that keeps every trace with an error or a
slow turn and a quarter of the rest, then sends them to any OTLP backend.