# Monitor (/docs/deploy/monitor)



```sh
docker compose -f examples/observability/compose.yaml up -d
OTEL_EXPORTER_OTLP_ENDPOINT=http://host.docker.internal:4318 \
OTEL_METRICS_EXPORTER=prometheus,otlp \
npx nylorun start --tenant app
```

That sends the Runtime's metrics and spans to Grafana's all-in-one OpenTelemetry
backend on your machine (`compose.yaml` is in
[examples/observability](https://github.com/nylorun/agents/tree/main/examples/observability)).
Open Grafana at `http://localhost:3001` and import `grafana-dashboard.json`.

The Runtime keeps two records for two readers. The **event log** is what the
agent did and what the Runtime did to it: exact, never sampled, and read by
developers through the Runtime API and Studio (see [Events](/docs/run/events)
and [Traces](/docs/run/traces)). **Telemetry** is how the Runtime runs: OpenTelemetry
metrics, spans, and logs, best effort, for the operator and the tools they
already run. The Runtime stores no telemetry: metrics live in each process's
memory and go out over OTLP; spans go out over OTLP only. When the two disagree,
for example on a model call's tokens, the log is right.

## Health [#health]

| Probe                                                           | Answers                                                             |
| --------------------------------------------------------------- | ------------------------------------------------------------------- |
| `GET /health` on the Runtime (port 4000) and the gateway (4100) | The process is up, with its version and protocol.                   |
| `GET /ready`                                                    | The Runtime can serve: its listener, Tenant, Postgres, and Restate. |
| `GET /v1/tenant` (Management API)                               | The Tenant's checks, counts, sandboxes, harness, and streams.       |
| `nylorun status`                                                | The local stack's services and Studio's URL.                        |

## Metrics [#metrics]

Every runtime and gateway process serves Prometheus metrics on `/metrics`, port
`9464`, from the moment it starts. On a local Tenant the port is on the Compose
network only (`http://runtime:9464/metrics`, `http://gateway:9464/metrics`); on
Kubernetes it is the `metrics` port of the `<name>-runtime-internal` and
`<name>-gateway` Services (see [Kubernetes](/docs/deploy/kubernetes#telemetry)).
Set an OTLP endpoint and the same metrics are pushed there every minute too.

| Metric                                                         | Type, unit              | Attributes                                                                                               | Made by                                                      |
| -------------------------------------------------------------- | ----------------------- | -------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------ |
| `http.server.request.duration`                                 | histogram, s            | `http.request.method`, `http.route`, `http.response.status_code`, `nylorun.client`, `error.type` (a 5xx) | The runtime, for every API request.                          |
| `gen_ai.client.inference.duration`                             | histogram, s            | `gen_ai.operation.name`, `gen_ai.provider.name`, `gen_ai.request.model`, `error.type`                    | The gateway, for every model call.                           |
| `gen_ai.client.inference.time_to_first_chunk`                  | histogram, s            | As above                                                                                                 | The same, when the provider streamed.                        |
| `gen_ai.client.inference.usage.input_tokens`, `.output_tokens` | counter, `{token}`      | As above                                                                                                 | The same, for an answered call.                              |
| `mcp.client.operation.duration`                                | histogram, s            | `mcp.method.name`, `gen_ai.tool.name` (a call), `error.type`                                             | The gateway, for every request to a remote MCP server.       |
| `http.client.request.duration`                                 | histogram, s            | `http.request.method`, `http.response.status_code`, `error.type`                                         | The gateway, for every HTTP tool request.                    |
| `gen_ai.invoke_agent.duration`                                 | histogram, s            | `gen_ai.agent.name` (the agent manifest's name), `error.type` (`cancelled`, or the failure's code)       | The runtime, as each turn ends, waits for a person included. |
| `nylorun.queue.wait`                                           | histogram, s            | `nylorun.queue.outcome` (`taken`, `failed`, `cancelled`)                                                 | The runtime, as each queued message leaves the queue.        |
| `nylorun.egress.connections`                                   | counter, `{connection}` | `nylorun.egress.outcome` (`allowed`, `refused`), `nylorun.egress.reason`                                 | The egress gate, once per sandbox connection.                |
| `nylorun.egress.bytes`                                         | counter, By             | `network.io.direction` (`transmit`, `receive`)                                                           | The egress gate, for an allowed connection as it closes.     |
| `nylorun.telemetry.dropped`                                    | counter, `{item}`       | `signal` (`metrics`, `traces`), `exporter` (`otlp`, or `projector` for turn traces)                      | Every process, for what an export failed to deliver.         |

* **Names on `/metrics`.** Dots become underscores and counters end in
  `_total`, with no unit suffix: `http_server_request_duration_bucket`,
  `gen_ai_client_inference_usage_input_tokens_total`,
  `nylorun_egress_connections_total`. Each point carries
  `otel_scope_name="@nylorun/runtime"`; `service_name` and `service_version` are
  on `target_info`.
* **Bounded labels.** `http.route` is the route's pattern
  (`/v1/sessions/{sessionId}/commands`), and `other` for a path that names no
  route. A metric never carries a session, turn, request, or Tenant id, a host
  name, a URL, or a person.
* **`nylorun.client`** says who sent a request: `studio`, `sdk`
  (`@nylorun/agents` and `@nylorun/admin`), `cli`, `a2a` and `ag-ui` (by route),
  else `other`. Clients send it as the `Nylorun-Client` header. A browser app
  that calls the Runtime may send it too: allow the header in your proxy's CORS.

### The golden signals [#the-golden-signals]

| Part                       | Rate                                                                   | Errors                                                    | Duration                             | Saturation                      |
| -------------------------- | ---------------------------------------------------------------------- | --------------------------------------------------------- | ------------------------------------ | ------------------------------- |
| Runtime API                | `http.server.request.duration` count by `http.route`, `nylorun.client` | Points with `error.type`                                  | Its histogram                        | `nylorun.queue.wait`            |
| Turns                      | `gen_ai.invoke_agent.duration` count                                   | Points with `error.type`                                  | Its histogram                        | `nylorun.queue.wait` by outcome |
| Model calls                | `gen_ai.client.inference.duration` count by model                      | `error.type` (`rate_limited`, `overloaded`, `timeout`, …) | Its histogram, `time_to_first_chunk` | Input and output tokens         |
| MCP servers and HTTP tools | `mcp.client.operation.duration`, `http.client.request.duration` counts | `error.type`                                              | Their histograms                     |                                 |
| Egress                     | `nylorun.egress.connections`                                           | `outcome="refused"` by `reason`                           |                                      | `nylorun.egress.bytes`          |
| Telemetry itself           |                                                                        | `nylorun.telemetry.dropped`                               |                                      |                                 |

## Spans [#spans]

Set an OTLP endpoint and the Runtime exports spans; without one it makes none.

* **Live spans** from each process: a server span `{method} {http.route}` for
  every API request, with `nylorun.client`, and a client span `tools/call {tool}`
  for each remote MCP tool call. A request's span is the parent of the turn it
  starts, so a `traceparent` your app sends with a command makes your span, the
  request's, and the turn's one trace.
* **Turn traces** from the log: `invoke_agent`, `chat`, and `execute_tool`, one
  trace per turn, the same spans [`session.trace(turnId)`](/docs/run/traces)
  reads. How they leave is in [Docker Compose, Traces](/docs/deploy/vm#traces).
* **Egress has no span**: a connection belongs to the Agent's computer, not to a
  turn. Count it with `nylorun.egress.connections`, or read its `agent.egress`
  event (see [Egress records](/docs/run/agents#egress-records)).
* **Sampling.** `OTEL_TRACES_SAMPLER` decides which live spans are kept:
  `parentbased_always_on` (the default), `always_on`, `always_off`,
  `traceidratio`, or `parentbased_traceidratio` with `OTEL_TRACES_SAMPLER_ARG`
  (`0` to `1`). Metrics are never sampled. Every turn's trace is exported; sample
  those in your collector.

A harness must not instrument its own model calls: the Runtime already makes
their spans, and a second instrumentation counts every call twice. Spans for
work the Runtime cannot see, such as a retrieval step inside the harness, are
welcome.

## Content [#content]

By default spans carry references, never content: what a person sent, the
model's output, tool arguments and results, and error messages stay in the log,
which a span points at with `nylorun.event.cursor`. One switch, from the
OpenTelemetry GenAI semantic conventions, copies content onto the turn traces'
spans. Set it on the runtime.

| `OTEL_INSTRUMENTATION_GENAI_CAPTURE_CONTENT` | Spans carry                                                                                                                     |
| -------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| `NO_CONTENT` (default)                       | No content.                                                                                                                     |
| `SPAN_ONLY`                                  | `gen_ai.input.messages`, `gen_ai.output.messages`, `gen_ai.tool.call.arguments`, `gen_ai.tool.call.result`, and error messages. |
| `EVENT_ONLY`                                 | No content, with a warning: the Runtime emits no GenAI events.                                                                  |
| `SPAN_AND_EVENT`                             | As `SPAN_ONLY`, with a warning.                                                                                                 |

An unknown value means `NO_CONTENT`, with a warning. Content is recorded as
structured values, and each text in it is cut at the Tenant's
`traceContentBytes` ([Runtime limits](/docs/run/limits), 8 KiB by default). A
span never carries a credential, a prompt, a person's id, or an idempotency key,
whatever the switch says. Turn content on only where your backend may hold what
people and models wrote.

`NYLORUN_TRACE_CONTENT` and `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT`
still work for this release, with a `trace_content_deprecated` warning at start,
and are ignored once the switch is set. See [Compatibility](/docs/more/compatibility).

## Variables [#variables]

Set them on the runtime and the gateway. `nylorun start` passes them from its
shell to both; on Kubernetes, put them in `extraEnv.runtime` and
`extraEnv.gateway`.

| Variable                                                                    | Default                                    | Meaning                                                                              |
| --------------------------------------------------------------------------- | ------------------------------------------ | ------------------------------------------------------------------------------------ |
| `OTEL_METRICS_EXPORTER`                                                     | `prometheus`, plus `otlp` with an endpoint | A comma list of `prometheus`, `otlp`, and `none`.                                    |
| `OTEL_EXPORTER_PROMETHEUS_HOST`, `OTEL_EXPORTER_PROMETHEUS_PORT`            | `0.0.0.0`, `9464`                          | Where `/metrics` listens.                                                            |
| `OTEL_EXPORTER_OTLP_ENDPOINT`                                               | None                                       | The collector's OTLP/HTTP base URL: metrics to `/v1/metrics`, spans to `/v1/traces`. |
| `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT`, `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` | None                                       | One signal's full URL instead.                                                       |
| `OTEL_EXPORTER_OTLP_HEADERS`                                                | None                                       | `key=value` pairs separated by commas, values percent-encoded.                       |
| `OTEL_TRACES_SAMPLER`, `OTEL_TRACES_SAMPLER_ARG`                            | `parentbased_always_on`                    | Which live spans are kept.                                                           |
| `OTEL_SERVICE_NAME`                                                         | `nylorun-runtime`, `nylorun-gateway`       | `service.name`.                                                                      |
| `OTEL_RESOURCE_ATTRIBUTES`                                                  | None                                       | More resource attributes.                                                            |
| `OTEL_INSTRUMENTATION_GENAI_CAPTURE_CONTENT`                                | `NO_CONTENT`                               | Content on spans (above). The runtime only.                                          |

An unknown exporter, sampler, or ratio is skipped or replaced by its default,
with a warning. A `/metrics` port that is not a number stops the start.

## Logs [#logs]

Each service writes JSON lines to stdout: `level`, `message`, `time`, and
fields. The runtime logs one `request` line per API request (method, redacted
path, status, duration); the gateway logs `model_call`, `model_retry`,
`mcp_request`, and `http_tool_call`, and the egress gate `egress_refused`. With
span export on, a line written during a request carries `trace_id` and
`span_id`, the ids of the request's span. On a local Tenant,
`nylorun local logs <service>` reads them.

At start each process logs `telemetry_config`, then `metrics_ready` with its
`/metrics` URL, or `metrics_listen_failed` when the port is taken (the process
runs on without it). While an export fails, it logs `telemetry_export_dropped`
at most once a minute.

## Audit [#audit]

Who changed what the Tenant allows, and which credentials its sessions used, is
on the Tenant's own stream (Host feature `tenant-audit`). Like a session's
events it is a record: written with the change, never sampled, and kept by a
reset.

```ts
import { createAdmin } from "@nylorun/admin";

const admin = createAdmin();
let cursor: string | undefined;
do {
  const page = await admin.tenant.audit({ types: ["credential.used"], since: "2026-11-20T00:00:00Z", cursor });
  for (const event of page.events) console.log(event.time, event.principal, event.payload.target, event.payload.outcome);
  cursor = page.nextCursor ?? undefined;
} while (cursor);
```

| Event                                                            | When                                                                                       | Payload                                                     |
| ---------------------------------------------------------------- | ------------------------------------------------------------------------------------------ | ----------------------------------------------------------- |
| `key.created`, `key.revoked`                                     | An application or management key is created, rotated, or deleted.                          | `keyId`, `role`, `rotated`                                  |
| `signing_key.rotated`, `signing_key.revoked`                     | The signing keys rotate, or one is revoked.                                                | The key ids in each state, `force`                          |
| `policy.changed`                                                 | The tool policy is replaced.                                                               | `rules` (the count), `updatedAt`                            |
| `settings.changed`                                               | A Tenant setting is replaced, such as `limits.config` or `model`.                          | `key`, never the value                                      |
| `credential.created`, `credential.rotated`, `credential.deleted` | An Agent's secret is written, replaced, or deleted.                                        | `vaultId`, `credentialId`, `target`, `outcome`              |
| `credential.used`                                                | A gate asks for a credential for a session's MCP server or HTTP tool, approved or refused. | `vaultId`, `credentialId`, `sessionId`, `target`, `outcome` |
| `studio.signed_in`                                               | A person signs in to Studio.                                                               | `method`, `subject` when there is one                       |

Every event names who acted (`principal`), taken from the request's credential:
a key's name, `<key>:<subject>` for a key acting for a person, `operate` for
`nylorun-operate`, and `host` when the Runtime acted itself. No payload holds a
secret value. `GET /v1/tenant/audit` takes `since`, `until`, `types`
(comma-separated), `cursor`, and `limit` (default 50); it needs a management
key, and an application key gets `403 key_role_mismatch`.

## In Studio [#in-studio]

* **Tenant › Traffic** reads the runtime's and the gateway's `/metrics` every
  10 seconds: API requests by route and client, model calls and tokens, MCP tool
  calls, turns, queue waits, egress, and dropped telemetry, with counts since
  each process started, the current rate, and p50 and p95. Studio stores none of
  it. It reads the sources in `NYLORUN_STUDIO_METRICS_URLS` (`name=url` pairs),
  which `nylorun start` and the Helm chart set.
* **Tenant › Audit** shows the Tenant's audit, with a type filter.

For history, alerts, and dashboards, send the telemetry to your own backend.
[examples/observability](https://github.com/nylorun/agents/tree/main/examples/observability)
has a Grafana dashboard of the golden signals, and `collector.yaml`, an
OpenTelemetry Collector for production that keeps every trace with an error or a
slow turn and a quarter of the rest, then sends them to any OTLP backend.
