# Traces (/docs/run/traces) ```ts const trace = await session.trace(turnId); for (const { scopeSpans } of trace.resourceSpans) for (const { spans } of scopeSpans) for (const span of spans) console.log(span.name, span.startTimeUnixNano, span.endTimeUnixNano); ``` Each turn is one OpenTelemetry trace, built from the session's log. The Runtime exports it to an OTLP collector when the operator sets one up; see [Docker Compose](/docs/deploy/vm#traces). You can read any turn's trace from the Runtime without a collector. ## Read a turn's trace [#read-a-turns-trace] `session.trace(turnId)` calls `GET /v1/sessions/{id}/turns/{turnId}/trace` and returns the turn's spans as OTLP/HTTP JSON (`resourceSpans`), the same spans the exporter posts. It needs the Host feature `traces` and an application key. Post the answer to a collector's `/v1/traces` to backfill a turn it missed: the ids are the same on every read and export. | Span | What it covers | | --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `invoke_agent ` | The turn, from its start to its end, named after its agent manifest. The root of every other span. `gen_ai.agent.name` is the manifest's name, `gen_ai.agent.id` the [Agent's](/docs/run/agents) id, and `nylorun.manifest.version` the version the turn ran. | | `chat ` | One model call, with the Model Gate's own start, end, usage, and cost. | | `execute_tool ` | One tool call, or one flow HTTP stage. `gen_ai.tool.name` is the name the model called it by, and `nylorun.tool.identity` the tool's identity (an MCP tool's own name, or an HTTP tool's declared name). An MCP or HTTP call has the Tool Gate's own start and end. | | `turn.paused`, `command.approve`, `command.respond` | A marker with no length, for each pause and answer. | A turn still running has no root span yet. A trace holds no egress: the connections a pod opens are `agent.egress` events on its Agent's stream, each naming the turns that were running. To see a turn's, read them with `client.agents.events(id, { types: ["agent.egress"], since, until })` around the turn; see [Egress records](/docs/run/agents#egress-records). Attribute names follow the OpenTelemetry GenAI semantic conventions; the rest are `nylorun.*`. When the harness reported other usage than the gate measured, the span carries `nylorun.usage.mismatch`. By default a span carries references, not content: `nylorun.event.cursor` points at the log event it came from, which [`history()`](/docs/run/events) reads. What a person said, the model's output, tool results, and error messages are left out unless the operator opts in per kind; see [Docker Compose](/docs/deploy/vm#traces). A span never carries a credential value, a prompt, a person's id, an identity header, or an idempotency key. ## Trace context [#trace-context] Send a W3C `traceparent` header, and `tracestate` if you have one, and the turn joins your trace: its root span becomes a child of your span. The Runtime reads the header on: | Request | Endpoint | | -------------------------- | -------------------------------------------------------------------------------------------- | | A message command | [`POST /v1/sessions/{id}/commands`](/reference/runtime/sessions/send-a-command-to-a-session) | | An AG-UI run on an Agent | [`POST /v1/ag-ui/agents/{agentId}`](/reference/runtime/sessions/ag-ui/run-an-agent) | | An A2A message to an Agent | [`POST /v1/a2a/agents/{agentId}`](/reference/runtime/sessions/a2a/call-an-agent-over-a2a) | `@nylorun/agents` has no trace option, and `createAgUiHandler()` and `createA2aHandler()` do not pass the browser's headers on. Give the client a `fetch` that adds the header from your tracer, for example with `@opentelemetry/api`: ```ts import { createClient } from "@nylorun/agents"; import { context, propagation } from "@opentelemetry/api"; const client = createClient({ url: process.env.NYLORUN_RUNTIME_URL!, key: process.env.NYLORUN_SERVER_KEY!, fetch: (input, init) => { const headers = new Headers(init?.headers); const carrier: Record = {}; propagation.inject(context.active(), carrier); for (const [name, value] of Object.entries(carrier)) headers.set(name, value); return fetch(input, { ...init, headers }); }, }); ``` Pass that client to `createAgUiHandler({ client })` or `createA2aHandler({ client })` and their runs join the trace active in your server. The Runtime reads the header only on the requests in the table; other requests ignore it. The turn hands its own context on, so your services see the calls as part of the same trace: | Receiver | How | | ------------------------------------------- | ------------------------------------------------- | | [HTTP tools](/docs/build/tools#the-request) | A `traceparent` header, beside `Idempotency-Key`. | | [MCP servers](/docs/build/mcp) | A `traceparent` header on the call's requests. | | Sandbox commands | The `TRACEPARENT` environment variable. | | A custom [harness](/docs/build/harness) | `trace` on the turn's start. | ## Per-call reads [#per-call-reads] For one session's calls without spans, page the call records: ```ts const tools = await session.toolCalls({ turnId }); for (const call of tools.calls) console.log(call.toolName, call.status, call.attempts, call.durationMs); const models = await session.modelCalls({ turnId }); for (const call of models.calls) console.log(call.model, call.outcome, call.usage?.costUsd); ``` | Call | Each entry | | --------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [`session.toolCalls()`](/reference/runtime/sessions/page-a-sessions-tool-calls) | `effectId`, `toolName`, `kind`, `server`, `tool` (the tool's identity), `skill`, `plugin`, `input`, `status` (`invoking`, `held`, `completed`, `failed`, or `cancelled`), `attempts`, `error`, and timing, in the order the calls were first sent. Host feature `tool-calls`. | | [`session.modelCalls()`](/reference/runtime/sessions/page-a-sessions-model-calls) | `effectId`, `provider`, `model`, `outcome`, `startedAt`, `endedAt`, and `usage` with its cost by token kind (`costSplit`), as the Model Gate metered them. | Both take `turnId`, `limit` (default 50, at most 200), and `cursor`; pass the answer's `nextCursor` to read on. The `effectId` joins a call to its span and to its row in the [session log](/docs/run/events). | Call | Endpoint | | ----------------------- | --------------------------------------------------------------------------------------- | | `session.trace(turnId)` | [Read a turn's trace](/reference/runtime/sessions/read-a-turns-trace) | | `session.toolCalls()` | [Page a session's tool calls](/reference/runtime/sessions/page-a-sessions-tool-calls) | | `session.modelCalls()` | [Page a session's model calls](/reference/runtime/sessions/page-a-sessions-model-calls) | ## Next step [#next-step]