# Models (/docs/run/models)



The Tenant holds the model credential. Your app and the session's Agent never
see it.

```sh
npx nylorun tenant models set
```

The command lists the Tenant's model catalog, asks for a provider, a model, and
its sign-in (an API key, or the provider's OAuth), and sets the Tenant's model.
It does not make a test request; send a message in Studio to verify access.
The **Models** tab of Studio's [Tenant page](/docs/run/studio#the-tenant) uses
the same routes. The first run installs the
provider sign-in library under `~/.nylorun/lib/`.

For CI, pass the provider and model, and pipe the API key in on standard input.
The key is never an argument, and a terminal on standard input is refused
(exit 2). OAuth providers need the prompts.

```sh
printenv OPENAI_API_KEY | npx nylorun tenant models set --provider openai --model gpt-5
```

| Command                                                             | What it does                                                                                                             |
| ------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ |
| `tenant models set`                                                 | Prompts for a provider, model, and sign-in.                                                                              |
| `tenant models set --provider <id> --model <id> [--base-url <url>]` | The same with the API key on standard input. `--base-url` is for a custom OpenAI-compatible endpoint, provider `custom`. |
| `tenant models select <provider> [--model <id>]`                    | Switches to a provider that already has a credential, keeping its model (or `--model`) and endpoint.                     |
| `tenant models get [--json]`                                        | The Tenant's model.                                                                                                      |
| `tenant models providers [--json]`                                  | The providers with a credential, and the one in use.                                                                     |
| `tenant models catalog [--json]`                                    | Every provider and model the Runtime knows.                                                                              |

## A session's model [#a-sessions-model]

`GET /v1/sessions/{id}` answers `model`: the model the session's turns call, and
why.

```json
{
  "model": {
    "provider": "google-vertex",
    "model": "gemini-4-argon",
    "source": "tenant",
    "reason": "provider_not_connected",
    "requested": { "provider": "anthropic", "model": "claude-opus-5-5", "by": "manifest" }
  }
}
```

| Field       | Meaning                                                                                                                                  |
| ----------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| `source`    | `manifest`: the model its Agent's version names (`model` in `AGENT.md`). `session`: its own choice. `tenant`: the Tenant's model.        |
| `reason`    | On a fallback to the Tenant's model, why the model asked for isn't used: `provider_not_connected`, `not_in_catalog`, or `invalid_model`. |
| `requested` | The model asked for, and `by` whom (`manifest` or `session`).                                                                            |

It is `null` for a flow, and while the Tenant has no model. Each change is a
[`session.model`](/docs/run/events#turn-payloads) event, with the `previous`
record. `SessionView.model` types it in `@nylorun/agents`.

## First-start seed [#first-start-seed]

On the first `nylorun start`, an empty Tenant can take these from `.env`:

```dotenv
MODEL_PROVIDER=openai
MODEL=gpt-4.1
MODEL_PROVIDER_API_KEY=sk-...
MODEL_PROVIDER_BASE_URL=https://your-provider.example/v1
```

`MODEL_PROVIDER_BASE_URL` is required only for a custom OpenAI-compatible
endpoint. After the first write, the Tenant's model vault is authoritative.

## Custom and local models [#custom-and-local-models]

The Runtime lists providers and models at `GET /v1/tenant/providers` and
`GET /v1/tenant/models`. Point a custom OpenAI-compatible endpoint at its base
URL. A local server such as Ollama is reached at `host.docker.internal`.

```sh
printenv MODEL_KEY | npx nylorun tenant models set --provider custom \
  --model llama3.2 --base-url http://host.docker.internal:11434/v1
```

The Runtime cannot learn some facts from a custom endpoint, so you give them
in `settings` when you set the model (`admin.models.put`, `PUT /v1/tenant/model`):

| Setting         | Default   | Meaning                                                                                                     |
| --------------- | --------- | ----------------------------------------------------------------------------------------------------------- |
| `contextWindow` | 32,768    | The model's context window, in tokens.                                                                      |
| `maxTokens`     | 8,192     | The most tokens one answer may hold.                                                                        |
| `reasoning`     | Unset     | Whether the model reasons.                                                                                  |
| `inputs`        | Text only | What else it reads: `image`, `pdf`, `audio`, or `video`. See [Inputs](#inputs).                             |
| `compat`        | Unset     | Compatibility settings for the endpoint's OpenAI dialect, such as `thinkingFormat` or `chatTemplateKwargs`. |
| `headers`       | Unset     | Headers sent on every call, such as a gateway's routing header.                                             |
| `callHeaders`   | Unset     | Headers whose value comes from the calling Agent, per call. See [Headers per call](#headers-per-call).      |

The defaults are small on purpose: an early compaction is better than a failed
call. Set them from the model's documentation.

### Headers per call [#headers-per-call]

A custom endpoint behind a model gateway, such as LiteLLM, can carry headers
that are not credentials. A Runtime that serves several organizations' Agents
can name each call's organization, so the gateway attributes and budgets it
while the call is made:

```ts
await admin.models.put({
  idempotencyKey: crypto.randomUUID(),
  provider: "custom",
  model: "claude",
  baseUrl: "https://litellm.internal/v1",
  auth: { type: "api_key", key: process.env.GATEWAY_KEY! },
  settings: {
    callHeaders: { "x-nylorun-org": { from: "agent.label", key: "org" } },
  },
});
```

A `callHeaders` value takes its source from the Agent that makes the call:

| Source                         | Sends                                     |
| ------------------------------ | ----------------------------------------- |
| `{ from: "agent.label", key }` | The value of the Agent's label `key`.     |
| `{ from: "agent.id" }`         | The Agent's id.                           |
| `{ from: "session.owner" }`    | The session's owner, when it is a person. |

A source with no value sends no header. Both maps are settings, read back by
`GET /v1/tenant/model`, never secrets: neither may set `authorization` (the
`auth` sends it), a hop-by-hop or transport header, or `Nylorun-*`, and a name
appears once across both. Studio keeps them when it saves the model. A Runtime
that serves one organization needs none of this: the gateway's key already
names it.

### The workload token [#the-workload-token]

A custom endpoint may take `auth: { type: "workload" }` in place of a key: the
Runtime sends its workload token, a short-lived token your platform rotates
through a file, as the endpoint's bearer, read at each call. Only provider
`custom` takes it. The operator gives the Runtime the token; see
[The workload token](/docs/deploy/vm#the-workload-token). `nylorun start` does
not pass the workload token's variables yet, so a local Tenant cannot use it.

## Inputs [#inputs]

Every model reads `text`. A model also reads `image`, `pdf`, `office` (Word,
Excel, PowerPoint), `audio`, or `video` when its catalog says so, or when you
add the input and its API carries it.

| Field           | Where                                       | Meaning                                                     |
| --------------- | ------------------------------------------- | ----------------------------------------------------------- |
| `inputs`        | Each model in `GET /v1/tenant/models`       | What it reads: its catalog's inputs and the ones you added. |
| `addableInputs` | Each model in `GET /v1/tenant/models`       | What its API carries that its catalog does not list.        |
| `inputs`        | Each provider in `GET /v1/tenant/providers` | What the provider's current model reads.                    |

| Input            | APIs that can carry it                                                             |
| ---------------- | ---------------------------------------------------------------------------------- |
| `image`          | Every API                                                                          |
| `pdf`            | Anthropic, OpenAI (Responses and OpenAI-compatible completions), Azure, and Google |
| `office`         | OpenAI Responses                                                                   |
| `audio`, `video` | Google, and OpenAI-compatible completions                                          |

Add inputs to a catalog model with the Management API:

```ts
import { createAdmin } from "@nylorun/admin";

const admin = createAdmin();

await admin.models.setInputs({
  provider: "openai",
  model: "gpt-4.1",
  inputs: ["pdf"],
});
```

`setInputs` (`PUT /v1/tenant/models/inputs`) takes only inputs in the model's
`addableInputs` and replaces the ones you added before; its catalog's inputs
always stay, and an empty list removes the additions. The Tenant keeps them in
its setting `model.inputs`.

**A custom endpoint reads text only by default.** Set what it reads in
`settings.inputs` when you set the model: `image`, `pdf`, `audio`, or `video`.
`office` is refused there.

```ts
await admin.models.put({
  idempotencyKey: crypto.randomUUID(),
  provider: "custom",
  model: "llava",
  baseUrl: "http://host.docker.internal:11434/v1",
  settings: { inputs: ["image"], contextWindow: 131_072 },
  auth: { type: "api_key", key: process.env.MODEL_KEY! },
});
```

On the Tenant page's **Models** tab in Studio, each connected provider's model has an **Inputs**
line: what its catalog reads is checked and fixed, and what you may add toggles
through `setInputs`. The custom endpoint's form has **Images**, **PDF**,
**Audio**, and **Video** checkboxes, saved in `settings.inputs`.

Any file may be sent in a message, whatever the model reads. The harness
learns the model's inputs when each turn starts (`turn.start.model`) and
decides how each file reaches the model: Nylorun's harness sends a file the
model reads as itself, inlines a text file, and writes a note for anything
else. A [custom harness](/docs/build/harness) that sends a kind the model
cannot read fails the call with the model failure `unsupported_input`, and a
file past the Runtime limit `modelFileBytes` fails it with `invalid_request`.
See [Files](/docs/run/files#what-the-model-reads).

## Timeouts and retries [#timeouts-and-retries]

The Model Gate makes each provider call within the Tenant's
[Runtime limits](/docs/run/limits), read for each call:

| Limit                  | Default | Meaning                                                                                               |
| ---------------------- | ------- | ----------------------------------------------------------------------------------------------------- |
| `modelRequestMs`       | 600 s   | The longest one provider request may take.                                                            |
| `modelIdleMs`          | 300 s   | How long a provider's stream may go without an event before the attempt is abandoned and tried again. |
| `modelAttempts`        | 3       | Attempts per model call, the first included.                                                          |
| `modelRetryDelayMs`    | 2 s     | The delay before the second attempt. It doubles each time, with jitter.                               |
| `modelProviderRetries` | 2       | Retries the provider's SDK makes within one attempt.                                                  |
| `modelRetryDelayMaxMs` | 60 s    | The longest delay the provider's SDK waits before one of those retries.                               |
| `modelCallBytes`       | 32 MiB  | The largest model call body the Model Gate reads.                                                     |
| `modelFileBytes`       | 8 MiB   | The largest file the Model Gate sends a model.                                                        |

A slow local model may need a longer `modelRequestMs`:

```sh
echo '{ "limits": { "modelRequestMs": 1200000 } }' > limits.json
npx nylorun tenant settings limits put limits.json
```

A `put` replaces every limit, so keep each limit the Tenant changes in that
file.

The hop to the Model Gate has its own timeout, set in the deployment's
[Host tuning](/docs/deploy/vm#host-tuning): a `modelRequestMs` longer than it
is cut there.

## Usage and budgets [#usage-and-budgets]

```ts
await admin.models.budgets.put({
  budgets: [
    { scope: "tenant", period: "month", limitUsd: 500 },
    { scope: "manifest", id: "triage", period: "day", limitUsd: 5 },
    { scope: "turn", limitTokens: 200_000 },
  ],
});

const used = await admin.models.usage({ scope: "manifest", id: "triage", period: "day" });
```

A budget is a hard cap on model spend. Before each call, the Model Gate checks
the scope's spend against it; once it is reached, the call fails and the turn
fails with `model.budget_exhausted`.

| Scope      | Caps                                                                 | Rules                                   |
| ---------- | -------------------------------------------------------------------- | --------------------------------------- |
| `tenant`   | Every session of the Tenant.                                         | Needs `period`, `day` or `month` (UTC). |
| `manifest` | The sessions of every Agent made from the agent manifest `id` names. | Needs `id` and `period`.                |
| `turn`     | Each turn on its own.                                                | Takes no `id` and no `period`.          |

Each budget sets `limitUsd`, `limitTokens`, or both. A custom endpoint's calls
cost $0, so cap them with `limitTokens`. `put` replaces every budget, and an
empty list removes them. Set one budget per scope and `id`, and at most the
Runtime limit `modelBudgets` (100 by default).

Usage takes the same scopes, with `period` `day`, `month`, or `total`. Each
recorded call that `admin.models.exportCalls()` and `nylorun tenant calls`
return carries the `manifestName` of the agent manifest it ran for.

### A session's usage by token kind [#a-sessions-usage-by-token-kind]

A session's usage splits its cost by the kind of token it paid for, at the
prices in force when each call was made:

```ts
const session = client.session("support-42");
const usage = await session.usage();
console.log(usage.costUsd, usage.costSplit); // { input, cacheRead, cacheWrite, output } in USD
console.log(usage.modelTimeMs); // the calls' durations, added up

const { calls } = await session.modelCalls();
for (const call of calls) console.log(call.startedAt, call.endedAt, call.usage?.costSplit);
```

| Field         | Meaning                                                                                                                                        |
| ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| `costSplit`   | Cost in USD of uncached input (`input`), input read from the provider's cache (`cacheRead`), input written to it (`cacheWrite`), and `output`. |
| `splitCalls`  | How many calls `costSplit` counts.                                                                                                             |
| `modelTimeMs` | The calls' durations, added up.                                                                                                                |
| `timedCalls`  | How many calls `modelTimeMs` counts.                                                                                                           |

Each model call carries `startedAt`, `endedAt`, and `usage.costSplit`, each
`null` for a call an older Runtime recorded; `splitCalls` and `timedCalls`
below `calls` say some calls lack them. The fields need protocol 18, which the
SDKs send. Studio's session **Usage** tab shows this ledger. See
[Read a session's recorded model usage](/reference/runtime/sessions/read-a-sessions-recorded-model-usage).

The Tenant page's **Usage** tab in [Studio](/docs/run/studio#the-tenant) shows
what model calls used today, this month, and in all, and edits the budgets.

| Route                            | Role                                            |
| -------------------------------- | ----------------------------------------------- |
| `GET /v1/tenant/usage`           | Recorded model usage, for one scope and period. |
| `GET` / `PUT /v1/tenant/budgets` | The hard caps.                                  |

These routes are the Management API: they take a management key. Use
`admin.models` from the [Admin SDK](/reference/management/admin-sdk), or
`nylorun tenant usage --scope manifest --id triage --period day` and
`nylorun tenant budgets get|put` from a terminal. Each endpoint's request and
response are on the [Management API](/reference/management/models/get-the-tenants-model)
tab, including [Set the Tenant's model budgets](/reference/management/models/usage/set-the-tenants-model-budgets)
and [Set what a catalog model reads](/reference/management/models/set-what-a-catalog-model-reads).

<Callout title="Provider keys stay in the Runtime">
  Do not put provider keys in agent manifests, session records, the project
  link, or source control. They never reach your app, the agent loop, or an
  Agent's sandbox.
</Callout>

## Next step [#next-step]

<Cards>
  <Card title="Sessions" description="Open a session that uses this model." href="/docs/run/sessions" />

  <Card title="Secrets and credentials" description="Where the model key and an Agent's secrets live." href="/docs/run/credentials" />
</Cards>
