Models
Choose a provider and model for the Tenant.
The Tenant holds the model credential. Your app and the session's Agent never see it.
npx nylorun tenant models setThe command lists the Tenant's model catalog, asks for a provider, a model, and
its sign-in (an API key, or the provider's OAuth), and sets the Tenant's model.
It does not make a test request; send a message in Studio to verify access.
The Models tab of Studio's Tenant page uses
the same routes. The first run installs the
provider sign-in library under ~/.nylorun/lib/.
For CI, pass the provider and model, and pipe the API key in on standard input. The key is never an argument, and a terminal on standard input is refused (exit 2). OAuth providers need the prompts.
printenv OPENAI_API_KEY | npx nylorun tenant models set --provider openai --model gpt-5| Command | What it does |
|---|---|
tenant models set | Prompts for a provider, model, and sign-in. |
tenant models set --provider <id> --model <id> [--base-url <url>] | The same with the API key on standard input. --base-url is for a custom OpenAI-compatible endpoint, provider custom. |
tenant models select <provider> [--model <id>] | Switches to a provider that already has a credential, keeping its model (or --model) and endpoint. |
tenant models get [--json] | The Tenant's model. |
tenant models providers [--json] | The providers with a credential, and the one in use. |
tenant models catalog [--json] | Every provider and model the Runtime knows. |
A session's model
GET /v1/sessions/{id} answers model: the model the session's turns call, and
why.
{
"model": {
"provider": "google-vertex",
"model": "gemini-4-argon",
"source": "tenant",
"reason": "provider_not_connected",
"requested": { "provider": "anthropic", "model": "claude-opus-5-5", "by": "manifest" }
}
}| Field | Meaning |
|---|---|
source | manifest: the model its Agent's version names (model in AGENT.md). session: its own choice. tenant: the Tenant's model. |
reason | On a fallback to the Tenant's model, why the model asked for isn't used: provider_not_connected, not_in_catalog, or invalid_model. |
requested | The model asked for, and by whom (manifest or session). |
It is null for a flow, and while the Tenant has no model. Each change is a
session.model event, with the previous
record. SessionView.model types it in @nylorun/agents.
First-start seed
On the first nylorun start, an empty Tenant can take these from .env:
MODEL_PROVIDER=openai
MODEL=gpt-4.1
MODEL_PROVIDER_API_KEY=sk-...
MODEL_PROVIDER_BASE_URL=https://your-provider.example/v1MODEL_PROVIDER_BASE_URL is required only for a custom OpenAI-compatible
endpoint. After the first write, the Tenant's model vault is authoritative.
Custom and local models
The Runtime lists providers and models at GET /v1/tenant/providers and
GET /v1/tenant/models. Point a custom OpenAI-compatible endpoint at its base
URL. A local server such as Ollama is reached at host.docker.internal.
printenv MODEL_KEY | npx nylorun tenant models set --provider custom \
--model llama3.2 --base-url http://host.docker.internal:11434/v1The Runtime cannot learn some facts from a custom endpoint, so you give them
in settings when you set the model (admin.models.put, PUT /v1/tenant/model):
| Setting | Default | Meaning |
|---|---|---|
contextWindow | 32,768 | The model's context window, in tokens. |
maxTokens | 8,192 | The most tokens one answer may hold. |
reasoning | Unset | Whether the model reasons. |
inputs | Text only | What else it reads: image, pdf, audio, or video. See Inputs. |
compat | Unset | Compatibility settings for the endpoint's OpenAI dialect, such as thinkingFormat or chatTemplateKwargs. |
headers | Unset | Headers sent on every call, such as a gateway's routing header. |
callHeaders | Unset | Headers whose value comes from the calling Agent, per call. See Headers per call. |
The defaults are small on purpose: an early compaction is better than a failed call. Set them from the model's documentation.
Headers per call
A custom endpoint behind a model gateway, such as LiteLLM, can carry headers that are not credentials. A Runtime that serves several organizations' Agents can name each call's organization, so the gateway attributes and budgets it while the call is made:
await admin.models.put({
idempotencyKey: crypto.randomUUID(),
provider: "custom",
model: "claude",
baseUrl: "https://litellm.internal/v1",
auth: { type: "api_key", key: process.env.GATEWAY_KEY! },
settings: {
callHeaders: { "x-nylorun-org": { from: "agent.label", key: "org" } },
},
});A callHeaders value takes its source from the Agent that makes the call:
| Source | Sends |
|---|---|
{ from: "agent.label", key } | The value of the Agent's label key. |
{ from: "agent.id" } | The Agent's id. |
{ from: "session.owner" } | The session's owner, when it is a person. |
A source with no value sends no header. Both maps are settings, read back by
GET /v1/tenant/model, never secrets: neither may set authorization (the
auth sends it), a hop-by-hop or transport header, or Nylorun-*, and a name
appears once across both. Studio keeps them when it saves the model. A Runtime
that serves one organization needs none of this: the gateway's key already
names it.
The workload token
A custom endpoint may take auth: { type: "workload" } in place of a key: the
Runtime sends its workload token, a short-lived token your platform rotates
through a file, as the endpoint's bearer, read at each call. Only provider
custom takes it. The operator gives the Runtime the token; see
The workload token. nylorun start does
not pass the workload token's variables yet, so a local Tenant cannot use it.
Inputs
Every model reads text. A model also reads image, pdf, office (Word,
Excel, PowerPoint), audio, or video when its catalog says so, or when you
add the input and its API carries it.
| Field | Where | Meaning |
|---|---|---|
inputs | Each model in GET /v1/tenant/models | What it reads: its catalog's inputs and the ones you added. |
addableInputs | Each model in GET /v1/tenant/models | What its API carries that its catalog does not list. |
inputs | Each provider in GET /v1/tenant/providers | What the provider's current model reads. |
| Input | APIs that can carry it |
|---|---|
image | Every API |
pdf | Anthropic, OpenAI (Responses and OpenAI-compatible completions), Azure, and Google |
office | OpenAI Responses |
audio, video | Google, and OpenAI-compatible completions |
Add inputs to a catalog model with the Management API:
import { createAdmin } from "@nylorun/admin";
const admin = createAdmin();
await admin.models.setInputs({
provider: "openai",
model: "gpt-4.1",
inputs: ["pdf"],
});setInputs (PUT /v1/tenant/models/inputs) takes only inputs in the model's
addableInputs and replaces the ones you added before; its catalog's inputs
always stay, and an empty list removes the additions. The Tenant keeps them in
its setting model.inputs.
A custom endpoint reads text only by default. Set what it reads in
settings.inputs when you set the model: image, pdf, audio, or video.
office is refused there.
await admin.models.put({
idempotencyKey: crypto.randomUUID(),
provider: "custom",
model: "llava",
baseUrl: "http://host.docker.internal:11434/v1",
settings: { inputs: ["image"], contextWindow: 131_072 },
auth: { type: "api_key", key: process.env.MODEL_KEY! },
});On the Tenant page's Models tab in Studio, each connected provider's model has an Inputs
line: what its catalog reads is checked and fixed, and what you may add toggles
through setInputs. The custom endpoint's form has Images, PDF,
Audio, and Video checkboxes, saved in settings.inputs.
Any file may be sent in a message, whatever the model reads. The harness
learns the model's inputs when each turn starts (turn.start.model) and
decides how each file reaches the model: Nylorun's harness sends a file the
model reads as itself, inlines a text file, and writes a note for anything
else. A custom harness that sends a kind the model
cannot read fails the call with the model failure unsupported_input, and a
file past the Runtime limit modelFileBytes fails it with invalid_request.
See Files.
Timeouts and retries
The Model Gate makes each provider call within the Tenant's Runtime limits, read for each call:
| Limit | Default | Meaning |
|---|---|---|
modelRequestMs | 600 s | The longest one provider request may take. |
modelIdleMs | 300 s | How long a provider's stream may go without an event before the attempt is abandoned and tried again. |
modelAttempts | 3 | Attempts per model call, the first included. |
modelRetryDelayMs | 2 s | The delay before the second attempt. It doubles each time, with jitter. |
modelProviderRetries | 2 | Retries the provider's SDK makes within one attempt. |
modelRetryDelayMaxMs | 60 s | The longest delay the provider's SDK waits before one of those retries. |
modelCallBytes | 32 MiB | The largest model call body the Model Gate reads. |
modelFileBytes | 8 MiB | The largest file the Model Gate sends a model. |
A slow local model may need a longer modelRequestMs:
echo '{ "limits": { "modelRequestMs": 1200000 } }' > limits.json
npx nylorun tenant settings limits put limits.jsonA put replaces every limit, so keep each limit the Tenant changes in that
file.
The hop to the Model Gate has its own timeout, set in the deployment's
Host tuning: a modelRequestMs longer than it
is cut there.
Usage and budgets
await admin.models.budgets.put({
budgets: [
{ scope: "tenant", period: "month", limitUsd: 500 },
{ scope: "manifest", id: "triage", period: "day", limitUsd: 5 },
{ scope: "turn", limitTokens: 200_000 },
],
});
const used = await admin.models.usage({ scope: "manifest", id: "triage", period: "day" });A budget is a hard cap on model spend. Before each call, the Model Gate checks
the scope's spend against it; once it is reached, the call fails and the turn
fails with model.budget_exhausted.
| Scope | Caps | Rules |
|---|---|---|
tenant | Every session of the Tenant. | Needs period, day or month (UTC). |
manifest | The sessions of every Agent made from the agent manifest id names. | Needs id and period. |
turn | Each turn on its own. | Takes no id and no period. |
Each budget sets limitUsd, limitTokens, or both. A custom endpoint's calls
cost $0, so cap them with limitTokens. put replaces every budget, and an
empty list removes them. Set one budget per scope and id, and at most the
Runtime limit modelBudgets (100 by default).
Usage takes the same scopes, with period day, month, or total. Each
recorded call that admin.models.exportCalls() and nylorun tenant calls
return carries the manifestName of the agent manifest it ran for.
A session's usage by token kind
A session's usage splits its cost by the kind of token it paid for, at the prices in force when each call was made:
const session = client.session("support-42");
const usage = await session.usage();
console.log(usage.costUsd, usage.costSplit); // { input, cacheRead, cacheWrite, output } in USD
console.log(usage.modelTimeMs); // the calls' durations, added up
const { calls } = await session.modelCalls();
for (const call of calls) console.log(call.startedAt, call.endedAt, call.usage?.costSplit);| Field | Meaning |
|---|---|
costSplit | Cost in USD of uncached input (input), input read from the provider's cache (cacheRead), input written to it (cacheWrite), and output. |
splitCalls | How many calls costSplit counts. |
modelTimeMs | The calls' durations, added up. |
timedCalls | How many calls modelTimeMs counts. |
Each model call carries startedAt, endedAt, and usage.costSplit, each
null for a call an older Runtime recorded; splitCalls and timedCalls
below calls say some calls lack them. The fields need protocol 18, which the
SDKs send. Studio's session Usage tab shows this ledger. See
Read a session's recorded model usage.
The Tenant page's Usage tab in Studio shows what model calls used today, this month, and in all, and edits the budgets.
| Route | Role |
|---|---|
GET /v1/tenant/usage | Recorded model usage, for one scope and period. |
GET / PUT /v1/tenant/budgets | The hard caps. |
These routes are the Management API: they take a management key. Use
admin.models from the Admin SDK, or
nylorun tenant usage --scope manifest --id triage --period day and
nylorun tenant budgets get|put from a terminal. Each endpoint's request and
response are on the Management API
tab, including Set the Tenant's model budgets
and Set what a catalog model reads.
Provider keys stay in the Runtime
Do not put provider keys in agent manifests, session records, the project link, or source control. They never reach your app, the agent loop, or an Agent's sandbox.