NylorunDocsBeta
RunConfigure

Models

Choose a provider and model for the Tenant.

The Tenant holds the model credential. Your app and the session's Agent never see it.

npx nylorun tenant models set

The command lists the Tenant's model catalog, asks for a provider, a model, and its sign-in (an API key, or the provider's OAuth), and sets the Tenant's model. It does not make a test request; send a message in Studio to verify access. The Models tab of Studio's Tenant page uses the same routes. The first run installs the provider sign-in library under ~/.nylorun/lib/.

For CI, pass the provider and model, and pipe the API key in on standard input. The key is never an argument, and a terminal on standard input is refused (exit 2). OAuth providers need the prompts.

printenv OPENAI_API_KEY | npx nylorun tenant models set --provider openai --model gpt-5
CommandWhat it does
tenant models setPrompts for a provider, model, and sign-in.
tenant models set --provider <id> --model <id> [--base-url <url>]The same with the API key on standard input. --base-url is for a custom OpenAI-compatible endpoint, provider custom.
tenant models select <provider> [--model <id>]Switches to a provider that already has a credential, keeping its model (or --model) and endpoint.
tenant models get [--json]The Tenant's model.
tenant models providers [--json]The providers with a credential, and the one in use.
tenant models catalog [--json]Every provider and model the Runtime knows.

A session's model

GET /v1/sessions/{id} answers model: the model the session's turns call, and why.

{
  "model": {
    "provider": "google-vertex",
    "model": "gemini-4-argon",
    "source": "tenant",
    "reason": "provider_not_connected",
    "requested": { "provider": "anthropic", "model": "claude-opus-5-5", "by": "manifest" }
  }
}
FieldMeaning
sourcemanifest: the model its Agent's version names (model in AGENT.md). session: its own choice. tenant: the Tenant's model.
reasonOn a fallback to the Tenant's model, why the model asked for isn't used: provider_not_connected, not_in_catalog, or invalid_model.
requestedThe model asked for, and by whom (manifest or session).

It is null for a flow, and while the Tenant has no model. Each change is a session.model event, with the previous record. SessionView.model types it in @nylorun/agents.

First-start seed

On the first nylorun start, an empty Tenant can take these from .env:

MODEL_PROVIDER=openai
MODEL=gpt-4.1
MODEL_PROVIDER_API_KEY=sk-...
MODEL_PROVIDER_BASE_URL=https://your-provider.example/v1

MODEL_PROVIDER_BASE_URL is required only for a custom OpenAI-compatible endpoint. After the first write, the Tenant's model vault is authoritative.

Custom and local models

The Runtime lists providers and models at GET /v1/tenant/providers and GET /v1/tenant/models. Point a custom OpenAI-compatible endpoint at its base URL. A local server such as Ollama is reached at host.docker.internal.

printenv MODEL_KEY | npx nylorun tenant models set --provider custom \
  --model llama3.2 --base-url http://host.docker.internal:11434/v1

The Runtime cannot learn some facts from a custom endpoint, so you give them in settings when you set the model (admin.models.put, PUT /v1/tenant/model):

SettingDefaultMeaning
contextWindow32,768The model's context window, in tokens.
maxTokens8,192The most tokens one answer may hold.
reasoningUnsetWhether the model reasons.
inputsText onlyWhat else it reads: image, pdf, audio, or video. See Inputs.
compatUnsetCompatibility settings for the endpoint's OpenAI dialect, such as thinkingFormat or chatTemplateKwargs.
headersUnsetHeaders sent on every call, such as a gateway's routing header.
callHeadersUnsetHeaders whose value comes from the calling Agent, per call. See Headers per call.

The defaults are small on purpose: an early compaction is better than a failed call. Set them from the model's documentation.

Headers per call

A custom endpoint behind a model gateway, such as LiteLLM, can carry headers that are not credentials. A Runtime that serves several organizations' Agents can name each call's organization, so the gateway attributes and budgets it while the call is made:

await admin.models.put({
  idempotencyKey: crypto.randomUUID(),
  provider: "custom",
  model: "claude",
  baseUrl: "https://litellm.internal/v1",
  auth: { type: "api_key", key: process.env.GATEWAY_KEY! },
  settings: {
    callHeaders: { "x-nylorun-org": { from: "agent.label", key: "org" } },
  },
});

A callHeaders value takes its source from the Agent that makes the call:

SourceSends
{ from: "agent.label", key }The value of the Agent's label key.
{ from: "agent.id" }The Agent's id.
{ from: "session.owner" }The session's owner, when it is a person.

A source with no value sends no header. Both maps are settings, read back by GET /v1/tenant/model, never secrets: neither may set authorization (the auth sends it), a hop-by-hop or transport header, or Nylorun-*, and a name appears once across both. Studio keeps them when it saves the model. A Runtime that serves one organization needs none of this: the gateway's key already names it.

The workload token

A custom endpoint may take auth: { type: "workload" } in place of a key: the Runtime sends its workload token, a short-lived token your platform rotates through a file, as the endpoint's bearer, read at each call. Only provider custom takes it. The operator gives the Runtime the token; see The workload token. nylorun start does not pass the workload token's variables yet, so a local Tenant cannot use it.

Inputs

Every model reads text. A model also reads image, pdf, office (Word, Excel, PowerPoint), audio, or video when its catalog says so, or when you add the input and its API carries it.

FieldWhereMeaning
inputsEach model in GET /v1/tenant/modelsWhat it reads: its catalog's inputs and the ones you added.
addableInputsEach model in GET /v1/tenant/modelsWhat its API carries that its catalog does not list.
inputsEach provider in GET /v1/tenant/providersWhat the provider's current model reads.
InputAPIs that can carry it
imageEvery API
pdfAnthropic, OpenAI (Responses and OpenAI-compatible completions), Azure, and Google
officeOpenAI Responses
audio, videoGoogle, and OpenAI-compatible completions

Add inputs to a catalog model with the Management API:

import { createAdmin } from "@nylorun/admin";

const admin = createAdmin();

await admin.models.setInputs({
  provider: "openai",
  model: "gpt-4.1",
  inputs: ["pdf"],
});

setInputs (PUT /v1/tenant/models/inputs) takes only inputs in the model's addableInputs and replaces the ones you added before; its catalog's inputs always stay, and an empty list removes the additions. The Tenant keeps them in its setting model.inputs.

A custom endpoint reads text only by default. Set what it reads in settings.inputs when you set the model: image, pdf, audio, or video. office is refused there.

await admin.models.put({
  idempotencyKey: crypto.randomUUID(),
  provider: "custom",
  model: "llava",
  baseUrl: "http://host.docker.internal:11434/v1",
  settings: { inputs: ["image"], contextWindow: 131_072 },
  auth: { type: "api_key", key: process.env.MODEL_KEY! },
});

On the Tenant page's Models tab in Studio, each connected provider's model has an Inputs line: what its catalog reads is checked and fixed, and what you may add toggles through setInputs. The custom endpoint's form has Images, PDF, Audio, and Video checkboxes, saved in settings.inputs.

Any file may be sent in a message, whatever the model reads. The harness learns the model's inputs when each turn starts (turn.start.model) and decides how each file reaches the model: Nylorun's harness sends a file the model reads as itself, inlines a text file, and writes a note for anything else. A custom harness that sends a kind the model cannot read fails the call with the model failure unsupported_input, and a file past the Runtime limit modelFileBytes fails it with invalid_request. See Files.

Timeouts and retries

The Model Gate makes each provider call within the Tenant's Runtime limits, read for each call:

LimitDefaultMeaning
modelRequestMs600 sThe longest one provider request may take.
modelIdleMs300 sHow long a provider's stream may go without an event before the attempt is abandoned and tried again.
modelAttempts3Attempts per model call, the first included.
modelRetryDelayMs2 sThe delay before the second attempt. It doubles each time, with jitter.
modelProviderRetries2Retries the provider's SDK makes within one attempt.
modelRetryDelayMaxMs60 sThe longest delay the provider's SDK waits before one of those retries.
modelCallBytes32 MiBThe largest model call body the Model Gate reads.
modelFileBytes8 MiBThe largest file the Model Gate sends a model.

A slow local model may need a longer modelRequestMs:

echo '{ "limits": { "modelRequestMs": 1200000 } }' > limits.json
npx nylorun tenant settings limits put limits.json

A put replaces every limit, so keep each limit the Tenant changes in that file.

The hop to the Model Gate has its own timeout, set in the deployment's Host tuning: a modelRequestMs longer than it is cut there.

Usage and budgets

await admin.models.budgets.put({
  budgets: [
    { scope: "tenant", period: "month", limitUsd: 500 },
    { scope: "manifest", id: "triage", period: "day", limitUsd: 5 },
    { scope: "turn", limitTokens: 200_000 },
  ],
});

const used = await admin.models.usage({ scope: "manifest", id: "triage", period: "day" });

A budget is a hard cap on model spend. Before each call, the Model Gate checks the scope's spend against it; once it is reached, the call fails and the turn fails with model.budget_exhausted.

ScopeCapsRules
tenantEvery session of the Tenant.Needs period, day or month (UTC).
manifestThe sessions of every Agent made from the agent manifest id names.Needs id and period.
turnEach turn on its own.Takes no id and no period.

Each budget sets limitUsd, limitTokens, or both. A custom endpoint's calls cost $0, so cap them with limitTokens. put replaces every budget, and an empty list removes them. Set one budget per scope and id, and at most the Runtime limit modelBudgets (100 by default).

Usage takes the same scopes, with period day, month, or total. Each recorded call that admin.models.exportCalls() and nylorun tenant calls return carries the manifestName of the agent manifest it ran for.

A session's usage by token kind

A session's usage splits its cost by the kind of token it paid for, at the prices in force when each call was made:

const session = client.session("support-42");
const usage = await session.usage();
console.log(usage.costUsd, usage.costSplit); // { input, cacheRead, cacheWrite, output } in USD
console.log(usage.modelTimeMs); // the calls' durations, added up

const { calls } = await session.modelCalls();
for (const call of calls) console.log(call.startedAt, call.endedAt, call.usage?.costSplit);
FieldMeaning
costSplitCost in USD of uncached input (input), input read from the provider's cache (cacheRead), input written to it (cacheWrite), and output.
splitCallsHow many calls costSplit counts.
modelTimeMsThe calls' durations, added up.
timedCallsHow many calls modelTimeMs counts.

Each model call carries startedAt, endedAt, and usage.costSplit, each null for a call an older Runtime recorded; splitCalls and timedCalls below calls say some calls lack them. The fields need protocol 18, which the SDKs send. Studio's session Usage tab shows this ledger. See Read a session's recorded model usage.

The Tenant page's Usage tab in Studio shows what model calls used today, this month, and in all, and edits the budgets.

RouteRole
GET /v1/tenant/usageRecorded model usage, for one scope and period.
GET / PUT /v1/tenant/budgetsThe hard caps.

These routes are the Management API: they take a management key. Use admin.models from the Admin SDK, or nylorun tenant usage --scope manifest --id triage --period day and nylorun tenant budgets get|put from a terminal. Each endpoint's request and response are on the Management API tab, including Set the Tenant's model budgets and Set what a catalog model reads.

Provider keys stay in the Runtime

Do not put provider keys in agent manifests, session records, the project link, or source control. They never reach your app, the agent loop, or an Agent's sandbox.

Next step

On this page