Model routing

A model router is a model adapter made of other adapters. It tries them in priority order and moves to the next one when a provider is down, slow, rate-limited or returns an unusable answer. Use it when an agent must keep answering through a provider outage, or to fall back from one region or deployment to another.

Agents, tools and the runtime do not change: give the router to defineAgent like any other model, and grant it once.

import { createModelRouter, createRuntime, defineAgent } from 'mayura';
import { anthropicMessages } from 'mayura/provider-anthropic';
import { openAIResponses } from 'mayura/provider-openai';

const model = createModelRouter({
  id: 'router.support',
  routes: [
    anthropicMessages({
      apiKey: process.env.ANTHROPIC_API_KEY!, model: process.env.ANTHROPIC_MODEL!,
      maxCostMicros: 20_000, pricing: claudePrices,
    }),
    openAIResponses({
      apiKey: process.env.OPENAI_API_KEY!, model: process.env.OPENAI_MODEL!,
      maxCostMicros: 20_000, pricing: openAIPrices,
    }),
  ],
  circuit: { failureThreshold: 3, cooldownMs: 30_000 },
  onAttempt: attempt => console.log(attempt.modelId, attempt.outcome, attempt.failure ?? attempt.reason ?? '', attempt.costMicros),
});

const agent = defineAgent({ id: 'support', version: '1', instructions, model, tools, input, output });
const runtime = createRuntime({
  profile: 'ephemeral',
  permissions: { allow: ['model:router.support', 'tool:orders.lookup'] },
  limits: { maxCostMicros: 200_000 },
});

#Options

#When it fails over

The router moves to the next route after a timeout, a transport or provider failure, a rate limit, a provider refusing the route's key, or a response it cannot use. It does not fail over when:

• the caller cancels the run: the call ends as cancelled;
• an adapter throws a MayuraError with code INVALID_CONFIG, PERMISSION_DENIED or INVALID_INPUT, such as a schema the provider refuses. These are errors every route would repeat, so the call fails.

A rejected key (HTTP 401 or 403) fails over, because the next route has its own key: the call keeps working, and the attempt reports failure: 'authentication'. Watch onAttempt for it, since a route that keeps failing that way needs a new key. When every route fails, the call fails with the last route's reason, for example "The model provider's rate limit or quota was reached (HTTP 429)".

The router does not retry the same route, and adapters never retry. For more attempts, add routes: a second region or deployment of the same model is a route too.

#Circuit breaker

A route that fails failureThreshold calls in a row is skipped for cooldownMs. After the cooldown it gets one trial call: success closes the circuit, failure opens it again. Skipped routes are reported to onAttempt with outcome: 'skipped' and reason: 'circuit_open'.

router.status() returns the state of each route:

for (const route of model.status()) {
  console.log(route.route, route.modelId, route.state, route.consecutiveFailures, route.openUntilMs);
}

state is closed, open or half_open. Circuit state lives in the process that created the router. Separate processes, or separate router instances, keep separate state.

#What each attempt reports

onAttempt receives one object per attempt, with no prompt, output, key or provider error text:

#Accounting

The router's per-call bound (router.maxCostMicros) is the sum of the largest maxAttempts route bounds, because one call may try that many routes. The runtime reserves that amount before each call, so the run's maxCostMicros must be at least that large. With two routes of 20,000 micros each, every model call reserves 40,000.

After the call, the router reports what was actually spent:

• a successful attempt is charged its reported cost;
• a failed attempt with a known cost (the provider reported usage) is charged that cost;
• a failed attempt with an unknown cost, such as a timeout after the request was sent, is charged its full bound.

The run's budget therefore sees the worst case of every attempt. If every route fails and all costs are known, the known total is charged; if any is unknown, the call keeps its full reservation.

#Streaming and continuation

The router streams when the agent has a stream policy (see Streaming). A route without streaming answers through its ordinary call. The router can fail over only until the first piece of text has been released to the reader. After that, a failure ends the call instead of splicing a second provider's answer onto the first.

Some providers keep state between the model calls of one run (for example OpenAI's reasoning items). A run stays on the route that holds that state. If that route fails, the call moves to another route with only Mayura's own message history; one provider's state is never sent to another.

#Choosing routes

• Granting the router grants every route. Put only destinations in a router that you would grant one by one.
• Give every route the same contract: models that support strict tool calls and structured output. Each route receives the agent's output schema with every call, and defineAgent checks the agent against every route. Different models can still answer differently; failover keeps the agent available, it does not make answers identical.
• A fallback may cost more. Its bound counts toward every reservation even when it is never used.

#Related

• Model providers
• Streaming
• Costs and budgets
• Observability