One conversation shape. The model is the routing address.

Write a conversation once:

{
  "model": "openai/gpt-6-sol",
  "messages": [
    { "role": "user", "content": "Explain ownership in one sentence." }
  ]
}

To send the same conversation to another provider, change one line:

- "model": "openai/gpt-6-sol"
+ "model": "anthropic/claude-opus-5-5"

llmshim selects the provider, translates the request into that provider's native API, sends it, and translates the result back. Change the model, not the conversation. That is the central idea.

At its core, llmshim is a Rust library. The CLI and HTTP proxy wrap the same library. The Python, TypeScript, Go, and Ruby clients are thin clients for the proxy; Python and TypeScript can also start a bundled proxy automatically.

flowchart LR
    Rust[Rust application] --> Core[llmshim Rust engine]
    Terminal[Terminal user] --> CLI[llmshim chat]
    CLI --> Core
    HTTP[HTTP callers] --> Proxy[llmshim proxy]
    Clients[Language clients] --> Proxy
    Proxy --> Core
    Core --> Providers[OpenAI / Anthropic / Gemini / xAI]

A translation boundary

Think of llmshim like a compiler with provider-specific backends. The input is an OpenAI Chat Completions-style conversation. The model string selects a backend. That backend translates the request to the provider's native API and normalizes the provider's response.

flowchart LR
    Request[Model + messages + controls] --> Router[Resolve model address]
    Router --> Outbound[Translate request]
    Outbound --> API[Provider-native API]
    API --> Inbound[Translate response]
    Inbound --> Result[Normalized result]

Three rules define the boundary:

1. Common features are translated where the target supports them. Messages, tools, images, streaming, and reasoning controls all pass through a provider-specific adapter.
2. Different APIs do not become identical. A provider may support fewer reasoning levels, require another image representation, or omit a feature. llmshim maps, clamps, or omits fields according to the target.
3. Native controls remain available. Use x-openai, x-anthropic, or x-gemini when a portable control is not enough. There is no x-xai namespace.

llmshim is not an agent framework. It does not execute tools or own conversation memory. Your application remains responsible for both.

Next, choose the surface that fits your application, or see how translation flows in more detail.

Choose a surface

Every llmshim surface reaches the same Rust translation engine. The choice is where that engine runs and who manages its process.

Availability: Rust: in-process · CLI: interactive · HTTP: separate proxy · Clients: through the proxy

A quick decision

• Building a Rust application? Use the crate. It is the engine and has no local server hop.
• Exploring models from a terminal? Use llmshim chat. The CLI keeps the current conversation in memory and streams the answer.
• Need HTTP or a shared deployment? Run the proxy. It exposes llmshim's own compact API, not an OpenAI-compatible endpoint.
• Using Python or TypeScript? Start with the language client. Its bundled binary starts a local proxy on the first call. TypeScript can also connect to a proxy URL you provide; the current Python API always manages its own proxy.
• Using Go or Ruby? Start a proxy first, then connect with the pure HTTP client.

These are process choices, not different translation implementations. The CLI and proxy wrap the crate, and all four language clients speak the proxy contract. See Two contracts, one engine for the data shapes at each boundary.

Whichever path you choose, configure at least one provider before making a request.

Configure providers

llmshim needs credentials only for the provider that handles a request. Configure one provider to begin; add others when you want to switch models.

Availability: Rust: environment, optional config load · CLI/proxy: environment + config file · Clients: the proxy owns provider keys

Option 1: environment variables

Set one or more variables in the environment of the process that runs llmshim:

export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...
export GEMINI_API_KEY=AIza...
export XAI_API_KEY=xai-...

The Rust crate reads these variables when Router::from_env() is constructed. The CLI and proxy read them at startup.

Option 2: ~/.llmshim/config.toml

Use the interactive command to create or update the shared config file:

llmshim configure

It prompts for all four provider keys, the proxy host, and the proxy port. Press Enter to keep an existing value. The resulting file is used by CLI chat and the proxy.

You can also manage individual values:

llmshim set anthropic sk-ant-...
llmshim get anthropic
llmshim list

Valid key names are openai, anthropic, gemini, xai, proxy.host, and proxy.port. get and list mask stored API keys. Be aware that a value passed to set may remain in your shell history; the interactive command avoids that shell-history exposure.

On Unix, loading the default config also repairs its file mode to 0600 and the owned .llmshim directory mode to 0700. The default file must be a regular file owned by the current user, with no symlink or additional hard link; unsafe paths are ignored with a diagnostic. Already-private files need no permission change, so read-only loads still work on read-only filesystems. Use environment variables for operator-managed secrets outside this default storage layout.

Option 3: ChatGPT subscription login

llmshim login chatgpt

Open the printed URL, sign in, and enter the device code. Select chatgpt/gpt-6-astra, chatgpt/gpt-6-sol, chatgpt/gpt-6.1-sol, or chatgpt/gpt-6-luna. Only those four models are supported. ChatGPT tokens are stored separately from API keys and refreshed automatically. Device login may need enabling in ChatGPT security settings or workspace permissions.

llmshim login chatgpt --status checks the saved session. See ChatGPT OAuth configuration for custom cache paths and container setup.

Precedence: environment wins

The CLI and proxy call llmshim::env::load_all(). It reads ~/.llmshim/config.toml, but fills only variables that are not already set:

environment variable  >  config file  >  not configured

This makes a shell, container, or deployment secret override the local file without editing it.

Router::from_env() does not load the file on its own. A Rust application that wants the same behavior must call llmshim::env::load_all() first. See Environment variables versus config.toml.

Check what is available

llmshim models

The command lists registry entries only for providers whose keys or OAuth cache are available after applying the precedence above. Runtime discovery is the canonical way to see the current curated model list.

Go and Ruby clients never receive provider keys directly; they connect to a proxy that owns them. Python and TypeScript auto-started proxies inherit the current environment and use the same config-file loading behavior.

Amazon Bedrock

The bedrock Cargo feature is off by default. Bundled release binaries leave it off. Library users enable it with features = ["bedrock"]; build a custom binary with cargo build --features bedrock when embedding a registered Bedrock provider. There is no automatic CLI or environment registration.

Register an explicit profile binding; constructing it performs no authentication or network call:

#![allow(unused)]
fn main() {
use llmshim::{providers::bedrock::{AwsProfile, Bedrock}, router::Router};

let router = Router::new().register(
    "bedrock",
    Box::new(Bedrock::new(AwsProfile::new("development", "us-east-1"))),
);
}

Address a model as bedrock/<model-id>. The suffix is the exact Bedrock model or inference- profile ID, or ARN; slashes and colons are preserved by routing and encoded as one path parameter. The explicit region selects https://bedrock-runtime.{region}.{suffix}. The supported partitions are aws and aws-us-gov (amazonaws.com), and aws-cn (amazonaws.com.cn). FIPS, dual-stack, isolated partitions and the European Sovereign Cloud partition are not supported. Region strings accept only lowercase letters, digits and hyphens; unsupported partitions are refused locally. There is no environment-driven registration or change to existing API-key precedence.

AwsProfile stores a name and region, not credentials. Every HTTP attempt, including a retry, checks a per-provider, shared SDK lazy identity cache. It resolves only the named SDK profile provider on a cache miss and refreshes temporary credentials before expiry. Clones share the cache; calling with_profile_files or with_credential_provider creates a new cache. The profile may explicitly select an assumed role, SSO or a credential process. A missing/invalid profile fails; it never falls through to exported keys, a default profile, container credentials or instance metadata. A profile that explicitly declares credential_source still selects that source as documented by the AWS SDK. with_profile_files selects both files without default files; an application can instead supply a refreshable SDK ProvideCredentials handle with with_credential_provider(provider, "stable-identity-reference"). This non-secret binding must stay stable across restarts/replicas and change when the identity changes. For a caller-supplied provider it replaces unused profile/file identity settings; its namespace is distinct from named profiles. The endpoint remains part of the replay boundary. llmshim does not export tokens or write them to application configuration. The AWS SDK's SSO provider may refresh and write its own AWS SSO cache. Resolution is bounded to 30 seconds and by the dispatch deadline. Resolution happens before policy admission, so credential failures reserve neither budget nor rate capacity. Signing inputs and header encodings are validated before admission; the final SigV4 pass follows admission with a fresh timestamp and uses the exact serialized bytes sent. Validation adds one extra SigV4 pass per attempt without credential network traffic on a cache hit. A residual local signer failure reports explicit zero usage; it never fabricates a provider bill, and existing unpriced-fee policy stays conservative. Response-header time starts after authorization; the total attempt deadline still applies. Redirects are refused. Converse traffic uses a reused client without environment-selected proxies by default. with_ambient_proxy(true) opts into the caller client's ambient proxy policy (a scoped client still has no ambient proxy). Credential traffic (STS, SSO and explicit profile credential sources) uses the SDK's separate ring-based HTTPS client and its own proxy rules; this option controls only Converse traffic. Applications requiring custom credential networking should supply a configured SDK credential provider.

Credential errors include the profile name, or the caller binding for custom providers, and a fixed reason with plain diagnostic wording, without SDK text or paths. Rustls uses ring for both reqwest and the SDK; the feature does not add a second crypto provider or aws-lc-sys. Timeouts and SDK provider failures are 503 and retry; missing profiles, invalid configuration and expired returned credentials stop. SDK errors with opaque causes use provider_failed: this SDK erases credential-process failure details, so these are not guessed from message text. ShimClient::send sends unsigned JSON. Use send_with_transport(&request, &provider) when calling the HTTP layer directly; completion and stream entry points select the transport.

Converse and ConverseStream support text, tool definitions, tool choice, tool results and signed reasoning replay. Anthropic model IDs map common reasoning controls through the shared Anthropic profile and model constraints, including clamped effort and budget tokens. Mapping requires reasoning_effort; reasoning_mode or reasoning_summary alone never enable thinking. Other families reject each unsupported reasoning control by name. Tool history without enabled definitions (tools missing or tool_choice: "none") is refused locally with a toolConfig error. Adjacent same-role turns are merged with their blocks in order, including parallel tool results and a following user message. Tool text-part arrays become text blocks; null/empty tool content and non-text tool parts are refused. Tools without parameters use an empty object schema. Empty assistant text is omitted. Content cache_control markers require native cachePoint support and are refused instead of silently dropped. System/developer instructions must precede conversation turns; appended system instructions are refused, including on bound-thinking models. Other content blocks and unsupported common controls return a local error. Native model controls may be supplied through x-bedrock.additionalModelRequestFields; the extension cannot replace messages, system instructions or tools. Stream framing uses the AWS EventStream codec, verifies both CRCs, bounds retained frame/block state, and rejects truncated or malformed streams. Tools are exposed only with complete JSON arguments. A modeled exception after HTTP 200 remains a failure, available as BedrockError::from_shim(&error) with a typed kind, machine code and status. Classification never reads message text. Signature-error payloads are replaced with fixed text because canonical-request diagnostics can echo credentials. Streaming and unary IDs retain the AWS request identifier; a headerless manual decoder generates its own stable stream ID.

Usage retains native Bedrock counters. inputTokens excludes cache reads and writes, so the normalized prompt total includes all three buckets, while uncached_input_tokens includes only inputTokens. Streaming and unary responses use the same convention; policy accounting also receives cache read/write counters. This transport adds no prices or regional availability entries to the catalog. Signed reasoning is bound to the configured transport and identity reference. Unknown model families cannot authorize reasoning replay. This includes ARN model identifiers: use a cataloged model or inference-profile ID for signed reasoning across tool turns.

Offline tests use recorded protocol examples, synthetic credentials, the official AWS signing vectors and in-memory HTTP responses. They make no cloud calls and need no AWS CLI or account. See tests/fixtures/bedrock/README.md for fixture sources.

Public transport structs and error enums are non-exhaustive. Construct bindings/providers with AwsProfile::new and Bedrock::new, service errors with BedrockError::service, and manual decoders with EventStreamDecoder::new. Match error enums with a wildcard for future variants.

Google Vertex

The vertex Cargo feature is off by default. Enable it explicitly for library use. Bundled release binaries leave it off; providers are registered by the application, with no environment registration or changes to existing API-key precedence.

#![allow(unused)]
fn main() {
use llmshim::{providers::vertex::{GoogleIdentity, Vertex}, router::Router};
let identity = GoogleIdentity::service_account(
    "example-project", "us-central1", "selected-account", "service-account.json".into(),
);
let router = Router::new()
    .register("vertex-gemini", Box::new(Vertex::gemini(identity.clone())))
    .register("vertex-anthropic", Box::new(Vertex::anthropic(identity)));
}

Use vertex-gemini/gemini-3.8-flash or a Vertex Anthropic ID such as vertex-anthropic/claude-sonnet-5-5@20261001. Model availability is project/location dependent; this transport adds no regional catalog inventory or prices. Cost uses only a priced catalog row for the exact vertex-gemini/<model> or vertex-anthropic/<model> target. Direct API prices and bare model-name fallbacks are never used for Vertex. Without a known Vertex price, cost is unknown (null) and strict budget admission treats the target as unpriceable. The configured project and location select the path. Regional endpoints use https://{location}-aiplatform.googleapis.com/v1; global uses https://aiplatform.googleapis.com/v1. Anthropic's us and eu multi-region endpoints use aiplatform.{location}.rep.googleapis.com. Project/location strings are validated and models are encoded as single path segments. Invalid values fail locally before credential resolution.

Gemini uses generateContent or streamGenerateContent?alt=sse under publisher google. Anthropic uses rawPredict or streamRawPredict under publisher anthropic, removes model from the Messages body and adds anthropic_version: "vertex-2023-10-16". The exact Vertex model version stays in the URL. Both reuse the direct provider's request, response, tool, reasoning and usage conversion, with a transport-specific identity/replay boundary. Native controls remain under x-gemini and x-anthropic. Images/tools and supported model constraints follow those converters. Reasoning and tool signatures cannot cross identity bindings.

Construction reads no file and performs no authentication. GoogleIdentity::service_account selects exactly one service-account JSON file; only type: "service_account" is supported. JWT assertions use RS256 and the cloud-platform scope at Google's fixed OAuth token endpoint. Files selecting another token URI are refused. The JSON may contain private key material, which stays in memory during refresh and is never exported to application configuration or diagnostics. No environment variable, ambient ADC search, metadata service, installed CLI or alternate file is consulted after failure. Authorized-user and workload-identity files are not implemented; supply an explicit caller-owned TokenSource for those identities. A stable non-secret binding is required for both file and custom token sources. Keep it stable across restarts/replicas; change it when the selected identity changes, even if the file path stays the same.

A clone-shared cache checks the selected source on every dispatch/retry, refreshes before expiry, and serializes concurrent refreshes. Cached tokens are read separately from the refresh lock. Tokens carry monotonic expiry and are never printed through Debug. Bearer headers are marked sensitive. Foreground credential resolution is bounded to 30 seconds and the existing dispatch deadline, before policy admission; credential failure consumes no budget or rate capacity. Background refresh has its own 30-second source bound. Transient token-service failures/timeouts retry as 503; invalid files or rejected credentials stop as 401. While a cached token has more than one second remaining, authorization returns it immediately. One caller starts a background refresh using a separate mutex; other callers keep using the cached token. A failed early refresh preserves that token and waits one second before another early attempt. Only callers without a usable token wait for source I/O; expiry or rejection bypasses the early-refresh cooldown. A timed-out refresh releases the mutex, so the next caller can refresh again. Expired or rejected tokens are never used as a fallback. An inference HTTP 401, or a 403 with a recognized token-rejection code, invalidates the token actually sent. Within the configured retry limit, the same dispatch refreshes the selected identity once and retries. A second rejection fails and invalidates that token too. With retries disabled, invalidation applies to the next dispatch. An ordinary IAM permission-denied 403 does not trigger refresh. Identity selection never changes.

Inference and token traffic use scoped HTTPS clients with redirects and ambient proxies disabled. with_ambient_proxy(true) opts inference traffic into the caller client's proxy policy; a scoped caller client remains scoped. A caller needing custom token networking can provide TokenExchange on ServiceAccount. The exchange must not log its JWT assertion. ShimClient::send is unsigned; use send_with_transport or the completion/stream entry points for call-time authorization.

Both stream paths use the existing bounded SSE parser. Google envelopes and Anthropic error frames remain typed failures even after HTTP 200. Classification uses status and code, never message text; upstream explanations are withheld because they can echo credentials or prompts. Prompt blocking remains content_filter, distinct from a failed transport. Gemini output usage includes thoughts once, exposes cache buckets and reasoning tokens, and retains native accounting through the existing usage path. Anthropic stream usage follows cumulative snapshot semantics.

All new public structs/enums are non-exhaustive and have constructors. Tests are offline and use synthetic tokens, fake exchanges and virtual time; see tests/fixtures/vertex/README.md.

Azure OpenAI and Foundry v1

The azure Cargo feature is off by default and adds no dependency crates. Bundled binaries leave it off. Register the provider explicitly; there is no environment registration or implicit credential chain.

#![allow(unused)]
fn main() {
use llmshim::{providers::azure::{Azure, AzureIdentity}, router::Router};
let identity = AzureIdentity::api_key("selected-resource", "your-key")?;
let provider = Azure::new("https://example.openai.azure.com", identity)?
    .with_deployment("chat-production", "gpt-6-sol")?;
let router = Router::new().register("azure", Box::new(provider));
Ok::<(), llmshim::error::ShimError>(())
}

Request azure/chat-production. The body model is the configured deployment name, rather than the catalog model. with_deployment explicitly records its underlying model for catalog family and replay metadata. Unknown deployments are refused locally. The endpoint accepts an HTTPS resource origin or its /openai/v1 base, including the Azure OpenAI https://{resource}.openai.azure.com and Foundry https://{resource}.services.ai.azure.com forms. Custom HTTPS resource domains can be selected explicitly. User information, query strings, fragments and other base paths are refused. The v1 operation is always POST /openai/v1/chat/completions; legacy /models and dated API versions are not used. See Microsoft's v1 reference.

AzureIdentity::api_key(binding, key) selects one explicit key. It is added as the sensitive api-key header at dispatch time; no credential appears in ProviderRequest.headers. AzureIdentity::entra(binding, audience, source) instead takes one caller-owned TokenSource. Its token(audience) method must resolve only that identity and return an AccessToken with its remaining lifetime. Integrate an explicitly configured Entra credential handle there; llmshim never creates a default credential chain, reads an environment key, probes metadata, launches a CLI or switches identity after failure.

Choose the audience explicitly:

The scopes follow Microsoft's Azure OpenAI and Foundry Entra documentation. They are passed to the selected source, never inferred from a deployment name. Keep the non-secret binding stable across restarts and replicas, and change it when the identity changes. Replay metadata hashes the binding with the endpoint and authentication kind/audience; it never contains the key or bearer. Identity and token structs intentionally do not implement Debug.

The clone-shared Entra cache uses the same private refresh mechanism as Vertex. A token with more than one second remaining is returned without waiting for source I/O. Inside the early margin (one quarter of its lifetime, capped at 30 seconds), one caller starts a background refresh using a separate mutex acquired without waiting. Other readers continue using the live token. Failed early refresh retains it and allows another early attempt after one second. Expired or rejected tokens cannot be reused; callers without a usable token wait for refresh and bypass that cooldown. Source calls have a 30-second bound. Foreground authorization also obeys the existing dispatch deadline and happens before budget/rate admission.

An HTTP 401, or a typed expired/invalid-token 403, invalidates the bearer actually sent. Within the configured retry limit the same dispatch refreshes the same selected source once. Repeated rejection stops; a delayed rejection cannot evict a newer different bearer. Ordinary permission 403 responses and API-key failures do not trigger Entra refresh. Invalid/rejected credentials fail as typed 401 errors; transient source failure or timeout is a typed 503.

The adapter reuses the Chat Completions conversion for messages, tools, tool results, vision, structured output, cache accounting and reasoning metadata. It preserves native token-detail objects, including cached and reasoning counters; completion counts already include reasoning. Streaming requests force stream_options.include_usage: true, including when native options are supplied. Empty-choice usage tails and prompt-filter annotation frames survive SSE framing. A choice stopped by filtering keeps finish_reason: content_filter; a rejected request or an error frame after HTTP 200 yields a typed AzureErrorKind::ContentFilter error. Classification uses status and allowlisted machine codes, never message text. Untrusted error explanations and nested details are withheld to avoid exposing credentials or prompts.

Native controls use x-azure. The fields model, messages, stream, and stream_options cannot be overridden there. Controls that have no mapping (reasoning_mode, reasoning_summary, top_k) are refused locally by name. Other Chat controls are forwarded to the selected deployment, whose actual support is enforced by Azure; this transport invents no deployment capability inventory.

Cost uses only a priced azure/<mapped model> catalog row from the explicit deployment map. Deployment labels never select prices, and an unmapped deployment is unpriceable. It never borrows a direct OpenAI rate. This transport adds no prices. Without a known Azure price, catalog cost is unknown (null), and strict budget admission treats it as unpriceable.

Inference uses a scoped client without ambient proxies or redirects. with_ambient_proxy(true) opts inference into the caller client's proxy policy; a scoped caller stays scoped. Token-source networking belongs to the caller's explicitly supplied source and is subject to its own proxy configuration. ShimClient::send is unsigned; use send_with_transport or the completion and stream entry points to resolve authorization on each attempt.

Tests use synthetic, public-example-derived fixtures and in-memory token/HTTP endpoints. No Azure credential, tenant, metadata service or real token endpoint is needed. The fixture README lists the public schema sources. All new public enums and structs are non-exhaustive and have constructors.

Rust quickstart

Use the crate when the application making LLM requests is already written in Rust. The translation engine runs in-process; there is no proxy to start.

Availability: Rust: direct engine API · CLI/HTTP/clients: not used in this path

1. Add the dependencies

cargo add llmshim tokio serde_json

tokio and serde_json are direct dependencies of your application. llmshim does not re-export them.

Set at least one provider key before running the program:

export ANTHROPIC_API_KEY=sk-ant-...

2. Send a completion

use serde_json::json;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let router = llmshim::router::Router::from_env();

    // The public request contract is a serde_json::Value.
    let request = json!({
        "model": "openai/gpt-6-sol",
        "messages": [
            {"role": "user", "content": "What is Rust in one sentence?"}
        ],
        "max_tokens": 128
    });

    let response = llmshim::completion(&router, &request).await?;
    let text = response["choices"][0]["message"]["content"]
        .as_str()
        .unwrap_or("");

    println!("{text}");
    Ok(())
}

Run it with cargo run. Router::from_env() registers the providers whose environment variables are present. Change only the model address to route the same conversation elsewhere.

The result uses an OpenAI Chat Completions-style shape, regardless of which provider answered. The assistant text is at response["choices"][0]["message"]["content"].

3. Stream content

Streaming uses the StreamExt trait, so add futures as a direct dependency:

cargo add futures

This complete example prints text deltas as they arrive:

use futures::StreamExt;
use serde_json::json;
use std::io::{self, Write};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let router = llmshim::router::Router::from_env();
    let request = json!({
        "model": "anthropic/claude-sonnet-5-5",
        "messages": [
            {"role": "user", "content": "Write a haiku about Rust."}
        ],
        "max_tokens": 128
    });

    let mut stream = llmshim::stream(&router, &request).await?;

    while let Some(chunk) = stream.next().await {
        let chunk = chunk?;
        let parsed: serde_json::Value = serde_json::from_str(&chunk)?;

        if let Some(text) = parsed
            .pointer("/choices/0/delta/content")
            .and_then(|value| value.as_str())
        {
            print!("{text}");
            io::stdout().flush()?;
        }
    }

    println!();
    Ok(())
}

Each item is a JSON string in the normalized Chat Completions delta shape. A chunk can carry something other than text, such as reasoning, usage, or a tool call, so production code should inspect the fields it needs.

For config-file loading, aliases, and model inference, see Models and the Router. Runnable repository examples are available in examples/chat.rs and examples/stream.rs.

CLI quickstart

Use the CLI for an interactive, streaming conversation from a terminal. It can also configure llmshim and start the HTTP proxy.

Availability: Rust: wrapped by the CLI · CLI: interactive workflow · HTTP: available through llmshim proxy · Clients: separate

1. Install the binary

On macOS with Homebrew:

brew install sanjay920/tap/llmshim

Or install from source. The proxy feature keeps the proxy command available:

cargo install llmshim --features proxy

Running llmshim with no subcommand prints the command overview.

2. Configure a provider

llmshim configure

You can use environment variables instead. See Configure providers for precedence and the four supported key names.

3. Start chatting

llmshim chat

Choose a model at the prompt, then enter a message. The CLI streams each answer as it arrives. It requests reasoning_effort: "high" on every turn; reasoning returned by the provider is shown in dim gray before the answer.

The CLI keeps conversation history in memory for this process. Switch the next turn to another provider without clearing that history:

/model

Use /clear when you want a new conversation.

Attach an image

Attach a local image before the next message:

/image ./diagram.png

The CLI reads the file, converts it to a base64 data URI, and adds it to the next user message. /paste attempts to attach an image from the system clipboard on supported desktop environments.

Common operational tasks

llmshim models               # models for configured providers
llmshim set proxy.port 8080  # update one config value
llmshim get proxy.port       # read one config value
llmshim list                 # show masked keys and proxy settings
llmshim proxy                # start the HTTP API

This page covers the normal workflow. The exhaustive command and interactive slash-command list belongs in the CLI reference.

HTTP proxy quickstart

The proxy puts a network boundary around the Rust engine. Use it from any language that can send JSON over HTTP.

Availability: Rust: engine inside the server · CLI: starts the server · HTTP: compact llmshim API · Clients: connect to this API

The proxy has its own compact contract. It is not an OpenAI-compatible proxy.

1. Install a proxy-enabled binary

The proxy is behind the crate's proxy feature when building from source:

cargo install llmshim --features proxy

The Homebrew binary also includes the proxy:

brew install sanjay920/tap/llmshim

Configure at least one provider before starting it:

llmshim configure

2. Start the server

llmshim proxy

The default bind address is 0.0.0.0:3000. Override it for the process with LLMSHIM_HOST and LLMSHIM_PORT:

LLMSHIM_HOST=127.0.0.1 LLMSHIM_PORT=8080 llmshim proxy

The examples below assume the default port and use localhost to reach it.

3. Send a chat request

curl http://localhost:3000/v1/chat \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "openai/gpt-6-sol",
    "messages": [
      {"role": "user", "content": "What is Rust in one sentence?"}
    ],
    "config": {
      "max_tokens": 128
    },
    "provider_config": {
      "x-anthropic": {
        "disable_1m_context": true
      }
    }
  }'

config contains portable controls. provider_config is optional and merges provider-specific fields into the core request; omit it when you do not need a native control.

The non-streaming response uses llmshim's compact ChatResponse:

{
  "id": "msg_...",
  "model": "gpt-6-sol",
  "provider": "openai",
  "message": {
    "role": "assistant",
    "content": "Rust is a systems programming language..."
  },
  "usage": {
    "input_tokens": 15,
    "output_tokens": 12,
    "total_tokens": 27
  },
  "latency_ms": 640
}

IDs, text, token counts, and timing vary by request. reasoning and message.tool_calls appear only when the provider returns them.

Set top-level "stream": true on the same request to receive typed SSE events instead of a JSON response. POST /v1/chat/stream is the always-streaming equivalent.

See the HTTP API for the full contract.

Do not expose the bare proxy publicly

The server has permissive CORS and provides no authentication or TLS. For local use, bind it to 127.0.0.1. For a public deployment, put it behind an external gateway that supplies authentication and TLS. The supported topology and deployment checklist are covered in Deploy the proxy safely.

Stateless Responses requests

POST /v1/responses accepts a Responses request with a configured model route:

curl http://localhost:3000/v1/responses \
  -H 'Content-Type: application/json' \
  -d '{"model":"openai/gpt-6-astra","input":"Hello","store":false}'

This endpoint supports JSON and streaming requests: string input or message items with text/image parts, instructions, function tools and tool-call/result items, tool_choice, max_output_tokens, temperature, top_p, reasoning.effort, and text.format (text, JSON object, or JSON Schema). Function arguments and tool outputs are strings. Send the full history on every request; call_id links a function call to its output. Images require image_url, with optional detail (auto, low, or high). File inputs and file-backed images are unsupported.

The result has object: "response", message/output-text or refusal items, function-call items, provider-supplied reasoning summaries, and input/output, cached, and reasoning token details. Token exhaustion returns status: "incomplete" and incomplete_details.reason: "max_output_tokens". The proxy and gateway share their existing admission, logging, error status, Retry-After, and credential behavior with this route.

Set stream: true for Responses SSE events with contiguous sequence numbers and a terminal response containing output and usage. Reasoning items returned by an earlier request can be included unchanged in input; scoped receipts preserve provider provenance. Unsupported or unissued reasoning is dropped with reasons in metadata.reasoning_dropped. Use include: ["reasoning.encrypted_content"] for encrypted output where supported.

previous_response_id, conversation, store: true, and hosted tools return a named 400. Unknown top-level fields are refused. See native APIs for the stream and replay contract.

Language-client quickstarts

All four language clients speak the proxy's compact HTTP contract. The key difference is whether the package starts that proxy for you.

Availability: Python/TypeScript: bundled proxy, auto-start by default · Go/Ruby: pure HTTP, running proxy required

The auto-started Python and TypeScript proxies are reused within the current process and stopped when it exits. TypeScript can instead connect to a proxy you already run by setting baseUrl; the current Python public API always uses its managed local proxy. Go and Ruby never spawn the Rust binary; start llmshim proxy before using their default clients.

Provider keys belong to the proxy process, not to the HTTP client. Configure them through environment variables or llmshim configure.

Python

pip install llmshim

import llmshim

response = llmshim.chat("gpt-6-sol", "What is Rust?")
print(response["message"]["content"])

The first call starts the bundled proxy automatically.

Python README · PyPI package

TypeScript / JavaScript

npm install llmshim

import { Client } from "llmshim";

const client = new Client();
const response = await client.chat({
  model: "openai/gpt-6-sol",
  messages: [{ role: "user", content: "What is Rust?" }],
});

console.log(response.message.content);

Constructing the client is synchronous. Its first request starts the bundled proxy. Pass baseUrl to new Client(...) to use an existing server instead.

TypeScript README · npm package

Go

go get github.com/sanjay920/llmshim/clients/go

package main

import (
	"context"
	"fmt"
	"log"

	llmshim "github.com/sanjay920/llmshim/clients/go"
)

func main() {
	client := llmshim.New()
	response, err := client.Chat(context.Background(), llmshim.ChatRequest{
		Model: "openai/gpt-6-sol",
		Messages: []llmshim.Message{
			{Role: "user", Content: "What is Rust?"},
		},
	})
	if err != nil {
		log.Fatal(err)
	}

	fmt.Println(response.Message.Content)
}

llmshim.New() targets http://localhost:3000; it does not start the proxy.

Go README · Go module

Ruby

gem install llmshim

require "llmshim"

response = Llmshim.chat(
  model: "openai/gpt-6-sol",
  messages: "What is Rust?"
)

puts response.message.content

The module-level client targets LLMSHIM_BASE_URL when set, otherwise http://localhost:3000. It does not start the proxy.

Ruby README · RubyGems package

These snippets cover one non-streaming request. Use the linked client README as the canonical reference for streaming, errors, configuration, and native types rather than relying on duplicated method lists here.

How translation flows

Translation happens twice: once before the network request and once after it.

sequenceDiagram
    participant Caller
    participant Boundary as Rust API / proxy adapter
    participant Router
    participant Adapter as Provider adapter
    participant API as Native provider API

    Caller->>Boundary: model + messages + controls
    Boundary->>Router: resolve(model)
    Router-->>Boundary: provider + raw model name
    Boundary->>Adapter: transform_request(model, value)
    Adapter->>API: provider-native HTTP request
    API-->>Adapter: provider-native response
    Adapter-->>Boundary: transform_response(model, response)
    Boundary-->>Caller: normalized result

1. Resolve the model address

Every request contains a model string. Router::resolve turns that string into two things:

• a registered provider adapter;
• the raw model name to send to that provider.

For example, anthropic/claude-sonnet-5-5 selects the adapter registered as anthropic and passes claude-sonnet-5-5 to it. A bare model name can use prefix inference instead. Models and the Router covers the exact rules.

Resolution chooses an adapter. It does not check the static model-discovery list or choose a model based on prompt content.

2. Translate the outbound request

The Rust API receives the request as serde_json::Value. There is no canonical Rust request struct between the caller and the providers. The selected adapter's transform_request method reads the fields it understands and builds the provider-native URL, headers, and JSON body.

This is where an OpenAI-style messages array becomes an Anthropic Messages request, Gemini contents, or an OpenAI/xAI Responses request. Common controls are translated here too.

3. Send the native HTTP request

The shared HTTP client sends the translated request to the selected provider. Provider authentication belongs to this outbound request; callers do not send one provider's key to another provider.

4. Translate the inbound response

The same adapter's transform_response method converts the provider-native response into llmshim's normalized OpenAI Chat Completions-style value. Text, tool calls, usage, and provider-returned reasoning are placed into that common shape when present.

The proxy then performs one additional conversion into its compact ChatResponse. This is why the Rust and proxy APIs share semantics without sharing the same response envelope.

Streaming follows the same boundary

Streaming changes the inbound step, not the model. Each provider emits its own SSE event format. The adapter's third method, transform_stream_chunk, converts individual provider events into normalized chunks. The Rust API yields those chunks directly; the proxy converts them again into typed content, reasoning, tool_call, usage, done, and error events.

Two contracts, one engine

llmshim exposes one set of capabilities through two data contracts. The Rust crate uses flexible OpenAI-shaped JSON. The HTTP proxy uses a smaller request and response envelope designed for llmshim's clients.

The contracts are related, but they are not wire-compatible.

Rust: OpenAI-shaped Value

llmshim::completion and llmshim::stream accept a serde_json::Value. Conversation fields and controls live at the top level:

#![allow(unused)]
fn main() {
let request = serde_json::json!({
    "model": "anthropic/claude-sonnet-5-5",
    "messages": [{ "role": "user", "content": "Hello" }],
    "max_tokens": 200,
    "reasoning_effort": "high"
});
}

A non-streaming result uses an OpenAI Chat Completions-style shape. The answer is normally at choices[0].message.content; translated tool calls live beside it, and provider-returned reasoning uses an ordered reasoning[] array with provenance.

Rust streaming yields JSON strings in the normalized Chat Completions chunk shape. It does not yield the proxy's typed event objects.

Proxy: a compact llmshim envelope

The proxy accepts model and messages, then groups portable controls under config:

{
  "model": "anthropic/claude-sonnet-5-5",
  "messages": [{ "role": "user", "content": "Hello" }],
  "config": {
    "max_tokens": 200,
    "reasoning_effort": "high"
  }
}

A non-streaming proxy response is a compact ChatResponse:

{
  "id": "...",
  "model": "claude-sonnet-5-5",
  "provider": "anthropic",
  "message": { "role": "assistant", "content": "..." },
  "usage": {
    "input_tokens": 8,
    "output_tokens": 12,
    "total_tokens": 20
  },
  "latency_ms": 640
}

When the provider returns reasoning content, the response also includes the optional reasoning field.

The streaming endpoints emit typed SSE events rather than Chat Completions chunks. Language clients decode the same HTTP contract into their native types.

Where this goes

Use this placement convention throughout the documentation:

The CLI is a workflow, not a fourth JSON contract. It builds requests, keeps history for the current process, and renders results for a person.

Portable core, native edges

Portable does not mean every provider has the same API. It means you can express the common intent once and let the selected adapter handle the provider's wire format.

What the portable layer does

Provider adapters recognize the common conversation fields they support. They translate message roles and content, common generation controls, tool definitions and results, image blocks, streaming events, and unified reasoning controls.

The request remains a serde_json::Value, but it is not arbitrary pass-through JSON. Each adapter builds a new native request from understood fields. An unknown top-level field is not automatically forwarded to every provider.

Provider differences have three possible outcomes:

• Translated: the provider supports the intent under another field or structure.
• Clamped or omitted: the provider has a smaller vocabulary or cannot accept that control.
• Sent natively: the caller deliberately uses a provider extension.

Reasoning illustrates the choice:

flowchart LR
    U[Unified effort + mode] --> P{Native override present?}
    P -- Yes --> N[x-provider native config]
    P -- No --> F[Model-family mapping]
    F --> C[Clamp to supported tier]
    C --> W[Provider-native reasoning control]

The exact reasoning mappings belong in the reasoning guide. The important concept is precedence: an explicit native control wins; otherwise llmshim maps the portable intent to what the selected model accepts.

Native extensions

Three built-in adapters expose explicit native namespaces:

There is no x-xai namespace. xAI's supported common controls are translated directly to its Responses API shape.

In the Rust contract, an extension is a top-level request key:

{
  "x-openai": {
    "reasoning": { "effort": "high", "summary": "auto" }
  }
}

In the proxy contract, put that same key inside provider_config because the proxy merges provider_config into the core request:

{
  "provider_config": {
    "x-openai": {
      "reasoning": { "effort": "high", "summary": "auto" }
    }
  }
}

Using a native extension intentionally reduces portability. A request that depends on x-openai should not be expected to retain that behavior after its model is changed to Anthropic or Gemini.

Models and the Router

The model string is a routing address. It tells llmshim which provider adapter should receive the request and which model name that adapter should send upstream.

Prefer explicit addresses

The most explicit form is provider/model:

openai/gpt-6-sol
anthropic/claude-opus-5-5
gemini/gemini-3.8-flash
xai/grok-4.7

The part before the first slash is the Router registration key. The remainder is sent to that provider as the model name. Explicit addresses are easiest to read and do not depend on naming conventions.

Bare-name inference

When there is no slash, llmshim lowercases the name for prefix matching while preserving the original model string:

A bare name outside those prefixes produces an unknown-provider error. Use an explicit address when inference cannot identify the provider.

Registration and discovery are different

Resolution succeeds only when the selected provider key is registered on the Router. The built-in Router::from_env() registers OpenAI, Anthropic, Gemini, and xAI only when their corresponding environment variables are present. ChatGPT is registered when its OAuth cache exists; sign in with llmshim login chatgpt before starting the proxy. Use chatgpt/<model> to select subscription access. Bare GPT names continue to use OpenAI API keys.

The static model registry powers llmshim models and GET /v1/models. Those commands are discovery aids, filtered to configured providers. The registry is not generally an allowlist: most providers accept models absent from that list. ChatGPT accepts only chatgpt/gpt-6-astra, chatgpt/gpt-6-sol, chatgpt/gpt-6.1-sol, and chatgpt/gpt-6-luna. The provider rejects other IDs before authentication or network calls, including through aliases.

For that reason, this documentation does not maintain another static model table. Use runtime discovery for the current curated list.

Aliases are a Rust Router feature

Rust applications can attach a one-level alias while building a Router:

#![allow(unused)]
fn main() {
let router = llmshim::router::Router::from_env()
    .alias("smart", "anthropic/claude-opus-5-5");
}

The Router checks an alias before parsing the provider address. An alias target may be an explicit address or a bare model name, but aliases do not recursively chain. If a points to b and b points to a model, resolving a does not perform the second lookup.

Aliases are not currently configurable through the CLI, config file, proxy API, or language clients.

Named routes

An alias renames a model. A named route goes further: it maps a caller-defined name to a model plus request settings, configured in ~/.llmshim/config.toml.

[routes.compaction]
model = "anthropic/claude-haiku-4-5-20251001"
reasoning_effort = "low"
max_tokens = 4096

Address it as a model:

{"model": "route/compaction", "messages": [{"role": "user", "content": "…"}]}

Because it reuses the provider/model grammar, a route works everywhere a model address does — the Rust API, the CLI, the proxy, the native endpoints and an unmodified OpenAI SDK.

The name is opaque to llmshim. A harness may call a route compaction, advisor or webSearch; llmshim never interprets it and has no built-in role vocabulary. The harness decides what a name means; llmshim provides only the mechanism.

Three rules:

• Settings are defaults. A key the request already carries wins, so a caller can pick a route and still raise reasoning_effort for one call.
• An unknown name is an error (HTTP 400), never a silent fall back to a default model.
• Routes do not chain. A route's model may not be another route/….

A Rust application can register routes directly:

#![allow(unused)]
fn main() {
use llmshim::config::Route;

let router = llmshim::router::Router::new()
    .route("compaction", Route { model: "anthropic/claude-haiku-4-5-20251001".into(), ..Default::default() });
}

Environment variables versus config.toml

Router::from_env() reads provider environment variables such as OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY, and XAI_API_KEY. It does not read API keys from ~/.llmshim/config.toml by itself; it does read that file's [routes] table, which has no environment equivalent. It also discovers the selected ChatGPT OAuth cache, whose default location is ~/.llmshim/chatgpt/auth.json.

The CLI and proxy call llmshim::env::load_all() before constructing their Router. That function loads the config file and fills only environment variables that are not already set, so environment variables take precedence.

A Rust application that wants the same config-file behavior must request it:

#![allow(unused)]
fn main() {
llmshim::env::load_all();
let router = llmshim::router::Router::from_env();
}

Applications that manage secrets themselves can call Router::from_env() directly or construct a Router by registering provider implementations.

Catalog refresh is the daemon's default, not the embedder's

Router::from_env() also schedules one background fetch of the model catalog (prices, context windows, capabilities). That is right for the proxy, which starts once and runs for days. A program that starts many times a day, or runs air-gapped, should build its router with Router::from_env_without_catalog_refresh() — identical, except that constructing it makes no network call — and refresh only when it decides to:

#![allow(unused)]
fn main() {
let router = llmshim::router::Router::from_env_without_catalog_refresh();
if user_asked_for_it {
    // Rides the caller's Tokio runtime; `None` when there is none, when
    // LLMSHIM_CATALOG_OFFLINE=1 is set, or when the local catalog is invalid.
    router.refresh_catalog_in_background();
}
}

The offline router still resolves and prices every model in the vendored snapshot, any earlier cached download, and the local override files; its data is simply never newer than the disk.

Conversations across models

llmshim does not store conversations. A conversation continues because the caller sends the previous messages again.

Start with a history:

[
  { "role": "user", "content": "What is a Rust closure?" },
  { "role": "assistant", "content": "A closure is..." },
  { "role": "user", "content": "Explain that another way." }
]

Send it to another provider by keeping messages and changing the address:

- "model": "anthropic/claude-opus-5-5"
+ "model": "openai/gpt-6-sol"

There is no session handoff between Anthropic and OpenAI. The new provider sees the history because it is present in the new request.

Who owns history?

• Rust applications keep and resend their own messages array.
• HTTP callers and language clients do the same. The proxy is stateless and does not assign conversation IDs.
• llmshim chat keeps history in memory for the interactive process. The /model command changes the model for the next request without clearing that history; /clear removes it.

Provider adapters translate recognized cross-provider artifacts before sending the history upstream. This includes common message roles and supported tool call/result shapes. They also remove provider-only response metadata that should not be sent to another API.

That translation makes a shared history usable across providers, but it does not make every provider-specific detail portable. A turn that depends on a native extension, unsupported content block, or provider-only behavior can still lose that behavior after a switch.

What model switching does not do

Changing the model does not:

• summarize or truncate history;
• run more than one model;
• choose a model automatically;
• persist messages after the caller or CLI process discards them.

Multi-model conversation is therefore a small operation: preserve the history, choose a new model address, and send the next request.

Streaming

Streaming exposes the same kinds of incremental output on every surface, but the Rust crate and the proxy encode them differently.

Availability: Rust: normalized JSON chunks · CLI: rendered live · Proxy: typed SSE · Clients: language-native stream iterators

The useful distinction is: same semantic events, two encodings.

Rust: normalized Chat Completions chunks

llmshim::stream returns a stream whose items are JSON strings. Each provider adapter and the shared per-response normalizer have translated native events into Chat Completions-style deltas. Tool calls are complete records emitted once at termination, after their argument and signature fragments have been assembled. Retain each call's wire_ids when storing it or sending its result back.

#![allow(unused)]
fn main() {
use futures::StreamExt;
use std::io::{self, Write};

let mut stream = llmshim::stream(&router, &request).await?;
let mut reasoning = llmshim::reasoning::ReasoningAccumulator::default();

while let Some(chunk) = stream.next().await {
    let chunk = chunk?;
    let parsed: serde_json::Value = serde_json::from_str(&chunk)?;

    let delta = &parsed["choices"][0]["delta"];
    reasoning.push(delta)?;
    eprint!("{}", llmshim::reasoning::reasoning_text(delta));

    if let Some(text) = parsed
        .pointer("/choices/0/delta/content")
        .and_then(|value| value.as_str())
    {
        print!("{text}");
        io::stdout().flush()?;
    }
}
}

Not every chunk contains text. Inspect only the fields your application needs, and keep handling Err items until the stream ends. Store reasoning.blocks() with the final assistant message only after successful completion. An EOF or [DONE] without a terminal provider chunk is an error. Cache usage events are cumulative snapshots; use the latest values, rather than summing events.

The Rust quickstart contains a complete runnable program.

Showing a tool call while it is written

A long argument, such as a file's new contents, can take many seconds to stream. To show the call before it completes, implement AttemptPolicy::observe_tool_call_progress and stream with a policy (ShimClient::stream_with_policy, stream_owned_with_policy or llmshim::stream_with_policy). Each call is reported in the same shape for every provider: Started with the tool's name, Arguments with the text so far, then Ended or Abandoned.

#![allow(unused)]
fn main() {
use llmshim::{policy::{AttemptIdentity, AttemptPolicy}, toolcall::{ToolCallProgress, ToolCallProgressEvent}};

impl AttemptPolicy for MyPolicy {
    // acquire, observe and observe_abandoned as before …

    fn observe_tool_call_progress(&self, attempt: &AttemptIdentity, progress: ToolCallProgress) {
        match progress.event {
            ToolCallProgressEvent::Started { name } => self.show(attempt.id(), progress.id, name),
            ToolCallProgressEvent::Arguments { text, .. } => self.preview(progress.id, text),
            ToolCallProgressEvent::Abandoned => self.discard(progress.id),
            _ => {}
        }
    }
}
}

The progress is provisional. The completed call in the stream is still the only call to execute, validated as before, and it carries the same id as its progress. Ended is reported before that chunk is produced. Arguments.text is partial JSON; appended is the new part, or None when the provider replaced the text. attempt separates one attempt's calls from another's. If the stream ends with an error, discard the progress of every call whose completed form did not arrive. A policy that does not implement the method gets the same stream as before. Calls that llmshim makes for its own output contracts (a forced answer tool, prompt-encoded calls) are never reported.

Proxy: typed SSE events

Send the compact request to the always-streaming endpoint:

curl -N http://localhost:3000/v1/chat/stream \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "anthropic/claude-sonnet-5-5",
    "messages": [{"role": "user", "content": "Write a haiku."}],
    "config": {"max_tokens": 128}
  }'

The response is an SSE stream. The SSE event name and the JSON type agree:

event: reasoning
data: {"type":"reasoning","text":"..."}

event: content
data: {"type":"content","text":"Rust shapes silent thought"}

event: usage
data: {"type":"usage","input_tokens":12,"output_tokens":7,"total_tokens":19}

event: done
data: {"type":"done"}

The six event types are:

Setting "stream": true on POST /v1/chat produces the same SSE encoding. An admission error can still arrive as an HTTP error before streaming begins; an error after the stream begins arrives as an error event.

Client ergonomics

The language clients decode the typed SSE stream without changing its event vocabulary:

• Python returns an iterator of event dictionaries.
• TypeScript returns an async iterator of typed events.
• Go returns a channel of typed events.
• Ruby yields typed events to a block.

See the canonical client guides for complete loops: Python, TypeScript, Go, and Ruby.

Tool use

llmshim translates tool definitions, tool calls, and tool results. It never executes a tool. Your application owns that part of the loop.

Availability: Rust: top-level tools · CLI: no tool loop · Proxy/clients: provider_config.tools

The loop is always:

define tools → receive tool_calls → execute in your app → send tool results → continue

1. Define tools

Use the OpenAI Chat Completions function format. The function object is nested inside the tool definition:

{
  "type": "function",
  "function": {
    "name": "get_weather",
    "description": "Get the current weather for a city",
    "parameters": {
      "type": "object",
      "properties": {
        "city": {"type": "string"}
      },
      "required": ["city"]
    }
  }
}

In a Rust request, tools is a top-level field:

{
  "model": "anthropic/claude-sonnet-5-5",
  "messages": [{"role": "user", "content": "What is the weather in Tokyo?"}],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get the current weather for a city",
        "parameters": {
          "type": "object",
          "properties": {"city": {"type": "string"}},
          "required": ["city"]
        }
      }
    }
  ]
}

Through the proxy, place the identical array at provider_config.tools:

{
  "model": "anthropic/claude-sonnet-5-5",
  "messages": [{"role": "user", "content": "What is the weather in Tokyo?"}],
  "provider_config": {
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "get_weather",
          "description": "Get the current weather for a city",
          "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"]
          }
        }
      }
    ]
  }
}

Python and Ruby expose convenience tools arguments that build this provider_config field. TypeScript and Go expose provider_config directly.

2. Receive a tool call

llmshim normalizes a provider's request to call a function into the OpenAI shape:

{
  "role": "assistant",
  "content": null,
  "tool_calls": [
    {
      "id": "call_ls_example",
      "wire_ids": [{"provider":"openai","wire":"openai-responses","scope":"response_example","part_id":"0","id":"call_provider_123","item_id":"fc_example"}],
      "type": "function",
      "function": {
        "name": "get_weather",
        "arguments": "{\"city\":\"Tokyo\"}"
      }
    }
  ]
}

arguments is a JSON-encoded string. Parse and validate it before calling your application code. Treat model-generated arguments as untrusted input.

3. Execute it in your application

Dispatch get_weather in your own code. llmshim does not have access to that function and does not decide whether it is safe to run.

Keep the returned assistant message in the conversation, including its tool_calls. Preserve any additional fields on those calls; for example, Gemini can return a thought_signature object with data and origin. Every generated call also has wire_ids; retain that complete array. Then append one tool-result message for each call:

[
  {
    "role": "assistant",
    "content": null,
    "tool_calls": [
      {
        "id": "call_ls_example",
      "wire_ids": [{"provider":"openai","wire":"openai-responses","scope":"response_example","part_id":"0","id":"call_provider_123","item_id":"fc_example"}],
        "type": "function",
        "function": {
          "name": "get_weather",
          "arguments": "{\"city\":\"Tokyo\"}"
        }
      }
    ]
  },
  {
    "role": "tool",
    "tool_call_id": "call_ls_example",
    "content": "{\"temperature_c\":24,\"conditions\":\"clear\"}"
  }
]

The tool_call_id connects the result to the request. Send the expanded history and tool definitions in another completion so the model can use the result and produce an answer.

Identity and validation

llmshim mints each canonical id. wire_ids keeps the native correlation id and, for Responses, its separate item id. Reply with the canonical id and keep the original assistant call unchanged; the adapter translates both sides back to the same wire identity. Switching providers creates a deterministic target wire mapping without changing stored history. Dropping the mapping from an owned call is a local error.

All adapters validate that each tool call is answered exactly once before the next assistant turn. Unknown results, unanswered calls, duplicate owned ids, and invalid argument JSON fail before the provider request. Anthropic thinking blocks must precede text/tools. Invalid history is reported, rather than silently removing calls or results.

Gemini preserves whether the original function call had an id. When it did, its function response gets the same id. Otherwise result ordering preserves correlation, including repeated calls to the same function. The first call in a current Gemini 3 parallel batch carries the required signature; later calls retain their original signature presence/absence. See Google's signature contract. A compacted tool history must retain the user turn that introduced its calls.

Streaming calls

llmshim::stream uses one per-response assembler for every transport, including ChatGPT. Argument fragments, ids, and signatures accumulate under stable part ids. Completed calls appear once after terminal metadata, with full JSON arguments and their wire mappings; incomplete JSON never becomes a callable record. This applies to typed proxy tool_call events too.

For manually read SSE, create provider.stream_normalizer(model) and feed native data strings to push. Call finish on EOF. The lower-level stateless transform_stream_chunk only handles ordinary content/reasoning; it cannot assemble tool calls across events. Rust chunks retain separate choices; the single-message proxy contract projects choice zero.

What llmshim translates

Responses and stream chunks are translated back to the OpenAI tool_calls shape before they cross the Rust boundary. The proxy then exposes the same call as message.tool_calls or a typed tool_call stream event.

For the broader rule behind the surface-specific placement, see Two contracts, one engine.

Tool schema normalization

Every native adapter uses the same option-driven schema walker, after native tool overrides. The target selects transport rules; schema logic is not copied into each provider.

#![allow(unused)]
fn main() {
use llmshim::schema::{normalize_for, Target};

let mut schema = serde_json::json!({
    "type": "object",
    "properties": {"name": {"type": ["string", "null"]}}
});
let report = normalize_for(Target::Google, &mut schema);
assert!(!report.used_fallback);
assert_eq!(schema["properties"]["name"]["nullable"], true);
}

The walker upgrades draft-07 tuple/dependency forms, resolves document-local references, embedded resources and anchors, removes unused definitions, and normalizes snake-case schema keys. Snake-case wins when both spellings exist. Property names and literal values in defaults/examples/enums are not treated as schema keywords. External resources are never fetched.

Responses rewrites oneOf to anyOf and removes regex lookarounds. Google uses nullable markers; Cloud Code Assist removes nullable markers and resolves supported homogeneous combinators. Incompatible residual schemas use a per-tool fallback: {"type":"object","properties":{}}. Other tools in the request remain available. Circular/unresolved references, invalid shapes, and bounded expansion failures take the same fallback path.

Strict mode sanitizes unsupported keywords, closes every object with additionalProperties:false, makes every property required, and represents optional properties as nullable alternatives. Human constraints such as defaults, formats, patterns, bounds and dependencies are lifted into descriptions before they are removed. A provider receives strict:true only after successful enforcement. Keep an original schema if your application validates tool inputs; wire sanitization can relax machine-enforced constraints.

Use normalize_mcp_tool(&mut tool) when ingesting an MCP inputSchema into a registry. Raw MCP-style definitions (name, description, inputSchema) are also accepted in the request's tools array and normalized before native dispatch. MCP tool parameters must describe an object.

normalize_output_for(target, schema, strict) resolves references before wrapping non-object output types under response. Its OutputSchema::unwrap reverses that wrapper. The result reports whether normalization fell back; it does not itself validate a model's generated instance or perform a repair call. The HTTP capability shim does both against the original schema. schema::validate::compile exposes local-only instance validation separately.

Two escape hatches are available:

• LLMSHIM_NO_SCHEMA_NORMALIZATION=1 bypasses schema rewriting and disables requested strict flags.
• LLMSHIM_NO_STRICT=1 disables strict enforcement while retaining transport normalization.

The public Options API also permits an explicit bypass and depth/node/literal budgets. Defaults bound expansion to depth 96, 32,768 schema nodes and 8 MiB of literal/URI work, including repeated reference expansion. No network or filesystem reads are part of schema normalization.

Memoization

Normalization reuses a process-local result for the same schema content and resolved Options, including the target, strict/nullable policies, rewrite flags, and all budgets. Function names and descriptions outside the parameter schema continue to come from the current tool definition. This covers provider tools, MCP input schemas, and output-schema normalization through the same boundary.

The cache holds up to 512 entries and 16 MiB of estimated owned input/output bytes, with least-recently-used eviction. Entries contain schemas and normalization reports only; there is no disk persistence or response/prompt cache. Hash hits also compare the input and options, including object order and signed zero, so a collision cannot substitute another result. Returned values are independent of cached values. Exact no-op hits leave the caller's value in place.

Schema walking and validation compilation run outside the cache lock. Fallback results are cacheable too, because reference resolution never fetches external resources. Environment overrides are resolved before lookup; changing LLMSHIM_NO_STRICT or LLMSHIM_NO_SCHEMA_NORMALIZATION still takes effect. Set LLMSHIM_NO_SCHEMA_CACHE=1 to compare uncached behavior without disabling normalization. Inputs beyond the cache's bounded key-work limits take the normal uncached path and retain the normalizer's existing validation/fallback behavior.

The cache removes repeated normalization work. Whole-request copying, hashing, and provider-specific translation still contribute to per-request cost. The release benchmark measures the full transform at several tool counts; README results specify the workload and measurement scope.

Capability shims

Send a JSON Schema once, regardless of the provider:

{
  "model": "openai/gpt-6-astra",
  "messages": [{"role": "user", "content": "Return the first three primes."}],
  "response_format": {
    "type": "json_schema",
    "json_schema": {
      "name": "primes",
      "schema": {"type": "array", "items": {"type": "integer"}, "minItems": 3, "maxItems": 3}
    }
  },
  "x-shim": {
    "structured_output": "auto",
    "tool_calling": "auto",
    "reasoning_capture": "off"
  }
}

The final message.content is JSON text, such as [2,3,5]. Native transports and forced functions wrap non-object schemas internally and unwrap the result. A successful final answer must validate against the original schema, including constraints relaxed by wire normalization. Strict-schema nulls for optional, non-nullable properties are restored to omission; explicitly nullable values stay null. External schema resources are never fetched. Invalid or oversized schemas fail before dispatch.

structured_output: auto uses a native format when the catalog asserts support, then a pinned function when tools are supported and forced choice is not known to be forbidden, otherwise a JSON instruction in the prompt. A native-format assertion takes precedence for Anthropic too. native, forced_tool, and prompt select a path explicitly. Explicit native/forced selection may be rejected by a provider that does not implement it. Unknown capability flags are not treated as verified support. One request holds one catalog snapshot.

Native mappings are Responses text.format, Chat Completions response_format, Anthropic output_config.format, and GenerateContent generationConfig.responseMimeType plus responseSchema. See the official OpenAI, Anthropic, and Google documentation for the provider contracts.

An invalid complete JSON answer gets one corrective request. Both attempts' reported usage contributes to the final totals, including cache tokens. A second validation failure is an error. Refusals are preserved without repair; incomplete responses and transport failures are not schema-repair requests. Ordinary native user tool calls can still be intermediate turns before a native or prompted structured final answer. Forced-output mode requests a final answer.

tool_calling: auto switches to a prompt protocol only when the catalog says the model has no tool API. prompt selects that protocol explicitly; native uses the provider's function API. Prompt calls are checked against the original tool names and argument schemas, and honor none, required, pinned function choice, and parallel_tool_calls:false. They receive ordinary owned call IDs and wire mappings. The library never executes tools. Keep the same declarations and mode on a follow-up request; paired tool history is rendered as text for a prompt protocol. Native-capable destinations can consume the same canonical history.

reasoning_capture: forced_tool requests a brief explanation through an internal function with one text argument. Its envelope contains the explanation and the answer or requested user tool calls. The public explanation appears in message.reasoning; internal functions never appear in the caller's tool list or returned calls. On endpoints that cannot force functions, the same envelope uses the prompt protocol. This is a generated rationale, not a provider's private reasoning trace. Native deliberation about hidden protocol calls is not exposed, and no replay signature is fabricated for the rationale.

Managed output streams buffer until validation succeeds and then emit canonical content, reasoning, tool calls, usage, and finish metadata. Invalid attempts and internal calls never become stream events or successful log entries. Ordinary streams remain incremental. Buffering is bounded to 32 MiB of normalized chunks. Top-level stream and ShimClient::stream_owned open the first HTTP response before buffering, allowing frontends to send keepalives; the borrowed-provider ShimClient::stream awaits validation before returning its buffered stream.

The HTTP client applies plans to completions, streams, and fallback attempts. Direct Provider::transform_request calls only translate native formats; they do not execute a shim or repair loop. Providers never receive x-shim. Native extension overrides cannot replace a managed tool/output protocol. Models with verified unsupported reasoning controls have those controls removed before serialization. Changing a schema or shim mode changes the prompt/cache identity; keep them stable for a conversation when relying on provider prefix caching.

/v1/chat and /v1/chat/stream accept response_format and x-shim at the top level. Python and Ruby provide shim= / shim: and response_format options. Go exposes ChatRequest.Shim and ResponseFormat; TypeScript uses the JSON keys.

Prompt cache annotations

The caller knows which prompt sections stay stable. Declare that explicitly; llmshim translates the declaration into the provider's caching mechanism.

{
  "model": "anthropic/claude-sonnet-5-5",
  "messages": [
    {"role":"system","content":"Stable instructions"},
    {"role":"user","content":"Session context"},
    {"role":"user","content":"The current question"}
  ],
  "x-cache": {
    "segments": [
      {"upto_message":0,"label":"instructions","stability":"static"},
      {"upto_message":1,"label":"context","stability":"session"},
      {"upto_message":2,"label":"question","stability":"turn"}
    ],
    "key":"session:branch"
  }
}

upto_message is a zero-based index into the messages array exactly as you sent it — system and developer messages count, and each role: "tool" result counts as its own entry — not into the provider-native array (Anthropic hoists the system prompt out, so native numbering differs). A caller whose own message model expands into more wire messages than it holds must remap before copying an index across. On Anthropic an index past the end is rejected with a 400, and one that lands too early silently caches less than intended; on every other provider segments are parsed but neither checked nor placed. Labels are informational and are never used to guess prompt semantics. Keep stable sections before volatile sections. The proxy accepts the same top-level x-cache field; Python and Ruby provide a cache=/cache: keyword, Go has ChatRequest.Cache, and TypeScript accepts the field directly.

For Anthropic, static ends with a one-hour marker and session with a five-minute marker. turn never creates a marker. Eligible boundaries are selected from the end, with at most four markers. Existing explicit markers consume that budget. Managed segments supersede automatic end-of-request caching, so a volatile tail does not accidentally become a breakpoint. Invalid indices and one-hour markers following five-minute markers fail locally. These limits and TTL ordering follow the Anthropic cache contract.

For Responses transports, key becomes prompt_cache_key. No key is derived when it is absent. Other transports drop the annotation. Without x-cache, llmshim adds no breakpoints and preserves existing native cache behavior.

Every response and usage event reports cache_read_tokens and cache_write_tokens. Values are provider-reported cumulative snapshots; do not add repeated streaming snapshots together. Zero means no cache tokens were reported, not that the request was free.

They are joined by uncached_input_tokens — the prompt minus whatever cache read the provider already counted inside it. Providers disagree on that: Anthropic's input_tokens excludes the cache read, while the OpenAI Responses, Chat Completions and Gemini prompt totals include it. llmshim resolves the disagreement at the transport boundary so one counter means one thing.

cost_source distinguishes three accounting meanings:

• cost_source: "provider" — the provider reported what it charged for this generation and cost_usd is that figure, not an estimate. OpenRouter does this (as usage.cost), unconditionally and on streams too. It needs no catalog entry, so it answers for aggregator slugs the catalog has never heard of.
• cost_source: "provider_floor" — a partial provider bill is the highest known lower bound, but no exact terminal provider bill survived. cost_usd is not a final invoice in this case.
• cost_source: "catalog" — computed here from catalog prices, as below.

A terminal provider bill always wins. A partial provider bill remains a floor that a later catalog estimate cannot undercut, without being mislabeled as an exact final invoice.

The catalog estimate charges standard token classes: uncached input, output, cache reads and cache writes each at their own catalog rate, so a cached prompt is never billed twice. null means the catalog carries no price for the model — it never means free. A model priced for input but not for the cache reads a response actually used charges that class at the model's highest published rate, yielding a conservative estimate. Context tiers include cached input when choosing the rate and apply to the entire request. For native Grok 4.7, rates double above 200,000 input tokens. Provider tool fees, regional premiums, and priority endpoint surcharges are not included in these catalog estimates.

ProviderRequest::can_continue_from compares endpoint, credential headers, settings and the full prior input prefix. include, store, reasoning, tool schemas and other settings participate. The library sends full stateless requests; this helper does not create a continuation cache or enable provider conversation storage.

Images and vision

Images travel inside a message's content array. llmshim translates recognized image blocks into the selected provider's native representation.

Availability: Rust/proxy/clients: message content blocks · CLI: file or clipboard attachment

Use a portable image block

The most convenient input is the OpenAI Chat Completions image_url block:

{
  "role": "user",
  "content": [
    {"type": "text", "text": "Describe this image."},
    {
      "type": "image_url",
      "image_url": {
        "url": "data:image/png;base64,iVBORw0KGgo..."
      }
    }
  ]
}

A data URI carries both the media type and base64 bytes. It is the portable choice when the same request may target OpenAI, Anthropic, Gemini, or xAI.

The current translators recognize these input block forms:

When Gemini is the target, llmshim emits Gemini's native inline_data part for base64 image bytes. A raw Gemini inline_data part is not currently recognized as a portable input block, so use one of the forms above at the llmshim boundary.

Base64 versus remote URLs

Both data URIs and plain remote URLs are accepted inside image_url and input_image blocks:

{
  "type": "image_url",
  "image_url": {"url": "https://example.com/photo.jpg"}
}

OpenAI, xAI, and Anthropic receive a provider-native URL image. llmshim does not download that URL itself.

Gemini limitation: the current Gemini adapter cannot send a remote image URL as inline image data. It replaces the image block with a text part such as [Image: https://example.com/photo.jpg]. The model receives the URL as text, not the image. Use a base64 data URI when targeting Gemini.

Send through each surface

For Rust, put the content array directly in the top-level messages value. For the proxy and language clients, use the same content array in the compact request's messages field. Image controls do not belong in config or provider_config.

The core does not read local paths. Applications must read and encode local files themselves before building the request.

The CLI provides that convenience:

/image ./diagram.png

It reads the file, builds a base64 data URI, and attaches it to the next user message. /paste attempts to attach an image from the clipboard on supported desktop environments. You can also include a readable image path directly in the text entered at the CLI prompt.

Generate images from Rust

Image generation is a separate non-streaming operation:

#![allow(unused)]
fn main() {
async fn example(router: &llmshim::router::Router) -> llmshim::error::Result<()> {
let response = llmshim::images(router, &serde_json::json!({
    "model": "openai/gpt-image-1",
    "prompt": "A watercolor of a lighthouse",
    "n": 1,
    "output_format": "png"
})).await?;
for image in response.images {
    // Choose the destination using image.media_type.
    std::fs::write("lighthouse.png", image.bytes).unwrap();
}
Ok(()) }
}

ShimClient::images(provider, model, request) uses an already resolved provider. The router entry point accepts aliases and named routes. OpenAI GPT Image models use /images/generations; supported controls are n, size, quality, background, output_format, output_compression, moderation, and user. Results contain bytes, media type, and an optional revised prompt. URL-only, empty, filtered, and malformed outputs fail; llmshim never downloads output URLs.

The Gemini adapter uses models/<model>:generateContent with a user text part and generationConfig.responseModalities: ["TEXT", "IMAGE"]. Supported model IDs are gemini-2.5-flash-image, gemini-3.1-flash-lite-image, gemini-3.1-flash-image, and gemini-3-pro-image, as listed in Google's image generation guide. Use x-gemini.imageConfig for native aspectRatio and imageSize controls; model-specific restrictions are enforced by the provider. Omit n or set it to 1: one candidate may contain multiple image parts, and Gemini does not promise an exact image count. Other counts fail locally. Retired Imagen models are not supported.

Images come from non-thought inlineData parts across the returned candidates. Their base64 data and mimeType become decoded bytes and media type. Draft thought images are excluded; text-only or blocked output and invalid image parts fail with status 502. A supplied non-STOP finish reason also fails. Gemini's usageMetadata is preserved as native usage, including token-modality details. No text-model price is substituted for image-output usage.

usage.cost_usd prefers a reported provider bill. Otherwise, pricing uses cost::for_target(provider, model) and reports cost_source: "catalog". Complete, uncached text-input and image-output modality counters are required. Mixed text/image output, image inputs, cached usage, missing or inconsistent counters, and uncatalogued models have null cost and cost_source: "unknown". The catalog's single output rate cannot price mixed output safely. Generation currently has no proxy/CLI endpoint, editing, streaming, or dispatch-policy callback surface.

Embeddings

Embeddings return one vector per input text, in input order, with the model identity and the tokens the provider billed.

Availability: Rust: llmshim::embeddings · proxy/clients: not exposed yet

Embed a batch from Rust

#![allow(unused)]
fn main() {
use llmshim::embeddings::EmbeddingRequest;

let router = llmshim::router::Router::from_env();
let request = EmbeddingRequest::new(
    "openai/text-embedding-3-small",
    vec!["the cat sat on the mat".to_string(), "a dog barked".to_string()],
);

let batch = llmshim::embeddings(&router, &request).await.unwrap();
assert_eq!(batch.vectors.len(), 2);
println!("{} {} {:?}", batch.provider, batch.model, batch.revision);
}

The model id is the same provider/model grammar a completion uses, so a named route, an alias and provider prefix inference all work the same way. batch.provider and batch.model name what produced the vectors; batch.revision is the model string the provider's own response reported — the pinned snapshot the request landed on — and is None when the provider's response names no model.

batch.usage.input_tokens is the provider's own count, and batch.usage.cost_usd / cost_source follow the same rules as a completion: the provider's bill where there is one, else the catalog's price, else null for unknown — never 0 for "we did not find out".

Narrower vectors

EmbeddingRequest::with_dimensions asks for a smaller vector on models that truncate:

#![allow(unused)]
fn main() {
let request = EmbeddingRequest::new("gemini/gemini-embedding-001", texts)
    .with_dimensions(512);
}

A width wider than the model's own is refused before dispatch: no provider widens, and a silent failure upstream costs a round trip for something llmshim can see locally.

Which providers embed

A refusal is local and names the provider, e.g. anthropic has no embeddings API. llmshim never forwards the batch to another vendor's model: a vector from a model the caller did not name is the wrong answer with no signal that a substitution happened.

Any server that speaks OpenAI's /v1/embeddings — a local Ollama, LM Studio, vLLM or SGLang instance — is reached through the self-hosted provider already configured for it by base URL. Since that server's own limits are its launch configuration, llmshim passes the batch through and surfaces the server's error.

The catalog decides which models embed

A model the catalog carries but does not mark as an embedding model is refused before anything is sent, naming the model:

openai/gpt-6-astra is not marked as an embedding model in the catalog;
assert `embedding = true` under [models."openai/gpt-6-astra"] to embed with it

That is deliberate. An OpenAI chat model sent to /v1/embeddings fails upstream; the catalog assertion turns a wasted round trip into a local answer, and the state is tri-state — a model the catalog has never heard of (a self-hosted embedding model, an OpenRouter slug) is sent, because the provider is the authority on what it serves.

The assertion comes from the layer that knew:

# ~/.llmshim/config.toml
[models."vllm/bge-m3"]
embedding = true
dimensions = 1024

Builtin assertions cover openai/text-embedding-3-small, openai/text-embedding-3-large, openai/text-embedding-ada-002 and gemini/gemini-embedding-001. A local entry outranks them, so a private endpoint can assert its own model and width.

Batching

A batch over the provider's bounds is refused rather than split: a second HTTP request is a second charge and a different rate-limit footprint from the one the caller asked for. OpenAI's documented 2048-input cap and a 1 MiB text ceiling apply there; every other provider gets llmshim's own wider ceiling (8192 texts, 4 MiB), which is a guard against a runaway batch rather than a claim about any server's real limit. Check Provider::embedding_bounds on the provider you are addressing if you need the number.

Split the batch yourself and issue one call per chunk when you have more texts than the bound — the vectors come back in input order, so concatenating chunks preserves the mapping to your texts.

Audio

Speech synthesis is available from Rust through llmshim::speech(router, request) and ShimClient::speech(provider, model, request). Router aliases and named routes work as they do for images. Other providers refuse locally.

#![allow(unused)]
fn main() {
async fn example(router: &llmshim::router::Router) -> llmshim::error::Result<()> {
let response = llmshim::speech(router, &serde_json::json!({
    "model": "openai/tts-1",
    "input": "Hello there",
    "voice": "alloy",
    "response_format": "mp3"
})).await?;
std::fs::write("hello.mp3", response.bytes).unwrap();
Ok(()) }
}

Speech JSON fields:

Other speech fields are ignored. OpenAI tts-1 and tts-1-hd use /audio/speech. Inputs contain at most 4096 Unicode characters. Voice names must contain 1 to 64 ASCII letters. The provider decides which names it supports, so new voices do not need a crate release; an unknown voice returns the provider's error. Formats are mp3 (default), wav, flac, opus, aac and pcm. Optional speed ranges from 0.25 to 4. Streaming is refused. Output carries bytes and media type; applications own playback. Empty bodies and unexpected content types fail. The shared transport bounds decoded output and retains its retry, deadline, redirect refusal and circuit-breaker behavior.

The endpoint returns binary audio without usage counters or a provider bill. usage.input_characters counts submitted Unicode scalar values, including whitespace; cost_usd estimates the charge at $15 or $30 per million characters, with cost_source: "catalog". These dedicated catalog rows live in llmshim_catalog::audio::SPEECH_MODELS, separate from token pricing. See OpenAI's TTS-1 price and TTS-1 HD price and speech reference. There is no proxy, CLI, streaming or dispatch-policy callback surface.

Transcription

llmshim::transcription(router, &request) and ShimClient::transcription(provider, model, &request) accept caller-owned bytes. The crate never reads a filename as a path. Model aliases work as they do for embeddings; route settings are not applied to this typed request.

#![allow(unused)]
fn main() {
async fn example(router: &llmshim::router::Router, bytes: Vec<u8>) -> llmshim::error::Result<()> {
let mut request = llmshim::audio::TranscriptionRequest::new(
    "openai/gpt-4o-mini-transcribe", bytes, "recording.wav", "audio/wav"
);
request.language = Some("en".into());
let response = llmshim::transcription(router, &request).await?;
println!("{}", response.text);
Ok(()) }
}

OpenAI whisper-1, gpt-4o-transcribe and gpt-4o-mini-transcribe use /audio/transcriptions. Uploads contain 1 to 25,000,000 bytes (a conservative decimal interpretation of the published 25 MB limit). The filename is a basename of at most 255 bytes. A MIME type of at most 128 bytes is checked by reqwest's MIME parser. Optional language is a two-letter lowercase ISO-639-1 code; prompt is bounded to 16 KiB; temperature ranges from 0 to 1.

The multipart file part retains the supplied bytes, filename and MIME type; other parts are model, response_format and the supplied optional controls. Reqwest builds the form, with a fresh boundary and replayable bytes per retry. The existing sender handles status errors, retries, header/attempt deadlines and redirect refusal. JSON and plain-text responses use the shared bounded body reader with idle/total deadlines and circuit-breaker classification.

response_format defaults to json. For OpenAI, only json and text are exposed here; the GPT transcription models support only json, while Whisper also supports text. OpenRouter accepts json and verbose_json. Plain text must have a text/plain media type and valid UTF-8. JSON must contain a text string. An empty string is valid for silence.

TranscriptionResponse.usage preserves native usage. A valid provider usage.cost wins (cost_source: "provider"). Otherwise the dedicated catalog transcription rows price reported tokens (cost_source: "catalog"). The audio-input/text-input/text-output catalog rates per million tokens are $6/$2.50/$10 for GPT-4o Transcribe and $3/$1.25/$5 for Mini Transcribe. Audio and text input counts are priced separately. Native type: "tokens", integer input/output/total counts and consistent audio/text input details are required; missing or inconsistent accounting stays null with cost_source: "unknown". Chat-model rates are never used to estimate audio costs.

The OpenAI Cookbook explicitly separates GPT-4o Transcribe audio and text prices. The model pages read on 2026-10-01 show a single input table instead. Mini's $3 audio-input rate retains the previously published price; that separate rate could not be reconfirmed from the current model page.

Whisper costs $0.006 per minute. Its JSON/text response does not supply duration, so cost is unknown by default. An optional duration_seconds supplied by the caller enables a catalog estimate, recorded separately as usage.caller_duration_seconds; it is neither provider-measured usage nor sent upstream. No duration is inferred from byte size or transcript length.

Sources: transcription reference, file transcription limits, GPT-4o Transcribe, Mini Transcribe and Whisper.

OpenRouter transcribes through its OpenAI-compatible /audio/transcriptions with the same multipart upload and bounds. Use an OpenRouter slug, such as openrouter/openai/whisper-1 or openrouter/openai/whisper-large-v3. Slugs are not checked locally; OpenRouter refuses a model it does not serve. Its schema has no prompt field and answers only json or verbose_json, so a prompt and text are refused locally. An error inside a 200 answer becomes an error with OpenRouter's own code and message, as on the chat path. Usage keeps OpenRouter's seconds and token counts. The reported usage.cost is the bill (cost_source: "provider"); without it the cost is null with cost_source: "unknown", and no catalog estimate is made. See OpenRouter's speech-to-text guide and transcription reference.

Choosing a transcription model

llmshim::transcription_models(router) lists the transcription models of every provider configured on the router, so a caller can offer a person a choice. Each entry has the routed id to pass as the request model, the provider key, the provider's own model id, and a label with the model's name and its provider. The same model through two providers has two entries:

#![allow(unused)]
fn main() {
async fn example(router: &llmshim::router::Router, bytes: Vec<u8>) -> llmshim::error::Result<()> {
for model in llmshim::transcription_models(router) {
    println!("{}  {}", model.id, model.label);
}
let request = llmshim::audio::TranscriptionRequest::new(
    "openrouter/openai/whisper-1", bytes, "recording.wav", "audio/wav"
);
println!("{}", llmshim::transcription(router, &request).await?.text);
Ok(()) }
}

The list is ordered by provider key, then by each provider's own order. An id carries the key the provider is registered under, so it routes back to that provider. Each provider owns its list through Provider::transcription_models, which is empty by default:

• OpenAI lists its three models, the same catalog rows that gate its requests.
• OpenRouter lists the models in its public speech-to-text listing, vendored in the crate as read on 2026-10-07. The list does not gate requests: a slug that is not on it is still sent, and OpenRouter decides.
• Other providers list nothing, as they refuse transcription.

Other providers refuse locally. Transcription has no proxy/CLI, streaming, translation, timestamp or dispatch-policy callback surface.

Gemini transcription design

Gemini audio input requires generateContent JSON with base64 inlineData, whereas the implemented transcription preparation returns multipart fields and one replayable file. Sending that form to Gemini is not a valid protocol, and forwarding audio as text would silently lose the input. This increment keeps Gemini's local refusal.

A future preparation variant must carry JSON as well as multipart, while both continue to build each attempt inside the shared sender. Encode inline audio only after admission, use a transcription instruction plus caller prompt and language, require a successful terminal candidate, and collect only non-thought text. Keep that response's usageMetadata with audio/text modality counters; add published Gemini audio rates to the catalog before estimating a charge. Reusing text-input rates or inferring duration from bytes is rejected.

Tests must inspect inline bytes and MIME type, pair successful terminal text with blocked/truncated/thought-only replies, exercise identical transport bounds and deadlines, and pair priced complete modality counters with mixed, cached, missing and inconsistent counters. This needs a distinct native preparation and accounting implementation; the multipart-only seam does not serve it cleanly.

Reasoning controls

llmshim exposes one reasoning vocabulary across providers. You state the desired depth; the selected provider adapter maps it to what that model accepts.

Availability: Rust: top-level fields · CLI: high by default · Proxy/clients: fields under config

The two knobs

In a Rust request, both are top-level fields:

{
  "model": "anthropic/claude-sonnet-5-5",
  "messages": [{"role": "user", "content": "Solve this carefully."}],
  "reasoning_effort": "high",
  "reasoning_mode": "pro"
}

In the proxy contract, put them under config:

{
  "model": "anthropic/claude-sonnet-5-5",
  "messages": [{"role": "user", "content": "Solve this carefully."}],
  "config": {
    "reasoning_effort": "high",
    "reasoning_mode": "pro"
  }
}

Preserve reasoning for replay

Responses carry an ordered message.reasoning array. Every block has kind (text, redacted, or encrypted) and origin: provider, upstream model, coarse model family, wire format, receipt time, and an optional issuer binding. Text blocks carry text; opaque blocks carry data. Signatures, Responses item ids, original structured payloads and container identity are retained when present. Preserve the complete assistant message when building the next request. The proxy exposes the same array inside message; its top-level reasoning string remains a display-only convenience.

One replay policy applies to all adapters and fallback attempts: the target family and wire must match, and encrypted blocks also require the same provider and account binding. Unknown families and missing origins fail closed. Signed Anthropic blocks precede text and tools in their original order. Responses requests use store:false, include encrypted reasoning, and replay those items locally. Native overrides cannot turn storage on or bypass reasoning provenance. The outbound projection removes both previous_response_id and conversation, including references supplied through native overrides. Send complete history in the request: a provider-side conversation can otherwise prepend stored items, as described in the Responses API reference. OpenAI and xAI use this stateless Responses path; see xAI's encrypted reasoning contract.

API-key account bindings hash the endpoint and credential; they conservatively stop matching after key rotation. OAuth bindings use the account identifier captured from the actual request. No credential or raw account identifier is emitted. Rejected replay is silent and increments a bounded-reason counter, available through llmshim::reasoning::replay_counters().

Gemini tool-call signatures are objects containing data and origin. Keep that entire object on the tool call. Legacy reasoning_content, reasoning_signature, redacted_reasoning_content, and string tool signatures are read for one migration release, but untracked data is dropped. An explicit reasoning_origin can accompany legacy message fields during migration; new writers emit only the block array. Models absent from the catalog need a local family assertion before their reasoning can be replayed.

Inbound reasoning history has a separate whole-request limit of 16 MiB of estimated derived metadata and 16,384 reasoning records. A record is one reasoning block or tool signature, rather than every JSON descendant. The limit is checked across all messages before legacy origins or payloads are repeated; an excessive request fails with a fixed 400 and no provider call. Accepted legacy and normalized blocks remain byte-for-byte intact. The public Rust filter_reasoning_for_target helper returns a Result for the same reason: propagate a budget error instead of treating it as an ordinary policy drop.

Unary reasoning blocks, tool signatures, wire identities, and their provenance share a 16 MiB estimated-owned-memory and 4,096-entry derived metadata limit. The limit is checked before model, provider, account, payload, or signature copies are created. A response that exceeds it fails as an upstream response; llmshim never returns a truncated or partially normalized assistant message. The public Rust reasoning::capture_response helper therefore returns a Result; callers must discard the response and propagate an error. Successful capture preserves every accepted origin string byte-for-byte.

Streaming reasoning uses the same blocks with a part index and an optional replace:true flag for completed item snapshots. ReasoningAccumulator assembles fragments and removes stream framing before storage:

#![allow(unused)]
fn main() {
let mut reasoning = llmshim::reasoning::ReasoningAccumulator::default();
// For each normalized chunk:
reasoning.push(&chunk["choices"][0]["delta"])?;
// When the stream has completed successfully:
assistant["reasoning"] = serde_json::json!(reasoning.blocks());
}

push returns a Result because reasoning text, signatures, encrypted data, and block identities share the per-response retained-state budget. Propagate that error; an over-budget accumulator never returns an apparently complete partial result. Each native stream frame also applies the derived metadata limit before eagerly normalizing multiple completed reasoning blocks; the retained-state limit remains the separate bound across frames.

Use reasoning_text(&message_or_delta) for display without inspecting opaque data. A provider may expose only a summary or no readable reasoning at all.

Portable intent or native control

Choose per request:

1. Use the unified controls. llmshim maps the requested effort and mode to the selected model family, clamping to the nearest accepted tier when the model has a smaller vocabulary.
2. Use the provider's native dialect. Pass an exact native object through x-openai, x-anthropic, or x-gemini. Native configuration wins over the unified controls.

flowchart LR
    U[Unified effort + mode] --> P{Native override present?}
    P -- Yes --> N[x-provider native config]
    P -- No --> F[Model-family mapping]
    F --> C[Clamp to supported tier]
    C --> W[Provider-native reasoning control]

See Native provider controls for passthrough examples.

Effort mapping tables

These mappings follow provider documentation and live checks. A bold value is a clamp rather than a direct name-for-name mapping.

OpenAI Responses API

OpenAI receives reasoning.effort. Legacy minimal input clamps to low on the three advertised GPT-6 models. See the OpenAI model catalog.

Anthropic Messages API

Adaptive models use thinking: {type} plus output_config: {effort}:

On models that can disable thinking, reasoning_effort: "none" maps to disabled thinking. Fable 5.1, Opus 5.5, and Sonnet 5.5 always use adaptive thinking, so none maps to low. Explicit native disabled/manual thinking is rejected locally for those models, as is forced tool choice. Opus 5.5 thinking blocks replay to Fable 5.1, while older Opus models cannot consume them. Sonnet 5.5 blocks replay only to Sonnet 5.5, and Sonnet 5.5 reads thinking from Sonnet 5, Opus 4.8, Haiku 4.5, and earlier, but not from Opus 5, Opus 5.5, or a Fable/Mythos model. Opus 5.5 and Sonnet 5.5 omit temperature, top_p, and top_k, as do Fable and Opus 5. See the Opus 5.5 migration guide and the Sonnet 5.5 model page.

Fable behavior and its five effort levels follow Anthropic's migration guide and Models API. The low-effort and rejected-parameter paths are also checked against the live API in this repository.

Haiku 4.5 uses enabled thinking with a token budget scaled from max_tokens and floored at 1024:

Gemini

Gemini uses the four-rung generationConfig.thinkingConfig.thinkingLevel enum:

Gemini 3.8 Flash rejects minimal and thinkingBudget: 0, so none clamps to low. Gemini 3.5 Flash-Lite accepts minimal and can disable thinking. Native controls remain available through x-gemini.thinkingConfig.

ChatGPT subscription

GPT-6 Astra preserves low, medium, high, xhigh, and max; none and minimal clamp to low because Astra cannot disable reasoning. GPT-6 Sol and Luna accept none and max; minimal clamps to low. The shared Responses translator applies these mappings through both ChatGPT subscription and API-key OpenAI. See the official OpenAI model catalog. Native x-chatgpt.reasoning overrides the unified mapping.

xAI Responses API

xAI receives the nested native shape reasoning: {effort}:

Grok 4.7 cannot disable reasoning, so none clamps to low.

Grok 4.7 always returns encrypted reasoning on Responses. llmshim preserves the opaque payload and its provenance in message.reasoning, requests store: false, and replays compatible payloads unchanged on subsequent turns. Persist the full assistant message, including tool wire IDs and reasoning; storing only its text loses this context. Decoding does not modify origin.model or determine routing.

The adapter translates named tool selection to the Responses shape {"type":"function","name":"lookup"}. Callers may continue to send the Chat Completions shape with function.name.

Mode mapping: reasoning_mode: "pro"

Rules that hold across providers:

• standard is the default and is not sent on the wire.
• An explicit effort of none wins over pro; a request to turn reasoning off is not bumped back on, except where the model itself cannot disable it.
• pro without an effort lets OpenAI native-mode models select their own effort. Other models behave as a default medium bumped to high.

The native GPT-6 mode follows the OpenAI reasoning guide.

Precedence

1. Native passthrough under x-openai, x-anthropic, or x-gemini wins and is not translated into another dialect. Anthropic also accepts top-level thinking and output_config in the Rust contract.
2. Otherwise, unified reasoning_effort and reasoning_mode are mapped and clamped according to the tables above.
3. With neither, the provider/model default applies. The provider decides whether and how much to reason unless an explicit setting is supplied.

The adapter mappings are pinned by provider unit tests. The tables above cover the advertised models; compatibility mappings for explicit older IDs remain in the adapters and tests. Provider capabilities can change, so the implementation and these tables must move together.

Signature observations

The default signature-introspection feature makes a best-effort read of an undocumented Anthropic signature header. Its public decoder, providers::anthropic_signature::serving_model_from_signature, returns Option; unknown formats (including headers without a model), malformed data, and the feature-disabled stub return None. Tests use synthetic protobufs only.

This never changes origin.model, a stored message, replay eligibility, routing, fallback, or cache decisions. The decoded value is unverified metadata, not authentication or proof of which model ran. Dotted/dashed spellings and dated snapshots compare as the same model line; an unrecognized internal codename may be reported as a mismatch. Logs record gateway_integrity as ok, mismatch (with requested/served names), or unknown. Unknowns produce no warning.

A mismatch increments providers::anthropic_signature::mismatch_count() and adds x-llmshim-served-model outside the assistant message. Streaming adds it to the terminal canonical chunk or the proxy's done event. Multiple observed names are comma separated. Disable inspection with --no-default-features; model requests and stored message contents remain identical, while observation fields and metrics can differ.

Anthropic effort and budget selection now comes from catalog reasoning_options. An effort entry selects adaptive thinking and clamps to an advertised tier; a budget entry selects enabled thinking and respects its minimum, maximum, and max_tokens - 1. An impossible budget fails before HTTP dispatch. Verified builtin entries preserve the established mappings; a bounded table of historical model lines supplies missing metadata. Unknown future names are not guessed.

Fallback chains

A fallback chain keeps the conversation fixed while trying an ordered list of model addresses. Retry the route, then change the route.

Availability: Rust: completion_with_fallback · CLI: not exposed · Proxy/clients: non-streaming fallback field

Fallback is useful for temporary provider failures. It is not model selection: llmshim never chooses or reorders the models for you.

Rust

The Rust FallbackConfig list contains the complete order, including the primary model first:

#![allow(unused)]
fn main() {
use llmshim::{completion_with_fallback, FallbackConfig};
use serde_json::json;

let request = json!({
    "model": "anthropic/claude-opus-5-5",
    "messages": [{"role": "user", "content": "Hello"}],
    "max_tokens": 128
});

let fallback = FallbackConfig::new(vec![
    "anthropic/claude-opus-5-5".into(),
    "openai/gpt-6-sol".into(),
    "gemini/gemini-3.8-flash".into(),
]);

let response = completion_with_fallback(
    &router,
    &request,
    &fallback,
    None,
).await?;
}

When FallbackConfig.models is non-empty, that list supplies the primary and all fallback addresses. The function rewrites the request's model for each attempt.

The builder also allows callers to change the fallback layer's retries per model and its initial backoff:

#![allow(unused)]
fn main() {
use std::time::Duration;

let fallback = FallbackConfig::new(models)
    .max_retries(1)
    .initial_backoff(Duration::from_millis(250));
}

Proxy and clients

The proxy keeps the primary in model; fallback contains only the addresses to try afterward:

{
  "model": "anthropic/claude-opus-5-5",
  "messages": [{"role": "user", "content": "Hello"}],
  "config": {"max_tokens": 128},
  "fallback": [
    "openai/gpt-6-sol",
    "gemini/gemini-3.8-flash"
  ]
}

The proxy prepends model to the supplied fallback array before calling the Rust fallback API. Language clients expose the same request field; some offer a convenience argument for it.

What triggers retry and failover

The default fallback statuses are:

429  500  502  503  529

Transport errors are retryable too. For an eligible failure, the fallback layer retries the current model with exponential backoff, doubling the delay between attempts. If the model still fails, llmshim advances to the next address. A success returns immediately. If every address fails, Rust returns ShimError::AllFailed with the collected errors; the proxy returns an all_failed error response.

Each fallback model must resolve to a provider registered on the Router. A chain can cross providers, but only when their keys are configured. A chain entry may also be a named route.

Open circuits are skipped, not retried

A chain also consults provider health, before every attempt rather than once per entry — the attempt that opens a circuit is usually the chain's own. When a provider's circuit is open, because of too many 5xx/transport failures in the window, the chain abandons that entry, records circuit open for provider … among the collected errors, and moves to the next address instead of spending the rest of its retry budget on a target it already knows is dead. A 429 never opens a circuit; that is rate limiting, handled by backoff. See Scaling and rate limits for the knobs.

Fallback is non-streaming only

There is no streaming fallback API in the Rust crate. The proxy's streaming path calls llmshim::stream directly and does not inspect fallback.

That means both of these ignore the fallback array:

• POST /v1/chat/stream;
• POST /v1/chat with "stream": true.

Do not send fallback on a streaming request expecting failover. Implement a new stream attempt in the caller only if your application can safely decide what to do with content already emitted by the failed model.

Native provider controls

Use native controls when the portable request cannot express a provider feature precisely enough. This is an intentional trade: exact provider behavior in exchange for portability.

Availability: Rust: top-level x-* key · CLI: not exposed · Proxy/clients: provider_config["x-*"]

The built-in namespaces are:

There is no x-xai namespace. llmshim's unified reasoning control already emits xAI's supported native reasoning: {effort} shape, including its model-family clamps.

OpenAI

x-openai fields are copied into the OpenAI Responses request body. This gives access to the full native reasoning object, including effort, mode, summary, and context when accepted by the selected model.

{
  "model": "openai/gpt-6-sol",
  "messages": [{"role": "user", "content": "Analyze this carefully."}],
  "x-openai": {
    "reasoning": {
      "effort": "high",
      "mode": "pro",
      "summary": "auto"
    }
  }
}

The object is native OpenAI configuration. llmshim does not clamp it, so the selected model must accept the values.

ChatGPT subscription

Use x-chatgpt in an engine request, or provider_config["x-chatgpt"] through the proxy. Supported fields are input, instructions, include, tools, tool_choice, reasoning, previous_response_id, and truncation. The routed model, stream: true, and store: false are enforced after native overrides; unsupported fields, including all token limits and metadata, are removed. reasoning.encrypted_content is always included in the upstream request. Returned chat history uses llmshim's existing normalized messages; it does not preserve opaque encrypted reasoning items. x-openai does not apply to this provider.

Anthropic

Most x-anthropic fields become Anthropic Messages body fields. Two special keys control headers instead:

• disable_1m_context disables llmshim's automatic 1M-context beta header;
• extra_betas appends and de-duplicates caller-supplied anthropic-beta tokens.

{
  "model": "anthropic/claude-sonnet-5-5",
  "messages": [{"role": "user", "content": "Analyze this carefully."}],
  "x-anthropic": {
    "thinking": {"type": "adaptive"},
    "output_config": {"effort": "high"},
    "disable_1m_context": true,
    "extra_betas": ["extended-cache-ttl-2025-04-11"]
  }
}

thinking and output_config are sent in the request body. The two control keys are consumed by llmshim while building Anthropic headers and are not sent as body fields.

The Rust contract also accepts top-level thinking and output_config, but the namespace keeps provider-native fields visibly grouped.

Gemini

x-gemini.thinkingConfig replaces the unified Gemini thinking configuration. It accepts Gemini's native thinking configuration:

{
  "model": "gemini/gemini-3.8-flash",
  "messages": [{"role": "user", "content": "Analyze this carefully."}],
  "x-gemini": {
    "thinkingConfig": {
      "thinkingLevel": "high",
      "includeThoughts": true
    }
  }
}

Other keys inside x-gemini are copied to the Gemini request body. As with all native controls, the provider validates their shapes and values.

Where the namespace goes

In the Rust serde_json::Value, use the namespace as a top-level key, as in the examples above.

The proxy reserves provider_config for fields that are merged into that core request. Wrap the same namespace inside it:

{
  "model": "openai/gpt-6-sol",
  "messages": [{"role": "user", "content": "Analyze this carefully."}],
  "provider_config": {
    "x-openai": {
      "reasoning": {
        "effort": "high",
        "mode": "pro",
        "summary": "auto"
      }
    }
  }
}

Language clients send the same object through their provider_config field. The CLI chat workflow does not expose native request configuration.

Precedence and portability

An explicit native reasoning object takes precedence over reasoning_effort/reasoning_mode. llmshim does not translate the object into another provider's dialect. Anthropic header controls are applied as headers; the other native fields are copied to their documented native destination.

If you change the model address to another provider, remove or replace the old provider namespace. Keeping x-openai in an Anthropic request does not recreate the OpenAI behavior. See Portable core, native edges for the underlying contract.

HTTP API

The proxy wraps llmshim's translation engine in a compact JSON API. It has a compact chat contract plus native API endpoints, with JSON and SSE responses.

Availability: HTTP JSON: non-streaming chat, models, health · HTTP SSE: streaming chat

Endpoints

The canonical machine-readable contract is api/openapi.yaml.

Chat request

Both chat endpoints accept the same body:

{
  "model": "anthropic/claude-sonnet-5-5",
  "messages": [{"role": "user", "content": "Explain ownership briefly."}],
  "stream": false,
  "config": {
    "max_tokens": 300,
    "temperature": 0.2,
    "reasoning_effort": "medium"
  },
  "provider_config": {},
  "fallback": ["openai/gpt-6-luna"]
}

config accepts exactly these fields:

Adapters use only controls supported by the selected provider. Reasoning is mapped and clamped by model family; see Reasoning controls.

Messages require role; content defaults to null and may be text or content blocks. Tool turns may also carry tool_calls or tool_call_id. Assistant messages carry reasoning[] blocks with provenance. Preserve the complete array and any tool-call thought_signature object for replay. Legacy reasoning text is accepted only as migration input; untracked blocks are dropped.

provider_config is merged as top-level engine fields. For example, tools go at provider_config.tools, while an OpenAI-native override goes at provider_config["x-openai"]. See the request field map. The typed model and messages fields remain authoritative: passthrough configuration cannot replace them through the selected provider's native model or main messages/input/contents container. Native system or instruction controls remain supported and are estimated when active. Validation happens after named-route defaults and aliases are resolved and covers every resolvable explicit fallback before the primary request runs. Another provider's namespace is inert only when no primary, fallback, or route can select that provider.

Fallback first retries an eligible failure on the current route, then moves through the listed routes. It is ignored by both streaming paths. See Fallback chains.

Non-streaming response

POST /v1/chat returns a compact response when stream is false:

{
  "id": "msg_123",
  "model": "claude-sonnet-5-5",
  "provider": "anthropic",
  "message": {
    "role": "assistant",
    "content": "Ownership gives each value one owner."
  },
  "usage": {
    "input_tokens": 14,
    "output_tokens": 9,
    "reasoning_tokens": 4,
    "total_tokens": 27
  },
  "latency_ms": 612
}

created_at is the original completion time as a Unix timestamp. If the provider omits an ID, the server assigns one; gateway idempotency replay preserves both.

message.tool_calls appears when the model requests tools. reasoning appears when the provider returns reasoning text. reasoning_tokens is omitted when zero; the other usage fields are always present.

Normalized token accounting

Every compact usage object and typed SSE usage event contains uncached_input_tokens and x-llmshim-usage. Native Chat Completions and Messages place the same extension inside usage; Gemini places it inside usageMetadata. It survives the final usage frame on every streaming facade.

The extension always contains five unsigned counts: uncached_input_tokens, cache_read_tokens, cache_write_tokens, output_tokens, and reasoning_tokens, including zero values. Uncached input excludes both cache reads and cache writes. Output and reasoning preserve the upstream counts; reasoning may be included in output, so do not add them together. Zero means the provider reported no count for that class. Streaming counts are cumulative snapshots, not increments.

Native field conventions remain native: Messages input_tokens excludes cache reads and writes, which travel as cache_read_input_tokens and cache_creation_input_tokens. Chat prompt_tokens and Gemini promptTokenCount include all input classes. Chat also exposes cache reads in prompt_tokens_details.cached_tokens and reasoning in completion_tokens_details.reasoning_tokens; Gemini uses cachedContentTokenCount and thoughtsTokenCount. Compact input_tokens retains the engine's original prompt convention for compatibility; use the extension when comparing providers.

Idempotency replay retention

For completed unary gateway requests, an idempotency key enables a process-local replay only when its tenant, credential, route, and canonical request match. This is best-effort replay: the local cache has shared limits for entry count, retained response footprint, and JSON complexity across the current scoped API and its deprecated generic compatibility API. An entry that exceeds those limits, or a new entry when live retained entries already fill them, is not stored. A TTL is an upper bound on replay availability, and some responses are never retained. Clients must therefore tolerate a fresh execution and must not treat this as an exactly-once guarantee.

Distributed requests also hold a finite origin lease. Dropping a unary request future or streaming response body requests durable cancellation; process loss is detected when the lease expires. Cancellation can stop queued or locally active work, but it cannot prove that an external provider did not accept a send that was already in flight. Known usage and uncertain spend liability are retained.

These local limits use a conservative in-process footprint estimate, not an RSS measurement. In a distributed deployment, completed responses remain in the bounded lifecycle terminal pool and the idempotency cache retains a bounded credential/request-scoped pointer to that terminal. Pointer values have native Redis expiry and share limits of 100,000 entries and 64 MiB; their TTL is capped at one day and terminal extension uses the same effective TTL without shortening an existing longer terminal retention. Concurrent first requests that both miss the cache are not coalesced and can both execute. Each job is independently admitted and charged, so callers must not use an idempotency key as a single-flight or exactly-once guarantee.

Streaming response

POST /v1/chat/stream always streams. POST /v1/chat does the same when the body contains "stream": true.

Each Server-Sent Event has an SSE event: name and JSON data: whose type matches that name:

event: reasoning
data: {"type":"reasoning","text":"..."}

event: content
data: {"type":"content","text":"Ownership gives each value one owner."}

event: tool_call
data: {"type":"tool_call","id":"call_1","name":"lookup","arguments":"{\"id\":7}"}

event: usage
data: {"type":"usage","input_tokens":14,"output_tokens":9,"total_tokens":23}

event: done
data: {"type":"done"}

Admission failures happen before SSE begins and return normal HTTP 429 or 503 responses. A failure after the stream starts is an error event. For consumption patterns, see Streaming.

Models and health

GET /v1/models returns the registry entries for configured providers, in two shapes from one body:

{
  "object": "list",
  "data": [
    {"id": "openai/gpt-6-luna", "object": "model", "created": 1790035200, "owned_by": "openai"}
  ],
  "models": [
    {"id": "openai/gpt-6-luna", "provider": "openai", "name": "gpt-6-luna"}
  ]
}

object + data is the OpenAI list envelope, so an OpenAI SDK's client.models.list() works against llmshim unmodified. models is llmshim's own shape and is unchanged. Both audiences issue the same GET /v1/models, so there is no second path to split on. A data[].id is the routing id and can be sent straight back as model; created is the catalog release date as a Unix timestamp, or 0 when unknown.

This is discovery, not an allowlist: an arbitrary provider model ID can still be routed explicitly. See Model discovery.

GET /health reports that the process can serve requests and names its configured providers:

{"status":"ok","providers":["openai","anthropic"]}

It does not probe upstream provider availability.

Errors

Before a stream begins, errors use an HTTP status and this JSON envelope:

{
  "error": {
    "code": "rate_limited",
    "message": "Upstream provider rate limit reached; retry after the suggested delay"
  }
}

Proactive 429 and 503 responses include Retry-After in whole seconds. Provider failures normally retain the provider's status. See Errors and retries for the complete mapping.

Native API endpoints

Both llmshim proxy and llmshim gateway serve:

Point a native SDK at the server base URL and use a configured model route, such as anthropic/claude-sonnet-4-6, openai/gpt-6-astra or gemini/gemini-3.8-flash. Provider credentials stay in the server. On an authenticated gateway, the client key is a gateway key: Authorization: Bearer …, Anthropic's x-api-key and Gemini's x-goog-api-key are accepted. An explicit Authorization header takes precedence. Native requests use the existing gateway queue, authenticated tier, quotas and request IDs, or the proxy's concurrency and rate-limit admission. Rejections preserve HTTP status and Retry-After. Native idempotency keys are scoped to credential and wire format.

Gemini names its model and its action in the path, so a routing id's slash survives (/v1beta/models/gemini/gemini-3.8-flash:generateContent) and the body carries neither model nor stream. Its turns are contents[].parts: text becomes a message, inlineData becomes an image block, functionCall and functionResponse become a tool call and its result, and a thought part is restored from its receipt like any other opaque reasoning. systemInstruction, functionDeclarations, toolConfig, generationConfig (including thinkingLevel) and responseSchema translate through the shared engine. safetySettings and cachedContent ride the x-gemini namespace to Google verbatim; a field this server cannot express — candidateCount other than 1, thinkingBudget, allowedFunctionNames, includeThoughts: false, a hosted tool, an unknown part, or x-cache, whose explicit markers the engine applies on the Anthropic wire only — is refused by name, because forwarding it would fail upstream or silently disappear. Errors are Google's error object; generateContent returns a GenerateContentResponse whose modelVersion is the model that answered, and streamGenerateContent returns Gemini SSE chunks with no event names and no terminator — the finish reason ends the stream. That SSE form is the only one served: the query string is not read, so alt=sse is accepted and ignored while a client asking for Google's JSON-array form would have to parse frames it did not ask for. The credential is the x-goog-api-key header (or an explicit Authorization, which wins); ?key= is not read. Any other action under /v1beta/models/ — :countTokens, :embedContent, a misspelled or missing action — is refused with Google's 404 error object instead of reaching the chat handler, because the path is routed by a wildcard. :countTokens, :embedContent and the cached-content endpoints are not served.

{
  "model": "anthropic/claude-sonnet-4-6",
  "max_tokens": 1024,
  "system": "Answer briefly.",
  "messages": [{"role": "user", "content": "Hello"}],
  "stream": true
}

Messages system/content blocks, custom function tools, tool choice, tool results (including is_error), stop sequences, cache markers, and native JSON Schema output configuration translate through the shared engine. Chat Completions accepts its standard message/function shapes and response_format; JSON-object mode also gets object validation. Generation fields supported by the destination retain their ordinary adapter behavior. The facade supports one completion (n:1) and custom function tools; it rejects provider-hosted tool definitions rather than turning them into client-executed functions. n > 1 is refused rather than emulated — the OpenAI backend is the Responses API, which has no n, and neither do Anthropic Messages or Gemini; fanning out N requests would change the cost, rate-limit footprint and cache behavior of what was asked for. The refusal is a properly shaped OpenAI error naming the parameter:

{"error": {"message": "Unsupported value: 'n' must be 1. …",
           "type": "invalid_request_error", "param": "n",
           "code": "unsupported_parameter"}}

Provider-specific endpoints such as batches, token counting, uploads and Responses are not served by these aliases; Gemini's :generateContent family is, and nothing else under /v1beta is.

With stream:true, text arrives incrementally. Chat Completions emits chat.completion.chunk records followed by [DONE]. Messages emits message_start, content block events, message_delta, and message_stop. Completed reasoning and tool blocks are emitted after text so their metadata is complete before exposure; a Messages stream can therefore place these blocks after the text block. Keepalives continue while the engine runs. Managed structured-output/capture modes retain their validation buffering, as described in capability shims. Errors during a stream are native error events, not successful completion frames.

Native clients usually discard canonical tool wire maps. The facade stores the metadata it issued in private local receipts under the OS data directory's llmshim/replay-receipts folder. Set LLMSHIM_REPLAY_RECEIPTS_DIR to choose the location or mount a shared durable directory for several server instances. Files are written atomically with private permissions; they contain tool-call arguments and reasoning/signature metadata, not provider credentials or whole prompts. Receipts are scoped to a digest of the inbound credential. Preserve that directory and credential when replaying native histories across restarts.

New receipts are retained for 30 days, with limits of 100,000 files and 256 MiB of serialized receipt data. The oldest issued receipts are removed first when a limit is reached. Configure these trusted-operator limits with LLMSHIM_REPLAY_RECEIPTS_TTL_SECS, LLMSHIM_REPLAY_RECEIPTS_MAX_ENTRIES, and LLMSHIM_REPLAY_RECEIPTS_MAX_BYTES. Receipt lock acquisition waits at most two seconds by default; LLMSHIM_REPLAY_RECEIPTS_LOCK_TIMEOUT_MS can set a trusted operator value from 1 to 30,000 milliseconds. Native HTTP conversion uses two bounded blocking slots per application, with at most one used by ingress so a completed response always has reserved egress capacity. The ingress and egress wait queues are separately capped; excess or timed-out work receives a retryable 503. Canceled queued work performs no I/O, while a started worker retains its capacity until it exits. The journal is capped at 64 MiB; crash recovery can also use one fixed receipt staging file of at most 4 MiB and one fixed journal staging file of at most 64 MiB.

One native request may restore at most 4,096 receipt occurrences, including repeated uses of the same valid handle. Those occurrences share limits of 2 MiB of encoded receipt data, 32,768 decoded JSON nodes, and 8 MiB of estimated owned JSON memory. Each occurrence is admitted before its record is read and parsed; repeating a key consumes the limits again because the canonical history repeats the value. The reconstructed canonical request is also limited to 32,768 nodes, 8 MiB estimated owned JSON, and 2 MiB when serialized for the shared handler. Excess restoration fails as a native 413 without truncating history or starting provider inference. These conservative estimates bound conversion allocations; they are not process RSS measurements.

On first use after an upgrade, llmshim counts existing receipt files once and includes them in admission without expiring or deleting them because they have no trustworthy issue time. If that bounded pass cannot establish a complete total, existing receipts remain readable but new writes fail closed. After raising a limit, set LLMSHIM_REPLAY_RECEIPTS_RESCAN_INCOMPLETE_BASELINE=1 for one explicit retry in that process. Managed replacements preserve existing files on disk and record that the older value must never resurface after expiry. All processes sharing a directory must run a version that participates in the retention lock; an older binary can write outside its accounting. Deleting a retained receipt makes its owned tool ID unavailable, and the call fails explicitly. A changed call's arguments cannot reuse an issued receipt.

Untracked native thinking has no provenance and is dropped. Issued reasoning restores its original typed block and still passes through the common family, wire and account replay filter. Cross-wire opaque reasoning uses a receipt handle in native Messages fields so the original block can be restored without pretending it originated on that wire. Chat Completions includes a reasoning_details extension (and readable reasoning_content); clients that discard them also discard replayable reasoning. Keep full native assistant messages and tool IDs when saving a conversation. Receipt metadata is separate from the caller-owned conversation history.

Requests and reconstructed canonical handler bodies are limited to 2 MiB, and buffered response conversion is limited to 32 MiB. x-cache segment indices on native requests refer to their input messages array; the facade remaps boundaries when system/tool-result conversion expands that array. x-shim applies through the shared client. A signature mismatch's x-llmshim-served-model observation appears in JSON and the HTTP header for unary responses, or in terminal stream metadata. It never changes message provenance.

Responses streams and reasoning history

POST /v1/responses accepts stream: true through the same engine and admission path as ordinary requests. The SSE event name matches the JSON type, and sequence_number increases across creation, progress, item/part addition, text or function-argument deltas, item completion, and the terminal response. Text arrives incrementally except when hosted search requires buffering for native item order; function arguments arrive once the shared engine has assembled and validated the complete call. The terminal event is response.completed or response.incomplete with final output and usage. Upstream failure or premature EOF emits response.failed. Disconnecting cancels the body and drops the upstream stream.

Reasoning input items require a summary array of summary_text parts and may carry string id and encrypted_content fields. Return the issued item unchanged in the next request. Credential-scoped receipts restore the original provider block, including encrypted content or an Anthropic thinking signature. The existing replay policy checks the target wire, model family and issuer binding; a client cannot supply its own provenance. Unknown, changed, expired or cross-credential items are dropped. Incompatible target reasoning is dropped by the engine. metadata.reasoning_dropped lists drop reasons rather than refusing the full request. Summary-only external items do not establish provenance.

Use include: ["reasoning.encrypted_content"] to request encrypted output from upstreams that support it. Other providers return replayable receipt-backed reasoning items without publishing their private thinking text or signatures. Receipt retention and restoration limits apply as on the Messages facade.

The endpoint remains stateless: previous_response_id, conversation, and store: true are refused by name. Send explicit full history. Custom and namespaced tools are supported, including Codex CLI request controls and additional_tools declarations. Hosted search declarations require a native Responses provider. See Responses requests for exact translation behavior and the accepted controls that may be ignored.

Agent CLI integration

This endpoint supplies the inference transport for an agent CLI; it does not launch that CLI or implement a host's parent-turn provider. That integration belongs in the embedding harness. Its turn provider should launch the CLI with an isolated home and workspace, direct inference to an authenticated metered endpoint, and translate the CLI's completion into the parent turn result. Cancellation must stop the child and its requests; failed or incomplete turns must remain distinguishable from successful completion. Preserve full request history, tool inputs, result items and issued reasoning receipts across turns.

Deploy the proxy safely

The llmshim proxy has no built-in authentication and no TLS. It rejects browser-originated requests by default. Do not expose it directly to an untrusted network.

The supported public topology puts an authentication and TLS gateway in front of the proxy:

flowchart LR
    C[Applications] -->|HTTPS + credentials| G[External gateway<br/>TLS + authn/authz]
    G -->|trusted network| P[llmshim proxy]
    P -->|provider API keys| O[LLM providers]

The gateway can be any reverse proxy or API gateway that terminates TLS, authenticates callers, authorizes access, and applies your network policy. Keep the llmshim listener reachable only from that trusted boundary.

Browser applications

SDKs and other HTTP clients that do not send an Origin header continue to work normally. A browser application must be explicitly trusted before the proxy or gateway will process its requests:

LLMSHIM_TRUSTED_ORIGINS=https://app.example,http://localhost:5173 llmshim proxy

The value is a comma-separated list of exact http or https origins. Paths, queries, fragments, credentials, wildcards, and opaque origins are rejected. Each allowed origin receives CORS headers, including the private-network preflight permission required by browsers that enforce Private Network Access. An unlisted origin is rejected before it can reach a proxy, gateway, or native API route. This browser control does not replace proxy authentication, TLS, or network access policy.

Install a proxy-enabled binary

The HTTP server is behind the Rust proxy feature:

cargo install llmshim --features proxy

The Homebrew tap also installs a proxy-enabled binary:

brew install sanjay920/tap/llmshim

Then configure at least one provider key and start the process:

export ANTHROPIC_API_KEY="..."
llmshim proxy

The default address is 0.0.0.0:3000. Override it with environment variables:

LLMSHIM_HOST=127.0.0.1 LLMSHIM_PORT=8080 llmshim proxy

LLMSHIM_HOST and LLMSHIM_PORT take precedence over the proxy host and port in ~/.llmshim/config.toml. Binding to 127.0.0.1 is a useful default when a same-host gateway connects to the proxy.

The bundled ordinary proxy and gateway listeners cap accepted connections and apply finite first-header and HTTP/1 header deadlines. Configure LLMSHIM_HTTP_MAX_CONNECTIONS and LLMSHIM_HTTP_HEADER_TIMEOUT_MS for your deployment; see listener configuration. Keep request-body upload and per-client traffic controls at the front gateway.

Keep provider keys server-side

The proxy process reads OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY, and XAI_API_KEY. Language clients send neither those keys nor provider authorization headers. Store the keys in the process environment or the proxy's local config file, using your deployment platform's secret mechanism.

Do not bake keys into an image, return them to clients, or place them in provider_config. The latter is request data forwarded to adapters, not a secret store.

Container boundary

A minimal deployment has one llmshim process per container or service unit:

public client
    -> HTTPS/auth gateway
        -> private llmshim:3000
            -> provider APIs

Give the process outbound HTTPS access to the configured providers and inbound access only from the gateway or trusted callers. Use GET /health for process health; it confirms the proxy is running and lists configured providers, but does not call upstream APIs.

For multiple replicas and coordinated limits, continue to Scaling and rate limits. For every deployment setting, see the configuration reference.

Scaling and rate limits

The proxy is stateless per request. Put replicas behind a load balancer and send the complete conversation history on every call. No sticky session is required for llmshim itself.

flowchart LR
    C[Clients] --> L[Load balancer]
    L --> P1[Proxy replica 1<br/>pool + admission state]
    L --> P2[Proxy replica 2<br/>pool + admission state]
    L --> P3[Proxy replica N<br/>pool + admission state]
    P1 --> U[Provider APIs]
    P2 --> U
    P3 --> U
    P1 -. optional shared limits .-> R[(Redis)]
    P2 -. optional shared limits .-> R
    P3 -. optional shared limits .-> R

Each process owns its connection pool, concurrency semaphore, and—unless Redis coordination is enabled—its rate-limit buckets.

Connection reuse and warmup

All calls in one process share a lazily initialized HTTP client. Its pool uses HTTP/2 where available, gzip/brotli/zstd/deflate compression, a 90-second idle timeout, up to four idle connections per host, a 30-second TCP keepalive, and TCP NODELAY.

For Rust services, call llmshim::warmup(&router).await after constructing the Router. It sends bounded HEAD requests to configured built-in provider origins to pre-establish TCP and TLS connections. Failure to warm a provider is ignored; normal request handling remains authoritative.

Reactive retries

The shared client retries transport failures and HTTP 429, 500, 502, 503, 504, and 529. The default is three retries after the initial request—up to four attempts total—with exponential backoff and jitter. Retry-After and recognized OpenAI or Anthropic reset headers take precedence over computed backoff. Invalid reset durations are ignored; oversized finite durations saturate safely before the configured cap and jitter are applied.

These retries stay on the same route. A non-streaming fallback chain adds a second layer: retry the route, then change the route. See Fallback chains.

Backpressure and proactive limits

Every logical request first acquires an instance preparation slot before route resolution, schema work, or provider preparation. Each actual provider attempt then acquires a separate concurrency slot immediately before the send. Both waits use the queue timeout and return 503 with Retry-After when saturated. The logical slot remains held for the request or stream lifetime. A retry or repair releases the completed attempt's slot and must acquire again; fallback acquires against the provider it actually targets without reacquiring the logical slot.

The authenticated gateway checks credentials before acquiring its short-lived prequeue preparation slot. It releases that slot after native conversion, route expansion, token estimation, keyed request fingerprinting, and policy checks. Distributed mode retains it through finite-queue admission and then releases it before waiting for execution or results; local mode releases before its synchronous in-memory enqueue. Queued dispatch and each final provider attempt keep their independent capacity and rate-policy gates. Post-queue gateway response projection is outside this short-lived prequeue slot.

Optional token buckets can reject work before it reaches a provider. A rejection is 429 with Retry-After.

When neither RPM nor TPM is set, proactive rate limiting is disabled; concurrency backpressure still applies. Token permits are estimates based on the final provider-native body and authoritative prepared target. They include the serialized native prompt and schema material, recognized native reasoning budgets, and the effective native output limit. For Chat Completions, a null max_completion_tokens does not hide a numeric max_tokens limit; a numeric max_completion_tokens retains precedence. Recognized output limits must use unsigned JSON integers or null for TPM estimation. Other representations, including quoted integers and floats that a backend may coerce, reserve the maximum token estimate instead of silently falling back to an omitted-limit default. This affects TPM admission only; the forwarded request is unchanged. Use integer JSON values for bounded output allowances. For native Chat Completions bodies, OpenAI's n is treated as a generated candidate count; Gemini generationConfig.candidateCount is treated the same way. The output allowance is multiplied by the final wire candidate count. The estimator also defensively recognizes best_of, bestOf, and candidate_count, including a generation_config container that llmshim can preserve from x-gemini, when a final body contains them; that compatibility handling does not imply universal provider support. Anthropic's omitted output limit uses the adapter's 8,192-token default; other known models use the catalog output ceiling when the provider leaves the limit unspecified. Unknown hosted models use a conservative provider-family ceiling; unknown self-hosted models retain the 1,024-token fallback because the server's launch configuration is not visible to llmshim. OpenRouter models fallbacks remain supported; when no explicit output limit is present, every listed model contributes its catalog ceiling and an unknown listed model uses a one-million-token ceiling.

These permits are conservative estimates, not provider billing measurements. The input side uses serialized characters divided by four; provider tokenizers, images, caching, and unknown self-hosted model defaults can differ. Admission occurs once per actual network attempt. Transport retries, managed schema repairs, and supported fallback targets each reacquire RPM and TPM from the lane for the provider/body that will be sent. A provider-wide refusal may advance a fallback chain only to a distinct provider; an authenticated tenant refusal terminates the request.

For an authenticated gateway identity, an omitted rpm or tpm field means that dimension is unlimited. An explicit 0 means that dimension admits no requests; the gateway rejects before dispatch and does not consume the other tenant bucket. The same deny-all behavior applies to global and per-provider rate-limit values: an explicit zero returns 429 before dispatch, while an omitted value remains unlimited.

Provider health

Rate limiting and health are different questions. A 429 means the provider is alive and asking for less, and the token buckets already slow it down. A circuit breaker counts what retrying cannot fix — 500, 502, 503, 504, 529 and transport failures — over a sliding window, opens the circuit at the threshold, and admits a single probe after the cooldown.

Every dispatch path observes outcomes, so health accrues from ordinary traffic. Only a fallback chain refuses: it skips a provider with an open circuit instead of spending its retry budget on a target it already knows is dead. A single-target request is still dispatched — with no alternative, refusing would only convert an upstream failure into a local one.

Spend caps

The experimental gateway enforces a per-identity USD cap beside the RPM/TPM buckets. A gateway key's identity may carry budget_usd and an optional budget_window_secs (default one day); over budget is a 429 with Retry-After set to the window reset.

{"sk-example": {"tenant": "acme", "tier": 1, "budget_usd": 100, "budget_window_secs": 86400}}

budget_usd must be finite, non-negative, and no greater than 9007199.254740992. The ledger stores nano-USD integers and keeps every Redis value within Lua's exact-integer range; invalid limits fail closed. An explicit zero freezes the identity, including models whose configured token rates are zero.

Every actual provider send reserves a conservative amount before it consumes RPM or TPM. The reservation uses the final provider-native model and body, trusted per-field catalog facts, the request's output/reasoning bound (or a trusted catalog output ceiling), and the applicable trusted price tier. Local operator policy is authoritative; verified built-in prices and built-in or provider-reported limits are accepted. Community models.dev values alone do not establish a hard-cap policy. The rate debits and USD reservation commit together, so a budget refusal consumes neither allowance. The full context ceiling is used for input rather than treating a tokenizer heuristic as a guarantee, so admission can be deliberately conservative even for a short prompt. Repeated usage snapshots upsert one attempt by UUID; they are never summed as separate bills. Partial or missing-counter usage may raise liability but cannot release it. A successful terminal response replaces its reservation only when the wire's required input/output counters are explicitly present, including explicit zeros, or when the provider reports an authoritative bill. A terminal provider bill also outranks an earlier catalog estimate. Failed repair responses are therefore charged even when the caller ultimately receives a local 502. Bounded provider error bodies are inspected for native usage or a provider-reported bill before retry or return; the original error body and size limit remain unchanged.

Any provider-reported partial charge also establishes a floor: a later terminal catalog estimate cannot settle below it. A genuine terminal provider bill may apply the provider-final correction rule. Public Chat stream aggregation exposes a nonterminal bill as usage.provider_cost_floor_usd instead of presenting it as an exact final cost_usd.

Streaming snapshots remain partial until the protocol normalizer validates a clean end. A private candidate must still cover every choice or candidate after all later output and newly seen indexes; early usage-only frames and mixed finished/unfinished results cannot authorize release.

Transport uncertainty, cancellation, stream abandonment, worker loss, and a failed settlement keep the original reservation in its acquisition window. The window is selected only after concurrency admission; Redis selects it from server time inside the atomic transaction. Rollover never moves that liability into a later window, and attempt tombstones remain available for late settlement for 24 hours after the acquisition window ends. This makes the cap a hard ceiling under the configured catalog pricing policy. Catalog prices are still estimates rather than provider invoices: an external price change or fee missing from the policy cannot be guaranteed by llmshim. If combined known liabilities exceed the fixed-point range, the window becomes irreversibly frozen; a later release from another overlapping attempt cannot reopen capacity after overflow information has been lost.

The gateway retains at most 100,000 attempt tombstones by default, configured by LLMSHIM_GATEWAY_MAX_RETAINED_ACCOUNTING_ATTEMPTS. This is an operational accounting-state limit, shared by the Redis fleet and per process in local mode. When it is full, a new budgeted attempt fails closed with 503 before any RPM, TPM, or spend debit; live reservations and the 24-hour settlement interval are never evicted to make room. Expired entries are reclaimed incrementally, so a large simultaneous expiry does not block the coordinator. Size this limit for the number of provider attempts that can begin during one budget window plus the 24-hour late-settlement interval. A high-throughput deployment can reach this state limit before it reaches its dollar cap.

Strict admission rejects a request when it cannot form that bound. This includes an unknown price or context/output ceiling, variable OpenRouter routing, multi-candidate output, priority/fast service controls, provider-hosted tools, unbounded media, unknown native controls, and cache-creation controls whose fee dimension is not bounded. The rejection happens before send:

400 {"error":{"code":"unpriceable_under_budget","param":"model", …}}

It is deliberately not a 429: retrying never clears it. Use a bounded model, add a complete local catalog policy, remove the unbounded control, or accept the risk explicitly per key:

{"sk-example": {"tenant": "acme", "budget_usd": 100, "budget_allow_unpriced": true}}

budget_allow_unpriced defaults to false. With it set, only requests lacking a defensible reservation receive the exception. Already-known spend must remain below the cap. Such an attempt conservatively holds the remaining window balance until final known usage can replace it; uncertainty or abandonment therefore exhausts the cap rather than releasing zero. Any later provider-reported or catalog-derived charge is still observed, but only a terminal provider bill can resolve the fee uncertainty and release an unpriced reservation. Priceable requests reserve normally even when the flag is set. Each exception increments llmshim_gateway_unpriced_under_cap_total{provider,model}; a non-zero counter means the configured policy cannot promise a finite bound for that target.

One replica or a coordinated fleet

The default buckets are in memory. With N replicas, each replica enforces its own configured limit. If the number represents a fleet-wide provider quota, divide it across instances or enable shared coordination.

To share one bucket, build the opt-in feature and set Redis:

cargo install llmshim --features redis-coordination
LLMSHIM_REDIS_URL=redis://redis.internal:6379 llmshim proxy

redis-coordination includes the proxy feature. Redis coordinates provider rate-limit buckets, provider health and — on the authenticated gateway — tenant RPM/TPM and spend, so these limits mean the same thing on every replica. The gateway checks provider and tenant RPM/TPM in one Lua decision: if any dimension rejects, none is debited. Its trusted per-attempt coordinator fails closed when Redis is unavailable. The compact proxy's standalone Redis limiter keeps its documented fail-open behavior. Both paths share the same provider bucket keys. Connection pools and concurrency limits remain per process. If the Redis client cannot be initialized—or the binary lacks the feature—the compact proxy warns and falls back to in-memory buckets.

Shared rate, health, and attempt coordination use the Redis driver's finite 500 ms response and one-second connection defaults. Healthy operations reuse a cached multiplexed connection. If an operation fails or its future is canceled after it may have sent a command, llmshim retires that exact cached connection generation; new operations create a fresh generation while existing healthy leases may finish on the old one. An uncertain mutating command is never replayed, refunded, or treated as a zero charge. The compact rate and health paths retain their documented fail-open behavior, while authenticated gateway attempt and spend coordination remains fail-closed.

With gateway-redis, admitting a new job checks the current fenced queues and the released legacy/scoped queues' combined provider waiting depth, then inserts the job in one Lua transaction. Concurrent current origins cannot claim the same remaining slot. LLMSHIM_GATEWAY_QUEUE_DEPTH defaults to 10,000 total waiting jobs per provider; a full queue refuses new work with 503 and Retry-After. Released origins cannot include the new queues in their older admission script, so the combined bound becomes exact after old ingress stops. The current two protocol queues keep independent priority ordering; one shared worker-capacity bound prevents them from doubling upstream concurrency.

Authenticated jobs use the fenced llmshim:gw:scoped:v2:* namespace. Trusted unscoped Rust submissions use llmshim:gw:fenced:v1:*. Current workers and reapers watch only those two namespaces, with one per-provider concurrency limit shared across them. Released workers cannot lease or acknowledge these jobs. The released llmshim:gw:* and llmshim:gw:scoped:v1:* records are preserved for their matching old workers; current workers do not silently migrate or execute them with a partially upgraded protocol.

Each leased delivery has a random owner token. An independent heartbeat keeps that lease current during rate and policy waits, provider preparation, unary calls, quiet streams, response publication, and terminal cleanup. Redis checks the token in the same transaction that refreshes, publishes, releases, completes, acknowledges, or moves the delivery to the dead-letter queue. A worker that has lost its lease can no longer publish or mutate the replacement delivery. Lease creation, refresh, and reaping use Redis TIME, so host clock skew cannot expire another worker's healthy lease.

Cancellation is ordered in the same Redis transitions as stream publication, terminal completion, rate release, reaping and DLQ movement. Once cancellation wins, later response chunks and provider content are suppressed and the durable terminal is the fixed cancellation envelope. Usage already observed from a provider remains settled; cancellation does not infer a refund.

Distributed worker jobs have a finite six-hour lifetime by default. Redis lease operations are bounded to five seconds or one quarter of the configured lease, whichever is shorter. Positive millisecond overrides are available as LLMSHIM_GATEWAY_WORKER_JOB_TIMEOUT_MS and LLMSHIM_GATEWAY_REDIS_OPERATION_TIMEOUT_MS; zero and invalid values retain the finite defaults. A worker deadline drops local provider work while retaining the heartbeat, then publishes its error, records completion, and removes lease state in one owner-fenced Redis transaction.

Caller-side Redis deadlines also cover distributed connection setup, readiness, queue and DLQ statistics, lifecycle reserve/activate/cancel/readback and the bounded generic and scoped idempotency caches. Each of those logical operations uses the same generation-retiring connection cache: failure or caller timeout retires that operation's exact generation before a later operation can reuse it. Pub/sub connections remain separately owned for the accepted response lifetime. A leased request and its metadata retain hard expiry headroom for the full worker lifetime plus cleanup grace.

The queue remains at-least-once across an actual worker crash or coordination partition. If a provider accepted a request before ownership became uncertain, redelivery can still make another external call; lease fencing prevents stale local publication and cleanup but cannot make an external provider exactly once.

New distributed submissions use lifecycle-v2 storage. Queue and processing indexes contain only an opaque job ID plus a Redis-owned generation; the exact serialized request is stored once and Lua never decodes or rewrites it. Reserve and activate are separate state transitions, so an uncertain reserve is not dispatchable until its matching generation activates. Delayed operations from an older generation cannot resurrect or mutate a replacement job.

Lifecycle-v2 adds a generation- and token-fenced origin lease. Origins renew it while awaiting unary results or relaying streams, but never beyond the immutable total deadline. LLMSHIM_GATEWAY_REQUEST_TIMEOUT_MS remains the distributed queue-plus-upstream total limit; the HTTP logical deadline may shorten it and cannot extend it. LLMSHIM_GATEWAY_ORIGIN_LEASE_MS optionally tunes crash detection and is capped at five minutes. Redis server time decides expiry. Dropped unary futures and quiet streaming bodies request cancellation through one bounded gateway pump; a fixed-batch Redis expiry index covers process loss and pump saturation without detached per-drop tasks. Unary terminals remain readable after response-bus loss. Stream chunks remain transient, so a lost stream bus is reported as an error rather than treating a durable End marker as proof that every chunk reached the origin.

Lifecycle storage defaults to 10,000 retained jobs, 512 MiB of request/metadata bytes, and a separate 512 MiB terminal pool. Environment overrides are LLMSHIM_GATEWAY_RETAINED_JOBS, LLMSHIM_GATEWAY_RETAINED_BASE_BYTES, and LLMSHIM_GATEWAY_RETAINED_TERMINAL_BYTES. Workers reserve terminal capacity before provider dispatch: 64 MiB for unary and 128 KiB for stream terminal state, configurable with LLMSHIM_GATEWAY_UNARY_TERMINAL_BYTES and LLMSHIM_GATEWAY_STREAM_TERMINAL_BYTES.

The 64 MiB unary value is a distributed-service limit on the serialized normalized terminal envelope. It is distinct from the 32 MiB decoded raw upstream limit because normalization and JSON escaping can expand data. A bounded serializer stops before allocating beyond the limit. An over-limit normalized result becomes a fixed size error; ordinary shared-capacity pressure cannot discard an otherwise valid paid result because the full headroom was reserved before send.

Terminal envelopes are retained durably for bus-loss readback. Scoped idempotency stores a bounded pointer to that single terminal copy instead of a second response. The deprecated generic Redis cache is separately bounded to 100,000 entries, 1 MiB per value, 64 MiB total, and at most one day of TTL. Both cache kinds use per-entry Redis expiry; bounded reservation indexes let a restarted janitor recover count and byte accounting after the value has expired. The pointer and its terminal extension use the same capped effective TTL; a terminal's existing longer default retention is not shortened. Concurrent first-request cache misses are not coalesced and may execute more than once. Expired lifecycle records are removed in fixed-size batches; authoritative bounded reservation indexes retain the state, scope and charges needed to release counters and queue/DLQ ownership even after payload and metadata hard expiry. State movement between waiting, processing, requeue, terminal, and DLQ does not double-count request bytes. Pending/waiting cancellation and DLQ diagnostic envelopes fit inside each job's 4 KiB base metadata allowance; processing cancellation and provider response terminals use pre-reserved terminal capacity. New DLQ retention is limited to 1,000 entries, 64 MiB and 24 hours.

Lifecycle-v2 ingress remains disabled while any known lifecycle-v1 or older waiting or processing namespace is nonempty. Stop old ingress and let matching old workers drain those fixed keys; lifecycle startup does not scan, migrate, or delete old members. Historical DLQ lists and deprecated generic-cache keys are not covered by the new bound until an operator explicitly inventories and removes them. Legacy scoped response-cache keys expire after their configured cutover TTL.

For a rolling upgrade, first stop old ingress. Keep the matching old workers until both the waiting and processing sorted sets in each released namespace are empty for every provider, then remove those workers. Old work retains its old lease and publication guarantees while draining. Do not delete its Redis keys to accelerate the transition: those entries represent accepted requests. New queue-depth metrics include released waiting entries, but operators must also inspect the released processing keys before declaring the drain complete.

During an upgrade from the post-charge spend counter, each new authenticated origin uses one Redis-time Lua transaction to read the active legacy llmshim:spend value and retain the largest rounded-up nano-USD floor before idempotency lookup, queueing, or any worker refusal. A replay, conflict, overloaded or cancelled submission, unpriceable request, semaphore timeout, and zero-rate policy therefore cannot forget spend that the origin already saw. The private descriptor carries only the Redis-time window index and retained nano-USD floor, never the raw tenant or bearer. Worker admission tracks the separately applied floor and adds only a later positive delta, alongside new reservations and settlements. Origin retention creates no attempt, rate debit, aggregate charge, or applied delta. This preserves known active-window spend without double charging repeated imports or forgetting intervening new spend. Known-floor records share the finite retained-accounting index. If that index cannot retain newly learned spend, the same origin transaction sets one fleet freeze marker through the longest affected retention horizon. It remains in force if a slot later frees, so a stale descriptor cannot exploit discarded knowledge. A malformed, negative, non-finite, or wrong-type active legacy value also sets or extends that marker before the origin fails closed; its amount is unknown, so older queued snapshots cannot safely proceed. Keep each identity's budget_window_secs unchanged through this transition so the old and new counters name the same tumbling window.

An origin running released code can still admit and record its own request after a new origin takes that point-in-time snapshot. Queue isolation prevents old workers from weakening new-origin jobs, but cannot retrofit atomic accounting into old ingress. Stop old ingress before claiming one fleet-wide hard cap for the rest of the transition window. After old ingress is stopped, queued legacy scoped jobs fail closed on new workers; custom unscoped jobs remain compatible. If Redis cannot execute or persist the origin transaction at all, it cannot durably record a freeze. Worker coordination fails while Redis is unavailable; after recovery, require a fresh successful origin retention before treating previously queued transition work as safe.

Do not infer capacity from llmshim's implementation details alone. The README benchmarks are the maintained performance snapshot; load-test your model mix, payload sizes, provider quotas, and gateway before choosing replica counts.

Surface capability matrix

The Rust crate is the engine. The CLI calls it in-process, the proxy places an HTTP boundary around it, and the language clients speak that proxy contract.

Availability: Crate: in-process Rust · CLI: human workflow · Proxy: language-neutral server · Clients: proxy SDKs

Choose the boundary

The caller resends conversation history; the CLI does that for its current interactive session. Native HTTP facades persist private replay metadata for issued reasoning and tool IDs; see native endpoints.

Where request features go

The proxy's provider_config object is merged into the OpenAI-shaped engine request. That is why tools and native namespaces move under it without changing their inner shapes. Root model and messages are reserved. After route and alias resolution, the selected provider's namespace also cannot replace its native model or main history/input container. Native system and instruction controls remain supported. Explicit fallback targets receive the same checks, so a namespace is inert only when no selected fallback or route can activate it.

Build features

The Homebrew package and the binaries bundled by Python and TypeScript include proxy support. Go and Ruby are pure HTTP clients and require a separately running proxy.

All language clients expose reasoning blocks and the tool-call signature object (data plus origin). Preserve those objects for replay.

Upstream response limits

The engine limits ordinary JSON completion bodies to 32 MiB and provider error bodies to 64 KiB, measured after HTTP decompression. It checks decoded chunks before adding them to its body buffer. Oversized final bodies return a fixed 502 error without including a truncated provider response. Retry bodies use the same error-body buffer limit; oversized bodies are dropped before another attempt.

These limits apply through the Rust completion API, CLI, proxy, and gateway. They bound decoded body bytes; JSON allocations and process memory have additional overhead. Successful raw responses returned by the low-level ShimClient::send API use the configured connect and response-header deadlines. After headers are returned, body size, decoding, idle time, total lifetime, and cancellation remain the caller's responsibility; llmshim does not attach a hidden body task or timer to that raw response.

Normal provider streams, ChatGPT's collected SSE replies, and native HTTP facades use one bounded SSE data decoder. It checks input before growing its line and frame buffers:

The decoder handles UTF-8 fragments, an initial byte-order mark, multiline data fields, comments, and LF/CRLF/CR line endings. Provider transforms consume the data fields; unused SSE metadata is discarded. Limits terminate the stream with a fixed error and release its input source. EOF does not manufacture a completed event from an unterminated frame. These are framing limits, not an RSS ceiling or a transport timeout.

The retained-state limits count only data held across events, including map and JSON-container overhead. Forwarded text is not charged after its chunk is released. Exceeding either retained-state limit ends the response with upstream stream retained state exceeds limit; no partial tool call or reasoning block is emitted. Rust embedders can inject smaller finite limits with StreamRetentionLimits for testing or a stricter deployment policy.

Collected text and refusals, reasoning text and opaque fragments, and native facade text append to their existing buffers. Completed reasoning snapshots still replace their matching block while preserving its original provenance.

For shape details, continue to the request field map and HTTP API.

Request field map

Rust and the proxy express the same request semantics through different envelopes. Rust uses an OpenAI-shaped serde_json::Value; the proxy groups its portable subset under config and everything else under provider_config.

Availability: Rust: flexible top-level JSON · Proxy/clients: typed compact envelope

Rust engine request

These are the stable, adapter-recognized fields. A field marked “mapped” is renamed or reshaped for the selected provider; unsupported controls are omitted.

There is no x-xai namespace. OpenAI additionally recognizes store, prompt_cache_key, prompt_cache_retention, safety_identifier, and speed. Anthropic recognizes cache_control, thinking, output_config, and speed. Those fields are provider-specific even when they are not wrapped in an x-* namespace.

The request is a Value, so Rust does not reject unknown keys up front. Adapters deliberately select only fields they understand. Do not assume an arbitrary top-level key reaches the provider; use a documented native namespace.

Reasoning values and family-specific clamps are canonicalized in Reasoning controls. Native namespaces and precedence are covered in Native provider controls.

Message fields

Content blocks may use OpenAI image_url, Anthropic image, or Gemini inline_data input forms. See Images and vision.

Lossless reasoning round-trip

Preserve the complete assistant message, including reasoning[] and tool-call signature objects. llmshim reconstructs native reasoning only when the stored origin matches the target family and wire; encrypted data also requires a matching issuer binding. Anthropic thinking remains first and ordered, and Responses items retain their encrypted payloads. Unknown origins/families are never replayed. See reasoning replay.

Proxy request

The exact config children are:

For example, this proxy fragment:

{
  "config": {"reasoning_effort": "high"},
  "provider_config": {
    "tools": [],
    "x-anthropic": {"disable_1m_context": true}
  }
}

becomes engine fields named reasoning_effort, tools, and x-anthropic. Because provider_config is merged after config, a same-named child there overrides the portable value. Prefer the x-* namespace for an intentional native override; it makes that loss of portability visible.

model and messages are reserved at the provider_config root. After named route defaults and aliases are resolved, the proxy validates only the selected provider's active namespace and wire. Every resolvable explicit fallback is validated the same way before the primary request runs. The proxy rejects fields that would replace the admitted model or main history/input container: x-openai.model/input, x-chatgpt.input, x-anthropic.model/messages, x-gemini.contents, x-openrouter.model/messages, and the corresponding model plus messages/input fields for any OpenAiCompatible provider's configured namespace and wire. A namespace is inactive only when neither the primary nor an explicit fallback or route can select its provider. Native system and instructions, tool definitions, schemas, output limits, OpenRouter routing preferences, and other documented controls remain valid and are included in the estimate when active.

Output contracts

response_format: {type:"json_schema", json_schema:{name?, schema, strict?}} requests validated JSON output. x-shim controls structured_output (auto|native|forced_tool|prompt), tool_calling (auto|native|prompt), and reasoning_capture (off|forced_tool). See capability shims for path selection, bounded repair, streaming behavior, and provenance.

Provider behavior matrix

Every adapter accepts the same OpenAI-shaped engine request, but it targets a different native API and translates only the fields that API understands.

An explicit address such as anthropic/claude-sonnet-5-5 avoids inference. The named provider must be registered in the Router—that normally means its API key is configured.

ChatGPT uses a saved device-code login instead: llmshim login chatgpt. Both normal and streaming calls use upstream SSE. Normal calls aggregate a terminal response into Chat Completions JSON; a failed or truncated stream returns an error. Tools, image inputs, and reasoning use the Responses translator. ChatGPT forces store: false and stream: true and strips token limits, metadata, and sampling fields even from native overrides, following LiteLLM's backend contract. It uses a short default instruction when no instructions are supplied. Only chatgpt/gpt-6-astra, chatgpt/gpt-6-sol, chatgpt/gpt-6.1-sol, and chatgpt/gpt-6-luna are accepted; older and unlisted models fail locally before authentication or network calls. Access to those models and limits depend on the signed-in account. ChatGPT responses preserve the full chatgpt/<model> ID, including streaming chunks, so server responses identify the subscription provider correctly when API-key OpenAI is also configured. Streamed function calls are emitted once their arguments are complete. Each call includes its ID, name, and full JSON arguments, including in the proxy's tool_call event; text and reasoning still arrive incrementally.

Observable differences

OpenAI and xAI do not receive temperature, top_p, top_k, or stop from the portable top-level request. Anthropic and Gemini do. This is the portable-core rule in practice: unsupported fields are omitted rather than forwarded blindly.

Tools and multimodal round trips

Claude Fable 5.1

Fable 5.1 uses always-on adaptive thinking, with low, medium, high, xhigh, and max effort. Unified none maps to low; explicit native disabled/manual thinking and assistant prefill return local errors. Sampling parameters are omitted for Fable 5.1, Opus 5, and Opus 5.5 because those models reject them.

Fable 5.1 and Opus 5.5 accept only auto and none: required or a named forced tool returns a local 400 rather than changing the request's meaning. Describe the desired tool in the prompt, or use structured output when the requirement is a JSON schema. Native stop_reason: "refusal" maps to finish_reason: "content_filter" for Fable and Opus 5/5.5.

Fable 5.1 binds thinking blocks to the preceding system prompt, tools, and conversation. Keep that prefix unchanged when replaying signed reasoning. Appending turns is supported; llmshim preserves appended system/developer instructions as system turns for Fable 5.1. If your application edits or compacts prior history, remove stale thinking or use Anthropic's explicit binding controls under x-anthropic. A binding failure remains an error. See Anthropic's migration and history rules.

Opus 5.5 also requires adaptive thinking, rejects assistant prefill, and binds thinking to the conversation prefix. It accepts thinking from older Opus, Sonnet, and Haiku models, but not Fable or Mythos. Its thinking blocks replay only to Opus 5.5, Fable 5.1, or Mythos 5.1. See the Opus 5.5 migration guide.

Sonnet 5.5 requires adaptive thinking and binds thinking to the conversation prefix, and it rejects forced tool choice. Assistant prefill is not documented as rejected, so it stays allowed. Sonnet 5.5 accepts thinking from Sonnet 5, Opus 4.8, Haiku 4.5, and earlier — but not from Opus 5, Opus 5.5, or a Fable/Mythos model — and its own thinking blocks replay only to Sonnet 5.5. See the Sonnet 5.5 model page.

The raw Rust response exposes normalized thinking signatures for replay. The compact proxy response does not expose every native thinking field; applications needing native history controls can supply exact messages under provider_config.messages.

Shared message formats

Tool calls are normalized back to the OpenAI tool_calls shape. Preserve the complete assistant tool call when sending the next turn; provider adapters may need fields beyond function name and arguments.

Gemini tool signatures carry data and origin; all typed clients expose this object. Reasoning is replayed only to a compatible family and wire, and encrypted blocks also require a matching account binding.

Image input accepts OpenAI image_url, Anthropic image, and Gemini inline_data blocks. Base64 data URIs can be translated among providers. Gemini's remote-URL fallback is literal text such as [Image: URL]; llmshim does not download the URL.

Embeddings

Embeddings are a separate operation rather than a chat translation. OpenAI's /v1/embeddings, Gemini's batchEmbedContents, and an OpenAI-compatible server's /v1/embeddings are reached through llmshim::embeddings; Anthropic, xAI, OpenRouter, and the ChatGPT subscription backend have no embeddings route and refuse by name. See the embeddings guide.

Reasoning and native controls

Unified reasoning_effort and reasoning_mode are mapped by model family, not merely by provider. Floors, ceilings, and name-locked models are listed in the reasoning tables.

Use native provider controls when exact native semantics matter. Native objects are not rewritten or made portable. Changing the provider address means reconsidering the namespace as well.

Model discovery

Runtime discovery is the canonical answer to “what can this configured installation offer?”

llmshim models
curl http://localhost:3000/v1/models

Both commands filter the built-in registry to providers with configured API keys or a saved ChatGPT login. The proxy returns id, provider, and unprefixed name; the CLI prints the ID and display label.

The standalone llmshim-catalog crate supplies broader metadata without changing this curated discovery list:

llmshim models --all --json
llmshim models --refresh
LLMSHIM_CATALOG_OFFLINE=1 llmshim models --all

Its owned ModelInfo adds model family, pricing, reasoning effort/budget options, modalities, dates, and field-level sources. llmshim::catalog::resolve reads the resident catalog; llmshim::models::spec retains the borrowed, compile-time API for curated and historical facts.

Startup reads the vendored floor, cached data, and local overrides without waiting for network. Refresh uses ETags and a 24-hour TTL; errors keep the old snapshot available. Router::from_env() schedules one such refresh; Router::from_env_without_catalog_refresh() schedules none and leaves it to Router::refresh_catalog_in_background(). Offline mode uses only vendored and local layers. User overrides live in ~/.config/llmshim/models.toml, project overrides in .llmshim/models.toml, and cached data in ~/.cache/llmshim/models.dev.json. Local overrides win; verified builtin assertions win over models.dev data. Provider discovery is explicit, accepts capability facts, and never supplies pricing. Catalog entries do not imply credentials or account entitlement.

Advertised catalog

The CLI picker, llmshim models, and /v1/models share this curated set of 13 routes. It keeps the current model in each retained tier. Google entries are stable releases only. Credentials determine which providers are listed.

OpenAI

Anthropic

Google Gemini

xAI

Grok 4.7 accepts text and images with a 500,000-token context window. Reasoning supports low, medium, high (the upstream default), and xhigh; llmshim maps none to low and max to xhigh. Explicit xai/grok-4.6 requests remain supported, but discovery advertises 4.7.

Native xAI standard rates, USD per million tokens:

The higher tier applies to the whole request. These are native xAI rates; OpenRouter has its own prices. See the release notes. Grok 4.7 Fast is available in Cursor/Grok Build, not as a public xAI API model.

For OpenRouter, use openrouter/x-ai/grok-4.7. You can append :nitro and set x-openrouter.provider.zdr = true together. Nitro prioritizes throughput and allows priority endpoints, whose rates can be higher. ZDR restricts eligible inference endpoints to those with a zero-retention policy. On /v1/chat, put x-openrouter inside provider_config:

{
  "model": "openrouter/x-ai/grok-4.7:nitro",
  "messages": [{"role": "user", "content": "Reply with only pong."}],
  "config": {"max_tokens": 1024, "reasoning_effort": "low"},
  "provider_config": {"x-openrouter": {"provider": {"zdr": true}}}
}

These routing options do not enforce ZDR on an llmshim fallback to a different provider. For throughput sorting without priority endpoints, omit :nitro and set x-openrouter.provider.sort to "throughput" alongside zdr.

The Nitro route has replay metadata but no fixed catalog price. When OpenRouter reports its bill, llmshim uses it for usage.cost_usd and sets usage.cost_source to "provider". Without a reported bill or locally configured rates, cost_usd remains null.

A partial stream bill that is not confirmed as the terminal provider bill uses cost_source: "provider_floor"; it is a known lower bound rather than an exact invoice.

The OpenAI model catalog lists GPT-6 Sol and Luna at $2/$10 and $0.10/$0.50 per million input/output tokens on the Standard tier. Prompts above 272,000 input tokens use higher rates for the whole request; the built-in catalog records those tiers.

Anthropic positions Claude Opus 5.5 at Fable 5.1 level on most work, and estimates around 40% less cost per typical task than Opus 5. Its $4/$20 input/output token prices are 20% below Opus 5. Cache reads cost $0.20 per million tokens. Five-minute cache writes cost $5 per million; one-hour writes cost $8. Because the catalog has one aggregate cache-write rate, its $8 estimate is the conservative one-hour ceiling. See the Anthropic model pricing.

Anthropic positions Claude Sonnet 5.5 as "the best combination of speed and intelligence". Its $2/$10 input/output token prices match Sonnet 5. Cache reads cost $0.20 per million tokens. Five-minute cache writes cost $2.50 per million; one-hour writes cost $4. Because the catalog has one aggregate cache-write rate, its $4 estimate is the conservative one-hour ceiling. Sonnet 5.5 thinks adaptively by default at high effort. See the Anthropic model pricing.

ChatGPT subscription

The ChatGPT provider accepts only these four model IDs. Older and unlisted IDs are rejected locally before authentication or network calls, including when selected through a Router alias. The list is not an account entitlement check. The new models are rolling out, so availability also depends on workspace settings and account entitlement; see the ChatGPT and Codex changelog. Context and output limits remain unspecified because they depend on the account/backend. Bare GPT names still route to API-key OpenAI.

Spec metadata

Each registry entry can also carry spec metadata beyond its identity, so a consumer can read a model's facts from one place instead of maintaining its own parallel table:

Support is tri-state: Supported, Unsupported, or Unknown. Unknown is honest, not a bug — llmshim never guesses a spec to fill a cell. Context window, output ceiling, and capabilities are versioned snapshots populated from official provider docs (platform.claude.com, developers.openai.com, ai.google.dev, docs.x.ai), and reasoning is cross-checked against the provider clamp logic. What's deliberately left Unknown/None:

• parallel_tool_calls for most models (providers rarely document it per model);
• xAI max_output_tokens (not published) and per-model streaming.

Note also that Gemini publishes an input limit rather than a combined total, so context_window_tokens is the input window with max_output_tokens separate.

reasoning is deliberately a single flag — "does this model accept a reasoning control at all." The detailed per-tier mapping is not duplicated here; it lives in the reasoning guide and the provider transforms.

Look up one model's full spec from the Rust crate:

#![allow(unused)]
fn main() {
if let Some(m) = llmshim::models::spec("openai/gpt-6-sol") {
    println!("{}: reasoning = {:?}", m.label, m.capabilities.reasoning);
}
}

spec() accepts a full id ("openai/gpt-6-sol") or a bare name ("gpt-6-sol"). It also retains historical metadata for explicit lookups; those models are absent from the advertised list. None means no metadata is recorded. Specs remain a point-in-time snapshot pinned by the crate version.

Routing beyond the catalog

Pruning the advertised list does not remove provider adapters or their compatibility behavior. Older explicit OpenAI and Anthropic IDs still reach their provider, subject to upstream availability. The ChatGPT subscription route accepts only the current GPT-6 models. Known historical IDs also remain selectable by exact ID in the CLI, and spec() keeps their metadata.

The Router does not check explicit model names against this registry. If a provider is registered, provider/arbitrary-model-id is routed to that provider with arbitrary-model-id unchanged. A bare, unregistered model name works only when its prefix identifies a provider (gpt, o1, o3, o4, claude, gemini, or grok).

The ChatGPT provider is the exception: it validates requests against the four supported subscription models above.

OpenRouter is intentionally not enumerated above — its catalog is large and dynamic. Any openrouter/<vendor>/<model> slug routes through (e.g. openrouter/anthropic/claude-sonnet-5, openrouter/meta-llama/llama-3.1-70b-instruct:nitro); the slug's internal slash and :variant suffix are preserved. Because its slugs collide with other providers' prefixes, OpenRouter has no bare-model inference — always address it explicitly as openrouter/….

Self-hosted vLLM / SGLang are likewise not enumerated. Set VLLM_BASE_URL or SGLANG_BASE_URL (with an optional *_API_KEY) and address the served model as vllm/<served-model> or sglang/<served-model> — e.g. sglang/Qwen/Qwen3.6-35B-A3B-FP8. Local vs remote is just the base-URL value.

Rust applications can also define one-level Router aliases with Router::alias. Those aliases are not part of the static registry and are not configured by the stock CLI or proxy. See Models and the Router.

Configuration reference

llmshim reads provider keys from environment variables. The CLI and proxy can also load ~/.llmshim/config.toml, filling only variables that are not already set. Therefore environment variables take precedence over the file.

Provider keys

The config file shape is:

[keys]
openai = "..."
anthropic = "..."
gemini = "..."
xai = "..."

[proxy]
host = "0.0.0.0"
port = 3000

Manage it without editing TOML by hand:

llmshim configure
llmshim set anthropic '...'
llmshim get anthropic
llmshim list
llmshim path

Valid set/get keys are openai, anthropic, gemini, xai, proxy.host, and proxy.port. Displayed API keys are masked.

Router::from_env() reads environment variables and discovers the ChatGPT OAuth cache. A Rust application that wants the file behavior must call llmshim::env::load_all() before constructing the Router. See Models and the Router.

ChatGPT OAuth

Run llmshim login chatgpt and complete the printed device-code login before using chatgpt/<model>. llmshim login chatgpt --status checks the cache locally; llmshim logout chatgpt removes it without revoking the upstream session. Device-code login may need enabling in ChatGPT security settings or workspace permissions. See OpenAI authentication.

Tokens use LiteLLM's flat JSON format (access_token, refresh_token, id_token, expires_at, account_id). The default cache is separate from LiteLLM and Codex. Set CHATGPT_AUTH_FILE to an absolute Codex CLI auth.json path to share that login. Nested Codex tokens are read and refreshed in place; unknown fields are retained, last_refresh is updated as a UTC timestamp, and tokens.expires_at retains the refreshed expiry for subsequent reads. Refreshes are serialized across processes sharing that file and saved atomically; new token files have Unix mode 0600. An expired or unreadable session returns an error, never a background interactive login.

On Unix, reads of the default cache also repair owned directory modes to 0700 and the file mode to 0600. Symlinks, additional hard links, nonregular files, and paths owned by another user are refused before token parsing. Already-private handles do not require a chmod. Explicit CHATGPT_TOKEN_DIR or CHATGPT_AUTH_FILE overrides remain operator-managed and are read without this default-path permission repair; secure those locations separately.

The router registers ChatGPT when the selected cache file exists. Create a new router or restart the proxy after the first login. Once registered, requests read the cache each time, so refreshed or replaced tokens need no restart. Container users must mount the cache directory writable, including space for its lock and temporary files. The stock Docker helper does not mount it.

Proxy listener

The environment overrides [proxy]. The proxy has no built-in authentication or TLS; the bind address is not a security boundary by itself. See Deploy the proxy safely. When LLMSHIM_TRUSTED_ORIGINS is unset, requests with an Origin header are rejected before dispatch; SDK requests without one are unchanged.

The ordinary proxy and gateway commands acquire a connection slot before accepting a socket and retain it through the response body and connection close. Excess sockets wait in the operating system's listen backlog. Invalid, zero, or unrepresentable limit values retain the defaults. The first-header deadline also covers silent sockets and incomplete HTTP/2 prefaces; it ends when the application receives the first request headers, so valid inference and streams can outlast it.

HTTP/1 retains keepalive with a 64 KiB read buffer and at most 100 headers. HTTP/2 advertises a 64 KiB header-list limit and at most 128 concurrent streams per connection, with 30-second keepalive probes and a 10-second acknowledgement timeout. These transport controls are separate from request-body upload limits, logical request lifetimes, and provider concurrency. Managed client processes retain their separate TLS listener limits. Rust applications serving the exported routers through their own listener must configure equivalent transport limits.

Retries

The retry policy is resolved when the shared client is initialized. See Errors and retries.

Upstream attempt deadlines

Values are resolved when ShimClient is created. Zero, invalid, and unrepresentable values retain the finite default. Rust callers can instead pass checked AttemptDeadlines through ShimClient::with_attempt_deadlines.

These clocks bound a physical provider send, its response body, normalization, accounting, and attempt-policy callbacks. The proxy's separate logical clocks below cover the complete HTTP operation. Gateway response and job lifetimes are separate. The distributed gateway's existing 120-second request timeout is unchanged and may be the narrower boundary.

Proxy admission and rate limits

<PROVIDER> is OPENAI, ANTHROPIC, GEMINI, or XAI. Per-provider values override their global dimension; an omitted dimension inherits its global value. Redis coordination requires a binary built with redis-coordination. See Scaling and rate limits.

The proxy logical clock includes request decoding, provider preparation, retries, repair and fallback, native response conversion, and final response body production. /v1/chat/stream selects the stream clock immediately. Routes whose mode comes from JSON begin on the unary clock while the body is read and extend to the stream deadline, anchored to the original admission time, only after canonical stream:true has been parsed. Zero, invalid, and unrepresentable values retain the finite defaults.

Inbound upload parsing consumes this total clock. It is not a separate upload header, body-idle, or per-chunk timeout; deployments still need their HTTP front end to enforce those narrower upload controls.

Each built-in provider attempt checks this logical clock after provider request preparation, before and after attempt-policy admission, and again immediately before initiating the HTTP request. If successful admission finishes after the deadline, the attempt is abandoned through the existing conservative policy accounting path. The check prevents a send once built-in code observes expiry; it cannot preempt synchronous code or retract a kernel operation that was already authorized while the clock was live.

Expiry before response headers returns 504 in the selected route's JSON shape. After SSE headers, expiry emits the route's error event when the prior event ended at a valid boundary; expiry in a partial event ends the body with a transport error. A unary response whose headers are already committed also ends with a transport error.

Gateway local job lifetimes

Values must be positive and form a finite deadline; otherwise the finite default remains active. The HTTP clock starts after authentication and ingress preparation admission. Its prequeue permit still ends when the scheduler or distributed queue accepts the job. For the local scheduler, the same absolute deadline spans queueing, provider preparation, retries, repair/fallback, stream opening, forwarding, and final HTTP-body production. Direct Rust Scheduler callers receive an equivalent submission-anchored job clock.

A local stream reserves one bounded channel slot for a terminal timeout error, so a retained unpolled receiver can resume through buffered chunks and observe the timeout instead of clean EOF. Dropping unary or stream receivers cancels cooperative local dispatch and releases scheduler capacity. Custom Dispatch implementations remain trusted code: the scheduler bounds their cooperative future and owned resources, but cannot preempt synchronous blocking code or provide the built-in client's pre-send guarantee inside that code.

Distributed HTTP origins use the outer HTTP clock to bound origin resources. That does not by itself cancel an already accepted remote worker job; worker lifetime and origin-to-worker cancellation use the distributed protocol's separate controls.

Parsed JSON limits

Network JSON keeps its existing decoded-byte limits and also has mandatory finite parse budgets. Unary provider responses allow 65,536 values and about 40 MiB of estimated owned JSON; each SSE event allows 16,384 values and 16 MiB; inbound HTTP requests allow 32,768 values and 8 MiB. All profiles retain a maximum depth of 128. Complexity failures return a content-free upstream 502 or stream error, and inbound requests receive route-correct 413 framing.

The owned-byte figure is an application estimate for Value nodes, strings, keys, arrays, and map entries. The transport input, decoder scratch space, allocator metadata, and later derived output can coexist with that estimate, so it is not a process RSS promise. Limits reject the whole value and never truncate JSON into a successful response.

JSONL request logging

Set LLMSHIM_LOG to append one JSON object per completed request to a file:

LLMSHIM_LOG=./llmshim.jsonl llmshim proxy

Interactive chat also accepts an explicit path, which wins over the variable:

llmshim chat --log ./chat.jsonl

Each line contains ts, model, provider, latency_ms, token counts, status, and optional error and request_id. Logging is local to the process; coordinate collection and retention in your deployment platform.

CLI reference

Run llmshim with no subcommand, or use llmshim help, --help, or -h, to print top-level help. Interactive chat starts only with llmshim chat.

Availability: Core commands: default build · proxy: requires --features proxy · Docker commands: require Docker

Commands

Valid config keys for set and get are openai, anthropic, gemini, xai, proxy.host, and proxy.port.

Interactive chat

The model picker shares the curated catalog used by llmshim models and the server's /v1/models endpoint. An exact historical model ID with recorded metadata can still be selected explicitly, but is omitted from the picker.

llmshim chat opens a model picker. Pressing Enter without a selection chooses openai/gpt-6-astra (the first advertised entry). Every answer streams, requests use reasoning_effort: "high", and reasoning text is rendered dimly before answer text.

Chat output, provider errors, pasted-text echoes, and plain-text catalog listings render terminal control characters as visible escapes. Newlines, tabs, and ordinary Unicode remain readable. This display protection preserves the original conversation content and JSON data; provider text cannot supply terminal commands such as clipboard updates or cursor movement.

The chat process owns and resends its current history. Switching models changes the next route without clearing that history.

Existing image paths can also appear inline in a prompt. In an interactive terminal, Ctrl-V pastes an image when the platform clipboard integration can read one; otherwise it pastes text.

After llmshim login chatgpt, choose a ChatGPT entry in the picker or enter one of the four supported IDs: chatgpt/gpt-6-astra, chatgpt/gpt-6-sol, chatgpt/gpt-6.1-sol, or chatgpt/gpt-6-luna. /model uses the same list.

--log <path> appends JSONL request records. If it is absent, chat checks LLMSHIM_LOG.

Proxy

llmshim proxy loads file-backed keys, requires at least one configured provider, and listens using --host/--port, then LLMSHIM_HOST/LLMSHIM_PORT, then saved configuration, in that order. The host must be an IPv4 or IPv6 address. --port 0 asks the OS for an available port; the startup banner prints that port. The gateway accepts the same options. A default-feature Cargo build prints an error for these server commands; build with --features proxy or --features gateway.

Every subcommand accepts --help without starting a server, loading credentials, or entering an interactive prompt. Unknown arguments, missing option values, and invalid ports exit with status 2. Bind failures exit with a readable diagnostic and status 1 instead of a panic.

See HTTP API and Deploy the proxy safely.

Docker helper

docker start passes configured provider keys into the container and maps the chosen host port to container port 3000. The helper manages only the fixed image and container names above; it is not a deployment orchestrator.

Errors and retries

There are two decisions to make when a request fails: how the current surface reports it, and whether repeating the request is safe.

Rust error type

Rust APIs return llmshim::error::Result<T>, whose error is ShimError:

llmshim::completion returns one final error. llmshim::stream can fail while opening the stream, and each yielded item is also a Result<String> because a failure can occur after streaming begins.

What the shared client retries

Provider requests do not follow HTTP redirects. A 3xx response returns a ProviderError with that status; configure the final endpoint URL directly. This keeps prompts and provider authentication headers at the configured destination for both completion and streaming requests.

Non-streaming native normalizers require supported terminal metadata. Missing, malformed, nonterminal or unrecognized status/finish-reason values produce a ProviderError with status 502 and a fixed diagnostic, rather than labeling partial text as a successful stop. The diagnostic excludes response text and the provider-supplied reason. Known completion, length, filtering and tool mappings remain supported. Streaming transformations are unchanged.

The shared provider client automatically retries:

• transport connect, timeout, request, and body failures;
• HTTP 429, 500, 502, 503, 504, and 529.

By default it performs three retries after the initial request, for at most four attempts. It honors usable Retry-After and recognized provider reset headers, otherwise uses exponential backoff with jitter. Configure the count and wait cap with LLMSHIM_MAX_RETRIES and LLMSHIM_MAX_BACKOFF_SECS.

Other statuses are terminal at this layer. In particular, 400, 401, 403, and 404 normally indicate a request, credential, permission, or route problem; retrying the same request will not repair them.

Non-streaming fallback has its own policy after the shared-client attempt. It can retry/fall through on transport errors and 429, 500, 502, 503, or 529; it does not add 504 to its default fallback list. See Fallback chains.

Proxy JSON errors

Before streaming begins, proxy errors use an HTTP status and a stable envelope:

{
  "error": {
    "code": "unknown_provider",
    "message": "Unknown provider or model: example"
  }
}

The proxy-generated 429 rate_limited and 503 overloaded responses include Retry-After in whole seconds. An upstream provider's 429 or 503 status is preserved, but its final response is not guaranteed to include that header.

Malformed JSON or a body that cannot be deserialized can be rejected by the HTTP framework before llmshim's error mapping runs; clients should use the HTTP status as well as the JSON body.

Errors during SSE

Admission happens before the SSE response is committed, so proactive 429 and 503 failures are ordinary JSON HTTP responses. Once streaming has begun, a failure is a final typed event:

event: error
data: {"type":"error","message":"API key is invalid."}

Do not expect an HTTP status change after headers have been sent. Consumers must handle both the initial HTTP response and error events in the stream. When the proxy logical lifetime expires after SSE headers, it emits the native or compact timeout event only at a complete SSE boundary. If an event is partially framed, the body ends with a transport error so the timeout cannot be misread as part of the unfinished data: field.

JSON and SSE messages on the proxy, gateway, and native endpoints share one normalizer. Recognized upstream error envelopes are unwrapped, and internal provider error (NNN):, stream error:, and upstream error: display prefixes are removed. Ordinary text and malformed envelopes retain their content. Nested JSON unwrapping is bounded to four levels. Fallback failures summarize each provider's readable message.

SSE errors may also include an optional error object with the original message, type, code, parameter and status. message remains the display field. Native endpoints retain this metadata when constructing their own error envelope. Malformed tool_calls must be an array or null; other shapes return local HTTP 400 before upstream dispatch, including requests to the streaming endpoints.

API references

Use the reference that matches your boundary:

The proxy specification is OpenAPI 3.1 and currently reports llmshim version 0.1.26. It describes the four HTTP endpoints, compact request and response schemas, typed SSE events, and documented HTTP errors. It does not describe the Rust serde_json::Value contract or CLI workflow.

Open api/openapi.yaml in any OpenAPI 3.1-compatible viewer for interactive schema browsing, or pass it to a compatible generator as the starting point for another HTTP client. Generated code still needs your deployment's gateway authentication and must respect the SSE and Retry-After behaviors described in HTTP API and Errors and retries.