One conversation shape. The model is the routing address.

Write a conversation once:

{
  "model": "openai/gpt-5.6-sol",
  "messages": [
    { "role": "user", "content": "Explain ownership in one sentence." }
  ]
}

To send the same conversation to another provider, change one line:

- "model": "openai/gpt-5.6-sol"
+ "model": "anthropic/claude-opus-4-8"

llmshim selects the provider, translates the request into that provider's native API, sends it, and translates the result back. Change the model, not the conversation. That is the central idea.

At its core, llmshim is a Rust library. The CLI and HTTP proxy wrap the same library. The Python, TypeScript, Go, and Ruby clients are thin clients for the proxy; Python and TypeScript can also start a bundled proxy automatically.

flowchart LR
    Rust[Rust application] --> Core[llmshim Rust engine]
    Terminal[Terminal user] --> CLI[llmshim chat]
    CLI --> Core
    HTTP[HTTP callers] --> Proxy[llmshim proxy]
    Clients[Language clients] --> Proxy
    Proxy --> Core
    Core --> Providers[OpenAI / Anthropic / Gemini / xAI]

A translation boundary

Think of llmshim like a compiler with provider-specific backends. The input is an OpenAI Chat Completions-style conversation. The model string selects a backend. That backend translates the request to the provider's native API and normalizes the provider's response.

flowchart LR
    Request[Model + messages + controls] --> Router[Resolve model address]
    Router --> Outbound[Translate request]
    Outbound --> API[Provider-native API]
    API --> Inbound[Translate response]
    Inbound --> Result[Normalized result]

Three rules define the boundary:

1. Common features are translated where the target supports them. Messages, tools, images, streaming, and reasoning controls all pass through a provider-specific adapter.
2. Different APIs do not become identical. A provider may support fewer reasoning levels, require another image representation, or omit a feature. llmshim maps, clamps, or omits fields according to the target.
3. Native controls remain available. Use x-openai, x-anthropic, or x-gemini when a portable control is not enough. There is no x-xai namespace.

llmshim is not an agent framework. It does not execute tools or own conversation memory. Your application remains responsible for both.

Next, choose the surface that fits your application, or see how translation flows in more detail.

Choose a surface

Every llmshim surface reaches the same Rust translation engine. The choice is where that engine runs and who manages its process.

Availability: Rust: in-process · CLI: interactive · HTTP: separate proxy · Clients: through the proxy

A quick decision

• Building a Rust application? Use the crate. It is the engine and has no local server hop.
• Exploring models from a terminal? Use llmshim chat. The CLI keeps the current conversation in memory and streams the answer.
• Need HTTP or a shared deployment? Run the proxy. It exposes llmshim's own compact API, not an OpenAI-compatible endpoint.
• Using Python or TypeScript? Start with the language client. Its bundled binary starts a local proxy on the first call. TypeScript can also connect to a proxy URL you provide; the current Python API always manages its own proxy.
• Using Go or Ruby? Start a proxy first, then connect with the pure HTTP client.

These are process choices, not different translation implementations. The CLI and proxy wrap the crate, and all four language clients speak the proxy contract. See Two contracts, one engine for the data shapes at each boundary.

Whichever path you choose, configure at least one provider before making a request.

Configure providers

llmshim needs a key only for the provider that handles a request. Configure one provider to begin; add others when you want to switch models.

Availability: Rust: environment, optional config load · CLI/proxy: environment + config file · Clients: the proxy owns provider keys

Option 1: environment variables

Set one or more variables in the environment of the process that runs llmshim:

export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...
export GEMINI_API_KEY=AIza...
export XAI_API_KEY=xai-...

The Rust crate reads these variables when Router::from_env() is constructed. The CLI and proxy read them at startup.

Option 2: ~/.llmshim/config.toml

Use the interactive command to create or update the shared config file:

llmshim configure

It prompts for all four provider keys, the proxy host, and the proxy port. Press Enter to keep an existing value. The resulting file is used by CLI chat and the proxy.

You can also manage individual values:

llmshim set anthropic sk-ant-...
llmshim get anthropic
llmshim list

Valid key names are openai, anthropic, gemini, xai, proxy.host, and proxy.port. get and list mask stored API keys. Be aware that a value passed to set may remain in your shell history; the interactive command avoids that shell-history exposure.

Precedence: environment wins

The CLI and proxy call llmshim::env::load_all(). It reads ~/.llmshim/config.toml, but fills only variables that are not already set:

environment variable  >  config file  >  not configured

This makes a shell, container, or deployment secret override the local file without editing it.

Router::from_env() does not load the file on its own. A Rust application that wants the same behavior must call llmshim::env::load_all() first. See Environment variables versus config.toml.

Check what is available

llmshim models

The command lists registry entries only for providers whose keys are available after applying the precedence above. Runtime discovery is the canonical way to see the current curated model list.

Go and Ruby clients never receive provider keys directly; they connect to a proxy that owns them. Python and TypeScript auto-started proxies inherit the current environment and use the same config-file loading behavior.

Rust quickstart

Use the crate when the application making LLM requests is already written in Rust. The translation engine runs in-process; there is no proxy to start.

Availability: Rust: direct engine API · CLI/HTTP/clients: not used in this path

1. Add the dependencies

cargo add llmshim tokio serde_json

tokio and serde_json are direct dependencies of your application. llmshim does not re-export them.

Set at least one provider key before running the program:

export ANTHROPIC_API_KEY=sk-ant-...

2. Send a completion

use serde_json::json;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let router = llmshim::router::Router::from_env();

    // The public request contract is a serde_json::Value.
    let request = json!({
        "model": "openai/gpt-5.6-sol",
        "messages": [
            {"role": "user", "content": "What is Rust in one sentence?"}
        ],
        "max_tokens": 128
    });

    let response = llmshim::completion(&router, &request).await?;
    let text = response["choices"][0]["message"]["content"]
        .as_str()
        .unwrap_or("");

    println!("{text}");
    Ok(())
}

Run it with cargo run. Router::from_env() registers the providers whose environment variables are present. Change only the model address to route the same conversation elsewhere.

The result uses an OpenAI Chat Completions-style shape, regardless of which provider answered. The assistant text is at response["choices"][0]["message"]["content"].

3. Stream content

Streaming uses the StreamExt trait, so add futures as a direct dependency:

cargo add futures

This complete example prints text deltas as they arrive:

use futures::StreamExt;
use serde_json::json;
use std::io::{self, Write};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let router = llmshim::router::Router::from_env();
    let request = json!({
        "model": "anthropic/claude-sonnet-5",
        "messages": [
            {"role": "user", "content": "Write a haiku about Rust."}
        ],
        "max_tokens": 128
    });

    let mut stream = llmshim::stream(&router, &request).await?;

    while let Some(chunk) = stream.next().await {
        let chunk = chunk?;
        let parsed: serde_json::Value = serde_json::from_str(&chunk)?;

        if let Some(text) = parsed
            .pointer("/choices/0/delta/content")
            .and_then(|value| value.as_str())
        {
            print!("{text}");
            io::stdout().flush()?;
        }
    }

    println!();
    Ok(())
}

Each item is a JSON string in the normalized Chat Completions delta shape. A chunk can carry something other than text, such as reasoning, usage, or a tool call, so production code should inspect the fields it needs.

For config-file loading, aliases, and model inference, see Models and the Router. Runnable repository examples are available in examples/chat.rs and examples/stream.rs.

CLI quickstart

Use the CLI for an interactive, streaming conversation from a terminal. It can also configure llmshim and start the HTTP proxy.

Availability: Rust: wrapped by the CLI · CLI: interactive workflow · HTTP: available through llmshim proxy · Clients: separate

1. Install the binary

On macOS with Homebrew:

brew install sanjay920/tap/llmshim

Or install from source. The proxy feature keeps the proxy command available:

cargo install llmshim --features proxy

Running llmshim with no subcommand prints the command overview.

2. Configure a provider

llmshim configure

You can use environment variables instead. See Configure providers for precedence and the four supported key names.

3. Start chatting

llmshim chat

Choose a model at the prompt, then enter a message. The CLI streams each answer as it arrives. It requests reasoning_effort: "high" on every turn; reasoning returned by the provider is shown in dim gray before the answer.

The CLI keeps conversation history in memory for this process. Switch the next turn to another provider without clearing that history:

/model

Use /clear when you want a new conversation.

Attach an image

Attach a local image before the next message:

/image ./diagram.png

The CLI reads the file, converts it to a base64 data URI, and adds it to the next user message. /paste attempts to attach an image from the system clipboard on supported desktop environments.

Common operational tasks

llmshim models               # models for configured providers
llmshim set proxy.port 8080  # update one config value
llmshim get proxy.port       # read one config value
llmshim list                 # show masked keys and proxy settings
llmshim proxy                # start the HTTP API

This page covers the normal workflow. The exhaustive command and interactive slash-command list belongs in the CLI reference.

HTTP proxy quickstart

The proxy puts a network boundary around the Rust engine. Use it from any language that can send JSON over HTTP.

Availability: Rust: engine inside the server · CLI: starts the server · HTTP: compact llmshim API · Clients: connect to this API

The proxy has its own compact contract. It is not an OpenAI-compatible proxy.

1. Install a proxy-enabled binary

The proxy is behind the crate's proxy feature when building from source:

cargo install llmshim --features proxy

The Homebrew binary also includes the proxy:

brew install sanjay920/tap/llmshim

Configure at least one provider before starting it:

llmshim configure

2. Start the server

llmshim proxy

The default bind address is 0.0.0.0:3000. Override it for the process with LLMSHIM_HOST and LLMSHIM_PORT:

LLMSHIM_HOST=127.0.0.1 LLMSHIM_PORT=8080 llmshim proxy

The examples below assume the default port and use localhost to reach it.

3. Send a chat request

curl http://localhost:3000/v1/chat \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "openai/gpt-5.6-sol",
    "messages": [
      {"role": "user", "content": "What is Rust in one sentence?"}
    ],
    "config": {
      "max_tokens": 128
    },
    "provider_config": {
      "x-anthropic": {
        "disable_1m_context": true
      }
    }
  }'

config contains portable controls. provider_config is optional and merges provider-specific fields into the core request; omit it when you do not need a native control.

The non-streaming response uses llmshim's compact ChatResponse:

{
  "id": "msg_...",
  "model": "gpt-5.6-sol",
  "provider": "openai",
  "message": {
    "role": "assistant",
    "content": "Rust is a systems programming language..."
  },
  "usage": {
    "input_tokens": 15,
    "output_tokens": 12,
    "total_tokens": 27
  },
  "latency_ms": 640
}

IDs, text, token counts, and timing vary by request. reasoning and message.tool_calls appear only when the provider returns them.

Set top-level "stream": true on the same request to receive typed SSE events instead of a JSON response. POST /v1/chat/stream is the always-streaming equivalent.

See the HTTP API for the full contract.

Do not expose the bare proxy publicly

The server has permissive CORS and provides no authentication or TLS. For local use, bind it to 127.0.0.1. For a public deployment, put it behind an external gateway that supplies authentication and TLS. The supported topology and deployment checklist are covered in Deploy the proxy safely.

Language-client quickstarts

All four language clients speak the proxy's compact HTTP contract. The key difference is whether the package starts that proxy for you.

Availability: Python/TypeScript: bundled proxy, auto-start by default · Go/Ruby: pure HTTP, running proxy required

The auto-started Python and TypeScript proxies are reused within the current process and stopped when it exits. TypeScript can instead connect to a proxy you already run by setting baseUrl; the current Python public API always uses its managed local proxy. Go and Ruby never spawn the Rust binary; start llmshim proxy before using their default clients.

Provider keys belong to the proxy process, not to the HTTP client. Configure them through environment variables or llmshim configure.

Python

pip install llmshim

import llmshim

response = llmshim.chat("gpt-5.6-sol", "What is Rust?")
print(response["message"]["content"])

The first call starts the bundled proxy automatically.

Python README · PyPI package

TypeScript / JavaScript

npm install llmshim

import { Client } from "llmshim";

const client = new Client();
const response = await client.chat({
  model: "openai/gpt-5.6-sol",
  messages: [{ role: "user", content: "What is Rust?" }],
});

console.log(response.message.content);

Constructing the client is synchronous. Its first request starts the bundled proxy. Pass baseUrl to new Client(...) to use an existing server instead.

TypeScript README · npm package

Go

go get github.com/sanjay920/llmshim/clients/go

package main

import (
	"context"
	"fmt"
	"log"

	llmshim "github.com/sanjay920/llmshim/clients/go"
)

func main() {
	client := llmshim.New()
	response, err := client.Chat(context.Background(), llmshim.ChatRequest{
		Model: "openai/gpt-5.6-sol",
		Messages: []llmshim.Message{
			{Role: "user", Content: "What is Rust?"},
		},
	})
	if err != nil {
		log.Fatal(err)
	}

	fmt.Println(response.Message.Content)
}

llmshim.New() targets http://localhost:3000; it does not start the proxy.

Go README · Go module

Ruby

gem install llmshim

require "llmshim"

response = Llmshim.chat(
  model: "openai/gpt-5.6-sol",
  messages: "What is Rust?"
)

puts response.message.content

The module-level client targets LLMSHIM_BASE_URL when set, otherwise http://localhost:3000. It does not start the proxy.

Ruby README · RubyGems package

These snippets cover one non-streaming request. Use the linked client README as the canonical reference for streaming, errors, configuration, and native types rather than relying on duplicated method lists here.

How translation flows

Translation happens twice: once before the network request and once after it.

sequenceDiagram
    participant Caller
    participant Boundary as Rust API / proxy adapter
    participant Router
    participant Adapter as Provider adapter
    participant API as Native provider API

    Caller->>Boundary: model + messages + controls
    Boundary->>Router: resolve(model)
    Router-->>Boundary: provider + raw model name
    Boundary->>Adapter: transform_request(model, value)
    Adapter->>API: provider-native HTTP request
    API-->>Adapter: provider-native response
    Adapter-->>Boundary: transform_response(model, response)
    Boundary-->>Caller: normalized result

1. Resolve the model address

Every request contains a model string. Router::resolve turns that string into two things:

• a registered provider adapter;
• the raw model name to send to that provider.

For example, anthropic/claude-sonnet-5 selects the adapter registered as anthropic and passes claude-sonnet-5 to it. A bare model name can use prefix inference instead. Models and the Router covers the exact rules.

Resolution chooses an adapter. It does not check the static model-discovery list or choose a model based on prompt content.

2. Translate the outbound request

The Rust API receives the request as serde_json::Value. There is no canonical Rust request struct between the caller and the providers. The selected adapter's transform_request method reads the fields it understands and builds the provider-native URL, headers, and JSON body.

This is where an OpenAI-style messages array becomes an Anthropic Messages request, Gemini contents, or an OpenAI/xAI Responses request. Common controls are translated here too.

3. Send the native HTTP request

The shared HTTP client sends the translated request to the selected provider. Provider authentication belongs to this outbound request; callers do not send one provider's key to another provider.

4. Translate the inbound response

The same adapter's transform_response method converts the provider-native response into llmshim's normalized OpenAI Chat Completions-style value. Text, tool calls, usage, and provider-returned reasoning are placed into that common shape when present.

The proxy then performs one additional conversion into its compact ChatResponse. This is why the Rust and proxy APIs share semantics without sharing the same response envelope.

Streaming follows the same boundary

Streaming changes the inbound step, not the model. Each provider emits its own SSE event format. The adapter's third method, transform_stream_chunk, converts individual provider events into normalized chunks. The Rust API yields those chunks directly; the proxy converts them again into typed content, reasoning, tool_call, usage, done, and error events.

Two contracts, one engine

llmshim exposes one set of capabilities through two data contracts. The Rust crate uses flexible OpenAI-shaped JSON. The HTTP proxy uses a smaller request and response envelope designed for llmshim's clients.

The contracts are related, but they are not wire-compatible.

Rust: OpenAI-shaped Value

llmshim::completion and llmshim::stream accept a serde_json::Value. Conversation fields and controls live at the top level:

#![allow(unused)]
fn main() {
let request = serde_json::json!({
    "model": "anthropic/claude-sonnet-5",
    "messages": [{ "role": "user", "content": "Hello" }],
    "max_tokens": 200,
    "reasoning_effort": "high"
});
}

A non-streaming result uses an OpenAI Chat Completions-style shape. The answer is normally at choices[0].message.content; translated tool calls live beside it, and provider-returned reasoning uses reasoning_content when available.

Rust streaming yields JSON strings in the normalized Chat Completions chunk shape. It does not yield the proxy's typed event objects.

Proxy: a compact llmshim envelope

The proxy accepts model and messages, then groups portable controls under config:

{
  "model": "anthropic/claude-sonnet-5",
  "messages": [{ "role": "user", "content": "Hello" }],
  "config": {
    "max_tokens": 200,
    "reasoning_effort": "high"
  }
}

A non-streaming proxy response is a compact ChatResponse:

{
  "id": "...",
  "model": "claude-sonnet-5",
  "provider": "anthropic",
  "message": { "role": "assistant", "content": "..." },
  "usage": {
    "input_tokens": 8,
    "output_tokens": 12,
    "total_tokens": 20
  },
  "latency_ms": 640
}

When the provider returns reasoning content, the response also includes the optional reasoning field.

The streaming endpoints emit typed SSE events rather than Chat Completions chunks. Language clients decode the same HTTP contract into their native types.

Where this goes

Use this placement convention throughout the documentation:

The CLI is a workflow, not a fourth JSON contract. It builds requests, keeps history for the current process, and renders results for a person.

Portable core, native edges

Portable does not mean every provider has the same API. It means you can express the common intent once and let the selected adapter handle the provider's wire format.

What the portable layer does

Provider adapters recognize the common conversation fields they support. They translate message roles and content, common generation controls, tool definitions and results, image blocks, streaming events, and unified reasoning controls.

The request remains a serde_json::Value, but it is not arbitrary pass-through JSON. Each adapter builds a new native request from understood fields. An unknown top-level field is not automatically forwarded to every provider.

Provider differences have three possible outcomes:

• Translated: the provider supports the intent under another field or structure.
• Clamped or omitted: the provider has a smaller vocabulary or cannot accept that control.
• Sent natively: the caller deliberately uses a provider extension.

Reasoning illustrates the choice:

flowchart LR
    U[Unified effort + mode] --> P{Native override present?}
    P -- Yes --> N[x-provider native config]
    P -- No --> F[Model-family mapping]
    F --> C[Clamp to supported tier]
    C --> W[Provider-native reasoning control]

The exact reasoning mappings belong in the reasoning guide. The important concept is precedence: an explicit native control wins; otherwise llmshim maps the portable intent to what the selected model accepts.

Native extensions

Three built-in adapters expose explicit native namespaces:

There is no x-xai namespace. xAI's supported common controls are translated directly to its Responses API shape.

In the Rust contract, an extension is a top-level request key:

{
  "x-openai": {
    "reasoning": { "effort": "high", "summary": "auto" }
  }
}

In the proxy contract, put that same key inside provider_config because the proxy merges provider_config into the core request:

{
  "provider_config": {
    "x-openai": {
      "reasoning": { "effort": "high", "summary": "auto" }
    }
  }
}

Using a native extension intentionally reduces portability. A request that depends on x-openai should not be expected to retain that behavior after its model is changed to Anthropic or Gemini.

Models and the Router

The model string is a routing address. It tells llmshim which provider adapter should receive the request and which model name that adapter should send upstream.

Prefer explicit addresses

The most explicit form is provider/model:

openai/gpt-5.6-sol
anthropic/claude-opus-4-8
gemini/gemini-3.5-flash
xai/grok-4.5

The part before the first slash is the Router registration key. The remainder is sent to that provider as the model name. Explicit addresses are easiest to read and do not depend on naming conventions.

Bare-name inference

When there is no slash, llmshim lowercases the name for prefix matching while preserving the original model string:

A bare name outside those prefixes produces an unknown-provider error. Use an explicit address when inference cannot identify the provider.

Registration and discovery are different

Resolution succeeds only when the selected provider key is registered on the Router. The built-in Router::from_env() registers OpenAI, Anthropic, Gemini, and xAI only when their corresponding environment variables are present.

The static model registry powers llmshim models and GET /v1/models. Those commands are discovery aids, filtered to configured providers. The registry is not an allowlist: routing does not reject a model merely because it is absent from that list.

For that reason, this documentation does not maintain another static model table. Use runtime discovery for the current curated list.

Aliases are a Rust Router feature

Rust applications can attach a one-level alias while building a Router:

#![allow(unused)]
fn main() {
let router = llmshim::router::Router::from_env()
    .alias("smart", "anthropic/claude-opus-4-8");
}

The Router checks an alias before parsing the provider address. An alias target may be an explicit address or a bare model name, but aliases do not recursively chain. If a points to b and b points to a model, resolving a does not perform the second lookup.

Aliases are not currently configurable through the CLI, config file, proxy API, or language clients.

Environment variables versus config.toml

Router::from_env() means exactly what its name says: it reads OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY, and XAI_API_KEY. It does not read ~/.llmshim/config.toml by itself.

The CLI and proxy call llmshim::env::load_all() before constructing their Router. That function loads the config file and fills only environment variables that are not already set, so environment variables take precedence.

A Rust application that wants the same config-file behavior must request it:

#![allow(unused)]
fn main() {
llmshim::env::load_all();
let router = llmshim::router::Router::from_env();
}

Applications that manage secrets themselves can call Router::from_env() directly or construct a Router by registering provider implementations.

Conversations across models

llmshim does not store conversations. A conversation continues because the caller sends the previous messages again.

Start with a history:

[
  { "role": "user", "content": "What is a Rust closure?" },
  { "role": "assistant", "content": "A closure is..." },
  { "role": "user", "content": "Explain that another way." }
]

Send it to another provider by keeping messages and changing the address:

- "model": "anthropic/claude-opus-4-8"
+ "model": "openai/gpt-5.6-sol"

There is no session handoff between Anthropic and OpenAI. The new provider sees the history because it is present in the new request.

Who owns history?

• Rust applications keep and resend their own messages array.
• HTTP callers and language clients do the same. The proxy is stateless and does not assign conversation IDs.
• llmshim chat keeps history in memory for the interactive process. The /model command changes the model for the next request without clearing that history; /clear removes it.

Provider adapters translate recognized cross-provider artifacts before sending the history upstream. This includes common message roles and supported tool call/result shapes. They also remove provider-only response metadata that should not be sent to another API.

That translation makes a shared history usable across providers, but it does not make every provider-specific detail portable. A turn that depends on a native extension, unsupported content block, or provider-only behavior can still lose that behavior after a switch.

What model switching does not do

Changing the model does not:

• summarize or truncate history;
• run more than one model;
• choose a model automatically;
• persist messages after the caller or CLI process discards them.

Multi-model conversation is therefore a small operation: preserve the history, choose a new model address, and send the next request.

Streaming

Streaming exposes the same kinds of incremental output on every surface, but the Rust crate and the proxy encode them differently.

Availability: Rust: normalized JSON chunks · CLI: rendered live · Proxy: typed SSE · Clients: language-native stream iterators

The useful distinction is: same semantic events, two encodings.

Rust: normalized Chat Completions chunks

llmshim::stream returns a stream whose items are JSON strings. Each provider adapter has already translated its native event into an OpenAI Chat Completions-style delta.

#![allow(unused)]
fn main() {
use futures::StreamExt;
use std::io::{self, Write};

let mut stream = llmshim::stream(&router, &request).await?;

while let Some(chunk) = stream.next().await {
    let chunk = chunk?;
    let parsed: serde_json::Value = serde_json::from_str(&chunk)?;

    if let Some(reasoning) = parsed
        .pointer("/choices/0/delta/reasoning_content")
        .and_then(|value| value.as_str())
    {
        eprint!("{reasoning}");
    }

    if let Some(text) = parsed
        .pointer("/choices/0/delta/content")
        .and_then(|value| value.as_str())
    {
        print!("{text}");
        io::stdout().flush()?;
    }
}
}

Not every chunk contains text. Inspect only the fields your application needs, and keep handling Err items until the stream ends.

The Rust quickstart contains a complete runnable program.

Proxy: typed SSE events

Send the compact request to the always-streaming endpoint:

curl -N http://localhost:3000/v1/chat/stream \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "anthropic/claude-sonnet-5",
    "messages": [{"role": "user", "content": "Write a haiku."}],
    "config": {"max_tokens": 128}
  }'

The response is an SSE stream. The SSE event name and the JSON type agree:

event: reasoning
data: {"type":"reasoning","text":"..."}

event: content
data: {"type":"content","text":"Rust shapes silent thought"}

event: usage
data: {"type":"usage","input_tokens":12,"output_tokens":7,"total_tokens":19}

event: done
data: {"type":"done"}

The six event types are:

Setting "stream": true on POST /v1/chat produces the same SSE encoding. An admission error can still arrive as an HTTP error before streaming begins; an error after the stream begins arrives as an error event.

Client ergonomics

The language clients decode the typed SSE stream without changing its event vocabulary:

• Python returns an iterator of event dictionaries.
• TypeScript returns an async iterator of typed events.
• Go returns a channel of typed events.
• Ruby yields typed events to a block.

See the canonical client guides for complete loops: Python, TypeScript, Go, and Ruby.

Tool use

llmshim translates tool definitions, tool calls, and tool results. It never executes a tool. Your application owns that part of the loop.

Availability: Rust: top-level tools · CLI: no tool loop · Proxy/clients: provider_config.tools

The loop is always:

define tools → receive tool_calls → execute in your app → send tool results → continue

1. Define tools

Use the OpenAI Chat Completions function format. The function object is nested inside the tool definition:

{
  "type": "function",
  "function": {
    "name": "get_weather",
    "description": "Get the current weather for a city",
    "parameters": {
      "type": "object",
      "properties": {
        "city": {"type": "string"}
      },
      "required": ["city"]
    }
  }
}

In a Rust request, tools is a top-level field:

{
  "model": "anthropic/claude-sonnet-5",
  "messages": [{"role": "user", "content": "What is the weather in Tokyo?"}],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get the current weather for a city",
        "parameters": {
          "type": "object",
          "properties": {"city": {"type": "string"}},
          "required": ["city"]
        }
      }
    }
  ]
}

Through the proxy, place the identical array at provider_config.tools:

{
  "model": "anthropic/claude-sonnet-5",
  "messages": [{"role": "user", "content": "What is the weather in Tokyo?"}],
  "provider_config": {
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "get_weather",
          "description": "Get the current weather for a city",
          "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"]
          }
        }
      }
    ]
  }
}

Python and Ruby expose convenience tools arguments that build this provider_config field. TypeScript and Go expose provider_config directly.

2. Receive a tool call

llmshim normalizes a provider's request to call a function into the OpenAI shape:

{
  "role": "assistant",
  "content": null,
  "tool_calls": [
    {
      "id": "call_123",
      "type": "function",
      "function": {
        "name": "get_weather",
        "arguments": "{\"city\":\"Tokyo\"}"
      }
    }
  ]
}

arguments is a JSON-encoded string. Parse and validate it before calling your application code. Treat model-generated arguments as untrusted input.

3. Execute it in your application

Dispatch get_weather in your own code. llmshim does not have access to that function and does not decide whether it is safe to run.

Keep the returned assistant message in the conversation, including its tool_calls. Preserve any additional fields on those calls; for example, Gemini can return a thought_signature that its adapter needs on the next turn. Then append one tool-result message for each call:

[
  {
    "role": "assistant",
    "content": null,
    "tool_calls": [
      {
        "id": "call_123",
        "type": "function",
        "function": {
          "name": "get_weather",
          "arguments": "{\"city\":\"Tokyo\"}"
        }
      }
    ]
  },
  {
    "role": "tool",
    "tool_call_id": "call_123",
    "content": "{\"temperature_c\":24,\"conditions\":\"clear\"}"
  }
]

The tool_call_id connects the result to the request. Send the expanded history and tool definitions in another completion so the model can use the result and produce an answer.

What llmshim translates

Responses and stream chunks are translated back to the OpenAI tool_calls shape before they cross the Rust boundary. The proxy then exposes the same call as message.tool_calls or a typed tool_call stream event.

For the broader rule behind the surface-specific placement, see Two contracts, one engine.

Images and vision

Images travel inside a message's content array. llmshim translates recognized image blocks into the selected provider's native representation.

Availability: Rust/proxy/clients: message content blocks · CLI: file or clipboard attachment

Use a portable image block

The most convenient input is the OpenAI Chat Completions image_url block:

{
  "role": "user",
  "content": [
    {"type": "text", "text": "Describe this image."},
    {
      "type": "image_url",
      "image_url": {
        "url": "data:image/png;base64,iVBORw0KGgo..."
      }
    }
  ]
}

A data URI carries both the media type and base64 bytes. It is the portable choice when the same request may target OpenAI, Anthropic, Gemini, or xAI.

The current translators recognize these input block forms:

When Gemini is the target, llmshim emits Gemini's native inline_data part for base64 image bytes. A raw Gemini inline_data part is not currently recognized as a portable input block, so use one of the forms above at the llmshim boundary.

Base64 versus remote URLs

Both data URIs and plain remote URLs are accepted inside image_url and input_image blocks:

{
  "type": "image_url",
  "image_url": {"url": "https://example.com/photo.jpg"}
}

OpenAI, xAI, and Anthropic receive a provider-native URL image. llmshim does not download that URL itself.

Gemini limitation: the current Gemini adapter cannot send a remote image URL as inline image data. It replaces the image block with a text part such as [Image: https://example.com/photo.jpg]. The model receives the URL as text, not the image. Use a base64 data URI when targeting Gemini.

Send through each surface

For Rust, put the content array directly in the top-level messages value. For the proxy and language clients, use the same content array in the compact request's messages field. Image controls do not belong in config or provider_config.

The core does not read local paths. Applications must read and encode local files themselves before building the request.

The CLI provides that convenience:

/image ./diagram.png

It reads the file, builds a base64 data URI, and attaches it to the next user message. /paste attempts to attach an image from the clipboard on supported desktop environments. You can also include a readable image path directly in the text entered at the CLI prompt.

Reasoning controls

llmshim exposes one reasoning vocabulary across providers. You state the desired depth; the selected provider adapter maps it to what that model accepts.

Availability: Rust: top-level fields · CLI: high by default · Proxy/clients: fields under config

The two knobs

In a Rust request, both are top-level fields:

{
  "model": "anthropic/claude-sonnet-5",
  "messages": [{"role": "user", "content": "Solve this carefully."}],
  "reasoning_effort": "high",
  "reasoning_mode": "pro"
}

In the proxy contract, put them under config:

{
  "model": "anthropic/claude-sonnet-5",
  "messages": [{"role": "user", "content": "Solve this carefully."}],
  "config": {
    "reasoning_effort": "high",
    "reasoning_mode": "pro"
  }
}

Provider-returned reasoning is normalized as reasoning_content in Rust responses and chunks, and as reasoning or a typed reasoning event through the proxy. A provider or model may keep its reasoning hidden or return only a summary.

Portable intent or native control

Choose per request:

1. Use the unified controls. llmshim maps the requested effort and mode to the selected model family, clamping to the nearest accepted tier when the model has a smaller vocabulary.
2. Use the provider's native dialect. Pass an exact native object through x-openai, x-anthropic, or x-gemini. Native configuration wins over the unified controls.

flowchart LR
    U[Unified effort + mode] --> P{Native override present?}
    P -- Yes --> N[x-provider native config]
    P -- No --> F[Model-family mapping]
    F --> C[Clamp to supported tier]
    C --> W[Provider-native reasoning control]

See Native provider controls for passthrough examples.

Effort mapping tables

These mappings were verified against the live provider APIs. A bold value is a clamp rather than a direct name-for-name mapping.

OpenAI Responses API

OpenAI receives reasoning.effort. Legacy minimal input is also accepted: it remains native on gpt-5.5 and clamps to low where other listed families would reject it.

Anthropic Messages API

Adaptive models use thinking: {type} plus output_config: {effort}:

Adaptive models think by default even when no reasoning config is sent. reasoning_effort: "none" maps to disabled thinking and is the way to request zero thinking tokens.

Pre-4.6 models such as Haiku 4.5 and Claude 3.7 use enabled thinking with a token budget scaled from max_tokens and floored at 1024:

Gemini

Gemini uses the four-rung generationConfig.thinkingConfig.thinkingLevel enum:

Gemini 3.1 Pro rejects both minimal and thinkingBudget: 0, so none clamps to its low floor. The legacy integer thinkingBudget remains available only through x-gemini.thinkingConfig.

xAI Responses API

xAI receives the nested native shape reasoning: {effort}:

grok-4.5 cannot disable reasoning, so none clamps to low. grok-4.20 models are name-locked: reasoning on or off is encoded in the model name, and the API rejects any reasoning parameter. llmshim therefore omits it for that family.

Mode mapping: reasoning_mode: "pro"

Rules that hold across providers:

• standard is the default and is not sent on the wire.
• An explicit effort of none wins over pro; a request to turn reasoning off is not bumped back on, except where the model itself cannot disable it.
• pro without an effort lets OpenAI native-mode models select their own effort. Other models behave as a default medium bumped to high.

Precedence

1. Native passthrough under x-openai, x-anthropic, or x-gemini wins and is not translated into another dialect. Anthropic also accepts top-level thinking and output_config in the Rust contract.
2. Otherwise, unified reasoning_effort and reasoning_mode are mapped and clamped according to the tables above.
3. With neither, the provider/model default applies. Anthropic adaptive models and Gemini 3.1 Pro think by default; most other families do not.

The non-obvious boundaries above were live-probed as of July 2026 and are pinned by provider unit tests. Provider capabilities can change, so the implementation and these tables must move together.

Fallback chains

A fallback chain keeps the conversation fixed while trying an ordered list of model addresses. Retry the route, then change the route.

Availability: Rust: completion_with_fallback · CLI: not exposed · Proxy/clients: non-streaming fallback field

Fallback is useful for temporary provider failures. It is not model selection: llmshim never chooses or reorders the models for you.

Rust

The Rust FallbackConfig list contains the complete order, including the primary model first:

#![allow(unused)]
fn main() {
use llmshim::{completion_with_fallback, FallbackConfig};
use serde_json::json;

let request = json!({
    "model": "anthropic/claude-opus-4-8",
    "messages": [{"role": "user", "content": "Hello"}],
    "max_tokens": 128
});

let fallback = FallbackConfig::new(vec![
    "anthropic/claude-opus-4-8".into(),
    "openai/gpt-5.6-sol".into(),
    "gemini/gemini-3.5-flash".into(),
]);

let response = completion_with_fallback(
    &router,
    &request,
    &fallback,
    None,
).await?;
}

When FallbackConfig.models is non-empty, that list supplies the primary and all fallback addresses. The function rewrites the request's model for each attempt.

The builder also allows callers to change the fallback layer's retries per model and its initial backoff:

#![allow(unused)]
fn main() {
use std::time::Duration;

let fallback = FallbackConfig::new(models)
    .max_retries(1)
    .initial_backoff(Duration::from_millis(250));
}

Proxy and clients

The proxy keeps the primary in model; fallback contains only the addresses to try afterward:

{
  "model": "anthropic/claude-opus-4-8",
  "messages": [{"role": "user", "content": "Hello"}],
  "config": {"max_tokens": 128},
  "fallback": [
    "openai/gpt-5.6-sol",
    "gemini/gemini-3.5-flash"
  ]
}

The proxy prepends model to the supplied fallback array before calling the Rust fallback API. Language clients expose the same request field; some offer a convenience argument for it.

What triggers retry and failover

The default fallback statuses are:

429  500  502  503  529

Transport errors are retryable too. For an eligible failure, the fallback layer retries the current model with exponential backoff, doubling the delay between attempts. If the model still fails, llmshim advances to the next address. A success returns immediately. If every address fails, Rust returns ShimError::AllFailed with the collected errors; the proxy returns an all_failed error response.

Each fallback model must resolve to a provider registered on the Router. A chain can cross providers, but only when their keys are configured.

Fallback is non-streaming only

There is no streaming fallback API in the Rust crate. The proxy's streaming path calls llmshim::stream directly and does not inspect fallback.

That means both of these ignore the fallback array:

• POST /v1/chat/stream;
• POST /v1/chat with "stream": true.

Do not send fallback on a streaming request expecting failover. Implement a new stream attempt in the caller only if your application can safely decide what to do with content already emitted by the failed model.

Native provider controls

Use native controls when the portable request cannot express a provider feature precisely enough. This is an intentional trade: exact provider behavior in exchange for portability.

Availability: Rust: top-level x-* key · CLI: not exposed · Proxy/clients: provider_config["x-*"]

The built-in namespaces are:

There is no x-xai namespace. llmshim's unified reasoning control already emits xAI's supported native reasoning: {effort} shape, including its model-family clamps.

OpenAI

x-openai fields are copied into the OpenAI Responses request body. This gives access to the full native reasoning object, including effort, mode, summary, and context when accepted by the selected model.

{
  "model": "openai/gpt-5.6-sol",
  "messages": [{"role": "user", "content": "Analyze this carefully."}],
  "x-openai": {
    "reasoning": {
      "effort": "high",
      "mode": "pro",
      "summary": "auto"
    }
  }
}

The object is native OpenAI configuration. llmshim does not clamp it, so the selected model must accept the values.

Anthropic

Most x-anthropic fields become Anthropic Messages body fields. Two special keys control headers instead:

• disable_1m_context disables llmshim's automatic 1M-context beta header;
• extra_betas appends and de-duplicates caller-supplied anthropic-beta tokens.

{
  "model": "anthropic/claude-sonnet-5",
  "messages": [{"role": "user", "content": "Analyze this carefully."}],
  "x-anthropic": {
    "thinking": {"type": "adaptive"},
    "output_config": {"effort": "high"},
    "disable_1m_context": true,
    "extra_betas": ["extended-cache-ttl-2025-04-11"]
  }
}

thinking and output_config are sent in the request body. The two control keys are consumed by llmshim while building Anthropic headers and are not sent as body fields.

The Rust contract also accepts top-level thinking and output_config, but the namespace keeps provider-native fields visibly grouped.

Gemini

x-gemini.thinkingConfig replaces the unified Gemini thinking configuration. It accepts Gemini's native object, including the legacy integer thinkingBudget:

{
  "model": "gemini/gemini-3.5-flash",
  "messages": [{"role": "user", "content": "Analyze this carefully."}],
  "x-gemini": {
    "thinkingConfig": {
      "thinkingBudget": 2048,
      "includeThoughts": true
    }
  }
}

Other keys inside x-gemini are copied to the Gemini request body. As with all native controls, the provider validates their shapes and values.

Where the namespace goes

In the Rust serde_json::Value, use the namespace as a top-level key, as in the examples above.

The proxy reserves provider_config for fields that are merged into that core request. Wrap the same namespace inside it:

{
  "model": "openai/gpt-5.6-sol",
  "messages": [{"role": "user", "content": "Analyze this carefully."}],
  "provider_config": {
    "x-openai": {
      "reasoning": {
        "effort": "high",
        "mode": "pro",
        "summary": "auto"
      }
    }
  }
}

Language clients send the same object through their provider_config field. The CLI chat workflow does not expose native request configuration.

Precedence and portability

An explicit native reasoning object takes precedence over reasoning_effort/reasoning_mode. llmshim does not translate the object into another provider's dialect. Anthropic header controls are applied as headers; the other native fields are copied to their documented native destination.

If you change the model address to another provider, remove or replace the old provider namespace. Keeping x-openai in an Anthropic request does not recreate the OpenAI behavior. See Portable core, native edges for the underlying contract.

HTTP API

The proxy wraps llmshim's translation engine in a compact JSON API. It has four endpoints and two response encodings.

Availability: HTTP JSON: non-streaming chat, models, health · HTTP SSE: streaming chat

Endpoints

The canonical machine-readable contract is api/openapi.yaml.

Chat request

Both chat endpoints accept the same body:

{
  "model": "anthropic/claude-sonnet-5",
  "messages": [{"role": "user", "content": "Explain ownership briefly."}],
  "stream": false,
  "config": {
    "max_tokens": 300,
    "temperature": 0.2,
    "reasoning_effort": "medium"
  },
  "provider_config": {},
  "fallback": ["openai/gpt-5.4-mini"]
}

config accepts exactly these fields:

Adapters use only controls supported by the selected provider. Reasoning is mapped and clamped by model family; see Reasoning controls.

Messages require role; content defaults to null and may be text or content blocks. Tool turns may also carry tool_calls or tool_call_id. reasoning_content is accepted on a message when prior reasoning must be round-tripped.

provider_config is merged as top-level engine fields. For example, tools go at provider_config.tools, while an OpenAI-native override goes at provider_config["x-openai"]. See the request field map.

Fallback first retries an eligible failure on the current route, then moves through the listed routes. It is ignored by both streaming paths. See Fallback chains.

Non-streaming response

POST /v1/chat returns a compact response when stream is false:

{
  "id": "msg_123",
  "model": "claude-sonnet-5",
  "provider": "anthropic",
  "message": {
    "role": "assistant",
    "content": "Ownership gives each value one owner."
  },
  "usage": {
    "input_tokens": 14,
    "output_tokens": 9,
    "reasoning_tokens": 4,
    "total_tokens": 27
  },
  "latency_ms": 612
}

message.tool_calls appears when the model requests tools. reasoning appears when the provider returns reasoning text. reasoning_tokens is omitted when zero; the other usage fields are always present.

Streaming response

POST /v1/chat/stream always streams. POST /v1/chat does the same when the body contains "stream": true.

Each Server-Sent Event has an SSE event: name and JSON data: whose type matches that name:

event: reasoning
data: {"type":"reasoning","text":"..."}

event: content
data: {"type":"content","text":"Ownership gives each value one owner."}

event: tool_call
data: {"type":"tool_call","id":"call_1","name":"lookup","arguments":"{\"id\":7}"}

event: usage
data: {"type":"usage","input_tokens":14,"output_tokens":9,"total_tokens":23}

event: done
data: {"type":"done"}

Admission failures happen before SSE begins and return normal HTTP 429 or 503 responses. A failure after the stream starts is an error event. For consumption patterns, see Streaming.

Models and health

GET /v1/models returns the registry entries for configured providers:

{
  "models": [
    {"id": "openai/gpt-5.4-mini", "provider": "openai", "name": "gpt-5.4-mini"}
  ]
}

This is discovery, not an allowlist: an arbitrary provider model ID can still be routed explicitly. See Model discovery.

GET /health reports that the process can serve requests and names its configured providers:

{"status":"ok","providers":["openai","anthropic"]}

It does not probe upstream provider availability.

Errors

Before a stream begins, errors use an HTTP status and this JSON envelope:

{"error":{"code":"rate_limited","message":"Upstream provider rate limit reached; retry after the suggested delay"}}

Proactive 429 and 503 responses include Retry-After in whole seconds. Provider failures normally retain the provider's status. See Errors and retries for the complete mapping.

Deploy the proxy safely

The llmshim proxy has no built-in authentication and no TLS. It also sends permissive CORS headers. Do not expose it directly to an untrusted network.

The supported public topology puts an authentication and TLS gateway in front of the proxy:

flowchart LR
    C[Applications] -->|HTTPS + credentials| G[External gateway<br/>TLS + authn/authz]
    G -->|trusted network| P[llmshim proxy]
    P -->|provider API keys| O[LLM providers]

The gateway can be any reverse proxy or API gateway that terminates TLS, authenticates callers, authorizes access, and applies your network policy. Keep the llmshim listener reachable only from that trusted boundary.

Install a proxy-enabled binary

The HTTP server is behind the Rust proxy feature:

cargo install llmshim --features proxy

The Homebrew tap also installs a proxy-enabled binary:

brew install sanjay920/tap/llmshim

Then configure at least one provider key and start the process:

export ANTHROPIC_API_KEY="..."
llmshim proxy

The default address is 0.0.0.0:3000. Override it with environment variables:

LLMSHIM_HOST=127.0.0.1 LLMSHIM_PORT=8080 llmshim proxy

LLMSHIM_HOST and LLMSHIM_PORT take precedence over the proxy host and port in ~/.llmshim/config.toml. Binding to 127.0.0.1 is a useful default when a same-host gateway connects to the proxy.

Keep provider keys server-side

The proxy process reads OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY, and XAI_API_KEY. Language clients send neither those keys nor provider authorization headers. Store the keys in the process environment or the proxy's local config file, using your deployment platform's secret mechanism.

Do not bake keys into an image, return them to clients, or place them in provider_config. The latter is request data forwarded to adapters, not a secret store.

Container boundary

A minimal deployment has one llmshim process per container or service unit:

public client
    -> HTTPS/auth gateway
        -> private llmshim:3000
            -> provider APIs

Give the process outbound HTTPS access to the configured providers and inbound access only from the gateway or trusted callers. Use GET /health for process health; it confirms the proxy is running and lists configured providers, but does not call upstream APIs.

For multiple replicas and coordinated limits, continue to Scaling and rate limits. For every deployment setting, see the configuration reference.

Scaling and rate limits

The proxy is stateless per request. Put replicas behind a load balancer and send the complete conversation history on every call. No sticky session is required for llmshim itself.

flowchart LR
    C[Clients] --> L[Load balancer]
    L --> P1[Proxy replica 1<br/>pool + admission state]
    L --> P2[Proxy replica 2<br/>pool + admission state]
    L --> P3[Proxy replica N<br/>pool + admission state]
    P1 --> U[Provider APIs]
    P2 --> U
    P3 --> U
    P1 -. optional shared limits .-> R[(Redis)]
    P2 -. optional shared limits .-> R
    P3 -. optional shared limits .-> R

Each process owns its connection pool, concurrency semaphore, and—unless Redis coordination is enabled—its rate-limit buckets.

Connection reuse and warmup

All calls in one process share a lazily initialized HTTP client. Its pool uses HTTP/2 where available, gzip/brotli/zstd/deflate compression, a 90-second idle timeout, up to four idle connections per host, a 30-second TCP keepalive, and TCP NODELAY.

For Rust services, call llmshim::warmup(&router).await after constructing the Router. It sends bounded HEAD requests to configured built-in provider origins to pre-establish TCP and TLS connections. Failure to warm a provider is ignored; normal request handling remains authoritative.

Reactive retries

The shared client retries transport failures and HTTP 429, 500, 502, 503, 504, and 529. The default is three retries after the initial request—up to four attempts total—with exponential backoff and jitter. Retry-After and recognized OpenAI or Anthropic reset headers take precedence over computed backoff.

These retries stay on the same route. A non-streaming fallback chain adds a second layer: retry the route, then change the route. See Fallback chains.

Backpressure and proactive limits

Every proxy request first acquires an instance concurrency slot. Waiting longer than the queue timeout returns 503 with Retry-After.

Optional token buckets can reject work before it reaches a provider. A rejection is 429 with Retry-After.

When neither RPM nor TPM is set, proactive rate limiting is disabled; concurrency backpressure still applies. Token permits are estimates based on request size and requested output, not provider billing measurements.

One replica or a coordinated fleet

The default buckets are in memory. With N replicas, each replica enforces its own configured limit. If the number represents a fleet-wide provider quota, divide it across instances or enable shared coordination.

To share one bucket, build the opt-in feature and set Redis:

cargo install llmshim --features redis-coordination
LLMSHIM_REDIS_URL=redis://redis.internal:6379 llmshim proxy

redis-coordination includes the proxy feature. Redis is used for rate-limit coordination; connection pools and concurrency limits remain per process. If Redis becomes unavailable at runtime, limiting fails open so requests continue. If the Redis client cannot be initialized—or the binary lacks the feature—the proxy warns and falls back to in-memory buckets.

Do not infer capacity from llmshim's implementation details alone. The README benchmarks are the maintained performance snapshot; load-test your model mix, payload sizes, provider quotas, and gateway before choosing replica counts.

Surface capability matrix

The Rust crate is the engine. The CLI calls it in-process, the proxy places an HTTP boundary around it, and the language clients speak that proxy contract.

Availability: Crate: in-process Rust · CLI: human workflow · Proxy: language-neutral server · Clients: proxy SDKs

Choose the boundary

No surface stores server-side conversation memory. The caller resends history; the CLI merely does that for its current interactive session.

Where request features go

The proxy's provider_config object is merged into the OpenAI-shaped engine request. That is why tools and native namespaces move under it without changing their inner shapes.

Build features

The Homebrew package and the binaries bundled by Python and TypeScript include proxy support. Go and Ruby are pure HTTP clients and require a separately running proxy.

One typed-wrapper limitation matters for Gemini tool loops: raw proxy JSON round-trips thought_signature, but the Go and Ruby typed structs do not currently surface it. Use raw JSON if that field is required.

For shape details, continue to the request field map and HTTP API.

Request field map

Rust and the proxy express the same request semantics through different envelopes. Rust uses an OpenAI-shaped serde_json::Value; the proxy groups its portable subset under config and everything else under provider_config.

Availability: Rust: flexible top-level JSON · Proxy/clients: typed compact envelope

Rust engine request

These are the stable, adapter-recognized fields. A field marked “mapped” is renamed or reshaped for the selected provider; unsupported controls are omitted.

There is no x-xai namespace. OpenAI additionally recognizes store, prompt_cache_key, prompt_cache_retention, safety_identifier, and speed. Anthropic recognizes cache_control, thinking, output_config, and speed. Those fields are provider-specific even when they are not wrapped in an x-* namespace.

The request is a Value, so Rust does not reject unknown keys up front. Adapters deliberately select only fields they understand. Do not assume an arbitrary top-level key reaches the provider; use a documented native namespace.

Reasoning values and family-specific clamps are canonicalized in Reasoning controls. Native namespaces and precedence are covered in Native provider controls.

Message fields

Content blocks may use OpenAI image_url, Anthropic image, or Gemini inline_data input forms. See Images and vision.

Proxy request

The exact config children are:

For example, this proxy fragment:

{
  "config": {"reasoning_effort": "high"},
  "provider_config": {
    "tools": [],
    "x-anthropic": {"disable_1m_context": true}
  }
}

becomes engine fields named reasoning_effort, tools, and x-anthropic. Because provider_config is merged after config, a same-named child there overrides the portable value. Prefer the x-* namespace for an intentional native override; it makes that loss of portability visible.

Provider behavior matrix

Every adapter accepts the same OpenAI-shaped engine request, but it targets a different native API and translates only the fields that API understands.

An explicit address such as anthropic/claude-sonnet-5 avoids inference. The named provider must be registered in the Router—that normally means its API key is configured.

Observable differences

OpenAI and xAI do not receive temperature, top_p, top_k, or stop from the portable top-level request. Anthropic and Gemini do. This is the portable-core rule in practice: unsupported fields are omitted rather than forwarded blindly.

Tools and multimodal round trips

Tool calls are normalized back to the OpenAI tool_calls shape. Preserve the complete assistant tool call when sending the next turn; provider adapters may need fields beyond function name and arguments.

In particular, raw proxy JSON round-trips Gemini thought_signature, but the Go and Ruby typed structs do not currently surface it. Use raw JSON if you need that field.

Image input accepts OpenAI image_url, Anthropic image, and Gemini inline_data blocks. Base64 data URIs can be translated among providers. Gemini's remote-URL fallback is literal text such as [Image: URL]; llmshim does not download the URL.

Reasoning and native controls

Unified reasoning_effort and reasoning_mode are mapped by model family, not merely by provider. Floors, ceilings, and name-locked models are listed in the reasoning tables.

Use native provider controls when exact native semantics matter. Native objects are not rewritten or made portable. Changing the provider address means reconsidering the namespace as well.

Model discovery

Runtime discovery is the canonical answer to “what can this configured installation offer?”

llmshim models
curl http://localhost:3000/v1/models

Both commands filter the built-in registry to providers with configured API keys. The proxy returns id, provider, and unprefixed name; the CLI prints the ID and display label.

Registered catalog

The current registry contains 26 entries, newest first within each provider. This page mirrors src/models.rs; use runtime discovery rather than parsing this table in applications.

OpenAI

Anthropic

Google Gemini

xAI

Catalog is not an allowlist

The Router does not check explicit model names against this registry. If a provider is registered, provider/arbitrary-model-id is routed to that provider with arbitrary-model-id unchanged. A bare, unregistered model name works only when its prefix identifies a provider (gpt, o1, o3, o4, claude, gemini, or grok).

Rust applications can also define one-level Router aliases with Router::alias. Those aliases are not part of the static registry and are not configured by the stock CLI or proxy. See Models and the Router.

Configuration reference

llmshim reads provider keys from environment variables. The CLI and proxy can also load ~/.llmshim/config.toml, filling only variables that are not already set. Therefore environment variables take precedence over the file.

Provider keys

The config file shape is:

[keys]
openai = "..."
anthropic = "..."
gemini = "..."
xai = "..."

[proxy]
host = "0.0.0.0"
port = 3000

Manage it without editing TOML by hand:

llmshim configure
llmshim set anthropic '...'
llmshim get anthropic
llmshim list
llmshim path

Valid set/get keys are openai, anthropic, gemini, xai, proxy.host, and proxy.port. Displayed API keys are masked.

Router::from_env() reads environment variables only. A Rust application that wants the file behavior must call llmshim::env::load_all() before constructing the Router. See Models and the Router.

Proxy listener

The environment overrides [proxy]. The proxy has no built-in authentication or TLS; the bind address is not a security boundary by itself. See Deploy the proxy safely.

Retries

The retry policy is resolved when the shared client is initialized. See Errors and retries.

Proxy admission and rate limits

<PROVIDER> is OPENAI, ANTHROPIC, GEMINI, or XAI. Per-provider values override their global dimension; an omitted dimension inherits its global value. Redis coordination requires a binary built with redis-coordination. See Scaling and rate limits.

JSONL request logging

Set LLMSHIM_LOG to append one JSON object per completed request to a file:

LLMSHIM_LOG=./llmshim.jsonl llmshim proxy

Interactive chat also accepts an explicit path, which wins over the variable:

llmshim chat --log ./chat.jsonl

Each line contains ts, model, provider, latency_ms, token counts, status, and optional error and request_id. Logging is local to the process; coordinate collection and retention in your deployment platform.

CLI reference

Run llmshim with no subcommand, or use llmshim help, --help, or -h, to print top-level help. Interactive chat starts only with llmshim chat.

Availability: Core commands: default build · proxy: requires --features proxy · Docker commands: require Docker

Commands

Valid config keys for set and get are openai, anthropic, gemini, xai, proxy.host, and proxy.port.

Interactive chat

llmshim chat opens a model picker. Pressing Enter without a selection chooses anthropic/claude-sonnet-5. Every answer streams, requests use reasoning_effort: "high", and reasoning text is rendered dimly before answer text.

The chat process owns and resends its current history. Switching models changes the next route without clearing that history.

Existing image paths can also appear inline in a prompt. In an interactive terminal, Ctrl-V pastes an image when the platform clipboard integration can read one; otherwise it pastes text.

--log <path> appends JSONL request records. If it is absent, chat checks LLMSHIM_LOG.

Proxy

llmshim proxy loads file-backed keys, requires at least one configured provider, and listens using LLMSHIM_HOST/LLMSHIM_PORT or the config file. It accepts no command-line flags. A default-feature Cargo build prints an error; install or build with --features proxy.

See HTTP API and Deploy the proxy safely.

Docker helper

docker start passes configured provider keys into the container and maps the chosen host port to container port 3000. The helper manages only the fixed image and container names above; it is not a deployment orchestrator.

Errors and retries

There are two decisions to make when a request fails: how the current surface reports it, and whether repeating the request is safe.

Rust error type

Rust APIs return llmshim::error::Result<T>, whose error is ShimError:

llmshim::completion returns one final error. llmshim::stream can fail while opening the stream, and each yielded item is also a Result<String> because a failure can occur after streaming begins.

What the shared client retries

The shared provider client automatically retries:

• transport connect, timeout, request, and body failures;
• HTTP 429, 500, 502, 503, 504, and 529.

By default it performs three retries after the initial request, for at most four attempts. It honors usable Retry-After and recognized provider reset headers, otherwise uses exponential backoff with jitter. Configure the count and wait cap with LLMSHIM_MAX_RETRIES and LLMSHIM_MAX_BACKOFF_SECS.

Other statuses are terminal at this layer. In particular, 400, 401, 403, and 404 normally indicate a request, credential, permission, or route problem; retrying the same request will not repair them.

Non-streaming fallback has its own policy after the shared-client attempt. It can retry/fall through on transport errors and 429, 500, 502, 503, or 529; it does not add 504 to its default fallback list. See Fallback chains.

Proxy JSON errors

Before streaming begins, proxy errors use an HTTP status and a stable envelope:

{
  "error": {
    "code": "unknown_provider",
    "message": "Unknown provider or model: example"
  }
}

The proxy-generated 429 rate_limited and 503 overloaded responses include Retry-After in whole seconds. An upstream provider's 429 or 503 status is preserved, but its final response is not guaranteed to include that header.

Malformed JSON or a body that cannot be deserialized can be rejected by the HTTP framework before llmshim's error mapping runs; clients should use the HTTP status as well as the JSON body.

Errors during SSE

Admission happens before the SSE response is committed, so proactive 429 and 503 failures are ordinary JSON HTTP responses. Once streaming has begun, a failure is a final typed event:

event: error
data: {"type":"error","message":"stream error: ..."}

Do not expect an HTTP status change after headers have been sent. Consumers must handle both the initial HTTP response and error events in the stream.

API references

Use the reference that matches your boundary:

The proxy specification is OpenAPI 3.1 and currently reports llmshim version 0.1.26. It describes the four HTTP endpoints, compact request and response schemas, typed SSE events, and documented HTTP errors. It does not describe the Rust serde_json::Value contract or CLI workflow.

Open api/openapi.yaml in any OpenAPI 3.1-compatible viewer for interactive schema browsing, or pass it to a compatible generator as the starting point for another HTTP client. Generated code still needs your deployment's gateway authentication and must respect the SSE and Retry-After behaviors described in HTTP API and Errors and retries.