Configuration reference

llmshim reads provider keys from environment variables. The CLI and proxy can also load ~/.llmshim/config.toml, filling only variables that are not already set. Therefore environment variables take precedence over the file.

Provider keys

The config file shape is:

[keys]
openai = "..."
anthropic = "..."
gemini = "..."
xai = "..."

[proxy]
host = "0.0.0.0"
port = 3000

Manage it without editing TOML by hand:

llmshim configure
llmshim set anthropic '...'
llmshim get anthropic
llmshim list
llmshim path

Valid set/get keys are openai, anthropic, gemini, xai, proxy.host, and proxy.port. Displayed API keys are masked.

Router::from_env() reads environment variables and discovers the ChatGPT OAuth cache. A Rust application that wants the file behavior must call llmshim::env::load_all() before constructing the Router. See Models and the Router.

ChatGPT OAuth

Run llmshim login chatgpt and complete the printed device-code login before using chatgpt/<model>. llmshim login chatgpt --status checks the cache locally; llmshim logout chatgpt removes it without revoking the upstream session. Device-code login may need enabling in ChatGPT security settings or workspace permissions. See OpenAI authentication.

Tokens use LiteLLM's flat JSON format (access_token, refresh_token, id_token, expires_at, account_id). The default cache is separate from LiteLLM and Codex. Set CHATGPT_AUTH_FILE to an absolute Codex CLI auth.json path to share that login. Nested Codex tokens are read and refreshed in place; unknown fields are retained, last_refresh is updated as a UTC timestamp, and tokens.expires_at retains the refreshed expiry for subsequent reads. Refreshes are serialized across processes sharing that file and saved atomically; new token files have Unix mode 0600. An expired or unreadable session returns an error, never a background interactive login.

On Unix, reads of the default cache also repair owned directory modes to 0700 and the file mode to 0600. Symlinks, additional hard links, nonregular files, and paths owned by another user are refused before token parsing. Already-private handles do not require a chmod. Explicit CHATGPT_TOKEN_DIR or CHATGPT_AUTH_FILE overrides remain operator-managed and are read without this default-path permission repair; secure those locations separately.

The router registers ChatGPT when the selected cache file exists. Create a new router or restart the proxy after the first login. Once registered, requests read the cache each time, so refreshed or replaced tokens need no restart. Container users must mount the cache directory writable, including space for its lock and temporary files. The stock Docker helper does not mount it.

Proxy listener

The environment overrides [proxy]. The proxy has no built-in authentication or TLS; the bind address is not a security boundary by itself. See Deploy the proxy safely. When LLMSHIM_TRUSTED_ORIGINS is unset, requests with an Origin header are rejected before dispatch; SDK requests without one are unchanged.

The ordinary proxy and gateway commands acquire a connection slot before accepting a socket and retain it through the response body and connection close. Excess sockets wait in the operating system's listen backlog. Invalid, zero, or unrepresentable limit values retain the defaults. The first-header deadline also covers silent sockets and incomplete HTTP/2 prefaces; it ends when the application receives the first request headers, so valid inference and streams can outlast it.

HTTP/1 retains keepalive with a 64 KiB read buffer and at most 100 headers. HTTP/2 advertises a 64 KiB header-list limit and at most 128 concurrent streams per connection, with 30-second keepalive probes and a 10-second acknowledgement timeout. These transport controls are separate from request-body upload limits, logical request lifetimes, and provider concurrency. Managed client processes retain their separate TLS listener limits. Rust applications serving the exported routers through their own listener must configure equivalent transport limits.

Retries

The retry policy is resolved when the shared client is initialized. See Errors and retries.

Upstream attempt deadlines

Values are resolved when ShimClient is created. Zero, invalid, and unrepresentable values retain the finite default. Rust callers can instead pass checked AttemptDeadlines through ShimClient::with_attempt_deadlines.

These clocks bound a physical provider send, its response body, normalization, accounting, and attempt-policy callbacks. The proxy's separate logical clocks below cover the complete HTTP operation. Gateway response and job lifetimes are separate. The distributed gateway's existing 120-second request timeout is unchanged and may be the narrower boundary.

Proxy admission and rate limits

<PROVIDER> is OPENAI, ANTHROPIC, GEMINI, or XAI. Per-provider values override their global dimension; an omitted dimension inherits its global value. Redis coordination requires a binary built with redis-coordination. See Scaling and rate limits.

The proxy logical clock includes request decoding, provider preparation, retries, repair and fallback, native response conversion, and final response body production. /v1/chat/stream selects the stream clock immediately. Routes whose mode comes from JSON begin on the unary clock while the body is read and extend to the stream deadline, anchored to the original admission time, only after canonical stream:true has been parsed. Zero, invalid, and unrepresentable values retain the finite defaults.

Inbound upload parsing consumes this total clock. It is not a separate upload header, body-idle, or per-chunk timeout; deployments still need their HTTP front end to enforce those narrower upload controls.

Each built-in provider attempt checks this logical clock after provider request preparation, before and after attempt-policy admission, and again immediately before initiating the HTTP request. If successful admission finishes after the deadline, the attempt is abandoned through the existing conservative policy accounting path. The check prevents a send once built-in code observes expiry; it cannot preempt synchronous code or retract a kernel operation that was already authorized while the clock was live.

Expiry before response headers returns 504 in the selected route's JSON shape. After SSE headers, expiry emits the route's error event when the prior event ended at a valid boundary; expiry in a partial event ends the body with a transport error. A unary response whose headers are already committed also ends with a transport error.

Gateway local job lifetimes

Values must be positive and form a finite deadline; otherwise the finite default remains active. The HTTP clock starts after authentication and ingress preparation admission. Its prequeue permit still ends when the scheduler or distributed queue accepts the job. For the local scheduler, the same absolute deadline spans queueing, provider preparation, retries, repair/fallback, stream opening, forwarding, and final HTTP-body production. Direct Rust Scheduler callers receive an equivalent submission-anchored job clock.

A local stream reserves one bounded channel slot for a terminal timeout error, so a retained unpolled receiver can resume through buffered chunks and observe the timeout instead of clean EOF. Dropping unary or stream receivers cancels cooperative local dispatch and releases scheduler capacity. Custom Dispatch implementations remain trusted code: the scheduler bounds their cooperative future and owned resources, but cannot preempt synchronous blocking code or provide the built-in client's pre-send guarantee inside that code.

Distributed HTTP origins use the outer HTTP clock to bound origin resources. That does not by itself cancel an already accepted remote worker job; worker lifetime and origin-to-worker cancellation use the distributed protocol's separate controls.

Parsed JSON limits

Network JSON keeps its existing decoded-byte limits and also has mandatory finite parse budgets. Unary provider responses allow 65,536 values and about 40 MiB of estimated owned JSON; each SSE event allows 16,384 values and 16 MiB; inbound HTTP requests allow 32,768 values and 8 MiB. All profiles retain a maximum depth of 128. Complexity failures return a content-free upstream 502 or stream error, and inbound requests receive route-correct 413 framing.

The owned-byte figure is an application estimate for Value nodes, strings, keys, arrays, and map entries. The transport input, decoder scratch space, allocator metadata, and later derived output can coexist with that estimate, so it is not a process RSS promise. Limits reject the whole value and never truncate JSON into a successful response.

JSONL request logging

Set LLMSHIM_LOG to append one JSON object per completed request to a file:

LLMSHIM_LOG=./llmshim.jsonl llmshim proxy

Interactive chat also accepts an explicit path, which wins over the variable:

llmshim chat --log ./chat.jsonl

Each line contains ts, model, provider, latency_ms, token counts, status, and optional error and request_id. Logging is local to the process; coordinate collection and retention in your deployment platform.