Errors and retries
There are two decisions to make when a request fails: how the current surface reports it, and whether repeating the request is safe.
Rust error type
Rust APIs return llmshim::error::Result<T>, whose error is ShimError:
| Variant | Meaning |
|---|---|
UnknownProvider(String) | The model address could not identify a registered provider |
MissingModel | The request has no string model field |
Http(reqwest::Error) | Provider transport failed |
Json(serde_json::Error) | A required JSON conversion failed |
ProviderError { status, body } | A provider returned a non-success status or native error |
Stream(String) | A provider stream could not be parsed or translated |
AllFailed(Vec<String>) | Every route in a fallback chain failed |
llmshim::completion returns one final error. llmshim::stream can fail while
opening the stream, and each yielded item is also a Result<String> because a
failure can occur after streaming begins.
What the shared client retries
Provider requests do not follow HTTP redirects. A 3xx response returns a
ProviderError with that status; configure the final endpoint URL directly.
This keeps prompts and provider authentication headers at the configured
destination for both completion and streaming requests.
Non-streaming native normalizers require supported terminal metadata. Missing,
malformed, nonterminal or unrecognized status/finish-reason values produce a
ProviderError with status 502 and a fixed diagnostic, rather than labeling
partial text as a successful stop. The diagnostic excludes response text
and the provider-supplied reason. Known completion, length, filtering and tool
mappings remain supported. Streaming transformations are unchanged.
The shared provider client automatically retries:
- transport connect, timeout, request, and body failures;
- HTTP
429,500,502,503,504, and529.
By default it performs three retries after the initial request, for at most
four attempts. It honors usable Retry-After and recognized provider reset
headers, otherwise uses exponential backoff with jitter. Configure the count
and wait cap with LLMSHIM_MAX_RETRIES and LLMSHIM_MAX_BACKOFF_SECS.
Other statuses are terminal at this layer. In particular, 400, 401, 403,
and 404 normally indicate a request, credential, permission, or route problem;
retrying the same request will not repair them.
Non-streaming fallback has its own policy after the shared-client attempt. It
can retry/fall through on transport errors and 429, 500, 502, 503, or
529; it does not add 504 to its default fallback list. See
Fallback chains.
Proxy JSON errors
Before streaming begins, proxy errors use an HTTP status and a stable envelope:
{
"error": {
"code": "unknown_provider",
"message": "Unknown provider or model: example"
}
}
| Source | HTTP status | error.code |
|---|---|---|
| Missing model | 400 | missing_model |
| Unknown provider/model | 400 | unknown_provider |
| Local request validation | 400 | invalid_request |
| Provider error | provider status when valid | Original code/type when available; otherwise invalid_request for 400, provider_error for other statuses |
| Provider HTTP transport failure | 502 | http_error |
| JSON conversion failure | 500 | json_error |
| Stream setup/translation failure | 500 | stream_error |
| Exhausted fallback chain | 502 | all_failed |
| Proactive rate-limit rejection | 429 | rate_limited |
| Concurrency queue timeout | 503 | overloaded |
| Proxy logical lifetime exceeded before headers | 504 | request_timeout |
The proxy-generated 429 rate_limited and 503 overloaded responses include
Retry-After in whole seconds. An upstream provider's 429 or 503 status is
preserved, but its final response is not guaranteed to include that header.
Malformed JSON or a body that cannot be deserialized can be rejected by the HTTP framework before llmshim's error mapping runs; clients should use the HTTP status as well as the JSON body.
Errors during SSE
Admission happens before the SSE response is committed, so proactive 429 and
503 failures are ordinary JSON HTTP responses. Once streaming has begun, a
failure is a final typed event:
event: error
data: {"type":"error","message":"API key is invalid."}
Do not expect an HTTP status change after headers have been sent. Consumers
must handle both the initial HTTP response and error events in the stream.
When the proxy logical lifetime expires after SSE headers, it emits the native
or compact timeout event only at a complete SSE boundary. If an event is
partially framed, the body ends with a transport error so the timeout cannot be
misread as part of the unfinished data: field.
JSON and SSE messages on the proxy, gateway, and native endpoints share one
normalizer. Recognized upstream error envelopes are unwrapped, and internal
provider error (NNN):, stream error:, and upstream error: display prefixes
are removed. Ordinary text and malformed envelopes retain their content. Nested
JSON unwrapping is bounded to four levels. Fallback failures summarize each
provider's readable message.
SSE errors may also include an optional error object with the original message,
type, code, parameter and status. message remains the display field. Native
endpoints retain this metadata when constructing their own error envelope.
Malformed tool_calls must be an array or null; other shapes return local HTTP 400
before upstream dispatch, including requests to the streaming endpoints.