Streaming

Streaming exposes the same kinds of incremental output on every surface, but the Rust crate and the proxy encode them differently.

Availability: Rust: normalized JSON chunks · CLI: rendered live · Proxy: typed SSE · Clients: language-native stream iterators

The useful distinction is: same semantic events, two encodings.

Rust: normalized Chat Completions chunks

llmshim::stream returns a stream whose items are JSON strings. Each provider adapter and the shared per-response normalizer have translated native events into Chat Completions-style deltas. Tool calls are complete records emitted once at termination, after their argument and signature fragments have been assembled. Retain each call's wire_ids when storing it or sending its result back.

#![allow(unused)]
fn main() {
use futures::StreamExt;
use std::io::{self, Write};

let mut stream = llmshim::stream(&router, &request).await?;
let mut reasoning = llmshim::reasoning::ReasoningAccumulator::default();

while let Some(chunk) = stream.next().await {
    let chunk = chunk?;
    let parsed: serde_json::Value = serde_json::from_str(&chunk)?;

    let delta = &parsed["choices"][0]["delta"];
    reasoning.push(delta)?;
    eprint!("{}", llmshim::reasoning::reasoning_text(delta));

    if let Some(text) = parsed
        .pointer("/choices/0/delta/content")
        .and_then(|value| value.as_str())
    {
        print!("{text}");
        io::stdout().flush()?;
    }
}
}

Not every chunk contains text. Inspect only the fields your application needs, and keep handling Err items until the stream ends. Store reasoning.blocks() with the final assistant message only after successful completion. An EOF or [DONE] without a terminal provider chunk is an error. Cache usage events are cumulative snapshots; use the latest values, rather than summing events.

The Rust quickstart contains a complete runnable program.

Showing a tool call while it is written

A long argument, such as a file's new contents, can take many seconds to stream. To show the call before it completes, implement AttemptPolicy::observe_tool_call_progress and stream with a policy (ShimClient::stream_with_policy, stream_owned_with_policy or llmshim::stream_with_policy). Each call is reported in the same shape for every provider: Started with the tool's name, Arguments with the text so far, then Ended or Abandoned.

#![allow(unused)]
fn main() {
use llmshim::{policy::{AttemptIdentity, AttemptPolicy}, toolcall::{ToolCallProgress, ToolCallProgressEvent}};

impl AttemptPolicy for MyPolicy {
    // acquire, observe and observe_abandoned as before …

    fn observe_tool_call_progress(&self, attempt: &AttemptIdentity, progress: ToolCallProgress) {
        match progress.event {
            ToolCallProgressEvent::Started { name } => self.show(attempt.id(), progress.id, name),
            ToolCallProgressEvent::Arguments { text, .. } => self.preview(progress.id, text),
            ToolCallProgressEvent::Abandoned => self.discard(progress.id),
            _ => {}
        }
    }
}
}

The progress is provisional. The completed call in the stream is still the only call to execute, validated as before, and it carries the same id as its progress. Ended is reported before that chunk is produced. Arguments.text is partial JSON; appended is the new part, or None when the provider replaced the text. attempt separates one attempt's calls from another's. If the stream ends with an error, discard the progress of every call whose completed form did not arrive. A policy that does not implement the method gets the same stream as before. Calls that llmshim makes for its own output contracts (a forced answer tool, prompt-encoded calls) are never reported.

Proxy: typed SSE events

Send the compact request to the always-streaming endpoint:

curl -N http://localhost:3000/v1/chat/stream \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "anthropic/claude-sonnet-5-5",
    "messages": [{"role": "user", "content": "Write a haiku."}],
    "config": {"max_tokens": 128}
  }'

The response is an SSE stream. The SSE event name and the JSON type agree:

event: reasoning
data: {"type":"reasoning","text":"..."}

event: content
data: {"type":"content","text":"Rust shapes silent thought"}

event: usage
data: {"type":"usage","input_tokens":12,"output_tokens":7,"total_tokens":19}

event: done
data: {"type":"done"}

The six event types are:

Setting "stream": true on POST /v1/chat produces the same SSE encoding. An admission error can still arrive as an HTTP error before streaming begins; an error after the stream begins arrives as an error event.

Client ergonomics

The language clients decode the typed SSE stream without changing its event vocabulary:

• Python returns an iterator of event dictionaries.
• TypeScript returns an async iterator of typed events.
• Go returns a channel of typed events.
• Ruby yields typed events to a block.

See the canonical client guides for complete loops: Python, TypeScript, Go, and Ruby.