LLM Streaming in Production: SSE, Backpressure, and Cancellation Patterns (2026)

Updated: August 1, 2026

LLM streaming in production means delivering model tokens over Server-Sent Events (SSE) while correctly handling three failure modes that most tutorials skip: proxy buffering that hides your first token for seconds, backpressure when a slow client stalls generation, and cancellation that has to reach the provider (or you keep paying for tokens no one will read). This guide is the checklist I use to ship OpenAI, Anthropic, and Gemini streams that survive nginx, mobile networks, and users who close the tab mid-response. I've hit every one of these bugs at least once, usually on a Friday.

• SSE has won the LLM transport war in 2026. OpenAI, Anthropic, Gemini, MCP, and A2A all use it; reach for WebSockets only when you truly need bidirectional mid-stream control.
• Nginx buffers responses by default. Send X-Accel-Buffering: no or set proxy_buffering off, or your first token arrives 5 to 10 seconds late.
• Cancellation must propagate from the browser's AbortController all the way to the provider's HTTP request, or you burn tokens for closed tabs.
• Backpressure in LLM streams is uncommon (models are slower than networks), but it matters when you buffer to Redis, S3, or a slow WebSocket client. Bound your queues.
• Anthropic emits typed events (message_start, content_block_delta); OpenAI emits choices[0].delta.content. Normalize into one internal event shape before it reaches your frontend.
• Instrument time-to-first-token (TTFT), inter-token latency, and cancellation rate. A stream that "works" locally can still be broken by a load balancer idle timeout.

Why streaming matters for production LLM apps

Streaming exists to hide latency. A GPT-4-class model may take 8 to 20 seconds to finish a 500-token reply, but the first token typically lands in under 500 ms. Rendering that token immediately gives users a 10x to 20x improvement in perceived responsiveness, even though the total wall-clock time hasn't moved. Every consumer-grade AI product (ChatGPT, Claude, Gemini, Perplexity) streams by default because non-streaming feels broken by comparison.

The payoff isn't only UX. Streaming lets you start post-processing tokens (moderation checks, PII redaction, JSON validation) before generation completes, and it lets users cancel a request when they see the answer heading the wrong direction. Combined with prompt caching to cut input costs, streaming is the second-biggest lever you have on perceived latency.

What makes it hard in production isn't the SDK call. Every provider gives you a one-liner. It's everything downstream: reverse proxies that batch your tokens, load balancers that kill idle connections at 30 seconds, browsers that don't forward disconnects, and Redis queues that grow unbounded when a client stalls. This article walks through each failure mode with the fix.

SSE vs WebSockets for LLM streaming in 2026

Server-Sent Events won. Every major provider ships SSE, and the emerging Model Context Protocol (MCP) and Google's A2A protocol both use it for streaming. SSE is a plain HTTP/1.1 response with Content-Type: text/event-stream, so it passes through any CDN, WAF, and reverse proxy that speaks HTTP. WebSockets require a protocol upgrade that some corporate proxies still drop.

Pick WebSockets only when you need bidirectional messaging inside a single connection: mid-stream tool approvals, live voice with barge-in, or a shared collaborative session. Otherwise SSE is smaller, simpler, and reconnects automatically via the browser's EventSource.

Dimension	Server-Sent Events	WebSockets
Direction	Server to client only	Full duplex
Transport	Plain HTTP/1.1 or HTTP/2	HTTP upgrade to WS/WSS
Auto-reconnect	Built into EventSource	You implement it
Proxy/CDN friendliness	High. Looks like any HTTP response	Medium. Upgrade blocked by some proxies
Headers on browser client	Limited (no custom auth headers with EventSource)	Full during handshake
Best fit	Chat, agent tokens, log tails	Voice, collaborative canvas, tool approvals

The EventSource API's inability to set custom headers is the one real SSE gotcha. You either move the auth token to a signed cookie, use a fetch-based SSE polyfill, or upgrade to WebSockets. In practice most teams use @microsoft/fetch-event-source or a small custom fetch reader, both covered later.

OpenAI, Anthropic, and Gemini SSE formats compared

Every provider streams SSE, but the event shapes differ. If you're writing multi-provider code, normalize them into one internal shape at the edge, so the rest of the stack (and your frontend) sees the same tokens regardless of upstream.

OpenAI sends a stream of chat.completion.chunk objects. Each data: line is a JSON payload, and the token text lives at choices[0].delta.content. The stream ends with data: [DONE].

data: {"id":"chatcmpl-...","choices":[{"delta":{"content":"Hello"},"index":0}]}

data: {"id":"chatcmpl-...","choices":[{"delta":{"content":" world"},"index":0}]}

data: [DONE]

Anthropic sends typed events. The tokens themselves ride in content_block_delta events; message_start, content_block_start, message_delta, and message_stop carry lifecycle metadata (including usage counts for billing). See the Anthropic streaming reference for the full event list.

event: message_start
data: {"type":"message_start","message":{"id":"msg_01...","usage":{"input_tokens":42}}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Hello"}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" world"}}

event: message_stop
data: {"type":"message_stop"}

Gemini's streamGenerateContent endpoint returns a stream of JSON GenerateContentResponse objects, with token text at candidates[0].content.parts[0].text. Google's SDK also exposes generate_content_stream which yields Python objects directly, no SSE parsing required server-side.

A production FastAPI streaming endpoint

Below is a minimal but production-shaped FastAPI endpoint that streams from Anthropic and forwards tokens as SSE to the browser. It handles client disconnects, emits the anti-buffering headers, and cancels the upstream call if the client goes away. This is the shape you want; everything else in the article layers on top of it.

from anthropic import AsyncAnthropic
from fastapi import FastAPI, Request
from fastapi.responses import StreamingResponse
import asyncio, json

app = FastAPI()
client = AsyncAnthropic()

SSE_HEADERS = {
    "Content-Type": "text/event-stream",
    "Cache-Control": "no-cache, no-transform",
    "Connection": "keep-alive",
    # Critical: disable nginx buffering for THIS response.
    "X-Accel-Buffering": "no",
}

async def token_stream(prompt: str, request: Request):
    # `async with` guarantees the upstream HTTP connection is closed
    # if this generator is garbage-collected mid-flight.
    async with client.messages.stream(
        model="claude-sonnet-5",
        max_tokens=1024,
        messages=[{"role": "user", "content": prompt}],
    ) as stream:
        async for event in stream:
            # Client hung up? Stop generation immediately.
            if await request.is_disconnected():
                break
            if event.type == "content_block_delta":
                payload = json.dumps({"t": event.delta.text})
                yield f"data: {payload}\n\n"
        # Final usage event lands in the message object after stream closes.
        final = await stream.get_final_message()
        yield f"data: {json.dumps({'done': True, 'usage': final.usage.model_dump()})}\n\n"

@app.post("/chat")
async def chat(request: Request):
    body = await request.json()
    return StreamingResponse(
        token_stream(body["prompt"], request),
        headers=SSE_HEADERS,
    )

Three details that matter and are frequently missed. The async with client.messages.stream(...) context manager is what actually cancels the upstream request; a bare async for loop leaks the connection. request.is_disconnected() is polled inside the loop; without it, the generator keeps pulling tokens after the browser closes the tab. And the X-Accel-Buffering: no header is what makes tokens arrive in real time behind an nginx or Kubernetes ingress. Without it, tokens pile up until the response is complete. (I lost most of an afternoon to that one, on a service that streamed perfectly on my laptop and shipped bursts on staging.)

Why is nginx buffering my SSE stream?

Because proxy_buffering is on by default. Nginx collects your upstream response into an internal buffer and forwards it to the client in bursts. For a normal JSON API that's a throughput win. For SSE it means the client sees nothing for 5 to 10 seconds and then the entire response arrives at once. Streaming that appears to work locally is silently broken behind the load balancer.

There are three places to fix it, and you generally want all three:

1. Application response header. Set X-Accel-Buffering: no on the streaming response. Nginx honors this per-response even if global buffering is on. This is the least-privilege fix and the one you should always ship.
2. Nginx location block. For the streaming path, add proxy_buffering off;, force proxy_http_version 1.1;, and clear Connection so chunked transfer works cleanly.
3. Every other proxy in the path. Cloudflare, AWS ALB, GCP HTTPS LB, YARP, Azure APIM, and API gateways each have their own buffering knob. Cloudflare responds to the same Cache-Control: no-transform and X-Accel-Buffering: no headers; ALB streams responses over HTTP/1.1 without special config; APIM historically buffered and now respects the header as of 2025.

# nginx.conf. The streaming location.
location /chat {
    proxy_pass http://app_upstream;
    proxy_buffering off;
    proxy_cache off;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    # Give long generations room; align with your app's max_tokens ceiling.
    proxy_read_timeout 300s;
    chunked_transfer_encoding on;
}

Nginx's own documentation on proxy_buffering is the authoritative reference. Read it before you change global defaults.

How do you cancel a streaming LLM response without wasting tokens?

The fastest way to blow through your budget is to keep generating tokens for browsers that closed the tab three seconds ago. Cancellation has to travel end-to-end: browser AbortController, then server request, then provider SDK, then upstream HTTP connection close. If any link breaks, you pay for tokens no one will read.

On the browser, use fetch with AbortController. The plain EventSource API auto-reconnects and doesn't cooperate well with abort semantics:

const controller = new AbortController();

// User clicks "Stop"
stopButton.addEventListener("click", () => controller.abort());

const res = await fetch("/chat", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({ prompt }),
  signal: controller.signal,
});

const reader = res.body.getReader();
const decoder = new TextDecoder();
let buf = "";
try {
  while (true) {
    const { value, done } = await reader.read();
    if (done) break;
    buf += decoder.decode(value, { stream: true });
    for (const frame of parseSseFrames(buf)) {
      render(frame.data);
      buf = buf.slice(frame.consumed);
    }
  }
} catch (e) {
  if (e.name !== "AbortError") throw e;
}

On the server, the FastAPI example above already handles this correctly: closing the fetch response causes request.is_disconnected() to return true, the generator breaks out of the loop, and the async with client.messages.stream(...) context manager tears down the upstream connection to Anthropic. The provider stops generating within one to two tokens of the disconnect.

Node backends follow the same pattern with the OpenAI SDK's stream.controller.abort() or by passing { signal: req.signal } when Express 5 forwards the incoming request's abort signal. For deeper reliability patterns like retry-after cancellation, see our guide on LLM rate limiting and 429 retries in production.

Backpressure: when the client can't keep up

Backpressure happens when your producer emits data faster than the consumer can drain it. For raw LLM streams to a browser this is rare. A 100 tok/s model produces roughly 400 bytes/s, well under any network. But it becomes real in three shapes:

• Fan-out through a queue. If you buffer tokens into Redis Pub/Sub, Kafka, or an in-memory asyncio.Queue to fan out to multiple subscribers, a slow subscriber grows the queue unboundedly. Cap the queue size and drop or block on overflow.
• WebSocket to a mobile client on 3G. The OS send buffer fills, the socket's write() starts blocking, and if you don't await it you build up an application-level backlog. Always await writes; treat a blocked write as a signal to pause upstream consumption.
• Multi-model aggregation. Merging tokens from two providers into one output stream can starve one model while the other drains. Use per-source bounded channels rather than a single shared queue.

The Python pattern is a bounded queue with an explicit await:

queue: asyncio.Queue[str] = asyncio.Queue(maxsize=64)

async def producer(stream):
    async for chunk in stream:
        # This will pause when the consumer is 64 chunks behind.
        await queue.put(chunk.text)

async def consumer(send):
    while True:
        chunk = await queue.get()
        await send(chunk)  # await propagates network backpressure.

Note the two awaits. queue.put blocks the producer once the queue is full, which in turn causes the LLM stream to pause receiving from the socket, and TCP backpressure propagates the pause all the way back to the provider. That's the mechanism that keeps memory bounded under a slow client.

Streaming structured outputs and partial JSON

When you're using OpenAI's response_format, Anthropic's tool use, or Instructor to force a JSON output, streaming tokens are still individual characters: {, ", n, a, m, e. Rendering that live requires a partial JSON parser that gracefully closes unbalanced brackets on every frame.

The three practical options are partial-json-parser (JavaScript) or partial-json in Python, which auto-complete incomplete structures; a streaming JSON event parser, which emits values as they appear; and Instructor's Partial[Model], which yields validated Pydantic objects on every field completion. For the general shape of validating streaming LLM output, see our guide on LLM observability in production, which covers tracing partial output through your pipeline.

from instructor import from_anthropic, Partial
from pydantic import BaseModel

class Ticket(BaseModel):
    title: str
    priority: str
    summary: str

client = from_anthropic(AsyncAnthropic())
stream = client.messages.create_partial(
    response_model=Partial[Ticket],
    messages=[{"role": "user", "content": prompt}],
    model="claude-sonnet-5",
    max_tokens=512,
)
async for partial in stream:
    # `partial.title` fills in first, then `priority`, then `summary`.
    # Fields that haven't arrived yet are None, never a parse error.
    yield f"data: {partial.model_dump_json()}\n\n"

Reconnection with Last-Event-ID

SSE has a built-in resume mechanism most implementations skip. If you emit an id: field on every event, browsers automatically send Last-Event-ID as a header when they reconnect after a network blip. Your server can then resume the stream from that offset instead of restarting the generation.

async def stream_with_ids(prompt: str, resume_from: int = 0):
    seq = 0
    async for token in generate(prompt):
        seq += 1
        if seq <= resume_from:
            continue  # Skip tokens the client already has.
        yield f"id: {seq}\ndata: {json.dumps({'t': token})}\n\n"

Resume-from-offset only works if you're persisting the completed prefix somewhere the second connection can reach: Redis, Postgres, or an in-memory dict keyed on a request ID. For most chat apps the simpler play is to store the completed message once generation finishes and let the client re-render from history on reconnect; true mid-stream resume is worth building only for very long-running generations (30+ seconds) where restarting is expensive. Honestly, that's the exact trade-off durable agent pipelines with LangGraph and Temporal exist to solve at the workflow level.

Observability: TTFT, inter-token latency, cancel rate

You can't debug a stream you don't measure. The four metrics that actually matter in production:

1. Time to first token (TTFT). Wall-clock milliseconds between request start and the first byte of token payload reaching the client. This is the metric users feel. Anything over 1 second on chat feels sluggish; over 3 seconds feels broken.
2. Inter-token latency (ITL). Median time between tokens. Sudden spikes usually mean proxy buffering has kicked in on part of the path.
3. Stream completion rate. The fraction of streams that reach [DONE] vs. get cancelled, error out, or time out. A rising cancellation rate is a leading indicator of quality problems.
4. Cancelled-but-billed tokens. Tokens the provider generated after the user cancelled. If this is non-zero, you have a cancellation propagation bug; fix it before you scale.

Structure your traces so each stream is a parent span with token-generation as child events. OpenTelemetry's GenAI semantic conventions define gen_ai.response.time_to_first_token and gen_ai.usage.output_tokens, so emit both. The OpenAI SDK's stream=True path exposes chunk.usage on the final chunk when you request stream_options={"include_usage": True}; Anthropic's message_delta event carries usage.output_tokens. Log those, not client-side estimates, because that's what shows up on the bill.

The MDN guide to SSE is worth bookmarking for the browser-side API details, and the OpenAI streaming reference covers the include_usage flag and edge cases like tool-call streaming that this article only touched briefly.

Frequently Asked Questions

Does streaming reduce LLM costs?

No. You're billed per input and output token regardless of whether the response streams or arrives all at once. The cost win from streaming is indirect: proper cancellation lets users stop a wrong-direction answer early, and that avoids paying for tokens they'd never read. Combine streaming with prompt caching if you want an actual bill reduction.

Should I use SSE or WebSockets for a Claude or GPT chat app?

Use SSE. It's what every provider ships, it passes through any HTTP proxy, and the browser's EventSource auto-reconnects. Reach for WebSockets only when you need to send messages back mid-stream (for example, live voice with barge-in or an approval-gated agent that pauses on tool calls).

Why do my SSE tokens arrive in one big burst instead of streaming?

Almost always proxy buffering. Nginx has proxy_buffering on by default and holds the response until it's complete. Fix it by sending X-Accel-Buffering: no as a response header, or by setting proxy_buffering off on the streaming location. Check every proxy in the path (Cloudflare, ALB, APIM), not just the first one.

How do I cancel an OpenAI or Anthropic streaming request?

Both SDKs cancel when the underlying HTTP connection closes. In Python, use the async streaming context manager (async with client.messages.stream(...)) so exiting the block tears down the connection. In Node, call stream.controller.abort(). On the browser side, wrap fetch in an AbortController; closing the socket from the browser propagates all the way to the provider within one to two tokens.

Can I stream JSON structured outputs and render them progressively?

Yes, but the raw token stream contains partial JSON that fails a strict parser. Use a partial JSON library like partial-json (Python) or partial-json-parser (JavaScript) that auto-closes unbalanced brackets, or use Instructor's Partial[Model] to yield typed Pydantic objects as fields fill in. Both approaches let you paint the UI on every token instead of waiting for the closing brace.