Skip to content

All lessons

Area 08 // BackendLesson 129 / 14014 min read

Streaming with SSE, and WebSockets

Server-sent events (SSE) keep one HTTP response open so the server can push tokens to the browser as the model writes them, which is how most chat apps stream answers, while WebSockets open a two-way connection for when the client also needs to send messages during the stream.

Think of it like this

Analogy

SSE is a live radio broadcast: once you tune in, the station keeps talking and you listen; you can't talk back on the same channel. A WebSocket is a phone call: both sides can speak at any time over the same open line. For an AI reading out its answer, radio is usually enough; for a conversation where you might interrupt, you want the phone.

The problem it solves

An LLM may take 10 seconds to write a full answer, but it produces the first words within a second. If your API waits for the whole response, users stare at a spinner. You need a way to push partial results to the browser as they're generated (streaming and latency), and sometimes to let the user send messages (like "stop") while the answer is still streaming.

How it works: streaming with SSE

  1. The client sends a request (often POST /chat with the question, or GET /stream?q= for simple cases).
  2. The server responds with Content-Type: text/event-stream and keeps the connection open.
  3. Each event is text: optional event: name and id: lines, then data: ... (often JSON), then a blank line.
  4. The server forwards tokens from the provider's stream (stream=True or the SDK's streaming helper) as events, then sends a final done event with metadata (citations, usage).
  5. Browsers read it with EventSource (GET only, auto-reconnect) or fetch with a stream reader (works with POST and auth headers).
  6. Operational details: disable proxy buffering, raise idle timeouts, send periodic comments (: ping) to keep connections alive, and cancel the LLM call if the client disconnects.
  7. Errors mid-stream: the status is already 200, so send an event: error message the UI can show.

How it works: WebSockets

  1. The client opens a WebSocket (ws:// or wss://) with an HTTP upgrade request; the connection stays open.
  2. Both sides send messages whenever they like: text or binary.
  3. Good fits: voice assistants (audio both ways), collaborative editors, agents that need user input mid-task, "stop generating" and other live controls, multiplayer apps.
  4. Costs: stateful connections are harder to load-balance and scale (sticky sessions or a shared pub/sub), need their own auth on connect, heartbeats and reconnection logic.
  5. In FastAPI: @app.websocket("/ws"), then await ws.accept(), receive_text() and send_json().

See it in code

Install
pip install fastapi httpx
Needs
Python 3.10+. The token list stands in for a provider's stream.

The same streamed answer over SSE and over a WebSocket.

streaming_api.py
import asyncio, json
from fastapi import FastAPI, WebSocket
from fastapi.responses import StreamingResponse
from fastapi.testclient import TestClient
 
app = FastAPI()
TOKENS = ["Annual", " leave", " is", " 25", " days", " [1]"]
 
# SSE: one HTTP response that stays open; each event is "data: ...\n\n".
@app.get("/stream")
async def stream(q: str):
    async def events():
        for t in TOKENS:                                # in real code: tokens from the LLM's stream
            await asyncio.sleep(0.01)
            yield f"data: {json.dumps({'token': t})}\n\n"
        yield "event: done\ndata: {}\n\n"
    return StreamingResponse(events(), media_type="text/event-stream")
 
# WebSocket: a two-way connection; the client can send while the server streams (e.g. "stop").
@app.websocket("/ws")
async def ws(socket: WebSocket):
    await socket.accept()
    question = await socket.receive_text()
    for t in TOKENS:
        await socket.send_json({"token": t})
    await socket.send_json({"done": True, "question": question})
    await socket.close()
 
client = TestClient(app)
with client.stream("GET", "/stream", params={"q": "leave?"}) as r:
    print("SSE content-type:", r.headers["content-type"])
    lines = [l for l in r.iter_lines() if l]
print("SSE lines:", lines[:2], "...", lines[-2:])
 
with client.websocket_connect("/ws") as s:
    s.send_text("How much leave do I get?")
    msgs = [s.receive_json() for _ in range(len(TOKENS) + 1)]
print("WebSocket answer:", "".join(m.get("token", "") for m in msgs), "|", msgs[-1])
Output
SSE content-type: text/event-stream; charset=utf-8
SSE lines: ['data: {"token": "Annual"}', 'data: {"token": " leave"}'] ... ['event: done', 'data: {}']
WebSocket answer: Annual leave is 25 days [1] | {'done': True, 'question': 'How much leave do I get?'}

In a real endpoint the loop reads from the provider, e.g. async with client.messages.stream(...) as s: async for text in s.text_stream: yield f"data: {json.dumps({'token': text})}\n\n".

A real example

Example

The stream that arrived in one lump

A team's chat streamed nicely on their laptops, but in production answers appeared all at once after eight seconds. Their reverse proxy was buffering responses, and the load balancer cut idle connections at 30 seconds, killing long answers. They added X-Accel-Buffering: no and Cache-Control: no-cache headers, raised idle timeouts, sent a keep-alive comment every 15 seconds, and cancelled the provider stream when the client disconnected. Time to first visible token dropped from 8 s to 0.7 s, and abandoned answers stopped costing tokens.

The easy way

Start here

Use SSE for chat answers: a FastAPI StreamingResponse with text/event-stream, forwarding the SDK's stream, and fetch with a stream reader in the browser. Reach for WebSockets only when the client must send messages during the stream.

When you grow: the production setup

Production streaming includes structured event types (token, tool-call, citation, error, done), resumable streams with event IDs, keep-alives, proxy and load-balancer tuning, cancellation on disconnect, auth on connect for WebSockets, sticky sessions or pub/sub for scaling WebSockets, and metrics for TTFT and stream completion rates.

Trade-offs

What you gain

  • Much better perceived speed.
  • SSE is plain HTTP: simple, proxy-friendly, easy to debug.
  • WebSockets enable real-time two-way experiences.

What it costs you

  • Long-lived connections need timeout and proxy tuning.
  • Errors after the first byte can't use status codes.
  • WebSockets are harder to scale and secure.

When not to use it

Common mistakes

  • Buffering proxies that hide the stream.
  • Not cancelling the LLM call when the user leaves.
  • Using WebSockets when SSE would do.
  • No error event, so failed streams just stop.
  • Forgetting auth on WebSocket connections.

Interview questions

Try answering out loud first, then open the model answer.

SSE vs WebSockets for an LLM chat app?

Model answer

SSE fits most chat apps: the client sends a request and the server streams the answer one way over plain HTTP, with simple infrastructure and automatic reconnection. WebSockets fit when both sides send messages during a session: voice, live interruptions, collaborative editing, agents asking for input. They cost more in scaling, auth and connection management.

How do you stream an LLM response through your backend?

Model answer

Call the provider with streaming enabled using an async client; in an async endpoint return a streaming response with text/event-stream that yields each token as an SSE data event, plus final events with citations and usage; set headers to disable buffering; send keep-alives; handle errors with an error event; and cancel the upstream stream if the client disconnects.

What breaks when streaming in production?

Model answer

Proxy or CDN buffering, idle timeouts on load balancers, missing keep-alives, connections dropped during deploys (drain them), lack of cancellation leading to wasted tokens, and mid-stream errors that the client can't see without explicit error events.

Quiz

Pick an answer to see whether it’s right and why.

  1. 01How does SSE send each piece of data?
  2. 02When would you choose WebSockets over SSE?
  3. 03Why does streaming improve perceived speed?
  4. 04What should happen when the user closes the tab mid-stream?
  5. 05Why might a stream arrive all at once in production but not locally?

0 of 5 answered

Key terms

Server-sent events (SSE)
A simple HTTP standard for a server to push a stream of text events to the browser over one long-lived response. Most LLM APIs stream tokens this way.
WebSocket
A long-lived, two-way connection between browser and server where either side can send messages at any time.
Streaming (responses)
Sending the model's reply token by token as it is generated, so the user starts reading within a second instead of waiting for the whole answer.
Time to first token (TTFT)
How long a user waits from sending a request until the first token of the reply arrives. Mostly set by queueing and prompt processing (prefill); it's the latency users feel most.
Event stream (text/event-stream)
The content type used by server-sent events, where each message is sent as data lines followed by a blank line.

Before and after

Read first

Read next

Progress is saved in this browser only.