15 · Streaming Architecture

This is a streaming architecture for real-time UX and lower perceived latency. The goal: return the first token as fast as possible and stream the rest efficiently and reliably.

How it works

1. 1User asks a question in the UI.
2. 2FastAPI accepts the request; LangServe opens an SSE/WebSocket connection.
3. 3LangGraph starts execution and retrieves context (RAG).
4. 4Bedrock returns tokens as a stream via the Converse API.
5. 5LangServe forwards tokens to the client, which renders them one by one.

Key points

• SSE (server-sent events) is simple one-way streaming; WebSockets are bidirectional.
• Prompt caching and shorter context reduce time-to-first-token.
• Parallel tool calls and semantic caching speed things up.
• Measure TTFT, tokens per second and total latency.

Why it matters

Streaming isn't just about speed — it's about perceived responsiveness. Showing the first token fast makes an AI feel instant, even when total generation takes seconds.

Frequently asked questions

What is TTFT?

Time To First Token — how long until the first piece of the response appears. Lower feels much faster.

SSE or WebSocket?

SSE for simple one-way token streaming; WebSockets when you need two-way, interactive communication.