Skip to content

Stream LLM tokens to the browser ​

A chat answer should appear as the model writes it, not after it has finished. The server asks the model with stream: true, reads its server-sent events as they arrive (res.events()), and passes each piece of text on to the page in a stream of its own (sse); the page reads that stream with events and appends as it goes.

The server ​

routes/api/chat/server.weft
weft
import { sse, fetch } from "std/http"
import type { Request, Response, Events } from "std/http"
import { env } from "std/process"
import * as json from "std/json"

type Token = { text: string }
/// The part of the model's events this reads: `content_block_delta` carries the text.
type Delta = { type: string, delta?: { type: string, text?: string } }

/// GET /api/chat?q=…: the answer, token by token, then an `end` event.
export async fn GET(req: Request): Response {
  let question = req.query.get("q") ?? ""
  return sse<Token>(async (events: Events<Token>) => {
    let res = try await fetch("https://api.anthropic.com/v1/messages", {
      method: "POST",
      headers: [
        ["x-api-key", env("ANTHROPIC_API_KEY") ?? ""],
        ["anthropic-version", "2026-10-21"],
        ["anthropic-beta", "server-side-fallback-2026-07-01"],
      ],
      json: {
        model: "claude-opus-5-5",
        max_tokens: 16000,
        stream: true,
        fallbacks: "default",
        messages: [{ role: "user", content: question }],
      },
      timeout: 300000,
    })
    if !res.ok {
      await events.send({ text: `The model answered ${res.status}: ${try await res.text()}` }, { event: "end" })
      return
    }
    let upstream = res.events()
    let mut e = try await upstream.next()
    while e != null && !events.closed() {
      if e.event == "content_block_delta" {
        let raw = try json.parse(e.data)
        let d = try raw as Delta
        let text = d.delta?.text
        if text != null {
          await events.send({ text })
        }
      }
      e = try await upstream.next()
    }
    await events.send({ text: "" }, { event: "end" })
  })
}

sse<Token> answers at once with text/event-stream and then runs the function: each events.send writes one event and waits until the connection has taken it, so a slow reader slows the model's reading instead of filling memory. When the reader goes away, events.closed() turns true and the loop stops (and with it the request to the model); a failure in it (the fetch, a malformed event) ends the stream and is logged. fallbacks: "default" lets the API answer with another model when this one declines a request.

The page ​

routes/page.weft
weft
import { events } from "std/http"
import { encodeComponent } from "std/url"
import * as json from "std/json"

type Token = { text: string }

let mut question = ""
let mut answer = ""
let mut busy = false

async fn ask() {
  if question.trim() == "" || busy { return }
  busy = true
  answer = ""
  let stream = events(`/api/chat?q=${try encodeComponent(question) else ""}`)
  while true {
    let e = try await stream.next() else null
    if e == null || e.event == "end" { break }
    let raw = try json.parse(e.data) else null
    let token = try raw as Token else null
    answer += token?.text ?? ""
  }
  busy = false
}
---
<Box class="flex flex-col gap-3 w-[32rem]">
  <Box class="flex gap-2">
    <Input bind:value={question} onsubmit={ask} placeholder="Ask anything" />
    <Button onpress={ask} disabled={busy}>Ask</Button>
  </Box>
  <Text class="whitespace-pre-wrap">{answer}</Text>
</Box>

A click handler can't pass an error on, so each try here has an else. Each answer += is a state change, so the text grows on screen as tokens arrive. events in the browser is a reader over fetch, not EventSource: it doesn't reconnect, and next() gives null when the stream ends.

sh
ANTHROPIC_API_KEY=… weft dev .

The same res.events() reads any server-sent events (other model APIs, a feed), and stream sends bodies that aren't events (a CSV built row by row, a large export). Request bodies can stream too, for uploads past the 1 MiB maxBody: std/http, Streaming.

Weft is in development. The reference is the spec as it stands today.