Skip to content

std/s3 ​

From the spec

This page is spec/std/s3.md, the specification the compiler and the backends are written from. It is rebuilt from the spec with every deploy (how).

Buckets on any S3-compatible store: AWS S3, MinIO, Cloudflare R2, Ceph, … (s3.weft beside http.weft declares it; the server runtime implements it, server.md, in core/crates/weft-runtime/src/server/s3.rs). Requests are signed with SigV4 by rusty-s3 and sent by the runtime's HTTP client on the I/O threads. Sandboxed user code can't import it (sandbox.md); an app hands it what it may reach instead.

weft
import { s3 } from "std/s3"

let photos = s3.bucket("photos")                              // endpoint and keys from the environment
try await photos.put("anna/cat.jpg", body, { contentType: "image/jpeg", metadata: Map([["owner", "anna"]]) })
let page = try await photos.list({ prefix: "anna/", limit: 100 })
let link = photos.presign("anna/cat.jpg", { expires: 600 })   // a GET anyone can use for 10 minutes

Buckets ​

bucket(name, options?) (also s3.bucket) gives a Bucket at once; no request is made until one is asked for. Each option left out comes from the environment (the program's, as std/process reads it):

optionenvironmentdefault
endpointAWS_ENDPOINT_URL_S3, AWS_ENDPOINT_URL, S3_ENDPOINThttps://s3.<region>.amazonaws.com
regionAWS_REGION, AWS_DEFAULT_REGIONus-east-1 (R2: auto)
accessKeyId, secretAccessKey, sessionTokenAWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKENnone: requests go unsigned (public buckets)
pathStylewith an endpoint https://host/bucket/key, without one https://bucket.host/key
timeout (ms, per request), partSize (bytes, ≥ 5 MiB), revalidate (ms)30,000; 8 MiB; 1,000

A key without its secret (or the other way round), a bad endpoint or a part size below 5 MiB panics at bucket (a misconfiguration).

Objects ​

get(key): Task<Bytes ! S3Error>, getText(key) (UTF-8, lossy), getJson(key) (std/json's SyntaxError too, when it isn't JSON)the whole body
head(key): Task<ObjectInfo? ! S3Error>null when there is no such object
put(key, body: Bytes | string, options?), putJson(key, value, options?): Task<Put ! S3Error>options = { contentType?, cacheControl?, metadata?: Map<string, string> }; a string defaults to text/plain; charset=utf-8, json to application/json
delete(key)deleting what isn't there succeeds (S3's own rule)
copy(from, to): Task<Put ! S3Error>on the server (x-amz-copy-source), metadata kept
list(options?): Task<ListPage ! S3Error>{ prefix?, delimiter?, cursor?, limit? (1–1,000) }; ListPage = { objects: Listed[], prefixes: string[], cursor: string? }; pass cursor back for the next page (null on the last); with a delimiter, keys below it are summed up in prefixes
presign(key, options?): stringa URL that lets anyone GET (default) or PUT the object until it expires: { method?, expires? (seconds, 3,600; 1–604,800, else a panic), contentType? (a PUT must then send it) }; no I/O
  • ObjectInfo = { key, size, etag, lastModified: number?, contentType: string?, metadata: Map<string, string> }: the ETag without its quotes, lastModified in ms since the epoch, metadata the x-amz-meta-* headers, names lowercase. Put = { etag }.
  • Keys are any non-empty strings (UTF-8); a / is only a character, which delimiter groups by.

Large objects ​

  • download(key): Task<Download ! S3Error>, Download = { info: ObjectInfo, chunk(): Task<Bytes? ! S3Error> }: the body as it arrives, null at the end (as std/http's chunk()).
  • upload(key, options?): Task<Upload ! S3Error>, Upload = { write(chunk), finish(): Task<Put>, abort() } (each failing with S3Error): a multipart upload. Writes are buffered and each whole part (partSize) goes up as it fills; finish sends the rest as the last part and completes it, or makes one plain PUT when the object never filled a part; abort drops the parts sent. Await each write before the next (a write while one is in flight panics): that is the backpressure.

Errors ​

Every request (each function above but presign) can fail with an S3Error:

weft
export type S3Error =
  | NotFound { key: string, message: string }            // no such key (`NoSuchKey`), bucket or upload; a 404
  | AccessDenied { message: string }                     // `AccessDenied`, a signature that doesn't hold, an unknown or expired key; a 401 or 403
  | Unreachable { message: string }                      // the request couldn't be made, or took longer than `timeout`
  | Rejected { status: number, code: string, message: string }   // anything else the store said: S3's code and message

key is the key asked for. Messages are S3's where it gave one.

A copy or completed upload that S3 answers with a 200 holding an error is that error.

User trees in a bucket ​

bucket.trees is a std/sandbox Store (sandbox.md) whose trees live under prefixes, and bucket.tree(name) is bucket.trees.tree(name): what app.serve and run take (spec/app.md, User trees):

weft
let sites = s3.bucket("sites")
try await sites.trees.save(`users/${user.id}`, manifest)      // checked and compiled first
try await app.serve(async req => [sites.tree(`users/${(try await session(req).user())?.id}`), "./routes"])
  • Layout (shared with every store, sandbox.md; user-files.md, Stores): <name>/manifest.json holds the version, the files and routes, and what the tree uses of the app; each revision of a file is <name>/f/<id>/<rev>.weft, its history <name>/h/<id>.json, the draft <name>/draft.json; the export index is $index/<key>/<name>. Saving writes the sources, the index's new entries, then the manifest: readers switch to the new version at once and never see half of one. Old revisions stay (history); delete(name) removes the manifest first, then everything under <name>/.
  • Caching by version: a process keeps each tree's compiled copy. Its own saves update it at once; before serving from (or running) a tree whose copy is older than revalidate, it asks for the manifest again with If-None-Match (a 304 costs no transfer) and loads and compiles only a version it hasn't got. Requests that come while a tree loads wait for that one load. A prefix with no tree falls through to the next tree.
  • Loading compiles (the bytecode isn't stored): a version is compiled once per process, and was checked when it was saved. The store's failures are std/sandbox's Storage { message } (S3's message).

Decided here ​

  1. rusty-s3 signs (query-string SigV4, every x-amz-* header signed) and the runtime's reqwest sends: one HTTP stack, no SDK.
  2. head gives null for a missing object, get fails with NotFound.
  3. Path-style URLs whenever an endpoint is given (MinIO, R2 and test servers want them), virtual-hosted on AWS's own.
  4. Streaming upload is multipart with awaited writes as backpressure; put is one request.
  5. User trees in a bucket keep each file's revisions beside a small manifest, the same layout as a folder, so copying a prefix to a device's folder (later std/sync) gives a tree that serves as it is.
  6. The tests run against s3s-fs (the S3 protocol over a folder, authentication on) in the test process: cargo test -p weft-cli --test s3.

Weft is in development. The reference is the spec as it stands today.