Appearance
std/s3
From the spec
This page is spec/std/s3.md, the specification the compiler and the backends are written from. It is rebuilt from the spec with every deploy (how).
Buckets on any S3-compatible store: AWS S3, MinIO, Cloudflare R2, Ceph, … (s3.weft beside http.weft declares it; the server runtime implements it, server.md, in core/crates/weft-runtime/src/server/s3.rs). Requests are signed with SigV4 by rusty-s3 and sent by the runtime's HTTP client on the I/O threads. Sandboxed user code can't import it (sandbox.md); an app hands it what it may reach instead.
weft
import { s3 } from "std/s3"
let photos = s3.bucket("photos") // endpoint and keys from the environment
try await photos.put("anna/cat.jpg", body, { contentType: "image/jpeg", metadata: Map([["owner", "anna"]]) })
let page = try await photos.list({ prefix: "anna/", limit: 100 })
let link = photos.presign("anna/cat.jpg", { expires: 600 }) // a GET anyone can use for 10 minutesBuckets
bucket(name, options?) (also s3.bucket) gives a Bucket at once; no request is made until one is asked for. Each option left out comes from the environment (the program's, as std/process reads it):
| option | environment | default |
|---|---|---|
endpoint | AWS_ENDPOINT_URL_S3, AWS_ENDPOINT_URL, S3_ENDPOINT | https://s3.<region>.amazonaws.com |
region | AWS_REGION, AWS_DEFAULT_REGION | us-east-1 (R2: auto) |
accessKeyId, secretAccessKey, sessionToken | AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN | none: requests go unsigned (public buckets) |
pathStyle | with an endpoint https://host/bucket/key, without one https://bucket.host/key | |
timeout (ms, per request), partSize (bytes, ≥ 5 MiB), revalidate (ms) | 30,000; 8 MiB; 1,000 |
A key without its secret (or the other way round), a bad endpoint or a part size below 5 MiB panics at bucket (a misconfiguration).
Objects
get(key): Task<Bytes ! S3Error>, getText(key) (UTF-8, lossy), getJson(key) (std/json's SyntaxError too, when it isn't JSON) | the whole body |
head(key): Task<ObjectInfo? ! S3Error> | null when there is no such object |
put(key, body: Bytes | string, options?), putJson(key, value, options?): Task<Put ! S3Error> | options = { contentType?, cacheControl?, metadata?: Map<string, string> }; a string defaults to text/plain; charset=utf-8, json to application/json |
delete(key) | deleting what isn't there succeeds (S3's own rule) |
copy(from, to): Task<Put ! S3Error> | on the server (x-amz-copy-source), metadata kept |
list(options?): Task<ListPage ! S3Error> | { prefix?, delimiter?, cursor?, limit? (1–1,000) }; ListPage = { objects: Listed[], prefixes: string[], cursor: string? }; pass cursor back for the next page (null on the last); with a delimiter, keys below it are summed up in prefixes |
presign(key, options?): string | a URL that lets anyone GET (default) or PUT the object until it expires: { method?, expires? (seconds, 3,600; 1–604,800, else a panic), contentType? (a PUT must then send it) }; no I/O |
ObjectInfo = { key, size, etag, lastModified: number?, contentType: string?, metadata: Map<string, string> }: the ETag without its quotes,lastModifiedin ms since the epoch,metadatathex-amz-meta-*headers, names lowercase.Put = { etag }.- Keys are any non-empty strings (UTF-8); a
/is only a character, whichdelimitergroups by.
Large objects
download(key): Task<Download ! S3Error>,Download = { info: ObjectInfo, chunk(): Task<Bytes? ! S3Error> }: the body as it arrives,nullat the end (asstd/http'schunk()).upload(key, options?): Task<Upload ! S3Error>,Upload = { write(chunk), finish(): Task<Put>, abort() }(each failing withS3Error): a multipart upload. Writes are buffered and each whole part (partSize) goes up as it fills;finishsends the rest as the last part and completes it, or makes one plainPUTwhen the object never filled a part;abortdrops the parts sent. Await eachwritebefore the next (a write while one is in flight panics): that is the backpressure.
Errors
Every request (each function above but presign) can fail with an S3Error:
weft
export type S3Error =
| NotFound { key: string, message: string } // no such key (`NoSuchKey`), bucket or upload; a 404
| AccessDenied { message: string } // `AccessDenied`, a signature that doesn't hold, an unknown or expired key; a 401 or 403
| Unreachable { message: string } // the request couldn't be made, or took longer than `timeout`
| Rejected { status: number, code: string, message: string } // anything else the store said: S3's code and messagekey is the key asked for. Messages are S3's where it gave one.
A copy or completed upload that S3 answers with a 200 holding an error is that error.
User trees in a bucket
bucket.trees is a std/sandbox Store (sandbox.md) whose trees live under prefixes, and bucket.tree(name) is bucket.trees.tree(name): what app.serve and run take (spec/app.md, User trees):
weft
let sites = s3.bucket("sites")
try await sites.trees.save(`users/${user.id}`, manifest) // checked and compiled first
try await app.serve(async req => [sites.tree(`users/${(try await session(req).user())?.id}`), "./routes"])- Layout (shared with every store,
sandbox.md;user-files.md, Stores):<name>/manifest.jsonholds the version, the files and routes, and what the tree uses of the app; each revision of a file is<name>/f/<id>/<rev>.weft, its history<name>/h/<id>.json, the draft<name>/draft.json; the export index is$index/<key>/<name>. Saving writes the sources, the index's new entries, then the manifest: readers switch to the new version at once and never see half of one. Old revisions stay (history);delete(name)removes the manifest first, then everything under<name>/. - Caching by version: a process keeps each tree's compiled copy. Its own saves update it at once; before serving from (or running) a tree whose copy is older than
revalidate, it asks for the manifest again withIf-None-Match(a 304 costs no transfer) and loads and compiles only a version it hasn't got. Requests that come while a tree loads wait for that one load. A prefix with no tree falls through to the next tree. - Loading compiles (the bytecode isn't stored): a version is compiled once per process, and was checked when it was saved. The store's failures are
std/sandbox'sStorage { message }(S3's message).
Decided here
rusty-s3signs (query-string SigV4, everyx-amz-*header signed) and the runtime'sreqwestsends: one HTTP stack, no SDK.headgivesnullfor a missing object,getfails withNotFound.- Path-style URLs whenever an endpoint is given (MinIO, R2 and test servers want them), virtual-hosted on AWS's own.
- Streaming upload is multipart with awaited writes as backpressure;
putis one request. - User trees in a bucket keep each file's revisions beside a small manifest, the same layout as a folder, so copying a prefix to a device's folder (later
std/sync) gives a tree that serves as it is. - The tests run against
s3s-fs(the S3 protocol over a folder, authentication on) in the test process:cargo test -p weft-cli --test s3.