Implement HTTP handlers / webhooks in Rust modules

[gefjon]

Description of Changes

Adds support to Rust modules and the SpacetimeDB host for defining HTTP handlers and registering them to routes.

User-facing API

In a Rust module, users can annotate functions with the new macro #[spacetimedb::http::handler]. A function annotated this way must accept exactly two arguments, of types &mut spacetimedb::http::HandlerContext and spacetimedb::http::Request (which is a type alias for http::Request<spacetimedb::http::Body>. It must also return spacetimedb::http::Response (which is a type alias for http::Response<spacetimedb::http::Body>).

Once the user has defined an HTTP handler, they can register it to a route by annotating a function with #[spacetimedb::http::router]. Such a function must take no arguments and return a spacetimedb::http::Router. (The original design put this annotation on a static variable rather than a function, but that turned out to be undesirable because it required that constructing a Router be const.) Router exposes various methods for registering handlers to routes.

All of a database's user-defined routes are exposed under /v1/database/:name_or_identity/route/{*path}.

Example

See the new smoketest for a more exhaustive example.

A simpler example, which stores arbitrary byte data in a table via a POST request, returns an ID, and then retrieves that same data via a GET request with a query parameter:

#[spacetimedb::table(accessor = data)]
struct Data {
    #[primary_key]
    #[auto_inc]
    id: u64,
    body: Vec<u8>,
}

#[spacetimedb::http::handler]
fn insert(ctx: &mut HandlerContext, request: Request) -> Response {
    let body: Vec<u8> = request.into_body().into_bytes().into();
    let id = ctx.with_tx(|tx| tx.db.data().insert(Data { id: 0, body: body.clone() }).id);
    Response::new(Body::from_bytes(format!("{id}")))
}

#[spacetimedb::http::handler]
fn retrieve(ctx: &mut HandlerContext, request: Request) -> Response {
    let id = request
        .uri()
        .query()
        .and_then(|query| query.strip_prefix("id="))
        .and_then(|id| u64::from_str(id).ok())
        .unwrap();
    let body = ctx.with_tx(|tx| tx.db.data().id().find(id).map(|data| data.body));
    if let Some(body) = body {
        Response::new(Body::from_bytes(body))
    } else {
        Response::builder().status(404).body(Body::empty()).unwrap()
    }
}

#[spacetimedb::http::router]
fn router() -> Router {
    Router::new().post("/insert", insert).get("/retrieve", retrieve)
}

Design and implementation notes

• As mentioned above, the router is registered via a function, not a static or const item. This is because static or const initializers must be const, and it turns out to be a pain to make all of the Router constructors be const fns.
• The #[handler] macro clobbers the original function name with a const variable of type HttpHandler. This is unfortunate, but AFAICT necessary, 'cause we need to pass the string identifier for the handler to the Router, not the function pointer, and Rust allows no (stable and reliable) way to get a unique string identifier out of a function item/value, nor to attach data or implement traits for function items/values. The alternative(s) would involve changing the signature of the Router methods to have uglier and more complex callsites, e.g. like .get("/retrieve", retrieve::handler()), .get("/retrieve", handler!(retrieve)) or .get::<retrieve>("/retrieve"). I believe that registering handlers will be much more common than calling their functions, so I've chosen to make it so that registering them gets the convenient syntax, even though the inability to call them directly will be somewhat surprising.
• I haven't wired up energy handling or timing metrics for handler execution to anywhere. Procedures are still in the same boat.
• HTTP requests to user-defined routes bypass the usual SpacetimeDB auth middleware, meaning that the host does not validate (or inspect in any way) Authorization headers in requests before invoking the user-defined handler. This is required to allow arbitrary user-programmable handling of Authorization headers, including those in formations which SpacetimeDB would reject. As a result of this, HandlerContext doesn't expose a sender or sender_connection_id.
• HTTP route paths may consist only of a very restrictive set of characters. I've chosen this set to keep our options open in the future to add additional syntax to routes, like for registering wildcard segments and path parameters:
  • ASCII digits.
  • ASCII letters.
  • -_~/.
• The internal data structure that represents a Router is currently a Vec<Route>, meaning that resolving a request to a route is O(num_routes). Registering a route checks against each previous route for uniqueness, meaning that constructing a router is O(num_routes ^ 2). There are TODO comments to use a trie, but I think this can wait, as I expect most databases to register few routes.
• Commit 999a7c3 contains a fix to a mostly-unrelated bug where a few bindings introduced by the SATS derive macros were unhygienic and not in a reserved namespace, leading to name conflicts. I discovered this 'cause I tried writing an HTTP handler named index to serve the index/root of a website and it broke.

Still TODO

• Resolve various TODO comments in the diff.
• Documentation.
• C# bindings support.
• C++ bindings support.
• V8 host support.
• TypeScript bindings support.

API and ABI breaking changes

New APIs, currently flagged as unstable, which will eventually need stability guarantees. No (intentional) breaking changes, or changes to existing APIs at all.

Expected complexity level and risk

3? Changes to our HTTP routing to support the user-defined routes.

Testing

• New smoketest of the behavior!
• I dunno, maybe try hosting a simple webpage and see how it works?
• Build a test app with Stripe integration.
  • @aasoni did this.

[bfops]

my code-owned files LGTM:

tools/ci/src/main.rs

[gefjon]

I had Codex vibe-code up a simple static webpage hosted entirely in a SpacetimeDB module. This was easy enough and mostly totally worked, but we ran into a few silly issues which I'll work to fix over the next few days. In particular:

• A name conflict between having an HTTP handler named index and a table. This would, I believe, manifest if any module code defined a const named index in the same file as a #[table] macro invocation, but that's unlikely to happen in the wild due to consts usually having all-caps names.
• The root route was accessible at /v1/database/:name/route, but not at /v1/database/:name/route/ (note the trailing slash). This was especially unintuitive as the root route is registered as "/".
• The request path visible inside the handler was different from what was expected. The AI said:

  Externally the page was served at /v1/database/{db}/route, but inside the handler the incoming path looked like /{db}/route..., [...]

• We get warnings from the #[handler] macro output warning about lowercase constants that should be uppercase. This just needs an #[allow].