Implement HTTP handlers / webhooks in Rust modules (#4636)

# Description of Changes

Adds support to Rust modules and the SpacetimeDB host for defining HTTP
handlers and registering them to routes.

## User-facing API

In a Rust module, users can annotate functions with the new macro
`#[spacetimedb::http::handler]`. A function annotated this way must
accept exactly two arguments, of types `&mut
spacetimedb::http::HandlerContext` and `spacetimedb::http::Request`
(which is a type alias for `http::Request<spacetimedb::http::Body>`. It
must also return `spacetimedb::http::Response` (which is a type alias
for `http::Response<spacetimedb::http::Body>`).

Once the user has defined an HTTP handler, they can register it to a
route by annotating a function with `#[spacetimedb::http::router]`. Such
a function must take no arguments and return a
`spacetimedb::http::Router`. (The original design put this annotation on
a `static` variable rather than a function, but that turned out to be
undesirable because it required that constructing a `Router` be
`const`.) `Router` exposes various methods for registering handlers to
routes.

All of a database's user-defined routes are exposed under
`/v1/database/:name_or_identity/route/{*path}`.

## Example

See [the new
smoketest](https://github.com/clockworklabs/SpacetimeDB/blob/phoebe/http-handlers-webhooks/crates/smoketests/tests/smoketests/http_routes.rs)
for a more exhaustive example.

A simpler example, which stores arbitrary byte data in a table via a
`POST` request, returns an ID, and then retrieves that same data via a
`GET` request with a query parameter:

```rust
#[spacetimedb::table(accessor = data)]
struct Data {
    #[primary_key]
    #[auto_inc]
    id: u64,
    body: Vec<u8>,
}

#[spacetimedb::http::handler]
fn insert(ctx: &mut HandlerContext, request: Request) -> Response {
    let body: Vec<u8> = request.into_body().into_bytes().into();
    let id = ctx.with_tx(|tx| tx.db.data().insert(Data { id: 0, body: body.clone() }).id);
    Response::new(Body::from_bytes(format!("{id}")))
}

#[spacetimedb::http::handler]
fn retrieve(ctx: &mut HandlerContext, request: Request) -> Response {
    let id = request
        .uri()
        .query()
        .and_then(|query| query.strip_prefix("id="))
        .and_then(|id| u64::from_str(id).ok())
        .unwrap();
    let body = ctx.with_tx(|tx| tx.db.data().id().find(id).map(|data| data.body));
    if let Some(body) = body {
        Response::new(Body::from_bytes(body))
    } else {
        Response::builder().status(404).body(Body::empty()).unwrap()
    }
}

#[spacetimedb::http::router]
fn router() -> Router {
    Router::new().post("/insert", insert).get("/retrieve", retrieve)
}
```

## Design and implementation notes

- As mentioned above, the router is registered via a function, not a
`static` or `const` item. This is because `static` or `const`
initializers must be `const`, and it turns out to be a pain to make all
of the `Router` constructors be `const fn`s.
- The `#[handler]` macro clobbers the original function name with a
`const` variable of type `HttpHandler`. This is unfortunate, but AFAICT
necessary, 'cause we need to pass the string identifier for the handler
to the `Router`, not the function pointer, and Rust allows no (stable
and reliable) way to get a unique string identifier out of a function
item/value, nor to attach data or implement traits for function
items/values. The alternative(s) would involve changing the signature of
the `Router` methods to have uglier and more complex callsites, e.g.
like `.get("/retrieve", retrieve::handler())`, `.get("/retrieve",
handler!(retrieve))` or `.get::<retrieve>("/retrieve")`. I believe that
registering handlers will be much more common than calling their
functions, so I've chosen to make it so that registering them gets the
convenient syntax, even though the inability to call them directly will
be somewhat surprising.
- I haven't wired up energy handling or timing metrics for handler
execution to anywhere. Procedures are still in the same boat.
- HTTP requests to user-defined routes bypass the usual SpacetimeDB auth
middleware, meaning that the host does not validate (or inspect in any
way) `Authorization` headers in requests before invoking the
user-defined handler. This is required to allow arbitrary
user-programmable handling of `Authorization` headers, including those
in formations which SpacetimeDB would reject. As a result of this,
`HandlerContext` doesn't expose a `sender` or `sender_connection_id`.
- HTTP route paths may consist only of a very restrictive set of
characters. I've chosen this set to keep our options open in the future
to add additional syntax to routes, like for registering wildcard
segments and path parameters:
  - ASCII digits.
  - ASCII letters.
  - `-_~/`.
- The internal data structure that represents a `Router` is currently a
`Vec<Route>`, meaning that resolving a request to a route is
`O(num_routes)`. Registering a route checks against each previous route
for uniqueness, meaning that constructing a router is `O(num_routes ^
2)`. There are TODO comments to use a trie, but I think this can wait,
as I expect most databases to register few routes.
- Commit 999a7c3 contains a fix to a mostly-unrelated bug where a few
bindings introduced by the SATS derive macros were unhygienic and not in
a reserved namespace, leading to name conflicts. I discovered this
'cause I tried writing an HTTP handler named `index` to serve the
index/root of a website and it broke.

## Still TODO

- [x] Resolve various TODO comments in the diff.
- [x] Documentation.
- [x] C# bindings support.
- [x] C++ bindings support.
- [x] V8 host support.
- [x] TypeScript bindings support.

# API and ABI breaking changes

New APIs, currently flagged as `unstable`, which will eventually need
stability guarantees. No (intentional) breaking changes, or changes to
existing APIs at all.

# Expected complexity level and risk

3? Changes to our HTTP routing to support the user-defined routes.

# Testing

<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->

- [x] New smoketest of the behavior!
- [x] I dunno, maybe try hosting a simple webpage and see how it works?
- [x] Build a test app with Stripe integration.
  - @aasoni did this.

---------

Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
Co-authored-by: Jason Larabie <jason@clockworklabs.io>

Author: 3 people

Cargo.lock

@@ -30,6 +30,7 @@ target_sources(spacetimedb_cpp_library PRIVATE ${LIBRARY_SOURCES})
 
 # Require C++20 for consumers of this library without forcing global flags
 target_compile_features(spacetimedb_cpp_library PUBLIC cxx_std_20)
+target_compile_definitions(spacetimedb_cpp_library PRIVATE SPACETIMEDB_UNSTABLE_FEATURES)
 
 # Set include directories
 target_include_directories(spacetimedb_cpp_library
@@ -60,46 +61,5 @@ if(PROJECT_IS_TOP_LEVEL)
     set(CMAKE_EXPORT_COMPILE_COMMANDS ON)
 endif()
 
-# ---- Tests ----
-# Default: ON only when building this project directly; OFF when used via FetchContent/add_subdirectory
-if(CMAKE_VERSION VERSION_LESS 3.21)
-    # Fallback heuristic for older CMake
-    set(_is_top_level FALSE)
-    if(CMAKE_SOURCE_DIR STREQUAL CMAKE_CURRENT_SOURCE_DIR)
-        set(_is_top_level TRUE)
-    endif()
-else()
-    set(_is_top_level ${PROJECT_IS_TOP_LEVEL})
-endif()
-
-option(BUILD_TESTS "Build the test suite" ${_is_top_level})
-
-if(BUILD_TESTS AND NOT CMAKE_SYSTEM_NAME STREQUAL "Emscripten")
-    enable_testing()
-    
-    # Add test executable
-    add_executable(test_bsatn tests/main.cpp tests/module_library_unit_tests.cpp)
-    
-    # Link against the module library
-    target_link_libraries(test_bsatn PRIVATE spacetimedb_cpp_library)
-    
-    # Set C++20 standard for tests
-    target_compile_features(test_bsatn PRIVATE cxx_std_20)
-    
-    # Add test to CTest
-    add_test(NAME bsatn_tests COMMAND test_bsatn)
-    
-    # Add verbose test variant
-    add_test(NAME bsatn_tests_verbose COMMAND test_bsatn -v)
-    
-    # Set test properties
-    set_tests_properties(bsatn_tests PROPERTIES 
-        TIMEOUT 30
-        LABELS "unit"
-    )
-    
-    set_tests_properties(bsatn_tests_verbose PROPERTIES 
-        TIMEOUT 30
-        LABELS "unit;verbose"
-    )
-endif()
+# Unit/compile/smoke test harnesses live under `tests/` as standalone runners
+# rather than being built through the top-level library CMake target.

crates/bindings-cpp/CMakeLists.txt

@@ -126,6 +126,11 @@
 
 // Procedure context and macros
 #include "spacetimedb/procedure_macros.h"
+#ifdef SPACETIMEDB_UNSTABLE_FEATURES
+#include "spacetimedb/handler_context.h"
+#include "spacetimedb/router.h"
+#include "spacetimedb/http_handler_macros.h"
+#endif
 
 // =============================================================================
 // VIEW SYSTEM

crates/bindings-cpp/include/spacetimedb.h

@@ -216,6 +216,37 @@ int16_t __call_reducer__(
     BytesSource args,
     BytesSink error);
 
+STDB_EXPORT(__call_view__)
+int16_t __call_view__(
+    uint32_t id,
+    uint64_t sender_0, uint64_t sender_1, uint64_t sender_2, uint64_t sender_3,
+    BytesSource args,
+    BytesSink result);
+
+STDB_EXPORT(__call_view_anon__)
+int16_t __call_view_anon__(
+    uint32_t id,
+    BytesSource args,
+    BytesSink result);
+
+STDB_EXPORT(__call_procedure__)
+int16_t __call_procedure__(
+    uint32_t id,
+    uint64_t sender_0, uint64_t sender_1, uint64_t sender_2, uint64_t sender_3,
+    uint64_t conn_id_0, uint64_t conn_id_1,
+    uint64_t timestamp_microseconds,
+    BytesSource args_source,
+    BytesSink result_sink);
+
+STDB_EXPORT(__call_http_handler__)
+int16_t __call_http_handler__(
+    uint32_t id,
+    uint64_t timestamp_microseconds,
+    BytesSource request_source,
+    BytesSource request_body_source,
+    BytesSink response_sink,
+    BytesSink response_body_sink);
+
 // ========================================================================
 // WASI SHIMS
 // ========================================================================

crates/bindings-cpp/include/spacetimedb/abi/abi.h

@@ -0,0 +1,92 @@
+#ifndef SPACETIMEDB_HANDLER_CONTEXT_H
+#define SPACETIMEDB_HANDLER_CONTEXT_H
+
+#ifndef SPACETIMEDB_UNSTABLE_FEATURES
+#error "spacetimedb/handler_context.h requires SPACETIMEDB_UNSTABLE_FEATURES to be enabled"
+#endif
+
+#include <spacetimedb/abi/FFI.h>
+#include <spacetimedb/bsatn/timestamp.h>
+#include <spacetimedb/bsatn/uuid.h>
+#include <spacetimedb/http.h>
+#include <spacetimedb/internal/tx_execution.h>
+#include <spacetimedb/random.h>
+#include <spacetimedb/tx_context.h>
+#include <array>
+#include <cstdint>
+#include <functional>
+#include <memory>
+#include <optional>
+#include <type_traits>
+
+namespace SpacetimeDB {
+
+struct HandlerContext {
+    Timestamp timestamp;
+    HttpClient http;
+
+private:
+    mutable std::shared_ptr<StdbRng> rng_instance;
+    mutable uint32_t counter_uuid_ = 0;
+
+public:
+    HandlerContext() = default;
+    explicit HandlerContext(Timestamp t) : timestamp(t) {}
+
+    Identity identity() const {
+        std::array<uint8_t, 32> id_bytes;
+        ::identity(id_bytes.data());
+        return Identity(id_bytes);
+    }
+
+    StdbRng& rng() const {
+        if (!rng_instance) {
+            rng_instance = std::make_shared<StdbRng>(timestamp);
+        }
+        return *rng_instance;
+    }
+
+    Uuid new_uuid_v4() const {
+        std::array<uint8_t, 16> random_bytes;
+        rng().fill_bytes(random_bytes.data(), random_bytes.size());
+        return Uuid::from_random_bytes_v4(random_bytes);
+    }
+
+    Uuid new_uuid_v7() const {
+        std::array<uint8_t, 4> random_bytes;
+        rng().fill_bytes(random_bytes.data(), random_bytes.size());
+        return Uuid::from_counter_v7(counter_uuid_, timestamp, random_bytes);
+    }
+
+#ifdef SPACETIMEDB_UNSTABLE_FEATURES
+    template<typename Func>
+    auto with_tx(Func&& body) -> decltype(body(std::declval<TxContext&>())) {
+        auto make_reducer_ctx = [](Timestamp tx_timestamp) {
+            return ReducerContext(
+                Identity{},
+                std::nullopt,
+                tx_timestamp,
+                AuthCtx::internal()
+            );
+        };
+        return Internal::with_tx(make_reducer_ctx, body);
+    }
+
+    template<typename Func>
+    auto try_with_tx(Func&& body) -> decltype(body(std::declval<TxContext&>())) {
+        auto make_reducer_ctx = [](Timestamp tx_timestamp) {
+            return ReducerContext(
+                Identity{},
+                std::nullopt,
+                tx_timestamp,
+                AuthCtx::internal()
+            );
+        };
+        return Internal::try_with_tx(make_reducer_ctx, body);
+    }
+#endif
+};
+
+} // namespace SpacetimeDB
+
+#endif // SPACETIMEDB_HANDLER_CONTEXT_H

crates/bindings-cpp/include/spacetimedb/handler_context.h

@@ -3,6 +3,10 @@
 
 #pragma once
 
+#ifndef SPACETIMEDB_UNSTABLE_FEATURES
+#error "spacetimedb/http.h requires SPACETIMEDB_UNSTABLE_FEATURES to be enabled"
+#endif
+
 #include <string>
 #include <vector>
 #include <optional>
@@ -312,8 +316,10 @@ class HttpClient {
 
 } // namespace SpacetimeDB
 
-// Include implementation after class definition to avoid circular dependencies
-#ifdef SPACETIMEDB_UNSTABLE_FEATURES
+// Include implementation dependencies after class definition to avoid circular dependencies
+#if defined(SPACETIMEDB_UNSTABLE_FEATURES) && !defined(SPACETIMEDB_HTTP_CONVERT_H)
+#include "spacetimedb/logger.h"
+#include "spacetimedb/http_convert.h"
 #include "spacetimedb/http_client_impl.h"
 #endif
 

crates/bindings-cpp/include/spacetimedb/http.h

@@ -7,7 +7,7 @@
 #include "spacetimedb/http_convert.h"
 #include "spacetimedb/abi/abi.h"
 #include "spacetimedb/bsatn/bsatn.h"
-#include "spacetimedb/internal/Module.h"
+#include "spacetimedb/internal/runtime_registration.h"
 
 namespace SpacetimeDB {
 
@@ -23,9 +23,9 @@ inline Outcome<HttpResponse> HttpClient::SendImpl(const HttpRequest& request) {
     // Prepare body bytes
     const std::vector<uint8_t>& body_bytes = request.body.bytes;
 
-    // Call host function
-    // Note: For empty body, we need to pass a valid pointer, not null
-    const uint8_t* body_ptr = body_bytes.empty() ? reinterpret_cast<const uint8_t*>("") : body_bytes.data();
+    // The host ABI requires a non-null, in-bounds body pointer even when body_len == 0.
+    static const uint8_t empty_sentinel = 0;
+    const uint8_t* body_ptr = body_bytes.empty() ? &empty_sentinel : body_bytes.data();
 
     BytesSource out[2] = {BytesSource{0}, BytesSource{0}};
     Status status = procedure_http_request(
@@ -40,15 +40,11 @@ inline Outcome<HttpResponse> HttpClient::SendImpl(const HttpRequest& request) {
         if (status.inner == 21) {
             // Read error message from out[0]
             std::vector<uint8_t> error_bytes = Internal::ConsumeBytes(out[0]);
-            
-            LOG_INFO("HTTP: Error bytes: " + std::to_string(error_bytes.size()));
-            
+           
             // Decode BSATN string
             bsatn::Reader reader(error_bytes.data(), error_bytes.size());
             std::string error_message = bsatn::deserialize<std::string>(reader);
 
-            LOG_INFO("HTTP: Error message: " + error_message);
-            
             return Err<HttpResponse>(std::move(error_message));
         }
 
@@ -57,7 +53,6 @@ inline Outcome<HttpResponse> HttpClient::SendImpl(const HttpRequest& request) {
             return Err<HttpResponse>("HTTP requests are blocked inside transactions. Call HTTP before with_tx() or try_with_tx().");
         }
 
-        LOG_INFO("HTTP: Unknown error code: " + std::to_string(status.inner));
         return Err<HttpResponse>("HTTP request failed with status code: " + std::to_string(status.inner));
     }
 

crates/bindings-cpp/include/spacetimedb/http_client_impl.h

@@ -237,6 +237,12 @@ inline HttpRequest from_wire(const wire::HttpRequest& request) {
     return result;
 }
 
+inline HttpRequest from_wire(const wire::HttpRequest& request, std::vector<uint8_t> body) {
+    HttpRequest result = from_wire(request);
+    result.body.bytes = std::move(body);
+    return result;
+}
+
 // ==================== HttpResponse Conversions ====================
 
 /**
@@ -268,7 +274,16 @@ inline HttpResponse from_wire(const wire::HttpResponse& response) {
     return result;
 }
 
+inline std::pair<wire::HttpResponse, std::vector<uint8_t>> to_wire_split(const HttpResponse& response) {
+    return {to_wire(response), response.body.bytes};
+}
+
 } // namespace convert
 } // namespace SpacetimeDB
 
+#ifdef SPACETIMEDB_UNSTABLE_FEATURES
+#include "spacetimedb/logger.h"
+#include "spacetimedb/http_client_impl.h"
+#endif
+
 #endif // SPACETIMEDB_HTTP_CONVERT_H

crates/bindings-cpp/include/spacetimedb/http_convert.h

@@ -0,0 +1,61 @@
+#pragma once
+
+#ifndef SPACETIMEDB_UNSTABLE_FEATURES
+#error "spacetimedb/http_handler_macros.h requires SPACETIMEDB_UNSTABLE_FEATURES to be enabled"
+#endif
+
+#include "spacetimedb/handler_context.h"
+#include "spacetimedb/http.h"
+#include "spacetimedb/internal/runtime_registration.h"
+#include "spacetimedb/internal/template_utils.h"
+#include "spacetimedb/internal/v10_builder.h"
+#include "spacetimedb/macros.h"
+#include "spacetimedb/router.h"
+
+namespace SpacetimeDB::Internal {
+
+template<typename Func>
+inline void RegisterHttpHandlerMacro(const char* handler_name, Func func) {
+    using traits = function_traits<Func>;
+    static_assert(traits::arity == 2, "HTTP handlers must take exactly two arguments");
+    using ContextType = typename traits::template arg_t<0>;
+    using RequestType = typename traits::template arg_t<1>;
+    using ReturnType = typename traits::result_type;
+    static_assert(std::is_same_v<ContextType, HandlerContext>, "First parameter of HTTP handler must be HandlerContext");
+    static_assert(std::is_same_v<RequestType, HttpRequest>, "Second parameter of HTTP handler must be HttpRequest");
+    static_assert(std::is_same_v<ReturnType, HttpResponse>, "HTTP handlers must return HttpResponse");
+
+    std::function<HttpResponse(HandlerContext&, HttpRequest)> handler =
+        [func](HandlerContext& ctx, HttpRequest request) -> HttpResponse {
+            return func(ctx, std::move(request));
+        };
+    RegisterHttpHandlerHandler(handler_name, func, std::move(handler));
+    getV10Builder().RegisterHttpHandlerDef(handler_name);
+}
+
+} // namespace SpacetimeDB::Internal
+
+#define SPACETIMEDB_HTTP_HANDLER(handler_name, ctx_param, request_param) \
+    SpacetimeDB::HttpResponse handler_name(ctx_param, request_param); \
+    __attribute__((export_name("__preinit__60_http_handler_" #handler_name))) \
+    extern "C" void CONCAT(_spacetimedb_preinit_register_http_handler_, handler_name)() { \
+        ::SpacetimeDB::Internal::RegisterHttpHandlerMacro(#handler_name, handler_name); \
+    } \
+    SpacetimeDB::HttpResponse handler_name(ctx_param, request_param)
+
+#define SPACETIMEDB_HTTP_HANDLER_NAMED(handler_name, canonical_name, ctx_param, request_param) \
+    SpacetimeDB::HttpResponse handler_name(ctx_param, request_param); \
+    __attribute__((export_name("__preinit__60_http_handler_" #handler_name))) \
+    extern "C" void CONCAT(_spacetimedb_preinit_register_http_handler_, handler_name)() { \
+        ::SpacetimeDB::Internal::RegisterHttpHandlerMacro(#handler_name, handler_name); \
+        SpacetimeDB::Module::RegisterExplicitFunctionName(#handler_name, canonical_name); \
+    } \
+    SpacetimeDB::HttpResponse handler_name(ctx_param, request_param)
+
+#define SPACETIMEDB_HTTP_ROUTER(router_name) \
+    SpacetimeDB::Router router_name(); \
+    __attribute__((export_name("__preinit__61_http_router_" #router_name))) \
+    extern "C" void CONCAT(_spacetimedb_preinit_register_http_router_, router_name)() { \
+        ::SpacetimeDB::Internal::getV10Builder().RegisterHttpRouter(router_name()); \
+    } \
+    SpacetimeDB::Router router_name()