chore: add otel support to e2e tests (#1756)

now we can setup otel to track performance of each test and have properly defined spans per test

Author: filipecabaco

test/e2e/README.md

@@ -11,28 +11,43 @@
 | `--port` | | Override URL port (useful for local) |
 | `--test` | | Comma-separated list of test categories to run (runs all if omitted) |
 | `--json` | | Output results as JSON to stdout (all other output goes to stderr) |
+| `--url` | | Override project URL (e.g. `http://127.0.0.1:54321`) |
+| `--db-url` | | Override database URL (e.g. `postgresql://postgres:postgres@127.0.0.1:54322/postgres`) |
+| `--otel` | `OTEL_EXPORTER_OTLP_ENDPOINT` | OTLP HTTP endpoint for tracing (e.g. `http://localhost:4318`) |
+| | `OTEL_API_TOKEN` | Bearer token for authenticated OTLP endpoints |
 
 Sensitive credentials (`--secret-key`, `SUPABASE_DB_PASSWORD`) should be set as environment variables to avoid them appearing in shell history.
 
 A random test user is created at the start of each run and deleted automatically when it finishes.
 
 ## Test categories
 
-Pass any combination to `--test` as a comma-separated list:
+Pass any combination to `--test` as a comma-separated list. Use `functional` to run all non-load suites, or `load` to run all load suites.
 
-| Category | Description |
-|---|---|
-| `connection` | WebSocket connect latency and broadcast throughput |
-| `load` | Postgres changes and presence throughput (INSERT / UPDATE / DELETE) |
-| `broadcast` | Self-broadcast and REST broadcast API |
-| `presence` | Presence join on public and private channels |
-| `authorization` | Private channel allow/deny checks |
-| `postgres-changes` | Filtered INSERT, UPDATE, DELETE events and concurrent changes |
-| `broadcast-changes` | Database-triggered broadcast INSERT, UPDATE, DELETE events |
+| Category | Suites | Tests |
+|---|---|---|
+| `connection` | connection | First connect latency; broadcast message throughput |
+| `load` | load-postgres-changes | Postgres system message latency; INSERT / UPDATE / DELETE throughput via postgres changes |
+| | load-presence | Presence join throughput |
+| | load-broadcast-from-db | Broadcast-from-database throughput |
+| | load-broadcast | Self-broadcast throughput; REST broadcast API throughput |
+| | load-broadcast-replay | Broadcast replay throughput on channel join |
+| `broadcast` | broadcast extension | Self-broadcast receive; REST broadcast API send-and-receive |
+| `presence` | presence extension | Presence join on public channels; presence join on private channels |
+| `authorization` | authorization check | Private channel denied without permissions; private channel allowed with permissions |
+| `postgres-changes` | postgres changes extension | Filtered INSERT, UPDATE, DELETE events; concurrent INSERT + UPDATE + DELETE |
+| `broadcast-changes` | broadcast changes | DB-triggered broadcast for INSERT, UPDATE, DELETE |
+| `broadcast-replay` | broadcast replay | Replayed messages delivered on join; `meta.replayed` flag set; messages before `since` not replayed |
 
 ```bash
 # Run only connection and broadcast tests
 ./realtime-check --env local --publishable-key <key> --secret-key <key> --test connection,broadcast
+
+# Run all load tests
+./realtime-check --env local --publishable-key <key> --secret-key <key> --test load
+
+# Run all functional (non-load) tests
+./realtime-check --env local --publishable-key <key> --secret-key <key> --test functional
 ```
 
 ## JSON output
@@ -51,12 +66,31 @@ The pre-built binary requires no runtime — just run it directly.
 
 ### Local project
 
+A `supabase/config.toml` is included, so `supabase start` works out of the box.
+
 ```bash
 supabase start
 SUPABASE_SERVICE_ROLE_KEY=<service-role-key> \
   ./realtime-check --env local --publishable-key <anon-key>
 ```
 
+### Local project with tracing
+
+```bash
+supabase start
+docker compose up -d  # starts Jaeger at http://localhost:16686
+SUPABASE_SERVICE_ROLE_KEY=<service-role-key> \
+  ./realtime-check --env local --publishable-key <anon-key> --otel http://localhost:4318
+```
+
+For authenticated OTLP endpoints, set `OTEL_API_TOKEN` and it will be sent as a `Bearer` token:
+
+```bash
+SUPABASE_SERVICE_ROLE_KEY=<service-role-key> \
+OTEL_API_TOKEN=<token> \
+  ./realtime-check --env local --publishable-key <anon-key> --otel https://otlp.example.com
+```
+
 ### Remote project
 
 ```bash
@@ -102,7 +136,7 @@ SUPABASE_SERVICE_ROLE_KEY=<key> SUPABASE_DB_PASSWORD=<pw> \
   ./result/bin/realtime-check --project <ref> --publishable-key <key>
 ```
 
-> **Note:** The nix build locks the dependency hash in `flake.nix`. If you update `package.json` or `bun.lock`, run `nix build` once — it will fail with the new hash in the error output — then update `outputHash` in `flake.nix` accordingly.
+`bun run nix` calls `nix-build.sh`, which automatically updates the `outputHash` in `flake.nix` when `package.json` or `bun.lock` change — no manual hash update needed.
 
 ---
 

test/e2e/bun.lock

@@ -0,0 +1,11 @@
+# E2e testing infrastructure. Requires `supabase start` to be running first.
+# Run from test/e2e:
+#   docker compose up -d
+#   OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 bun run realtime-check.ts --env local
+services:
+  jaeger:
+    image: jaegertracing/jaeger:2.5.0
+    container_name: e2e-jaeger
+    ports:
+      - "16686:16686"
+      - "4318:4318"

test/e2e/docker-compose.yml

@@ -30,7 +30,7 @@
           installPhase = "cp -r node_modules $out";
           outputHashMode = "recursive";
           outputHashAlgo = "sha256";
-          outputHash = "sha256-MK55AYy2z5nY7B30o8vt34+wk+86Zruz/q2ZDmA951c=";
+          outputHash = "sha256-I7ZNZkyK83Lk+Ut3j6FngvWNfIl8JaW2cF4bfyVf5TQ=";
         };
       in {
         packages.default = pkgs.stdenv.mkDerivation {

test/e2e/flake.nix

@@ -0,0 +1,36 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+FLAKE="flake.nix"
+FAKE_HASH="sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA="
+
+# Replace current hash with fake to force nix to reveal the correct one
+sed -i.bak "s|outputHash = \"sha256-.*\";|outputHash = \"${FAKE_HASH}\";|" "$FLAKE"
+rm -f "${FLAKE}.bak"
+
+echo "Probing for correct node_modules hash..."
+NIX_OUT=$(nix build 2>&1 || true)
+
+REAL_HASH=$(echo "$NIX_OUT" | grep "got:" | awk '{print $2}')
+
+if [[ -z "$REAL_HASH" ]]; then
+  # No hash mismatch error — either build succeeded or failed for another reason
+  if echo "$NIX_OUT" | grep -q "error:"; then
+    echo "Build failed:"
+    echo "$NIX_OUT"
+    git checkout -- "$FLAKE" 2>/dev/null || true
+    exit 1
+  else
+    echo "Hash was already correct. Build succeeded."
+    echo "Done. Binary available at ./result/bin/realtime-check"
+    exit 0
+  fi
+fi
+
+echo "Updating hash to: $REAL_HASH"
+sed -i.bak "s|outputHash = \"${FAKE_HASH}\";|outputHash = \"${REAL_HASH}\";|" "$FLAKE"
+rm -f "${FLAKE}.bak"
+
+echo "Building with correct hash..."
+nix build
+echo "Done. Binary available at ./result/bin/realtime-check"