feat(dev,perf): add JsRuntime performance harness

Author: azasypkin

.github/workflows/ci.yml

@@ -73,6 +73,54 @@ jobs:
 
       - name: Clippy
         run: cargo clippy --all-targets -- -D warnings
+  ci-perf:
+    name: JS Runtime Perf Harness (Linux)
+    runs-on: ubuntu-latest
+    timeout-minutes: 20
+    permissions:
+      contents: write
+    steps:
+      - uses: actions/checkout@v6
+        with:
+          ref: ${{ github.ref }}
+
+      - name: Install Protoc
+        uses: arduino/setup-protoc@v3
+
+      - name: Install Rust toolchain
+        run: |
+          rustup toolchain install stable
+          rustup override set stable
+
+      - name: Cache Dependencies
+        uses: Swatinem/rust-cache@v2
+        with:
+          shared-key: "ci-perf"
+          cache-all-crates: true
+
+      - name: Set Node.js 22.x
+        uses: actions/setup-node@v6
+        with:
+          node-version: 22.x
+
+      - name: Run perf harness and record history
+        run: make perf ANALYZE=1
+
+      - name: Upload perf report JSON
+        uses: actions/upload-artifact@v7
+        with:
+          name: perf-report
+          path: /tmp/perf.json
+          retention-days: 30
+
+      - name: Commit perf history
+        if: success() && github.ref == 'refs/heads/main'
+        run: |
+          git config user.name "github-actions[bot]"
+          git config user.email "github-actions[bot]@users.noreply.github.com"
+          git add .perf/history.jsonl
+          git diff --cached --quiet || git commit -m "chore: update JS runtime perf history [skip ci]" && git push
+
   ci-web-scraper:
     name: Web Scraper Build (Linux)
     runs-on: ubuntu-latest

.perf/config.json

@@ -0,0 +1,14 @@
+{
+  "thresholds": {
+    "p50": 15,
+    "p99": 20,
+    "throughput": 15,
+    "peakRssDeltaKb": 25
+  },
+  "scenarios": [
+    "cold_start_trivial",
+    "steady_state_trivial",
+    "steady_state_extractor",
+    "concurrent_extractors_8x"
+  ]
+}

.perf/history.jsonl

@@ -0,0 +1,145 @@
+# AGENTS.md
+
+## JS Runtime Performance Harness (`benches/js-runtime-perf/`)
+
+### Overview
+
+Retrack embeds a Deno/V8 runtime to execute user-supplied extractor and formatter scripts.
+Retrack keeps a single long-lived worker thread that owns one V8 isolate and receives work
+over an `mpsc` channel. This harness measures the latency, throughput, and peak RSS delta 
+of that runtime so changes to the architecture (context-per-call, pooling, shared HTTP client,
+startup snapshots, etc.) can be evaluated with real numbers.
+
+The harness is self-contained: it lives inside the `retrack` workspace, links against the
+real `retrack::js_runtime::JsRuntime`.
+
+The harness is **advisory / warn-only**. CI records a new history entry on every push to
+`main` and prints a table with per-metric deltas, but it never fails a build on
+regressions. Thresholds in `.perf/config.json` only control when warnings are emitted.
+
+### Scenario catalogue
+
+All scenarios use a default `JsRuntimeConfig` with a 10 MiB heap and a 10s execution
+budget, matching production settings.
+
+| Scenario                   | What it measures                                                                                                       |
+|----------------------------|------------------------------------------------------------------------------------------------------------------------|
+| `cold_start_trivial`       | Full worker-thread startup: `JsRuntime::init()` + fresh V8 isolate + first script execution, trivial script.           |
+| `steady_state_trivial`     | Serial executions of a trivial script through a single long-lived `JsRuntime`.                                         |
+| `steady_state_extractor`   | Realistic extractor: decodes a `Uint8Array` response body, parses JSON, filters/maps, re-encodes the result.           |
+| `concurrent_extractors_8x` | `tokio::spawn` burst of `N` extractor calls sharing one `Arc<JsRuntime>`; exposes the single-worker-thread bottleneck. |
+
+The last scenario is deliberately designed to show that Retrack's current mpsc-based
+architecture serialises concurrent work onto one worker thread, which is the exact shape
+of the bottleneck we want any future optimisation to address.
+
+### Running locally
+
+```bash
+# Full run + comparison table + history append (from components/retrack/)
+make perf ANALYZE=1
+
+# Run only, no history touch (useful when iterating locally and discarding results)
+make perf
+
+# Re-analyze an existing /tmp/perf.json (e.g. downloaded from CI) without rerunning
+make perf-analyze
+
+# Smoke test (fast)
+make perf ANALYZE=1 PERF_ITERATIONS=20 PERF_WARMUP=5
+
+# Single scenario
+make perf ANALYZE=1 PERF_SCENARIOS=steady_state_extractor
+
+# Custom output path
+make perf PERF_OUTPUT=/tmp/perf-baseline.json
+
+# View HTML report (opens scripts/perf-report.html, then load .perf/history.jsonl)
+make perf-report
+```
+
+`make perf` produces `/tmp/perf.json` and prints a one-line summary per scenario. When
+`ANALYZE=1` is set it then invokes `scripts/analyze-perf.ts`, which compares the fresh
+report to the last entry in `.perf/history.jsonl`, prints a table with Δp50/Δp99/Δops/Δrss
+columns, and appends to history **only when at least one tracked metric moved by more
+than 0.1 %** (see "History append gating" below). `make perf-analyze` is the same
+analyze-only tail, exposed separately for re-analyzing a file without rerunning the
+harness.
+
+### Interpreting the output
+
+The printed table uses the last recorded history entry as the baseline:
+
+```
+Scenario                             p50       p99    throughput       rss      Δp50      Δp99      Δops      Δrss
+steady_state_extractor            1.45ms    1.82ms       688.9/s     512KB     -2.1%     -3.0%     +1.4%      0.0%
+```
+
+- **Δp50 / Δp99**: percentage change in latency vs the previous run. Warnings fire when
+  these exceed the thresholds in `.perf/config.json` (`p50`, `p99`).
+- **Δops**: percentage change in throughput. Warnings fire on a _decrease_ below
+  `-thresholds.throughput` (i.e. getting slower).
+- **Δrss**: percentage change in peak RSS delta. Warnings fire above
+  `thresholds.peakRssDeltaKb`.
+
+A first run prints "First run recorded - no comparison available." and establishes the
+baseline.
+
+### History append gating
+
+`scripts/analyze-perf.ts` does not append unconditionally. It diffs the fresh report
+against the last entry in `.perf/history.jsonl` across a whitelist of tracked metrics
+(`p50_us`, `p90_us`, `p99_us`, `max_us`, `throughput_ops_per_sec`, `peak_rss_delta_kb`).
+If every tracked metric on every scenario is within ±0.1 % of the previous entry, the
+file is left untouched and the CLI prints `All tracked metrics within ±0.1% of the
+previous run; history not updated.` When something moves, the append happens and the
+output names the scenario/metric that tripped the threshold.
+
+This matters for the CI commit step: because `history.jsonl` is modified only on
+material movement, the `git diff --cached --quiet || git commit` check becomes an
+effective "commit only if something changed" — pushes with steady-state numbers no
+longer produce noisy chore commits on `main`.
+
+The threshold is hard-coded at `HISTORY_APPEND_THRESHOLD_PCT = 0.1` in
+`scripts/analyze-perf.ts`. Adjust there if it proves too tight or too loose.
+Scenario additions/removals are treated as unconditionally material (always appended).
+Structural zero-valued metrics (e.g. `peak_rss_delta_kb = 0`) are handled explicitly —
+`0 → 0` is unchanged, `0 → anything` or `anything → 0` triggers an append.
+
+### CI contract
+
+- `.github/workflows/ci.yml` has a `ci-perf` job that runs on every push to `main`.
+- It builds the harness in release mode, runs `make perf ANALYZE=1` (which produces
+  the report, prints the delta table, and appends to history only on material
+  movement), uploads `/tmp/perf.json` as an artefact, and commits the updated
+  `.perf/history.jsonl` back to `main` with `[skip ci]` in the commit message.
+- The commit step is a no-op when nothing moved — `history.jsonl` is unmodified, so
+  `git diff --cached --quiet` is true.
+- The job **never fails on regressions**. Warnings are visible in the job log; acting on
+  them is a human decision.
+
+### File locations
+
+```
+benches/js-runtime-perf/Cargo.toml               # Workspace member, depends on `retrack` + `retrack-types`
+benches/js-runtime-perf/src/main.rs              # CLI driver
+benches/js-runtime-perf/src/measure.rs           # hdrhistogram recorder, peak RSS probe
+benches/js-runtime-perf/src/report.rs            # JSON output shape (camelCase top-level)
+benches/js-runtime-perf/src/scenarios/*.rs       # One scenario per file
+benches/js-runtime-perf/scripts/*.js             # JS fixtures loaded via `include_str!`
+src/lib.rs                                       # Minimal library target exposing `js_runtime` + `config`
+.perf/config.json                                # Scenario list + warning thresholds
+.perf/history.jsonl                              # Append-only history (one JSON per run)
+scripts/analyze-perf.ts                          # Node 22 analyzer (reads /tmp/perf.json)
+scripts/perf-report.html                         # Standalone HTML viewer for history.jsonl
+```
+
+### Tuning
+
+- To relax or tighten warnings, edit `.perf/config.json`. Values are percentages.
+- To add a scenario: create a module under `benches/js-runtime-perf/src/scenarios/`,
+  register it in `scenarios.rs` (both the `ALL` slice and the `run` dispatcher), and add
+  its name to `.perf/config.json`.
+- Benchmark results are platform-sensitive. History entries include `env.os`, `env.arch`,
+  and `env.cpuModel` for this reason; absolute numbers from a laptop are not directly
+  comparable to those from a CI runner.

AGENTS.md

@@ -5,12 +5,17 @@ authors = ["Aleh Zasypkin <dev@retrack.dev>"]
 description = "Tracks changes in a web page, API, or file."
 edition = "2024"
 
+[lib]
+name = "retrack"
+path = "src/lib.rs"
+
 [[bin]]
 name = "retrack"
 path = "src/main.rs"
 
 [workspace]
 members = [
+    "benches/js-runtime-perf",
     "components/retrack-types"
 ]
 

Cargo.lock

@@ -2,7 +2,7 @@ COMPOSE_DB   := dev/docker/docker-compose.yml
 ENV_FILE     := .env
 CHROME_PATH  ?= /Applications/Google Chrome.app/Contents/MacOS/Google Chrome
 
-.PHONY: dev-up dev-down api scraper-setup scraper scraper-debug db-reset db-migrate test test-api test-scraper fmt clippy check docker-api docker-scraper docker-scraper-camoufox docker-pin-digests clean help
+.PHONY: dev-up dev-down api scraper-setup scraper scraper-debug db-reset db-migrate test test-api test-scraper fmt clippy check docker-api docker-scraper docker-scraper-camoufox docker-pin-digests clean help perf perf-analyze perf-report
 
 ## ---------- Development ----------
 
@@ -82,6 +82,31 @@ docker-scraper-camoufox: ## Build the Web Scraper (Camoufox/Firefox) Docker imag
 docker-pin-digests: ## Re-pin base images in Dockerfiles to current SHA256 digests.
 	./dev/scripts/docker-pin-digests.sh
 
+## ---------- JS Runtime Perf Harness ----------
+
+PERF_OUTPUT ?= /tmp/perf.json
+PERF_ITERATIONS ?= 500
+PERF_WARMUP ?= 50
+PERF_CONCURRENCY ?= 8
+PERF_SCENARIOS ?= all
+
+perf: ## Run the JS runtime perf harness. Use ANALYZE=1 to also print the comparison table and record to .perf/history.jsonl (ARGS='--scenarios cold_start_trivial --iterations 100').
+	cargo run --release -p js-runtime-perf -- \
+		--scenarios $(PERF_SCENARIOS) \
+		--iterations $(PERF_ITERATIONS) \
+		--warmup $(PERF_WARMUP) \
+		--concurrency $(PERF_CONCURRENCY) \
+		--output $(PERF_OUTPUT) $(ARGS) \
+		$(if $(ANALYZE),&& node scripts/analyze-perf.ts $(PERF_OUTPUT))
+
+perf-analyze: ## Analyze an existing $(PERF_OUTPUT) without rerunning the harness (equivalent to the ANALYZE=1 tail of `make perf`).
+	node scripts/analyze-perf.ts $(PERF_OUTPUT)
+
+perf-report: ## Open the HTML perf viewer. Load .perf/history.jsonl inside it.
+	@open scripts/perf-report.html 2>/dev/null || \
+		xdg-open scripts/perf-report.html 2>/dev/null || \
+		echo 'Open scripts/perf-report.html in your browser'
+
 ## ---------- Misc ----------
 
 clean: ## Remove build artifacts.

Cargo.toml

@@ -0,0 +1,23 @@
+[package]
+name = "js-runtime-perf"
+version = "0.1.0"
+edition = "2024"
+publish = false
+description = "Performance harness for the Retrack JS runtime."
+
+[[bin]]
+name = "js-runtime-perf"
+path = "src/main.rs"
+
+[dependencies]
+anyhow = "1.0.102"
+clap = { version = "4.6.0", features = ["derive"] }
+futures = "0.3.32"
+hdrhistogram = "7.5.4"
+http = "1.4.0"
+libc = "0.2.185"
+retrack = { path = "../.." }
+retrack-types = { path = "../../components/retrack-types" }
+serde = { version = "1.0.228", features = ["derive"] }
+serde_json = "1.0.149"
+tokio = { version = "1.50.0", features = ["macros", "rt", "rt-multi-thread", "sync", "time"] }

Makefile

@@ -0,0 +1,11 @@
+// Realistic extractor: decodes the first response body (Uint8Array), parses
+// JSON, filters items, and re-encodes the result. Exercises the same code
+// paths a production extractor script would.
+(() => {
+  const response = context.responses[0];
+  const payload = JSON.parse(Deno.core.decode(new Uint8Array(response.body)));
+  const filtered = (payload.items || [])
+    .filter((item) => item.value > 10)
+    .map((item) => ({ id: item.id, value: item.value * 2 }));
+  return { body: Deno.core.encode(JSON.stringify({ status: response.status, filtered })) };
+})();

benches/js-runtime-perf/Cargo.toml

@@ -0,0 +1,4 @@
+// Minimal extractor-shaped script: returns an empty body envelope, just
+// enough for ExtractorScriptResult to deserialise. Used by the
+// cold-start and steady-state trivial scenarios.
+(() => ({ body: Deno.core.encode("{}") }))();