feat(bench): publication — first pinned-host run + canonical docs (SQLR-16, sub-phase 9.6) (#107)

Closes out the benchmark suite. Three deliverables:

  1. scripts/compare.py — pure-stdlib Python diff renderer.
     Reads two JSON envelopes, matches samples by (workload,
     driver), prints a Markdown table with deltas + ratios.
     Same-version-only per Q8; cross-host pairs flagged.

  2. First "official" pinned-host run committed under
     benchmarks/results/ — 35 samples across all 12 workloads
     × {SQLRite, SQLite-WAL+NORMAL, DuckDB on Group B}. M1 Pro,
     macOS 23.5.0, criterion defaults (3 s warm-up + 5 s
     measurement, 100 samples on light workloads / 10 on heavy
     ones). The header `dirty=true` flag reflects 9.6's working-
     tree state at run time (this PR's docs uncommitted); the
     measurements only depend on the bench binary built from
     the committed bench-9.5-duckdb tip.

  3. docs/benchmarks.md — canonical user-facing reference.
     Mirrors how docs/fts.md is the canonical FTS reference
     for Phase 8: TOC, quick-start, what-the-suite-measures
     summary, reading-the-numbers methodology (Q3 SQLite
     tuning, parser tax, DuckDB OLTP weakness), the headline
     table, the engineering debts surfaced (SQLR-18 / 19 /
     20 / 21), reproducing-a-run + comparing-two-runs sections,
     and a "what's NOT measured" section.

Top-level README.md gains a "Benchmarks" section pointing
at docs/benchmarks.md. docs/_index.md promotes benchmarks.md
into a top-level reference (alongside fts.md), and demotes
benchmarks-plan.md to a "design rationale, historical" pointer.

Plan-doc status banner updated to "all six sub-phases shipped."

The benchmarks/results/.gitignore is rewritten — instead of
the strict "ignore everything until 9.6" rule, it now
documents the convention ("only pinned-host runs get
committed; PR review enforces it") and allows *.json files
to be added explicitly. Local runs still show up as
untracked in `git status`, which is the prompt to either
commit them as official or leave them out.

Verification:
  - cargo fmt --all -- --check clean
  - cargo clippy -p sqlrite-benchmarks --all-targets clean
  - Full CI-equivalent build / test / doc green
    (--exclude desktop / python / nodejs / benchmarks)
  - cargo test -p sqlrite-benchmarks passes (5 tests)
  - compare.py syntax-checked; round-trips a synthetic
    envelope pair into a Markdown diff
  - Official run completed: 35 samples, see
    benchmarks/results/2026-05-07-apple-9ffd55a5.json

Refs: SQLR-16 (parent execution).

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: joaoh82

README.md

@@ -114,6 +114,12 @@ In-depth documentation lives under [`docs/`](docs/). Start at [`docs/_index.md`]
 - [Design decisions](docs/design-decisions.md), [Roadmap](docs/roadmap.md)
 - Internals: [File format](docs/file-format.md), [Pager](docs/pager.md), [Storage model](docs/storage-model.md), [SQL engine](docs/sql-engine.md)
 
+### Benchmarks
+
+SQLRite ships with a [curated benchmark suite](docs/benchmarks.md) that pits the engine against `SQLite` (and optionally `DuckDB`) on twelve OLTP, SQL-feature-scaling, and AI-era workloads (vector top-10, BM25, hybrid retrieval). Run locally with `make bench` (lean, ~5 min) or `make bench-duckdb` (with the analytical comparator on Group B, ~30 min). Pinned-host JSON envelopes get committed under [`benchmarks/results/`](benchmarks/results/) so trends can be diffed mechanically; `benchmarks/scripts/compare.py` renders any two runs as a Markdown diff.
+
+The [canonical reference](docs/benchmarks.md) carries headline numbers, methodology notes (Q3 SQLite tuning, parser-tax framing), and the engineering debts (SQLR-18 / 19 / 20 / 21) the suite surfaced honestly.
+
 ### Requirements
 Before you begin, ensure you have met the following requirements:
 * Rust (latest stable) – [How to install Rust](https://www.rust-lang.org/en-US/install.html)

benchmarks/README.md

@@ -2,7 +2,7 @@
 
 Benchmark suite for SQLRite vs SQLite (and friends). Tracks task **SQLR-4** / **SQLR-16**.
 
-> **Status (2026-05-07):** sub-phases 9.1 + 9.2 + 9.3 + 9.4 + 9.5 landed — harness + all 12 workloads + DuckDB driver. The first official pinned-host JSON + canonical `docs/benchmarks.md` reference land in 9.6. See [`docs/benchmarks-plan.md`](../docs/benchmarks-plan.md) for the canonical design + the resolved Q1–Q8 decisions.
+> **Status (2026-05-07):** all six sub-phases (9.1–9.6) shipped — harness + 12 workloads + DuckDB driver + canonical [`docs/benchmarks.md`](../docs/benchmarks.md) reference + first official pinned-host run committed under [`results/`](results/). See [`docs/benchmarks-plan.md`](../docs/benchmarks-plan.md) for the design rationale + the resolved Q1–Q8 decisions.
 
 ## Quick start
 
@@ -30,6 +30,8 @@ Three groups (full descriptions in [`docs/benchmarks-plan.md`](../docs/benchmark
 - **Group B — SQL-feature scaling.** W7 aggregate, W8 GROUP BY, W9 INNER JOIN. ✅ shipped (9.3).
 - **Group C — Differentiators.** W10 vector top-10, W11 BM25 top-10, W12 hybrid retrieval. ✅ shipped (9.4).
 
+For headline numbers + reading-the-numbers methodology + the engineering debts the suite surfaced, see the canonical [**`docs/benchmarks.md`**](../docs/benchmarks.md).
+
 Workloads are versioned (`W1.v1`, `W1.v2`, …) per Q8 — bumping the version is the explicit "I changed the benchmark" gesture.
 
 ## Drivers
@@ -150,20 +152,6 @@ Single-statement aggregate. No WHERE, no GROUP BY. All three engines walk every
 
 **Read this as:** GROUP BY at low cardinality is fine on SQLRite; high-cardinality work is broken. DuckDB's vectorized hash aggregator dominates SQLite at every cardinality — that's the analytical-engine value proposition in one workload.
 
-### W8 — `SELECT k, COUNT(*) FROM big GROUP BY k` (three cardinalities)
-
-1M-row table, GROUP BY at 10 / 1k / 100k distinct keys.
-
-| Cardinality | SQLRite | SQLite (rusqlite, WAL+NORMAL) | Notes |
-|---|---|---|---|
-| 10 groups | ~204 ms | ~460 ms | SQLRite faster — likely SQLite's planner picks a sort-then-group path here (small group counts often slower than a hash). Not the headline; small samples. |
-| 1,000 groups | ~1.50 s | ~242 ms | ~6.2× |
-| 100,000 groups | **skipped** | ~241 ms | SQLRite skipped by default — see note below |
-
-**SQLRite × 100k-cardinality is skipped by default** in `make bench`. A first measurement clocked ~245 s/iter (~41 min for 10 samples), strongly suggesting an O(n × cardinality) path inside the GROUP BY executor (a hash aggregator should be O(n + groups)). **SQLR-19** tracks the investigation. Set `SQLRITE_BENCH_W8_CARD_100K_SQLRITE=1` once that lands to re-include the bucket.
-
-**Read this as:** GROUP BY at low cardinality is fine; high-cardinality work is broken. The fix is probably a `Vec`-backed group store → `HashMap`-backed.
-
 ### W9 — INNER JOIN, customer ↔ order, probe by customer PK
 
 Plan target was two **100k-row tables**. v1 ships at **10k rows** because SQLRite's join executor doesn't push the ON predicate down to an index probe on the inner side — at the 100k scale, per-iter cost was >5 minutes (88 minutes of measured runtime didn't produce a single sample before the smoke run was killed). **SQLR-20** tracks the join-planner / inner-side index-probe fix. Until then, v1 = 10k rows; bumping to 100k follows the fix + a `W9.v2` tag.

benchmarks/results/.gitignore

@@ -1,7 +1,15 @@
-# Local bench runs land here. Only the "official" pinned-host runs
-# get committed (lands in 9.6 — see `docs/benchmarks-plan.md`). Until
-# 9.6, ignore everything except this rule + .gitkeep so the directory
-# stays in git.
-*
-!.gitkeep
-!.gitignore
+# Local bench runs land here.
+#
+# Convention: only "official" pinned-host runs get committed —
+# the project owner's reproducible host, with `dirty=false` in
+# the JSON envelope, suitable for the headline numbers in
+# `docs/benchmarks.md`. Casual / WIP runs aren't.
+#
+# This `.gitignore` doesn't enforce that — `*.json` files are
+# committable. Discipline lives in PR review: don't merge a
+# JSON that wasn't run from a clean tree on the canonical host.
+#
+# Local runs from a dirty tree are still on disk; just decline
+# to `git add` them. `git status` shows them as "untracked,"
+# which is the prompt to either commit them as official or
+# leave them out.