Skip to content

Commit 3e16c69

Browse files
joaoh82claude
andauthored
feat(bench): v2 republish — post-SQLR-23 pinned-host run + headline update (SQLR-25) (#114)
Reruns the full 12-workload suite (SQLRite + SQLite + DuckDB on Group B) on the canonical Apple M1 Pro / macOS 23.5.0 host after SQLR-23 (parser plan cache + bound `?` params + `Value::Vector`) bumped every workload from `v1 → v2` and SQLR-28 took W10 → `v3`. The v1 numbers in `docs/benchmarks.md` were dominated by per-iter SQL parsing on the bench driver's SQLRite path; v2 measures the post-binding reality. Headline: **W10 HNSW: 126.81 ms → 2.40 ms (~53× faster)**, with brute-force-vs-HNSW gap now ~50× at 10k corpus (was ~1.1× in v1, masked by parser tax + L2-only probe). Other parser-tax-dominated paths tightened: W1 read-by-PK ~2.5×, W6 secondary-index ~2.6×, W3 bulk insert ~1.7×, W11 BM25 top-10 ~2.2× faster than v1. Compute-bound paths (W4 single-row insert, W5 mixed OLTP, W7 SUM, W8 GROUP BY) moved a few percent — parser cost was a small slice of total time there. Changes: - `benchmarks/results/2026-05-08-apple-ac84d560.json` — canonical v2 envelope, 35 samples (Group A SQLRite+SQLite + Group B SQLRite+SQLite+DuckDB + Group C SQLRite-only). - `docs/benchmarks.md` — Headline numbers now reflect v2; v1 retained in a "Historical (v1, retired)" section so the methodology shift is visible. "Parser tax" sidebar updated to point at the new run. - `benchmarks/README.md` — adds W10.v3 result table (the v1 retired table stays alongside) + status banner bumped to 2026-05-08. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent ac84d56 commit 3e16c69

3 files changed

Lines changed: 461 additions & 24 deletions

File tree

benchmarks/README.md

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
Benchmark suite for SQLRite vs SQLite (and friends). Tracks task **SQLR-4** / **SQLR-16**.
44

5-
> **Status (2026-05-07):** all six sub-phases (9.1–9.6) shipped — harness + 12 workloads + DuckDB driver + canonical [`docs/benchmarks.md`](../docs/benchmarks.md) reference + first official pinned-host run committed under [`results/`](results/). See [`docs/benchmarks-plan.md`](../docs/benchmarks-plan.md) for the design rationale + the resolved Q1–Q8 decisions.
5+
> **Status (2026-05-08):** all six sub-phases (9.1–9.6) shipped — harness + 12 workloads + DuckDB driver + canonical [`docs/benchmarks.md`](../docs/benchmarks.md) reference + pinned-host runs committed under [`results/`](results/). The v2 republish (SQLR-25) lands the post-SQLR-23 / SQLR-28 numbers; the v1 baseline is retained as historical. See [`docs/benchmarks-plan.md`](../docs/benchmarks-plan.md) for the design rationale + the resolved Q1–Q8 decisions.
66
77
## Quick start
88

@@ -172,7 +172,7 @@ Even at 10k scale, the gap is large:
172172

173173
10k 384-dim vectors. Two variants per the plan: brute-force (no index) and HNSW (`CREATE INDEX … USING hnsw (embedding) WITH (metric = 'cosine')`, per SQLR-28). SQLRite-only — `sqlite-vec` extension wiring is a follow-up (`rusqlite[bundled]` doesn't ship it; loading a pre-compiled `.dylib` at runtime is non-trivial and was out of scope for v1).
174174

175-
The **W10.v1** numbers below were taken before SQLR-23 (parser-bound) and SQLR-28 (HNSW probe was L2-only, so the HNSW variant silently fell through to brute-force on cosine queries) — they are retained for historical context only. **W10.v3** ships with the cosine-built index + cosine-aware optimizer probe; republish under SQLR-25.
175+
The **W10.v1** numbers below were taken before SQLR-23 (parser-bound) and SQLR-28 (HNSW probe was L2-only, so the HNSW variant silently fell through to brute-force on cosine queries) — they are retained for historical context only. **W10.v3** ships with the cosine-built index + cosine-aware optimizer probe and was republished under SQLR-25.
176176

177177
| Variant | SQLRite median (v1, retired) | Throughput |
178178
|---|---|---|
@@ -183,7 +183,15 @@ The **W10.v1** numbers below were taken before SQLR-23 (parser-bound) and SQLR-2
183183
1. **Per-iter SQL parse cost** — the 384-element bracket-array literal in the `ORDER BY` clause was ~4 KB of SQL the parser walked every iteration. Fixed in SQLR-23 (`Value::Vector` bind).
184184
2. **Cosine queries silently brute-forced on the HNSW path** — the optimizer's `try_hnsw_probe` was L2-only; cosine queries never hit the graph. Fixed in SQLR-28 (per-index metric + matching probe).
185185

186-
W10.v3 measures the *actual* HNSW-vs-brute-force gap with both fixes in place.
186+
#### W10.v3 — post-SQLR-23 + SQLR-28 (canonical pinned-host run, [`results/2026-05-08-apple-ac84d560.json`](results/2026-05-08-apple-ac84d560.json))
187+
188+
| Variant | SQLRite median | Throughput |
189+
|---|---|---|
190+
| brute-force | 120.88 ms | ~8 ops/s |
191+
| **hnsw** | **2.40 ms** | **~417 ops/s** |
192+
| **HNSW vs brute-force** | **~50× faster** | |
193+
194+
**Read this as:** with both fixes in place, the HNSW shortcut delivers what the algorithm promises. Brute-force barely moved (compute-bound: 10k × 384-dim cosine distances per query; the parser tax was a small slice of total time). HNSW dropped ~53× — bounded log-probe vs O(N×D) — and the brute-vs-HNSW gap is now ~50× at 10k corpus, the right order-of-magnitude for the algorithm and the corpus size. Bigger corpora widen the gap further; the existing harness can be tuned via the W10 corpus knob without an envelope-version bump.
187195

188196
### W11 — BM25 top-10
189197

0 commit comments

Comments
 (0)