diff --git a/benchmarks/README.md b/benchmarks/README.md index 1892ff0..962b213 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -2,7 +2,7 @@ Benchmark suite for SQLRite vs SQLite (and friends). Tracks task **SQLR-4** / **SQLR-16**. -> **Status (2026-05-07):** all six sub-phases (9.1–9.6) shipped — harness + 12 workloads + DuckDB driver + canonical [`docs/benchmarks.md`](../docs/benchmarks.md) reference + first official pinned-host run committed under [`results/`](results/). See [`docs/benchmarks-plan.md`](../docs/benchmarks-plan.md) for the design rationale + the resolved Q1–Q8 decisions. +> **Status (2026-05-08):** all six sub-phases (9.1–9.6) shipped — harness + 12 workloads + DuckDB driver + canonical [`docs/benchmarks.md`](../docs/benchmarks.md) reference + pinned-host runs committed under [`results/`](results/). The v2 republish (SQLR-25) lands the post-SQLR-23 / SQLR-28 numbers; the v1 baseline is retained as historical. See [`docs/benchmarks-plan.md`](../docs/benchmarks-plan.md) for the design rationale + the resolved Q1–Q8 decisions. ## Quick start @@ -172,7 +172,7 @@ Even at 10k scale, the gap is large: 10k 384-dim vectors. Two variants per the plan: brute-force (no index) and HNSW (`CREATE INDEX … USING hnsw (embedding) WITH (metric = 'cosine')`, per SQLR-28). SQLRite-only — `sqlite-vec` extension wiring is a follow-up (`rusqlite[bundled]` doesn't ship it; loading a pre-compiled `.dylib` at runtime is non-trivial and was out of scope for v1). -The **W10.v1** numbers below were taken before SQLR-23 (parser-bound) and SQLR-28 (HNSW probe was L2-only, so the HNSW variant silently fell through to brute-force on cosine queries) — they are retained for historical context only. **W10.v3** ships with the cosine-built index + cosine-aware optimizer probe; republish under SQLR-25. +The **W10.v1** numbers below were taken before SQLR-23 (parser-bound) and SQLR-28 (HNSW probe was L2-only, so the HNSW variant silently fell through to brute-force on cosine queries) — they are retained for historical context only. **W10.v3** ships with the cosine-built index + cosine-aware optimizer probe and was republished under SQLR-25. | Variant | SQLRite median (v1, retired) | Throughput | |---|---|---| @@ -183,7 +183,15 @@ The **W10.v1** numbers below were taken before SQLR-23 (parser-bound) and SQLR-2 1. **Per-iter SQL parse cost** — the 384-element bracket-array literal in the `ORDER BY` clause was ~4 KB of SQL the parser walked every iteration. Fixed in SQLR-23 (`Value::Vector` bind). 2. **Cosine queries silently brute-forced on the HNSW path** — the optimizer's `try_hnsw_probe` was L2-only; cosine queries never hit the graph. Fixed in SQLR-28 (per-index metric + matching probe). -W10.v3 measures the *actual* HNSW-vs-brute-force gap with both fixes in place. +#### W10.v3 — post-SQLR-23 + SQLR-28 (canonical pinned-host run, [`results/2026-05-08-apple-ac84d560.json`](results/2026-05-08-apple-ac84d560.json)) + +| Variant | SQLRite median | Throughput | +|---|---|---| +| brute-force | 120.88 ms | ~8 ops/s | +| **hnsw** | **2.40 ms** | **~417 ops/s** | +| **HNSW vs brute-force** | **~50× faster** | | + +**Read this as:** with both fixes in place, the HNSW shortcut delivers what the algorithm promises. Brute-force barely moved (compute-bound: 10k × 384-dim cosine distances per query; the parser tax was a small slice of total time). HNSW dropped ~53× — bounded log-probe vs O(N×D) — and the brute-vs-HNSW gap is now ~50× at 10k corpus, the right order-of-magnitude for the algorithm and the corpus size. Bigger corpora widen the gap further; the existing harness can be tuned via the W10 corpus knob without an envelope-version bump. ### W11 — BM25 top-10 diff --git a/benchmarks/results/2026-05-08-apple-ac84d560.json b/benchmarks/results/2026-05-08-apple-ac84d560.json new file mode 100644 index 0000000..e7a5f39 --- /dev/null +++ b/benchmarks/results/2026-05-08-apple-ac84d560.json @@ -0,0 +1,405 @@ +{ + "schema_version": 1, + "run_started_at": "2026-05-08T15:41:43Z", + "run_duration_secs": 1789.0, + "host": { + "cpu": "Apple M1 Pro", + "cpus": 10, + "ram_mib": 32768, + "os_kind": "macos", + "os_release": "23.5.0", + "arch": "aarch64" + }, + "commit": { + "sha": "ac84d560bb21fbfef4070b1a0f8f6aac24644ca7", + "branch": "sqlr-25-bench-v2-republish", + "dirty": true + }, + "samples": [ + { + "workload": "W1.v2", + "driver": "sqlite", + "median_ns": 2091.65205166995, + "median_ci_lower_ns": 2023.8649766899769, + "median_ci_upper_ns": 2149.422558849305, + "mean_ns": 2268.981926961545, + "std_dev_ns": 616.5457398860442, + "samples": 100, + "ops_per_s": 478090.9899433856 + }, + { + "workload": "W1.v2", + "driver": "sqlrite", + "median_ns": 3923.587906076011, + "median_ci_lower_ns": 3880.506020423716, + "median_ci_upper_ns": 3948.6301907968577, + "mean_ns": 3987.278735531798, + "std_dev_ns": 366.3605867425403, + "samples": 100, + "ops_per_s": 254868.7639829388 + }, + { + "workload": "W10.v3/brute-force", + "driver": "sqlrite", + "median_ns": 120875002.05, + "median_ci_lower_ns": 119432916.0, + "median_ci_upper_ns": 125855060.22222222, + "mean_ns": 122598499.79623017, + "std_dev_ns": 4064582.227447615, + "samples": 10, + "ops_per_s": 8.273009166827974 + }, + { + "workload": "W10.v3/hnsw", + "driver": "sqlrite", + "median_ns": 2398411.4191743825, + "median_ci_lower_ns": 2348086.659722222, + "median_ci_upper_ns": 2491667.824074074, + "mean_ns": 2427491.8649724424, + "std_dev_ns": 102366.68421099371, + "samples": 10, + "ops_per_s": 416.9426446210947 + }, + { + "workload": "W11.v2", + "driver": "sqlite", + "median_ns": 23654.906768367524, + "median_ci_lower_ns": 23547.650315623356, + "median_ci_upper_ns": 24341.97455725057, + "mean_ns": 23913.829927356528, + "std_dev_ns": 571.199883546743, + "samples": 10, + "ops_per_s": 42274.52721721347 + }, + { + "workload": "W11.v2", + "driver": "sqlrite", + "median_ns": 501626.1366459627, + "median_ci_lower_ns": 492887.9399585921, + "median_ci_upper_ns": 512927.4139061422, + "mean_ns": 504163.97041062807, + "std_dev_ns": 11955.032558858413, + "samples": 10, + "ops_per_s": 1993.5165394018916 + }, + { + "workload": "W12.v2", + "driver": "sqlrite", + "median_ns": 607903.9145171958, + "median_ci_lower_ns": 596500.7953317901, + "median_ci_upper_ns": 652024.4496527778, + "mean_ns": 623609.7411017416, + "std_dev_ns": 42204.63168403889, + "samples": 10, + "ops_per_s": 1644.9968097247925 + }, + { + "workload": "W2.v2/range-100", + "driver": "sqlite", + "median_ns": 66616.52817729535, + "median_ci_lower_ns": 65785.63596491228, + "median_ci_upper_ns": 67869.87708333334, + "mean_ns": 72454.0231253191, + "std_dev_ns": 23822.42156687678, + "samples": 100, + "ops_per_s": 15011.289650799095 + }, + { + "workload": "W2.v2/range-100", + "driver": "sqlrite", + "median_ns": 24267208.25, + "median_ci_lower_ns": 24089291.75, + "median_ci_upper_ns": 24668000.0, + "mean_ns": 26632353.79, + "std_dev_ns": 7816605.337903695, + "samples": 100, + "ops_per_s": 41.20787153174078 + }, + { + "workload": "W2.v2/range-10k", + "driver": "sqlite", + "median_ns": 7008678.571428571, + "median_ci_lower_ns": 6810946.428571428, + "median_ci_upper_ns": 7349125.0, + "mean_ns": 7982842.618571429, + "std_dev_ns": 4533022.6613204405, + "samples": 100, + "ops_per_s": 142.68024846746127 + }, + { + "workload": "W2.v2/range-10k", + "driver": "sqlrite", + "median_ns": 30725625.0, + "median_ci_lower_ns": 30444260.5, + "median_ci_upper_ns": 31246729.5, + "mean_ns": 33352340.84, + "std_dev_ns": 8837394.73525212, + "samples": 100, + "ops_per_s": 32.546123960049634 + }, + { + "workload": "W2.v2/range-1k", + "driver": "sqlite", + "median_ns": 649299.8587656973, + "median_ci_lower_ns": 632725.6785714286, + "median_ci_upper_ns": 666337.9666666667, + "mean_ns": 724870.9891784526, + "std_dev_ns": 342182.18138643814, + "samples": 100, + "ops_per_s": 1540.1204643736944 + }, + { + "workload": "W2.v2/range-1k", + "driver": "sqlrite", + "median_ns": 26641854.0, + "median_ci_lower_ns": 25977895.75, + "median_ci_upper_ns": 28121125.0, + "mean_ns": 32502029.355, + "std_dev_ns": 16703632.194522006, + "samples": 100, + "ops_per_s": 37.534925309627475 + }, + { + "workload": "W3.v2", + "driver": "sqlite", + "median_ns": 183962555.66666663, + "median_ci_lower_ns": 176328152.83333334, + "median_ci_upper_ns": 196417444.33333337, + "mean_ns": 196211279.2, + "std_dev_ns": 40356044.64014864, + "samples": 10, + "ops_per_s": 5.435888821918539 + }, + { + "workload": "W3.v2", + "driver": "sqlrite", + "median_ns": 606197645.5, + "median_ci_lower_ns": 584494562.5, + "median_ci_upper_ns": 704306666.5, + "mean_ns": 653136737.5, + "std_dev_ns": 113923173.00654095, + "samples": 10, + "ops_per_s": 1.6496269944684236 + }, + { + "workload": "W4.v2", + "driver": "sqlite", + "median_ns": 11345.171333649738, + "median_ci_lower_ns": 10823.96616869208, + "median_ci_upper_ns": 12253.197674418603, + "mean_ns": 12955.781165492392, + "std_dev_ns": 4281.784390672776, + "samples": 100, + "ops_per_s": 88143.2259232616 + }, + { + "workload": "W4.v2", + "driver": "sqlrite", + "median_ns": 6566570.3125, + "median_ci_lower_ns": 6381117.1875, + "median_ci_upper_ns": 6989635.375, + "mean_ns": 7118857.9175, + "std_dev_ns": 1249959.472460705, + "samples": 100, + "ops_per_s": 152.2864984931965 + }, + { + "workload": "W5.v2", + "driver": "sqlite", + "median_ns": 9653.03723172696, + "median_ci_lower_ns": 9316.82389937107, + "median_ci_upper_ns": 9931.765474081025, + "mean_ns": 9959.92855058162, + "std_dev_ns": 2523.674162182306, + "samples": 100, + "ops_per_s": 103594.3378228426 + }, + { + "workload": "W5.v2", + "driver": "sqlrite", + "median_ns": 57996395.5, + "median_ci_lower_ns": 54083.0, + "median_ci_upper_ns": 122323125.0, + "mean_ns": 78927419.17, + "std_dev_ns": 86714859.4560829, + "samples": 100, + "ops_per_s": 17.242450869209623 + }, + { + "workload": "W6.v2", + "driver": "sqlite", + "median_ns": 2557.089588304828, + "median_ci_lower_ns": 2487.471022520414, + "median_ci_upper_ns": 2671.0153098150695, + "mean_ns": 3130.0094644723804, + "std_dev_ns": 1759.0674821253504, + "samples": 100, + "ops_per_s": 391069.5990369779 + }, + { + "workload": "W6.v2", + "driver": "sqlrite", + "median_ns": 4036.203656965382, + "median_ci_lower_ns": 4010.437058204241, + "median_ci_upper_ns": 4074.5239910184255, + "mean_ns": 4139.053163401543, + "std_dev_ns": 513.0900021217542, + "samples": 100, + "ops_per_s": 247757.56750387803 + }, + { + "workload": "W7.v2", + "driver": "duckdb", + "median_ns": 478782.5882523148, + "median_ci_lower_ns": 449912.841796875, + "median_ci_upper_ns": 498924.1322916667, + "mean_ns": 476860.1179152199, + "std_dev_ns": 28720.836846339804, + "samples": 10, + "ops_per_s": 2088.6306739981273 + }, + { + "workload": "W7.v2", + "driver": "sqlite", + "median_ns": 31565314.89814815, + "median_ci_lower_ns": 31119865.277777776, + "median_ci_upper_ns": 32048895.904761903, + "mean_ns": 31687712.042526457, + "std_dev_ns": 728682.2777543392, + "samples": 10, + "ops_per_s": 31.680342908876455 + }, + { + "workload": "W7.v2", + "driver": "sqlrite", + "median_ns": 103615758.35, + "median_ci_lower_ns": 102500069.33333331, + "median_ci_upper_ns": 106036615.66666669, + "mean_ns": 104174221.40059523, + "std_dev_ns": 2038138.1371251731, + "samples": 10, + "ops_per_s": 9.651041655480004 + }, + { + "workload": "W8.v2/card-10", + "driver": "duckdb", + "median_ns": 949747.5743670886, + "median_ci_lower_ns": 868905.1209563995, + "median_ci_upper_ns": 1016511.3080168776, + "mean_ns": 985983.3841947962, + "std_dev_ns": 190947.5551140088, + "samples": 10, + "ops_per_s": 1052.911349277622 + }, + { + "workload": "W8.v2/card-10", + "driver": "sqlite", + "median_ns": 366520646.0, + "median_ci_lower_ns": 358092323.0, + "median_ci_upper_ns": 417579302.0, + "mean_ns": 390339495.85, + "std_dev_ns": 52685148.480203815, + "samples": 10, + "ops_per_s": 2.7283592641054115 + }, + { + "workload": "W8.v2/card-10", + "driver": "sqlrite", + "median_ns": 197315986.1666667, + "median_ci_lower_ns": 194357388.66666663, + "median_ci_upper_ns": 200913701.3333333, + "mean_ns": 199149333.3333333, + "std_dev_ns": 7646561.872182194, + "samples": 10, + "ops_per_s": 5.068013086153755 + }, + { + "workload": "W8.v2/card-100k", + "driver": "duckdb", + "median_ns": 22933157.638888888, + "median_ci_lower_ns": 21351330.45625, + "median_ci_upper_ns": 24902944.964285713, + "mean_ns": 23765551.31545635, + "std_dev_ns": 3616439.877410352, + "samples": 10, + "ops_per_s": 43.604985224723286 + }, + { + "workload": "W8.v2/card-100k", + "driver": "sqlite", + "median_ns": 239719076.5, + "median_ci_lower_ns": 235210569.33333337, + "median_ci_upper_ns": 247224875.0, + "mean_ns": 243697679.20000005, + "std_dev_ns": 13289513.311869564, + "samples": 10, + "ops_per_s": 4.171549526222207 + }, + { + "workload": "W8.v2/card-1k", + "driver": "duckdb", + "median_ns": 1039376.0295698924, + "median_ci_lower_ns": 916913.9784946236, + "median_ci_upper_ns": 1189012.9293394778, + "mean_ns": 1066015.2289703875, + "std_dev_ns": 175616.45954191365, + "samples": 10, + "ops_per_s": 962.1157036051844 + }, + { + "workload": "W8.v2/card-1k", + "driver": "sqlite", + "median_ns": 240643416.75, + "median_ci_lower_ns": 236370562.5, + "median_ci_upper_ns": 275085770.75, + "mean_ns": 255594200.0, + "std_dev_ns": 26866890.198058955, + "samples": 10, + "ops_per_s": 4.155526103749106 + }, + { + "workload": "W8.v2/card-1k", + "driver": "sqlrite", + "median_ns": 1379751479.5, + "median_ci_lower_ns": 1360678041.0, + "median_ci_upper_ns": 1397903000.0, + "mean_ns": 1383032920.7, + "std_dev_ns": 28090665.379418075, + "samples": 10, + "ops_per_s": 0.7247682027218294 + }, + { + "workload": "W9.v2", + "driver": "duckdb", + "median_ns": 484971.75645682006, + "median_ci_lower_ns": 457281.68418079097, + "median_ci_upper_ns": 537153.288606403, + "mean_ns": 501575.2209037306, + "std_dev_ns": 70232.06875399526, + "samples": 10, + "ops_per_s": 2061.9757474248627 + }, + { + "workload": "W9.v2", + "driver": "sqlite", + "median_ns": 2157.433404757092, + "median_ci_lower_ns": 2113.9801819284417, + "median_ci_upper_ns": 2289.301211123625, + "mean_ns": 2194.158215899006, + "std_dev_ns": 107.69245606104188, + "samples": 10, + "ops_per_s": 463513.7278374492 + }, + { + "workload": "W9.v2", + "driver": "sqlrite", + "median_ns": 30296536875.5, + "median_ci_lower_ns": 29889488645.5, + "median_ci_upper_ns": 31381315042.0, + "mean_ns": 30614205587.6, + "std_dev_ns": 863788548.43593, + "samples": 10, + "ops_per_s": 0.03300707285817454 + } + ] +} \ No newline at end of file diff --git a/docs/benchmarks.md b/docs/benchmarks.md index aed9c68..a5a9963 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -90,7 +90,7 @@ A few methodology notes that change how you read the table. - `Statement::query_with_params(&[Value])` / `Statement::execute_with_params(&[Value])` — bind `?` placeholders at execute time without re-running sqlparser. - `Value::Vector(Vec)` as a first-class bind type — the 4 KB query vector for W10 is now bound directly instead of being re-lexed every iteration. The HNSW probe optimizer still recognizes the bound shape, so the algorithmic shortcut keeps firing. -The bench harness `Driver::query_one` / `query_all` paths route through `prepare_cached` + the bound API. Every workload's `WorkloadId.version` was bumped `v1 → v2` in lockstep — old JSON envelopes keep the v1 tag and stay readable, but cross-version comparisons require an explicit acknowledgment in the comparison script. The next official pinned-host run will land the post-binding numbers; treat the v1 row above as "before" and watch this section for the "after" once republished. +The bench harness `Driver::query_one` / `query_all` paths route through `prepare_cached` + the bound API. Every workload's `WorkloadId.version` was bumped `v1 → v2` in lockstep — old JSON envelopes keep the v1 tag and stay readable, but cross-version comparisons require an explicit acknowledgment in the comparison script. The headline table below carries the v2 numbers from the post-SQLR-23 republished run (SQLR-25); the retired v1 baseline lives in the historical section underneath. **Where DuckDB is misleading.** Per-PK-probe single-row OLTP queries (W9) are SQLite's home turf, not DuckDB's. The plan flags this as "apples-to-oranges"; we still publish the number because the directional comparison is informative. @@ -100,29 +100,53 @@ The bench harness `Driver::query_one` / `query_all` paths route through `prepare ## Headline numbers -Median latency from the first official pinned-host run — [`benchmarks/results/2026-05-07-apple-9ffd55a5.json`](../benchmarks/results/2026-05-07-apple-9ffd55a5.json), Apple M1 Pro / macOS 23.5.0, criterion defaults (3 s warm-up, 5 s measurement, 100 samples on light workloads / 10 samples on heavy ones — see the JSON envelope's per-sample `samples` field). Only medians here; the JSON carries 95 % CIs, mean, std-dev, ops/s. +Median latency from the post-SQLR-23 pinned-host run — [`benchmarks/results/2026-05-08-apple-ac84d560.json`](../benchmarks/results/2026-05-08-apple-ac84d560.json), Apple M1 Pro / macOS 23.5.0, criterion defaults (3 s warm-up, 5 s measurement, 100 samples on light workloads / 10 samples on heavy ones — see the JSON envelope's per-sample `samples` field). Only medians here; the JSON carries 95 % CIs, mean, std-dev, ops/s. | Workload | SQLRite | SQLite (WAL+NORMAL) | DuckDB | Notes | |---|---|---|---|---| -| **W1** read-by-PK | 9.87 µs | 2.05 µs | — | ~5× — parser tax | -| **W2** range-100 | 23.99 ms | 60.50 µs | — | ~400× — full-scan vs index range probe | -| **W2** range-1k | 24.92 ms | 585.21 µs | — | ~43× | -| **W2** range-10k | 30.15 ms | 6.24 ms | — | ~5× — converges as scan dominates | -| **W3** bulk insert (100k/txn) | 1.029 s | 166.43 ms | — | ~6.2× | -| **W4** single-row insert | 6.76 ms | 9.78 µs | — | **~691× ⚠️** SQLR-18 | -| **W5** mixed OLTP | 55.63 ms | 9.96 µs | — | **~5,580× ⚠️** SQLR-18 | -| **W6** index lookup | 10.45 µs | 2.50 µs | — | ~4× — parser tax | -| **W7** SUM (1M rows) | 109.47 ms | 31.14 ms | 468.74 µs | DuckDB ~66× faster than SQLite | -| **W8** GROUP BY card-10 | 201.80 ms | 438.09 ms | 761.40 µs | DuckDB ~575× faster than SQLite | -| **W8** GROUP BY card-1k | 1.372 s | 251.13 ms | 871.80 µs | DuckDB ~288× faster than SQLite | -| **W8** GROUP BY card-100k | _skipped_ | 238.96 ms | 19.58 ms | **SQLRite skipped ⚠️** SQLR-19; DuckDB ~12× faster than SQLite | -| **W9** INNER JOIN (10k×10k) | 34.25 s | 2.23 µs | 699.23 µs | **~15M× ⚠️** SQLR-20; DuckDB ~313× slower than SQLite (analytical-engine OLTP weakness) | -| **W10** vector top-10 (brute-force, 10k×384) | 138.66 ms | — | — | parser cost dominates | -| **W10** vector top-10 (HNSW) | 126.81 ms | — | — | masked by parser cost | -| **W11** BM25 top-10 (1k docs) | 1.079 ms | 25.03 µs | — | ~43× | -| **W12** hybrid (1k docs) | 713.53 µs | — | — | RAG headline | - -> The **canonical run** is [`benchmarks/results/2026-05-07-apple-9ffd55a5.json`](../benchmarks/results/2026-05-07-apple-9ffd55a5.json). The `dirty=true` flag in the commit metadata reflects the working-tree state when 9.6 PR was being authored (this doc + README updates uncommitted at run time); the **measurements themselves only depend on the bench binary**, which was built from the committed bench-9.5-duckdb tip. Subsequent official runs land alongside this file with their own date / host / commit. +| **W1** read-by-PK | 3.92 µs | 2.09 µs | — | ~1.9× — gap closed by SQLR-23 (was ~4.8× in v1) | +| **W2** range-100 | 24.27 ms | 66.62 µs | — | ~364× — full-scan vs index range probe | +| **W2** range-1k | 26.64 ms | 649.30 µs | — | ~41× | +| **W2** range-10k | 30.73 ms | 7.01 ms | — | ~4.4× — converges as scan dominates | +| **W3** bulk insert (100k/txn) | 606.20 ms | 183.96 ms | — | ~3.3× — 100k INSERT plan parsed once, not per-row (was ~6.2× in v1) | +| **W4** single-row insert | 6.57 ms | 11.35 µs | — | **~579× ⚠️** SQLR-18 | +| **W5** mixed OLTP | 58.00 ms | 9.65 µs | — | **~6,010× ⚠️** SQLR-18 | +| **W6** index lookup | 4.04 µs | 2.56 µs | — | ~1.6× — gap closed by SQLR-23 (was ~4.2× in v1) | +| **W7** SUM (1M rows) | 103.62 ms | 31.57 ms | 478.78 µs | DuckDB ~66× faster than SQLite | +| **W8** GROUP BY card-10 | 197.32 ms | 366.52 ms | 949.75 µs | DuckDB ~386× faster than SQLite | +| **W8** GROUP BY card-1k | 1.380 s | 240.64 ms | 1.039 ms | DuckDB ~232× faster than SQLite | +| **W8** GROUP BY card-100k | _skipped_ | 239.72 ms | 22.93 ms | **SQLRite skipped ⚠️** SQLR-19; DuckDB ~10× faster than SQLite | +| **W9** INNER JOIN (10k×10k) | 30.30 s | 2.16 µs | 484.97 µs | **~14M× ⚠️** SQLR-20; DuckDB ~225× slower than SQLite (analytical-engine OLTP weakness) | +| **W10** vector top-10 (brute-force, 10k×384) | 120.88 ms | — | — | compute-bound; modest ~13% drop vs v1 | +| **W10** vector top-10 (HNSW) | **2.40 ms** | — | — | **~53× faster than v1** ⭐ — SQLR-23 + SQLR-28 unmasked the index; HNSW now ~50× faster than brute-force | +| **W11** BM25 top-10 (1k docs) | 501.63 µs | 23.65 µs | — | ~21× — `fts_match` / `bm25_score` no longer re-parsed (was ~43× in v1) | +| **W12** hybrid (1k docs) | 607.90 µs | — | — | RAG headline (~15% faster than v1) | + +> The **canonical v2 run** is [`benchmarks/results/2026-05-08-apple-ac84d560.json`](../benchmarks/results/2026-05-08-apple-ac84d560.json). It supersedes the v1 baseline (table below) end-to-end: every workload was rerun on the same canonical Apple M1 Pro host after [SQLR-23](https://github.com/joaoh82/rust_sqlite/pulls?q=SQLR-23) bumped `WorkloadId.version` from `v1 → v2` in lockstep (W10 → `v3` after [SQLR-28](https://github.com/joaoh82/rust_sqlite/pulls?q=SQLR-28) widened the HNSW probe to cosine + dot). The `dirty=true` flag reflects the working-tree state at run time (this doc update + the new envelope itself uncommitted); the **measurements themselves only depend on the bench binary**, which was built from the clean `ac84d560` tip. Subsequent official runs land alongside this file with their own date / host / commit. + +### Historical (v1, retired) + +The pre-SQLR-23 baseline from [`benchmarks/results/2026-05-07-apple-9ffd55a5.json`](../benchmarks/results/2026-05-07-apple-9ffd55a5.json), retained so the methodology shift is visible. The v1→v2 jump is not an algorithmic improvement — it's the bench-driver methodology change (per-iter `inline_params` → `prepare_cached` + bound `?` parameters; `Value::Vector` for HNSW-eligible KNN). Cross-version comparisons (`W1.v1` vs `W1.v2`) are flagged in the comparison script per Q8; the [`compare.py`](../benchmarks/scripts/compare.py) v1↔v2 report walks each one. + +| Workload | SQLRite (v1) | SQLite (v1) | DuckDB (v1) | +|---|---|---|---| +| **W1** read-by-PK | 9.87 µs | 2.05 µs | — | +| **W2** range-100 | 23.99 ms | 60.50 µs | — | +| **W2** range-1k | 24.92 ms | 585.21 µs | — | +| **W2** range-10k | 30.15 ms | 6.24 ms | — | +| **W3** bulk insert (100k/txn) | 1.029 s | 166.43 ms | — | +| **W4** single-row insert | 6.76 ms | 9.78 µs | — | +| **W5** mixed OLTP | 55.63 ms | 9.96 µs | — | +| **W6** index lookup | 10.45 µs | 2.50 µs | — | +| **W7** SUM (1M rows) | 109.47 ms | 31.14 ms | 468.74 µs | +| **W8** GROUP BY card-10 | 201.80 ms | 438.09 ms | 761.40 µs | +| **W8** GROUP BY card-1k | 1.372 s | 251.13 ms | 871.80 µs | +| **W8** GROUP BY card-100k | _skipped_ | 238.96 ms | 19.58 ms | +| **W9** INNER JOIN (10k×10k) | 34.25 s | 2.23 µs | 699.23 µs | +| **W10** brute-force | 138.66 ms | — | — | +| **W10** HNSW | 126.81 ms | — | — | +| **W11** BM25 top-10 (1k docs) | 1.079 ms | 25.03 µs | — | +| **W12** hybrid (1k docs) | 713.53 µs | — | — | ---