@@ -141,30 +141,46 @@ tests/
141141- Parser tests use inline code strings parsed directly with tree-sitter
142142- Always run the full suite (` npm test ` ) before submitting a PR
143143
144- ## Benchmarks
144+ ## Regression Benchmarks
145145
146- Two benchmark scripts live in ` scripts/ ` . Run them before and after your changes
147- to make sure you're not degrading performance.
146+ Two regression benchmark scripts live in ` scripts/ ` . These are ** not** unit
147+ tests — they measure performance metrics that reviewers use to judge whether a
148+ change is acceptable. If your PR touches code covered by a benchmark, you
149+ ** must** run it before and after your changes and include the results in the PR
150+ description.
148151
149- | Script | What it measures | When to run |
150- | --------| -----------------| -------------|
152+ | Benchmark | What it measures | When to run |
153+ | ----------- | -----------------| -------------|
151154| ` node scripts/benchmark.js ` | Build speed (native vs WASM), query latency | Changes to ` builder.js ` , ` parser.js ` , ` queries.js ` , ` resolve.js ` , ` db.js ` , or the native engine |
152- | ` node scripts/embedding-benchmark.js ` | Embedding search recall (Hit@1/3/5/10) across models | Changes to ` embedder.js ` or embedding strategies |
155+ | ` node scripts/embedding-benchmark.js ` | Search recall (Hit@1/3/5/10) across models | Changes to ` embedder.js ` or embedding strategies |
153156
154- Both scripts output JSON to stdout (progress goes to stderr), so you can save
155- results for comparison:
157+ ### How to report results
158+
159+ Both scripts output JSON to stdout (progress goes to stderr). Run the relevant
160+ benchmark on ` main ` (before), then on your branch (after), and paste both in
161+ your PR description:
156162
157163``` bash
158- # Before your changes
164+ git stash && git checkout main
159165node scripts/benchmark.js > before.json
160- node scripts/embedding-benchmark.js > before-embed.json
161166
162- # After your changes
167+ git checkout - && git stash pop
163168node scripts/benchmark.js > after.json
164- node scripts/embedding-benchmark.js > after-embed.json
165169```
166170
167- If recall or build speed regresses, investigate before opening a PR.
171+ In the PR, include a table like:
172+
173+ ```
174+ ## Benchmark results
175+
176+ | Metric | Before | After | Delta |
177+ |--------------|--------|--------|-------|
178+ | Build (ms) | 1200 | 1180 | -20 |
179+ | Hit@1 | 75.5% | 76.2% | +0.7% |
180+ ```
181+
182+ Regressions are not automatically blocking, but unexplained drops in speed or
183+ recall will be questioned during review.
168184
169185## Common Contribution Types
170186
0 commit comments