Skip to content

Commit fcc19d6

Browse files
docs: reframe benchmarks as regression benchmarks with PR reporting
1 parent 8395059 commit fcc19d6

1 file changed

Lines changed: 29 additions & 13 deletions

File tree

CONTRIBUTING.md

Lines changed: 29 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -141,30 +141,46 @@ tests/
141141
- Parser tests use inline code strings parsed directly with tree-sitter
142142
- Always run the full suite (`npm test`) before submitting a PR
143143

144-
## Benchmarks
144+
## Regression Benchmarks
145145

146-
Two benchmark scripts live in `scripts/`. Run them before and after your changes
147-
to make sure you're not degrading performance.
146+
Two regression benchmark scripts live in `scripts/`. These are **not** unit
147+
tests — they measure performance metrics that reviewers use to judge whether a
148+
change is acceptable. If your PR touches code covered by a benchmark, you
149+
**must** run it before and after your changes and include the results in the PR
150+
description.
148151

149-
| Script | What it measures | When to run |
150-
|--------|-----------------|-------------|
152+
| Benchmark | What it measures | When to run |
153+
|-----------|-----------------|-------------|
151154
| `node scripts/benchmark.js` | Build speed (native vs WASM), query latency | Changes to `builder.js`, `parser.js`, `queries.js`, `resolve.js`, `db.js`, or the native engine |
152-
| `node scripts/embedding-benchmark.js` | Embedding search recall (Hit@1/3/5/10) across models | Changes to `embedder.js` or embedding strategies |
155+
| `node scripts/embedding-benchmark.js` | Search recall (Hit@1/3/5/10) across models | Changes to `embedder.js` or embedding strategies |
153156

154-
Both scripts output JSON to stdout (progress goes to stderr), so you can save
155-
results for comparison:
157+
### How to report results
158+
159+
Both scripts output JSON to stdout (progress goes to stderr). Run the relevant
160+
benchmark on `main` (before), then on your branch (after), and paste both in
161+
your PR description:
156162

157163
```bash
158-
# Before your changes
164+
git stash && git checkout main
159165
node scripts/benchmark.js > before.json
160-
node scripts/embedding-benchmark.js > before-embed.json
161166

162-
# After your changes
167+
git checkout - && git stash pop
163168
node scripts/benchmark.js > after.json
164-
node scripts/embedding-benchmark.js > after-embed.json
165169
```
166170

167-
If recall or build speed regresses, investigate before opening a PR.
171+
In the PR, include a table like:
172+
173+
```
174+
## Benchmark results
175+
176+
| Metric | Before | After | Delta |
177+
|--------------|--------|--------|-------|
178+
| Build (ms) | 1200 | 1180 | -20 |
179+
| Hit@1 | 75.5% | 76.2% | +0.7% |
180+
```
181+
182+
Regressions are not automatically blocking, but unexplained drops in speed or
183+
recall will be questioned during review.
168184

169185
## Common Contribution Types
170186

0 commit comments

Comments
 (0)