Skip to content

Commit 17804b9

Browse files
xiaoxiaojxclaude
andcommitted
chore(bench): add tinybench-based performance benchmark suite
Introduces benchmark infrastructure for enhanced-resolve, wired to CodSpeed via @codspeed/tinybench-plugin so the same `npm run benchmark` command works both locally (walltime fallback) and in CI (instrumentation). Adds two initial cases: `realistic-midsize`, a synthetic mid-size project exercising relative/bare/scoped/exports lookups plus nested node_modules, and `pathological-deep-stack`, a 50-level alias chain that specifically stresses the doResolve recursion-check path that PR #443 is targeting. Also excludes benchmark/ from eslint (synthetic fixtures + ESM entry point conflict with the base config) and adds `codspeed`/`tinybench` to the spellcheck dictionary. Refs: #443 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 8148159 commit 17804b9

32 files changed

Lines changed: 706 additions & 3 deletions

File tree

.cspell.json

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,8 @@
33
"words": [
44
"abcxyz",
55
"addrs",
6+
"codspeed",
7+
"tinybench",
68
"abortable",
79
"anotherhashishere",
810
"arcanis",

benchmark/README.md

Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,107 @@
1+
# Benchmarks
2+
3+
Performance benchmarks for `enhanced-resolve`, tracked over time via
4+
[CodSpeed](https://codspeed.io/).
5+
6+
Runner stack: [tinybench](https://github.com/tinylibs/tinybench) +
7+
[`@codspeed/tinybench-plugin`](https://www.npmjs.com/package/@codspeed/tinybench-plugin).
8+
The plugin is a drop-in wrapper: locally it falls back to plain tinybench
9+
wall-clock measurements, and under `CodSpeedHQ/action` in CI it automatically
10+
switches to CodSpeed's instrumentation mode.
11+
12+
## Running locally
13+
14+
```sh
15+
npm run benchmark
16+
```
17+
18+
Optional substring filter to run only matching cases:
19+
20+
```sh
21+
npm run benchmark -- realistic
22+
BENCH_FILTER=pathological npm run benchmark
23+
```
24+
25+
Locally the runner uses tinybench's wall-clock measurements and prints a
26+
table of ops/s, mean, p99, and relative margin of error per task. Under CI,
27+
the plugin detects the CodSpeed runner environment and switches to
28+
instruction-counting mode automatically.
29+
30+
The V8 flags in `package.json` (`--no-opt --predictable --hash-seed=1` etc.)
31+
are required by CodSpeed's instrumentation mode for deterministic results —
32+
do not drop them.
33+
34+
### Optional: running real instruction counts locally
35+
36+
If you want to reproduce CI's exact instrument-count numbers on your own
37+
machine (Linux only — the underlying Valgrind tooling has no macOS backend),
38+
install the standalone CodSpeed CLI and wrap `npm run benchmark` with it:
39+
40+
```sh
41+
curl -fsSL https://codspeed.io/install.sh | bash
42+
codspeed run npm run benchmark
43+
```
44+
45+
This is only useful if you want to debug an instruction-count regression
46+
outside CI. Day-to-day benchmark iteration should use `npm run benchmark`
47+
directly (wall-clock mode).
48+
49+
## Layout
50+
51+
```
52+
benchmark/
53+
├── run.mjs # entry point: discovers cases, runs bench
54+
└── cases/
55+
└── <case-name>/
56+
├── index.bench.mjs # default export: register(bench, ctx)
57+
└── fixture/ # optional: project tree to resolve against
58+
```
59+
60+
Each case directory must contain `index.bench.mjs` exporting a default
61+
function with the signature:
62+
63+
```js
64+
export default function register(bench, { caseName, caseDir, fixtureDir }) {
65+
bench.add("my case: descriptive name", async () => {
66+
// ... resolve calls ...
67+
});
68+
}
69+
```
70+
71+
`fixtureDir` is the absolute path to the case's `fixture/` subdirectory
72+
(which may or may not exist). `caseDir` is the parent directory containing
73+
`index.bench.mjs`.
74+
75+
## Existing cases
76+
77+
| Case | What it measures |
78+
| ------------------------- | ------------------------------------------------------------------------------------------------------------ |
79+
| `realistic-midsize` | Mixed batch of relative/bare/scoped/exports/nested-`node_modules` requests against a synthetic mid-size tree |
80+
| `pathological-deep-stack` | 50-deep alias chain, specifically stresses the `doResolve` recursion-check path |
81+
82+
Add new cases by creating a new directory under `cases/``run.mjs` will
83+
pick it up automatically on the next run.
84+
85+
## Adding a CodSpeed-friendly case
86+
87+
A few rules of thumb:
88+
89+
1. **Keep each bench body long enough to be measurable** (batch many resolves
90+
per iteration). CodSpeed's simulation mode runs the body exactly once under
91+
instrumentation, so a body that does one `resolver.resolve()` will be
92+
dominated by instrumentation overhead.
93+
2. **Avoid randomness.** Fixed request lists, fixed seeds. CodSpeed compares
94+
against the base commit and expects identical work across runs.
95+
3. **Pre-build anything expensive (resolvers, alias lists, fixture paths) outside
96+
the `bench.add` callback.** The goal is to measure resolve, not setup.
97+
4. **Prefer one focused case per bench file**, so the PR report is easy to
98+
read.
99+
100+
## CI integration
101+
102+
CI is driven by `.github/workflows/benchmarks.yml`, which uses
103+
`CodSpeedHQ/action@v4` in `mode: "simulation"` and authenticates via the
104+
`CODSPEED_TOKEN` repo secret.
105+
106+
Both `CODSPEED_TOKEN` and CodSpeed repo enablement must be configured by a
107+
repo admin once — see the top-level PR description for the handoff.
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
module.exports = {};
Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
{
2+
"name": "pathological-deep-stack-fixture",
3+
"version": "1.0.0"
4+
}
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
module.exports = "target";
Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
/*
2+
* pathological-deep-stack
3+
*
4+
* Stress the doResolve recursion-check path specifically. Configures a long
5+
* chain of alias rewrites so every resolution of `chain-0` produces a stack
6+
* of depth O(N) before landing on `./target.js`.
7+
*
8+
* This is the case that the linked-list rewrite in PR #443 is most likely
9+
* to regress, because hasStackEntry walks the parent chain. Keeping the
10+
* number here high enough to matter (50+ levels) but low enough to finish
11+
* quickly is the point.
12+
*
13+
* If you're benchmarking the #443 PR, compare this case before and after:
14+
* - "before" (current main): Set<string> clone per level, O(n) cost per
15+
* doResolve, O(n^2) per full chain, but cheap per-step comparison
16+
* - "after" (#443): single linked-list node per level, O(1) alloc, but
17+
* hasStackEntry walks parents -> O(n) per comparison, O(n^2) per chain
18+
* Both are quadratic on paper; the question is the constant factor and the
19+
* amount of GC pressure. That's exactly what this case measures.
20+
*/
21+
22+
import fs from "fs";
23+
import path from "path";
24+
import enhanced from "../../../lib/index.js";
25+
26+
const { ResolverFactory, CachedInputFileSystem } = enhanced;
27+
28+
const CHAIN_LENGTH = 50;
29+
30+
/**
31+
* Build an AliasPlugin-compatible alias list that forms a chain:
32+
* chain-0 -> chain-1 -> chain-2 -> ... -> chain-49 -> ./target
33+
*
34+
* Each entry rewrites one specifier to the next, so a single resolve of
35+
* `chain-0` forces enhanced-resolve to re-enter the pipeline CHAIN_LENGTH
36+
* times before bottoming out.
37+
*/
38+
function buildChainAliases() {
39+
const aliases = [];
40+
for (let i = 0; i < CHAIN_LENGTH - 1; i++) {
41+
aliases.push({ name: `chain-${i}`, alias: `chain-${i + 1}` });
42+
}
43+
aliases.push({ name: `chain-${CHAIN_LENGTH - 1}`, alias: "./target" });
44+
return aliases;
45+
}
46+
47+
/**
48+
* @param {import('tinybench').Bench} bench
49+
* @param {{ fixtureDir: string }} ctx
50+
*/
51+
export default function register(bench, { fixtureDir }) {
52+
const fileSystem = new CachedInputFileSystem(fs, 4000);
53+
const aliases = buildChainAliases();
54+
55+
const resolver = ResolverFactory.createResolver({
56+
fileSystem,
57+
extensions: [".js"],
58+
alias: aliases,
59+
});
60+
61+
const resolve = () =>
62+
new Promise((resolve, reject) => {
63+
resolver.resolve({}, fixtureDir, "chain-0", {}, (err, result) => {
64+
if (err) return reject(err);
65+
if (!result) return reject(new Error("no result"));
66+
resolve(result);
67+
});
68+
});
69+
70+
bench.add(
71+
`pathological-deep-stack: alias chain of ${CHAIN_LENGTH} (warm)`,
72+
async () => {
73+
// 10 resolves per iteration so the bench body is long enough to be
74+
// measurable but short enough to finish quickly.
75+
for (let i = 0; i < 10; i++) {
76+
await resolve();
77+
}
78+
},
79+
);
80+
}

benchmark/cases/realistic-midsize/fixture/node_modules/@scope/pkg/dist/index.js

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

benchmark/cases/realistic-midsize/fixture/node_modules/@scope/pkg/dist/sub.js

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

benchmark/cases/realistic-midsize/fixture/node_modules/@scope/pkg/package.json

Lines changed: 8 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

benchmark/cases/realistic-midsize/fixture/node_modules/debug/package.json

Lines changed: 5 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)