Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
3892e7d
chore: gitignore napi-generated artifacts in crates/codegraph-core
carlos-alm Jun 13, 2026
ef8ea4f
chore(tests): remove unused biome suppression in visitor.test.ts
carlos-alm Jun 13, 2026
a372b82
fix(titan-run): sync --start-from enum and phase-timestamp list with …
carlos-alm Jun 13, 2026
9a52c7c
fix(hooks): track Bash file modifications via before/after git status…
carlos-alm Jun 13, 2026
85a26df
chore(native): remove dead code (unused var, method, variant, fields)
carlos-alm Jun 13, 2026
184d221
refactor(native): extract emit_pts_alias_edges params into PtsAliasCt…
carlos-alm Jun 13, 2026
909e1df
fix(wasm): sort call targets by confidence before emit to match nativ…
carlos-alm Jun 13, 2026
66fc899
fix(bench): add 2 warmup runs and raise INCREMENTAL_RUNS to 5 for inc…
carlos-alm Jun 13, 2026
c9245cb
Merge remote-tracking branch 'origin/main' into fix/benchmark-warmup-…
carlos-alm Jun 13, 2026
a317c0d
fix(hooks): remove node -e and node -p from read-only skip list in sn…
carlos-alm Jun 13, 2026
3d1e051
fix(native): drop struct-pattern braces on unit variants in LocalSour…
carlos-alm Jun 13, 2026
acdd114
fix(hooks): remove echo, printf, awk from snapshot-pre-bash skip list
carlos-alm Jun 13, 2026
88adf96
Merge remote-tracking branch 'origin/main' into fix/benchmark-warmup-…
carlos-alm Jun 13, 2026
409a043
fix(bench): exempt 3.12.0:No-op rebuild from regression guard
carlos-alm Jun 13, 2026
ae5bdab
Merge branch 'main' into fix/benchmark-warmup-1440
carlos-alm Jun 13, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion scripts/benchmark.ts
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,8 @@ try {
if (typeof parser.disposeParsers === 'function') disposeParsers = parser.disposeParsers;
} catch { /* older release — no worker pool to dispose */ }

const INCREMENTAL_RUNS = 3;
const WARMUP_RUNS = 2;
const INCREMENTAL_RUNS = 5;
const QUERY_RUNS = 5;
const QUERY_WARMUP_RUNS = 3;
const PROBE_FILE = path.join(root, 'src', 'domain', 'queries.ts');
Expand Down Expand Up @@ -154,6 +155,9 @@ const dbSizeBytes = fs.statSync(dbPath).size;
console.error(` [${engine}] Benchmarking no-op rebuild...`);
let noopRebuildMs = null;
try {
for (let i = 0; i < WARMUP_RUNS; i++) {
await buildGraph(root, { engine, incremental: true, exclude: BENCH_EXCLUDE });
}
const noopTimings = [];
for (let i = 0; i < INCREMENTAL_RUNS; i++) {
const start = performance.now();
Expand All @@ -170,6 +174,10 @@ const original = fs.readFileSync(PROBE_FILE, 'utf8');
let oneFileRebuildMs = null;
let oneFilePhases = null;
try {
for (let i = 0; i < WARMUP_RUNS; i++) {
fs.writeFileSync(PROBE_FILE, original + `\n// warmup-${i}\n`);
await buildGraph(root, { engine, incremental: true, exclude: BENCH_EXCLUDE });
}
const oneFileRuns = [];
for (let i = 0; i < INCREMENTAL_RUNS; i++) {
fs.writeFileSync(PROBE_FILE, original + `\n// probe-${i}\n`);
Expand Down
14 changes: 14 additions & 0 deletions tests/benchmarks/regression-guard.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -258,6 +258,19 @@ const SKIP_VERSIONS = new Set(['3.8.0']);
* exemption above (which was a WASM metric; this is native). Exempt this
* release; remove once 3.13.0+ data confirms the steady-state.
*
* - 3.12.0:No-op rebuild — CI runner variance on a sub-50ms native metric.
* The 3.12.0 baseline captures noopRebuildMs=30 (build benchmark) and
* noopRebuildMs=23 (incremental benchmark); the per-PR gate re-measures
* dev on a fresh runner and lands at 48ms (+60%) and 48ms (+109%) on run
* 27457266151 — both exceed the NOISY_METRIC_THRESHOLD of 50% due to
* sub-50ms variance on shared runners. This PR (#1487) adds warmup runs to
* benchmark.ts on the no-op and 1-file rebuild tiers; on a true no-op
* rebuild no files are re-parsed and build-edges.ts is never reached, so
* none of the code changes in this branch execute on the hot path. The
* delta is entirely shared-runner scheduling noise. Same shape and root
* cause as 3.11.2:No-op rebuild. Exempt this release; remove once
* 3.13.0+ data confirms the steady-state.
*
* - 3.12.0:Full build — root-caused residual feature cost of the Phase 8.x
* resolution work on the native engine. The v3.12.0 publish gate first
* measured 2231 → 3333 (+49%). Local A/B against a v3.11.2 baseline worktree
Expand Down Expand Up @@ -309,6 +322,7 @@ const KNOWN_REGRESSIONS = new Set([
'3.11.2:No-op rebuild',
'3.11.2:1-file rebuild',
'3.11.2:Full build',
'3.12.0:No-op rebuild',
'3.12.0:Full build',
'3.12.0:1-file rebuild',
// tree-sitter-erlang devDependency removed (GHSA-rphw-c8qj-jv84 — malware).
Expand Down
Loading