Skip to content

Commit 8199db7

Browse files
carlos-almcarlos-alm
authored
feat: child-process isolation for benchmarks (#512)
* feat: child-process isolation for benchmarks Run each engine/model benchmark in a forked subprocess so that segfaults (e.g. from the native Rust addon) only kill the child — the parent survives and collects partial results from whichever engines succeeded. - New shared utility: scripts/lib/fork-engine.js (isWorker, workerEngine, forkEngines) handles subprocess lifecycle, timeout, crash recovery, and partial JSON result extraction - benchmark.js, query-benchmark.js, incremental-benchmark.js: fork per engine (wasm/native) via forkEngines() - embedding-benchmark.js: fork per model (ONNX runtime can OOM/segfault) - JSON output contract unchanged — report scripts and CI need no changes * fix: extract shared forkWorker, remove dual timeout, add settle guard (#512) - Extract forkWorker() as a generic reusable subprocess helper - Remove fork()'s built-in timeout option so the manual SIGKILL timer is the sole timeout mechanism (they previously fired simultaneously) - Add a settled flag to prevent double Promise resolution on spawn failure - forkEngines() now delegates to forkWorker() instead of inline runWorker() * fix: capture and call cleanup from resolveBenchmarkSource in parent (#512) The first resolveBenchmarkSource() call (for version extraction) never had its cleanup invoked, potentially leaking temporary resources. * fix: replace duplicated forkModel with shared forkWorker (#512) embedding-benchmark.js had a near-identical copy of runWorker() from fork-engine.js (including the dual-timeout bug). Now imports and uses the shared forkWorker() helper instead. * fix: call versionCleanup on error exit path in benchmark.js (#512) versionCleanup() was only called on the success path. When both engines fail and the parent exits early via process.exit(1), temporary resources from resolveBenchmarkSource() were leaked. * fix: add parent cleanup and git safety net in query-benchmark.js (#512) Two issues addressed: 1. resolveBenchmarkSource() cleanup was never captured or called in the parent process, leaking temporary resources on every run. 2. benchDiffImpact() runs git-add inside the worker — if a segfault kills the worker mid-execution, the finally block never runs and the git staging area is left dirty. The parent now checks for leftover staged files after workers exit and cleans them up. * fix: replace process.exit in forkEngines with thrown error (#512) forkEngines() called process.exit(1) when no engines were available, which aborted the call-stack immediately and prevented callers from invoking their own cleanup callbacks (versionCleanup, parentCleanup). Now throws an Error instead; all three callers catch it, run cleanup, and exit gracefully. Impact: 1 functions changed, 3 affected
1 parent 01e5b96 commit 8199db7

5 files changed

Lines changed: 631 additions & 505 deletions

File tree

‎scripts/benchmark.js‎

Lines changed: 147 additions & 169 deletions
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,9 @@
33
/**
44
* Benchmark runner — measures codegraph performance on itself (dogfooding).
55
*
6-
* Runs both native (Rust) and WASM engines, outputs JSON to stdout
7-
* with raw and per-file normalized metrics for each.
6+
* Each engine (native / WASM) runs in a forked subprocess so that a segfault
7+
* in the native addon only kills the child — the parent survives and collects
8+
* partial results from whichever engines succeeded.
89
*
910
* Usage: node scripts/benchmark.js
1011
*/
@@ -15,25 +16,82 @@ import { performance } from 'node:perf_hooks';
1516
import { fileURLToPath } from 'node:url';
1617
import Database from 'better-sqlite3';
1718
import { resolveBenchmarkSource, srcImport } from './lib/bench-config.js';
19+
import { isWorker, workerEngine, forkEngines } from './lib/fork-engine.js';
20+
21+
// ── Parent process: fork one child per engine, assemble final output ─────
22+
if (!isWorker()) {
23+
const { version, cleanup: versionCleanup } = await resolveBenchmarkSource();
24+
let wasm, native;
25+
try {
26+
({ wasm, native } = await forkEngines(import.meta.url, process.argv.slice(2)));
27+
} catch (err) {
28+
console.error(`Error: ${err.message}`);
29+
versionCleanup();
30+
process.exit(1);
31+
}
32+
33+
const primary = wasm || native;
34+
if (!primary) {
35+
console.error('Error: Both engines failed. No results to report.');
36+
versionCleanup();
37+
process.exit(1);
38+
}
39+
40+
const result = {
41+
version,
42+
date: new Date().toISOString().slice(0, 10),
43+
files: primary.files,
44+
wasm: wasm
45+
? {
46+
buildTimeMs: wasm.buildTimeMs,
47+
queryTimeMs: wasm.queryTimeMs,
48+
nodes: wasm.nodes,
49+
edges: wasm.edges,
50+
dbSizeBytes: wasm.dbSizeBytes,
51+
perFile: wasm.perFile,
52+
noopRebuildMs: wasm.noopRebuildMs,
53+
oneFileRebuildMs: wasm.oneFileRebuildMs,
54+
oneFilePhases: wasm.oneFilePhases,
55+
queries: wasm.queries,
56+
phases: wasm.phases,
57+
}
58+
: null,
59+
native: native
60+
? {
61+
buildTimeMs: native.buildTimeMs,
62+
queryTimeMs: native.queryTimeMs,
63+
nodes: native.nodes,
64+
edges: native.edges,
65+
dbSizeBytes: native.dbSizeBytes,
66+
perFile: native.perFile,
67+
noopRebuildMs: native.noopRebuildMs,
68+
oneFileRebuildMs: native.oneFileRebuildMs,
69+
oneFilePhases: native.oneFilePhases,
70+
queries: native.queries,
71+
phases: native.phases,
72+
}
73+
: null,
74+
};
75+
76+
console.log(JSON.stringify(result, null, 2));
77+
versionCleanup();
78+
process.exit(0);
79+
}
80+
81+
// ── Worker process: benchmark a single engine, write JSON to stdout ──────
82+
const engine = workerEngine();
1883

1984
const __dirname = path.dirname(fileURLToPath(import.meta.url));
2085
const root = path.resolve(__dirname, '..');
2186

22-
const { version, srcDir, cleanup } = await resolveBenchmarkSource();
87+
const { srcDir, cleanup } = await resolveBenchmarkSource();
2388

2489
const dbPath = path.join(root, '.codegraph', 'graph.db');
2590

26-
// Import programmatic API (use file:// URLs for Windows compatibility)
2791
const { buildGraph } = await import(srcImport(srcDir, 'builder.js'));
2892
const { fnDepsData, fnImpactData, pathData, rolesData, statsData } = await import(
2993
srcImport(srcDir, 'queries.js')
3094
);
31-
const { isNativeAvailable } = await import(
32-
srcImport(srcDir, 'native.js')
33-
);
34-
const { isWasmAvailable } = await import(
35-
srcImport(srcDir, 'parser.js')
36-
);
3795

3896
const INCREMENTAL_RUNS = 3;
3997
const QUERY_RUNS = 5;
@@ -49,9 +107,6 @@ function round1(n) {
49107
return Math.round(n * 10) / 10;
50108
}
51109

52-
/**
53-
* Pick hub (most-connected) and leaf (least-connected) non-test symbols from the DB.
54-
*/
55110
function selectTargets() {
56111
const db = new Database(dbPath, { readonly: true });
57112
const rows = db
@@ -67,183 +122,106 @@ function selectTargets() {
67122
db.close();
68123

69124
if (rows.length === 0) return { hub: 'buildGraph', leaf: 'median' };
70-
71125
return { hub: rows[0].name, leaf: rows[rows.length - 1].name };
72126
}
73127

74128
// Redirect console.log to stderr so only JSON goes to stdout
75129
const origLog = console.log;
76130
console.log = (...args) => console.error(...args);
77131

78-
async function benchmarkEngine(engine) {
79-
// Clean DB for a full build
80-
if (fs.existsSync(dbPath)) fs.unlinkSync(dbPath);
81-
82-
const buildStart = performance.now();
83-
const buildResult = await buildGraph(root, { engine, incremental: false });
84-
const buildTimeMs = performance.now() - buildStart;
85-
86-
const queryStart = performance.now();
87-
fnDepsData('buildGraph', dbPath);
88-
const queryTimeMs = performance.now() - queryStart;
89-
90-
const stats = statsData(dbPath);
91-
const totalFiles = stats.files.total;
92-
const totalNodes = stats.nodes.total;
93-
const totalEdges = stats.edges.total;
94-
const dbSizeBytes = fs.statSync(dbPath).size;
95-
96-
// ── Incremental build tiers (reuse existing DB from full build) ─────
97-
console.error(` [${engine}] Benchmarking no-op rebuild...`);
98-
const noopTimings = [];
132+
// Clean DB for a full build
133+
if (fs.existsSync(dbPath)) fs.unlinkSync(dbPath);
134+
135+
const buildStart = performance.now();
136+
const buildResult = await buildGraph(root, { engine, incremental: false });
137+
const buildTimeMs = performance.now() - buildStart;
138+
139+
const queryStart = performance.now();
140+
fnDepsData('buildGraph', dbPath);
141+
const queryTimeMs = performance.now() - queryStart;
142+
143+
const stats = statsData(dbPath);
144+
const totalFiles = stats.files.total;
145+
const totalNodes = stats.nodes.total;
146+
const totalEdges = stats.edges.total;
147+
const dbSizeBytes = fs.statSync(dbPath).size;
148+
149+
// ── Incremental build tiers ─────────────────────────────────────────
150+
console.error(` [${engine}] Benchmarking no-op rebuild...`);
151+
const noopTimings = [];
152+
for (let i = 0; i < INCREMENTAL_RUNS; i++) {
153+
const start = performance.now();
154+
await buildGraph(root, { engine, incremental: true });
155+
noopTimings.push(performance.now() - start);
156+
}
157+
const noopRebuildMs = Math.round(median(noopTimings));
158+
159+
console.error(` [${engine}] Benchmarking 1-file rebuild...`);
160+
const original = fs.readFileSync(PROBE_FILE, 'utf8');
161+
let oneFileRebuildMs;
162+
let oneFilePhases = null;
163+
try {
164+
const oneFileRuns = [];
99165
for (let i = 0; i < INCREMENTAL_RUNS; i++) {
166+
fs.writeFileSync(PROBE_FILE, original + `\n// probe-${i}\n`);
100167
const start = performance.now();
101-
await buildGraph(root, { engine, incremental: true });
102-
noopTimings.push(performance.now() - start);
168+
const res = await buildGraph(root, { engine, incremental: true });
169+
oneFileRuns.push({ ms: performance.now() - start, phases: res?.phases || null });
103170
}
104-
const noopRebuildMs = Math.round(median(noopTimings));
105-
106-
console.error(` [${engine}] Benchmarking 1-file rebuild...`);
107-
const original = fs.readFileSync(PROBE_FILE, 'utf8');
108-
let oneFileRebuildMs;
109-
let oneFilePhases = null;
110-
try {
111-
const oneFileRuns = [];
112-
for (let i = 0; i < INCREMENTAL_RUNS; i++) {
113-
fs.writeFileSync(PROBE_FILE, original + `\n// probe-${i}\n`);
114-
const start = performance.now();
115-
const res = await buildGraph(root, { engine, incremental: true });
116-
oneFileRuns.push({ ms: performance.now() - start, phases: res?.phases || null });
117-
}
118-
oneFileRuns.sort((a, b) => a.ms - b.ms);
119-
const medianRun = oneFileRuns[Math.floor(oneFileRuns.length / 2)];
120-
oneFileRebuildMs = Math.round(medianRun.ms);
121-
oneFilePhases = medianRun.phases;
122-
} finally {
123-
fs.writeFileSync(PROBE_FILE, original);
124-
await buildGraph(root, { engine, incremental: true });
125-
}
126-
127-
// ── Query benchmarks (median of QUERY_RUNS each) ────────────────────
128-
console.error(` [${engine}] Benchmarking queries...`);
129-
const targets = selectTargets();
130-
console.error(` hub=${targets.hub}, leaf=${targets.leaf}`);
131-
132-
function benchQuery(fn, ...args) {
133-
const timings = [];
134-
for (let i = 0; i < QUERY_RUNS; i++) {
135-
const start = performance.now();
136-
fn(...args);
137-
timings.push(performance.now() - start);
138-
}
139-
return round1(median(timings));
140-
}
141-
142-
const queries = {
143-
fnDepsMs: fnDepsData ? benchQuery(fnDepsData, targets.hub, dbPath, { depth: 3, noTests: true }) : null,
144-
fnImpactMs: fnImpactData ? benchQuery(fnImpactData, targets.hub, dbPath, { depth: 3, noTests: true }) : null,
145-
pathMs: pathData ? benchQuery(pathData, targets.hub, targets.leaf, dbPath, { noTests: true }) : null,
146-
rolesMs: rolesData ? benchQuery(rolesData, dbPath, { noTests: true }) : null,
147-
};
148-
149-
return {
150-
buildTimeMs: Math.round(buildTimeMs),
151-
queryTimeMs: Math.round(queryTimeMs * 10) / 10,
152-
nodes: totalNodes,
153-
edges: totalEdges,
154-
files: totalFiles,
155-
dbSizeBytes,
156-
perFile: {
157-
buildTimeMs: Math.round((buildTimeMs / totalFiles) * 10) / 10,
158-
nodes: Math.round((totalNodes / totalFiles) * 10) / 10,
159-
edges: Math.round((totalEdges / totalFiles) * 10) / 10,
160-
dbSizeBytes: Math.round(dbSizeBytes / totalFiles),
161-
},
162-
noopRebuildMs,
163-
oneFileRebuildMs,
164-
oneFilePhases,
165-
queries,
166-
phases: buildResult?.phases || null,
167-
};
171+
oneFileRuns.sort((a, b) => a.ms - b.ms);
172+
const medianRun = oneFileRuns[Math.floor(oneFileRuns.length / 2)];
173+
oneFileRebuildMs = Math.round(medianRun.ms);
174+
oneFilePhases = medianRun.phases;
175+
} finally {
176+
fs.writeFileSync(PROBE_FILE, original);
177+
await buildGraph(root, { engine, incremental: true });
168178
}
169179

170-
// ── Run benchmarks ───────────────────────────────────────────────────────
171-
const hasWasm = isWasmAvailable();
172-
const hasNative = isNativeAvailable();
173-
174-
if (!hasWasm && !hasNative) {
175-
console.error('Error: Neither WASM grammars nor native engine are available.');
176-
console.error('Run "npm run build:wasm" to build WASM grammars, or install the native platform package.');
177-
process.exit(1);
178-
}
180+
// ── Query benchmarks ────────────────────────────────────────────────
181+
console.error(` [${engine}] Benchmarking queries...`);
182+
const targets = selectTargets();
183+
console.error(` hub=${targets.hub}, leaf=${targets.leaf}`);
179184

180-
let wasm = null;
181-
if (hasWasm) {
182-
try {
183-
wasm = await benchmarkEngine('wasm');
184-
} catch (err) {
185-
console.error(`WASM benchmark failed: ${err?.message ?? String(err)}`);
185+
function benchQuery(fn, ...args) {
186+
const timings = [];
187+
for (let i = 0; i < QUERY_RUNS; i++) {
188+
const start = performance.now();
189+
fn(...args);
190+
timings.push(performance.now() - start);
186191
}
187-
} else {
188-
console.error('WASM grammars not built — skipping WASM benchmark');
192+
return round1(median(timings));
189193
}
190194

191-
let native = null;
192-
if (hasNative) {
193-
try {
194-
native = await benchmarkEngine('native');
195-
} catch (err) {
196-
console.error(`Native benchmark failed: ${err?.message ?? String(err)}`);
197-
}
198-
} else {
199-
console.error('Native engine not available — skipping native benchmark');
200-
}
195+
const queries = {
196+
fnDepsMs: fnDepsData ? benchQuery(fnDepsData, targets.hub, dbPath, { depth: 3, noTests: true }) : null,
197+
fnImpactMs: fnImpactData ? benchQuery(fnImpactData, targets.hub, dbPath, { depth: 3, noTests: true }) : null,
198+
pathMs: pathData ? benchQuery(pathData, targets.hub, targets.leaf, dbPath, { noTests: true }) : null,
199+
rolesMs: rolesData ? benchQuery(rolesData, dbPath, { noTests: true }) : null,
200+
};
201201

202202
// Restore console.log for JSON output
203203
console.log = origLog;
204204

205-
const primary = wasm || native;
206-
if (!primary) {
207-
console.error('Error: Both engines failed. No results to report.');
208-
cleanup();
209-
process.exit(1);
210-
}
211-
const result = {
212-
version,
213-
date: new Date().toISOString().slice(0, 10),
214-
files: primary.files,
215-
wasm: wasm
216-
? {
217-
buildTimeMs: wasm.buildTimeMs,
218-
queryTimeMs: wasm.queryTimeMs,
219-
nodes: wasm.nodes,
220-
edges: wasm.edges,
221-
dbSizeBytes: wasm.dbSizeBytes,
222-
perFile: wasm.perFile,
223-
noopRebuildMs: wasm.noopRebuildMs,
224-
oneFileRebuildMs: wasm.oneFileRebuildMs,
225-
oneFilePhases: wasm.oneFilePhases,
226-
queries: wasm.queries,
227-
phases: wasm.phases,
228-
}
229-
: null,
230-
native: native
231-
? {
232-
buildTimeMs: native.buildTimeMs,
233-
queryTimeMs: native.queryTimeMs,
234-
nodes: native.nodes,
235-
edges: native.edges,
236-
dbSizeBytes: native.dbSizeBytes,
237-
perFile: native.perFile,
238-
noopRebuildMs: native.noopRebuildMs,
239-
oneFileRebuildMs: native.oneFileRebuildMs,
240-
oneFilePhases: native.oneFilePhases,
241-
queries: native.queries,
242-
phases: native.phases,
243-
}
244-
: null,
205+
const workerResult = {
206+
buildTimeMs: Math.round(buildTimeMs),
207+
queryTimeMs: Math.round(queryTimeMs * 10) / 10,
208+
nodes: totalNodes,
209+
edges: totalEdges,
210+
files: totalFiles,
211+
dbSizeBytes,
212+
perFile: {
213+
buildTimeMs: Math.round((buildTimeMs / totalFiles) * 10) / 10,
214+
nodes: Math.round((totalNodes / totalFiles) * 10) / 10,
215+
edges: Math.round((totalEdges / totalFiles) * 10) / 10,
216+
dbSizeBytes: Math.round(dbSizeBytes / totalFiles),
217+
},
218+
noopRebuildMs,
219+
oneFileRebuildMs,
220+
oneFilePhases,
221+
queries,
222+
phases: buildResult?.phases || null,
245223
};
246224

247-
console.log(JSON.stringify(result, null, 2));
225+
console.log(JSON.stringify(workerResult));
248226

249227
cleanup();

0 commit comments

Comments
 (0)