Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions .cursor/rules/benchmarking.mdc
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,21 @@ Use this mapping when deciding which React benchmark scenarios are relevant to a

Regressions >5% on stable scenarios or >15% on volatile scenarios are worth investigating.

### Profiling / tracing (opt + deopt investigation)

The React benchmark supports the same V8 opt/deopt investigation as the Node benchmark, via Chromium's `--js-flags`:

- **`bench:trace`** (`BENCH_V8_TRACE=true`): launches Chromium with `--trace-opt --trace-deopt`; browser process output piped to `v8-trace.log`. Equivalent to `examples/benchmark`'s `start:trace`.
- **`bench:deopt`** (`BENCH_V8_DEOPT=true`): launches Chromium with `--prof`; V8 writes per-process logs to `v8-logs/v8-<pid>.log`. Process the renderer log (largest file) with `node --prof-process`.

Both default to `--lib data-client --size small` for focused runs. Override with additional flags:

```bash
yarn workspace example-benchmark-react bench:trace
yarn workspace example-benchmark-react bench:deopt
BENCH_V8_TRACE=true yarn workspace example-benchmark-react bench --scenario update-entity
```

### When to use Node vs React benchmark

- **Core/normalizr/endpoint changes only** (no rendering impact): Run `examples/benchmark` (Node). Faster iteration, no browser needed.
Expand Down
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# Logs
logs
*.log
v8-logs/
npm-debug.log*
yarn-debug.log*
yarn-error.log*
Expand Down
2 changes: 2 additions & 0 deletions examples/benchmark-react/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,3 +73,5 @@ Filtering: `yarn bench --lib data-client --size small --action update`
| `BENCH_LABEL=<tag>` | Appends `[<tag>]` to result names |
| `BENCH_PORT` | Preview port (default 5173) |
| `BENCH_TRACE=true` | Chrome tracing for duration scenarios |
| `BENCH_V8_TRACE=true` | Launch Chromium with `--trace-opt --trace-deopt`; output to `v8-trace.log` |
| `BENCH_V8_DEOPT=true` | Launch Chromium with `--prof`; V8 logs to `v8-logs/` |
30 changes: 29 additions & 1 deletion examples/benchmark-react/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -210,6 +210,34 @@ The runner prints a JSON array in `customBiggerIsBetter` format (name, unit, val

To view results locally, open `bench/report-viewer.html` in a browser and paste the JSON (or upload `react-bench-output.json`) to see a comparison table and bar chart.

## Optional: Chrome trace
## Profiling

### Chrome trace (timeline duration)

Set `BENCH_TRACE=true` when running the bench to enable Chrome tracing for duration scenarios. Trace files are written to disk; parsing and reporting trace duration is best-effort and may require additional tooling for the trace zip format.

### V8 opt/deopt investigation

For the same granularity of V8 optimization investigation available in `examples/benchmark` (Node), two modes pass `--js-flags` to Chromium via Playwright:

```bash
yarn bench:trace # --trace-opt --trace-deopt → v8-trace.log
yarn bench:deopt # --prof → v8-logs/v8-<pid>.log
```

Both default to `--lib data-client --size small` for focused, fast investigation. Add other flags as needed (e.g. `--scenario update-entity`).

**`bench:trace`** (`BENCH_V8_TRACE=true`) launches Chromium with `--js-flags="--trace-opt --trace-deopt"`. The runner uses Playwright `launchServer` (not `launch`) so the root browser process stdout/stderr can be piped to `v8-trace.log` — `Browser` from `launch()` does not expose `process()`. This is the browser equivalent of `examples/benchmark`'s `start:trace` — look for optimization and deoptimization lines for functions of interest.

**`bench:deopt`** (`BENCH_V8_DEOPT=true`) launches Chromium with `--js-flags="--prof"`. V8 writes per-process profiling logs to `v8-logs/v8-<pid>.log`. Chromium is multi-process, so several files are created; the renderer log (typically the largest) contains the benchmark's hot path. Process it with:

```bash
node --prof-process v8-logs/v8-<pid>.log > processed.txt
```

Both env vars can be combined for simultaneous trace output and profiling logs. The convenience scripts can be overridden:

```bash
BENCH_V8_TRACE=true yarn bench --lib data-client --scenario update-entity
BENCH_V8_DEOPT=true yarn bench --lib data-client --size large
```
108 changes: 105 additions & 3 deletions examples/benchmark-react/bench/runner.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,14 @@
/// <reference types="node" />
import * as fs from 'node:fs';
import * as path from 'node:path';
import { chromium } from 'playwright';
import type { Browser, CDPSession, Locator, Page } from 'playwright';
import type {
Browser,
BrowserServer,
CDPSession,
Locator,
Page,
} from 'playwright';

import { collectMeasures, getMeasureDuration } from './measure.js';
import { collectHeapUsed } from './memory.js';
Expand Down Expand Up @@ -141,6 +149,9 @@ const BASE_URL =
const BENCH_LABEL =
process.env.BENCH_LABEL ? ` [${process.env.BENCH_LABEL}]` : '';
const USE_TRACE = process.env.BENCH_TRACE === 'true';
const BENCH_V8_TRACE = process.env.BENCH_V8_TRACE === 'true';
const BENCH_V8_DEOPT = process.env.BENCH_V8_DEOPT === 'true';
const V8_LOG_DIR = path.resolve('v8-logs');
const MEMORY_WARMUP = 1;
const MEMORY_MEASUREMENTS = 3;

Expand Down Expand Up @@ -649,6 +660,93 @@ function shuffle<T>(arr: T[]): T[] {
return out;
}

/** Chromium from `launch()` does not expose `process()`; use `launchServer` when piping V8 trace output. */
async function launchBenchChromium(): Promise<{
browser: Browser;
closeBenchBrowser: () => Promise<void>;
}> {
const launchOpts = {
headless: true,
args: buildV8LaunchArgs(),
};

if (BENCH_V8_TRACE) {
const server: BrowserServer = await chromium.launchServer(launchOpts);
let v8TraceStream: fs.WriteStream | undefined;
const proc = server.process();
if (proc?.stderr ?? proc?.stdout) {
v8TraceStream = fs.createWriteStream('v8-trace.log');
proc.stderr?.pipe(v8TraceStream, { end: false });
proc.stdout?.pipe(v8TraceStream, { end: false });
process.stderr.write(
'V8 trace output → v8-trace.log (root browser process stderr/stdout)\n',
);
} else {
process.stderr.write(
'Warning: BENCH_V8_TRACE but browser server process streams unavailable; v8-trace.log may be empty.\n',
);
}
const browser = await chromium.connect({ wsEndpoint: server.wsEndpoint() });
return {
browser,
closeBenchBrowser: async () => {
await browser.close();
await server.close();
if (v8TraceStream) {
v8TraceStream.end();
process.stderr.write(
'\nV8 opt/deopt trace written to v8-trace.log\n',
);
}
},
Comment thread
cursor[bot] marked this conversation as resolved.
};
}

const browser = await chromium.launch(launchOpts);
return {
browser,
closeBenchBrowser: () => browser.close(),
};
}

function buildV8LaunchArgs(): string[] {
const jsFlags: string[] = [];
if (BENCH_V8_TRACE) {
jsFlags.push('--trace-opt', '--trace-deopt');
}
if (BENCH_V8_DEOPT) {
fs.rmSync(V8_LOG_DIR, { recursive: true, force: true });
fs.mkdirSync(V8_LOG_DIR, { recursive: true });
jsFlags.push('--prof', `--logfile=${V8_LOG_DIR}/v8-%p.log`);
}
if (jsFlags.length === 0) return [];
return [`--js-flags=${jsFlags.join(' ')}`];
}

function reportV8Logs(): void {
if (!BENCH_V8_DEOPT) return;
try {
const logs = fs.readdirSync(V8_LOG_DIR).filter(f => f.endsWith('.log'));
if (logs.length === 0) return;
process.stderr.write(`\nV8 profiling logs written to ${V8_LOG_DIR}/:\n`);
for (const log of logs) {
const size = fs.statSync(path.join(V8_LOG_DIR, log)).size;
process.stderr.write(` ${log} (${(size / 1024).toFixed(1)} KB)\n`);
}
const largest = logs.reduce((a, b) => {
const sa = fs.statSync(path.join(V8_LOG_DIR, a)).size;
const sb = fs.statSync(path.join(V8_LOG_DIR, b)).size;
return sa >= sb ? a : b;
});
process.stderr.write(
`\nProcess the renderer log (typically the largest file) with:\n` +
` node --prof-process ${V8_LOG_DIR}/${largest}\n\n`,
);
} catch {
// best-effort reporting
}
}
Comment thread
cursor[bot] marked this conversation as resolved.

function scenarioUnit(scenario: Scenario): string {
if (isRefStabilityScenario(scenario)) return 'count';
if (scenario.resultMetric === 'heapDelta') return 'bytes';
Expand Down Expand Up @@ -778,7 +876,10 @@ async function main() {
samples.set(s.name, { value: [], reactCommit: [], trace: [] });
}

const browser = await chromium.launch({ headless: true });
const { browser, closeBenchBrowser } = await launchBenchChromium();
if (BENCH_V8_DEOPT) {
process.stderr.write(`V8 profiling logs → ${V8_LOG_DIR}/\n`);
}

// Deterministic scenarios: run once, no warmup
const deterministicNames = new Set<string>();
Expand Down Expand Up @@ -924,7 +1025,8 @@ async function main() {
}
}

await browser.close();
await closeBenchBrowser();
reportV8Logs();

// ---------------------------------------------------------------------------
// Report
Expand Down
2 changes: 2 additions & 0 deletions examples/benchmark-react/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@
"bench:small": "npx tsx bench/runner.ts --size small",
"bench:large": "npx tsx bench/runner.ts --size large",
"bench:dc": "npx tsx bench/runner.ts --lib data-client",
"bench:trace": "BENCH_V8_TRACE=true npx tsx bench/runner.ts --lib data-client --size small",
"bench:deopt": "BENCH_V8_DEOPT=true npx tsx bench/runner.ts --lib data-client --size small",
"bench:run": "yarn build && (yarn preview &) && sleep 5 && yarn bench",
"bench:run:no-compiler": "yarn build:no-compiler && (yarn preview &) && sleep 5 && yarn bench:no-compiler",
"validate": "npx tsx bench/validate.ts",
Expand Down