Skip to content

Commit 13161f1

Browse files
committed
feat: cut simulator over to generated bb-avm-sim IPC service
Replace the in-process NAPI AVM with an out-of-process bb-avm-sim binary reached over IPC. The simulator drives a pool of bb-avm-sim processes (AvmSimulatorPool) behind an AvmExecutor that also runs the CDB server answering the C++ AVM's contract-data callbacks; per-fork work goes through AvmExecutor.forFork. The pool prewarms one process on spawn so the first simulate() runs at steady-state cost. Also adds the bench_compare script + bench-regression skill for PR perf checks.
1 parent acf7f98 commit 13161f1

86 files changed

Lines changed: 1441 additions & 2422 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
---
2+
name: bench-regression
3+
description: Check a PR for performance regressions by comparing its benchmark run against its merge-base on next. Use when a dev asks "did my PR regress anything?" or wants to vet benchmark impact before merge.
4+
argument-hint: [pr-ref] (defaults to HEAD)
5+
---
6+
7+
# PR benchmark regression check
8+
9+
Compares a PR's benchmark run against its baseline (merge-base on `next`, optionally a window of
10+
baseline runs) and reports only the regressions that (a) are statistically real and (b) plausibly
11+
relate to what the PR changed. The data-pulling and diff math are done by `ci3/bench_compare`; your
12+
job is to get a run, invoke the script, then filter the output with judgement.
13+
14+
## How benchmark data is stored (context)
15+
16+
Each CI benchmark run uploads a single `bench-out/bench.json` (a flat array of `{name, unit, value}`,
17+
`customSmallerIsBetter` — higher = worse) to the build-cache, keyed by the commit's **git tree hash**
18+
(`bench-<treehash>.tar.gz`, ~40 KB). A PR only produces one on an *uploadable* run: labels
19+
`ci-full-no-test-cache`, `ci-full`, or a merge-queue run. `ci3/bench_compare` pulls these per-run
20+
blobs — never the multi-MB per-branch `data.js` graph history.
21+
22+
## Steps
23+
24+
### 1. Get a benchmark run for the PR
25+
26+
- Default to `HEAD` (or the ref the user gives). Try step 2 directly — `ci3/bench_compare` fails
27+
clearly if the commit has no bench data.
28+
- **If there's no run and the user is fine using the last one:** walk back the branch for the most
29+
recent benched commit — `for c in $(git rev-list -n 30 HEAD); do ci3/bench_compare "$c" >/dev/null 2>&1 && echo "$c" && break; done` — and use that commit.
30+
- **If a fresh run is needed:** tell the user to add the `ci-full-no-test-cache` label to the PR (that
31+
triggers x-bench on a dedicated metal box and uploads) and to re-run this once CI finishes. Do not
32+
block waiting unless asked.
33+
34+
### 2. Determine the baseline branch — do NOT assume `next`
35+
36+
The comparison is `merge-base(PR, baseline)`, so the baseline MUST be the PR's *actual* target branch.
37+
A PR onto `v5-next` or a `merge-train/*` branch compared against `next` would report the entire
38+
branch-vs-branch perf delta as bogus regressions. Get it from the PR itself:
39+
40+
```bash
41+
gh pr view <pr-or-branch> --json baseRefName -q .baseRefName # e.g. next, v5-next, merge-train/spartan
42+
git fetch origin <baseRefName> # ensure origin/<baseRefName> is current
43+
```
44+
45+
(You do NOT need to worry about cross-branch data mixups: the bench cache is content-addressed by git
46+
tree hash, so `next`, `v5-next`, etc. already occupy different keys — a next commit's tree is only ever
47+
benched by next-content. The only thing you control, and must get right, is *which branch's lineage*
48+
you merge-base against.)
49+
50+
### 3. Run the comparison
51+
52+
```bash
53+
ci3/bench_compare <pr-commit> --baseline origin/<baseRefName> --window 5 --out /tmp/bench-report.json
54+
```
55+
56+
- Always pass `--baseline` explicitly (the script auto-detects via `gh` if omitted, but be explicit).
57+
It prints `baseline = …` to stderr — confirm it matches the PR's target before trusting the numbers.
58+
- `--window 5` averages the merge-base + 4 preceding baseline runs, yielding `stddev` and `z` per bench
59+
— essential for separating real regressions from flaky benches. Use `--window 1` if history is sparse.
60+
- JSON on `--out` is `{ meta, benches:[{name,unit,pr,baseline,n,pct,stddev,z,status}] }`, sorted by
61+
`pct` desc. Human summary on stderr.
62+
63+
### 4. Filter to what matters (your judgement)
64+
65+
Read the JSON and keep a regression only if **all** hold:
66+
67+
- **Statistically real:** `z >= ~4` (the PR value is ≥4 baseline-stddevs above the mean). A large
68+
`pct` with small/absent `z` is baseline noise — a flaky bench (e.g. `p2p/BatchTxRequester/.../duration`
69+
hitting a 30 s timeout, `pct` in the thousands) — **ignore it**. When window is 1 (no `z`), fall back
70+
to requiring a large `pct` AND a meaningful absolute magnitude.
71+
- **Material magnitude:** skip tiny absolutes (e.g. `0.01 ms -> 0.02 ms` = +100% but noise).
72+
- **Plausibly caused by the PR:** map the bench `name` prefix to the PR's changed areas. Get the diff
73+
with `git diff --name-only $(git merge-base HEAD origin/next)..HEAD | cut -d/ -f1-2 | sort -u`, then
74+
keep benches whose names start with a touched area (`yarn-project/simulator`, `barretenberg/cpp`,
75+
`avm-transpiler`, …) and treat far-away regressions (a TS-only PR moving a C++ proving bench) as
76+
drift/noise, not this PR.
77+
78+
### 5. Report
79+
80+
List the surviving regressions with `baseline -> pr unit`, `pct`, and `z`, grouped by area, and give a
81+
one-line verdict (clean / N real regressions in <area>). Note anything you filtered out and why, so
82+
the dev can override your judgement.
83+
84+
## Notes
85+
86+
- Baseline is `origin/next` by default; `git fetch` it first if stale (`ci3/bench_compare` errors if
87+
the ref is missing).
88+
- Requires `jq`, `python3`, and read access to the build-cache (the public `build-cache.aztec-labs.com`
89+
endpoint needs no creds; the in-VPC S3 path is a fallback).
90+
- To author or wire up new benchmarks, see the `adding-benchmarks` skill; this skill only *checks* them.

barretenberg/cpp/src/barretenberg/avm/avm_execute.cpp

Lines changed: 22 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -13,8 +13,24 @@ namespace bb::avm {
1313
using namespace bb::avm2;
1414
using namespace bb::world_state;
1515

16-
// Global cancellation token for the currently active simulation.
17-
// Set before simulation starts, cleared after. SIGUSR1 handler reads this to cancel.
16+
// Cancellation for the single in-flight simulation. bb-avm-sim runs exactly one
17+
// simulation at a time; the SIGUSR1 handler (which may run on any thread) cancels
18+
// it through g_active_cancellation_token.
19+
//
20+
// The token is process-lifetime and never freed. A per-request token would let
21+
// the signal handler dereference a pointer to a token the completing request had
22+
// already freed (use-after-free), since the handler can run concurrently with the
23+
// request thread unwinding. g_active_cancellation_token points at this token only
24+
// while a simulation runs and is null otherwise, so a signal between simulations
25+
// is a safe no-op.
26+
//
27+
// NOTE: cancellation is process-scoped (a signal), not request-scoped. A signal
28+
// delivered late — after its target finished and the next simulation began on this
29+
// process — would cancel the wrong simulation. The pool runs one simulation per
30+
// process and signals only the in-flight one, so this isn't exercised today;
31+
// hardening it would need a request-scoped cancel channel rather than a signal.
32+
const avm2::simulation::CancellationTokenPtr g_sim_cancellation_token =
33+
std::make_shared<avm2::simulation::CancellationToken>();
1834
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
1935
std::atomic<avm2::simulation::CancellationToken*> g_active_cancellation_token{ nullptr };
2036

@@ -36,7 +52,10 @@ template <typename T> static T deserialize_from_msgpack(const std::vector<uint8_
3652
template <>
3753
void handle_simulate(AvmRequest& request, wire::AvmSimulate&& command, Responder<wire::AvmSimulateResponse> respond)
3854
{
39-
auto cancellation_token = std::make_shared<avm2::simulation::CancellationToken>();
55+
// Reuse the process-lifetime token (cleared of any prior cancellation) instead
56+
// of allocating a per-request one the signal handler could outlive.
57+
g_sim_cancellation_token->reset();
58+
const avm2::simulation::CancellationTokenPtr& cancellation_token = g_sim_cancellation_token;
4059
try {
4160
auto sim_inputs = deserialize_from_msgpack<AvmFastSimulationInputs>(command.inputs);
4261

barretenberg/cpp/src/barretenberg/nodejs_module/CMakeLists.txt

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@ string(REGEX REPLACE "[\r\n\"]" "" NODE_API_HEADERS_DIR ${NODE_API_HEADERS_DIR})
2727
add_library(nodejs_module SHARED ${SOURCE_FILES})
2828
set_target_properties(nodejs_module PROPERTIES PREFIX "" SUFFIX ".node")
2929
target_include_directories(nodejs_module PRIVATE ${NODE_API_HEADERS_DIR} ${NODE_ADDON_API_DIR})
30-
target_link_libraries(nodejs_module PRIVATE ipc ipc_runtime vm2_sim wsdb_ipc_merkle_db)
30+
target_link_libraries(nodejs_module PRIVATE ipc ipc_runtime lmdblib)
3131

3232
# On macOS, Node.js N-API symbols are provided by the runtime, not at link time
3333
if(CMAKE_SYSTEM_NAME STREQUAL "Darwin")

0 commit comments

Comments
 (0)