chore(port): forward-port v5-next e2e/testing backlog to next#24930
Merged
Conversation
Updates every README in the e2e test section to describe the suite as it will look once the nine open round-2 consolidation PRs land. This is docs-only — no code, config, or `bootstrap.sh` changes. ## What changed per README - **`end-to-end/README.md`** (root): repointed the `LOG_LEVEL` example off the deleted `proving/empty_blocks.test.ts` to `proving/default_node.test.ts`; linked the new p2p README from the category table (the p2p row now reads "yes" and points at `src/p2p/README.md`); and renamed the `setupBlockProducer` submission-window literal `1024` to the named constant `NO_REORG_SUBMISSION_EPOCHS`. - **`end-to-end/src/single-node/README.md`**: rewrote the `cross-chain/` row for the merged layout (`CrossChainMessagingTest` + `message_test_helpers.ts`, and the `l1_to_l2` / `l1_to_l2_inbox_drift` / `l2_to_l1` / `token_bridge` files); folded `public_payments`/`sponsored_payments` into `private_payments.parallel` and fixed the stale `failures.parallel` name to `failures`; replaced `slasher_config`/`sequencer_config` with the merged `runtime_config`; replaced `world_state_pruning`/`empty_blocks` with the merged `default_node`; added the previously-missing `snapshot_sync` to the `sync/` row; and applied the `NO_REORG_SUBMISSION_EPOCHS` rename. - **`end-to-end/src/multi-node/README.md`**: named `equivocation_offenses` in the slashing row (in place of the separate `duplicate_proposal`/`duplicate_attestation` files) and applied the `NO_REORG_SUBMISSION_EPOCHS` rename in the `MOCK_GOSSIP_MULTI_VALIDATOR_OPTS` preset. - **`end-to-end/src/p2p/README.md`** (new): created with the content added by #24500. - **`end-to-end/src/automine/README.md`** and **`end-to-end/src/infra/README.md`**: untouched — both are behavior-level (no file names) or cover a directory none of the nine PRs change, so they remain accurate as-landed. --------- Co-authored-by: AztecBot <tech@aztec-labs.com> (cherry picked from commit 1cc5f91)
## What / why
Round-4 e2e speedup, PR 3 of the effort. The `single-node/bot` suite's
file-level `beforeAll` costs
~178s on CI with 97% untagged, and the residual proven-chain waits in
the `private_payments`/`failures`
fees suites (~32s/shard) were also invisible. This PR instruments both
so the cost is attributable, and
diagnoses the bot residual.
Spans are pure passthrough when `TEST_TIMING_FILE` is unset (see
`fixtures/timing.ts`), so this is zero
behavior change for normal runs and CI.
## Commit
- `test(e2e): instrument bot suite setup and fees proven-chain
residuals`
- Bot suite (`single-node/bot/bot.test.ts`): `setup:wallet` around
`EmbeddedWallet.create`,
`wallet:create` around `createSchnorrInitializerlessAccount`, and
`setup:bot` around the four nested
`describe` beforeAll hooks (`Bot.create` / `AmmBot.create` /
`CrossChainBot.create` / `BotStore`).
- Fees harness (`single-node/fees`): `wait:proven-checkpoint` around
`catchUpProvenChain`'s poll loop,
and `warp:proven-checkpoint-epoch` around `advanceToNextEpoch` via a new
instrumented
`FeesTest.advanceToNextEpoch()` wrapper; `waitForEpochProven` and the
direct call sites in
`failures`/`private_payments` now route through it.
- Reuses existing tags where the concept matches (`wallet:create` is
already what
`createFundedInitializerlessAccounts` fires;
`warp:proven-checkpoint-epoch` is already what
`advanceToNextEpoch` fires in `block-production/setup.ts`) rather than
minting near-duplicates.
## Diagnosis
The timing environment (`shared/timing_env.mjs`) folds *every*
`beforeAll` in a file — the file-level
one plus each nested `describe`'s — into a single suite-scoped
`beforeHooksMs`, and every span fired
during any beforeAll lands on that same suite line. The shortlist read
the whole number as the
file-level hook and, seeing only `setup:env:*` tagged, attributed the
rest to the one visible untagged
file-level call (`createSchnorrInitializerlessAccount`). It is actually
the nested hooks.
Local read (`TEST_TIMING_SPANS=1`, `PIPELINING_SETUP_OPTS` = 12s L2
slots, production sequencer, fake
prover), suite `beforeHooksMs = 153,733 ms`:
- `setup:env:*` (file-level `setup()`): ~4.6s — already tagged
- `setup:wallet` (`EmbeddedWallet.create`, ephemeral): **137 ms** — the
PXE has `autoSync:false`, no
genesis walk
- `wallet:create` (`createSchnorrInitializerlessAccount`): **69 ms** —
initializerless, no deploy tx, as
designed
- `setup:bot` (4 nested hooks): **149,582 ms = 97.3%** of the file's
beforeAll
- transaction-bot `Bot.create`: 32.2s (token deploy class+instance, then
private+public mint)
- bridge-resume `new BotStore(...)`: 0.9 ms
- amm-bot `AmmBot.create`: 73.9s (deploys 4 contracts + set_minter +
mint + add_liquidity)
- cross-chain-bot `CrossChainBot.create`: 43.5s (TestContract deploy +
seed 2 L1→L2 msgs + wait ready)
**Root cause: structural, not a bug.** The bot suite runs the production
Sequencer to exercise
CHECKPOINTED/PROPOSED follow modes and L1 fee-juice bridging, so each
`Bot.create` deploys real
contracts and mints serially at slot cadence (`bot/src/factory.ts`). The
two calls the shortlist
suspected are ~0.1s each.
No fix is shipped here because there is no safe one-liner: speeding this
up means batching/parallelizing
the `bot/src/factory.ts` deploy/mint paths (production code,
PR-6-shaped), genesis-seeding (PR 1/2), or
cutting the CI slot time (PR 5) — none belong in the instrumentation PR.
Tracked as a round-5 item; the
new `setup:bot` span makes that win measurable. Full write-up in
`tmp/SPEEDUP_ROUND4_FINDINGS.md`.
## Expected effect
No wall-clock change (passthrough spans). The deliverable is visibility:
the previously-invisible
`setup:bot` / `setup:wallet` / `wallet:create` decomposition of the bot
beforeAll, and the
`wait:proven-checkpoint` / `warp:proven-checkpoint-epoch` residual in
the fees suites.
Measured numbers from a full green CI run will be appended below once
available.
## Measured impact
Both runs are full CI runs. PR run: `ci/x-fast` on head `1f7898cd2b` (CI
1783345614459429, 1,799 test
lines). Baseline: `ci/x-full-no-test-cache` on the exact merge-base
`a8cfa93df5` (CI 1783341726388943,
1,863 test lines). No wall-clock change claimed (spans are passthrough;
suite-level noise floor is
~11s median / ~52s p90) — the deliverable is attribution, measured as
span `busyMs`.
Bot suite (`suite` line, beforeAll = 181.5s on the PR run vs 182.1s
baseline):
- Baseline: 7.4s tagged (`setup:env:*` only) → **174.8s / 96%
invisible**.
- PR run: `setup:bot` **174,203 ms** (4 occurrences, max 86,961 ms = the
amm-bot hook),
`setup:wallet` **250 ms**, `wallet:create` **144 ms**; residual now
**−1.5s ≈ 0** →
**100% of the beforeAll is attributed**.
- This confirms the diagnosis: `EmbeddedWallet.create` +
`createSchnorrInitializerlessAccount` are
~0.4s combined; the "invisible 178s" is the four nested describes'
bot-factory setup
(deploys + mints at production 12s-slot cadence), folded into the file's
suite line by the
hook accounting.
Fees suites (`private_payments.parallel` × 8 shards + `failures`):
- Baseline: avg **32.3s/shard untagged residual** (zero
`wait:proven-checkpoint` /
`warp:proven-checkpoint-epoch` lines on these suites).
- PR run: `wait:proven-checkpoint` fires on all 9 suite hooks, **293,888
ms total**
(avg 32.7s/shard — matches the residual exactly), plus one in-test
occurrence (32.1s);
`warp:proven-checkpoint-epoch` 19 occurrences, 361 ms total (the warp
RPC itself is
near-free — the cost is entirely the proven-chain poll that follows).
Per-shard residual is now
≈ −1s, i.e. fully attributed.
Net: **~469s/run of previously invisible setup/wait cost is now tagged
and attributable**
(174.2s `setup:bot` + 0.4s wallet spans + 294.3s fees proven-chain
waits), with no behavior or
wall-clock change.
Fixes A-1407
(cherry picked from commit 1476179)
…24566) PR 4 of the round-4 e2e speedup effort: extend the merged `warpWithSequencersPaused` primitive (from #24475, proven on `pipeline_prune`) to the remaining production-sequencer-bound dead waits. One commit per converted site so a CI-hostile one reverts in isolation. On the fresh `merge-train/spartan-v5` base, most of the sites the plan targeted have already been warp-optimized (they use `warpToBuildWindowForSlot` / `warpToEpochStart` / `waitUntilNextEpochStarts`, and `pipeline_prune` already uses `warpWithSequencersPaused`). The residual waits in the remaining sites ride real-time consensus/production that a warp cannot safely collapse — warping would skip the checkpoints/attestations/offenses the tests are verifying. After reading each file on the fresh base, only one genuinely-dead wall-clock wait was left to convert; the rest are honest skips (per the round-3 skip rule: an honest skip beats a flaky convert). ## Converted (one commit) - **`multi-node/recovery/proposal_failure_recovery` — missed-L1-publish test.** Between "proposed chain reached slotTwo" and the prune, the test waited in wall-clock (`waitUntilL1Timestamp`) for the L1 clock to roll past slotOne so the archiver prunes the uncheckpointed slotOne/slotTwo blocks. The pipelined slotTwo broadcast has already reached every node and slotThree does not build until slotTwo, so nothing must be produced in that gap — a genuine dead wait, the direct `pipeline_prune` analog. Replaced with `warpWithSequencersPaused(..., { restart: false })`: pause across the warp (warping under a running sequencer would interrupt in-flight builds), keep the sequencers stopped until the prune is confirmed so no proposer builds on the unpruned tip, then restart them for recovery. Expected saving: up to ~1 L2 slot (aztecSlot=36s on this suite's cadence) per run of that test. ## Skipped, with reasons - **`proposal_failure_recovery` — orphan-prune test (second `it`).** The prune wait overlaps P2's concurrent S2 checkpoint rebuild inside the same slot (pipelining builds S2 during S1 while the orphan is pruned). Pausing the sequencers to warp would stop the S2 rebuild the test then asserts on. No pausable dead gap. - **`multi-node/recovery/equivocation_recovery`.** The dominant waits are the heal phase (`waitUntilCheckpointNumber` / `waitForAllNodesToReachCheckpoint` for baseline+2) which requires B/C to actually produce two checkpoints after A is stopped, plus `waitForOffenseOnNodes` which requires real slashing-round detection. Warping skips the very production being verified. - **`multi-node/slashing/slash_veto_demo`.** `waitForSubmittableRound` accrues inactivity offenses to quorum over proven blocks, and the veto/expiry logic rides real slashing-round consensus. This is proof-submission/consensus real time (the prover must actually run) — warp does not help, same as the fees `catchUpProvenChain` finding from round 3. - **`multi-node/slashing/attested_invalid_proposal`** (now under `slashing/`, not `invalid-attestations/` — base drift). Epoch advances already use cheatcode warps (`advanceToEpoch` / `advanceToNextEpoch`); the remaining waits are real-time committee/attestation/offense-detection rounds (bad proposer must build and broadcast, lazy validator must attest, honest node must detect the offense). - **`multi-node/block-production/multi_validator_node`.** Cost is real-time attestation + checkpoint building via an actual deploy tx; the only clock jump (`advanceToEpoch` past the validator-set lag) is already a cheatcode warp, and this test was just flake-fixed (#24543) so it is left untouched. - **`single-node/proving/optimistic.parallel` (`wait:epoch` tails, pool 2).** Already warp-optimized on this base: every epoch-boundary skip uses `warpToEpochStart` / `waitUntilNextEpochStarts` / `warpToBuildWindowForSlot`. The residual `wait:epoch` is the deliberate ~2-slot real-time tail (the epoch's final checkpoint must land and the sequencer must settle) — pausing across it would suppress that checkpoint. The `waitUntilCheckpointNumber(midCheckpoint)` waits require the sequencer to produce two mid-epoch checkpoints, and `waitUntilProvenCheckpointNumber(..., 240)` rides real prover latency. Nothing left to safely convert. - **`wait:tx-mined` sites (pool 3).** These are inclusion waits for in-flight builds, not waits for a future slot, so they do not qualify (as anticipated in the plan). Not converted. ## Local verification - `yarn build` (full TS project): passes. - `yarn format end-to-end` / `yarn lint end-to-end`: clean. - The converted test is a 4-validator mock-gossip pipelining suite, too heavy to run on the dev machine, so it was not executed locally. CI (`ci-no-fail-fast`) is the verifier; per-commit isolation contains any flake. ## Measured impact The conversion landed as scoped: `warp:sequencers-paused` goes from 1 occurrence run-wide (pipeline_prune, pre-existing from #24475) to 2 — the new one on the converted `proposal_failure_recovery` test. - Converted test ("prune and recover when proposer fails to publish to L1"): total 130.8s → 122.5s (−8.3s), body 114.6s → 109.2s (−5.4s). The added `warp:sequencers-paused` span is 36.1s. - The net saving is modest: the pause+warp+drain+deferred-restart cycle itself costs ~36s, only ~8s less than the real-time prune-boundary wait it replaced — not a full slot. The clean attribution is the span count (1→2); the ~8s test-total delta sits near the multi-node noise floor. - Untouched sites are flat within noise: the sibling orphan-prune test 93.7s → 91.5s, and pipeline_prune 260.4s → 258.7s. This is consistent with the PR's own finding that the warp lever is largely exhausted on this base — one convertible site, a single-test gain. Baseline CI run 1783374468714213 (merge-base e1711d3, full). PR CI run 1783387686257121 (x-fast). ## Notes for the rest of round 4 - No new shared helper was added — the converted site reuses the existing `warpWithSequencersPaused` on `SingleNodeTestContext` (inherited by `MultiNodeTestContext`). PRs 1a/1b/6/5/7 have nothing new to reuse from here. - Finding relevant to the round's premise: on this drifted base the multi-node recovery/slashing suites' large `wait:checkpoint`/`wait:epoch` pools are dominated by real production (checkpoint building, slashing rounds, prover latency), not dead clock time. The warp lever is largely exhausted there; PR 5 (shrink slot times) is the remaining cadence lever for those suites. Fixes A-1183 (cherry picked from commit cfb6c7f)
## What
PR 5 of the round-4 e2e speedup effort: shrink e2e slot times, the
highest-leverage remaining lever
now that the warp lever is exhausted (PR 4) and the fees setup costs are
pure inclusion latency at
CI cadence (PR 6). Every cadence-bound cost — setup txs, inclusion
waits, epoch walks — scales
linearly with what these commits cut.
Config-only. Three commits, in **ascending risk order** so the
CI-survey/fix phase can revert from
the top. A fourth candidate (WIDE_SLOT 72s → 48s) was investigated and
**dropped** — see below.
All guard math is recomputed from `stdlib/src/timetable/budgets.ts` +
`stdlib/src/timetable/proposer_timetable.ts`:
maxBlocksPerCheckpoint = floor((S - init - D - 2P - prepCp) / D) must be
>= 1
where `S` = aztecSlotDuration, `init` = checkpointProposalInitTime (1s,
never clamped), `D` =
blockDurationMs/1000, `P` = p2pPropagationTime, `prepCp` =
checkpointProposalPrepareTime. Fast-profile
clamping (`P -> 0.5`, `prepCp -> 0.5`, `minBlock -> 1`) applies only
when `ethereumSlotDuration < 8`
(`FAST_PROFILE_ETHEREUM_SLOT_DURATION`). At or above 8s the production
budgets apply verbatim.
## Commit 1 (lowest risk) — `perf(e2e): cut default CI L1 block time to
8s`
`DEFAULT_L1_BLOCK_TIME`: `process.env.CI ? 12 : 8` → `8`. Local dev
already ran at 8, so this only
removes a CI-vs-local cadence asymmetry. Default single-node L2 slot
goes 24s → 16s.
Guard (default single-node: D = DEFAULT_BLOCK_DURATION_MS/1000 = 3,
production budgets P=2,
prepCp=1, init=1):
- before (eth 12): S = 2x12 = 24 → floor((24 - 1 - 3 - 4 - 1)/3) =
floor(15/3) = **5** blocks/cp
- after (eth 8): S = 2x8 = 16 → floor((16 - 1 - 3 - 4 - 1)/3) =
floor(7/3) = **2** blocks/cp
Both >= 1. eth=8 is AT the fast-profile boundary, so budgets stay at
production values (no clamping).
**Blast radius is much smaller than the round-4 plan assumed.** The plan
expected this to cut the
fees / cross-chain / bot families, but the base has drifted: those
suites now run on
`PIPELINING_SETUP_OPTS` / `AUTOMINE_E2E_OPTS` (both set
`ethereumSlotDuration: 4` explicitly) and no
longer read `DEFAULT_L1_BLOCK_TIME`. The genuine default-cadence
consumers (eth from
`SingleNodeTestContext.getSlotDurations`, no explicit eth) are four
single-node proving/recovery
suites: `proving/default_node`, `proving/cross_chain_public_message`,
`recovery/sync_after_reorg`,
`partial-proofs/multi_root`. The only other `process.env.CI` timing
coupling
(`multi-node/slashing/inactivity_setup.ts`, `process.env.CI ? 8 : 4`)
sets its own eth slot and does
not read `DEFAULT_L1_BLOCK_TIME`, so it is untouched.
## Commit 2 (medium risk) — `perf(e2e): cut reorg cadence to 24s slots`
`REORG_TIMING_BASE`: aztecSlotDuration 36 → 24, blockDurationMs 8000 →
5000 (epoch stays 4 slots).
Shared by both reorg profiles, both below the fast-profile boundary:
- `FAST_REORG_TIMING` (eth 4): proving/optimistic reorg describes,
l1-reorgs/
- `MULTI_VALIDATOR_REORG_TIMING` (eth 6, attestationPropagationTime
0.5): recovery/proposal_failure_recovery,
recovery/equivocation_recovery, high-availability/ha_sync,
high-availability/ha_checkpoint_handoff
Both run fast-profile: P = 0.5, prepCp = 0.5, init = 1.
Guard, FAST_REORG_TIMING and MULTI_VALIDATOR_REORG_TIMING (identical
budgets):
- before: S = 36, D = 8 → floor((36 - 1 - 8 - 1 - 0.5)/8) =
floor(25.5/8) = **3** blocks/cp
- after: S = 24, D = 5 → floor((24 - 1 - 5 - 1 - 0.5)/5) = floor(16.5/5)
= **3** blocks/cp
blockDurationMs drops 8000 → 5000 specifically to preserve 3
blocks/checkpoint. Leaving it at 8000
would give floor(13.5/8) = **1**, which would break the l1-reorgs
suites' `assertMultipleBlocksPerSlot(2)`
(they push TX_COUNT=8 with maxTxsPerBlock=1 to force multi-block
checkpoints). All other timing scales
off `test.constants.slotDuration` (waits, warps).
equivocation_recovery's manual fit calc still holds
(3 x 5s = 15 + 0.5 init + 5 final block + 2.5 finalization = 23s <=
24s); its comment is updated.
## Commit 3 (higher risk) — `perf(e2e): cut multi-validator
block-production cadence to 24s slots`
`MULTI_VALIDATOR_BLOCK_PRODUCTION_TIMING`: aztecSlotDurationInL1Slots 3
→ 2 (36s → 24s at eth 12),
blockDurationMs 6000 → 4000. eth stays 12 (production budgets).
Consumers: block-production/simple,
first_slot, proof_boundary.
Guard (eth 12, production budgets prepCp=1, init=1; P = per-test
attestationPropagationTime), S=24, D=4:
- simple (P=default 2): floor((24 - 1 - 4 - 4 - 1)/4) = floor(14/4) =
**3** (was 4 at 36/6)
- proof_boundary (P=default 2): floor((24 - 1 - 4 - 4 - 1)/4) = **3**
(was 4)
- first_slot (P=0.5): floor((24 - 1 - 4 - 1 - 1)/4) = floor(17/4) =
**4** (was 4)
All >= 1. simple asserts no block-count (only no-sequencer-failures);
proof_boundary asserts
proof-vs-boundary timing that scales off `test.constants`; first_slot
keeps 4 blocks/checkpoint
(its comment stays accurate). So the drop 4 → 3 is harmless for those
three.
`high_tps` is **pinned to the old 36s/6s cadence at its call site**
(`aztecSlotDurationInL1Slots: 3`,
`blockDurationMs: 6000` in its setupOpts, overriding the shared
profile). Its checkpoint packing is
2 txs x 2.5s = 5s per block, which needs a 6s block sub-slot; a 4s block
cannot hold it, and the suite
hard-asserts `max-checkpoint-length == 4` and `max-txs-per-block == 2`.
Pinning keeps high_tps at its
tuned cadence rather than rewriting its whole timing model. The other
three suites still take the cut.
## Dropped candidate — WIDE_SLOT 72s → 48s
Investigated per the plan's stretch item and **dropped**.
`blob_promotion` hard-asserts a checkpoint
with >= `PIPELINE_EXPECTED_BLOCKS_PER_CHECKPOINT = 8` blocks. At the
profile's D = 5.5s:
- current 72/12: floor((72 - 1 - 5.5 - 4 - 1)/5.5) = floor(60.5/5.5) =
**11** blocks/cp (fits 8)
- 48/12: floor((48 - 1 - 5.5 - 4 - 1)/5.5) = floor(36.5/5.5) = **6**
blocks/cp (cannot fit 8)
Forcing 8 blocks would require shrinking D to ~4.5s, which lands the
guard at exactly 8 with zero
margin — under the deliberate `mockGossipSubNetworkLatency: 500ms` the
suite runs with, and on top of
the profile's documented A-914 constraint (pipelined
multiple-blocks-per-slot starves non-proposer
nodes with `CheckpointNumberNotSequentialError` when the L2 slot is
tightened, worsened by a shorter
slot). This is the "provably can't fit" case, and it cannot be verified
locally (heavy multi-node
pipelining), so it is not shipped. proposed_chain / cross_chain_messages
(only need 2 blocks/cp) would
survive the guard but share the A-914 risk; not worth the exposure for
one profile.
## Local verification
- **Commit 1: verified.** Ran `single-node/proving/default_node.test.ts
-t 'returns initial block data'`
locally (eth=8 = the new CI cadence, which local already uses). The node
came up at
`ethereumSlotDuration: 8, aztecSlotDuration: 16, blockDurationMs: 3000`
and logged
`Sequencer timetable initialized with 2 blocks per slot
{"maxNumberOfBlocks":2}` — exactly the guard
value above. No "Invalid timing configuration", no missed-slot warnings,
test passed. Budgets stayed
at production values (`attestationPropagationTime: 2, minBlockDuration:
2`), confirming eth=8 does not
trip fast-profile clamping.
- **Commits 2 and 3: guard math + CI only.** The reorg and
multi-validator block-production suites are
too heavy for the dev machine (multi-node + real-time waits + proving).
The guard math above is the
verification; CI validates end-to-end. The formula was confirmed exact
against commit 1's live run.
## Revert protocol for the fix phase
Commits are ordered by ascending risk; revert individual commits from
the top on flake rather than
fighting them:
1. **Any block-production suite flake** (simple / first_slot /
proof_boundary) → revert commit 3 first.
proof_boundary is the most real-time-sensitive (it warps to N-3 then
runs the boundary in real time
at the tighter 24s cadence); watch it first.
2. **Any reorg / prune / HA flake** (l1-reorgs, proving/optimistic reorg
cases,
recovery/proposal_failure_recovery, recovery/equivocation_recovery,
high-availability/*) → revert
commit 2. Watch the l1-reorgs `assertMultipleBlocksPerSlot(2)`
assertions and equivocation_recovery's
slashing-round timing.
3. **Any of the four default-cadence single-node proving/recovery
suites** → revert commit 1 (lowest
probability; the cadence is what local already runs).
## Notes for PR 7 (bot suite cadence)
The bot suite (`single-node/bot/bot.test.ts`) uses `setup(0, {
...PIPELINING_SETUP_OPTS, ... })`, which
sets `ethereumSlotDuration: 4` explicitly. It does not read
`DEFAULT_L1_BLOCK_TIME`, so **commit 1 does
not affect the bot suite** — its cadence lever remains its own
PIPELINING preset, unchanged here.
## Measured impact
The cadence cuts land a run-wide −6.4%: −1,600s summed across shards
over the 135 suites present in
both runs (−2,046s of improvements against +446s of regressions). The
wall-clock benefit is smaller,
since shards run in parallel across workers.
Controls confirm the deltas are real cadence effects, not run-type
artifacts. `high_tps` (deliberately
pinned at the old cadence) is unchanged (+0.1s), and the WIDE_SLOT /
explicit-preset proving suites are
flat — pipeline_prune −1.2s, blob_promotion −0.1s, long_proving_time
−1.1s, prune_when_cannot_build
−0.1s. Only the default-cadence and reorg/block-production timings
moved.
Largest per-suite wall cuts (default-cadence proving/recovery from
commit 1; reorg from commit 2):
- optimistic.parallel −399.9s (−23.5%, ×8 shards)
- proof_boundary.parallel −346.0s (−25.6%, ×5 shards)
- full −202.7s; blocks.parallel −196.3s (−28.8%, ×5 shards)
- invalidate_block.parallel −118.5s (×11 shards); sync_after_reorg
−108.3s
- data_withholding_slash −72.0s; proposal_failure_recovery.parallel
−60.1s;
cross_chain_public_message −48.2s; equivocation_recovery −46.9s;
late_prover_tx_collection −43.3s
Regressions, all on high-variance multi-node suites that are not
cadence-cut here (run-to-run noise,
not attributable to this change): multi_root +74.1s, inactivity_slash
+64.4s,
attested_invalid_proposal.parallel +32.1s, proposed_chain.parallel
+26.1s, bot +22.3s.
The metric is whole-suite wall time (sum of test durations + hooks); a
cadence change carries no new
span to attribute against, so these are run-vs-run wall deltas — the
flat pinned/preset controls above
are what make the cadence attribution defensible.
Baseline CI run 1783417125211175 (merge-base 13a53f1, full). PR CI
run 1783426164237985
(x-fast).
Fixes A-1184
(cherry picked from commit eedf4eb)
…24564) ## What / why PR 2 of the round-4 e2e speedup effort. The fees-family suites pay a `setup:bridge` span (~48s ≈ 4 production slots per process) in `FeesTest.applyFPCSetup`, bridging fee juice from L1 to the BananaFPC via the gas portal (`bridgeFromL1ToL2` = prepare-on-L1 + advance 2 blocks + a claim tx). That funding is pure setup plumbing — the fees tests snapshot the FPC's gas balance and assert deltas, they don't assert on the bridge itself. Fee-juice balances are just public-data leaves at `computeFeePayerBalanceLeafSlot(address)`, which `getGenesisValues` (`world-state/src/testing.ts`) already prefills for genesis-funded accounts — the same mechanism that funds the initial accounts and (already, on this base) the sponsored FPC. This PR seeds the BananaFPC's fee juice at genesis instead of bridging it during setup. Note: the plan's premise that `fundSponsoredFPC: true` bridges was stale — the sponsored FPC has been genesis-funded since #19532 (setup pushes its deterministic address into the genesis-funded list). The remaining `setup:bridge` cost in the fees family is the BananaFPC bridge, which this PR removes. ## Mechanism - The BananaFPC address depends on the BananaCoin address and the FPC admin (the first setup account), and the accounts are generated *inside* `setup()` with random secrets — so the FPC address can't be precomputed by the test before genesis. Fixed deploy salts are added for BananaCoin and BananaFPC so both addresses become deterministic once the admin is known, and a `computeExtraGenesisFundedAddresses(defaultAccounts)` hook on `SetupOptions` runs after the accounts are generated and before genesis values are computed. `FeesTest` uses it to derive the BananaFPC address from the first account and add it to the genesis-funded list. - The returned address flows through the existing initial-accounts path (`addressesToFund` → `getGenesisValues`), so it is funded with the same fee juice as an initial account (`10^22`) and automatically included in the L1 `FeeJuicePortal` `feeJuicePortalInitialBalance` (`fundingNeeded`) accounting — the portal's locked L1 balance stays consistent with total L2 fee-juice supply, exactly as for every other genesis-funded balance. - `applyFPCSetup` deploys the FPC with the fixed salt and asserts the deployed address equals the seeded one, so any drift in the deterministic deploy params surfaces as a clear error instead of a downstream "insufficient fee payer balance". The `bridgeFromL1ToL2` call is removed. - The `setup:bridge` span wrapper is kept in place for the bridging that remains, so its disappearance from the fees setup hooks is the measured proof. ## Scope decisions (site by site) - **`FeesTest.applyFPCSetup` BananaFPC funding** → moved to genesis. Used by `account_init`, `private_payments.parallel`, `failures`, `gas_estimation.parallel`. This is the only funding that moved. - **`FeesTest.mintAndBridgeFeeJuice`** (bridges to an arbitrary recipient) → kept. It fires in `account_init` test bodies that test the bridge/claim flow ("pays natively in the Fee Juice after Alice bridges funds") — real behavior under test. - **`account_init` `FeeJuicePaymentMethodWithClaim` / `prepareTokensOnL1`** → kept. Tests the atomic claim-in-deploy flow. - **Sponsored FPC** → already genesis-funded via `fundSponsoredFPC: true` on this base; no change. - **`cross_chain_messaging_test`** (`fundSponsoredFPC: true`) → already genesis-funded; its own cross-chain bridging harness is the behavior under test, kept. - **`bench/client_flows_benchmark`** bridges to its BananaFPC → out of scope (benchmark harness, not in the fees e2e timing family), kept. - Production/sandbox funding paths (cli, `aztec sandbox`) → untouched. The new hook defaults to unset; only the e2e fixtures opt in. ## Expected effect ~48s × the number of fees processes that ran the FPC bridge in setup (account_init, private_payments, failures, gas_estimation). No C++, no new config, no change to production genesis roots. ## Local verification - `single-node/fees/account_init -t 'pays privately through an FPC'` passes; the span leaderboard for the run shows **zero `setup:bridge`** occurrences (was ~48s), while `deploy:fpc`/`deploy:token` remain. The FPC-paying test asserts the FPC's gas balance decreases by the fee, so the genesis funding is exercised end to end, and the deployed-address assertion held. - `single-node/fees/account_init -t 'after Alice bridges funds'` passes with `setup:bridge` present in the test body, confirming the retained bridge/claim harness still works. ## Measured impact The `setup:bridge` span is eliminated across the run: 17 occurrences totalling 646.4s of bridge setup time in the baseline drop to zero in the PR run. The 13 fees-hook bridges (~48s each) are the bulk of it. - Fees beforeHooks per shard: `private_payments.parallel` −51.1s (×8 shards), `gas_estimation.parallel` −48.1s (×3), `failures` −47.8s, `account_init` −47.8s — each matching the ~48s (4-slot) bridge cost. - `setup:bridge` busyMs removed per suite: account_init 48.1s, failures 48.2s, gas_estimation ×3 = 144.6s, private_payments ×8 = 384.7s (13 fees-hook bridges ≈ 625.5s), plus the ~5s sponsored-FPC bridges on `amm` and `storage_proof`. - Total beforeHooks reduction over the suites present in both runs: −907.4s summed across shards (wall-clock benefit is smaller, since the fees shards run in parallel). - One suite moved the wrong way within noise: `fee_juice_payments` +11.0s. It keeps its real bridging/claim txs (no `setup:bridge` span removed), so this is run-to-run variance, not a regression from this change. Baseline CI run 1783374468714213 (merge-base e1711d3, full). PR CI run 1783375878950264 (x-fast — the comparison is per-suite over the suites present in both runs). Fixes A-1405 (cherry picked from commit 1321e98)
Round-4 e2e speedup, PR 1b (adoption half of the AuthRegistry prize). **Stacked on #24567** (`spl/genesis-prefilled-nullifiers`, the genesis-nullifier mechanism). This PR targets that branch; retarget to `merge-train/spartan-v5` once #24567 merges. ## What Make e2e environments start with the standard contracts (AuthRegistry, PublicChecks, HandshakeRegistry) already "published", so `ensureAuthRegistryPublished` and its two siblings skip their two publish txs each. At production cadence `setup:auth-registry` alone was ~16 min/run (2 sequential txs per process, paid per-process across the suite). This collapses that span to sub-second. ## Mechanism (three parts, all default-off outside e2e) - **Genesis nullifiers.** Per standard contract, seed `siloNullifier(ContractClassRegistry, classId)` and `siloNullifier(ContractInstanceRegistry, instanceAddress)` (the real derived address — standard contracts are not magic-address protocol contracts) into the fixtures' `getGenesisValues` call, via PR 1a's new 5th param. These are exactly the nullifiers the publish txs would emit, so the AVM's deployment-nullifier check passes when the contracts are called publicly. A single helper (`getStandardContractGenesisNullifiers`) feeds all four e2e genesis builders (`fixtures/setup.ts`, `e2e_prover_test.ts`, `p2p/p2p_network.ts`, `multi-node/governance/add_rollup.test.ts`) — the latter three recompute a genesis that must reproduce the L1-deployed archive root, so they must seed the identical set. Non-e2e callers (cli, sandbox/local-network) are untouched. - **Archiver preload.** `registerStandardContracts` mirrors `registerProtocolContracts`: it seeds each contract's class (with recomputed `publicBytecodeCommitment`), instance, and public-function signatures into the contract store at block 0, reading the bundled `@aztec/standard-contracts` artifacts. It is idempotent (skips already-registered classes on restart) and gated behind a new archiver config flag `testPreloadStandardContracts` (env `TEST_PRELOAD_STANDARD_CONTRACTS`, default **false**). The flag is set from the e2e node config so every spawned node (validators, prover nodes) picks it up through the normal config path. This adds `@aztec/standard-contracts` as an archiver dependency. - **Guard short-circuit.** The `ensure*Published` helpers are unchanged. Their guards read `wallet.getContractClassMetadata(id).isContractClassPubliclyRegistered` (to `aztecNode.getContractClass`) and `wallet.getContractMetadata(addr).isContractPublished` (to `aztecNode.getContract`) — both are archiver-store reads, not nullifier-tree reads. The store preload alone makes both short-circuit; the genesis nullifiers are what make the contract actually callable in the AVM afterwards. Keeping the helpers doubles as a regression check: if seeding ever breaks, the tx path re-engages and tests still pass, just slow. ## Why the flag is test-only (A-1257 rationale) Preloading unconditionally in production would recreate the A-1257 collision #24254 fixed: a real on-chain publish of a preloaded class would collide with the block-0 preload, because production genesis will **not** carry the matching nullifiers. The flag is only set by the e2e fixtures, which also seed the nullifiers, keeping the store and the nullifier tree consistent. A single source of truth (`getPublishableStandardContracts`) drives both the preloaded set and the seeded-nullifier set so they cannot drift. ## State-write verification Inspected the two publish txs on this base: - `ContractClassRegistry.publish` pushes one nullifier (`classId`, siloed to the class-registry address) and broadcasts a `ContractClassPublished` contract-class log carrying the packed bytecode. No public-data writes. - `ContractInstanceRegistry.publish_for_public_execution` asserts the class-registration nullifier exists, then pushes one nullifier (`address`, siloed to the instance-registry address) and broadcasts a `ContractInstancePublished` private log. No public-data writes. So the full state-write set is two nullifiers + two broadcast logs. The nullifiers are replicated via genesis `prefilledNullifiers`; the archiver normally learns the class/instance from those two logs (`data_store_updater.ts`), and the block-0 preload replaces that path exactly. No `genesisPublicData` is required. PXE-side registration (`wallet.registerContract`) is unaffected and still runs in the helpers. ## Local verification - `automine/accounts/authwit.test.ts` (public-authwit path against the standard AuthRegistry), timing on: passes; `setup:auth-registry` span = **10 ms** (was ~33 s at production cadence). No on-chain AuthRegistry publish (the only `ContractClassRegistry.publish` executions are the test's own AuthWitTest/GenericProxy deploys); the public AuthRegistry contract executes fine against the seeded nullifier. `testPreloadStandardContracts: true` confirmed in the node config dump. - `single-node/fees/account_init.test.ts` (production sequencer + simulated prover node), timing on: 5/5 pass; `setup:auth-registry` span = **10 ms**. Prover node syncs against the seeded genesis (no root divergence), and the flag propagates to it. No duplicate-nullifier errors. - New unit test in `archiver/src/modules/data_store_updater.test.ts`: `registerStandardContracts` preloads each publishable standard contract's class + instance into the store and is idempotent on a second call. ## Expected impact ~14–16 min/run from auth-registry alone, plus more from the two untagged siblings (PublicChecks / HandshakeRegistry) wherever used. ## Measured impact `setup:auth-registry` collapses from 702.2s of setup time run-wide (39 occurrences, 5–35s each) to 482ms total (max 87ms per occurrence) — sub-second run-wide, every shard clean, no seeding or flag-propagation slow path. - The measured beforeHooks reduction is larger than the auth-registry span alone, because the two untagged sibling publishes (PublicChecks, HandshakeRegistry) are removed too. Example: account_init beforeHooks −49.6s vs a 30.9s tagged auth-registry span; the extra ~19s is the two siblings. - Per-suite beforeHooks deltas: account_init −49.6s, fee_settings −49.5s, failures −48.7s, private_payments.parallel −41.8s/shard (×8), gas_estimation.parallel −41.5s/shard (×3), l1_to_l2 −38.1s, l2_to_l1 −37.5s, l1_to_l2_inbox_drift −38.2s, token_bridge −37.7s, fee_juice_payments −37.4s, bot −34.3s; automine/parallel suites shed 5–15s each (their fast-cadence publishes). - Total beforeHooks reduction over the suites present in both runs: −1,649.7s summed across shards (wall-clock benefit is smaller, since shards run in parallel across workers). Baseline CI run 1783389062016888 (#24567 `ci/x-fast`; mechanism-only, so timing-neutral vs merge-base). PR CI run 1783391194852197 (#24568 `ci/x-fast`). ## Notes for downstream - **PR 6 (fees harness):** `applyEnsureAuthRegistryPublished` now short-circuits; the fees setup chain loses the ~32 s/shard auth-registry step, so rebase the overlap/batch shape onto what remains (token deploy, FPC, mints). - **Fix phase / timing capture:** expected span assertion is `setup:auth-registry` around sub-second across the run (10 ms observed locally per process). Any shard still showing tens of seconds means the guards took the slow path (a seeding or flag-propagation bug), not noise. Fixes A-1404 (cherry picked from commit f320049)
Adds `yarn-project/.claude/skills/writing-e2e-tests/SKILL.md`: a skill
that gives an agent everything it needs to place, structure, and write a
robust e2e test with little guidance. Written against the v5-next e2e
layout (category directories with per-category READMEs and context
classes).
## What it covers
- **Step 0 decision ladder**: unit test → new expectation in an existing
test → new `it` in an existing suite → new file on an existing
context/harness → new standalone test.
- **Where to place the test**: recap of the category directories
(`automine/` via `AutomineTestContext`, `single-node/` via
`setupWithProver`/`setupBlockProducer`, `multi-node/` via
`MultiNodeTestContext` on the mock-gossip bus, `p2p/` via
`P2PNetworkTest` on real libp2p, plus `composed/`, `infra/`, `spartan/`,
`bench/`), the multi-node-vs-p2p decision rule, pointers to the
per-category READMEs as the authoritative reference, file naming
(`describe` matches path, header comment), automatic CI registration
through `bootstrap.sh` globs, `.parallel.test.ts` semantics, and
jest/bash timeout sync.
- **Setup reuse**: the three layers (category contexts → domain
harnesses → root `SetupOptions`), the standard suite file shape,
one-environment-per-file, preset spread order, guarded teardown.
- **Readability**: intent-only test bodies, the named-waiter surface
(`fixtures/wait_helpers.ts`, context waiters, `ChainMonitor`), shared
helpers and co-located `*_test_helpers.ts`, simulators with `afterEach`
checks, shared error constants.
- **Speed**: themes distilled from the recent
`test(e2e)`/`perf(e2e)`/`chore(e2e)` speedup PRs and their tracking
issues — avoid unneeded setup, genesis seeding over setup txs,
`BatchCall` batching over `Promise.all` overlap (with the
PXE-serialization caveat), warping dead waits with
`markProvenAndWarp`/`warpWithSequencersPaused` ("an honest wait beats a
flaky warp"), named timing profiles over ad-hoc cadences, and measuring
with the `testSpan` instrumentation + `track-e2e-times` skill before
optimizing.
- **Flakiness**: twelve golden rules synthesized from six months of
deflake PRs and the accumulated flaky-test gotchas (poll-don't-sleep via
named waiters, receipt-anchored assertions, mined ≠ checkpointed ≠
proven, `syncChainTip` tag awareness, pipelining timing margins +
`findSlotsWithProposers`, invariant-not-exact-value assertions, L1
account/nonce and port hygiene, freeze-L1-across-restarts, gossip-mesh
readiness via `runGossipScenario`, fee padding, determinism, honest
timeouts), plus `deflaker.sh` validation and `.test_patterns.yml` as a
last resort.
- Final pre-ship checklist.
All helper/API names cited were verified to exist on
`merge-train/spartan-v5` (`AutomineTestContext.setup`,
`setupWithProver`/`setupBlockProducer`,
`MOCK_GOSSIP_MULTI_VALIDATOR_OPTS`, `wait_helpers.ts` waiters,
`warpWithSequencersPaused`, `markProvenAndWarp`,
`testSpan`/`TEST_TIMING_FILE`, `runGossipScenario`,
`waitForP2PMeshConnectivity`, `findSlotsWithProposers`,
`getPaddedMaxFeesPerGas`, etc.). Genesis-prefill techniques that only
exist in still-open speedup PRs are described as themes to look for
rather than as concrete APIs.
(cherry picked from commit 4c2b6d8)
(cherry picked from commit 23e8e1e)
## What Fixes a low-frequency (~1 in 140) startup-race flake in the `proof_boundary` multi-node e2e suite (CI hash `c06563e7d2b7e067`, group `e2e-p2p-epoch-flakes`). All five scenarios time out in the shared `computeBoundarySlot()` helper at `waitUntilCheckpointNumber(1)` — the 3-validator committee never produces its first checkpoint. ## Why `setupTest` created the three validator nodes sequentially (`asyncMap`) and started each sequencer at node construction. A node subscribes to the gossip topics when its P2P client starts (unconditionally, at node creation), but the in-memory mock gossip bus has no message replay. So when the first-created validator happened to draw the first slot's proposer, it broadcast the inaugural block proposal before the other two committee members had been created and subscribed — the proposal was lost, the 3-of-3 quorum was missed, and the chain forked onto genesis and never produced checkpoint 1. It's randao-dependent (fatal only when the first-created node is the first proposer), hence the low frequency. ## Fix Create the validator nodes with `dontStartSequencer: true`, then `startSequencers(nodes)` once all three exist. All peers subscribe to gossip at creation, so no proposal is broadcast before the whole committee is on the bus. This matches the pattern already used by the non-flaky sibling tests `first_slot.test.ts` and `block-production/setup.ts`. ## Scope Test-only change. Related to A-1419, which tracks the underlying product-level recovery race (competing block-1 re-executions surfacing as `BlockNumberNotSequentialError`). This PR implements that issue's "keep the test startup barrier" acceptance criterion but does **not** resolve A-1419 — the product-side race is unchanged. (cherry picked from commit 54edf7a)
(cherry picked from commit 1d280af)
(cherry picked from commit da9ac1c) CONFLICTS LEFT INTENTIONALLY for follow-up resolution.
) Adapt the upstream NAPI forwarding to the current IPC world-state backend. (cherry picked from commit a4e3a44)
Required for the ported fee-payer phase assertion to exercise the supplied deferred payment method. (cherry picked from commit 019608c)
Use the direct merged-payload simulation path on next, where BatchCall.simulate enters the phase before the first app call.
spalladino
marked this pull request as ready for review
July 23, 2026 21:29
spalladino
approved these changes
Jul 23, 2026
spalladino
enabled auto-merge
July 23, 2026 21:29
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Forward-ports the end-to-end / testing slice of the v5-next → next backlog (work merged to
v5-nextafter the ~2026-07-08 cut that reshapednext).Applied (clean cherry-picks, chronological)
next; not included here)Cherry-pick onto this branch and resolve:
git cherry-pick -x 1d280af721— perf(e2e): overlap and batch setup txs in e2e harnessesgit cherry-pick -x da9ac1c883— feat: assert non revertible phase when setting fee payerPart of the manual v5-next backlog sweep. Draft until owners resolve the conflicts above and CI is green.