Commit 356e036
authored
perf(e2e): warp proven-checkpoint waits, tighten in-process polling, and instrument setup spans (#24452)
## Motivation
Span-instrumented full CI runs (since #24407 landed) now show where the
remaining e2e wall-clock goes: a large uninstrumented setup layer, a
cluster of proven-checkpoint/epoch waits that burn real time while
nothing is being built, and a 1s in-process poll cadence that adds
~0.5-0.9s of dead sleep to nearly every awaited tx. This PR takes the
low-risk warp folds and polling wins that are safe today, and adds span
instrumentation to the setup layer so the next round can attack the
~5,000s of currently-invisible beforeAll work. Everything multi-node is
validated on CI (hence `ci-no-fail-fast`); the last commit is a
deliberate slot-cut experiment kept isolated so it can be reverted on
its own.
## Changes
- **poll in-process nodes at 250ms instead of 1s** — TestWallet now
defaults its `send().wait()` poll interval to 0.25s (in-process nodes
reach CHECKPOINTED synchronously under automine and cheaply otherwise),
with the spartan/worker sites explicitly restored to the 1s default
since they talk to remote JSON-RPC nodes. The e2e-local wait helpers
(`wait_helpers.ts`, `waitForProvenChain`, gas-portal block advance,
L1->L2 message poll) drop from 1s to 0.25s. Removes ~0.5-0.9s of dead
sleep per awaited tx across ~600 tests. Production aztec.js/wallet-sdk
defaults are untouched.
- **warp past the epoch boundary in waitForProvenCheckpoint** — after
the multi-node block-production fixture stops its sequencers, warp the
L1 clock one epoch forward (forward-only, skipped if already proven) so
the epoch closes and the fake prover can prove+submit without waiting
the epoch out in real time. Targets ~716s of suite-summed
`wait:proven-checkpoint` across proposed_chain (~430s),
deploy_and_call_ordering (~143s), cross_chain_messages (~143s), plus
blob_promotion.
- **warp epoch waits in multi_proof and upload_failed_proof** — replace
`waitUntilEpochStarts` with `warpToEpochStart` in these two 12s-slot
proving tests (both passed under warp in the round-1 CI sweep).
proof_fails is deliberately left untouched.
- **register-only TestContract in automine pxe test** — the test only
calls the noinitcheck private `emit_nullifier`, so register the contract
instead of deploying it, dropping a deployment tx and its checkpoint
cycle.
- **compute genesis values on an ephemeral world state** —
`generateGenesisValues` used a full fsync-on
`NativeWorldStateService.tmp` per e2e container just to read one tree
root; switch to the fsync-off `ephemeral` backend. Adds a unit test
asserting `tmp` and `ephemeral` produce identical archive roots for a
funded-accounts genesis with non-zero timestamp (this path is
consensus-critical — CLI deploy paths compute the on-chain genesis root
through it).
- **instrument setup-layer deploys and mints with spans** — wrap the
top-offender setup helpers (fees harness token/FPC deploys + mints +
fee-juice bridge, cross-chain token/bridge deploys + mints, shared
auth-registry publish, gas-portal bridge) in `testSpan` under the #24407
taxonomy (`deploy:*`, `tx:mint`, `setup:bridge`, `setup:auth-registry`).
Zero behavior change (testSpan is a passthrough without
`TEST_TIMING_FILE`); this is the data source for round 4's attack on the
~5,000s of uninstrumented beforeAll work.
- **cut multi_validator_node slots 36s -> 16s** (experiment, last
commit) — lower `aztecSlotDuration` 36->16 and `blockDurationMs`
6000->2000 together (eth stays 8). Small expected saving; kept as the
final isolated commit so it can be reverted alone if CI shows
committee/attestation trouble on this file.
One planned item was dropped: an opt-in warp for
`ChainMonitor.waitUntilL2Slot`. All three candidate call sites turned
out to cover deliberate real-time building (live-sequencer coordination,
inactivity accumulation across an epoch, and the proof-boundary critical
window), so the opt-in API would have had no safe callers.
## Verification
Locally: full `yarn build`, `yarn format --check`, and `yarn lint` on
the touched packages all pass. The new world-state genesis-equivalence
unit test passes (`tmp` and `ephemeral` roots identical). The
`automine/pxe.test.ts` e2e passes as a smoke test for the register-only
change and the 250ms polling. Everything multi-node (the warp folds, the
timing cut) is validated on CI. Note the final commit is a deliberate
slot-cut experiment that can be reverted in isolation.
## Measured impact
Full green CI run of this PR (`9b4cc967`, CI `1782961631439529`) vs the
base-proxy full run (`d160265b`, CI `1782938936852228` — the branch
point plus one unrelated one-file test change). Identical test
populations: 2051 rows, all passed, in both runs. Sums are across
parallel processes, not wall-clock (methodology in the Linear "Times
tracking" doc).
| Bucket | Base | This PR | Δ |
|---|---|---|---|
| Overall | 7h 26m 03s | 6h 36m 20s | **−49m 43s (−11.1%)** |
| Setup (before-hooks) | 2h 14m 31s | 1h 45m 44s | −21.4% |
| — of which setup.ts | 42m 53s | 34m 13s | −20.2% |
| Body | 5h 05m 14s | 4h 45m 03s | −6.6% |
| Teardown | 6m 17s | 5m 33s | −11.6% |
By mechanism:
- **Proven-checkpoint warp**: `wait:proven-checkpoint` 14m 12s → 3m 32s
(**−75%**), worst single wait 215s → 79s. Suite deltas match span deltas
~1:1 — proposed_chain −67%, deploy_and_call_ordering −45%,
cross_chain_messages −21%. Hard attribution.
- **Epoch warps**: multi_proof −52% (wait:epoch 92s → 40s).
upload_failed_proof's warp also worked (35s → 20s) but was masked in the
suite total by one-shot proving noise in that run.
- **250ms polling + ephemeral genesis**: ~30–35 min spread across the
whole suite — 141 of 152 suites improved; every `setup:env:*` span
dropped ~16–50% with identical counts (same work, less waiting).
- **Slot-cut experiment**: multi_validator_node 103s → 84s (−18.6%) and
passed CI — the experiment survives.
- **Register-only pxe test**: 18.2s → 10.1s.
- **New setup spans**: ~44 min/run of previously untagged setup time is
now attributable (`setup:auth-registry` 16m, `setup:bridge` 10m,
`tx:mint` 8m, `deploy:token` 7m, `deploy:fpc` 2m) — the target list for
round 4.
11 suites regressed (~8 min total vs ~58 min of improvements), all in
untagged real-time slashing/proving bodies whose tagged waits are
unchanged — consistent with run-to-run variance, not PR effects
(per-suite noise floor between same-day runs is median ~11s / p90 ~52s).1 parent f397aa5 commit 356e036
17 files changed
Lines changed: 183 additions & 58 deletions
File tree
- yarn-project
- end-to-end/src
- automine
- fixtures
- multi-node/block-production
- single-node
- fees
- proving
- spartan
- test-wallet
- wallet-sdk/src/base-wallet
- world-state/src
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
| 21 | + | |
21 | 22 | | |
22 | 23 | | |
23 | 24 | | |
24 | 25 | | |
25 | | - | |
26 | | - | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
27 | 30 | | |
28 | 31 | | |
29 | 32 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
821 | 821 | | |
822 | 822 | | |
823 | 823 | | |
824 | | - | |
| 824 | + | |
| 825 | + | |
| 826 | + | |
| 827 | + | |
| 828 | + | |
| 829 | + | |
825 | 830 | | |
826 | 831 | | |
827 | 832 | | |
| |||
923 | 928 | | |
924 | 929 | | |
925 | 930 | | |
926 | | - | |
927 | | - | |
928 | | - | |
929 | | - | |
930 | | - | |
931 | | - | |
932 | | - | |
933 | | - | |
| 931 | + | |
| 932 | + | |
| 933 | + | |
| 934 | + | |
| 935 | + | |
| 936 | + | |
| 937 | + | |
| 938 | + | |
| 939 | + | |
| 940 | + | |
934 | 941 | | |
935 | 942 | | |
936 | 943 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
21 | 21 | | |
22 | 22 | | |
23 | 23 | | |
24 | | - | |
| 24 | + | |
25 | 25 | | |
26 | 26 | | |
27 | 27 | | |
| |||
45 | 45 | | |
46 | 46 | | |
47 | 47 | | |
48 | | - | |
| 48 | + | |
49 | 49 | | |
50 | 50 | | |
51 | 51 | | |
| |||
118 | 118 | | |
119 | 119 | | |
120 | 120 | | |
121 | | - | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
122 | 125 | | |
123 | 126 | | |
124 | 127 | | |
| |||
129 | 132 | | |
130 | 133 | | |
131 | 134 | | |
132 | | - | |
| 135 | + | |
133 | 136 | | |
134 | 137 | | |
135 | 138 | | |
| |||
153 | 156 | | |
154 | 157 | | |
155 | 158 | | |
156 | | - | |
| 159 | + | |
157 | 160 | | |
158 | 161 | | |
159 | 162 | | |
| |||
174 | 177 | | |
175 | 178 | | |
176 | 179 | | |
177 | | - | |
| 180 | + | |
178 | 181 | | |
179 | 182 | | |
180 | 183 | | |
| |||
183 | 186 | | |
184 | 187 | | |
185 | 188 | | |
186 | | - | |
| 189 | + | |
187 | 190 | | |
188 | 191 | | |
189 | 192 | | |
| |||
207 | 210 | | |
208 | 211 | | |
209 | 212 | | |
210 | | - | |
| 213 | + | |
211 | 214 | | |
212 | 215 | | |
213 | 216 | | |
| |||
231 | 234 | | |
232 | 235 | | |
233 | 236 | | |
234 | | - | |
| 237 | + | |
235 | 238 | | |
236 | 239 | | |
237 | 240 | | |
| |||
256 | 259 | | |
257 | 260 | | |
258 | 261 | | |
259 | | - | |
| 262 | + | |
260 | 263 | | |
261 | 264 | | |
262 | 265 | | |
| |||
Lines changed: 3 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
27 | 27 | | |
28 | 28 | | |
29 | 29 | | |
30 | | - | |
| 30 | + | |
31 | 31 | | |
32 | 32 | | |
33 | 33 | | |
| |||
51 | 51 | | |
52 | 52 | | |
53 | 53 | | |
54 | | - | |
| 54 | + | |
55 | 55 | | |
56 | 56 | | |
57 | | - | |
| 57 | + | |
58 | 58 | | |
59 | 59 | | |
60 | 60 | | |
| |||
Lines changed: 12 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
| 27 | + | |
27 | 28 | | |
28 | 29 | | |
29 | 30 | | |
| |||
209 | 210 | | |
210 | 211 | | |
211 | 212 | | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
212 | 224 | | |
213 | 225 | | |
214 | 226 | | |
| |||
Lines changed: 15 additions & 8 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
| 27 | + | |
27 | 28 | | |
28 | 29 | | |
29 | 30 | | |
| |||
76 | 77 | | |
77 | 78 | | |
78 | 79 | | |
79 | | - | |
80 | | - | |
81 | | - | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
82 | 85 | | |
83 | 86 | | |
84 | | - | |
85 | | - | |
86 | | - | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
87 | 92 | | |
88 | 93 | | |
89 | 94 | | |
| |||
241 | 246 | | |
242 | 247 | | |
243 | 248 | | |
244 | | - | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
245 | 252 | | |
246 | 253 | | |
247 | 254 | | |
248 | | - | |
| 255 | + | |
249 | 256 | | |
250 | 257 | | |
251 | 258 | | |
| |||
Lines changed: 13 additions & 5 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
10 | 10 | | |
11 | 11 | | |
12 | 12 | | |
| 13 | + | |
13 | 14 | | |
14 | 15 | | |
15 | 16 | | |
| |||
159 | 160 | | |
160 | 161 | | |
161 | 162 | | |
162 | | - | |
163 | | - | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
164 | 166 | | |
165 | | - | |
166 | | - | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
167 | 170 | | |
168 | 171 | | |
169 | 172 | | |
| |||
181 | 184 | | |
182 | 185 | | |
183 | 186 | | |
184 | | - | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
185 | 193 | | |
186 | 194 | | |
187 | 195 | | |
| |||
Lines changed: 1 addition & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
21 | | - | |
| 21 | + | |
22 | 22 | | |
23 | 23 | | |
Lines changed: 21 additions & 11 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
| 27 | + | |
27 | 28 | | |
28 | 29 | | |
29 | 30 | | |
| |||
158 | 159 | | |
159 | 160 | | |
160 | 161 | | |
161 | | - | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
162 | 165 | | |
163 | 166 | | |
164 | 167 | | |
| |||
167 | 170 | | |
168 | 171 | | |
169 | 172 | | |
170 | | - | |
| 173 | + | |
171 | 174 | | |
172 | 175 | | |
173 | 176 | | |
| |||
236 | 239 | | |
237 | 240 | | |
238 | 241 | | |
239 | | - | |
240 | | - | |
241 | | - | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
242 | 247 | | |
243 | 248 | | |
244 | 249 | | |
| |||
257 | 262 | | |
258 | 263 | | |
259 | 264 | | |
260 | | - | |
261 | | - | |
262 | | - | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
263 | 270 | | |
264 | 271 | | |
265 | 272 | | |
| 273 | + | |
266 | 274 | | |
267 | 275 | | |
268 | 276 | | |
| |||
345 | 353 | | |
346 | 354 | | |
347 | 355 | | |
348 | | - | |
349 | | - | |
350 | | - | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
| 359 | + | |
| 360 | + | |
351 | 361 | | |
352 | 362 | | |
353 | 363 | | |
| |||
Lines changed: 3 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
80 | 80 | | |
81 | 81 | | |
82 | 82 | | |
83 | | - | |
84 | | - | |
| 83 | + | |
| 84 | + | |
85 | 85 | | |
86 | 86 | | |
87 | 87 | | |
88 | | - | |
| 88 | + | |
89 | 89 | | |
90 | 90 | | |
91 | 91 | | |
| |||
0 commit comments