test(e2e): instrument and diagnose bot suite setup cost#24534
Merged
Conversation
spalladino
added
the
ci-no-fail-fast
Adds timing spans so the untagged residual of the bot suite's file-level beforeAll (~178s on CI, 97% invisible) can be decomposed and the ~32s/shard proven-chain residual in the private_payments/failures fees suites becomes visible. Bot suite (single-node/bot/bot.test.ts): - setup:wallet around EmbeddedWallet.create (stands up the ephemeral PXE) - wallet:create around createSchnorrInitializerlessAccount (same concept as createFundedInitializerlessAccounts, already tagged wallet:create) - setup:bot around the nested describes' Bot.create/AmmBot.create/ CrossChainBot.create/BotStore creation, which the suite-line accounting folds into the same suite beforeHooksMs as the file hook Fees harness (single-node/fees): - wait:proven-checkpoint around catchUpProvenChain's poll loop - warp:proven-checkpoint-epoch around advanceToNextEpoch, via a new instrumented FeesTest.advanceToNextEpoch() wrapper that waitForEpochProven and the direct call sites in failures/private_payments now route through Reuses existing tags where the concept matches rather than inventing near-duplicates. Spans are passthrough when TEST_TIMING_FILE is unset, so this is zero behavior change.
spalladino
force-pushed
the
spl/e2e-bot-setup-spans
branch
from
July 6, 2026 21:50
1f7898c to
46074c2
Compare
PhilWindle
approved these changes
Jul 8, 2026
PhilWindle
pushed a commit
that referenced
this pull request
Jul 8, 2026
Parallelizes the independent setup steps in the bot factory (`@aztec/bot`, `bot/src/factory.ts`). This is production bot code that runs against live networks, so the changes are strictly behavior-preserving: same contracts at the same (salt-derived) addresses, same amounts, same final state. Only the ordering/concurrency of provably-independent setup steps changes. The bot factory is the single largest cost in the `single-node/bot` e2e suite: each `*.create` runs the production sequencer at 12s L2 slots (`PIPELINING_SETUP_OPTS`) and deploys/mints one tx per slot, fully serial. `AmmBot.create` alone was ~74s (7 serial txs). ## Dependency graph per bot type Edges below are "must be mined before", verified against the Noir contract source (`token_contract`, `amm_contract`). ### Transaction bot -- `setup()` (unchanged) `setupAccount` -> register recipient -> `ensureFeeJuiceBalance` -> deploy token -> mint. This chain is fully data-dependent: funding gates the deploy, the deploy gates the mint. The recipient (`createSchnorrAccount`) is register-only (no tx), and the private+public mints are already batched into one tx. Nothing is independent, so `setup()` is left serial. ### AMM bot -- `setupAmm()` Steps: deploy token0 (D0), token1 (D1), LP token (DL), AMM (DA); `set_minter` granting the AMM rights over the LP token (SM); mint token0+token1 to the provider (M, one batched tx); `add_liquidity` (AL). - D0, D1, DL: independent -- distinct contracts, no cross-references. - DA: the AMM constructor only stores the three token addresses (no calls into the tokens). Those addresses are salt-derived, so DA needs only the (pre-derived) addresses, not the token deploys mined. - SM: `Token::set_minter` just writes `minters[amm] = true`; it does not call or validate the AMM contract. It needs the LP token deployed and the (derivable) AMM address only -- not the AMM deployed. The deployer is authorized because the Token constructor sets `minters[admin] = true`. - M: `mint_to_private`/`mint_to_public` require the caller be a minter; the deployer is admin=minter from the constructor, so the token0/token1 mints need only those tokens deployed (no `set_minter`). - AL: `add_liquidity` transfers token0/token1 from the provider (needs M for balances), mints LP tokens (needs SM so the AMM is an LP minter), and targets the AMM (needs DA). Restructured from 7 serial txs into 3 slot-phases: - Phase 1: `Promise.all([D0, D1, DL])` - Phase 2: `Promise.all([DA, SM, M])` - Phase 3: `AL` ### Cross-chain bot -- `setupCrossChain()` Steps: deploy TestContract (an L2 tx), seed N L1->L2 messages (L1 txs), wait for the first message ready. - The seeds reference only the L2 recipient (TestContract) address, which is salt-derived. L1->L2 messages are queued on L1 and do not require the L2 contract to exist yet; they are consumed later in `bot.run()`, after setup completes. So the L2 deploy and the L1 seeding are independent. - The L2 deploy pays via the L2 wallet/PXE account; the seeds use the bot's L1 client -- different accounts, so there is no L1 nonce interaction between them. Restructured to overlap the deploy with the seed loop: `Promise.all([deploy TestContract, seed loop])`, then the message-ready wait. ## What stayed serial, and why - The whole transaction-bot `setup()` (data-dependent chain, nothing independent). - `ensureFeeJuiceBalance` before every deploy (funding must precede spending). - `add_liquidity` after its mints + minter grant + AMM deploy. - The individual L1->L2 seeds inside the loop: they share the bot's single L1 account, so concurrent sends would race on the L1 nonce. The suite already documents this nonce hazard. Only the loop as a whole overlaps the (L2) TestContract deploy. ## Production safety / idempotent re-entry - Every deploy still goes through `registerOrDeployContract`, which checks `getContractMetadata(address).isContractPublished` per contract and only registers (no tx) if already deployed. If one parallel deploy fails and the others succeed, the next `*.create` recovers: the succeeded contracts register-only, the failed one redeploys. Addresses are salt-derived and identical across runs. - `set_minter` is idempotent (writes `true` again). The AMM path's mint and `add_liquidity` always run (no balance guard) -- unchanged from the previous `fundAmm`. - Same-sender concurrent `.send()`s are safe by construction: the PXE serializes simulate/prove through its `SerialQueue` and setup txs pay with random tx nonces. The bot runs the same `EmbeddedWallet` -> PXE path in the e2e suite and in production. - Deliberate, benign failure-path change: because Phase 2 runs DA/SM/M concurrently, a failure of one no longer prevents the others from being sent. Their effects (a deployed-but-unused AMM, a minter grant, an extra mint) are all idempotent-safe, and `add_liquidity` uses fixed amounts, so re-entry converges to the same final state. - `withNoMinTxsPerBlock` (the `flushSetupTransactions` save/zero/restore wrapper around each setup tx) was not safe to re-enter concurrently: interleaved save/zero/restore could read the already-zeroed value and "restore" `minTxsPerBlock` to 0 permanently. It now reference-counts entrants -- the first saves and zeroes, the last restores -- with unit tests covering the overlapping and failure paths (`bot/src/factory.test.ts`). Serial callers see the exact same RPC sequence as before. Final state is identical in every path: same contract addresses, same minter grant, same minted amounts, same liquidity. ## Local evidence Two full local runs of `single-node/bot/bot.test.ts` (production sequencer, 12s L2 slots), all 10 tests passing in both. Hook durations from factory log timestamps, against the round-4 local baseline measured on the same machine at the same cadence (findings from #24534): - `Bot.create` (transaction-bot hook, untouched): 31.8s vs 32.2s baseline -- unchanged, as intended. - `AmmBot.create`: 57.2s vs 73.9s baseline (-16.7s). The three token deploy txs go out within 300ms of each other; the AMM deploy, `set_minter` and the mint batch all go out within the following slot. - `CrossChainBot.create`: 24.0s vs 43.5s baseline (-19.5s). The TestContract deploy overlaps both L1 seeds; the first message is ready almost immediately after the deploy is mined. - Suite-level `beforeHooksMs`: 116.9s vs 153.7s baseline (-36.8s), matching the per-hook deltas. - Unit: `bot/src/factory.test.ts` (4 tests) covering the `withNoMinTxsPerBlock` reentrancy fix. ## Expected effect on CI - `AmmBot.create`: 7 serial txs -> 3 slot-phases (~87s on CI per #24534's spans -> ~55-60s). - `CrossChainBot.create`: deploy overlaps the seed pipeline (~43s -> ~25s). - Transaction bot: unchanged. - Aggregate: roughly -35 to -45s on the bot suite's beforeAll wall time. - Fix-phase proof metric (once #24534's `setup:bot` instrumentation is on this base): the amm-bot and cross-chain `setup:bot` occurrences drop per the numbers above. Measured CI numbers to be appended when a full run lands. ## Measured impact The bot suite (a single shard) drops 216.3s → 118.3s total (**−98.0s, −45%**); beforeHooks 215.9s → 118.0s. - No `setup:bot` spans exist on this base (that instrumentation lives in the unmerged #24534), so the metric is the suite-level hook fold, which covers all three nested bot factory setups (Bot, AmmBot, CrossChainBot). - Noise context from the same run pair: untouched light suites move −2 to −3s per shard (private_initialization −2.9s/shard over 20 shards, deploy_method −2.1s/shard over 14), and the largest counter-move is validators_sentinel +24s (multi-node variance). A −98s single-shard delta is far outside that envelope. - The CI delta exceeds the local −36.8s because the CI baseline pays more missed-slot penalties per serial tx (baseline beforeHooks 215.9s on CI vs 153.7s locally at the same 12s cadence); collapsing 7 serial txs into 3 slot-phases removes proportionally more where slots are more often missed. Baseline CI run 1783393028773172 (merge-base 30966a4, full). PR CI run 1783427250198215 (x-fast). Fixes A-1408
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What / why
Round-4 e2e speedup, PR 3 of the effort. The
single-node/botsuite's file-levelbeforeAllcosts~178s on CI with 97% untagged, and the residual proven-chain waits in the
private_payments/failuresfees suites (~32s/shard) were also invisible. This PR instruments both so the cost is attributable, and
diagnoses the bot residual.
Spans are pure passthrough when
TEST_TIMING_FILEis unset (seefixtures/timing.ts), so this is zerobehavior change for normal runs and CI.
Commit
test(e2e): instrument bot suite setup and fees proven-chain residualssingle-node/bot/bot.test.ts):setup:walletaroundEmbeddedWallet.create,wallet:createaroundcreateSchnorrInitializerlessAccount, andsetup:botaround the four nesteddescribebeforeAll hooks (Bot.create/AmmBot.create/CrossChainBot.create/BotStore).single-node/fees):wait:proven-checkpointaroundcatchUpProvenChain's poll loop,and
warp:proven-checkpoint-epocharoundadvanceToNextEpochvia a new instrumentedFeesTest.advanceToNextEpoch()wrapper;waitForEpochProvenand the direct call sites infailures/private_paymentsnow route through it.wallet:createis already whatcreateFundedInitializerlessAccountsfires;warp:proven-checkpoint-epochis already whatadvanceToNextEpochfires inblock-production/setup.ts) rather than minting near-duplicates.Diagnosis
The timing environment (
shared/timing_env.mjs) folds everybeforeAllin a file — the file-levelone plus each nested
describe's — into a single suite-scopedbeforeHooksMs, and every span firedduring any beforeAll lands on that same suite line. The shortlist read the whole number as the
file-level hook and, seeing only
setup:env:*tagged, attributed the rest to the one visible untaggedfile-level call (
createSchnorrInitializerlessAccount). It is actually the nested hooks.Local read (
TEST_TIMING_SPANS=1,PIPELINING_SETUP_OPTS= 12s L2 slots, production sequencer, fakeprover), suite
beforeHooksMs = 153,733 ms:setup:env:*(file-levelsetup()): ~4.6s — already taggedsetup:wallet(EmbeddedWallet.create, ephemeral): 137 ms — the PXE hasautoSync:false, nogenesis walk
wallet:create(createSchnorrInitializerlessAccount): 69 ms — initializerless, no deploy tx, asdesigned
setup:bot(4 nested hooks): 149,582 ms = 97.3% of the file's beforeAllBot.create: 32.2s (token deploy class+instance, then private+public mint)new BotStore(...): 0.9 msAmmBot.create: 73.9s (deploys 4 contracts + set_minter + mint + add_liquidity)CrossChainBot.create: 43.5s (TestContract deploy + seed 2 L1→L2 msgs + wait ready)Root cause: structural, not a bug. The bot suite runs the production Sequencer to exercise
CHECKPOINTED/PROPOSED follow modes and L1 fee-juice bridging, so each
Bot.createdeploys realcontracts and mints serially at slot cadence (
bot/src/factory.ts). The two calls the shortlistsuspected are ~0.1s each.
No fix is shipped here because there is no safe one-liner: speeding this up means batching/parallelizing
the
bot/src/factory.tsdeploy/mint paths (production code, PR-6-shaped), genesis-seeding (PR 1/2), orcutting the CI slot time (PR 5) — none belong in the instrumentation PR. Tracked as a round-5 item; the
new
setup:botspan makes that win measurable. Full write-up intmp/SPEEDUP_ROUND4_FINDINGS.md.Expected effect
No wall-clock change (passthrough spans). The deliverable is visibility: the previously-invisible
setup:bot/setup:wallet/wallet:createdecomposition of the bot beforeAll, and thewait:proven-checkpoint/warp:proven-checkpoint-epochresidual in the fees suites.Measured numbers from a full green CI run will be appended below once available.
Measured impact
Both runs are full CI runs. PR run:
ci/x-faston head1f7898cd2b(CI 1783345614459429, 1,799 testlines). Baseline:
ci/x-full-no-test-cacheon the exact merge-basea8cfa93df5(CI 1783341726388943,1,863 test lines). No wall-clock change claimed (spans are passthrough; suite-level noise floor is
~11s median / ~52s p90) — the deliverable is attribution, measured as span
busyMs.Bot suite (
suiteline, beforeAll = 181.5s on the PR run vs 182.1s baseline):setup:env:*only) → 174.8s / 96% invisible.setup:bot174,203 ms (4 occurrences, max 86,961 ms = the amm-bot hook),setup:wallet250 ms,wallet:create144 ms; residual now −1.5s ≈ 0 →100% of the beforeAll is attributed.
EmbeddedWallet.create+createSchnorrInitializerlessAccountare~0.4s combined; the "invisible 178s" is the four nested describes' bot-factory setup
(deploys + mints at production 12s-slot cadence), folded into the file's suite line by the
hook accounting.
Fees suites (
private_payments.parallel× 8 shards +failures):wait:proven-checkpoint/warp:proven-checkpoint-epochlines on these suites).wait:proven-checkpointfires on all 9 suite hooks, 293,888 ms total(avg 32.7s/shard — matches the residual exactly), plus one in-test occurrence (32.1s);
warp:proven-checkpoint-epoch19 occurrences, 361 ms total (the warp RPC itself isnear-free — the cost is entirely the proven-chain poll that follows). Per-shard residual is now
≈ −1s, i.e. fully attributed.
Net: ~469s/run of previously invisible setup/wait cost is now tagged and attributable
(174.2s
setup:bot+ 0.4s wallet spans + 294.3s fees proven-chain waits), with no behavior orwall-clock change.
Fixes A-1407