perf(e2e): warp dead waits in multi-node recovery and proving tests (#24566)

PR 4 of the round-4 e2e speedup effort: extend the merged
`warpWithSequencersPaused` primitive (from #24475, proven on
`pipeline_prune`) to the remaining production-sequencer-bound dead
waits. One commit per converted site so a CI-hostile one reverts in
isolation.

On the fresh `merge-train/spartan-v5` base, most of the sites the plan
targeted have already been warp-optimized (they use
`warpToBuildWindowForSlot` / `warpToEpochStart` /
`waitUntilNextEpochStarts`, and `pipeline_prune` already uses
`warpWithSequencersPaused`). The residual waits in the remaining sites
ride real-time consensus/production that a warp cannot safely collapse —
warping would skip the checkpoints/attestations/offenses the tests are
verifying. After reading each file on the fresh base, only one
genuinely-dead wall-clock wait was left to convert; the rest are honest
skips (per the round-3 skip rule: an honest skip beats a flaky convert).

## Converted (one commit)

- **`multi-node/recovery/proposal_failure_recovery` — missed-L1-publish
test.** Between "proposed chain reached slotTwo" and the prune, the test
waited in wall-clock (`waitUntilL1Timestamp`) for the L1 clock to roll
past slotOne so the archiver prunes the uncheckpointed slotOne/slotTwo
blocks. The pipelined slotTwo broadcast has already reached every node
and slotThree does not build until slotTwo, so nothing must be produced
in that gap — a genuine dead wait, the direct `pipeline_prune` analog.
Replaced with `warpWithSequencersPaused(..., { restart: false })`: pause
across the warp (warping under a running sequencer would interrupt
in-flight builds), keep the sequencers stopped until the prune is
confirmed so no proposer builds on the unpruned tip, then restart them
for recovery. Expected saving: up to ~1 L2 slot (aztecSlot=36s on this
suite's cadence) per run of that test.

## Skipped, with reasons

- **`proposal_failure_recovery` — orphan-prune test (second `it`).** The
prune wait overlaps P2's concurrent S2 checkpoint rebuild inside the
same slot (pipelining builds S2 during S1 while the orphan is pruned).
Pausing the sequencers to warp would stop the S2 rebuild the test then
asserts on. No pausable dead gap.
- **`multi-node/recovery/equivocation_recovery`.** The dominant waits
are the heal phase (`waitUntilCheckpointNumber` /
`waitForAllNodesToReachCheckpoint` for baseline+2) which requires B/C to
actually produce two checkpoints after A is stopped, plus
`waitForOffenseOnNodes` which requires real slashing-round detection.
Warping skips the very production being verified.
- **`multi-node/slashing/slash_veto_demo`.** `waitForSubmittableRound`
accrues inactivity offenses to quorum over proven blocks, and the
veto/expiry logic rides real slashing-round consensus. This is
proof-submission/consensus real time (the prover must actually run) —
warp does not help, same as the fees `catchUpProvenChain` finding from
round 3.
- **`multi-node/slashing/attested_invalid_proposal`** (now under
`slashing/`, not `invalid-attestations/` — base drift). Epoch advances
already use cheatcode warps (`advanceToEpoch` / `advanceToNextEpoch`);
the remaining waits are real-time
committee/attestation/offense-detection rounds (bad proposer must build
and broadcast, lazy validator must attest, honest node must detect the
offense).
- **`multi-node/block-production/multi_validator_node`.** Cost is
real-time attestation + checkpoint building via an actual deploy tx; the
only clock jump (`advanceToEpoch` past the validator-set lag) is already
a cheatcode warp, and this test was just flake-fixed (#24543) so it is
left untouched.
- **`single-node/proving/optimistic.parallel` (`wait:epoch` tails, pool
2).** Already warp-optimized on this base: every epoch-boundary skip
uses `warpToEpochStart` / `waitUntilNextEpochStarts` /
`warpToBuildWindowForSlot`. The residual `wait:epoch` is the deliberate
~2-slot real-time tail (the epoch's final checkpoint must land and the
sequencer must settle) — pausing across it would suppress that
checkpoint. The `waitUntilCheckpointNumber(midCheckpoint)` waits require
the sequencer to produce two mid-epoch checkpoints, and
`waitUntilProvenCheckpointNumber(..., 240)` rides real prover latency.
Nothing left to safely convert.
- **`wait:tx-mined` sites (pool 3).** These are inclusion waits for
in-flight builds, not waits for a future slot, so they do not qualify
(as anticipated in the plan). Not converted.

## Local verification

- `yarn build` (full TS project): passes.
- `yarn format end-to-end` / `yarn lint end-to-end`: clean.
- The converted test is a 4-validator mock-gossip pipelining suite, too
heavy to run on the dev machine, so it was not executed locally. CI
(`ci-no-fail-fast`) is the verifier; per-commit isolation contains any
flake.

## Measured impact

The conversion landed as scoped: `warp:sequencers-paused` goes from 1
occurrence run-wide
(pipeline_prune, pre-existing from #24475) to 2 — the new one on the
converted
`proposal_failure_recovery` test.

- Converted test ("prune and recover when proposer fails to publish to
L1"): total 130.8s → 122.5s
(−8.3s), body 114.6s → 109.2s (−5.4s). The added
`warp:sequencers-paused` span is 36.1s.
- The net saving is modest: the pause+warp+drain+deferred-restart cycle
itself costs ~36s, only ~8s
less than the real-time prune-boundary wait it replaced — not a full
slot. The clean attribution is
the span count (1→2); the ~8s test-total delta sits near the multi-node
noise floor.
- Untouched sites are flat within noise: the sibling orphan-prune test
93.7s → 91.5s, and
  pipeline_prune 260.4s → 258.7s.

This is consistent with the PR's own finding that the warp lever is
largely exhausted on this base —
one convertible site, a single-test gain.

Baseline CI run 1783374468714213 (merge-base e1711d3, full). PR CI
run 1783387686257121 (x-fast).

## Notes for the rest of round 4

- No new shared helper was added — the converted site reuses the
existing `warpWithSequencersPaused` on `SingleNodeTestContext`
(inherited by `MultiNodeTestContext`). PRs 1a/1b/6/5/7 have nothing new
to reuse from here.
- Finding relevant to the round's premise: on this drifted base the
multi-node recovery/slashing suites' large
`wait:checkpoint`/`wait:epoch` pools are dominated by real production
(checkpoint building, slashing rounds, prover latency), not dead clock
time. The warp lever is largely exhausted there; PR 5 (shrink slot
times) is the remaining cadence lever for those suites.

Fixes A-1183

(cherry picked from commit cfb6c7f)

Author: spalladino

yarn-project/end-to-end/src/multi-node/recovery/proposal_failure_recovery.parallel.test.ts

@@ -2,7 +2,6 @@ import type { Archiver } from '@aztec/archiver';
 import type { AztecNodeService } from '@aztec/aztec-node';
 import { EthAddress } from '@aztec/aztec.js/addresses';
 import type { Logger } from '@aztec/aztec.js/log';
-import { waitUntilL1Timestamp } from '@aztec/ethereum/l1-tx-utils';
 import { asyncMap } from '@aztec/foundation/async-map';
 import { BlockNumber, CheckpointNumber, SlotNumber } from '@aztec/foundation/branded-types';
 import { retryUntil } from '@aztec/foundation/retry';
@@ -188,11 +187,16 @@ describe('multi-node/recovery/proposal_failure_recovery', () => {
     logger.warn(`Waiting for proposed chain to reach slot ${slotTwo} on all nodes (build during slotOne)`);
     await test.waitForAllNodesToReachBlockAtSlot(slotTwo, 'proposed', undefined, { timeout: slotAdvanceTimeout });
 
-    // (3) Wait until slotOne has fully ended on L1 — the archiver only prunes once slotAtNextL1Block > slotOne.
-    // The end-of-slotOne timestamp equals the start-of-slotTwo timestamp.
+    // (3) Collapse the dead gap where the chain just waits for the L1 clock to roll past slotOne so the
+    // archiver prunes the uncheckpointed slotOne/slotTwo blocks (it only prunes once slotAtNextL1Block >
+    // slotOne, and the end-of-slotOne timestamp equals the start-of-slotTwo timestamp). The pipelined slotTwo
+    // broadcast has already reached every node (step 2) and slotThree does not build until slotTwo, so nothing
+    // needs to be produced in this window. The sequencers are paused across the warp — warping under a running
+    // sequencer would interrupt in-flight builds — and kept stopped (restart: false) until the prune is
+    // confirmed, so no proposer builds against the still-unpruned tip; they are restarted for recovery below.
     const slotOneEndTimestamp = getTimestampForSlot(slotTwo, test.constants);
-    logger.warn(`Waiting until L1 timestamp ${slotOneEndTimestamp} (end of slot ${slotOne})`);
-    await waitUntilL1Timestamp(test.l1Client, slotOneEndTimestamp, undefined, test.L2_SLOT_DURATION_IN_S * 3);
+    logger.warn(`Warping past the end of slot ${slotOne} (L1 timestamp ${slotOneEndTimestamp}) to trigger the prune`);
+    await test.warpWithSequencersPaused(nodes, test.context.cheatCodes, slotOneEndTimestamp, { restart: false });
 
     // (4) After slotOne ends without a checkpoint, all nodes should prune.
     // Verify rollback via the prune event itself: the pruned slot must equal slotOne, and the
@@ -216,9 +220,13 @@ describe('multi-node/recovery/proposal_failure_recovery', () => {
       expect(prunedSlots).toContain(slotOne);
     }
 
-    // (5) Allow the formerly suppressed node to publish again so the chain can recover.
+    // (5) Allow the formerly suppressed node to publish again, then restart the paused sequencers so the
+    // chain can build the recovery checkpoint. Restarting only now (after the prune is confirmed) keeps any
+    // proposer from building on the still-unpruned tip.
     logger.warn(`Re-enabling checkpoint publishing on node ${proposerOneNodeIndex}`);
     await nodes[proposerOneNodeIndex].setConfig({ skipPublishingCheckpointsPercent: 0 });
+    await test.startSequencers(nodes);
+    logger.warn('Restarted all sequencers for recovery');
 
     // (6) During slotTwo: the pipelined proposer for slotThree builds and broadcasts → proposed advances again.
     // The chain must have rewound past slotOne and slotTwo and now build on whatever was