fix: deflake HA full e2e suite by switching to in-proc interval-mining anvil (#23979)

Fixes the flaky HA full suite (`e2e_ha_full`) seen in
http://ci.aztec-labs.com/8e1e980c4886df0d, where "should distribute work
across multiple HA nodes" timed out awaiting a trigger tx. Also
re-enables the suite, which #23976 had skipped.

## Root cause

The HA compose suite was the only block-building suite running against
an L1 with no self-advancing clock. Its anvil container ran in automine
with no `--block-time`, and being external, it was excluded from the
`TestDateProvider` sync that locally-spawned anvils get. L1 chain time
only moved when something mined, while the shared sequencer clock
free-ran. #23821 removed the `AnvilTestWatcher` that used to couple the
two clocks in this mode and replaced it with per-iteration nudges in the
test (clock warp + blind `mine(8)`).

Two consequences, both visible in the failed run's logs:

- The `mine(8)` overshoot put L1 ~1.5 slots ahead of the test clock, so
each iteration's first propose raced its slot boundary and was silently
dropped, followed by a prune that destroyed the pipelined builders'
forks (`Fork not found` on all surviving nodes). This race was lost in
passing runs too.
- Recovery then required the proposers' archiver-sync gate to clear, but
the gate's deadline runs on the free-running test clock while nothing
mines L1 during the test's `waitForTx` — `Archiver did not sync L1 past
slot 109 before slot 110 expired, discarding pipelined work`, repeated
until the jest timeout. Whether a run passed or failed came down to
seconds of margin on this gate.

## Fix

Stop emulating L1 time in the test and run the suite in the same regime
as every other block-building e2e (e.g. `e2e_epochs`):

- Drop the anvil container and `ETHEREUM_HOSTS` from the HA compose
file. With no external L1 configured, `setup()` spawns anvil in-proc
with interval mining (`--block-time = ethereumSlotDuration`) and keeps
the `TestDateProvider` snapped to L1 block timestamps via the existing
stdout listener. The sibling web3signer compose suite already works this
way.
- Add `automineL1Setup: true` so L1 contract deployment runs under
temporary automine before interval mining starts.
- Delete all time scaffolding from the test (clock warps, cheat-mining
heartbeats, archiver sync nudges). Tests submit a tx and wait, in real
time. No assertions change.

No production code changes: with a self-advancing L1, the sequencer and
publisher behave exactly as on a real network.

## Parallelization

The suite file is renamed to `e2e_ha_full.parallel.test.ts`, so CI runs
each of its 8 tests as an isolated job in its own compose stack instead
of one 15+ minute serial job:

- `bootstrap.sh` expands the HA suite per test name (same mechanism as
the existing `.parallel` simple tests).
- `run_test.sh` forwards the test name into the compose stack and
namespaces the docker compose project per test so concurrent jobs on one
host don't collide.
- `sendTriggerTx` now starts the HA sequencers idempotently, since under
per-test isolation the governance/reload/distribute tests run without
the first test (previously the only caller of `startHASequencers`).
- Three clock-skew test titles contained parentheses, which jest's
`--testNamePattern` interprets as regex groups (the filter would
silently match nothing); they are retitled.

## Teardown fix (follow-up to the first CI round)

The first CI round passed every test body but three jobs
(produce-blocks, governance, reload) hung in `afterAll` until the job
timeout. Two compounding causes, both fixed here:

- `afterAll` reset the shared `TestDateProvider` *before* stopping
nodes. The reset rewinds the clock from chain time to wall time —
minutes apart after the automine deploy burst — so vote submissions
armed against the rewound clock pushed sequencer stops out by that gap.
The old 30s abandon-race then gave up, and the abandoned nodes outlived
the jest environment, keeping the worker alive until the CI timeout
(jest runs without `forceExit`). `afterAll` now stops sequencers first,
awaits every node stop fully, and resets the clock last. These three
jobs are the ones whose tests end with sequencers still running; the
distribute test (which stops nodes in-test, before any reset) passed for
the same reason.
- Ports #23990 from `merge-train/spartan` (not previously on the v5
line): `CheckpointProposalJob.interrupt()` now propagates to the
publisher, cancelling the `sendRequestsAt` slot-deadline sleep on
sequencer stop, so a pending vote submission can never block shutdown.
The original PR's `e2e_ha_full` teardown changes are superseded by the
rework above and were not ported.

## Verification

- Three full local runs of the suite via `run_test.sh ha` (all 8 tests
each): green in 255s / 254s / 268s of jest time (the old warp-based
suite ran 10+ minutes), with zero occurrences of the old failure
signatures (`Fork not found`, `Archiver did not sync`, `discarding
pipelined work`) — passing runs of the old code showed 12+ `Fork not
found` errors even when green.
- One per-test CI-style run (`run_test.sh ha <file> "should distribute
work across multiple HA nodes"`): the originally flaky test passes
standalone in its own compose stack (7 skipped, 1 passed), exercising
the full `TEST_NAME` plumbing.
- `yarn build`, `yarn format`, `yarn lint` clean; `sequencer-client`
unit tests pass (back to the pre-change suite after the revert).

Author: spalladino

.test_patterns.yml

@@ -371,7 +371,7 @@ tests:
     owners:
       - *palla
 
-  - regex: "yarn-project/end-to-end/scripts/run_test.sh ha src/composed/ha/e2e_ha_full.test.ts"
+  - regex: "yarn-project/end-to-end/scripts/run_test.sh ha src/composed/ha/e2e_ha_full.parallel.test.ts"
     owners:
       - *spyros
 

yarn-project/end-to-end/bootstrap.sh

@@ -96,7 +96,13 @@ function test_cmds {
   )
   for test in "${tests[@]}"; do
     # We must set ONLY_TERM_PARENT=1 to allow the script to fully control cleanup process.
-    echo "$hash:ONLY_TERM_PARENT=1:TIMEOUT=30m $run_test_script ha $test"
+    if [[ "$test" == *.parallel.test.ts ]]; then
+      while IFS= read -r test_name; do
+        echo "$hash:ONLY_TERM_PARENT=1:TIMEOUT=30m $run_test_script ha $test \"$test_name\""
+      done < <(extract_test_names "$test")
+    else
+      echo "$hash:ONLY_TERM_PARENT=1:TIMEOUT=30m $run_test_script ha $test"
+    fi
   done
 
   #echo "$hash:ONLY_TERM_PARENT=1 $run_test_script simple src/e2e_multi_validator/e2e_multi_validator_node.test.ts"

yarn-project/end-to-end/scripts/ha/docker-compose.yml

@@ -29,12 +29,6 @@ services:
     volumes:
       - web3signer_keys:/keys
 
-  anvil:
-    image: aztecprotocol/build:3.0
-    cpus: 1
-    mem_limit: 2G
-    entrypoint: 'anvil --silent -p 8545 --host 0.0.0.0 --chain-id 31337'
-
   end-to-end:
     image: aztecprotocol/build:3.0
     cpus: 4
@@ -51,7 +45,8 @@ services:
     environment:
       JEST_CACHE_DIR: /tmp-jest
       LOG_LEVEL: ${LOG_LEVEL:-verbose}
-      ETHEREUM_HOSTS: http://anvil:8545
+      TEST: ${TEST:-./src/composed/ha/e2e_ha_full.parallel.test.ts}
+      TEST_NAME: ${TEST_NAME:-}
       L1_CHAIN_ID: 31337
       DATABASE_URL: postgresql://aztec:aztec@postgres:5432/aztec_ha_test
       WEB3_SIGNER_URL: http://web3signer:9000
@@ -70,10 +65,6 @@ services:
         while ! nc -z web3signer 9000; do sleep 1; done;
         echo "Web3Signer is ready"
 
-        # Wait for anvil to be ready
-        while ! nc -z anvil 8545; do sleep 1; done;
-        echo "Anvil is ready"
-
         # Run database migrations
         echo "Running database migrations..."
         cd /root/aztec-packages/yarn-project/aztec
@@ -84,7 +75,7 @@ services:
         cd /root/aztec-packages/yarn-project/end-to-end
 
         # Run the test
-        setsid ./scripts/test_simple.sh ${TEST:-./src/composed/ha/e2e_ha_sequencer.test.ts} &
+        setsid ./scripts/test_simple.sh "$${TEST}" "$${TEST_NAME}" &
         pid=$$!
         pgid=$$(($$(ps -o pgid= -p $$pid)))
         trap "kill -SIGTERM -$$pgid" SIGTERM
@@ -96,8 +87,6 @@ services:
         condition: service_healthy
       web3signer:
         condition: service_started
-      anvil:
-        condition: service_started
 
 volumes:
   postgres_data:

yarn-project/end-to-end/scripts/run_test.sh

@@ -25,7 +25,10 @@ case "$type" in
     TEST=$test exec run_compose_test $test end-to-end $PWD/web3signer
   ;;
   "ha")
-    # Remove volumes on cleanup for HA tests to ensure clean database state on retries
-    TEST=$test REMOVE_COMPOSE_VOLUMES=1 exec run_compose_test $test end-to-end $PWD/ha
+    # Remove volumes on cleanup for HA tests to ensure clean database state on retries.
+    # NAME_POSTFIX namespaces the compose project per test so parallel per-test jobs don't collide.
+    # Compose project names must be lowercase alphanumerics, hyphens, and underscores.
+    postfix=$(echo "$test_name" | tr '[:upper:]' '[:lower:]' | sed 's/[^a-z0-9]/_/g')
+    TEST=$test TEST_NAME=$test_name NAME_POSTFIX=${postfix:+_$postfix} REMOVE_COMPOSE_VOLUMES=1 exec run_compose_test $test end-to-end $PWD/ha
   ;;
 esac