fix(test): wait for full gossip mesh before committee produces (A-1219) (#24149)

## Problem

`e2e_p2p_network › should rollup txs from all peers (and add the
validators without cheating)` (in `gossip_network_no_cheat.test.ts`)
intermittently fails with `TimeoutError: Timeout awaiting first
checkpoint published` — the chain never gets a first checkpoint onto L1
within 120s.

## Log analysis

From CI run
[`6d6e74a70fce8826`](http://ci.aztec-labs.com/6d6e74a70fce8826):

- The test did a blind `sleep(8000)` for peer discovery, then waited for
the first checkpoint. On the 2-CPU runner the gossipsub
**proposal/checkpoint meshes were not fully formed** 8s in.
- The first checkpoint attempt (slot 97, proposer validator-3) reached
only **2 of 3** attestations — validators 1 and 2 never received the
slot-97 proposal at all (only validator-4 had a live gossip path). No L1
publish was attempted; the proposer aborted locally on the
attestation-collection timeout.
- Because that checkpoint never landed, the L1-confirmed chain stayed at
genesis, so every later slot rebuilt a *competing* un-checkpointed block
1 (new archive). The blocks **are** pruned (`archiver:l1-sync: Pruning
blocks after block 0 ...`), but the prune lands ~1.5 slots after the
block is built — later than the next proposal arrives. So peers still
holding a not-yet-pruned block 1 rejected the new proposal with
`block_number_already_exists`, never re-executed, never attested —
capping every round at 2/3 forever.

Root cause: the gossip mesh wasn't formed when the committee started
producing, so the first proposal reached only a subset of the committee.
That both starved the first checkpoint of quorum and split the
validators onto competing block-1 forks that never re-converge.

## Fix

Replace the blind `sleep(8000)` with `waitForP2PMeshConnectivity` on the
`block_proposal`, `checkpoint_proposal`, and `checkpoint_attestation`
topics, requiring a **full mesh (N-1 peers per node)** so the first
proposal reaches the whole committee. The first checkpoint then reaches
quorum and lands — after which the chain advances to block 2 and no
competing block 1 is ever built.

Also adds a `minMeshPeerCount` parameter to `waitForP2PMeshConnectivity`
(default `1`, preserving existing callers — the helper otherwise only
requires a single mesh peer per node, which can leave some committee
members unreached at first). Quorum-from-genesis tests pass `N-1` for a
full mesh.

This is the test-side fix that addresses the trigger. There is a
separate, more fundamental product-robustness gap — a single missed
checkpoint at the chain tip is unrecoverable because of the
`block_number_already_exists` guard vs. the prune latency — which is
consensus-sensitive and tracked separately (related to A-1218); it is
intentionally **not** addressed here.

## Testing

- Build, format, lint clean; only the test and its helper changed.
- **Not yet run:** the full e2e (`gossip_network_no_cheat.test.ts`,
real-time-dependent, ideally under a 2-CPU constraint). The real
validation is running it repeatedly and confirming the committee reaches
3/3 and the first checkpoint publishes within the gate.

Closes A-1219.

Author: PhilWindle

yarn-project/end-to-end/src/e2e_p2p/gossip_network_no_cheat.test.ts

@@ -13,7 +13,7 @@ import { sleep } from '@aztec/foundation/sleep';
 import { MockZKPassportVerifierAbi } from '@aztec/l1-artifacts/MockZKPassportVerifierAbi';
 import { RollupAbi } from '@aztec/l1-artifacts/RollupAbi';
 import type { SequencerClient } from '@aztec/sequencer-client';
-import { CheckpointAttestation, ConsensusPayload } from '@aztec/stdlib/p2p';
+import { CheckpointAttestation, ConsensusPayload, TopicType } from '@aztec/stdlib/p2p';
 import { ZkPassportProofParams } from '@aztec/stdlib/zkpassport';
 
 import { jest } from '@jest/globals';
@@ -201,8 +201,21 @@ describe('e2e_p2p_network', () => {
       shouldCollectMetrics(),
     );
 
-    // wait a bit for peers to discover each other
-    await sleep(8000);
+    // Wait for the gossipsub mesh to fully form before the committee starts producing. With
+    // skipInitialSequencer, the first blocks are built by this committee, and the first checkpoint
+    // must reach quorum (all 4 validators) to land on L1. If the proposal/checkpoint meshes are only
+    // partly formed, some committee members miss the first proposal, the first checkpoint stalls at
+    // 2/3, and every later slot rebuilds a competing un-checkpointed block 1 that peers reject as
+    // `block_number_already_exists` — a permanent 2/3 deadlock. Require a full mesh (N-1 peers per
+    // node) on the proposal/checkpoint topics so the first proposal reaches the whole committee.
+    await t.waitForP2PMeshConnectivity(
+      nodes,
+      NUM_VALIDATORS,
+      60,
+      0.5,
+      [TopicType.block_proposal, TopicType.checkpoint_proposal, TopicType.checkpoint_attestation],
+      NUM_VALIDATORS - 1,
+    );
 
     // Wait for the first checkpoint to be published to L1 before submitting transactions.
     // With skipInitialSequencer, no blocks exist from setup, so the first blocks are built by the

yarn-project/end-to-end/src/e2e_p2p/p2p_network.ts

@@ -431,6 +431,7 @@ export class P2PNetworkTest {
     timeoutSeconds = 30,
     checkIntervalSeconds = 0.1,
     topics: TopicType[] = [TopicType.tx],
+    minMeshPeerCount = 1,
   ) {
     const nodeCount = expectedNodeCount ?? nodes.length;
     const minPeerCount = nodeCount - 1;
@@ -457,27 +458,29 @@ export class P2PNetworkTest {
 
     this.logger.warn('All nodes connected to P2P mesh');
 
-    // Wait for GossipSub mesh to form for all specified topics.
-    // We only require at least 1 mesh peer per node because GossipSub
-    // stops grafting once it reaches Dlo peers and won't fill the mesh to all available peers.
+    // Wait for the GossipSub mesh to form for all specified topics. By default we only require at
+    // least 1 mesh peer per node, since GossipSub stops grafting once it reaches Dlo peers and won't
+    // fill the mesh to every available peer. Callers that need a proposal to reach the whole
+    // committee within a slot (e.g. quorum-from-genesis tests) raise `minMeshPeerCount` so the mesh
+    // is fully formed — a single mesh peer can leave some committee members unreached at first.
     for (const topic of topics) {
-      this.logger.warn(`Waiting for GossipSub mesh to form for ${topic} topic...`);
+      this.logger.warn(`Waiting for GossipSub mesh (>= ${minMeshPeerCount} peers per node) for ${topic} topic...`);
       await Promise.all(
         nodes.map(async (node, index) => {
           const p2p = node.getP2P();
           await retryUntil(
             async () => {
               const meshPeers = await p2p.getGossipMeshPeerCount(topic);
               this.logger.debug(`Node ${index} has ${meshPeers} gossip mesh peers for ${topic} topic`);
-              return meshPeers >= 1 ? true : undefined;
+              return meshPeers >= minMeshPeerCount ? true : undefined;
             },
-            `Node ${index} to have gossip mesh peers for ${topic} topic`,
+            `Node ${index} to have >= ${minMeshPeerCount} gossip mesh peers for ${topic} topic`,
             timeoutSeconds,
             checkIntervalSeconds,
           );
         }),
       );
-      this.logger.warn(`All nodes have gossip mesh peers for ${topic} topic`);
+      this.logger.warn(`All nodes have >= ${minMeshPeerCount} gossip mesh peers for ${topic} topic`);
     }
   }