Skip to content

Commit eae69fb

Browse files
spalladinoaztec-bot
authored andcommitted
test: fix proof_boundary startup race (#24671)
## What Fixes a low-frequency (~1 in 140) startup-race flake in the `proof_boundary` multi-node e2e suite (CI hash `c06563e7d2b7e067`, group `e2e-p2p-epoch-flakes`). All five scenarios time out in the shared `computeBoundarySlot()` helper at `waitUntilCheckpointNumber(1)` — the 3-validator committee never produces its first checkpoint. ## Why `setupTest` created the three validator nodes sequentially (`asyncMap`) and started each sequencer at node construction. A node subscribes to the gossip topics when its P2P client starts (unconditionally, at node creation), but the in-memory mock gossip bus has no message replay. So when the first-created validator happened to draw the first slot's proposer, it broadcast the inaugural block proposal before the other two committee members had been created and subscribed — the proposal was lost, the 3-of-3 quorum was missed, and the chain forked onto genesis and never produced checkpoint 1. It's randao-dependent (fatal only when the first-created node is the first proposer), hence the low frequency. ## Fix Create the validator nodes with `dontStartSequencer: true`, then `startSequencers(nodes)` once all three exist. All peers subscribe to gossip at creation, so no proposal is broadcast before the whole committee is on the bus. This matches the pattern already used by the non-flaky sibling tests `first_slot.test.ts` and `block-production/setup.ts`. ## Scope Test-only change. Related to A-1419, which tracks the underlying product-level recovery race (competing block-1 re-executions surfacing as `BlockNumberNotSequentialError`). This PR implements that issue's "keep the test startup barrier" acceptance criterion but does **not** resolve A-1419 — the product-side race is unchanged. (cherry picked from commit 54edf7a)
1 parent 41f0427 commit eae69fb

1 file changed

Lines changed: 13 additions & 2 deletions

File tree

yarn-project/end-to-end/src/multi-node/block-production/proof_boundary.parallel.test.ts

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -58,9 +58,10 @@ describe('multi-node/block-production/proof_boundary', () => {
5858

5959
const minTxsPerBlock = validatorOverrides.minTxsPerBlock ?? 0;
6060
const maxTxsPerBlock = validatorOverrides.maxTxsPerBlock ?? 1;
61-
logger.warn(`Initial setup complete. Starting ${NODE_COUNT} validator nodes.`);
61+
logger.warn(`Initial setup complete. Creating ${NODE_COUNT} validator nodes with sequencers paused.`);
62+
// Ensure every validator can handle the first proposal before any sequencer begins its polling loop.
6263
nodes = await asyncMap(validators, ({ privateKey }) =>
63-
test.createValidatorNode([privateKey], { minTxsPerBlock, maxTxsPerBlock }),
64+
test.createValidatorNode([privateKey], { dontStartSequencer: true, minTxsPerBlock, maxTxsPerBlock }),
6465
);
6566

6667
proverNode = await test.createProverNode({
@@ -73,6 +74,11 @@ describe('multi-node/block-production/proof_boundary', () => {
7374
logger.warn(`Test setup completed.`, { validators: validators.map(v => v.attester.toString()) });
7475
};
7576

77+
const startSequencers = async () => {
78+
await test.startSequencers(nodes);
79+
logger.warn(`Started ${NODE_COUNT} validator sequencers.`);
80+
};
81+
7682
const collectSequencerEvents = (sequencers: SequencerClient[]) => {
7783
const published: PublishedEvent[] = [];
7884
const preparing: PreparingEvent[] = [];
@@ -216,6 +222,7 @@ describe('multi-node/block-production/proof_boundary', () => {
216222

217223
const sequencers = nodes.map(node => node.getSequencer()!);
218224
const events = collectSequencerEvents(sequencers);
225+
await startSequencers();
219226

220227
const { boundarySlot, boundaryTs } = await computeBoundarySlot();
221228

@@ -264,6 +271,7 @@ describe('multi-node/block-production/proof_boundary', () => {
264271

265272
const sequencers = nodes.map(node => node.getSequencer()!);
266273
const events = collectSequencerEvents(sequencers);
274+
await startSequencers();
267275

268276
const { boundarySlot, boundaryTs } = await computeBoundarySlot();
269277

@@ -298,6 +306,7 @@ describe('multi-node/block-production/proof_boundary', () => {
298306

299307
const sequencers = nodes.map(node => node.getSequencer()!);
300308
const events = collectSequencerEvents(sequencers);
309+
await startSequencers();
301310

302311
const { boundarySlot, boundaryEpoch } = await computeBoundarySlot();
303312

@@ -335,6 +344,7 @@ describe('multi-node/block-production/proof_boundary', () => {
335344

336345
const sequencers = nodes.map(node => node.getSequencer()!);
337346
const events = collectSequencerEvents(sequencers);
347+
await startSequencers();
338348

339349
const { boundarySlot } = await computeBoundarySlot();
340350
const slotN = SlotNumber(Number(boundarySlot) - 1);
@@ -374,6 +384,7 @@ describe('multi-node/block-production/proof_boundary', () => {
374384

375385
const sequencers = nodes.map(node => node.getSequencer()!);
376386
const events = collectSequencerEvents(sequencers);
387+
await startSequencers();
377388

378389
const { boundarySlot } = await computeBoundarySlot();
379390
const slotN = SlotNumber(Number(boundarySlot) - 1);

0 commit comments

Comments
 (0)