Skip to content

fix(test): reliably find target proposer in sentinel_status_slash (A-1217)#24143

Merged
PhilWindle merged 2 commits into
merge-train/spartan-v5from
phil/a-1217-fix-e2e_p2p-sentinel_status_slash
Jun 17, 2026
Merged

fix(test): reliably find target proposer in sentinel_status_slash (A-1217)#24143
PhilWindle merged 2 commits into
merge-train/spartan-v5from
phil/a-1217-fix-e2e_p2p-sentinel_status_slash

Conversation

@PhilWindle

@PhilWindle PhilWindle commented Jun 16, 2026

Copy link
Copy Markdown
Collaborator

Problem

The e2e helper warpToSlotBeforeTargetProposer (in sentinel_status_slash.parallel.test.ts) intermittently threw Target proposer ... not found with sufficient buffer within 20 epochs, surfacing as an e2e_p2p flake. It's a deterministic helper bug, not network flakiness.

The search window was derived as searchStart = currentSlot + minBufferSlots, searchEnd = (currentEpoch + 2) * AZTEC_EPOCH_DURATION - 1. With AZTEC_EPOCH_DURATION = 2 and minBufferSlots = 2, the buffer offset consumes a full epoch, so whenever currentSlot is odd the window collapses to a single, always-odd slot. Because the proposer for each slot is a different RANDAO-shuffled committee member, probing only odd slots never examines the even slot of any epoch — where the 1-of-6 target can be the proposer — so the loop exhausts all attempts and throws.

Log verification

Confirmed against CI run cc8c935bb167ed37:

  • All 20 attempts probed a 1-slot window, every probed slot odd: 13, 15, 17, … 51. Zero hit lines.
  • The target 0x90f79b… was on the committee every epoch (RANDAO-shuffled into different positions) and attested at slots 11–12 — reachable, just never at a probed (odd) slot.
  • Zero EpochNotStable reverts during the scan — the search ran clean, just structurally blind to even slots.

Fix

Extract the proposer search into a shared findUpcomingProposerSlot in e2e_p2p/shared.ts, parameterized by minLeadSlots, and have the sentinel test use it:

  • Scans forward one slot at a time from currentSlot + minLeadSlots, so it examines both epoch parities (fixing the odd-only blindness) and finds the RANDAO-shuffled target wherever it proposes.
  • Guarantees the returned slot is at least minLeadSlots ahead, so the sentinel can warp to targetSlot - minLeadSlots for its settle buffer with no risk of a backwards warp — the lead comes from the scan start, not from a per-epoch position constraint.
  • Handles EpochNotStable by warping one epoch forward and continuing, keeping the lead.

The sentinel helper becomes a thin wrapper: findUpcomingProposerSlot({ minLeadSlots: 2 }) then advanceToSlot(targetSlot - 2).

Kept separate from the existing advanceToEpochBeforeProposer, which serves a genuinely different pattern: its four callers stay one epoch before the target to start sequencers, then warp to the epoch boundary, and rely on warmupSlots for their warm-up margin (load-bearing — without it their proposals serialize past the slot boundary and are rejected as late). That helper is unchanged, so its callers are unaffected.

Testing

  • Build, format, and lint clean; only shared.ts and the sentinel test changed.
  • The proposer search now examines both parities and guarantees the lead by construction.
  • Not yet run: the full e2e (sentinel_status_slash.parallel.test.ts, ~20 min, real-time-dependent). The real validation is running it repeatedly to confirm the proposer is found and the suite's other two tests (which share the helper) still pass.

Closes A-1217.

The e2e helper `warpToSlotBeforeTargetProposer` intermittently threw
"Target proposer ... not found with sufficient buffer within 20 epochs",
masked as an e2e_p2p flake. It is a deterministic helper bug.

The search window was `[currentSlot + minBufferSlots, (currentEpoch + 2)
* AZTEC_EPOCH_DURATION - 1]`. With AZTEC_EPOCH_DURATION = 2 and
minBufferSlots = 2 the buffer offset consumes a full epoch, so whenever
the loop sits on an odd slot the window collapses to a single, always-odd
slot. The proposer for each slot is a different RANDAO-shuffled committee
member, so probing only odd slots never examines the even slot of any
epoch — where the 1-of-6 target can be the proposer — and the loop
exhausts all 20 attempts.

Verified against CI run cc8c935bb167ed37: all 20 attempts probed a
1-slot window (13, 15, 17, ... 51 — all odd), the target 0x90f79b... was
on the committee every epoch, and there were zero reverts. The window
arithmetic, not network flakiness, is the cause.

Fix: scan the full next epoch `[(currentEpoch + 1) * AZTEC_EPOCH_DURATION,
(currentEpoch + 2) * AZTEC_EPOCH_DURATION - 1]`, covering both slot
parities, and drop the search-time buffer entirely. The settle buffer is
already provided by landing at `targetSlot - 2` (clamped so we never warp
backwards), so no separate pre-search buffer is needed. The next epoch
remains the only window scanned (the current epoch is partly elapsed and
the second-next can revert with EpochNotStable).
Move the forward proposer-slot search into a shared `findUpcomingProposerSlot`
in e2e_p2p/shared.ts, parameterized by `minLeadSlots`, and have the sentinel
test use it.

The helper scans forward one slot at a time from `currentSlot + minLeadSlots`,
so it examines both epoch parities (the per-slot proposer is a different
RANDAO-shuffled committee member, so a fixed per-epoch offset can leave the
1-of-N target unexamined when the epoch is short) and guarantees the returned
slot is at least `minLeadSlots` ahead — the sentinel warps to
`targetSlot - minLeadSlots` for its settle buffer with no risk of a backwards
warp. EpochNotStable reverts are handled by warping one epoch forward and
keeping the lead.

Kept separate from `advanceToEpochBeforeProposer`, which serves a different
pattern: those callers stay one epoch before the target to start sequencers,
then warp to the epoch boundary, and rely on `warmupSlots` for their margin.
That helper is unchanged, so its four callers are unaffected.
@PhilWindle PhilWindle marked this pull request as ready for review June 17, 2026 09:16
@PhilWindle PhilWindle changed the title fix(test): probe full epoch in warpToSlotBeforeTargetProposer (A-1217) fix(test): reliably find target proposer in sentinel_status_slash (A-1217) Jun 17, 2026
@AztecBot

Copy link
Copy Markdown
Collaborator

Flakey Tests

🤖 says: This CI run detected 2 tests that failed, but were tolerated due to a .test_patterns.yml entry.

\033FLAKED\033 (8;;http://ci.aztec-labs.com/f4f8e30e112388c8�f4f8e30e112388c88;;�): yarn-project/kv-store/scripts/run_test.sh src/bench/sqlite-opfs-encrypted/map_bench.test.ts (1s) (code: 0)
\033FLAKED\033 (8;;http://ci.aztec-labs.com/7d7cf097f2b0014d�7d7cf097f2b0014d8;;�): yarn-project/kv-store/scripts/run_test.sh src/sqlite-opfs/internal/ordered-binary-browser.test.ts (1s) (code: 0)

@PhilWindle PhilWindle merged commit 269e6d0 into merge-train/spartan-v5 Jun 17, 2026
21 checks passed
@PhilWindle PhilWindle deleted the phil/a-1217-fix-e2e_p2p-sentinel_status_slash branch June 17, 2026 10:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants