From b428973994b07a21007e2dd13496d08c04b2aa4f Mon Sep 17 00:00:00 2001 From: Santiago Palladino Date: Fri, 12 Jun 2026 09:24:34 -0300 Subject: [PATCH 01/14] refactor(stdlib)!: thin chain-checkpointed event, collapse sync (#24007) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Motivation `L2BlockStream.work()` reconciled the four chain tips through independent code paths — a reorg walk, two checkpoint loops with a shared cursor, a block loop, and a final proven/finalized diff — with optimizations (`startingBlock`, `skipFinalized`) that muted some paths but not others. Nothing enforced that what one poll emitted was tier-consistent, which produced a family of bugs (detailed below). The root enabler was the fat `chain-checkpointed` event: because it carried a full `PublishedCheckpoint`, the stream had to fetch, order, and replay every checkpoint individually, and every special case had to reason about when that replay could be skipped. This PR makes `chain-checkpointed` a thin tip event like `chain-proven`/`chain-finalized` and collapses `work()` into three symmetric steps: reorg detection, one block-download loop, and unconditional end-of-pass tier reconciliation from a single source snapshot. The two consumers that relied on per-checkpoint payload delivery are first decoupled: the sentinel stops needing a block stream at all, and the prover-node drives its own checkpoint catch-up from a cursor. ## Bugs and races fixed - **Checkpointed cursor stuck at genesis under `startingBlock` (A-1061).** The `startingBlock` fast-forward arm suppressed all checkpoint emission while the end-of-pass `chain-proven` still fired, leaving the local store with `proven > checkpointed` and the checkpointed cursor at genesis until the source's checkpointed tip passed `startingBlock`. In p2p this degraded `isEpochPrune` (epoch prunes misclassified as ordinary prunes, so `txPoolDeleteTxsAfterReorg` was not honored during the window); the same pattern affected the sentinel and prover-node streams. Tier reconciliation is now unconditional against the source snapshot, so the first pass converges. Regression: *emits the source checkpointed tip on the FIRST pass even when startingBlock is past it*. - **Duplicate / out-of-order checkpoint events after a reorg (stale-snapshot guard).** Loop 1's "blocks already local" guard compared against the `localTips` snapshot taken before the prune handler rewrote the store, so a node lagging a checkpoint that detected a reorg in the same pass could emit `chain-checkpointed` for a new-fork checkpoint before its blocks, then re-emit checkpoints from the prune target upward. Loop 1 is deleted; reconciliation re-reads the local tips after a prune so it compares against the just-clamped cursors. Regression: *does not re-emit the checkpointed tip after pruning to a block ahead of it*. - **Same-height stale cursors never refreshed (number-only comparisons).** The proven/finalized gates compared block numbers only, so a same-number/different-hash tip after a reorg kept a stale (hash, checkpoint-id) pair until the tip next advanced. Gates now compare (number, hash) — skipping the hash comparison when the local hash is `undefined`, which world-state legitimately reports for tips ahead of its synced range (comparing against it would re-emit the event on every poll). Regressions: *re-emits the proven tip when numbers match but the known local hash differs* and *does not re-emit when the local hash is undefined*. - **Sentinel credited attestors on reorged-out checkpoints.** Its `slotNumberToCheckpoint` map had no `chain-pruned` handling, so a reorged-out checkpoint's attestation entry lingered and `getSlotActivity` could classify validators against a non-canonical checkpoint. The map (and the stream feeding it) is gone; the sentinel fetches the checkpoint for a slot on demand, so the answer is always canonical. - **Read-skew race between `getL2Tips` and `getCheckpoints`.** The old checkpoint loops fetched payloads in separate reads from the tips snapshot; a source-side reorg between the two reads could plant inconsistent state in the local store (the reason catch-up code needed validation-and-abort machinery). Eliminated structurally: every tier event is now built from the same `getL2Tips` snapshot, with no second fetch to skew against. - **Prover-node restart could have skipped unproven checkpoints (prevented by design here).** With catch-up driven by a cursor, seeding it from a checkpointed tip would silently skip the unproven checkpoints of a partially-proven epoch on restart. The cursor seeds from the last checkpoint of the last fully-proven epoch (or 0), and advances only after both `checkpointStore.addOrUpdate` and `sessionManager.onCheckpointAdded` succeed, preserving the existing at-least-once retry semantics (A-1041). Epoch expiry additionally gets a periodic tick, since it previously piggybacked on per-checkpoint event volume. ## Approach - **Sentinel (decoupled first):** deletes its `L2BlockStream`, `L2TipsMemoryStore`, slot→checkpoint map, and the manual stream sync in its work loop. `getSlotActivity` fetches `archiver.getCheckpoint({ slot })` on demand (the by-slot query already existed) and derives the same attestation data; the p2p-sync gate reads the archiver tips directly instead of a stale local mirror. - **Prover-node (decoupled second):** on a thin `chain-checkpointed` tip event, walks every checkpoint between its cursor and the tip: light `getCheckpointsData` metadata first, whole-epoch relevance filtering (an epoch is skipped only if fully proven or past its proof-submission window — never individual checkpoints inside a provable epoch, which the SessionManager's full-coverage contract requires), then a heavy `getCheckpoint` fetch only for checkpoints it will actually register. This is strictly cheaper than the old stream replay, which transferred every full checkpoint payload before the prover-node could decide to skip it. - **Stream rewrite:** `chain-checkpointed` becomes `{ block: L2BlockId, checkpoint: CheckpointId }`, emitted at most once per pass — symmetric with `chain-proven`/`chain-finalized`. `work()` is now: reorg walk-back + `chain-pruned`; one `getBlocks` download loop (start incorporates `startingBlock`/`skipFinalized`); end-of-pass reconciliation checkpointed → proven → finalized from one snapshot. PXE, the one remaining payload consumer, fetches its anchor header by hash (reorg-safe) and skips the update if the block vanished. ## API changes Internal API only (no RPC schema changes): - `L2BlockStreamEvent`: `chain-checkpointed` carries `{ block, checkpoint }` ids instead of a `PublishedCheckpoint`, and fires at most once per sync pass instead of once per checkpoint. Consumers needing payloads fetch them from the block source. - `L2BlockStream` source narrows to `Pick`; the `checkpointPrefetchLimit` option and `CHECKPOINT_PREFETCH_LIMIT` export are gone. - `LocalChainTips.checkpointed` widens to `{ block, checkpoint }` so the checkpointed tier can hash-gate like proven/finalized (still structurally assignable from `LocalL2Tips`). ## Simplifications - `L2BlockStream.work()`: ~220 → ~115 lines. Deleted: Loop 1 (already-local checkpoint backfill), Loop 2 + prefetch buffer (checkpoint-transport block download), `nextCheckpointToEmit`, both `startingBlock` checkpoint fast-forward arms, and the checkpoint payload fetching altogether. - Catch-up emits at most one checkpointed event per pass regardless of lag — no per-checkpoint replay, no multi-emission warning path, no anti-spam special cases. - Sentinel: net ~50 lines removed plus a whole subsystem dependency (stream + tips store) — replaced by one ~25-line on-demand fetch. - PXE node adapter: the ~25-line `getCheckpoints` implementation is deleted; the telemetry stream wrapper narrows accordingly. - Tips stores: `handleChainCheckpointed` reads ids straight off the event instead of recomputing the checkpoint hash from the payload. - Stream test suite: rewritten from 1,828 to ~640 lines while adding four regression tests for the bugs above. - Net across the branch: **−1,228 lines** over 15 files. ## Changes - **stdlib**: thin `chain-checkpointed` event; collapsed `work()`; hash-aware tier gates; narrowed stream source type; tips-store handler reads event ids directly. - **stdlib (tests)**: stream suite rewritten around the new event semantics + regression tests for A-1061, post-prune reconciliation, and both hash-gate behaviors. - **prover-node**: cursor-driven checkpoint catch-up (`processCheckpointJump`/`registerCheckpoint`/ `computeStartingCheckpoint`), whole-epoch relevance filtering, prune clamping, periodic epoch-expiry tick. - **aztec-node (sentinel)**: block stream, tips store, and slot→checkpoint map deleted; on-demand canonical checkpoint fetch; direct archiver read for the p2p-sync gate. - **pxe**: anchor header fetched by hash on checkpointed-tip events; node adapter loses `getCheckpoints`. - **kv-store / telemetry-client**: test-suite and wrapper-type adjustments to the new event shape. Fixes A-1061 --------- Co-authored-by: Claude Fable 5 --- .../aztec-node/src/sentinel/README.md | 6 +- .../aztec-node/src/sentinel/sentinel.test.ts | 43 +- .../aztec-node/src/sentinel/sentinel.ts | 106 +- .../prover-node/src/prover-node.test.ts | 130 +- yarn-project/prover-node/src/prover-node.ts | 126 +- .../block_synchronizer/block_stream_source.ts | 46 +- .../block_synchronizer.test.ts | 52 +- .../block_synchronizer/block_synchronizer.ts | 14 +- .../schema_tests.ts | 5 +- .../src/block/l2_block_stream/interfaces.ts | 11 +- .../l2_block_stream/l2_block_stream.test.ts | 1691 +++-------------- .../block/l2_block_stream/l2_block_stream.ts | 212 +-- .../l2_block_stream/l2_tips_store_base.ts | 6 +- .../block/test/l2_tips_store_test_suite.ts | 12 +- .../src/wrappers/l2_block_stream.ts | 2 +- 15 files changed, 642 insertions(+), 1820 deletions(-) diff --git a/yarn-project/aztec-node/src/sentinel/README.md b/yarn-project/aztec-node/src/sentinel/README.md index 5ab68d8d8cff..9f114ea65c0f 100644 --- a/yarn-project/aztec-node/src/sentinel/README.md +++ b/yarn-project/aztec-node/src/sentinel/README.md @@ -17,10 +17,10 @@ The sentinel is one of several watchers registered with the slasher; it does not | Source | What it provides | |---|---| | `EpochCache` | Slot/epoch helpers, committee + proposer for a slot, escape-hatch state | -| `L2BlockSource` (archiver) | Synced slot, `chain-checkpointed` events, block headers | +| `L2BlockSource` (archiver) | Synced slot, `getCheckpoint({ slot })`, `getL2Tips()`, block headers | | `P2PClient` | `getCheckpointAttestationsForSlot(slot, payloadHash)`, `hasBlockProposalsForSlot(slot)` | | `CheckpointReexecutionTracker` | Local re-execution outcome for the proposal at each slot (`valid` / `invalid` / `unvalidated`) — populated by the validator client's `ProposalHandler` | -| L1-checkpointed events | `chain-checkpointed` populates `slotNumberToCheckpoint` with the canonical attestor set | +| L1-confirmed checkpoints | Fetched on demand per slot via `archiver.getCheckpoint({ slot })`, yielding the canonical attestor set | ## Two cadences @@ -49,7 +49,7 @@ For each slot, the proposer is assigned one of six statuses, ranked highest-conf | # | Status | Trigger | Inactive party | |---|---|---|---| -| 6 | `checkpoint-mined` | `slotNumberToCheckpoint.has(slot)` (a checkpoint covering this slot has landed on L1) | Attestors who didn't attest | +| 6 | `checkpoint-mined` | `archiver.getCheckpoint({ slot })` returns a checkpoint (one covering this slot has landed on L1) | Attestors who didn't attest | | 5 | `checkpoint-valid` | `tracker.getOutcomeForSlot(slot) === 'valid'` | Attestors who didn't attest | | 4 | `checkpoint-invalid` | `tracker.getOutcomeForSlot(slot) === 'invalid'` (re-executed and rejected) | Proposer | | 3 | `checkpoint-unvalidated` | `tracker.getOutcomeForSlot(slot) === 'unvalidated'` (validation aborted: missing data, timeout, etc.) | Proposer | diff --git a/yarn-project/aztec-node/src/sentinel/sentinel.test.ts b/yarn-project/aztec-node/src/sentinel/sentinel.test.ts index e2c489e37254..ee95f5af5187 100644 --- a/yarn-project/aztec-node/src/sentinel/sentinel.test.ts +++ b/yarn-project/aztec-node/src/sentinel/sentinel.test.ts @@ -11,7 +11,6 @@ import { GENESIS_BLOCK_HEADER_HASH, L2Block, type L2BlockSource, - type L2BlockStream, getAttestationInfoFromPublishedCheckpoint, } from '@aztec/stdlib/block'; import { @@ -44,7 +43,6 @@ describe('sentinel', () => { let epochCache: MockProxy; let archiver: MockProxy; let p2p: MockProxy; - let blockStream: MockProxy; let reexecutionTracker: CheckpointReexecutionTracker; let kvStore: AztecLMDBStoreV2; @@ -70,7 +68,6 @@ describe('sentinel', () => { archiver = mock(); archiver.getGenesisBlockHash.mockReturnValue(GENESIS_BLOCK_HEADER_HASH); p2p = mock(); - blockStream = mock(); reexecutionTracker = new CheckpointReexecutionTracker(); kvStore = await openTmpStore('sentinel-test'); @@ -100,7 +97,7 @@ describe('sentinel', () => { epochCache.getEpochNow.mockReturnValue(epoch); epochCache.getL1Constants.mockReturnValue(l1Constants); - sentinel = new TestSentinel(epochCache, archiver, p2p, store, reexecutionTracker, config, blockStream); + sentinel = new TestSentinel(epochCache, archiver, p2p, store, reexecutionTracker, config); }); afterEach(async () => { @@ -142,16 +139,22 @@ describe('sentinel', () => { let proposer: EthAddress; let committee: EthAddress[]; - /** Helper to create and emit a chain-checkpointed event */ - const emitCheckpointEvent = async (checkpoint: Checkpoint, checkpointAttestations: CommitteeAttestation[] = []) => { + /** + * Stubs the archiver so the slot's on-demand `getCheckpoint({ slot })` lookup returns this checkpoint, mirroring + * a checkpoint that has landed on L1 covering the slot. + */ + const mineCheckpointForSlot = (checkpoint: Checkpoint, checkpointAttestations: CommitteeAttestation[] = []) => { const published = new PublishedCheckpoint(checkpoint, L1PublishedData.random(), checkpointAttestations); - const lastBlock = checkpoint.blocks.at(-1)!; - const block = { number: lastBlock.number, hash: (await lastBlock.hash()).toString() }; - await sentinel.handleBlockStreamEvent({ type: 'chain-checkpointed', checkpoint: published, block }); + const checkpointSlot = checkpoint.header.slotNumber; + archiver.getCheckpoint.mockImplementation(query => + Promise.resolve('slot' in query && query.slot === checkpointSlot ? published : undefined), + ); return published; }; beforeEach(async () => { + // No L1 checkpoint by default; individual tests mine one via mineCheckpointForSlot. + archiver.getCheckpoint.mockResolvedValue(undefined); signers = times(4, Secp256k1Signer.random); validators = signers.map(signer => signer.address); block = await L2Block.random(BlockNumber(1), { slotNumber: slot }); @@ -165,7 +168,7 @@ describe('sentinel', () => { it('flags checkpoint as mined when L1 has it (case 6)', async () => { // Create a checkpoint with a block at the target slot and emit chain-checkpointed event const checkpoint = await Checkpoint.random(CheckpointNumber(1), { numBlocks: 1, slotNumber: slot }); - await emitCheckpointEvent(checkpoint); + mineCheckpointForSlot(checkpoint); const activity = await sentinel.getSlotActivity(slot, epoch, proposer, committee); expect(activity[proposer.toString()]).toEqual('checkpoint-mined'); @@ -208,7 +211,7 @@ describe('sentinel', () => { it('prefers L1-mined over tracker outcome', async () => { reexecutionTracker.recordOutcome(slot, block.archive.root, 'invalid', CheckpointNumber(1)); const checkpoint = await Checkpoint.random(CheckpointNumber(1), { numBlocks: 1, slotNumber: slot }); - await emitCheckpointEvent(checkpoint); + mineCheckpointForSlot(checkpoint); const activity = await sentinel.getSlotActivity(slot, epoch, proposer, committee); expect(activity[proposer.toString()]).toEqual('checkpoint-mined'); @@ -224,7 +227,7 @@ describe('sentinel', () => { }); // Emit the chain-checkpointed event with attestations from signers 0 and 1 - publishedCheckpoint = await emitCheckpointEvent(checkpoint, checkpointAttestations); + publishedCheckpoint = mineCheckpointForSlot(checkpoint, checkpointAttestations); const attestorsFromCheckpoint = compactArray( getAttestationInfoFromPublishedCheckpoint(publishedCheckpoint, TEST_COORDINATION_SIGNATURE_CONTEXT).map(info => @@ -267,7 +270,7 @@ describe('sentinel', () => { // Emit chain-checkpointed event with both signed and placeholder attestations // The Sentinel should only count the recovered-from-signature ones - publishedCheckpoint = await emitCheckpointEvent(checkpoint, allAttestations); + publishedCheckpoint = mineCheckpointForSlot(checkpoint, allAttestations); // Verify that getAttestationInfoFromPublishedCheckpoint returns 4 entries total: // - 2 with status 'recovered-from-signature' (actual attestations with valid signatures) @@ -299,7 +302,7 @@ describe('sentinel', () => { it('identifies missed attestors if block is mined', async () => { // Create checkpoint with a block at the target slot const checkpoint = await Checkpoint.random(CheckpointNumber(1), { numBlocks: 1, slotNumber: slot }); - await emitCheckpointEvent(checkpoint); + mineCheckpointForSlot(checkpoint); // P2P provides attestations from validators 0, 1, 2 (not validator 3) p2p.getCheckpointAttestationsForSlot.mockResolvedValue(attestations.slice(0, -1)); @@ -1030,18 +1033,6 @@ describe('sentinel', () => { }); class TestSentinel extends Sentinel { - constructor( - epochCache: EpochCache, - archiver: L2BlockSource, - p2p: P2PClient, - store: SentinelStore, - reexecutionTracker: CheckpointReexecutionTracker, - config: SentinelRuntimeConfig, - protected override blockStream: L2BlockStream, - ) { - super(epochCache, archiver, p2p, store, reexecutionTracker, config); - } - public override init() { this.initialSlot = this.epochCache.getEpochAndSlotNow().slot; return Promise.resolve(); diff --git a/yarn-project/aztec-node/src/sentinel/sentinel.ts b/yarn-project/aztec-node/src/sentinel/sentinel.ts index 78a50ff8c075..ef2fc3c69340 100644 --- a/yarn-project/aztec-node/src/sentinel/sentinel.ts +++ b/yarn-project/aztec-node/src/sentinel/sentinel.ts @@ -1,16 +1,9 @@ import type { EpochCache } from '@aztec/epoch-cache'; -import { - BlockNumber, - CheckpointNumber, - CheckpointProposalHash, - EpochNumber, - SlotNumber, -} from '@aztec/foundation/branded-types'; +import { CheckpointNumber, CheckpointProposalHash, EpochNumber, SlotNumber } from '@aztec/foundation/branded-types'; import { countWhile, filterAsync, fromEntries, getEntries, mapValues } from '@aztec/foundation/collection'; import { EthAddress } from '@aztec/foundation/eth-address'; import { createLogger } from '@aztec/foundation/log'; import { RunningPromise } from '@aztec/foundation/running-promise'; -import { L2TipsMemoryStore, type L2TipsStore } from '@aztec/kv-store/stores'; import type { P2PClient } from '@aztec/p2p'; import { OffenseType, @@ -21,13 +14,7 @@ import { getOffenseTypeName, } from '@aztec/slasher'; import type { SlasherConfig } from '@aztec/slasher/config'; -import { - type L2BlockSource, - L2BlockStream, - type L2BlockStreamEvent, - type L2BlockStreamEventHandler, - getAttestationInfoFromPublishedCheckpoint, -} from '@aztec/stdlib/block'; +import { type L2BlockSource, getAttestationInfoFromPublishedCheckpoint } from '@aztec/stdlib/block'; import type { CheckpointReexecutionTracker } from '@aztec/stdlib/checkpoint'; import type { ChainConfig } from '@aztec/stdlib/config'; import { getEpochAtSlot, getSlotRangeForEpoch, getTimestampForSlot } from '@aztec/stdlib/epoch-helpers'; @@ -97,7 +84,7 @@ function statusToCategory(status: ValidatorStatusInSlot): ValidatorStatusType { * first: * * - `checkpoint-mined` — a checkpoint covering this slot has landed on L1 - * (`slotNumberToCheckpoint` populated from `chain-checkpointed`). + * (fetched on demand via `archiver.getCheckpoint({ slot })`). * - `checkpoint-valid` — the local node re-executed a checkpoint proposal for this slot * successfully (consulted via `CheckpointReexecutionTracker`). * - `checkpoint-invalid` — the local node re-executed a checkpoint proposal for this slot @@ -135,25 +122,13 @@ function statusToCategory(status: ValidatorStatusInSlot): ValidatorStatusType { * (no history entries for that slot) and per-epoch evaluation writes an empty performance map * (no slashing). */ -export class Sentinel extends (EventEmitter as new () => WatcherEmitter) implements L2BlockStreamEventHandler, Watcher { +export class Sentinel extends (EventEmitter as new () => WatcherEmitter) implements Watcher { protected runningPromise: RunningPromise; - protected blockStream!: L2BlockStream; - protected l2TipsStore: L2TipsStore; protected initialSlot: SlotNumber | undefined; protected lastProcessedSlot: SlotNumber | undefined; /** Largest epoch number for which the end-of-epoch aggregator has run. */ protected lastEvaluatedEpoch: EpochNumber | undefined; - protected slotNumberToCheckpoint: Map< - SlotNumber, - { - checkpointNumber: CheckpointNumber; - archive: string; - /** Hex keccak256 of the consensus payload bytes; used to fetch matching p2p attestations. */ - proposalPayloadHash: CheckpointProposalHash; - attestors: EthAddress[]; - } - > = new Map(); constructor( protected epochCache: EpochCache, @@ -165,7 +140,6 @@ export class Sentinel extends (EventEmitter as new () => WatcherEmitter) impleme protected logger = createLogger('node:sentinel'), ) { super(); - this.l2TipsStore = new L2TipsMemoryStore(archiver.getGenesisBlockHash()); const interval = (epochCache.getL1Constants().ethereumSlotDuration * 1000) / 4; this.runningPromise = new RunningPromise(this.work.bind(this), logger, interval); } @@ -187,17 +161,14 @@ export class Sentinel extends (EventEmitter as new () => WatcherEmitter) impleme } /** - * Loads initial slot and initializes blockstream. We will not process anything at or before - * the initial slot. Floors at the archiver's synced L2 slot so the sentinel keeps making - * forward progress when L1 is advancing but L2 has no activity (the synced slot is driven by - * L1 sync, not by L2 blocks). Falls back to the wallclock if the archiver isn't ready yet - * (cold start). + * Loads the initial slot. We will not process anything at or before the initial slot. Floors at the + * archiver's synced L2 slot so the sentinel keeps making forward progress when L1 is advancing but L2 has no + * activity (the synced slot is driven by L1 sync, not by L2 blocks). Falls back to the wallclock if the + * archiver isn't ready yet (cold start). */ protected async init() { this.initialSlot = await this.getCurrentSlot(); - const startingBlock = BlockNumber(await this.archiver.getBlockNumber()); - this.logger.info(`Starting validator sentinel with initial slot ${this.initialSlot} and block ${startingBlock}`); - this.blockStream = new L2BlockStream(this.archiver, this.l2TipsStore, this, this.logger, { startingBlock }); + this.logger.info(`Starting validator sentinel with initial slot ${this.initialSlot}`); } /** @@ -215,44 +186,38 @@ export class Sentinel extends (EventEmitter as new () => WatcherEmitter) impleme return this.runningPromise.stop(); } - public async handleBlockStreamEvent(event: L2BlockStreamEvent): Promise { - await this.l2TipsStore.handleBlockStreamEvent(event); - if (event.type === 'chain-checkpointed') { - this.handleCheckpoint(event); - } - } - - protected handleCheckpoint(event: L2BlockStreamEvent) { - if (event.type !== 'chain-checkpointed') { - return; + /** + * Fetches the L1-confirmed checkpoint covering a slot (if any) and derives the slot-level data the + * activity classifier needs: the checkpoint number, archive root, consensus payload hash (used to fetch + * matching p2p attestations regardless of feeAssetPriceModifier variants), and the recovered attestor set. + * Reads on demand so the result is always against the canonical chain — a reorged-out checkpoint simply + * stops being returned, with no stale mapping to clean up. + */ + protected async getCheckpointForSlot(slot: SlotNumber): Promise< + | { + checkpointNumber: CheckpointNumber; + archive: string; + proposalPayloadHash: CheckpointProposalHash; + attestors: EthAddress[]; + } + | undefined + > { + const checkpoint = await this.archiver.getCheckpoint({ slot }); + if (!checkpoint) { + return undefined; } - const checkpoint = event.checkpoint; - - // Store mapping from slot to archive, checkpoint number, attestors, and the consensus payload - // hash (used to query matching p2p attestations regardless of feeAssetPriceModifier variants). const signatureContext = this.getSignatureContext(); const proposalPayloadHash = CheckpointProposalHash.fromBuffer( ConsensusPayload.fromCheckpoint(checkpoint.checkpoint, signatureContext).getPayloadHash(), ); - this.slotNumberToCheckpoint.set(checkpoint.checkpoint.header.slotNumber, { + return { checkpointNumber: checkpoint.checkpoint.number, archive: checkpoint.checkpoint.archive.root.toString(), proposalPayloadHash, attestors: getAttestationInfoFromPublishedCheckpoint(checkpoint, signatureContext) .filter(a => a.status === 'recovered-from-signature') .map(a => a.address!), - }); - - // Prune the archive map to only keep at most N entries - const historyLength = this.store.getHistoryLength(); - if (this.slotNumberToCheckpoint.size > historyLength) { - const toDelete = Array.from(this.slotNumberToCheckpoint.keys()) - .sort((a, b) => Number(a - b)) - .slice(0, this.slotNumberToCheckpoint.size - historyLength); - for (const key of toDelete) { - this.slotNumberToCheckpoint.delete(key); - } - } + }; } /** @@ -387,10 +352,6 @@ export class Sentinel extends (EventEmitter as new () => WatcherEmitter) impleme public async work() { const currentSlot = await this.getCurrentSlot(); try { - // Manually sync the block stream to ensure we have the latest data. - // Note we never `start` the blockstream, so it loops at the same pace as we do. - await this.blockStream.sync(); - // Per-slot activity recording (lag = 2 slots for P2P attestation settlement). const targetSlot = await this.isReadyToProcess(currentSlot); if (targetSlot !== false) { @@ -478,7 +439,7 @@ export class Sentinel extends (EventEmitter as new () => WatcherEmitter) impleme return false; } - const archiverLastBlockHash = await this.l2TipsStore.getL2Tips().then(tip => tip.proposed.hash); + const archiverLastBlockHash = await this.archiver.getL2Tips().then(tip => tip.proposed.hash); const p2pLastBlockHash = await this.p2p.getL2Tips().then(tips => tips.proposed.hash); const isP2pSynced = archiverLastBlockHash === p2pLastBlockHash; if (!isP2pSynced) { @@ -534,8 +495,9 @@ export class Sentinel extends (EventEmitter as new () => WatcherEmitter) impleme this.logger.debug(`Computing stats for slot ${slot} at epoch ${epoch}`, { slot, epoch, proposer, committee }); // Gather attestors from both p2p (live attestations) and the archiver (signers on the - // checkpoint if one has landed on L1). Used regardless of which case applies. - const checkpoint = this.slotNumberToCheckpoint.get(slot); + // checkpoint if one has landed on L1). Fetched on demand so it always reflects the canonical chain. + // Used regardless of which case applies. + const checkpoint = await this.getCheckpointForSlot(slot); const p2pAttested = await this.p2p.getCheckpointAttestationsForSlot(slot, checkpoint?.proposalPayloadHash); const p2pAttestors = p2pAttested.map(a => a.getSender()).filter((s): s is EthAddress => s !== undefined); const attestors = new Set( diff --git a/yarn-project/prover-node/src/prover-node.test.ts b/yarn-project/prover-node/src/prover-node.test.ts index b9e23f2846c0..1725c0c13f3e 100644 --- a/yarn-project/prover-node/src/prover-node.test.ts +++ b/yarn-project/prover-node/src/prover-node.test.ts @@ -77,6 +77,7 @@ describe('ProverNode', () => { // exercise. proverNode.setSessionManager(sessionManager); proverNode.setPublishingService(publishingService); + mined.clear(); }); // ---------------- event dispatch ---------------- @@ -87,14 +88,9 @@ describe('ProverNode', () => { checkpoint: { number: CheckpointNumber(n), hash: `0x0${n}` }, }); - it('dispatches chain-checkpointed to handleCheckpointEvent', async () => { + it('dispatches chain-checkpointed: catches up and registers the checkpoint', async () => { setupNotFullyProven(); - const checkpoint = makeCheckpoint(1, 1, 1); - const event: L2BlockStreamEvent = { - type: 'chain-checkpointed', - checkpoint: makePublishedCheckpoint(checkpoint), - block: { number: BlockNumber(1), hash: '0x01' }, - }; + const event = mineCheckpoint(makeCheckpoint(1, 1, 1)); await proverNode.handleBlockStreamEvent(event); @@ -102,6 +98,32 @@ describe('ProverNode', () => { expect(sessionManager.onCheckpointAdded).toHaveBeenCalledWith(EpochNumber(1)); }); + it('caps the catch-up fetch at two epochs when resyncing far behind the checkpointed tip', async () => { + // epochDuration=1 ⇒ two epochs' worth of checkpoints is 2. Cursor sits at checkpoint 0 while the + // checkpointed tip has jumped to 100 (e.g. the prover node was offline for a long time). We must not + // fetch all 100 checkpoints: epochs that far back are past their proof-submission window and cannot be + // proven anyway, so the catch-up should fetch only the most recent two epochs' worth (checkpoints 99 and + // 100) and skip the rest, leaving the cursor advanced past them so they are never retried. + setupNotFullyProven(); + const fetchSpy = l2BlockSource.getCheckpointsData; + mineCheckpoint(makeCheckpoint(99, 99, 99)); + const event = mineCheckpoint(makeCheckpoint(100, 100, 100)); + proverNode.setLastProcessedCheckpoint(CheckpointNumber.ZERO); + + await proverNode.handleBlockStreamEvent(event); + + // Only the most recent two epochs' worth were fetched and registered; the cursor lands at the tip. + const fetchRanges = fetchSpy.mock.calls.map(([q]) => q as any).filter(q => 'from' in q); + expect(fetchRanges).toEqual([{ from: CheckpointNumber(99), limit: 2 }]); + expect( + proverNode + .getCheckpointStore() + .listAll() + .map(p => p.id), + ).toHaveLength(2); + expect(proverNode.getLastProcessedCheckpoint()).toEqual(CheckpointNumber(100)); + }); + it('dispatches chain-pruned through markPrunedAfter and notifies the session manager only when affected', async () => { // No registered checkpoints — nothing to prune. await proverNode.handleBlockStreamEvent({ @@ -114,11 +136,7 @@ describe('ProverNode', () => { // Register a checkpoint, then prune. setupNotFullyProven(); - await proverNode.handleBlockStreamEvent({ - type: 'chain-checkpointed', - checkpoint: makePublishedCheckpoint(makeCheckpoint(2, 2, 2)), - block: { number: BlockNumber(2), hash: '0x02' }, - }); + await proverNode.handleBlockStreamEvent(mineCheckpoint(makeCheckpoint(2, 2, 2))); await proverNode.handleBlockStreamEvent({ type: 'chain-pruned', @@ -207,18 +225,16 @@ describe('ProverNode', () => { // the tips stay put for the L2BlockStream to retry. worldState.syncImmediate.mockRejectedValue(new Error('boom')); - const event: L2BlockStreamEvent = { - type: 'chain-checkpointed', - checkpoint: makePublishedCheckpoint(makeCheckpoint(1, 1, 1)), - block: { number: BlockNumber(1), hash: '0x01' }, - }; + const event = mineCheckpoint(makeCheckpoint(1, 1, 1)); await expect(proverNode.handleBlockStreamEvent(event)).rejects.toThrow('boom'); - // Tips left unadvanced; nothing was registered and the session manager wasn't notified. + // Tips left unadvanced; nothing was registered, the session manager wasn't notified, and the catch-up + // cursor stays behind so the next pass retries this checkpoint. expect(await proverNode.getTipsStore().getL2BlockHash(1)).toBeUndefined(); expect(proverNode.getCheckpointStore().listAll()).toHaveLength(0); expect(sessionManager.onCheckpointAdded).not.toHaveBeenCalled(); + expect(proverNode.getLastProcessedCheckpoint()).toEqual(CheckpointNumber.ZERO); }); it('leaves the tips store unadvanced when a handler propagates an error (A-1041)', async () => { @@ -227,11 +243,7 @@ describe('ProverNode', () => { // tips-store update, so the error surfaces to the L2BlockStream and the tips stay put. l2BlockSource.getSyncedL2SlotNumber.mockRejectedValue(new Error('archiver down')); - const event: L2BlockStreamEvent = { - type: 'chain-checkpointed', - checkpoint: makePublishedCheckpoint(makeCheckpoint(1, 1, 1)), - block: { number: BlockNumber(1), hash: '0x01' }, - }; + const event = mineCheckpoint(makeCheckpoint(1, 1, 1)); await expect(proverNode.handleBlockStreamEvent(event)).rejects.toThrow('archiver down'); @@ -251,25 +263,19 @@ describe('ProverNode', () => { ); l2BlockSource.isEpochComplete.mockResolvedValue(true); - await proverNode.handleBlockStreamEvent({ - type: 'chain-checkpointed', - checkpoint: makePublishedCheckpoint(makeCheckpoint(1, 1, 1)), - block: { number: BlockNumber(1), hash: '0x01' }, - }); + await proverNode.handleBlockStreamEvent(mineCheckpoint(makeCheckpoint(1, 1, 1))); expect(proverNode.getCheckpointStore().listAll().length).toBe(0); expect(sessionManager.onCheckpointAdded).not.toHaveBeenCalled(); + // The whole-epoch skip still advances the cursor so we don't re-evaluate it next pass. + expect(proverNode.getLastProcessedCheckpoint()).toEqual(CheckpointNumber(1)); }); it('content-addresses the prover by the checkpoint archive root', async () => { setupNotFullyProven(); const archiveRoot = Fr.random(); - await proverNode.handleBlockStreamEvent({ - type: 'chain-checkpointed', - checkpoint: makePublishedCheckpoint(makeCheckpoint(1, 1, 2, archiveRoot)), - block: { number: BlockNumber(2), hash: '0x02' }, - }); + await proverNode.handleBlockStreamEvent(mineCheckpoint(makeCheckpoint(1, 1, 2, archiveRoot))); const prover = proverNode.getCheckpointStore().listAll()[0]; expect(prover.id).toContain(archiveRoot.toString()); @@ -370,16 +376,8 @@ describe('ProverNode', () => { setupRegistrationSuccess(); // Register two checkpoints at slots 6 and 7 (both in epoch 3). - await proverNode.handleBlockStreamEvent({ - type: 'chain-checkpointed', - checkpoint: makePublishedCheckpoint(makeCheckpoint(1, 6, 6)), - block: { number: BlockNumber(6), hash: '0x06' }, - }); - await proverNode.handleBlockStreamEvent({ - type: 'chain-checkpointed', - checkpoint: makePublishedCheckpoint(makeCheckpoint(2, 7, 7)), - block: { number: BlockNumber(7), hash: '0x07' }, - }); + await proverNode.handleBlockStreamEvent(mineCheckpoint(makeCheckpoint(1, 6, 6))); + await proverNode.handleBlockStreamEvent(mineCheckpoint(makeCheckpoint(2, 7, 7))); expect(proverNode.getCheckpointStore().listAll().length).toBe(2); // Pruning above checkpoint 0 marks both as pruned — onPrune must receive [EpochNumber(3)], @@ -617,6 +615,44 @@ describe('ProverNode', () => { function makePublishedCheckpoint(checkpoint: Checkpoint): PublishedCheckpoint { return { checkpoint, attestations: [] } as unknown as PublishedCheckpoint; } + + /** Registry of mined checkpoints. */ + const mined = new Map(); + + /** + * Registers `checkpoint` with the block source mocks: its light metadata is returned by `getCheckpointsData` + * range queries, and its full payload by `getCheckpoint({ number })`. Returns the thin `chain-checkpointed` + * tip event that points at it — the block stream now delivers only the tip, and the prover-node fetches + * everything between its cursor and the tip itself. + */ + function mineCheckpoint(checkpoint: Checkpoint): L2BlockStreamEvent { + mined.set(Number(checkpoint.number), checkpoint); + l2BlockSource.getCheckpoint.mockImplementation((query: any) => + Promise.resolve('number' in query ? makeMaybePublished(mined.get(Number(query.number))) : undefined), + ); + l2BlockSource.getCheckpointsData.mockImplementation((query: any) => { + if (!('from' in query)) { + return Promise.resolve([]); + } + const data = []; + for (let n = Number(query.from); n < Number(query.from) + query.limit; n++) { + const cp = mined.get(n); + if (cp) { + data.push({ checkpointNumber: cp.number, header: cp.header } as any); + } + } + return Promise.resolve(data); + }); + return { + type: 'chain-checkpointed', + block: { number: BlockNumber(checkpoint.blocks[0].number), hash: '0x01' }, + checkpoint: { number: checkpoint.number, hash: checkpoint.hash().toString() }, + }; + } + + function makeMaybePublished(checkpoint: Checkpoint | undefined): PublishedCheckpoint | undefined { + return checkpoint ? makePublishedCheckpoint(checkpoint) : undefined; + } }); /** ProverNode subclass that exposes hooks for injecting a mocked SessionManager + reads. */ @@ -660,4 +696,12 @@ class TestProverNode extends ProverNode { public getLastExpiredEpoch(): EpochNumber | undefined { return this.lastExpiredEpoch; } + + public getLastProcessedCheckpoint(): CheckpointNumber { + return this.lastProcessedCheckpoint; + } + + public setLastProcessedCheckpoint(checkpoint: CheckpointNumber): void { + this.lastProcessedCheckpoint = checkpoint; + } } diff --git a/yarn-project/prover-node/src/prover-node.ts b/yarn-project/prover-node/src/prover-node.ts index 9e5b7fbd7508..296ec231a724 100644 --- a/yarn-project/prover-node/src/prover-node.ts +++ b/yarn-project/prover-node/src/prover-node.ts @@ -5,6 +5,7 @@ import { BlockNumber, CheckpointNumber, EpochNumber } from '@aztec/foundation/br import { assertRequired, compact, pick } from '@aztec/foundation/collection'; import { memoize } from '@aztec/foundation/decorators'; import { createLogger } from '@aztec/foundation/log'; +import { RunningPromise } from '@aztec/foundation/running-promise'; import { DateProvider, executeTimeout } from '@aztec/foundation/timer'; import type { EpochProverFactory } from '@aztec/prover-client'; import { getLastSiblingPath } from '@aztec/prover-client/helpers'; @@ -95,6 +96,17 @@ export class ProverNode implements L2BlockStreamEventHandler, ProverNodeApi, Tra */ protected lastExpiredEpoch: EpochNumber | undefined; + /** + * Highest checkpoint number whose proving-side handling has completed (or that was legitimately skipped). + * The catch-up loop walks from here to each `chain-checkpointed` tip event. Seeded at start() from the last + * checkpoint of the last fully-proven epoch (or 0), so a restart reprocesses the partially-proven epoch rather + * than trusting a checkpointed tip that may sit ahead of unproven checkpoints. Clamped down on a prune. + */ + protected lastProcessedCheckpoint: CheckpointNumber = CheckpointNumber.ZERO; + + /** Periodic tick that runs the epoch-expiry sweep during idle periods when no block-stream events arrive. */ + private expiryTicker: RunningPromise | undefined; + public readonly tracer: Tracer; protected publishingService: ProofPublishingService | undefined; @@ -219,7 +231,7 @@ export class ProverNode implements L2BlockStreamEventHandler, ProverNodeApi, Tra public async handleBlockStreamEvent(event: L2BlockStreamEvent): Promise { switch (event.type) { case 'chain-checkpointed': - await this.handleCheckpointEvent(event.checkpoint); + await this.processCheckpointJump(event.checkpoint.number); break; case 'chain-pruned': await this.handlePruneEvent(event.checkpointed.checkpoint); @@ -239,32 +251,83 @@ export class ProverNode implements L2BlockStreamEventHandler, ProverNodeApi, Tra await this.tipsStore.handleBlockStreamEvent(event); } - /** Register a new checkpoint with the store and notify the session manager. */ - private async handleCheckpointEvent(publishedCheckpoint: PublishedCheckpoint) { - const checkpoint = publishedCheckpoint.checkpoint; - const slotNumber = checkpoint.header.slotNumber; - const l1Constants = await this.getL1Constants(); - const epochNumber = getEpochAtSlot(slotNumber, l1Constants); - - if (await this.isEpochFullyProven(epochNumber, l1Constants)) { - this.log.debug(`Skipping checkpoint ${checkpoint.number} for already-proven epoch ${epochNumber}`); + /** + * Walks every checkpoint between the local cursor and the newly-reported checkpointed tip, registering + * each one that belongs to an epoch that can still be proven. The block stream now delivers a single thin + * `chain-checkpointed` tip event per pass rather than one fat event per checkpoint, so this drives the + * catch-up itself: light metadata first (`getCheckpointsData`) to decide relevance per epoch, then a heavy + * `getCheckpoints` fetch only for checkpoints in provable epochs. + * + * The cursor advances one checkpoint at a time and only after that checkpoint's proving-side handling has + * fully succeeded, preserving the A-1041 at-least-once semantics: a mid-jump failure leaves the cursor + * behind so the next pass retries from the first checkpoint that did not complete. + */ + private async processCheckpointJump(targetCheckpoint: CheckpointNumber): Promise { + if (targetCheckpoint <= this.lastProcessedCheckpoint) { return; } + const l1Constants = await this.getL1Constants(); - if (await this.isEpochPastProofSubmissionWindow(epochNumber, l1Constants)) { - this.log.debug( - `Skipping checkpoint ${checkpoint.number} for epoch ${epochNumber} past its proof-submission window`, - ); - return; + // Cap the catch-up at the two most recent epochs' worth of checkpoints. With at most one checkpoint per + // slot, an epoch spans at most `epochDuration` checkpoints, so two epochs is `2 * epochDuration`. When the + // cursor is much further behind (e.g. resyncing after a long time offline), fetching the whole gap could + // load thousands of checkpoints we cannot act on: anything older than the last two epochs is already past + // its proof-submission window, so we skip it and jump the cursor forward to the start of the capped range. + const maxCheckpoints = 2 * l1Constants.epochDuration; + let from = CheckpointNumber(this.lastProcessedCheckpoint + 1); + if (Number(targetCheckpoint - from) + 1 > maxCheckpoints) { + const cappedFrom = CheckpointNumber(targetCheckpoint - maxCheckpoints + 1); + this.log.warn(`Skipping unprovable checkpoints during catch-up; the prover node is far behind`, { + from, + cappedFrom, + targetCheckpoint, + maxCheckpoints, + }); + // Advance the cursor past the skipped checkpoints so they are never retried. + this.lastProcessedCheckpoint = CheckpointNumber(cappedFrom - 1); + from = cappedFrom; } + const limit = Number(targetCheckpoint - from) + 1; + const metadatas = await this.l2BlockSource.getCheckpointsData({ from, limit }); + + // Per-epoch relevance is cached so a multi-checkpoint epoch resolves it once. Skipping is whole-epoch + // only: the SessionManager requires an epoch's checkpoints fully covered before it opens a session, so we + // never drop an individual checkpoint inside an epoch we will prove. + const epochSkippable = new Map(); + for (const metadata of metadatas) { + const epochNumber = getEpochAtSlot(metadata.header.slotNumber, l1Constants); + let skippable = epochSkippable.get(epochNumber); + if (skippable === undefined) { + skippable = + (await this.isEpochFullyProven(epochNumber, l1Constants)) || + (await this.isEpochPastProofSubmissionWindow(epochNumber, l1Constants)); + epochSkippable.set(epochNumber, skippable); + } + if (skippable) { + this.log.debug(`Skipping checkpoint ${metadata.checkpointNumber} for unprovable epoch ${epochNumber}`); + } else { + await this.registerCheckpoint(metadata.checkpointNumber, epochNumber); + } + // Advance only after the checkpoint's handling succeeded (or it was legitimately skipped). registerCheckpoint + // throws on failure, which leaves the cursor here for the next pass to retry (A-1041). + this.lastProcessedCheckpoint = metadata.checkpointNumber; + } + } + /** Heavy-fetch a single checkpoint, register it with the store, and notify the session manager. */ + private async registerCheckpoint(checkpointNumber: CheckpointNumber, epochNumber: EpochNumber): Promise { + const published = await this.l2BlockSource.getCheckpoint({ number: checkpointNumber }); + if (!published) { + throw new Error(`Checkpoint ${checkpointNumber} not found in block source during catch-up`); + } + const checkpoint = published.checkpoint; this.log.info(`New checkpoint ${checkpoint.number} for epoch ${epochNumber}`, { checkpointNumber: checkpoint.number, epochNumber, - slotNumber, + slotNumber: checkpoint.header.slotNumber, }); - const registerData = await this.collectRegisterData(checkpoint, publishedCheckpoint.attestations); + const registerData = await this.collectRegisterData(checkpoint, published.attestations); await this.checkpointStore.addOrUpdate(checkpoint, registerData); await this.sessionManager?.onCheckpointAdded(epochNumber); } @@ -298,6 +361,11 @@ export class ProverNode implements L2BlockStreamEventHandler, ProverNodeApi, Tra /** Mark every prover above the prune threshold as pruned and notify the session manager. */ private async handlePruneEvent(prunedCheckpoint: { number: CheckpointNumber; hash: string }) { this.log.warn(`Chain pruned to checkpoint ${prunedCheckpoint.number}`, { prunedCheckpoint }); + // Clamp the catch-up cursor down to the (post-prune) checkpointed tip so reprocessing resumes from the + // first checkpoint above the prune target rather than from a stale, now-orphaned cursor. + if (this.lastProcessedCheckpoint > prunedCheckpoint.number) { + this.lastProcessedCheckpoint = prunedCheckpoint.number; + } const affected = this.checkpointStore.markPrunedAfter(prunedCheckpoint.number); if (affected.length === 0) { return; @@ -411,12 +479,22 @@ export class ProverNode implements L2BlockStreamEventHandler, ProverNodeApi, Tra const { startingBlock, lastFullyProvenEpoch } = await this.computeStartupState(); this.lastExpiredEpoch = lastFullyProvenEpoch; + this.lastProcessedCheckpoint = await this.computeStartingCheckpoint(lastFullyProvenEpoch); this.blockStream = new L2BlockStream(this.l2BlockSource, this.tipsStore, this, this.log, { pollIntervalMS: this.config.proverNodePollingIntervalMs, startingBlock, }); this.blockStream.start(); + // With thin once-per-pass tip events, the expiry sweep no longer fires once per checkpoint; drive it + // from a periodic tick so epochs still expire during idle/no-event periods. + this.expiryTicker = new RunningPromise( + () => this.checkEpochExpiry(), + this.log, + this.config.proverNodePollingIntervalMs, + ); + this.expiryTicker.start(); + await this.rewardsMetrics.start(); this.l1Metrics.start(); this.log.info(`Started Prover Node with prover id ${this.prover.getProverId().toString()}`, this.config); @@ -426,6 +504,7 @@ export class ProverNode implements L2BlockStreamEventHandler, ProverNodeApi, Tra this.log.info('Stopping ProverNode'); this.jobMetrics.stopObservingState(); await this.blockStream?.stop(); + await this.expiryTicker?.stop(); if (this.sessionManager) { await this.sessionManager.stop(); } @@ -586,6 +665,19 @@ export class ProverNode implements L2BlockStreamEventHandler, ProverNodeApi, Tra return { startingBlock: firstBlockOfEpoch, lastFullyProvenEpoch }; } + /** + * Resolves the catch-up cursor seed: the last checkpoint of the last fully-proven epoch, or 0 if none. Seeding + * from a checkpoint (rather than a checkpointed tip) guarantees a restart reprocesses every checkpoint of the + * partially-proven epoch, since the checkpointed tip can sit ahead of the last fully-proven checkpoint. + */ + protected async computeStartingCheckpoint(lastFullyProvenEpoch: EpochNumber | undefined): Promise { + if (lastFullyProvenEpoch === undefined) { + return CheckpointNumber.ZERO; + } + const checkpoints = await this.l2BlockSource.getCheckpointsData({ epoch: lastFullyProvenEpoch }); + return checkpoints.at(-1)?.checkpointNumber ?? CheckpointNumber.ZERO; + } + private async gatherPreviousBlockHeader(previousBlockNumber: number) { const data = await this.l2BlockSource.getBlockData({ number: BlockNumber(previousBlockNumber) }); if (!data?.header) { diff --git a/yarn-project/pxe/src/block_synchronizer/block_stream_source.ts b/yarn-project/pxe/src/block_synchronizer/block_stream_source.ts index 2009d350b2a1..cbd6ed780cb2 100644 --- a/yarn-project/pxe/src/block_synchronizer/block_stream_source.ts +++ b/yarn-project/pxe/src/block_synchronizer/block_stream_source.ts @@ -1,24 +1,15 @@ -import { Fr } from '@aztec/foundation/curves/bn254'; -import { - type BlockData, - type BlockQuery, - type BlocksQuery, - type CheckpointsQuery, - L2Block, - type L2BlockSource, -} from '@aztec/stdlib/block'; -import { Checkpoint, L1PublishedData, PublishedCheckpoint } from '@aztec/stdlib/checkpoint'; +import { type BlockData, type BlockQuery, type BlocksQuery, L2Block, type L2BlockSource } from '@aztec/stdlib/block'; import type { AztecNode } from '@aztec/stdlib/interfaces/client'; /** * Lifts an {@link AztecNode} RPC client into the shape {@link L2BlockStream} expects. - * `getBlocks` requests transaction bodies so that real `L2Block` instances can be constructed; - * `getCheckpoints` requests blocks + L1 info + attestations so that `PublishedCheckpoint` - * instances are fully populated. + * `getBlocks` requests transaction bodies so that real `L2Block` instances can be constructed. The stream no + * longer fetches checkpoint payloads (the `chain-checkpointed` event is a thin tip event), so `getCheckpoints` + * is not part of the lifted shape. */ export function blockStreamSourceFromAztecNode( node: AztecNode, -): Pick { +): Pick { return { getL2Tips: async () => { const tips = await node.getChainTips(); @@ -50,32 +41,5 @@ export function blockStreamSourceFromAztecNode( const responses = await node.getBlocks(query.from, query.limit, { includeTransactions: true }); return responses.map(r => new L2Block(r.archive, r.header, r.body!, r.checkpointNumber, r.indexWithinCheckpoint)); }, - - async getCheckpoints(query: CheckpointsQuery): Promise { - if (!('from' in query)) { - throw new Error('getCheckpoints with epoch query not supported via AztecNode RPC'); - } - const { from, limit } = query; - const responses = await node.getCheckpoints(from, limit, { - includeBlocks: true, - includeTransactions: true, - includeL1PublishInfo: true, - includeAttestations: true, - }); - return responses.map(r => { - const checkpoint = new Checkpoint( - r.archive, - r.header, - r.blocks!.map(b => new L2Block(b.archive, b.header, b.body!, b.checkpointNumber, b.indexWithinCheckpoint)), - r.number, - r.feeAssetPriceModifier, - ); - const l1 = - r.l1?.published === true - ? new L1PublishedData(r.l1.blockNumber, r.l1.timestamp, r.l1.blockHash) - : new L1PublishedData(0n, 0n, Fr.ZERO.toString()); - return new PublishedCheckpoint(checkpoint, l1, r.attestations ?? []); - }); - }, }; } diff --git a/yarn-project/pxe/src/block_synchronizer/block_synchronizer.test.ts b/yarn-project/pxe/src/block_synchronizer/block_synchronizer.test.ts index bbd6d2ae9729..c15898ba2109 100644 --- a/yarn-project/pxe/src/block_synchronizer/block_synchronizer.test.ts +++ b/yarn-project/pxe/src/block_synchronizer/block_synchronizer.test.ts @@ -17,7 +17,6 @@ import { makeL2BlockId, makeL2CheckpointId, } from '@aztec/stdlib/block'; -import { Checkpoint, L1PublishedData, PublishedCheckpoint } from '@aztec/stdlib/checkpoint'; import type { AztecNode, BlockResponse } from '@aztec/stdlib/interfaces/client'; import { NoteDao, NoteStatus } from '@aztec/stdlib/note'; import { TxHash } from '@aztec/stdlib/tx'; @@ -376,24 +375,53 @@ describe('BlockSynchronizer', () => { const initialBlock = await L2Block.random(BlockNumber(0)); await anchorBlockStore.setHeader(initialBlock.header); - // Create a checkpoint with a block + // The checkpointed tip block, fetched by hash from the node when the thin event arrives. const checkpointBlock = await L2Block.random(BlockNumber(1)); - const checkpoint = await Checkpoint.random(CheckpointNumber(1), { numBlocks: 1 }); - // Replace the random block with our known block - checkpoint.blocks[0] = checkpointBlock; - - const publishedCheckpoint = new PublishedCheckpoint(checkpoint, L1PublishedData.random(), []); + const checkpointBlockHash = await checkpointBlock.hash(); + aztecNode.getBlockData.mockImplementation(query => + Promise.resolve( + query instanceof BlockHash && query.equals(checkpointBlockHash) + ? ({ + header: checkpointBlock.header, + archive: checkpointBlock.archive, + blockHash: checkpointBlockHash, + checkpointNumber: checkpointBlock.checkpointNumber, + indexWithinCheckpoint: checkpointBlock.indexWithinCheckpoint, + } as BlockData) + : undefined, + ), + ); await synchronizer.handleBlockStreamEvent({ type: 'chain-checkpointed', - checkpoint: publishedCheckpoint, - block: { number: BlockNumber(1), hash: '0x456' }, + block: { number: BlockNumber(1), hash: checkpointBlockHash.toString() }, + checkpoint: makeL2CheckpointId(CheckpointNumber(1), Fr.random().toString()), }); const obtainedHeader = await anchorBlockStore.getBlockHeader(); expect(obtainedHeader.equals(checkpointBlock.header)).toBe(true); }); + it('skips the anchor update on chain-checkpointed when the block was reorged out (missing by hash)', async () => { + synchronizer = createSynchronizer({ syncChainTip: 'checkpointed' }); + + const initialBlock = await L2Block.random(BlockNumber(0)); + await anchorBlockStore.setHeader(initialBlock.header); + + // The node no longer serves the checkpointed block at that hash (transient reorg). + aztecNode.getBlockData.mockResolvedValue(undefined); + + await synchronizer.handleBlockStreamEvent({ + type: 'chain-checkpointed', + block: { number: BlockNumber(1), hash: Fr.random().toString() }, + checkpoint: makeL2CheckpointId(CheckpointNumber(1), Fr.random().toString()), + }); + + // Anchor is left untouched; a later event corrects it. + const obtainedHeader = await anchorBlockStore.getBlockHeader(); + expect(obtainedHeader.equals(initialBlock.header)).toBe(true); + }); + it('does not update anchor on chain-checkpointed when syncChainTip is proposed', async () => { synchronizer = createSynchronizer({ syncChainTip: 'proposed' }); @@ -401,14 +429,10 @@ describe('BlockSynchronizer', () => { const initialBlock = await L2Block.random(BlockNumber(1)); await synchronizer.handleBlockStreamEvent({ type: 'blocks-added', blocks: [initialBlock] }); - // Create a different checkpoint - const checkpoint = await Checkpoint.random(CheckpointNumber(1), { numBlocks: 1 }); - const publishedCheckpoint = new PublishedCheckpoint(checkpoint, L1PublishedData.random(), []); - await synchronizer.handleBlockStreamEvent({ type: 'chain-checkpointed', - checkpoint: publishedCheckpoint, block: { number: BlockNumber(1), hash: '0x456' }, + checkpoint: makeL2CheckpointId(CheckpointNumber(1), Fr.random().toString()), }); // Anchor should still be the initial block, not the checkpoint block diff --git a/yarn-project/pxe/src/block_synchronizer/block_synchronizer.ts b/yarn-project/pxe/src/block_synchronizer/block_synchronizer.ts index f0e0735f9a0d..3bfdc6141def 100644 --- a/yarn-project/pxe/src/block_synchronizer/block_synchronizer.ts +++ b/yarn-project/pxe/src/block_synchronizer/block_synchronizer.ts @@ -74,9 +74,17 @@ export class BlockSynchronizer implements L2BlockStreamEventHandler { } case 'chain-checkpointed': { if (this.config.syncChainTip === 'checkpointed') { - // Get the last block header from the checkpoint - const lastBlock = event.checkpoint.checkpoint.blocks.at(-1)!; - await this.updateAnchorBlockHeader(lastBlock.header); + // Fetch the checkpointed tip header by hash. By-hash is safer than by-number against a + // same-height reorg; a missing result means the block was reorged out between the event + // and this fetch, so we skip the anchor update and let a later event correct it. + const block = await this.node.getBlockData(BlockHash.fromString(event.block.hash)); + if (block) { + await this.updateAnchorBlockHeader(block.header); + } else { + this.log.warn( + `Block header not found for checkpointed block ${event.block.number}, skipping anchor update`, + ); + } } break; } diff --git a/yarn-project/pxe/src/storage/backwards_compatibility_tests/schema_tests.ts b/yarn-project/pxe/src/storage/backwards_compatibility_tests/schema_tests.ts index e6e6fc8ac71d..9b9f037de3e8 100644 --- a/yarn-project/pxe/src/storage/backwards_compatibility_tests/schema_tests.ts +++ b/yarn-project/pxe/src/storage/backwards_compatibility_tests/schema_tests.ts @@ -261,7 +261,10 @@ export const SCHEMA_TESTS: readonly SchemaTest[] = [ await l2TipsStore.handleBlockStreamEvent({ type: 'chain-checkpointed', block: { number: BlockNumber(71), hash: new Fr(73n).toString() }, - checkpoint: publishedCheckpoint, + checkpoint: { + number: publishedCheckpoint.checkpoint.number, + hash: publishedCheckpoint.checkpoint.hash().toString(), + }, }); // `'chain-proven'` writes the 'proven' tag. `'finalized'` is omitted because its handler runs delete-before // logic that would depend on the order of preceding events. diff --git a/yarn-project/stdlib/src/block/l2_block_stream/interfaces.ts b/yarn-project/stdlib/src/block/l2_block_stream/interfaces.ts index e9e13deca97a..6ed09b019739 100644 --- a/yarn-project/stdlib/src/block/l2_block_stream/interfaces.ts +++ b/yarn-project/stdlib/src/block/l2_block_stream/interfaces.ts @@ -1,6 +1,5 @@ import type { BlockNumber } from '@aztec/foundation/branded-types'; -import type { PublishedCheckpoint } from '../../checkpoint/published_checkpoint.js'; import type { L2Block } from '../l2_block.js'; import type { CheckpointId, L2BlockId, L2TipId, LocalL2Tips } from '../l2_block_source.js'; @@ -21,7 +20,7 @@ export type LocalL2BlockId = { number: BlockNumber; hash?: string }; */ export type LocalChainTips = { proposed: LocalL2BlockId; - checkpointed?: { checkpoint: CheckpointId }; + checkpointed?: { block: LocalL2BlockId; checkpoint: CheckpointId }; proven: { block: LocalL2BlockId }; finalized: { block: LocalL2BlockId }; }; @@ -46,10 +45,14 @@ export type L2BlockStreamEvent = type: 'blocks-added'; blocks: L2Block[]; } - | /** Emits checkpoints published to L1. */ { + | /** + * Reports a new checkpointed tip. Emitted at most once per sync pass when the source's checkpointed tip + * leads the local one. Carries only the block + checkpoint ids; consumers that need the full checkpoint + * payload fetch it on demand from the block source. + */ { type: 'chain-checkpointed'; - checkpoint: PublishedCheckpoint; block: L2BlockId; + checkpoint: CheckpointId; } | /** * Reports last correct block (new tip of the proposed chain). Note that this is not necessarily the anchor block diff --git a/yarn-project/stdlib/src/block/l2_block_stream/l2_block_stream.test.ts b/yarn-project/stdlib/src/block/l2_block_stream/l2_block_stream.test.ts index 1b17a151c4f5..a123e9160682 100644 --- a/yarn-project/stdlib/src/block/l2_block_stream/l2_block_stream.test.ts +++ b/yarn-project/stdlib/src/block/l2_block_stream/l2_block_stream.test.ts @@ -3,22 +3,14 @@ import { compactArray } from '@aztec/foundation/collection'; import { Fr } from '@aztec/foundation/curves/bn254'; import type { Logger } from '@aztec/foundation/log'; -import { jest } from '@jest/globals'; import { type MockProxy, mock } from 'jest-mock-extended'; import times from 'lodash.times'; -import type { PublishedCheckpoint } from '../../checkpoint/published_checkpoint.js'; import type { BlockHeader } from '../../tx/block_header.js'; import type { BlockData } from '../block_data.js'; -import { BlockHash, GENESIS_BLOCK_HEADER_HASH } from '../block_hash.js'; +import { BlockHash } from '../block_hash.js'; import type { L2Block } from '../l2_block.js'; -import { - type BlocksQuery, - GENESIS_CHECKPOINT_HEADER_HASH, - type L2BlockId, - type L2BlockSource, - type LocalL2Tips, -} from '../l2_block_source.js'; +import type { BlocksQuery, L2BlockId, L2BlockSource, LocalL2Tips } from '../l2_block_source.js'; import type { L2BlockStreamEvent, L2BlockStreamEventHandler, @@ -32,7 +24,6 @@ describe('L2BlockStream', () => { let blockSource: MockProxy; let latest: number = 0; - let checkpointed: number = 0; const makeHash = (number: number) => new Fr(number).toString(); @@ -43,15 +34,6 @@ describe('L2BlockStream', () => { indexWithinCheckpoint: 0, }) as L2Block; - /** Makes a block with hash method (for use in mocks that need hash) */ - const makeBlockWithHash = (number: number) => - ({ - number: BlockNumber(number), - checkpointNumber: CheckpointNumber(number), - indexWithinCheckpoint: 0, - hash: () => Promise.resolve(new BlockHash(new Fr(number))), - }) as L2Block; - const makeBlockData = (number: number, checkpointNum: number): BlockData => ({ header: makeHeader(number), @@ -64,29 +46,6 @@ describe('L2BlockStream', () => { const makeBlockId = (number: number): L2BlockId => ({ number: BlockNumber(number), hash: makeHash(number) }); - /** Helper to match a blocks-added event with blocks that may have extra properties like hash */ - const expectBlocksAdded = (blockNumbers: number[]) => - expect.objectContaining({ - type: 'blocks-added', - blocks: blockNumbers.map(n => - expect.objectContaining({ - number: BlockNumber(n), - }), - ), - }); - - /** Helper to match a chain-checkpointed event */ - const expectCheckpointed = (checkpointNumber?: number) => - checkpointNumber !== undefined - ? expect.objectContaining({ - type: 'chain-checkpointed', - checkpoint: expect.objectContaining({ - checkpoint: expect.objectContaining({ number: checkpointNumber }), - }), - block: expect.objectContaining({ number: expect.any(Number) }), - }) - : expect.objectContaining({ type: 'chain-checkpointed' }); - const makeCheckpointId = (number: number) => ({ number: CheckpointNumber(number), hash: makeHash(number) }); const makeTipId = (number: number) => ({ @@ -94,6 +53,13 @@ describe('L2BlockStream', () => { checkpoint: { number: CheckpointNumber(number), hash: makeHash(number) }, }); + /** A thin chain-checkpointed event for the source's checkpointed tip at `number`. */ + const checkpointedEvent = (number: number): L2BlockStreamEvent => ({ + type: 'chain-checkpointed', + block: makeBlockId(number), + checkpoint: makeCheckpointId(number), + }); + /** Sets the remote tips. All tips default to 0 except latest. */ const setRemoteTips = ( latest_: number, @@ -107,7 +73,6 @@ describe('L2BlockStream', () => { proven = proven ?? 0; finalized = finalized ?? 0; latest = latest_; - checkpointed = checkpointed_; blockSource.getL2Tips.mockResolvedValue({ proposed: { number: BlockNumber(latest), hash: makeHash(latest) }, @@ -121,7 +86,7 @@ describe('L2BlockStream', () => { beforeEach(() => { blockSource = mock(); - // Returns blocks up until what was reported as the latest block (for uncheckpointed blocks) + // Returns blocks up until what was reported as the latest block. blockSource.getBlocks.mockImplementation((query: BlocksQuery) => 'from' in query ? Promise.resolve( @@ -137,31 +102,6 @@ describe('L2BlockStream', () => { } return Promise.resolve(query.number > latest ? undefined : makeBlockData(query.number, query.number)); }); - - // Returns published checkpoints - each checkpoint contains just the one block for simplicity - // Respects the limit parameter and returns up to `limit` checkpoints - blockSource.getCheckpoints.mockImplementation(query => { - if (!('from' in query)) { - return Promise.resolve([]); - } - const { from: checkpointNumber, limit } = query; - return Promise.resolve( - compactArray( - times(limit, i => { - const cpNum = checkpointNumber + i; - return cpNum > checkpointed - ? undefined - : ({ - checkpoint: { - number: cpNum, - hash: () => new Fr(cpNum), - blocks: [makeBlockWithHash(cpNum)], - }, - } as unknown as PublishedCheckpoint); - }), - ), - ); - }); }); describe('with mock local data provider', () => { @@ -172,10 +112,7 @@ describe('L2BlockStream', () => { beforeEach(() => { localData = new TestL2BlockStreamLocalDataProvider(); handler = new TestL2BlockStreamEventHandler(); - blockStream = new TestL2BlockStream(blockSource, localData, handler, undefined, { - batchSize: 10, - checkpointPrefetchLimit: 1, - }); + blockStream = new TestL2BlockStream(blockSource, localData, handler, undefined, { batchSize: 10 }); }); it('pulls new blocks from start', async () => { @@ -267,50 +204,41 @@ describe('L2BlockStream', () => { ] satisfies L2BlockStreamEvent[]); }); - it('fetches checkpointed blocks and emits chain-checkpointed events', async () => { - // All blocks are checkpointed (checkpointed=5, proposed=5) + it('emits a single chain-checkpointed event carrying the source checkpointed tip', async () => { + // All blocks are checkpointed (checkpointed=5, proposed=5). Download all 5 blocks in one batch, then a + // single thin checkpointed event for the source tip. setRemoteTips(5, 5); await blockStream.work(); - // Each checkpointed block triggers a blocks-added and chain-checkpointed event - // (since each checkpoint contains one block in our mock) expect(handler.events).toEqual([ - expectBlocksAdded([1]), - expectCheckpointed(), - expectBlocksAdded([2]), - expectCheckpointed(), - expectBlocksAdded([3]), - expectCheckpointed(), - expectBlocksAdded([4]), - expectCheckpointed(), - expectBlocksAdded([5]), - expectCheckpointed(), + { type: 'blocks-added', blocks: times(5, i => makeBlock(i + 1)) }, + checkpointedEvent(5), ]); - // 2 calls: one for block 0 in reorg detection (hash compare at genesis), one for block 1 in loop 2. - expect(blockSource.getBlockData).toHaveBeenCalledTimes(2); - expect(blockSource.getBlocks).not.toHaveBeenCalled(); + // No checkpoint payloads are fetched anymore. + expect(blockSource.getBlocks).toHaveBeenCalledWith({ from: BlockNumber(1), limit: 5 }); }); - it('fetches checkpointed blocks first, then uncheckpointed blocks', async () => { - // Blocks 1-3 are checkpointed, blocks 4-5 are uncheckpointed + it('emits checkpointed once even when the checkpointed tip trails the proposed tip', async () => { + // Blocks 1-3 checkpointed, blocks 4-5 uncheckpointed. setRemoteTips(5, 3); await blockStream.work(); - // First 3 blocks come via checkpoints, last 2 via getBlocks + // Download all 5 blocks, then a single checkpointed event for checkpoint 3. expect(handler.events).toEqual([ - expectBlocksAdded([1]), - expectCheckpointed(), - expectBlocksAdded([2]), - expectCheckpointed(), - expectBlocksAdded([3]), - expectCheckpointed(), - expectBlocksAdded([4, 5]), + { type: 'blocks-added', blocks: times(5, i => makeBlock(i + 1)) }, + checkpointedEvent(3), ]); - // 2 calls: one for block 0 in reorg detection (hash compare at genesis), one for block 1 in loop 2. - expect(blockSource.getBlockData).toHaveBeenCalledTimes(2); - expect(blockSource.getBlocks).toHaveBeenCalledWith({ from: BlockNumber(4), limit: 2 }); + }); + + it('does not re-emit the checkpointed event once the local tip matches the source', async () => { + setRemoteTips(5, 5); + localData.setProposed(5); + localData.setCheckpointed(5, 5); + + await blockStream.work(); + expect(handler.events.filter(e => e.type === 'chain-checkpointed')).toEqual([]); }); it('handles reorg with uncheckpointed reason when pruned to checkpointed tip', async () => { @@ -346,22 +274,60 @@ describe('L2BlockStream', () => { // The reorg-search loop must NOT walk past block 0; it should throw a clear error // pointing at the genesis-hash mismatch instead of cascading into "block hash not found - // for -1" further down. The error is caught and logged by `work` rather than rethrown, - // so we assert via the logged error and ensure no events were emitted. - const errorSpy = jest.spyOn( - (blockStream as unknown as { log: { error: (...args: any[]) => void } }).log, - 'error', - ); + // for -1" further down. The error is caught and logged by `work` rather than rethrown. + const log = mock(); + blockStream = new TestL2BlockStream(blockSource, localData, handler, log, { batchSize: 10 }); await blockStream.work(); expect(handler.events).toEqual([]); - expect(errorSpy).toHaveBeenCalledWith( - expect.stringContaining('Error processing block stream'), - expect.objectContaining({ - message: expect.stringContaining('Genesis block hash mismatch'), - }), - ); + expect(log.error).toHaveBeenCalledWith(expect.stringContaining('Genesis block hash mismatch'), expect.anything()); + }); + }); + + describe('A-1061 regression: startingBlock past the source checkpointed tip', () => { + let localData: TestL2BlockStreamLocalDataProvider; + let handler: TestL2BlockStreamEventHandler; + let blockStream: TestL2BlockStream; + + beforeEach(() => { + localData = new TestL2BlockStreamLocalDataProvider(); + handler = new TestL2BlockStreamEventHandler(); + }); + + it('emits the source checkpointed tip on the FIRST pass even when startingBlock is past it', async () => { + // Source: checkpointed=30, proven=25, proposed=35. The node restarts with startingBlock=33 (past the + // checkpointed tip of block 30). Pre-rewrite, the startingBlock fast-forward suppressed all checkpoint + // emission, leaving the local checkpointed cursor stuck at genesis while proven still advanced. + setRemoteTips(35, 30, 25, 10); + blockStream = new TestL2BlockStream(blockSource, localData, handler, undefined, { + batchSize: 10, + startingBlock: 33, + }); + + await blockStream.work(); + + const checkpointEvents = handler.events.filter(e => e.type === 'chain-checkpointed'); + expect(checkpointEvents).toEqual([checkpointedEvent(30)]); + + // proven is resolvable (block 25), and the checkpointed cursor is NOT stuck at genesis. + const provenEvents = handler.events.filter(e => e.type === 'chain-proven'); + expect(provenEvents).toEqual([ + { type: 'chain-proven', block: makeBlockId(25), checkpoint: makeCheckpointId(25) }, + ]); + }); + + it('downloads blocks from startingBlock, not from genesis', async () => { + setRemoteTips(35, 30, 25, 10); + blockStream = new TestL2BlockStream(blockSource, localData, handler, undefined, { + batchSize: 10, + startingBlock: 33, + }); + + await blockStream.work(); + + // First block download begins at startingBlock (33), skipping 1..32. + expect(blockSource.getBlocks).toHaveBeenCalledWith({ from: BlockNumber(33), limit: 3 }); }); }); @@ -384,13 +350,12 @@ describe('L2BlockStream', () => { startingBlock: 30, }); - // We first seed a few blocks into the blockstream - // Block 30 comes via checkpoint, blocks 31-35 via uncheckpointed + // We first seed a few blocks into the blockstream: blocks 30-35 (startingBlock=30), plus the + // checkpointed/proven/finalized reconciliation events. await blockStream.work(); expect(handler.events).toEqual([ - expectBlocksAdded([30]), - expectCheckpointed(30), - { type: 'blocks-added', blocks: times(5, i => makeBlock(i + 31)) }, + { type: 'blocks-added', blocks: times(6, i => makeBlock(i + 30)) }, + checkpointedEvent(30), { type: 'chain-proven', block: makeBlockId(25), checkpoint: makeCheckpointId(25) }, { type: 'chain-finalized', block: makeBlockId(10), checkpoint: makeCheckpointId(10) }, ]); @@ -404,19 +369,16 @@ describe('L2BlockStream', () => { ]); }); - // Regression test for the checkpoint-replay storm: pruning to an uncheckpointed block ahead of - // the checkpointed tip must not reset the checkpointed cursor, otherwise the next work() replays - // every checkpoint from 1 to the source tip. - it('does not replay checkpoints after pruning to an uncheckpointed block ahead of the checkpointed tip', async () => { - // Sync blocks 1-7: blocks 1-5 are checkpointed (checkpoints 1-5), blocks 6-7 uncheckpointed. + // Regression: pruning to an uncheckpointed block ahead of the checkpointed tip must not reset the + // checkpointed cursor, otherwise the next work() re-emits the checkpointed tip. + it('does not re-emit the checkpointed tip after pruning to a block ahead of it', async () => { + // Sync blocks 1-7: blocks 1-5 are checkpointed (checkpoint 5), blocks 6-7 uncheckpointed. setRemoteTips(7, 5); await blockStream.work(); - const checkpointEventsOnSync = handler.events.filter(e => e.type === 'chain-checkpointed'); - expect(checkpointEventsOnSync).toHaveLength(5); + expect(handler.events.filter(e => e.type === 'chain-checkpointed')).toEqual([checkpointedEvent(5)]); handler.clearEvents(); // Source drops its proposed tip to block 6 (uncheckpointed, still ahead of checkpointed=5). - // The stream prunes the local proposed tip from 7 back to 6. setRemoteTips(6, 5); await blockStream.work(); expect(handler.events).toEqual([ @@ -424,1231 +386,161 @@ describe('L2BlockStream', () => { ]); handler.clearEvents(); - // The next sync must NOT re-emit any chain-checkpointed events: the checkpointed cursor was - // left at block 5 / checkpoint 5, so there is nothing to replay. + // The next sync must NOT re-emit a chain-checkpointed event: the checkpointed cursor was left at + // block 5 / checkpoint 5. await blockStream.work(); expect(handler.events.filter(e => e.type === 'chain-checkpointed')).toEqual([]); }); - }); - - describe('multiple blocks per checkpoint', () => { - let localData: TestL2BlockStreamLocalDataProvider; - let handler: TestL2BlockStreamEventHandler; - let blockStream: TestL2BlockStream; - - // Configuration for checkpoint structure: each checkpoint contains 3 blocks - const blocksPerCheckpoint = 3; - - /** Gets the checkpoint number for a given block number */ - const getCheckpointForBlock = (blockNum: number) => Math.ceil(blockNum / blocksPerCheckpoint); - - /** Gets the first block number in a checkpoint */ - const getFirstBlockInCheckpoint = (checkpointNum: number) => (checkpointNum - 1) * blocksPerCheckpoint + 1; - - /** Gets the last block number in a checkpoint */ - const getLastBlockInCheckpoint = (checkpointNum: number) => checkpointNum * blocksPerCheckpoint; - - /** Makes a block with hash method (for use in mocks that need hash) */ - const makeBlockInCheckpointWithHash = (blockNum: number) => { - const checkpointNum = getCheckpointForBlock(blockNum); - const firstBlockInCheckpoint = getFirstBlockInCheckpoint(checkpointNum); - return { - number: BlockNumber(blockNum), - checkpointNumber: CheckpointNumber(checkpointNum), - indexWithinCheckpoint: blockNum - firstBlockInCheckpoint, - hash: () => Promise.resolve(new BlockHash(new Fr(blockNum))), - } as L2Block; - }; - - /** Makes block data for a checkpointed block */ - const makeBlockDataInCheckpoint = (blockNum: number): BlockData => - ({ - header: makeHeader(blockNum), - checkpointNumber: CheckpointNumber(getCheckpointForBlock(blockNum)), - indexWithinCheckpoint: blockNum - getFirstBlockInCheckpoint(getCheckpointForBlock(blockNum)), - }) as unknown as BlockData; - - /** Sets the remote tips with correct checkpoint numbers for multi-block checkpoints. */ - const setRemoteTipsMultiBlock = ( - latest_: number, - checkpointedBlock?: number, - proven?: number, - finalized?: number, - proposedCheckpointBlock?: number, - ) => { - checkpointedBlock = checkpointedBlock ?? 0; - proven = proven ?? 0; - finalized = finalized ?? 0; - proposedCheckpointBlock = proposedCheckpointBlock ?? 0; - latest = latest_; - checkpointed = checkpointedBlock; - - const checkpointedCheckpointNum = checkpointedBlock > 0 ? getCheckpointForBlock(checkpointedBlock) : 0; - const provenCheckpointNum = proven > 0 ? getCheckpointForBlock(proven) : 0; - const finalizedCheckpointNum = finalized > 0 ? getCheckpointForBlock(finalized) : 0; - const proposedCheckpointNum = proposedCheckpointBlock > 0 ? getCheckpointForBlock(proposedCheckpointBlock) : 0; - - blockSource.getL2Tips.mockResolvedValue({ - proposed: { number: BlockNumber(latest), hash: makeHash(latest) }, - checkpointed: { - block: { number: BlockNumber(checkpointedBlock), hash: makeHash(checkpointedBlock) }, - checkpoint: { - number: CheckpointNumber(checkpointedCheckpointNum), - hash: makeHash(checkpointedCheckpointNum), - }, - }, - proposedCheckpoint: { - block: { number: BlockNumber(proposedCheckpointBlock), hash: makeHash(proposedCheckpointBlock) }, - checkpoint: { - number: CheckpointNumber(proposedCheckpointNum), - hash: makeHash(proposedCheckpointNum), - }, - }, - proven: { - block: { number: BlockNumber(proven), hash: makeHash(proven) }, - checkpoint: { number: CheckpointNumber(provenCheckpointNum), hash: makeHash(provenCheckpointNum) }, - }, - finalized: { - block: { number: BlockNumber(finalized), hash: makeHash(finalized) }, - checkpoint: { number: CheckpointNumber(finalizedCheckpointNum), hash: makeHash(finalizedCheckpointNum) }, - }, - }); - }; - - beforeEach(() => { - localData = new TestL2BlockStreamLocalDataProvider(); - handler = new TestL2BlockStreamEventHandler(); - blockStream = new TestL2BlockStream(blockSource, localData, handler, undefined, { batchSize: 10 }); - - // Override the mock to support multiple blocks per checkpoint - blockSource.getBlockData.mockImplementation(query => { - if (!('number' in query)) { - return Promise.resolve(undefined); - } - return Promise.resolve(query.number > latest ? undefined : makeBlockDataInCheckpoint(query.number)); - }); - - // Returns published checkpoints with multiple blocks each, respecting the limit parameter - blockSource.getCheckpoints.mockImplementation(query => { - if (!('from' in query)) { - return Promise.resolve([]); - } - const { from: checkpointNumber, limit } = query; - const checkpoints: PublishedCheckpoint[] = []; - for (let i = 0; i < limit; i++) { - const cpNum = CheckpointNumber(checkpointNumber + i); - const firstBlock = getFirstBlockInCheckpoint(cpNum); - const lastBlock = getLastBlockInCheckpoint(cpNum); - // Only include checkpoints that are within the checkpointed range - if (lastBlock > checkpointed) { - break; - } - checkpoints.push({ - checkpoint: { - number: cpNum, - hash: () => new Fr(cpNum), - blocks: times(blocksPerCheckpoint, j => makeBlockInCheckpointWithHash(firstBlock + j)), - }, - } as unknown as PublishedCheckpoint); - } - return Promise.resolve(checkpoints); - }); - }); - - it('emits all blocks in a checkpoint before chain-checkpointed event', async () => { - // Set up: 6 blocks in 2 checkpoints (blocks 1-3 in checkpoint 1, blocks 4-6 in checkpoint 2) - setRemoteTipsMultiBlock(6, 6); + // prune + same-pass reconciliation: a prune walk-back and the catch-up tier events fire in one pass. + it('emits the prune event and the new tier events in the same pass', async () => { + // Sync up to a checkpointed/proven chain: proposed=9, checkpointed=9, proven=6, finalized=3. + setRemoteTips(9, 9, 6, 3); await blockStream.work(); + handler.clearEvents(); - // Should emit blocks 1-3, then checkpoint 1, then blocks 4-6, then checkpoint 2 - expect(handler.events).toEqual([ - expectBlocksAdded([1, 2, 3]), - expectCheckpointed(1), - expectBlocksAdded([4, 5, 6]), - expectCheckpointed(2), - ]); - }); - - it('handles partial checkpoint at the end (uncheckpointed blocks)', async () => { - // Set up: 5 blocks total, but only first 3 are checkpointed (checkpoint 1 complete) - // Blocks 4-5 are uncheckpointed - setRemoteTipsMultiBlock(5, 3); - + // Reorg: the source drops its proposed/checkpointed tip to block 6 (the memory store still holds 7-9, + // which the source no longer serves) and finalized advances to 6 within the same snapshot. + setRemoteTips(6, 6, 6, 6); await blockStream.work(); - // Should emit checkpoint 1 blocks, then checkpoint event, then uncheckpointed blocks 4-5 - expect(handler.events).toEqual([expectBlocksAdded([1, 2, 3]), expectCheckpointed(1), expectBlocksAdded([4, 5])]); + // First the prune to block 6, then the finalized reconciliation event for the advanced finalized tip. + expect(handler.events[0]).toEqual({ + type: 'chain-pruned', + block: makeBlockId(6), + checkpointed: makeTipId(6), + proven: makeTipId(6), + }); + const finalizedEvents = handler.events.filter(e => e.type === 'chain-finalized'); + expect(finalizedEvents).toEqual([ + { type: 'chain-finalized', block: makeBlockId(6), checkpoint: makeCheckpointId(6) }, + ]); }); + }); - it('handles starting from middle of a checkpoint', async () => { - // Set up: 9 blocks in 3 checkpoints, but we start from block 5 (middle of checkpoint 2) - // Local has blocks 1-4, local checkpointed = 0 - setRemoteTipsMultiBlock(9, 9); - localData.proposed.number = BlockNumber(4); + describe('hash-gated tier reconciliation', () => { + // World-state-shaped provider: reports `undefined` block hashes for its proven/finalized tips. The + // reconciliation must skip the hash comparison so it does not re-emit on every poll. + class WorldStateShapedProvider implements L2BlockStreamLocalDataProvider { + public proposedNumber = BlockNumber.ZERO; + public provenNumber = BlockNumber.ZERO; + public finalizedNumber = BlockNumber.ZERO; - await blockStream.work(); + public getL2BlockHash(number: number): Promise { + return Promise.resolve(number > this.proposedNumber ? undefined : new Fr(number).toString()); + } - // Should first emit checkpoint 1 (blocks 1-4 already local) - // Then continue from block 5, which is in checkpoint 2 - // Blocks 5-6 complete checkpoint 2, then blocks 7-9 complete checkpoint 3 - expect(handler.events).toEqual([ - expectCheckpointed(1), // checkpoint 1 for already-local blocks 1-4 - expectBlocksAdded([5, 6]), - expectCheckpointed(2), // checkpoint 2 - expectBlocksAdded([7, 8, 9]), - expectCheckpointed(3), // checkpoint 3 - ]); + public getL2Tips(): Promise { + return Promise.resolve({ + proposed: { number: this.proposedNumber, hash: new Fr(this.proposedNumber).toString() }, + // proven/finalized hashes are intentionally undefined, as world-state reports them. + proven: { block: { number: this.provenNumber } }, + finalized: { block: { number: this.finalizedNumber } }, + }); + } + } - // Verify checkpoint order - const checkpointEvents = handler.events.filter(e => e.type === 'chain-checkpointed'); - expect(checkpointEvents).toHaveLength(3); - expect((checkpointEvents[0] as any).checkpoint.checkpoint.number).toBe(CheckpointNumber(1)); - expect((checkpointEvents[1] as any).checkpoint.checkpoint.number).toBe(CheckpointNumber(2)); - expect((checkpointEvents[2] as any).checkpoint.checkpoint.number).toBe(CheckpointNumber(3)); - }); + it('does not re-emit proven/finalized when the local hash is undefined and numbers match', async () => { + const localData = new WorldStateShapedProvider(); + const handler = new TestL2BlockStreamEventHandler(); + const blockStream = new TestL2BlockStream(blockSource, localData, handler, undefined, { + batchSize: 10, + ignoreCheckpoints: true, + }); - it('correctly identifies checkpoint number in chain-checkpointed events', async () => { - // Set up: 6 blocks in 2 checkpoints - setRemoteTipsMultiBlock(6, 6); + // Source proven/finalized are at the same numbers the local provider already tracks. + setRemoteTips(9, 0, 6, 3); + localData.proposedNumber = BlockNumber(9); + localData.provenNumber = BlockNumber(6); + localData.finalizedNumber = BlockNumber(3); await blockStream.work(); - // Extract the chain-checkpointed events - const checkpointEvents = handler.events.filter(e => e.type === 'chain-checkpointed'); - expect(checkpointEvents).toHaveLength(2); - expect((checkpointEvents[0] as any).checkpoint.checkpoint.number).toBe(CheckpointNumber(1)); - expect((checkpointEvents[1] as any).checkpoint.checkpoint.number).toBe(CheckpointNumber(2)); + // Numbers match and local hashes are undefined ⇒ no re-emission. + expect(handler.events.filter(e => e.type === 'chain-proven')).toEqual([]); + expect(handler.events.filter(e => e.type === 'chain-finalized')).toEqual([]); }); - it('handles many checkpoints with batching', async () => { - // Set up: 12 blocks in 4 checkpoints (3 blocks each), with batch size of 5 - // Batch size doesn't align with checkpoint boundaries, so the stream must - // respect checkpoint boundaries and emit events correctly - blockStream = new TestL2BlockStream(blockSource, localData, handler, undefined, { batchSize: 5 }); - setRemoteTipsMultiBlock(12, 12); - - await blockStream.work(); - - // Even though batch size is 5, checkpoint boundaries (every 3 blocks) take precedence - // Expected sequence: - // - Blocks 1-3 (checkpoint 1), then checkpoint 1 event - // - Blocks 4-6 (checkpoint 2), then checkpoint 2 event - // - Blocks 7-9 (checkpoint 3), then checkpoint 3 event - // - Blocks 10-12 (checkpoint 4), then checkpoint 4 event - expect(handler.events).toEqual([ - expectBlocksAdded([1, 2, 3]), - expectCheckpointed(1), - expectBlocksAdded([4, 5, 6]), - expectCheckpointed(2), - expectBlocksAdded([7, 8, 9]), - expectCheckpointed(3), - expectBlocksAdded([10, 11, 12]), - expectCheckpointed(4), - ]); - }); + // Finding 3: a same-number, different-hash proven tip IS re-emitted. + it('re-emits the proven tip when numbers match but the known local hash differs', async () => { + const localData = new TestL2BlockStreamLocalDataProvider(); + const handler = new TestL2BlockStreamEventHandler(); + const blockStream = new TestL2BlockStream(blockSource, localData, handler, undefined, { + batchSize: 10, + ignoreCheckpoints: true, + }); - it('does not emit more than batchSize blocks in a single blocks-added event for checkpointed blocks', async () => { - // Set up: 12 blocks in 4 checkpoints (3 blocks each), but batch size is 2 - // Each checkpoint has 3 blocks, but we should never emit more than 2 at once - blockStream = new TestL2BlockStream(blockSource, localData, handler, undefined, { batchSize: 2 }); - setRemoteTipsMultiBlock(12, 12); + setRemoteTips(9, 0, 6, 3); + localData.proposed.number = BlockNumber(9); + // Local proven sits at the same block number but a stale hash (e.g. a same-height reorg). + localData.proven.block.number = BlockNumber(6); + localData.proven.block.hash = '0xstale6'; + localData.finalized.block.number = BlockNumber(3); + localData.finalized.block.hash = makeHash(3); await blockStream.work(); - // Verify no blocks-added event has more than 2 blocks - const blocksAddedEvents = handler.events.filter(e => e.type === 'blocks-added'); - for (const event of blocksAddedEvents) { - if (event.type === 'blocks-added') { - expect(event.blocks.length).toBeLessThanOrEqual(2); - } - } - - // Expected sequence with batchSize=2: - // - Blocks 1-2, then blocks 3, then checkpoint 1 - // - Blocks 4-5, then block 6, then checkpoint 2 - // etc. - expect(handler.events).toEqual([ - expectBlocksAdded([1, 2]), - expectBlocksAdded([3]), - expectCheckpointed(1), - expectBlocksAdded([4, 5]), - expectBlocksAdded([6]), - expectCheckpointed(2), - expectBlocksAdded([7, 8]), - expectBlocksAdded([9]), - expectCheckpointed(3), - expectBlocksAdded([10, 11]), - expectBlocksAdded([12]), - expectCheckpointed(4), + expect(handler.events.filter(e => e.type === 'chain-proven')).toEqual([ + { type: 'chain-proven', block: makeBlockId(6), checkpoint: makeCheckpointId(6) }, ]); + // Finalized matched on both number and hash ⇒ not re-emitted. + expect(handler.events.filter(e => e.type === 'chain-finalized')).toEqual([]); }); + }); - it('emits checkpoint event when blocks become checkpointed after being added as uncheckpointed', async () => { - // Phase 1: Start with 3 checkpointed blocks (checkpoint 1), then add blocks 4-6 as uncheckpointed - setRemoteTipsMultiBlock(6, 3); - - await blockStream.work(); - - // Expect: blocks 1-3 via checkpoint, then uncheckpointed blocks 4-6 - expect(handler.events).toEqual([ - expectBlocksAdded([1, 2, 3]), - expectCheckpointed(1), - expectBlocksAdded([4, 5, 6]), - ]); - - handler.clearEvents(); - - // Update local state to reflect what the handler would have stored - localData.proposed.number = BlockNumber(6); - localData.checkpointed.block.number = BlockNumber(3); - localData.checkpointed.checkpoint.number = CheckpointNumber(1); - - // Phase 2: Now checkpoint 2 completes (blocks 4-6 become checkpointed) - setRemoteTipsMultiBlock(6, 6); - - await blockStream.work(); + describe('ignoreCheckpoints', () => { + let localData: TestL2BlockStreamLocalDataProvider; + let handler: TestL2BlockStreamEventHandler; + let blockStream: TestL2BlockStream; - // Should emit a checkpoint event for checkpoint 2 (blocks 4-6), even though blocks were already added - const checkpointEvents = handler.events.filter(e => e.type === 'chain-checkpointed'); - expect(checkpointEvents).toHaveLength(1); - expect((checkpointEvents[0] as any).checkpoint.checkpoint.number).toBe(CheckpointNumber(2)); + beforeEach(() => { + localData = new TestL2BlockStreamLocalDataProvider(); + handler = new TestL2BlockStreamEventHandler(); + blockStream = new TestL2BlockStream(blockSource, localData, handler, undefined, { + batchSize: 10, + ignoreCheckpoints: true, + }); }); - it('emits checkpoint event BEFORE new uncheckpointed blocks when checkpoint completes', async () => { - // Phase 1: Start with 3 checkpointed blocks (checkpoint 1), then add blocks 4-6 as uncheckpointed - setRemoteTipsMultiBlock(6, 3); - - await blockStream.work(); - - // Expect: blocks 1-3 via checkpoint, then uncheckpointed blocks 4-6 - expect(handler.events).toEqual([ - expectBlocksAdded([1, 2, 3]), - expectCheckpointed(1), - expectBlocksAdded([4, 5, 6]), - ]); - - handler.clearEvents(); - - // Update local state to reflect what the handler would have stored - localData.proposed.number = BlockNumber(6); - localData.checkpointed.block.number = BlockNumber(3); - localData.checkpointed.checkpoint.number = CheckpointNumber(1); - - // Phase 2: Checkpoint 2 completes (blocks 4-6) AND a new block 7 arrives - setRemoteTipsMultiBlock(7, 6); + it('does not emit checkpoint events for new checkpointed blocks', async () => { + setRemoteTips(6, 6); await blockStream.work(); - // Should emit checkpoint 2 FIRST, then the new uncheckpointed block 7 - // NOT: block 7 first, then checkpoint 2 - expect(handler.events).toEqual([expectCheckpointed(2), expectBlocksAdded([7])]); + expect(handler.events).toEqual([{ type: 'blocks-added', blocks: times(6, i => makeBlock(i + 1)) }]); }); - it('emits checkpoint as soon as last block in checkpoint arrives', async () => { - // This tests the realistic scenario where checkpoints are published as blocks arrive. - // Uncheckpointed blocks are always just a partial checkpoint (the current incomplete one). - - // Sync 1: Source has checkpointed=6 (checkpoint 2), proposed=9 - // Client gets blocks 1-6 via checkpoints, blocks 7-9 as uncheckpointed - setRemoteTipsMultiBlock(9, 6); - + it('still emits prune events but no checkpoint events', async () => { + setRemoteTips(9, 9); await blockStream.work(); - - expect(handler.events).toEqual([ - expectBlocksAdded([1, 2, 3]), - expectCheckpointed(1), - expectBlocksAdded([4, 5, 6]), - expectCheckpointed(2), - expectBlocksAdded([7, 8, 9]), // uncheckpointed - ]); - handler.clearEvents(); - // Update local state localData.proposed.number = BlockNumber(9); - localData.checkpointed.block.number = BlockNumber(6); - localData.checkpointed.checkpoint.number = CheckpointNumber(2); - - // Sync 2: Checkpoint 3 is now published (blocks 7-9), new blocks 10-12 are uncheckpointed - setRemoteTipsMultiBlock(12, 9); - - await blockStream.work(); - - // Should emit checkpoint 3 for already-local blocks 7-9, then uncheckpointed blocks 10-12 - expect(handler.events).toEqual([ - expectCheckpointed(3), - expectBlocksAdded([10, 11, 12]), // uncheckpointed - ]); - - handler.clearEvents(); - - // Update local state - localData.proposed.number = BlockNumber(12); localData.checkpointed.block.number = BlockNumber(9); - localData.checkpointed.checkpoint.number = CheckpointNumber(3); - - // Sync 3: Checkpoint 4 is now published (blocks 10-12), no new blocks - setRemoteTipsMultiBlock(12, 12); - - await blockStream.work(); - - // Should emit checkpoint 4 for already-local blocks 10-12 - expect(handler.events).toEqual([expectCheckpointed(4)]); - }); - - it('emits all checkpoints when source jumps ahead with multiple new checkpoints', async () => { - // Phase 1: Start with checkpoint 1 complete (blocks 1-3), blocks 4-6 uncheckpointed - setRemoteTipsMultiBlock(6, 3); + localData.checkpointed.checkpoint.number = CheckpointNumber(9); + for (let i = 4; i <= 9; i++) { + localData.blockHashes[i] = `0xbad${i}`; + } + setRemoteTips(3, 3); await blockStream.work(); expect(handler.events).toEqual([ - expectBlocksAdded([1, 2, 3]), - expectCheckpointed(1), - expectBlocksAdded([4, 5, 6]), + { type: 'chain-pruned', block: makeBlockId(3), checkpointed: makeTipId(3), proven: makeTipId(0) }, ]); + }); - handler.clearEvents(); - - // Update local state - localData.proposed.number = BlockNumber(6); - localData.checkpointed.block.number = BlockNumber(3); - localData.checkpointed.checkpoint.number = CheckpointNumber(1); - - // Phase 2: Source jumps to block 12 with checkpoints at 6, 9, and 12 - // - Checkpoint 2 (blocks 4-6) - blocks already local, needs checkpoint event - // - Checkpoint 3 (blocks 7-9) - new blocks + checkpoint event - // - Checkpoint 4 (blocks 10-12) - new blocks + checkpoint event - setRemoteTipsMultiBlock(12, 12); + it('still emits proven and finalized events', async () => { + setRemoteTips(9, 9, 6, 3); await blockStream.work(); - // Should emit: - // 1. Checkpoint 2 event (blocks 4-6 were already local) - // 2. Blocks 7-9 + checkpoint 3 event - // 3. Blocks 10-12 + checkpoint 4 event expect(handler.events).toEqual([ - expectCheckpointed(2), - expectBlocksAdded([7, 8, 9]), - expectCheckpointed(3), - expectBlocksAdded([10, 11, 12]), - expectCheckpointed(4), + { type: 'blocks-added', blocks: times(9, i => makeBlock(i + 1)) }, + { type: 'chain-proven', block: makeBlockId(6), checkpoint: makeCheckpointId(6) }, + { type: 'chain-finalized', block: makeBlockId(3), checkpoint: makeCheckpointId(3) }, ]); }); - - describe('startingBlock with stale checkpoint state', () => { - // When a node restarts with startingBlock set and has local blocks but no checkpoint - // state (e.g. checkpoint tracking is new, or checkpoint state was reset), Loop 1 - // should not spam checkpoint events for all historical checkpoints. - - it('skips historical checkpoint events before startingBlock on restart with stale checkpoint state', async () => { - // node has blocks 1-15 locally (proposed=15) but no checkpoint state. - // Checkpoint 5 covers blocks 13-15 (the last checkpoint). - setRemoteTipsMultiBlock(15, 15); - localData.proposed.number = BlockNumber(15); - // localData.checkpointed starts at 0 - simulating stale/missing checkpoint state - - blockStream = new TestL2BlockStream(blockSource, localData, handler, undefined, { - batchSize: 10, - startingBlock: 13, // start from checkpoint 5 (blocks 13-15) - }); - - await blockStream.work(); - - // Should only emit checkpoint 5 (the one containing startingBlock=13), not all 5 checkpoints - expect(handler.events).toEqual([expectCheckpointed(5)]); - // Verify we don't spam checkpoints 1-4 - const checkpointEvents = handler.events.filter(e => e.type === 'chain-checkpointed'); - expect(checkpointEvents).toHaveLength(1); - }); - - it('without startingBlock emits all historical checkpoints for already-local blocks', async () => { - // Same scenario without startingBlock: should emit all 5 checkpoints (correct catch-up behavior) - setRemoteTipsMultiBlock(15, 15); - localData.proposed.number = BlockNumber(15); - // localData.checkpointed starts at 0 - - await blockStream.work(); - - // All 5 checkpoints should be emitted since they're all for already-local blocks - const checkpointEvents = handler.events.filter(e => e.type === 'chain-checkpointed'); - expect(checkpointEvents).toHaveLength(5); - }); - - it('skips Loop 1 entirely when startingBlock is past the checkpointed tip', async () => { - // proposed=15, checkpointed=9 (ckpt 3 covers blocks 7-9). startingBlock=12 is past the - // checkpointed tip, in the proposed range. Loop 1 must skip without an RPC for block 12. - setRemoteTipsMultiBlock(15, 9); - blockStream = new TestL2BlockStream(blockSource, localData, handler, undefined, { - batchSize: 10, - startingBlock: 12, - }); - - await blockStream.work(); - - // No chain-checkpointed events because startingBlock is past the checkpointed tip. - const checkpointEvents = handler.events.filter(e => e.type === 'chain-checkpointed'); - expect(checkpointEvents).toHaveLength(0); - // Loop 1 must not query block 12 — past-the-tip is decided from sourceTips alone. - const loop1Calls = blockSource.getBlockData.mock.calls.filter( - c => 'number' in c[0] && (c[0] as { number: number }).number === 12, - ); - expect(loop1Calls).toHaveLength(0); - }); - - it('calls getBlockData for block 0 only for reorg detection, not checkpoint lookup, when startingBlock is 0', async () => { - // With startingBlock=0, the stream skips the checkpoint-number lookup (line 121 path) - // so getBlockData is called for block 0 only once: for the genesis reorg detection. - setRemoteTipsMultiBlock(15, 15); - blockStream = new TestL2BlockStream(blockSource, localData, handler, undefined, { - batchSize: 10, - startingBlock: 0, - }); - - await blockStream.work(); - - const calls = blockSource.getBlockData.mock.calls; - const block0Calls = calls.filter(c => 'number' in c[0] && (c[0] as { number: number }).number === 0); - // Only the genesis reorg-detection call — not an additional checkpoint-lookup call. - expect(block0Calls).toHaveLength(1); - }); - }); - - describe('checkpoint prefetching', () => { - it('prefetches multiple checkpoints in a single RPC call', async () => { - // Set up: 9 blocks in 3 checkpoints - setRemoteTipsMultiBlock(9, 9); - - // Create a stream with prefetch limit of 10 (will fetch all 3 checkpoints in one call) - // This also tests that we handle getting fewer checkpoints (3) than requested (10) - const prefetchStream = new TestL2BlockStream(blockSource, localData, handler, undefined, { - batchSize: 10, - checkpointPrefetchLimit: 10, - }); - - await prefetchStream.work(); - - // Should have fetched all 3 checkpoints in a single call (Loop 2 makes 1 call with limit 10) - // Even though we requested 10, only 3 exist - verify we handle this correctly - const calls = blockSource.getCheckpoints.mock.calls; - const loop2Calls = calls.filter(([query]) => 'limit' in query && query.limit === 10); - expect(loop2Calls.length).toBe(1); - expect((loop2Calls[0][0] as { from: number }).from).toBe(1); // Starting from checkpoint 1 - - // All 3 checkpoints should be emitted correctly (not 10) - const checkpointEvents = handler.events.filter(e => e.type === 'chain-checkpointed'); - expect(checkpointEvents).toHaveLength(3); - - // Verify correct event order - expect(handler.events).toEqual([ - expectBlocksAdded([1, 2, 3]), - expectCheckpointed(1), - expectBlocksAdded([4, 5, 6]), - expectCheckpointed(2), - expectBlocksAdded([7, 8, 9]), - expectCheckpointed(3), - ]); - }); - - it('prefetches checkpoints correctly when starting from an offset', async () => { - // Set up: 15 blocks in 5 checkpoints, but we already have blocks 1-6 locally (checkpoints 1-2) - setRemoteTipsMultiBlock(15, 15); - localData.proposed.number = BlockNumber(6); - localData.checkpointed.checkpoint.number = CheckpointNumber(2); - - // Create a stream with prefetch limit of 10 - const prefetchStream = new TestL2BlockStream(blockSource, localData, handler, undefined, { - batchSize: 10, - checkpointPrefetchLimit: 10, - }); - - await prefetchStream.work(); - - // Loop 2 should start fetching from checkpoint 3 (block 7 is in checkpoint 3) - const calls = blockSource.getCheckpoints.mock.calls; - const loop2Calls = calls.filter(([query]) => 'limit' in query && query.limit === 10); - expect(loop2Calls.length).toBe(1); - expect((loop2Calls[0][0] as { from: number }).from).toBe(3); // Starting from checkpoint 3, not 1 - - // Should only emit blocks 7-15 and checkpoints 3-5 (not 1-2, those are already local) - expect(handler.events).toEqual([ - expectBlocksAdded([7, 8, 9]), - expectCheckpointed(3), - expectBlocksAdded([10, 11, 12]), - expectCheckpointed(4), - expectBlocksAdded([13, 14, 15]), - expectCheckpointed(5), - ]); - - // Verify only 3 new checkpoints emitted - const checkpointEvents = handler.events.filter(e => e.type === 'chain-checkpointed'); - expect(checkpointEvents).toHaveLength(3); - }); - - it('prefetches correctly when starting from middle of a checkpoint', async () => { - // Local has blocks 1-7: checkpoints 1-2 complete, block 7 is first block of checkpoint 3 - setRemoteTipsMultiBlock(15, 15); - localData.proposed.number = BlockNumber(7); - localData.checkpointed.checkpoint.number = CheckpointNumber(2); - - const prefetchStream = new TestL2BlockStream(blockSource, localData, handler, undefined, { - batchSize: 10, - checkpointPrefetchLimit: 10, - }); - - await prefetchStream.work(); - - // Should start prefetching from checkpoint 3 - const calls = blockSource.getCheckpoints.mock.calls; - const loop2Calls = calls.filter(([query]) => 'limit' in query && query.limit === 10); - expect(loop2Calls.length).toBe(1); - expect((loop2Calls[0][0] as { from: number }).from).toBe(3); // Starting from checkpoint 3 - - // Should emit only blocks 8-9 from checkpoint 3 (block 7 is already local) - expect(handler.events).toEqual([ - expectBlocksAdded([8, 9]), // Rest of checkpoint 3 - expectCheckpointed(3), - expectBlocksAdded([10, 11, 12]), - expectCheckpointed(4), - expectBlocksAdded([13, 14, 15]), - expectCheckpointed(5), - ]); - - // Verify only 3 checkpoints emitted - const checkpointEvents = handler.events.filter(e => e.type === 'chain-checkpointed'); - expect(checkpointEvents).toHaveLength(3); - }); - - it('refills prefetch buffer when exhausted', async () => { - // Set up: 15 blocks in 5 checkpoints - setRemoteTipsMultiBlock(15, 15); - - // Create a stream with prefetch limit of 2 (will need 3 calls to fetch 5 checkpoints) - const prefetchStream = new TestL2BlockStream(blockSource, localData, handler, undefined, { - batchSize: 10, - checkpointPrefetchLimit: 2, - }); - - await prefetchStream.work(); - - // Should have made 3 calls with limit 2 to fetch 5 checkpoints - const calls = blockSource.getCheckpoints.mock.calls; - const loop2Calls = calls.filter(([query]) => 'limit' in query && query.limit === 2); - expect(loop2Calls.length).toBe(3); // ceil(5/2) = 3 - expect((loop2Calls[0][0] as { from: number }).from).toBe(1); // First batch: checkpoints 1-2 - expect((loop2Calls[1][0] as { from: number }).from).toBe(3); // Second batch: checkpoints 3-4 - expect((loop2Calls[2][0] as { from: number }).from).toBe(5); // Third batch: checkpoint 5 - - // All 5 checkpoints should be emitted - const checkpointEvents = handler.events.filter(e => e.type === 'chain-checkpointed'); - expect(checkpointEvents).toHaveLength(5); - }); - - it('uses default prefetch limit when not specified', async () => { - // Set up: 9 blocks in 3 checkpoints - setRemoteTipsMultiBlock(9, 9); - - // Create a stream without specifying checkpointPrefetchLimit (should use default of 50) - const defaultPrefetchStream = new TestL2BlockStream(blockSource, localData, handler, undefined, { - batchSize: 10, - }); - - await defaultPrefetchStream.work(); - - // Should have used default limit of 50 for Loop 2 calls - const calls = blockSource.getCheckpoints.mock.calls; - const loop2Calls = calls.filter(([query]) => 'limit' in query && query.limit === 50); - expect(loop2Calls.length).toBeGreaterThanOrEqual(1); - }); - }); - - describe('prune scenarios', () => { - it('prunes proposed chain back to checkpointed tip, then continues', async () => { - // Phase 1: Sync blocks 1-9 with checkpoints 1-2, blocks 7-9 uncheckpointed - setRemoteTipsMultiBlock(9, 6); - - await blockStream.work(); - - expect(handler.events).toEqual([ - expectBlocksAdded([1, 2, 3]), - expectCheckpointed(1), - expectBlocksAdded([4, 5, 6]), - expectCheckpointed(2), - expectBlocksAdded([7, 8, 9]), // uncheckpointed - ]); - - handler.clearEvents(); - - // Update local state to reflect what the handler stored - localData.proposed.number = BlockNumber(9); - localData.checkpointed.block.number = BlockNumber(6); - localData.checkpointed.checkpoint.number = CheckpointNumber(2); - - // Phase 2: Prune - proposed chain pruned back to checkpointed tip (block 6) - // This happens when uncheckpointed blocks (7-9) are invalid - // Mess up hashes for blocks 7-9 to simulate reorg - localData.blockHashes[7] = '0xbad7'; - localData.blockHashes[8] = '0xbad8'; - localData.blockHashes[9] = '0xbad9'; - - // Source now has proposed=6, checkpointed=6 - setRemoteTipsMultiBlock(6, 6); - - await blockStream.work(); - - // Should emit chain-pruned back to block 6 - expect(handler.events).toEqual([ - { - type: 'chain-pruned', - block: makeBlockId(6), - checkpointed: expect.objectContaining({ - checkpoint: expect.objectContaining({ number: CheckpointNumber(2) }), - }), - proven: expect.objectContaining({ - block: expect.objectContaining({ number: BlockNumber(0) }), - }), - }, - ]); - - handler.clearEvents(); - - // Update local state after prune - localData.proposed.number = BlockNumber(6); - delete localData.blockHashes[7]; - delete localData.blockHashes[8]; - delete localData.blockHashes[9]; - - // Phase 3: Chain continues - new blocks 7-12 arrive with checkpoints 3-4 - setRemoteTipsMultiBlock(12, 12); - - await blockStream.work(); - - // Should continue normally: blocks 7-9 + checkpoint 3, blocks 10-12 + checkpoint 4 - expect(handler.events).toEqual([ - expectBlocksAdded([7, 8, 9]), - expectCheckpointed(3), - expectBlocksAdded([10, 11, 12]), - expectCheckpointed(4), - ]); - }); - - it('prunes proposed and checkpointed chains back to proven tip, then continues', async () => { - // Phase 1: Sync blocks 1-12 with checkpoints 1-3, proven at checkpoint 2 (block 6) - setRemoteTipsMultiBlock(12, 9, 6); - - await blockStream.work(); - - expect(handler.events).toEqual([ - expectBlocksAdded([1, 2, 3]), - expectCheckpointed(1), - expectBlocksAdded([4, 5, 6]), - expectCheckpointed(2), - expectBlocksAdded([7, 8, 9]), - expectCheckpointed(3), - expectBlocksAdded([10, 11, 12]), - { type: 'chain-proven', block: makeBlockId(6), checkpoint: makeCheckpointId(2) }, - ]); - - handler.clearEvents(); - - // Update local state - localData.proposed.number = BlockNumber(12); - localData.checkpointed.block.number = BlockNumber(9); - localData.checkpointed.checkpoint.number = CheckpointNumber(3); - localData.proven.block.number = BlockNumber(6); - localData.proven.checkpoint.number = CheckpointNumber(2); - - // Phase 2: Prune - checkpoint 3 failed to prove, prune back to proven tip (block 6) - // Mess up hashes for blocks 7-12 to simulate reorg - for (let i = 7; i <= 12; i++) { - localData.blockHashes[i] = `0xbad${i}`; - } - - // Source now has proposed=6, checkpointed=6, proven=6 - setRemoteTipsMultiBlock(6, 6, 6); - - await blockStream.work(); - - // Should emit chain-pruned back to block 6, carrying the source proven tip (block 6 / ckpt 2) - expect(handler.events).toEqual([ - { - type: 'chain-pruned', - block: makeBlockId(6), - checkpointed: expect.objectContaining({ - checkpoint: expect.objectContaining({ number: CheckpointNumber(2) }), - }), - proven: expect.objectContaining({ - block: expect.objectContaining({ number: BlockNumber(6) }), - checkpoint: expect.objectContaining({ number: CheckpointNumber(2) }), - }), - }, - ]); - - handler.clearEvents(); - - // Update local state after prune - localData.proposed.number = BlockNumber(6); - localData.checkpointed.block.number = BlockNumber(6); - localData.checkpointed.checkpoint.number = CheckpointNumber(2); - for (let i = 7; i <= 12; i++) { - delete localData.blockHashes[i]; - } - - // Phase 3: Chain continues with new blocks and checkpoints - // New blocks 7-15 arrive with checkpoints 3-5, proven advances to checkpoint 3 - setRemoteTipsMultiBlock(15, 15, 9); - - await blockStream.work(); - - // Should continue normally with new blocks and checkpoints - expect(handler.events).toEqual([ - expectBlocksAdded([7, 8, 9]), - expectCheckpointed(3), - expectBlocksAdded([10, 11, 12]), - expectCheckpointed(4), - expectBlocksAdded([13, 14, 15]), - expectCheckpointed(5), - { type: 'chain-proven', block: makeBlockId(9), checkpoint: makeCheckpointId(3) }, - ]); - }); - - it('prunes uncheckpointed blocks and immediately receives new ones', async () => { - // Phase 1: Sync blocks 1-6 with checkpoint 1, blocks 4-6 uncheckpointed - setRemoteTipsMultiBlock(6, 3); - - await blockStream.work(); - - expect(handler.events).toEqual([ - expectBlocksAdded([1, 2, 3]), - expectCheckpointed(1), - expectBlocksAdded([4, 5, 6]), - ]); - - handler.clearEvents(); - - // Update local state - localData.proposed.number = BlockNumber(6); - localData.checkpointed.block.number = BlockNumber(3); - localData.checkpointed.checkpoint.number = CheckpointNumber(1); - - // Phase 2: Prune blocks 4-6 due to bad hashes - localData.blockHashes[4] = '0xbad4'; - localData.blockHashes[5] = '0xbad5'; - localData.blockHashes[6] = '0xbad6'; - - // Source still at checkpointed=3 (no new checkpoints yet) - setRemoteTipsMultiBlock(3, 3); - - await blockStream.work(); - - // Should emit prune back to block 3 - expect(handler.events).toEqual([ - { - type: 'chain-pruned', - block: makeBlockId(3), - checkpointed: expect.objectContaining({ - checkpoint: expect.objectContaining({ number: CheckpointNumber(1) }), - }), - proven: expect.objectContaining({ - block: expect.objectContaining({ number: BlockNumber(0) }), - }), - }, - ]); - - handler.clearEvents(); - - // Update local state after prune - localData.proposed.number = BlockNumber(3); - delete localData.blockHashes[4]; - delete localData.blockHashes[5]; - delete localData.blockHashes[6]; - - // Phase 3: New blocks 4-9 arrive with checkpoints 2-3 - setRemoteTipsMultiBlock(9, 9); - - await blockStream.work(); - - // Should continue normally with new blocks and checkpoints - expect(handler.events).toEqual([ - expectBlocksAdded([4, 5, 6]), - expectCheckpointed(2), - expectBlocksAdded([7, 8, 9]), - expectCheckpointed(3), - ]); - }); - - it('prunes proposed chain back to genesis when no checkpoints exist', async () => { - // Phase 1: Sync blocks 1-6, no checkpoints (checkpointed=0, proven=0, finalized=0) - setRemoteTipsMultiBlock(6, 0); - - await blockStream.work(); - - // All blocks come as uncheckpointed - expect(handler.events).toEqual([expectBlocksAdded([1, 2, 3, 4, 5, 6])]); - - handler.clearEvents(); - - // Update local state - localData.proposed.number = BlockNumber(6); - - // Phase 2: All blocks are invalid, prune back to genesis (block 0) - for (let i = 1; i <= 6; i++) { - localData.blockHashes[i] = `0xbad${i}`; - } - - // Source now has proposed=0, checkpointed=0 - setRemoteTipsMultiBlock(0, 0); - - await blockStream.work(); - - // Should emit chain-pruned back to block 0 - expect(handler.events).toEqual([ - { - type: 'chain-pruned', - block: makeBlockId(0), - checkpointed: expect.objectContaining({ - checkpoint: expect.objectContaining({ number: CheckpointNumber(0) }), - }), - proven: expect.objectContaining({ - block: expect.objectContaining({ number: BlockNumber(0) }), - }), - }, - ]); - - handler.clearEvents(); - - // Update local state after prune - localData.proposed.number = BlockNumber(0); - for (let i = 1; i <= 6; i++) { - delete localData.blockHashes[i]; - } - - // Phase 3: New blocks 1-6 arrive with checkpoints 1-2 - setRemoteTipsMultiBlock(6, 6); - - await blockStream.work(); - - // Should continue normally from genesis - expect(handler.events).toEqual([ - expectBlocksAdded([1, 2, 3]), - expectCheckpointed(1), - expectBlocksAdded([4, 5, 6]), - expectCheckpointed(2), - ]); - }); - - it('prunes both proposed and checkpointed chains back to genesis', async () => { - // Phase 1: Sync blocks 1-6 with checkpoint 1 (blocks 1-3), blocks 4-6 uncheckpointed - setRemoteTipsMultiBlock(6, 3); - - await blockStream.work(); - - expect(handler.events).toEqual([ - expectBlocksAdded([1, 2, 3]), - expectCheckpointed(1), - expectBlocksAdded([4, 5, 6]), - ]); - - handler.clearEvents(); - - // Update local state - localData.proposed.number = BlockNumber(6); - localData.checkpointed.block.number = BlockNumber(3); - localData.checkpointed.checkpoint.number = CheckpointNumber(1); - - // Phase 2: All blocks are invalid (even checkpointed ones), prune back to genesis - for (let i = 1; i <= 6; i++) { - localData.blockHashes[i] = `0xbad${i}`; - } - - // Source now has proposed=0, checkpointed=0 (full chain reset) - setRemoteTipsMultiBlock(0, 0); - - await blockStream.work(); - - // Should emit chain-pruned back to block 0 - expect(handler.events).toEqual([ - { - type: 'chain-pruned', - block: makeBlockId(0), - checkpointed: expect.objectContaining({ - checkpoint: expect.objectContaining({ number: CheckpointNumber(0) }), - }), - proven: expect.objectContaining({ - block: expect.objectContaining({ number: BlockNumber(0) }), - }), - }, - ]); - - handler.clearEvents(); - - // Update local state after prune - localData.proposed.number = BlockNumber(0); - localData.checkpointed.block.number = BlockNumber(0); - localData.checkpointed.checkpoint.number = CheckpointNumber(0); - for (let i = 1; i <= 6; i++) { - delete localData.blockHashes[i]; - } - - // Phase 3: New chain starts fresh with blocks 1-9 and checkpoints 1-3 - setRemoteTipsMultiBlock(9, 9); - - await blockStream.work(); - - // Should continue normally from genesis - expect(handler.events).toEqual([ - expectBlocksAdded([1, 2, 3]), - expectCheckpointed(1), - expectBlocksAdded([4, 5, 6]), - expectCheckpointed(2), - expectBlocksAdded([7, 8, 9]), - expectCheckpointed(3), - ]); - }); - }); - - describe('ignoreCheckpoints', () => { - beforeEach(() => { - blockStream = new TestL2BlockStream(blockSource, localData, handler, undefined, { - batchSize: 10, - ignoreCheckpoints: true, - }); - }); - - it('does not emit checkpoint events for new checkpointed blocks', async () => { - // 6 blocks in 2 checkpoints (blocks 1-3 in checkpoint 1, blocks 4-6 in checkpoint 2) - setRemoteTipsMultiBlock(6, 6); - - await blockStream.work(); - - // Should emit blocks-added events but NO chain-checkpointed events - expect(handler.events).toEqual([expectBlocksAdded([1, 2, 3]), expectBlocksAdded([4, 5, 6])]); - }); - - it('does not emit checkpoint events when blocks become checkpointed after being added', async () => { - // Phase 1: Blocks 1-3 checkpointed (checkpoint 1), blocks 4-6 uncheckpointed - setRemoteTipsMultiBlock(6, 3); - - await blockStream.work(); - - // Blocks 1-3 via checkpoint, blocks 4-6 as uncheckpointed, no checkpoint events - expect(handler.events).toEqual([expectBlocksAdded([1, 2, 3]), expectBlocksAdded([4, 5, 6])]); - - handler.clearEvents(); - - // Update local state - localData.proposed.number = BlockNumber(6); - localData.checkpointed.block.number = BlockNumber(3); - localData.checkpointed.checkpoint.number = CheckpointNumber(1); - - // Phase 2: Now blocks 4-6 become checkpointed too (checkpoint 2) - setRemoteTipsMultiBlock(6, 6); - - await blockStream.work(); - - // Checkpoint 2 event would normally be emitted for blocks 4-6, but not with ignoreCheckpoints - expect(handler.events).toEqual([]); - }); - - it('does not emit checkpoint events but still emits prune events for uncheckpointed blocks', async () => { - // Phase 1: Sync blocks 1-9, blocks 1-6 checkpointed (checkpoints 1-2), blocks 7-9 uncheckpointed - setRemoteTipsMultiBlock(9, 6); - - await blockStream.work(); - - expect(handler.events).toEqual([ - expectBlocksAdded([1, 2, 3]), - expectBlocksAdded([4, 5, 6]), - expectBlocksAdded([7, 8, 9]), - ]); - - handler.clearEvents(); - - // Update local state - localData.proposed.number = BlockNumber(9); - localData.checkpointed.block.number = BlockNumber(6); - localData.checkpointed.checkpoint.number = CheckpointNumber(2); - - // Phase 2: Prune uncheckpointed blocks 7-9 (bad hashes) - localData.blockHashes[7] = '0xbad7'; - localData.blockHashes[8] = '0xbad8'; - localData.blockHashes[9] = '0xbad9'; - - setRemoteTipsMultiBlock(6, 6); - - await blockStream.work(); - - // Should emit chain-pruned event (prune events are always emitted), no checkpoint events - expect(handler.events).toEqual([ - { - type: 'chain-pruned', - block: makeBlockId(6), - checkpointed: expect.objectContaining({ - checkpoint: expect.objectContaining({ number: CheckpointNumber(2) }), - }), - proven: expect.objectContaining({ - block: expect.objectContaining({ number: BlockNumber(0) }), - }), - }, - ]); - }); - - it('does not emit checkpoint events but still emits prune events for checkpointed blocks', async () => { - // Phase 1: Sync blocks 1-9, all checkpointed (checkpoints 1-3) - setRemoteTipsMultiBlock(9, 9); - - await blockStream.work(); - - expect(handler.events).toEqual([ - expectBlocksAdded([1, 2, 3]), - expectBlocksAdded([4, 5, 6]), - expectBlocksAdded([7, 8, 9]), - ]); - - handler.clearEvents(); - - // Update local state - localData.proposed.number = BlockNumber(9); - localData.checkpointed.block.number = BlockNumber(9); - localData.checkpointed.checkpoint.number = CheckpointNumber(3); - - // Phase 2: Prune checkpointed blocks (reorg of checkpointed chain back to checkpoint 1) - localData.blockHashes[4] = '0xbad4'; - localData.blockHashes[5] = '0xbad5'; - localData.blockHashes[6] = '0xbad6'; - localData.blockHashes[7] = '0xbad7'; - localData.blockHashes[8] = '0xbad8'; - localData.blockHashes[9] = '0xbad9'; - - setRemoteTipsMultiBlock(3, 3); - - await blockStream.work(); - - // Should emit chain-pruned event - expect(handler.events).toEqual([ - { - type: 'chain-pruned', - block: makeBlockId(3), - checkpointed: expect.objectContaining({ - checkpoint: expect.objectContaining({ number: CheckpointNumber(1) }), - }), - proven: expect.objectContaining({ - block: expect.objectContaining({ number: BlockNumber(0) }), - }), - }, - ]); - }); - - it('does not emit checkpoint events but still emits proven and finalized events', async () => { - // 9 blocks in 3 checkpoints, proven at block 6 (checkpoint 2), finalized at block 3 (checkpoint 1) - setRemoteTipsMultiBlock(9, 9, 6, 3); - - await blockStream.work(); - - // Should have blocks-added, chain-proven, chain-finalized, but NO checkpoint events - expect(handler.events).toEqual([ - expectBlocksAdded([1, 2, 3]), - expectBlocksAdded([4, 5, 6]), - expectBlocksAdded([7, 8, 9]), - { type: 'chain-proven', block: makeBlockId(6), checkpoint: makeCheckpointId(2) }, - { type: 'chain-finalized', block: makeBlockId(3), checkpoint: makeCheckpointId(1) }, - ]); - }); - - it('does not emit checkpoint events but still emits proven and finalized events with skipFinalized', async () => { - // Use skipFinalized in addition to ignoreCheckpoints - blockStream = new TestL2BlockStream(blockSource, localData, handler, undefined, { - batchSize: 10, - ignoreCheckpoints: true, - skipFinalized: true, - }); - - // 12 blocks in 4 checkpoints, proven at block 9 (checkpoint 3), finalized at block 6 (checkpoint 2) - setRemoteTipsMultiBlock(12, 12, 9, 6); - - // Local is behind - at block 3, finalized at block 3 - localData.proposed.number = BlockNumber(3); - localData.checkpointed.block.number = BlockNumber(3); - localData.checkpointed.checkpoint.number = CheckpointNumber(1); - localData.proven.block.number = BlockNumber(3); - localData.proven.checkpoint.number = CheckpointNumber(1); - localData.finalized.block.number = BlockNumber(3); - localData.finalized.checkpoint.number = CheckpointNumber(1); - - await blockStream.work(); - - // With skipFinalized, we skip to the finalized tip (block 6), then sync from there - // Should emit blocks 6-12, proven, finalized, but NO checkpoint events - expect(handler.events).toEqual([ - expectBlocksAdded([6]), - expectBlocksAdded([7, 8, 9]), - expectBlocksAdded([10, 11, 12]), - { type: 'chain-proven', block: makeBlockId(9), checkpoint: makeCheckpointId(3) }, - { type: 'chain-finalized', block: makeBlockId(6), checkpoint: makeCheckpointId(2) }, - ]); - }); - - it('does not emit checkpoint events after prune and re-sync with new blocks', async () => { - // Phase 1: Sync blocks 1-9, all checkpointed (checkpoints 1-3) - setRemoteTipsMultiBlock(9, 9); - - await blockStream.work(); - - expect(handler.events).toEqual([ - expectBlocksAdded([1, 2, 3]), - expectBlocksAdded([4, 5, 6]), - expectBlocksAdded([7, 8, 9]), - ]); - - handler.clearEvents(); - - // Update local state - localData.proposed.number = BlockNumber(9); - localData.checkpointed.block.number = BlockNumber(9); - localData.checkpointed.checkpoint.number = CheckpointNumber(3); - - // Phase 2: Prune back to block 6 (checkpoint 2) - localData.blockHashes[7] = '0xbad7'; - localData.blockHashes[8] = '0xbad8'; - localData.blockHashes[9] = '0xbad9'; - - setRemoteTipsMultiBlock(6, 6); - - await blockStream.work(); - - expect(handler.events).toEqual([ - { - type: 'chain-pruned', - block: makeBlockId(6), - checkpointed: expect.objectContaining({ - checkpoint: expect.objectContaining({ number: CheckpointNumber(2) }), - }), - proven: expect.objectContaining({ - block: expect.objectContaining({ number: BlockNumber(0) }), - }), - }, - ]); - - handler.clearEvents(); - - // Update local state after prune - localData.proposed.number = BlockNumber(6); - localData.checkpointed.block.number = BlockNumber(6); - localData.checkpointed.checkpoint.number = CheckpointNumber(2); - delete localData.blockHashes[7]; - delete localData.blockHashes[8]; - delete localData.blockHashes[9]; - - // Phase 3: New blocks 7-12 arrive, all checkpointed (checkpoints 3-4) - setRemoteTipsMultiBlock(12, 12); - - await blockStream.work(); - - // Should have new blocks-added events but still no checkpoint events - expect(handler.events).toEqual([expectBlocksAdded([7, 8, 9]), expectBlocksAdded([10, 11, 12])]); - }); - }); }); describe('skipFinalized', () => { @@ -1668,9 +560,9 @@ describe('L2BlockStream', () => { it('skips ahead to the latest finalized block', async () => { setRemoteTips(40, 0, 38, 35); - localData.proposed.number = BlockNumber(5); - localData.proven.block.number = BlockNumber(2); - localData.finalized.block.number = BlockNumber(2); + localData.setProposed(5); + localData.setProven(2); + localData.setFinalized(2); await blockStream.work(); @@ -1685,133 +577,17 @@ describe('L2BlockStream', () => { it('does not skip if already ahead of finalized', async () => { setRemoteTips(40, 0, 38, 35); - localData.proposed.number = BlockNumber(38); - localData.proven.block.number = BlockNumber(38); - localData.finalized.block.number = BlockNumber(35); + localData.setProposed(38); + localData.setProven(38); + localData.setFinalized(35); await blockStream.work(); + // proven and finalized tips already match the source on (number, hash), so only new blocks are emitted. expect(handler.events).toEqual([ { type: 'blocks-added', blocks: times(2, i => makeBlock(i + 39)) }, ] satisfies L2BlockStreamEvent[]); }); - - it('emits the finalized checkpoint when fetching the finalized block (inconsistency)', async () => { - // This test demonstrates an inconsistency: Loop 1 skips finalized checkpoints via - // nextCheckpointToEmit adjustment, but Loop 2 still emits the finalized checkpoint - // when we fetch the finalized block. - // - // Scenario: Fresh start with skipFinalized=true - // - Checkpoint 6 (block 6) is finalized - // - Checkpoints 7-9 are checkpointed - // - Blocks 10-12 are uncheckpointed - // - // Expected (if consistent): Skip checkpoint 6, only emit checkpoints 7+ - // Actual: Checkpoint 6 IS emitted because Loop 2 fetches block 6 and emits its checkpoint - - // 12 blocks total, checkpointed up to 9, proven at 9, finalized at 6 - setRemoteTips(12, 9, 9, 6); - - // Fresh start - no local blocks - localData.proposed.number = BlockNumber(0); - localData.checkpointed.block.number = BlockNumber(0); - localData.checkpointed.checkpoint.number = CheckpointNumber(0); - localData.proven.block.number = BlockNumber(0); - localData.proven.checkpoint.number = CheckpointNumber(0); - localData.finalized.block.number = BlockNumber(0); - localData.finalized.checkpoint.number = CheckpointNumber(0); - - await blockStream.work(); - - // With skipFinalized, nextCheckpointToEmit starts at checkpoint 6 (finalized checkpoint) - // Loop 1 skips immediately (no local blocks) - // Loop 2 starts at block 6 (finalized block), finds checkpoint 6, emits block 6, then emits checkpoint 6 - // This IS the inconsistency: we emit checkpoint 6 even though it's finalized - expect(handler.events).toEqual([ - expectBlocksAdded([6]), - expectCheckpointed(6), // <-- This is the finalized checkpoint being emitted! - expectBlocksAdded([7]), - expectCheckpointed(7), - expectBlocksAdded([8]), - expectCheckpointed(8), - expectBlocksAdded([9]), - expectCheckpointed(9), - { type: 'blocks-added', blocks: times(3, i => makeBlock(i + 10)) }, - { type: 'chain-proven', block: makeBlockId(9), checkpoint: makeCheckpointId(9) }, - { type: 'chain-finalized', block: makeBlockId(6), checkpoint: makeCheckpointId(6) }, - ]); - }); - - it('does not emit finalized checkpointed blocks when skipFinalized is true', async () => { - // Source: finalized=35, proven=38, checkpointed=38, proposed=40 - // All blocks up to 38 are checkpointed (each block is its own checkpoint), blocks 39-40 are uncheckpointed - setRemoteTips(40, 38, 38, 35); - - // Local is at block 5, finalized at 2 - localData.proposed.number = BlockNumber(5); - localData.checkpointed.block.number = BlockNumber(2); - localData.checkpointed.checkpoint.number = CheckpointNumber(2); - localData.proven.block.number = BlockNumber(2); - localData.proven.checkpoint.number = CheckpointNumber(2); - localData.finalized.block.number = BlockNumber(2); - localData.finalized.checkpoint.number = CheckpointNumber(2); - - await blockStream.work(); - - // With skipFinalized=true, we should skip to block 35 (finalized tip) - // We should NOT emit blocks 6-34 since they are finalized - // We should only emit blocks from 35 onwards, and checkpoint events from checkpoint 36 onwards - // (checkpoint 35 is the finalized checkpoint, so we skip it too) - expect(handler.events).toEqual([ - expect.objectContaining({ - type: 'blocks-added', - blocks: [expect.objectContaining({ number: BlockNumber(35) })], - }), - expect.objectContaining({ - type: 'chain-checkpointed', - block: expect.objectContaining({ number: BlockNumber(35) }), - checkpoint: expect.objectContaining({ - checkpoint: expect.objectContaining({ number: CheckpointNumber(35) }), - }), - }), - expect.objectContaining({ - type: 'blocks-added', - blocks: [expect.objectContaining({ number: BlockNumber(36) })], - }), - expect.objectContaining({ - type: 'chain-checkpointed', - block: expect.objectContaining({ number: BlockNumber(36) }), - checkpoint: expect.objectContaining({ - checkpoint: expect.objectContaining({ number: CheckpointNumber(36) }), - }), - }), - expect.objectContaining({ - type: 'blocks-added', - blocks: [expect.objectContaining({ number: BlockNumber(37) })], - }), - expect.objectContaining({ - type: 'chain-checkpointed', - block: expect.objectContaining({ number: BlockNumber(37) }), - checkpoint: expect.objectContaining({ - checkpoint: expect.objectContaining({ number: CheckpointNumber(37) }), - }), - }), - expect.objectContaining({ - type: 'blocks-added', - blocks: [expect.objectContaining({ number: BlockNumber(38) })], - }), - expect.objectContaining({ - type: 'chain-checkpointed', - block: expect.objectContaining({ number: BlockNumber(38) }), - checkpoint: expect.objectContaining({ - checkpoint: expect.objectContaining({ number: CheckpointNumber(38) }), - }), - }), - { type: 'blocks-added', blocks: times(2, i => makeBlock(i + 39)) }, - { type: 'chain-proven', block: makeBlockId(38), checkpoint: makeCheckpointId(38) }, - { type: 'chain-finalized', block: makeBlockId(35), checkpoint: makeCheckpointId(35) }, - ]); - }); }); describe('local provider without checkpointed tip', () => { @@ -1832,14 +608,8 @@ describe('L2BlockStream', () => { await blockStream.work(); - // All 5 blocks are synced (one per checkpoint in the mock) and no checkpoint events are emitted. - expect(handler.events).toEqual([ - expectBlocksAdded([1]), - expectBlocksAdded([2]), - expectBlocksAdded([3]), - expectBlocksAdded([4]), - expectBlocksAdded([5]), - ]); + // All 5 blocks are synced and no checkpoint events are emitted. + expect(handler.events).toEqual([{ type: 'blocks-added', blocks: times(5, i => makeBlock(i + 1)) }]); expect(handler.events.every(e => e.type === 'blocks-added')).toBe(true); }); @@ -1859,6 +629,11 @@ describe('L2BlockStream', () => { }); }); +/** Builds a checkpoint id from a plain number, isolated so the branded-type lint rule sees no BlockNumber flow. */ +function makeTipCheckpointId(checkpointNumber: number) { + return { number: CheckpointNumber(checkpointNumber), hash: new Fr(checkpointNumber).toString() }; +} + class TestL2BlockStreamEventHandler implements L2BlockStreamEventHandler { public readonly events: L2BlockStreamEvent[] = []; public throwing: boolean = false; @@ -1885,20 +660,48 @@ class TestL2BlockStreamEventHandler implements L2BlockStreamEventHandler { class TestL2BlockStreamLocalDataProvider implements L2BlockStreamLocalDataProvider { public readonly blockHashes: Record = {}; - public proposed = { number: BlockNumber.ZERO, hash: GENESIS_BLOCK_HEADER_HASH.toString() }; + // Genesis tip hashes match `getL2BlockHash(0)` (`new Fr(0)`) and the mock source's genesis tips + // (`makeHash(0)`), so the tier reconciliation finds no spurious difference at genesis. + public proposed = { number: BlockNumber.ZERO, hash: new Fr(0).toString() }; public checkpointed = { - block: { number: BlockNumber.ZERO, hash: GENESIS_BLOCK_HEADER_HASH.toString() }, - checkpoint: { number: CheckpointNumber.ZERO, hash: GENESIS_CHECKPOINT_HEADER_HASH.toString() }, + block: { number: BlockNumber.ZERO, hash: new Fr(0).toString() }, + checkpoint: { number: CheckpointNumber.ZERO, hash: new Fr(0).toString() }, }; public proven = { - block: { number: BlockNumber.ZERO, hash: GENESIS_BLOCK_HEADER_HASH.toString() }, - checkpoint: { number: CheckpointNumber.ZERO, hash: GENESIS_CHECKPOINT_HEADER_HASH.toString() }, + block: { number: BlockNumber.ZERO, hash: new Fr(0).toString() }, + checkpoint: { number: CheckpointNumber.ZERO, hash: new Fr(0).toString() }, }; public finalized = { - block: { number: BlockNumber.ZERO, hash: GENESIS_BLOCK_HEADER_HASH.toString() }, - checkpoint: { number: CheckpointNumber.ZERO, hash: GENESIS_CHECKPOINT_HEADER_HASH.toString() }, + block: { number: BlockNumber.ZERO, hash: new Fr(0).toString() }, + checkpoint: { number: CheckpointNumber.ZERO, hash: new Fr(0).toString() }, }; + /** Sets a tip's number and a matching hash, so the tier reconciliation sees consistent (number, hash) pairs. */ + public setProposed(number: number) { + this.proposed = { number: BlockNumber(number), hash: new Fr(number).toString() }; + } + + public setCheckpointed(blockNumber: number, checkpointNumber: number) { + this.checkpointed = { + block: { number: BlockNumber(blockNumber), hash: new Fr(blockNumber).toString() }, + checkpoint: makeTipCheckpointId(checkpointNumber), + }; + } + + public setProven(blockNumber: number) { + this.proven = { + block: { number: BlockNumber(blockNumber), hash: new Fr(blockNumber).toString() }, + checkpoint: makeTipCheckpointId(blockNumber), + }; + } + + public setFinalized(blockNumber: number) { + this.finalized = { + block: { number: BlockNumber(blockNumber), hash: new Fr(blockNumber).toString() }, + checkpoint: makeTipCheckpointId(blockNumber), + }; + } + public getL2BlockHash(number: number): Promise { return Promise.resolve( number > this.proposed.number ? undefined : (this.blockHashes[number] ?? new Fr(number).toString()), @@ -1919,9 +722,9 @@ class TestL2BlockStreamLocalDataProvider implements L2BlockStreamLocalDataProvid class TestLocalChainTipsProvider implements L2BlockStreamLocalDataProvider { public readonly blockHashes: Record = {}; - public proposed = { number: BlockNumber.ZERO, hash: GENESIS_BLOCK_HEADER_HASH.toString() }; - public proven = { block: { number: BlockNumber.ZERO, hash: GENESIS_BLOCK_HEADER_HASH.toString() } }; - public finalized = { block: { number: BlockNumber.ZERO, hash: GENESIS_BLOCK_HEADER_HASH.toString() } }; + public proposed = { number: BlockNumber.ZERO, hash: new Fr(0).toString() }; + public proven = { block: { number: BlockNumber.ZERO, hash: new Fr(0).toString() } }; + public finalized = { block: { number: BlockNumber.ZERO, hash: new Fr(0).toString() } }; public getL2BlockHash(number: number): Promise { return Promise.resolve( diff --git a/yarn-project/stdlib/src/block/l2_block_stream/l2_block_stream.ts b/yarn-project/stdlib/src/block/l2_block_stream/l2_block_stream.ts index 3d5280e1684d..33f648c03694 100644 --- a/yarn-project/stdlib/src/block/l2_block_stream/l2_block_stream.ts +++ b/yarn-project/stdlib/src/block/l2_block_stream/l2_block_stream.ts @@ -1,14 +1,18 @@ -import { BlockNumber, CheckpointNumber } from '@aztec/foundation/branded-types'; +import { BlockNumber } from '@aztec/foundation/branded-types'; import { AbortError } from '@aztec/foundation/error'; import { createLogger } from '@aztec/foundation/log'; import { RunningPromise } from '@aztec/foundation/running-promise'; -import type { PublishedCheckpoint } from '../../checkpoint/published_checkpoint.js'; -import { type L2BlockId, type L2BlockSource, makeL2BlockId } from '../l2_block_source.js'; -import type { L2BlockStreamEvent, L2BlockStreamEventHandler, L2BlockStreamLocalDataProvider } from './interfaces.js'; +import { type L2BlockId, type L2BlockSource, type L2TipId, makeL2BlockId } from '../l2_block_source.js'; +import type { + L2BlockStreamEvent, + L2BlockStreamEventHandler, + L2BlockStreamLocalDataProvider, + LocalL2BlockId, +} from './interfaces.js'; -/** Maximum number of checkpoints to prefetch at once during sync. Matches MAX_RPC_CHECKPOINTS_LEN. */ -export const CHECKPOINT_PREFETCH_LIMIT = 50; +/** Subset of the block source the stream depends on. Checkpoint payloads are no longer fetched here. */ +type L2BlockStreamSource = Pick; /** Creates a stream of events for new blocks, chain tips updates, and reorgs, out of polling an archiver or a node. */ export class L2BlockStream { @@ -17,7 +21,7 @@ export class L2BlockStream { private hasStarted = false; constructor( - private l2BlockSource: Pick, + private l2BlockSource: L2BlockStreamSource, private localData: L2BlockStreamLocalDataProvider, private handler: L2BlockStreamEventHandler, private readonly log = createLogger('types:block_stream'), @@ -27,10 +31,8 @@ export class L2BlockStream { startingBlock?: number; /** Instead of downloading all blocks, only fetch the smallest subset that results in reliable reorg detection. */ skipFinalized?: boolean; - /** When true, checkpoint events will not be emitted. Blocks are still fetched via checkpoints but only blocks-added events are emitted. */ + /** When true, checkpoint events will not be emitted. Blocks are still fetched but only blocks-added events are emitted. */ ignoreCheckpoints?: boolean; - /** Maximum number of checkpoints to prefetch at once during sync. Defaults to CHECKPOINT_PREFETCH_LIMIT (50). */ - checkpointPrefetchLimit?: number; } = {}, ) { // Note that RunningPromise is in stopped state by default. This promise won't run until someone invokes `start`, @@ -100,6 +102,7 @@ export class L2BlockStream { latestBlockNumber--; } + let pruned = false; if (latestBlockNumber < localTips.proposed.number) { latestBlockNumber = BlockNumber(Math.min(latestBlockNumber, sourceTips.proposed.number)); // see #13471 const hash = sourceCache.get(latestBlockNumber) ?? (await this.getBlockHashFromSource(latestBlockNumber)); @@ -115,149 +118,34 @@ export class L2BlockStream { checkpointed: sourceTips.checkpointed, proven: sourceTips.proven, }); + pruned = true; } - // If we are just starting, use the starting block number from the options. - const startingBlock = this.opts.startingBlock !== undefined ? BlockNumber(this.opts.startingBlock) : undefined; - if (latestBlockNumber === 0 && startingBlock !== undefined) { - latestBlockNumber = BlockNumber(Math.max(startingBlock - 1, 0)); - } - - // Only log this entry once (for sanity) - if (!this.hasStarted) { - this.log.verbose(`Starting sync from block number ${latestBlockNumber}`); - this.hasStarted = true; - } - + // The post-prune cursor: the highest block number both sides agree on. Block downloads resume from here. let nextBlockNumber = latestBlockNumber + 1; - // When checkpoints are ignored the local provider may omit `checkpointed`; in that case the fallback to - // CheckpointNumber.ZERO is harmless because `nextCheckpointToEmit` is never consumed for emission (Loop 1 and - // the startingBlock/skipFinalized adjustments below only feed checkpoint emission, which is gated off). - let nextCheckpointToEmit = CheckpointNumber( - (localTips.checkpointed?.checkpoint.number ?? CheckpointNumber.ZERO) + 1, - ); - // When startingBlock is set, also skip ahead for checkpoints. - if ( - startingBlock !== undefined && - startingBlock >= 1 && - nextCheckpointToEmit <= sourceTips.checkpointed.checkpoint.number - ) { - if (startingBlock > sourceTips.checkpointed.block.number) { - // startingBlock is past all checkpointed blocks; skip Loop 1 entirely. - nextCheckpointToEmit = CheckpointNumber(sourceTips.checkpointed.checkpoint.number + 1); - } else { - const startingBlockData = await this.l2BlockSource.getBlockData({ number: startingBlock }); - if (startingBlockData) { - nextCheckpointToEmit = CheckpointNumber(Math.max(nextCheckpointToEmit, startingBlockData.checkpointNumber)); - } - } + // If we are just starting from a fresh local store, fast-forward the download cursor to the configured + // starting block so we skip the history the consumer doesn't care about. + const startingBlock = this.opts.startingBlock !== undefined ? BlockNumber(this.opts.startingBlock) : undefined; + if (latestBlockNumber === 0 && startingBlock !== undefined) { + nextBlockNumber = Math.max(startingBlock, 1); } if (this.opts.skipFinalized) { // When skipping finalized blocks we need to provide reliable reorg detection while fetching as few blocks as // possible. Finalized blocks cannot be reorged by definition, so we can skip most of them. We do need the very // last finalized block however in order to guarantee that we will eventually find a block in which our local - // store matches the source. - // If the last finalized block is behind our local tip, there is nothing to skip. + // store matches the source. If the last finalized block is behind our local tip, there is nothing to skip. nextBlockNumber = Math.max(sourceTips.finalized.block.number, nextBlockNumber); - // If the next checkpoint to emit is behind the finalized tip then skip forward - nextCheckpointToEmit = CheckpointNumber(Math.max(nextCheckpointToEmit, sourceTips.finalized.checkpoint.number)); - } - - // Loop 1: Emit checkpoint events for checkpoints whose blocks are already in local storage. - // This handles the case where blocks were synced as uncheckpointed and later became checkpointed. - // The guard `lastBlockInCheckpoint.number > localTips.proposed.number` ensures we don't emit - // checkpoints for blocks we don't have (e.g., when startingBlock skips earlier blocks). - // Since only one checkpoint can ever be uncheckpointed, this loop should iterate at most once. - if (!this.opts.ignoreCheckpoints) { - let loop1Iterations = 0; - while (nextCheckpointToEmit <= sourceTips.checkpointed.checkpoint.number) { - const checkpoints = await this.l2BlockSource.getCheckpoints({ from: nextCheckpointToEmit, limit: 1 }); - if (checkpoints.length === 0) { - break; - } - const lastBlockInCheckpoint = checkpoints[0].checkpoint.blocks.at(-1)!; - // If this checkpoint has blocks we haven't seen yet, stop - they need to be fetched first - if (lastBlockInCheckpoint.number > localTips.proposed.number) { - break; - } - loop1Iterations++; - if (loop1Iterations > 1) { - this.log.warn( - `Emitting multiple checkpoints (${loop1Iterations}) for already-local blocks. ` + - `Next checkpoint: ${nextCheckpointToEmit}, source checkpointed: ${sourceTips.checkpointed.checkpoint.number}`, - ); - } - const lastBlockHash = await lastBlockInCheckpoint.hash(); - await this.emitEvent({ - type: 'chain-checkpointed', - checkpoint: checkpoints[0], - block: makeL2BlockId(lastBlockInCheckpoint.number, lastBlockHash.toString()), - }); - nextCheckpointToEmit = CheckpointNumber(nextCheckpointToEmit + 1); - } } - // Loop 2: Fetch new checkpointed blocks. For each checkpoint, emit all blocks - // from that checkpoint that we need, then emit the checkpoint event. - // We prefetch multiple checkpoints, then process them one by one. - let prefetchedCheckpoints: PublishedCheckpoint[] = []; - let prefetchIdx = 0; - let nextCheckpointNumber: CheckpointNumber | undefined; - - // Find the starting checkpoint number - if (nextBlockNumber <= sourceTips.checkpointed.block.number) { - const blockData = await this.l2BlockSource.getBlockData({ number: BlockNumber(nextBlockNumber) }); - if (blockData) { - nextCheckpointNumber = blockData.checkpointNumber; - } - } - - while (nextBlockNumber <= sourceTips.checkpointed.block.number && nextCheckpointNumber !== undefined) { - // Refill the prefetch buffer when exhausted - if (prefetchIdx >= prefetchedCheckpoints.length) { - const prefetchLimit = this.opts.checkpointPrefetchLimit ?? CHECKPOINT_PREFETCH_LIMIT; - prefetchedCheckpoints = await this.l2BlockSource.getCheckpoints({ - from: nextCheckpointNumber, - limit: prefetchLimit, - }); - prefetchIdx = 0; - if (prefetchedCheckpoints.length === 0) { - break; - } - } - - const checkpoint = prefetchedCheckpoints[prefetchIdx]!; - - // Get all blocks from this checkpoint that we need, respecting batchSize - const limit = Math.min(this.opts.batchSize ?? 50, sourceTips.checkpointed.block.number - nextBlockNumber + 1); - const blocksForCheckpoint = checkpoint.checkpoint.blocks - .filter(b => b.number >= nextBlockNumber) - .slice(0, limit); - if (blocksForCheckpoint.length === 0) { - break; - } - await this.emitEvent({ type: 'blocks-added', blocks: blocksForCheckpoint }); - nextBlockNumber = blocksForCheckpoint.at(-1)!.number + 1; - - // If we've reached the end of this checkpoint, emit the checkpoint event and move to next - const lastBlockInCheckpoint = checkpoint.checkpoint.blocks.at(-1)!; - if (nextBlockNumber > lastBlockInCheckpoint.number) { - if (!this.opts.ignoreCheckpoints) { - const lastBlockHash = await lastBlockInCheckpoint.hash(); - await this.emitEvent({ - type: 'chain-checkpointed', - checkpoint, - block: makeL2BlockId(lastBlockInCheckpoint.number, lastBlockHash.toString()), - }); - } - prefetchIdx++; - nextCheckpointNumber = CheckpointNumber(nextCheckpointNumber + 1); - } + // Only log this entry once (for sanity) + if (!this.hasStarted) { + this.log.verbose(`Starting sync from block number ${nextBlockNumber - 1}`); + this.hasStarted = true; } - // Loop 3: Fetch any remaining uncheckpointed (proposed) blocks. + // Download every block up to the source's proposed tip, batched by `batchSize`. while (nextBlockNumber <= sourceTips.proposed.number) { const limit = Math.min(this.opts.batchSize ?? 50, sourceTips.proposed.number - nextBlockNumber + 1); this.log.trace(`Requesting blocks from ${nextBlockNumber} limit ${limit}`); @@ -269,15 +157,30 @@ export class L2BlockStream { nextBlockNumber = blocks.at(-1)!.number + 1; } - // Update the proven and finalized tips. - if (localTips.proven !== undefined && sourceTips.proven.block.number !== localTips.proven.block.number) { + // End-of-pass tier reconciliation. For each tier, emit a single event iff the source tip differs from the + // local one. All three source tips come from the SAME `sourceTips` snapshot, so no extra source fetches are + // needed. We re-read the local tips after a prune because the prune handler has already clamped the local + // cursors back; the `localTips` snapshot taken before the prune would be stale and would mis-drive the tier + // comparison (emitting events relative to cursors that no longer exist). + const reconcileTips = pruned ? await this.localData.getL2Tips() : localTips; + if (!this.opts.ignoreCheckpoints && this.tipDiffers(reconcileTips.checkpointed?.block, sourceTips.checkpointed)) { + await this.emitEvent({ + type: 'chain-checkpointed', + block: sourceTips.checkpointed.block, + checkpoint: sourceTips.checkpointed.checkpoint, + }); + } + if (reconcileTips.proven !== undefined && this.tipDiffers(reconcileTips.proven.block, sourceTips.proven)) { await this.emitEvent({ type: 'chain-proven', block: sourceTips.proven.block, checkpoint: sourceTips.proven.checkpoint, }); } - if (localTips.finalized !== undefined && sourceTips.finalized.block.number !== localTips.finalized.block.number) { + if ( + reconcileTips.finalized !== undefined && + this.tipDiffers(reconcileTips.finalized.block, sourceTips.finalized) + ) { await this.emitEvent({ type: 'chain-finalized', block: sourceTips.finalized.block, @@ -292,6 +195,25 @@ export class L2BlockStream { } } + /** + * Returns whether the source tip differs from the local one and therefore warrants a tier event. Compares block + * number and, when both hashes are known, block hash. The hash comparison is skipped when the local hash is + * undefined or missing: world-state legitimately reports `undefined` hashes for tips ahead of its synced range, + * and comparing against an undefined hash would re-emit the event on every poll. + */ + private tipDiffers(localBlock: LocalL2BlockId | undefined, sourceTip: L2TipId): boolean { + if (localBlock === undefined) { + return true; + } + if (sourceTip.block.number !== localBlock.number) { + return true; + } + if (localBlock.hash === undefined) { + return false; + } + return sourceTip.block.hash !== localBlock.hash; + } + /** * Returns whether the source and local agree on the block hash at a given height. * @param blockNumber - The block number to test. @@ -327,7 +249,13 @@ export class L2BlockStream { private async emitEvent(event: L2BlockStreamEvent) { this.log.debug( - `Emitting ${event.type} (${event.type === 'blocks-added' ? event.blocks.length : event.type === 'chain-checkpointed' ? event.checkpoint.checkpoint.number : event.block.number})`, + `Emitting ${event.type} (${ + event.type === 'blocks-added' + ? event.blocks.length + : event.type === 'chain-checkpointed' + ? event.checkpoint.number + : event.block.number + })`, ); await this.handler.handleBlockStreamEvent(event); if (!this.isRunning() && !this.isSyncing) { diff --git a/yarn-project/stdlib/src/block/l2_block_stream/l2_tips_store_base.ts b/yarn-project/stdlib/src/block/l2_block_stream/l2_tips_store_base.ts index 37a576424b29..4a2a317e18bb 100644 --- a/yarn-project/stdlib/src/block/l2_block_stream/l2_tips_store_base.ts +++ b/yarn-project/stdlib/src/block/l2_block_stream/l2_tips_store_base.ts @@ -153,12 +153,8 @@ export abstract class L2TipsStoreBase implements L2BlockStreamEventHandler, L2Bl return; } await this.runInTransaction(async () => { - const checkpointId: CheckpointId = { - number: event.checkpoint.checkpoint.number, - hash: event.checkpoint.checkpoint.hash().toString(), - }; await this.saveTag('checkpointed', event.block); - await this.setTipCheckpoint('checkpointed', checkpointId); + await this.setTipCheckpoint('checkpointed', event.checkpoint); }); } diff --git a/yarn-project/stdlib/src/block/test/l2_tips_store_test_suite.ts b/yarn-project/stdlib/src/block/test/l2_tips_store_test_suite.ts index 7335058c452f..e8a928e5fb3d 100644 --- a/yarn-project/stdlib/src/block/test/l2_tips_store_test_suite.ts +++ b/yarn-project/stdlib/src/block/test/l2_tips_store_test_suite.ts @@ -94,14 +94,18 @@ export function testL2TipsStore(makeTipsStore: () => Promise) { return new PublishedCheckpoint(checkpoint, L1PublishedData.random(), []); }; - /** Creates a chain-checkpointed event with the required block field */ - const makeCheckpointedEvent = async (checkpoint: PublishedCheckpoint) => { - const lastBlock = checkpoint.checkpoint.blocks.at(-1)!; + /** Creates a thin chain-checkpointed event carrying the block + checkpoint ids of the checkpoint's last block. */ + const makeCheckpointedEvent = async (published: PublishedCheckpoint) => { + const lastBlock = published.checkpoint.blocks.at(-1)!; const blockId: L2BlockId = { number: lastBlock.number, hash: (await lastBlock.hash()).toString(), }; - return { type: 'chain-checkpointed' as const, checkpoint, block: blockId }; + const checkpointId = { + number: published.checkpoint.number, + hash: published.checkpoint.hash().toString(), + }; + return { type: 'chain-checkpointed' as const, checkpoint: checkpointId, block: blockId }; }; it('returns zero if no tips are stored', async () => { diff --git a/yarn-project/telemetry-client/src/wrappers/l2_block_stream.ts b/yarn-project/telemetry-client/src/wrappers/l2_block_stream.ts index 90b58b6a5a62..c6f49966cdb5 100644 --- a/yarn-project/telemetry-client/src/wrappers/l2_block_stream.ts +++ b/yarn-project/telemetry-client/src/wrappers/l2_block_stream.ts @@ -11,7 +11,7 @@ import { type Traceable, type Tracer, trackSpan } from '@aztec/telemetry-client' /** Extends an L2BlockStream with a tracer to create a new trace per iteration. */ export class TraceableL2BlockStream extends L2BlockStream implements Traceable { constructor( - l2BlockSource: Pick, + l2BlockSource: Pick, localData: L2BlockStreamLocalDataProvider, handler: L2BlockStreamEventHandler, public readonly tracer: Tracer, From 45ea78d349436804fa96025fcd941afb82d00ba3 Mon Sep 17 00:00:00 2001 From: Santiago Palladino Date: Fri, 12 Jun 2026 09:58:03 -0300 Subject: [PATCH 02/14] refactor!: remove proposedCheckpoint tip (#24008) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Motivation The `proposedCheckpoint` tip on `L2Tips` was the checkpoint frontier including L1-submitted-but-unconfirmed proposals, sitting between `proposed` (blocks) and `checkpointed` (confirmed). It existed only for proposer pipelining and had outlived its usefulness as a tip: - It was internal-only — never carried on the public RPC surface (`ChainTips` already excluded it), so no schema or storage migration is involved in removing it. - It was degenerate everywhere except the archiver: local stores (world-state, l2-tips-store) could never represent it (no stream event carries it; the cursor was always equal to `checkpointed`), which is why earlier work had already narrowed `LocalL2Tips` to omit it. - It was fabricated by shims purely to satisfy the type: the PXE node adapter and the TXE archiver both invented a `proposedCheckpoint: checkpointed` value. - Most importantly, the tip and its payload were read from two independent archiver snapshots — a JS-side tips cache for the tip vs. a direct store read on `#proposedCheckpoints` for the payload. A concurrent archiver write could be observed split, which forced explicit race-detection/refuse-to-proceed code in the sequencer's `checkSync`. ## Approach Drop the `proposedCheckpoint` tip from `L2Tips` entirely. The few consumers that needed the proposed-checkpoint frontier (the sequencer pipelining path, `automine`, and the node's `simulatePublicCalls`) read the payload directly from the existing archiver method: ```ts // L2BlockSource / archiver getProposedCheckpointData(): Promise; ``` With no argument this returns the latest proposed (not-yet-L1-confirmed) checkpoint payload, which is always the leading one: `addProposedCheckpoint` only stores a checkpoint number beyond the confirmed frontier (sequentiality is enforced), and every confirmation path (`addCheckpoints`, `promoteProposedToCheckpointed`) deletes the matching proposed entry in the same transaction. Consumers derive the proposed tip from the payload — checkpoint number, and last block = `startBlock + blockCount - 1` — so there is no separate tip to read and no tip-vs-payload split to reconcile. This removes the snapshot-race reconciliation code that previously lived in the sequencer's `checkSync`. The proposed-but-unconfirmed checkpoint frontier is archiver-internal. It is **not** exposed on the public node RPC surface, so the `'proposed'` checkpoint tag is removed from `getCheckpointNumber` and `getCheckpoint` (see breaking change below). ## Breaking change (RPC) The `'proposed'` checkpoint tag is no longer accepted by the node's `getCheckpointNumber(tip?)` and `getCheckpoint(param)` (and is dropped from `CheckpointParameter`). The proposed-but-unconfirmed checkpoint frontier was never a public concept; checkpoint-tip selectors are now `'checkpointed' | 'proven' | 'finalized'`. The block-level `'proposed'` tag (latest proposed block) on `getBlockNumber` / `getChainTips` is unchanged. ## API changes (internal) - `proposedCheckpoint` removed from `L2Tips` and `L2TipsSchema`; `'proposedCheckpoint'` removed from `L2BlockTag`. - `L2BlockSource.getProposedCheckpointData(query?)` is unchanged; its no-arg form returns the leading proposed-checkpoint payload (used by the sequencer / `simulatePublicCalls`), and its by-number/by-slot forms back the node's confirmed→proposed `getCheckpoint` fallback. - The `ChainTip` / `ChainTips` / `ChainTipSchema` / `ChainTipsSchema` aliases are removed. Block-level call sites use `L2BlockTag` / `L2Tips` / `L2TipsSchema` / `BlockTagWithoutLatestSchema` directly; checkpoint-tip selectors use the new `CheckpointTag` / `CheckpointTagSchema` (`'checkpointed' | 'proven' | 'finalized'`). `LocalL2Tips` continues to alias `L2Tips`. ## Changes - **stdlib**: removed the tip from `L2Tips`/`L2TipsSchema`/`L2BlockTag`; replaced the `ChainTip(s)` aliases with `CheckpointTag`/`CheckpointTagSchema` and direct `L2*` usage; narrowed `getCheckpointNumber`/`CheckpointParameter` to checkpoint tags; updated interface tests. - **archiver**: removed the `proposedCheckpoint` cursor from `getL2TipsData` (tips cache); rewrote `pruneOrphanProposedBlocks` to derive the proposed-checkpoint frontier block number from the payload store; cleaned the trace log and the mocks. - **sequencer-client**: `checkSync` consumes the proposed payload via `getProposedCheckpointData()`; **deleted the snapshot-race reconciliation block** (the inconsistent-proposed-checkpoint refuse-to-proceed guard) that only existed because tip and payload were read separately; `hasProposedCheckpoint` is derived from the payload. Migrated `automine_sequencer` next-checkpoint-number computation. - **aztec-node**: `getCheckpointNumber` and `getCheckpoint`/`resolveCheckpointParameter` no longer resolve the `'proposed'` checkpoint tag; `simulatePublicCalls` derives the proposed-checkpoint frontier from the payload (no extra block fetch in the leading case). `getChainTips` simplified. - **pxe / txe**: deleted the `proposedCheckpoint: tips.checkpointed` fabrication shims in the PXE node adapter and the TXE archiver. Fixes A-978 --- .../archiver/src/archiver-misc.test.ts | 5 +- .../archiver/src/archiver-sync.test.ts | 16 +- yarn-project/archiver/src/archiver.ts | 20 ++- .../archiver/src/store/block_store.test.ts | 35 ++-- .../archiver/src/store/block_store.ts | 37 ++-- .../src/test/mock_l1_to_l2_message_source.ts | 1 - .../archiver/src/test/mock_l2_block_source.ts | 23 +-- .../register_node_rpc_handlers.test.ts | 14 +- .../aztec-node/src/aztec-node/server.test.ts | 170 +++++++----------- .../aztec-node/src/aztec-node/server.ts | 57 +++--- .../epochs_missed_l1_publish.test.ts | 4 +- .../epochs_orphan_block_prune.test.ts | 4 +- .../e2e_l1_publisher/e2e_l1_publisher.test.ts | 1 - yarn-project/prover-node/src/prover-node.ts | 7 +- .../block_synchronizer/block_stream_source.ts | 5 +- .../sequencer/automine/automine_sequencer.ts | 8 +- .../sequencer/checkpoint_proposal_job.test.ts | 8 - .../checkpoint_proposal_job.timing.test.ts | 4 - .../src/sequencer/sequencer.test.ts | 63 ++----- .../src/sequencer/sequencer.ts | 35 +--- .../stdlib/src/block/l2_block_source.ts | 25 ++- .../l2_block_stream/l2_block_stream.test.ts | 10 +- .../stdlib/src/interfaces/archiver.test.ts | 2 - .../stdlib/src/interfaces/aztec-node.test.ts | 10 +- .../stdlib/src/interfaces/aztec-node.ts | 35 ++-- .../stdlib/src/interfaces/chain_tips.ts | 24 +-- .../src/interfaces/checkpoint_parameter.ts | 6 +- .../stdlib/src/interfaces/prover-node.test.ts | 2 - yarn-project/stdlib/src/tests/factories.ts | 4 - .../txe/src/state_machine/archiver.ts | 1 - 30 files changed, 229 insertions(+), 407 deletions(-) diff --git a/yarn-project/archiver/src/archiver-misc.test.ts b/yarn-project/archiver/src/archiver-misc.test.ts index 6fce5a928814..14612a3bff0f 100644 --- a/yarn-project/archiver/src/archiver-misc.test.ts +++ b/yarn-project/archiver/src/archiver-misc.test.ts @@ -210,16 +210,13 @@ describe('Archiver misc', () => { describe('isPruneDueAtSlot', () => { /** * Builds a fake L2Tips. `pending` is the L1-confirmed pending checkpoint (= `tips.checkpointed` - * in production). `proposedCheckpoint` is set to `pending + 1` to catch any implementation that - * accidentally reads the local-optimistic proposed checkpoint instead of the L1-confirmed one. + * in production). */ function makeTips(pending: CheckpointNumber, proven: CheckpointNumber): L2Tips { const block = { number: BlockNumber(0), hash: '0x' }; const tip = (n: CheckpointNumber) => ({ block, checkpoint: { number: n, hash: '0x' } }); - const proposedAhead = CheckpointNumber(Number(pending) + 1); return { proposed: block, - proposedCheckpoint: tip(proposedAhead), checkpointed: tip(pending), proven: tip(proven), finalized: tip(proven), diff --git a/yarn-project/archiver/src/archiver-sync.test.ts b/yarn-project/archiver/src/archiver-sync.test.ts index a5e1e2d6e886..f75422e67168 100644 --- a/yarn-project/archiver/src/archiver-sync.test.ts +++ b/yarn-project/archiver/src/archiver-sync.test.ts @@ -2075,11 +2075,15 @@ describe('Archiver Sync', () => { // Proposed checkpoint should still be set expect(await archiverStore.blocks.getLastProposedCheckpoint()).toBeDefined(); - // Proposed tip should be ahead of the checkpointed tip + // Proposed checkpoint should lead the checkpointed tip const tips = await archiver.getL2Tips(); - expect(tips.proposedCheckpoint.checkpoint.number).toEqual(CheckpointNumber(2)); + const proposedCheckpointResult = await archiver.getProposedCheckpointData(); + expect(proposedCheckpointResult).toBeDefined(); + expect(proposedCheckpointResult!.checkpointNumber).toEqual(CheckpointNumber(2)); expect(tips.checkpointed.checkpoint.number).toEqual(CheckpointNumber(1)); - expect(tips.proposedCheckpoint.block.number).toBeGreaterThan(tips.checkpointed.block.number); + const proposedCheckpointLastBlock = + proposedCheckpointResult!.startBlock + proposedCheckpointResult!.blockCount - 1; + expect(proposedCheckpointLastBlock).toBeGreaterThan(tips.checkpointed.block.number); }, 15_000); it('prunes blocks and clears stale pending checkpoint when slot ends', async () => { @@ -2142,11 +2146,9 @@ describe('Archiver Sync', () => { expect(await archiver.getBlockNumber()).toEqual(lastBlockInCheckpoint1); expect(await archiver.getCheckpointNumber()).toEqual(CheckpointNumber(1)); - // Proposed checkpoint should be cleared, so proposed tip falls back to checkpointed tip + // Proposed checkpoint should be cleared, so no proposed checkpoint leads the checkpointed tip expect(await archiverStore.blocks.getLastProposedCheckpoint()).toBeUndefined(); - const tips = await archiver.getL2Tips(); - expect(tips.proposedCheckpoint.checkpoint.number).toEqual(tips.checkpointed.checkpoint.number); - expect(tips.proposedCheckpoint.block.number).toEqual(tips.checkpointed.block.number); + expect(await archiver.getProposedCheckpointData()).toBeUndefined(); }, 15_000); }); diff --git a/yarn-project/archiver/src/archiver.ts b/yarn-project/archiver/src/archiver.ts index 8ea62a41dd2d..5619c2ce0224 100644 --- a/yarn-project/archiver/src/archiver.ts +++ b/yarn-project/archiver/src/archiver.ts @@ -5,7 +5,7 @@ import type { L1ContractAddresses } from '@aztec/ethereum/l1-contract-addresses' import type { ViemPublicClient, ViemPublicDebugClient } from '@aztec/ethereum/types'; import { BlockNumber, CheckpointNumber, EpochNumber, SlotNumber } from '@aztec/foundation/branded-types'; import { Buffer32 } from '@aztec/foundation/buffer'; -import { merge, pick } from '@aztec/foundation/collection'; +import { merge } from '@aztec/foundation/collection'; import { Fr } from '@aztec/foundation/curves/bn254'; import { EthAddress } from '@aztec/foundation/eth-address'; import { type Logger, createLogger } from '@aztec/foundation/log'; @@ -435,19 +435,23 @@ export class Archiver extends ArchiverDataSourceBase implements L2BlockSink, Tra const tips = await this.getL2Tips(); const now = BigInt(this.dateProvider.nowInSeconds()); + // Frontier block covered by a proposed (or, falling back, confirmed) checkpoint. Blocks beyond it + // have no enclosing checkpoint proposal and are the orphan-pruning candidates. + const proposedCheckpointBlockNumber = await this.stores.blocks.getProposedCheckpointL2BlockNumber(); + // The proposed tip is a proposed-checkpointed block, so there are no orphan proposed blocks to prune - if (tips.proposedCheckpoint.block.number === tips.proposed.number) { - this.log.trace( - `No orphan proposed blocks to prune: proposed tip ${tips.proposed.number} is checkpointed`, - pick(tips, 'proposed', 'proposedCheckpoint'), - ); + if (proposedCheckpointBlockNumber === tips.proposed.number) { + this.log.trace(`No orphan proposed blocks to prune: proposed tip ${tips.proposed.number} is checkpointed`, { + proposed: tips.proposed, + proposedCheckpointBlockNumber, + }); return; } // Load the blocks that are candidates for pruning (ie blocks without a proposed checkpoint covering them) const blocksWithoutProposedCheckpoint = await this.stores.blocks.getBlocksData({ - from: BlockNumber(tips.proposedCheckpoint.block.number + 1), - limit: tips.proposed.number - tips.proposedCheckpoint.block.number, + from: BlockNumber(proposedCheckpointBlockNumber + 1), + limit: tips.proposed.number - proposedCheckpointBlockNumber, }); // Iterate through them in order, the first one with a slot that should have received a proposed checkpoint diff --git a/yarn-project/archiver/src/store/block_store.test.ts b/yarn-project/archiver/src/store/block_store.test.ts index d7b3eb65d62e..31dbb3fe47a6 100644 --- a/yarn-project/archiver/src/store/block_store.test.ts +++ b/yarn-project/archiver/src/store/block_store.test.ts @@ -13,7 +13,6 @@ import { BlockHash, CommitteeAttestation, EthAddress, - GENESIS_BLOCK_HEADER_HASH, L2Block, type ValidateCheckpointResult, } from '@aztec/stdlib/block'; @@ -40,7 +39,6 @@ import { makeStateForBlock, } from '../test/mock_structs.js'; import { BlockStore } from './block_store.js'; -import { L2TipsCache } from './l2_tips_cache.js'; async function addProposedBlocks( blockStore: BlockStore, @@ -2627,26 +2625,19 @@ describe('BlockStore', () => { }); }); - describe('L2TipsCache proposedCheckpoint', () => { - it('returns proposedCheckpoint equal to checkpointed when no pending exists', async () => { - // Add checkpoint 1 with blocks 1-3 + describe('getLastProposedCheckpoint', () => { + it('returns undefined when there is no pending checkpoint', async () => { + // Add checkpoint 1 with blocks 1-3; confirmation deletes any matching proposed entry. const checkpoint1 = makePublishedCheckpoint( await Checkpoint.random(CheckpointNumber(1), { numBlocks: 3, startBlockNumber: 1 }), 10, ); await blockStore.addCheckpoints([checkpoint1]); - const l2TipsCache = new L2TipsCache(blockStore, GENESIS_BLOCK_HEADER_HASH); - const tips = await l2TipsCache.getL2Tips(); - - // proposedCheckpoint should always be defined - expect(tips.proposedCheckpoint).toBeDefined(); - // With no proposed checkpoint, it should equal the checkpointed tip - expect(tips.proposedCheckpoint!.block.number).toBe(tips.checkpointed.block.number); - expect(tips.proposedCheckpoint!.checkpoint.number).toBe(tips.checkpointed.checkpoint.number); + expect(await blockStore.getLastProposedCheckpoint()).toBeUndefined(); }); - it('returns proposedCheckpoint ahead of checkpointed when pending is set', async () => { + it('returns the leading proposed checkpoint payload', async () => { // Add checkpoint 1 const checkpoint1 = makePublishedCheckpoint( await Checkpoint.random(CheckpointNumber(1), { numBlocks: 1, startBlockNumber: 1 }), @@ -2663,21 +2654,23 @@ describe('BlockStore', () => { await blockStore.addProposedBlock(block2, { force: true }); // Set proposed checkpoint + const header = CheckpointHeader.empty(); await blockStore.addProposedCheckpoint({ checkpointNumber: CheckpointNumber(2), - header: CheckpointHeader.empty(), + header, startBlock: BlockNumber(2), blockCount: 1, totalManaUsed: 100n, feeAssetPriceModifier: 50n, }); - const l2TipsCache = new L2TipsCache(blockStore, GENESIS_BLOCK_HEADER_HASH); - const tips = await l2TipsCache.getL2Tips(); - - expect(tips.proposedCheckpoint).toBeDefined(); - expect(tips.proposedCheckpoint!.block.number).toBeGreaterThan(tips.checkpointed.block.number); - expect(tips.proposedCheckpoint!.checkpoint.number).toBeGreaterThan(tips.checkpointed.checkpoint.number); + const proposedCheckpoint = await blockStore.getLastProposedCheckpoint(); + expect(proposedCheckpoint).toBeDefined(); + // Callers derive the tip from the payload: last block = startBlock + blockCount - 1. + expect(proposedCheckpoint!.checkpointNumber).toBe(CheckpointNumber(2)); + expect(proposedCheckpoint!.startBlock).toBe(BlockNumber(2)); + expect(proposedCheckpoint!.blockCount).toBe(1); + expect(proposedCheckpoint!.totalManaUsed).toBe(100n); }); }); diff --git a/yarn-project/archiver/src/store/block_store.ts b/yarn-project/archiver/src/store/block_store.ts index 1901119016c4..c1615ef601e5 100644 --- a/yarn-project/archiver/src/store/block_store.ts +++ b/yarn-project/archiver/src/store/block_store.ts @@ -1167,14 +1167,13 @@ export class BlockStore { } /** - * Resolves all five L2 chain tips (proposed, proposedCheckpoint, checkpointed, proven, finalized) - * in a single read-only transaction so the snapshot is internally consistent. Each underlying - * record is read at most once: latest block, latest confirmed checkpoint, and latest pending - * checkpoint are each loaded directly (no separate "find the number, then look up data" hop), - * the proven/finalized checkpoint singletons are read once and their storage entries are - * reused if they coincide with the latest checkpoint, and per-tip block hashes are deduped - * when two tips land on the same block (e.g. finalized == proven, or proposedCheckpoint falls - * back to checkpointed when no pending checkpoint exists). + * Resolves all four L2 chain tips (proposed, checkpointed, proven, finalized) in a single + * read-only transaction so the snapshot is internally consistent. Each underlying record is + * read at most once: latest block and latest confirmed checkpoint are loaded directly (no + * separate "find the number, then look up data" hop), the proven/finalized checkpoint + * singletons are read once and their storage entries are reused if they coincide with the + * latest checkpoint, and per-tip block hashes are deduped when two tips land on the same block + * (e.g. finalized == proven). * * The result is guaranteed to satisfy `finalized <= proven <= checkpointed <= proposed` (by * block number). Genesis is represented by `(INITIAL_L2_BLOCK_NUM - 1)` and the supplied @@ -1197,9 +1196,6 @@ export class BlockStore { // Load latest block and checkpoint entries const [latestBlockEntry] = await toArray(this.#blocks.entriesAsync({ reverse: true, limit: 1 })); - const [proposedCheckpointEntry] = await toArray( - this.#proposedCheckpoints.entriesAsync({ reverse: true, limit: 1 }), - ); const [latestCheckpointEntry] = await toArray(this.#checkpoints.entriesAsync({ reverse: true, limit: 1 })); const latestCheckpointNumber = latestCheckpointEntry ? CheckpointNumber(latestCheckpointEntry[0]) @@ -1285,14 +1281,6 @@ export class BlockStore { const provenTip = await buildTipFromCheckpoint(provenCheckpoint); const finalizedTip = await buildTipFromCheckpoint(finalizedCheckpoint); - // Proposed checkpoint falls back to the checkpoint tip if it's not set. And if local storage is - // inconsistent and the proposed checkpoint is behind the checkpointed tip, we patch that and - // report the checkpointed tip as the proposed checkpoint to maintain the invariant. - const proposedCheckpointTip = - proposedCheckpointEntry === undefined || proposedCheckpointEntry[0] <= latestCheckpointNumber - ? checkpointedTip - : await buildTipFromCheckpoint(proposedCheckpointEntry[1]); - // A checkpointed block past the latest stored block would mean a checkpoint // references blocks that aren't in blocks. if (proposedBlockId.number < checkpointedTip.block.number) { @@ -1304,11 +1292,10 @@ export class BlockStore { // Assert that checkpoint numbers are increasing if ( finalizedTip.checkpoint.number > provenTip.checkpoint.number || - provenTip.checkpoint.number > checkpointedTip.checkpoint.number || - checkpointedTip.checkpoint.number > proposedCheckpointTip.checkpoint.number + provenTip.checkpoint.number > checkpointedTip.checkpoint.number ) { throw new Error( - `Inconsistent checkpoint numbers in chain tips: finalized=${finalizedTip.checkpoint.number} proven=${provenTip.checkpoint.number} checkpointed=${checkpointedTip.checkpoint.number} proposed=${proposedCheckpointTip.checkpoint.number}`, + `Inconsistent checkpoint numbers in chain tips: finalized=${finalizedTip.checkpoint.number} proven=${provenTip.checkpoint.number} checkpointed=${checkpointedTip.checkpoint.number}`, ); } @@ -1316,17 +1303,15 @@ export class BlockStore { if ( finalizedTip.block.number > provenTip.block.number || provenTip.block.number > checkpointedTip.block.number || - checkpointedTip.block.number > proposedCheckpointTip.block.number || - proposedCheckpointTip.block.number > proposedBlockId.number + checkpointedTip.block.number > proposedBlockId.number ) { throw new Error( - `Inconsistent block numbers in chain tips: finalized=${finalizedTip.block.number} proven=${provenTip.block.number} checkpointed=${checkpointedTip.block.number} proposedCheckpoint=${proposedCheckpointTip.block.number} proposed=${proposedBlockId.number}`, + `Inconsistent block numbers in chain tips: finalized=${finalizedTip.block.number} proven=${provenTip.block.number} checkpointed=${checkpointedTip.block.number} proposed=${proposedBlockId.number}`, ); } return { proposed: proposedBlockId, - proposedCheckpoint: proposedCheckpointTip, checkpointed: checkpointedTip, proven: provenTip, finalized: finalizedTip, diff --git a/yarn-project/archiver/src/test/mock_l1_to_l2_message_source.ts b/yarn-project/archiver/src/test/mock_l1_to_l2_message_source.ts index d2edd93167d7..f3b72a1c940f 100644 --- a/yarn-project/archiver/src/test/mock_l1_to_l2_message_source.ts +++ b/yarn-project/archiver/src/test/mock_l1_to_l2_message_source.ts @@ -44,7 +44,6 @@ export class MockL1ToL2MessageSource implements L1ToL2MessageSource { checkpointed: tip, proven: tip, finalized: tip, - proposedCheckpoint: tip, }); } } diff --git a/yarn-project/archiver/src/test/mock_l2_block_source.ts b/yarn-project/archiver/src/test/mock_l2_block_source.ts index 312c35e61c07..a9cee47f922f 100644 --- a/yarn-project/archiver/src/test/mock_l2_block_source.ts +++ b/yarn-project/archiver/src/test/mock_l2_block_source.ts @@ -55,7 +55,6 @@ export class MockL2BlockSource implements L2BlockSource, ContractDataSource { private provenBlockNumber: number = 0; private finalizedBlockNumber: number = 0; private checkpointedBlockNumber: number = 0; - private proposedCheckpointBlockNumber: number = 0; private initialHeader: BlockHeader = BlockHeader.empty(); private initialHeaderHash: BlockHash = GENESIS_BLOCK_HEADER_HASH; @@ -164,7 +163,6 @@ export class MockL2BlockSource implements L2BlockSource, ContractDataSource { }); // Keep tip numbers consistent with remaining blocks. this.checkpointedBlockNumber = Math.min(this.checkpointedBlockNumber, maxBlockNum); - this.proposedCheckpointBlockNumber = Math.min(this.proposedCheckpointBlockNumber, maxBlockNum); this.provenBlockNumber = Math.min(this.provenBlockNumber, maxBlockNum); this.finalizedBlockNumber = Math.min(this.finalizedBlockNumber, maxBlockNum); this.log.verbose(`Removed ${numBlocks} blocks from the mock L2 block source`); @@ -181,17 +179,9 @@ export class MockL2BlockSource implements L2BlockSource, ContractDataSource { this.finalizedBlockNumber = finalizedBlockNumber; } - public setProposedCheckpointBlockNumber(blockNumber: number) { - this.proposedCheckpointBlockNumber = blockNumber; - } - public setCheckpointedBlockNumber(checkpointedBlockNumber: number) { const prevCheckpointed = this.checkpointedBlockNumber; this.checkpointedBlockNumber = checkpointedBlockNumber; - // Proposed checkpoint is always at least as advanced as checkpointed - if (this.proposedCheckpointBlockNumber < checkpointedBlockNumber) { - this.proposedCheckpointBlockNumber = checkpointedBlockNumber; - } // Auto-create single-block checkpoints for newly checkpointed blocks that don't have one yet. // This handles blocks added via addProposedBlocks that are now being marked as checkpointed. const newCheckpoints: Checkpoint[] = []; @@ -255,10 +245,6 @@ export class MockL2BlockSource implements L2BlockSource, ContractDataSource { return block ? block.header.globalVariables.blockNumber : undefined; } - public getProposedCheckpointL2BlockNumber() { - return Promise.resolve(BlockNumber(this.proposedCheckpointBlockNumber)); - } - public getCheckpoint(query: CheckpointQuery): Promise { const checkpoint = this.resolveCheckpointQuery(query); if (!checkpoint) { @@ -373,19 +359,17 @@ export class MockL2BlockSource implements L2BlockSource, ContractDataSource { } async getL2Tips(): Promise { - const [latest, proven, finalized, checkpointed, proposedCheckpoint] = [ + const [latest, proven, finalized, checkpointed] = [ await this.getBlockNumber(), this.provenBlockNumber, this.finalizedBlockNumber, this.checkpointedBlockNumber, - await this.getProposedCheckpointL2BlockNumber(), ] as const; const latestBlock = this.l2Blocks[latest - 1]; const provenBlock = this.l2Blocks[proven - 1]; const finalizedBlock = this.l2Blocks[finalized - 1]; const checkpointedBlock = this.l2Blocks[checkpointed - 1]; - const proposedCheckpointBlock = this.l2Blocks[proposedCheckpoint - 1]; // For genesis tips (block number 0) report the dynamic initial header hash so consumers // running L2BlockStream against this mock agree at block 0 with their local tip store. @@ -413,10 +397,6 @@ export class MockL2BlockSource implements L2BlockSource, ContractDataSource { number: BlockNumber(checkpointed), hash: await tipHash(checkpointedBlock, checkpointed), }; - const proposedCheckpointBlockId = { - number: BlockNumber(proposedCheckpoint), - hash: await tipHash(proposedCheckpointBlock, proposedCheckpoint), - }; const makeTipId = (blockId: typeof latestBlockId) => { const checkpointNumber = this.findCheckpointNumberForBlock(blockId.number) ?? CheckpointNumber(0); @@ -435,7 +415,6 @@ export class MockL2BlockSource implements L2BlockSource, ContractDataSource { checkpointed: makeTipId(checkpointedBlockId), proven: makeTipId(provenBlockId), finalized: makeTipId(finalizedBlockId), - proposedCheckpoint: makeTipId(proposedCheckpointBlockId), }; } diff --git a/yarn-project/aztec-node/src/aztec-node/register_node_rpc_handlers.test.ts b/yarn-project/aztec-node/src/aztec-node/register_node_rpc_handlers.test.ts index dd71b0e627ce..d35a9d531ddc 100644 --- a/yarn-project/aztec-node/src/aztec-node/register_node_rpc_handlers.test.ts +++ b/yarn-project/aztec-node/src/aztec-node/register_node_rpc_handlers.test.ts @@ -6,19 +6,15 @@ import { createNamespacedSafeJsonRpcServer, startHttpRpcServer, } from '@aztec/foundation/json-rpc/server'; -import { - AztecNodeAdminApiSchema, - AztecNodeApiSchema, - AztecNodeDebugApiSchema, - type ChainTips, -} from '@aztec/stdlib/interfaces/client'; +import type { L2Tips } from '@aztec/stdlib/block'; +import { AztecNodeAdminApiSchema, AztecNodeApiSchema, AztecNodeDebugApiSchema } from '@aztec/stdlib/interfaces/client'; import { P2PApiSchema } from '@aztec/stdlib/interfaces/server'; import type { ApiSchemaFor } from '@aztec/stdlib/schemas'; import { registerAztecNodeRpcHandlers } from './register_node_rpc_handlers.js'; import type { AztecNodeService } from './server.js'; -type GetChainTipsOnly = { getChainTips(): Promise }; +type GetChainTipsOnly = { getChainTips(): Promise }; const GetChainTipsOnlySchema: ApiSchemaFor = { getChainTips: AztecNodeApiSchema.getChainTips, @@ -28,7 +24,7 @@ const p2p = {}; const mockNode = { getP2P: () => p2p, - getChainTips(): Promise { + getChainTips(): Promise { const tipId = { block: { number: BlockNumber(1), hash: `0x01` }, checkpoint: { number: CheckpointNumber(1), hash: `0x01` }, @@ -90,7 +86,7 @@ describe('registerAztecNodeRpcHandlers', () => { headers: { 'content-type': 'application/json' }, body: jsonStringify({ jsonrpc: '2.0', id: 1, method: 'node_getChainTips', params: [] }), }); - const body = (await response.json()) as { result: ChainTips }; + const body = (await response.json()) as { result: L2Tips }; expect(body.result).toEqual(expected); httpServer.close(); diff --git a/yarn-project/aztec-node/src/aztec-node/server.test.ts b/yarn-project/aztec-node/src/aztec-node/server.test.ts index 8d84ab641270..24d6df5b158f 100644 --- a/yarn-project/aztec-node/src/aztec-node/server.test.ts +++ b/yarn-project/aztec-node/src/aztec-node/server.test.ts @@ -744,16 +744,14 @@ describe('aztec node', () => { const checkpointNumber = CheckpointNumber(1); const proposedCheckpointBlockNumber = BlockNumber(9); const targetSlot = SlotNumber(10); - l2BlockSource.getL2Tips.mockResolvedValue( - makeTips({ - proposed: proposedCheckpointBlockNumber, - proposedCheckpoint: checkpointNumber, - proposedCheckpointBlock: proposedCheckpointBlockNumber, + l2BlockSource.getL2Tips.mockResolvedValue(makeTips({ proposed: proposedCheckpointBlockNumber })); + l2BlockSource.getProposedCheckpointData.mockResolvedValue( + makeProposedCheckpoint({ + checkpointNumber, + blockNumber: proposedCheckpointBlockNumber, + slotNumber: SlotNumber(9), }), ); - l2BlockSource.getBlockData.mockResolvedValue( - makeSimulationBlockData(proposedCheckpointBlockNumber, SlotNumber(9), checkpointNumber), - ); mockNextL1Slot(SlotNumber(5)); globalVariablesBuilder.buildCheckpointGlobalVariables.mockResolvedValue({ chainId, @@ -767,7 +765,7 @@ describe('aztec node', () => { await expect(node.simulatePublicCalls(tx)).rejects.toThrow(); - expect(l2BlockSource.getBlockData).toHaveBeenCalledWith({ number: proposedCheckpointBlockNumber }); + // Slot is read from the proposed checkpoint payload header, so no block fetch is needed for it. expect(globalVariablesBuilder.buildGlobalVariables).not.toHaveBeenCalled(); expect(globalVariablesBuilder.buildCheckpointGlobalVariables).toHaveBeenCalledWith( EthAddress.ZERO, @@ -781,16 +779,14 @@ describe('aztec node', () => { const checkpointNumber = CheckpointNumber(1); const proposedCheckpointBlockNumber = BlockNumber(9); const targetSlot = SlotNumber(12); - l2BlockSource.getL2Tips.mockResolvedValue( - makeTips({ - proposed: proposedCheckpointBlockNumber, - proposedCheckpoint: checkpointNumber, - proposedCheckpointBlock: proposedCheckpointBlockNumber, + l2BlockSource.getL2Tips.mockResolvedValue(makeTips({ proposed: proposedCheckpointBlockNumber })); + l2BlockSource.getProposedCheckpointData.mockResolvedValue( + makeProposedCheckpoint({ + checkpointNumber, + blockNumber: proposedCheckpointBlockNumber, + slotNumber: SlotNumber(9), }), ); - l2BlockSource.getBlockData.mockResolvedValue( - makeSimulationBlockData(proposedCheckpointBlockNumber, SlotNumber(9), checkpointNumber), - ); mockNextL1Slot(targetSlot); globalVariablesBuilder.buildCheckpointGlobalVariables.mockResolvedValue({ chainId, @@ -804,7 +800,6 @@ describe('aztec node', () => { await expect(node.simulatePublicCalls(tx)).rejects.toThrow(); - expect(l2BlockSource.getBlockData).toHaveBeenCalledWith({ number: proposedCheckpointBlockNumber }); expect(globalVariablesBuilder.buildGlobalVariables).not.toHaveBeenCalled(); expect(globalVariablesBuilder.buildCheckpointGlobalVariables).toHaveBeenCalledWith( EthAddress.ZERO, @@ -819,24 +814,17 @@ describe('aztec node', () => { const proposedCheckpointBlockNumber = BlockNumber(9); const latestProposedBlockNumber = BlockNumber(12); const targetSlot = SlotNumber(12); - l2BlockSource.getL2Tips.mockResolvedValue( - makeTips({ - proposed: latestProposedBlockNumber, - proposedCheckpoint: checkpointNumber, - proposedCheckpointBlock: proposedCheckpointBlockNumber, + l2BlockSource.getL2Tips.mockResolvedValue(makeTips({ proposed: latestProposedBlockNumber })); + l2BlockSource.getProposedCheckpointData.mockResolvedValue( + makeProposedCheckpoint({ + checkpointNumber, + blockNumber: proposedCheckpointBlockNumber, + slotNumber: SlotNumber(9), }), ); - l2BlockSource.getBlockData.mockImplementation(query => { - if (!('number' in query)) { - return Promise.resolve(undefined); - } - if (query.number === proposedCheckpointBlockNumber) { - return Promise.resolve( - makeSimulationBlockData(proposedCheckpointBlockNumber, SlotNumber(9), checkpointNumber), - ); - } - return Promise.resolve(makeSimulationBlockData(latestProposedBlockNumber, targetSlot, checkpointNumber)); - }); + l2BlockSource.getBlockData.mockResolvedValue( + makeSimulationBlockData(latestProposedBlockNumber, targetSlot, checkpointNumber), + ); mockNextL1Slot(SlotNumber(5)); globalVariablesBuilder.buildCheckpointGlobalVariables.mockResolvedValue({ chainId, @@ -850,7 +838,7 @@ describe('aztec node', () => { await expect(node.simulatePublicCalls(tx)).rejects.toThrow(); - expect(l2BlockSource.getBlockData).toHaveBeenCalledWith({ number: proposedCheckpointBlockNumber }); + // The latest proposed block is ahead of the proposed checkpoint, so its slot is fetched. expect(l2BlockSource.getBlockData).toHaveBeenCalledWith({ number: latestProposedBlockNumber }); expect(globalVariablesBuilder.buildGlobalVariables).not.toHaveBeenCalled(); expect(globalVariablesBuilder.buildCheckpointGlobalVariables).toHaveBeenCalledWith( @@ -866,11 +854,12 @@ describe('aztec node', () => { const proposedCheckpointBlockNumber = BlockNumber(9); const latestProposedBlockNumber = BlockNumber(12); const targetSlot = SlotNumber(13); - l2BlockSource.getL2Tips.mockResolvedValue( - makeTips({ - proposed: latestProposedBlockNumber, - proposedCheckpoint: checkpointNumber, - proposedCheckpointBlock: proposedCheckpointBlockNumber, + l2BlockSource.getL2Tips.mockResolvedValue(makeTips({ proposed: latestProposedBlockNumber })); + l2BlockSource.getProposedCheckpointData.mockResolvedValue( + makeProposedCheckpoint({ + checkpointNumber, + blockNumber: proposedCheckpointBlockNumber, + slotNumber: SlotNumber(9), }), ); l2BlockSource.getBlockData.mockResolvedValue(undefined); @@ -887,7 +876,7 @@ describe('aztec node', () => { await expect(node.simulatePublicCalls(tx)).rejects.toThrow(); - expect(l2BlockSource.getBlockData).toHaveBeenCalledWith({ number: proposedCheckpointBlockNumber }); + // Latest proposed block slot is unavailable; falls back to the next L1 timestamp slot. expect(l2BlockSource.getBlockData).toHaveBeenCalledWith({ number: latestProposedBlockNumber }); expect(globalVariablesBuilder.buildGlobalVariables).not.toHaveBeenCalled(); expect(globalVariablesBuilder.buildCheckpointGlobalVariables).toHaveBeenCalledWith( @@ -902,13 +891,12 @@ describe('aztec node', () => { const checkpointNumber = CheckpointNumber(0); const proposedCheckpointBlockNumber = BlockNumber(0); const targetSlot = SlotNumber(1); + // No proposed checkpoint leads the frontier; the proposed-checkpoint frontier falls back to the + // checkpointed tip (block 0, slot 0), whose slot is read via getBlockData. l2BlockSource.getL2Tips.mockResolvedValue( - makeTips({ - proposed: proposedCheckpointBlockNumber, - proposedCheckpoint: checkpointNumber, - proposedCheckpointBlock: proposedCheckpointBlockNumber, - }), + makeTips({ proposed: proposedCheckpointBlockNumber, checkpointed: checkpointNumber }), ); + l2BlockSource.getProposedCheckpointData.mockResolvedValue(undefined); l2BlockSource.getBlockData.mockResolvedValue( makeSimulationBlockData(proposedCheckpointBlockNumber, SlotNumber(0), checkpointNumber), ); @@ -1405,9 +1393,8 @@ describe('aztec node', () => { /** Builds an L2Tips stub with the given checkpoint numbers per tip. */ function makeTips(args: { proposed?: BlockNumber; - proposedCheckpointBlock?: BlockNumber; - proposedCheckpoint?: CheckpointNumber; checkpointed?: CheckpointNumber; + checkpointedBlock?: BlockNumber; proven?: CheckpointNumber; finalized?: CheckpointNumber; }): L2Tips { @@ -1418,13 +1405,30 @@ describe('aztec node', () => { }); return { proposed: makeBlockId(args.proposed), - checkpointed: makeTipId(args.checkpointed ?? CheckpointNumber(0)), - proposedCheckpoint: makeTipId(args.proposedCheckpoint ?? CheckpointNumber(0), args.proposedCheckpointBlock), + checkpointed: makeTipId(args.checkpointed ?? CheckpointNumber(0), args.checkpointedBlock), proven: makeTipId(args.proven ?? CheckpointNumber(0)), finalized: makeTipId(args.finalized ?? CheckpointNumber(0)), }; } + /** Builds the payload of the atomic leading-proposed-checkpoint read (last block = startBlock). */ + function makeProposedCheckpoint(args: { + checkpointNumber: CheckpointNumber; + blockNumber: BlockNumber; + slotNumber: SlotNumber; + }): ProposedCheckpointData { + return { + checkpointNumber: args.checkpointNumber, + header: CheckpointHeader.random({ slotNumber: args.slotNumber }), + archive: AppendOnlyTreeSnapshot.empty(), + checkpointOutHash: Fr.ZERO, + startBlock: args.blockNumber, + blockCount: 1, + totalManaUsed: 0n, + feeAssetPriceModifier: 0n, + }; + } + describe('getCheckpoint', () => { /** Builds a minimal ProposedCheckpointData stub. */ function makeProposedCheckpointData( @@ -1459,26 +1463,6 @@ describe('aztec node', () => { } describe('throw guards', () => { - it('throws BadRequestError when "proposed" resolves to a proposed entry and includeL1PublishInfo is requested', async () => { - l2BlockSource.getL2Tips.mockResolvedValue(makeTips({ proposedCheckpoint: CheckpointNumber(5) })); - l2BlockSource.getCheckpointData.mockResolvedValue(undefined); - l2BlockSource.getProposedCheckpointData.mockResolvedValue( - makeProposedCheckpointData(CheckpointNumber(5), SlotNumber(10)), - ); - - await expect(node.getCheckpoint('proposed', { includeL1PublishInfo: true })).rejects.toThrow(BadRequestError); - }); - - it('throws BadRequestError when "proposed" resolves to a proposed entry and includeAttestations is requested', async () => { - l2BlockSource.getL2Tips.mockResolvedValue(makeTips({ proposedCheckpoint: CheckpointNumber(5) })); - l2BlockSource.getCheckpointData.mockResolvedValue(undefined); - l2BlockSource.getProposedCheckpointData.mockResolvedValue( - makeProposedCheckpointData(CheckpointNumber(5), SlotNumber(10)), - ); - - await expect(node.getCheckpoint('proposed', { includeAttestations: true })).rejects.toThrow(BadRequestError); - }); - it('throws BadRequestError when number lookup resolves to a proposed entry and includeL1PublishInfo is requested', async () => { l2BlockSource.getCheckpointData.mockResolvedValue(undefined); l2BlockSource.getProposedCheckpointData.mockResolvedValue( @@ -1503,30 +1487,6 @@ describe('aztec node', () => { }); describe('fallback semantics', () => { - it('getCheckpoint("proposed") returns the projected proposed entry when one exists at the proposed-tip number', async () => { - l2BlockSource.getL2Tips.mockResolvedValue(makeTips({ proposedCheckpoint: CheckpointNumber(2) })); - l2BlockSource.getCheckpointData.mockResolvedValue(undefined); - const proposed = makeProposedCheckpointData(CheckpointNumber(2), SlotNumber(5)); - l2BlockSource.getProposedCheckpointData.mockResolvedValue(proposed); - - const result = await node.getCheckpoint('proposed'); - expect(result).toBeDefined(); - expect(result!.number).toEqual(CheckpointNumber(2)); - }); - - it('getCheckpoint("proposed") returns the latest confirmed checkpoint when no proposed entry exists', async () => { - // When no proposed entry exists, the proposedCheckpoint tip falls back to the confirmed tip. - l2BlockSource.getL2Tips.mockResolvedValue( - makeTips({ proposedCheckpoint: CheckpointNumber(3), checkpointed: CheckpointNumber(3) }), - ); - const confirmed = makeCheckpointData(CheckpointNumber(3)); - l2BlockSource.getCheckpointData.mockResolvedValue(confirmed); - - const result = await node.getCheckpoint('proposed'); - expect(result).toBeDefined(); - expect(result!.number).toEqual(CheckpointNumber(3)); - }); - it('getCheckpoint({ number }) returns the confirmed entry when one exists', async () => { const confirmed = makeCheckpointData(CheckpointNumber(3)); l2BlockSource.getCheckpointData.mockResolvedValue(confirmed); @@ -1636,22 +1596,23 @@ describe('aztec node', () => { }); describe('getCheckpointNumber', () => { - it('returns the proposed checkpoint number from proposedCheckpoint tip', async () => { + beforeEach(() => { l2BlockSource.getL2Tips.mockResolvedValue( - makeTips({ proposedCheckpoint: CheckpointNumber(7), checkpointed: CheckpointNumber(5) }), + makeTips({ checkpointed: CheckpointNumber(5), proven: CheckpointNumber(3), finalized: CheckpointNumber(2) }), ); + }); - const result = await node.getCheckpointNumber('proposed'); - expect(result).toEqual(CheckpointNumber(7)); + it('returns the checkpointed number by default', async () => { + expect(await node.getCheckpointNumber()).toEqual(CheckpointNumber(5)); + expect(await node.getCheckpointNumber('checkpointed')).toEqual(CheckpointNumber(5)); }); - it('returns the proposedCheckpoint tip number when it equals the confirmed checkpoint (fallback already baked in)', async () => { - l2BlockSource.getL2Tips.mockResolvedValue( - makeTips({ proposedCheckpoint: CheckpointNumber(5), checkpointed: CheckpointNumber(5) }), - ); + it('returns the proven checkpoint number', async () => { + expect(await node.getCheckpointNumber('proven')).toEqual(CheckpointNumber(3)); + }); - const result = await node.getCheckpointNumber('proposed'); - expect(result).toEqual(CheckpointNumber(5)); + it('returns the finalized checkpoint number', async () => { + expect(await node.getCheckpointNumber('finalized')).toEqual(CheckpointNumber(2)); }); }); @@ -1691,7 +1652,6 @@ describe('aztec node', () => { return { proposed: blockId(args.proposed), checkpointed: tipId(args.checkpointed), - proposedCheckpoint: tipId(args.checkpointed), proven: tipId(args.proven), finalized: tipId(args.finalized), }; diff --git a/yarn-project/aztec-node/src/aztec-node/server.ts b/yarn-project/aztec-node/src/aztec-node/server.ts index deea63b5e72f..06ab33134232 100644 --- a/yarn-project/aztec-node/src/aztec-node/server.ts +++ b/yarn-project/aztec-node/src/aztec-node/server.ts @@ -70,6 +70,7 @@ import { type CommitteeAttestation, type DataInBlock, type L2BlockSource, + type L2BlockTag, type L2Tips, type NormalizedBlockParameter, inspectBlockParameter, @@ -98,11 +99,10 @@ import type { BlockIncludeOptions, BlockResponse, BlocksIncludeOptions, - ChainTip, - ChainTips, CheckpointIncludeOptions, CheckpointParameter, CheckpointResponse, + CheckpointTag, GetTxByHashOptions, PeerInfo, ProposalsForSlot, @@ -249,9 +249,8 @@ export class AztecNodeService implements AztecNode, AztecNodeAdmin, AztecNodeDeb return status.syncSummary; } - public async getChainTips(): Promise { - const { proposed, checkpointed, proven, finalized } = await this.blockSource.getL2Tips(); - return { proposed, checkpointed, proven, finalized }; + public getChainTips(): Promise { + return this.blockSource.getL2Tips(); } public getL1Constants() { @@ -274,21 +273,19 @@ export class AztecNodeService implements AztecNode, AztecNodeAdmin, AztecNodeDeb return this.blockSource.getCheckpointsData(query); } - public async getBlockNumber(tip?: ChainTip): Promise { + public async getBlockNumber(tip?: L2BlockTag): Promise { if (tip === undefined || tip === 'proposed') { return this.blockSource.getBlockNumber(); } return (await this.blockSource.getBlockNumber({ tag: tip })) ?? BlockNumber.ZERO; } - public async getCheckpointNumber(tip?: ChainTip): Promise { + public async getCheckpointNumber(tip?: CheckpointTag): Promise { const tips = await this.blockSource.getL2Tips(); switch (tip) { case undefined: case 'checkpointed': return tips.checkpointed.checkpoint.number; - case 'proposed': - return tips.proposedCheckpoint.checkpoint.number; case 'proven': return tips.proven.checkpoint.number; case 'finalized': @@ -296,8 +293,8 @@ export class AztecNodeService implements AztecNode, AztecNodeAdmin, AztecNodeDeb } } - private isChainTip(value: unknown): value is ChainTip { - return value === 'proposed' || value === 'checkpointed' || value === 'proven' || value === 'finalized'; + private isCheckpointTag(value: unknown): value is CheckpointTag { + return value === 'checkpointed' || value === 'proven' || value === 'finalized'; } /** @@ -345,10 +342,10 @@ export class AztecNodeService implements AztecNode, AztecNodeAdmin, AztecNodeDeb /** * Resolves a {@link CheckpointParameter} into a concrete `{ number }` or `{ slot }` query. * - * Tag-based parameters (`'proposed'`, `'checkpointed'`, `'proven'`, `'finalized'`) are - * translated up-front to the corresponding tip's checkpoint number via {@link L2BlockSource.getL2Tips}. - * After resolution the unified {@link getCheckpoint} flow can perform a single - * confirmed→proposed lookup against either store. + * Tag-based parameters (`'checkpointed'`, `'proven'`, `'finalized'`) are translated up-front to the + * corresponding tip's checkpoint number via {@link L2BlockSource.getL2Tips}. After resolution the + * unified {@link getCheckpoint} flow can perform a single confirmed→proposed lookup against either + * store. */ private async resolveCheckpointParameter( param: CheckpointParameter, @@ -356,11 +353,9 @@ export class AztecNodeService implements AztecNode, AztecNodeAdmin, AztecNodeDeb if (typeof param === 'number') { return { number: param as CheckpointNumber }; } - if (this.isChainTip(param)) { + if (this.isCheckpointTag(param)) { const tips = await this.blockSource.getL2Tips(); switch (param) { - case 'proposed': - return { number: tips.proposedCheckpoint.checkpoint.number }; case 'checkpointed': return { number: tips.checkpointed.checkpoint.number }; case 'proven': @@ -1666,19 +1661,27 @@ export class AztecNodeService implements AztecNode, AztecNodeAdmin, AztecNodeDeb const coinbase = EthAddress.ZERO; const feeRecipient = AztecAddress.ZERO; + // Resolve the proposed-checkpoint frontier (latest proposed checkpoint, which leads the + // checkpointed tip, falling back to the checkpointed tip when none exists). The proposed payload + // carries its header slot, so no extra block fetch is needed to derive the slot. + const proposedCheckpoint = await this.blockSource.getProposedCheckpointData(); + const proposedCheckpointBlockNumber = proposedCheckpoint + ? BlockNumber(proposedCheckpoint.startBlock + proposedCheckpoint.blockCount - 1) + : l2Tips.checkpointed.block.number; + const proposedCheckpointNumber = proposedCheckpoint?.checkpointNumber ?? l2Tips.checkpointed.checkpoint.number; + // Define the slot for simulation as the max of the next L1 timestamp slot, the slot after the proposed // checkpoint, and the latest proposed block's slot. - const proposedCheckpointBlockData = await this.blockSource.getBlockData({ - number: l2Tips.proposedCheckpoint.block.number, - }); - const proposedCheckpointSlot = proposedCheckpointBlockData?.header.getSlot(); + const proposedCheckpointSlot = + proposedCheckpoint?.header.slotNumber ?? + (await this.blockSource.getBlockData({ number: proposedCheckpointBlockNumber }))?.header.getSlot(); let slotAfterProposedCheckpoint: SlotNumber | undefined; if (proposedCheckpointSlot !== undefined) { slotAfterProposedCheckpoint = SlotNumber.fromBigInt(BigInt(proposedCheckpointSlot) + 1n); } let latestProposedBlockSlot: SlotNumber | undefined; - if (l2Tips.proposed.number > l2Tips.proposedCheckpoint.block.number) { + if (l2Tips.proposed.number > proposedCheckpointBlockNumber) { latestProposedBlockSlot = ( await this.blockSource.getBlockData({ number: l2Tips.proposed.number }) )?.header.getSlot(); @@ -1715,11 +1718,9 @@ export class AztecNodeService implements AztecNode, AztecNodeAdmin, AztecNodeDeb // the world state tree so simulation can take them into account. We detect if the next block would // start a new checkpoint by checking if the proposed checkpoint's block number matches the latest block number, // which means the next block would be the first block of the next checkpoint. - const targetCheckpoint = CheckpointNumber( - (l2Tips.proposedCheckpoint.checkpoint.number ?? CheckpointNumber.ZERO) + 1, - ); + const targetCheckpoint = CheckpointNumber(proposedCheckpointNumber + 1); const nextCheckpointMessages: Fr[] | undefined = - l2Tips.proposedCheckpoint.block.number === l2Tips.proposed.number + proposedCheckpointBlockNumber === l2Tips.proposed.number ? await this.l1ToL2MessageSource.getL1ToL2Messages(targetCheckpoint).catch(err => { if (isErrorClass(err, L1ToL2MessagesNotReadyError)) { this.log.warn( @@ -1741,7 +1742,7 @@ export class AztecNodeService implements AztecNode, AztecNodeAdmin, AztecNodeDeb if (nextCheckpointMessages !== undefined) { this.log.debug( `Appending ${nextCheckpointMessages.length} L1-to-L2 messages to the world state tree for the next checkpoint`, - { checkpointNumber: l2Tips.proposedCheckpoint.checkpoint.number + 1 }, + { checkpointNumber: targetCheckpoint }, ); await appendL1ToL2MessagesToTree(merkleTreeFork, nextCheckpointMessages); } diff --git a/yarn-project/end-to-end/src/e2e_epochs/epochs_missed_l1_publish.test.ts b/yarn-project/end-to-end/src/e2e_epochs/epochs_missed_l1_publish.test.ts index fb5aa4afd1ce..d7cfa7be8bd6 100644 --- a/yarn-project/end-to-end/src/e2e_epochs/epochs_missed_l1_publish.test.ts +++ b/yarn-project/end-to-end/src/e2e_epochs/epochs_missed_l1_publish.test.ts @@ -12,8 +12,8 @@ import { retryUntil } from '@aztec/foundation/retry'; import { bufferToHex } from '@aztec/foundation/string'; import { timeoutPromise } from '@aztec/foundation/timer'; import { type L2Block, L2BlockSourceEvents } from '@aztec/stdlib/block'; +import type { L2Tips } from '@aztec/stdlib/block'; import { getTimestampForSlot } from '@aztec/stdlib/epoch-helpers'; -import type { ChainTips } from '@aztec/stdlib/interfaces/server'; import { jest } from '@jest/globals'; import { privateKeyToAccount } from 'viem/accounts'; @@ -178,7 +178,7 @@ describe('e2e_epochs/epochs_missed_l1_publish', () => { // We capture the L2 tips synchronously inside the handler — the archiver has already removed // the pruned blocks at emit time, so this snapshot reflects the rolled-back state before any // new pipelined block can be applied. - type PruneObservation = { slotNumber: SlotNumber; blocks: L2Block[]; tipsAtPrune: ChainTips }; + type PruneObservation = { slotNumber: SlotNumber; blocks: L2Block[]; tipsAtPrune: L2Tips }; const prunePromises: Promise[] = nodes.map( (node, idx) => new Promise(resolve => { diff --git a/yarn-project/end-to-end/src/e2e_epochs/epochs_orphan_block_prune.test.ts b/yarn-project/end-to-end/src/e2e_epochs/epochs_orphan_block_prune.test.ts index dbf1a088b277..cca27d9b2b23 100644 --- a/yarn-project/end-to-end/src/e2e_epochs/epochs_orphan_block_prune.test.ts +++ b/yarn-project/end-to-end/src/e2e_epochs/epochs_orphan_block_prune.test.ts @@ -11,8 +11,8 @@ import { retryUntil } from '@aztec/foundation/retry'; import { bufferToHex } from '@aztec/foundation/string'; import { timeoutPromise } from '@aztec/foundation/timer'; import { type L2Block, L2BlockSourceEvents } from '@aztec/stdlib/block'; +import type { L2Tips } from '@aztec/stdlib/block'; import { getTimestampForSlot } from '@aztec/stdlib/epoch-helpers'; -import type { ChainTips } from '@aztec/stdlib/interfaces/server'; import { jest } from '@jest/globals'; import { privateKeyToAccount } from 'viem/accounts'; @@ -157,7 +157,7 @@ describe('e2e_epochs/epochs_orphan_block_prune', () => { // Subscribe to the prune event on every node before sequencers start, so we never miss it. We capture the chain // tips asynchronously inside the handler for log context, but do not assert on them — by the time the snapshot is // read, P2's rebuild may already have landed. - type PruneObservation = { slotNumber: SlotNumber; blocks: L2Block[]; tipsAtPrune: ChainTips }; + type PruneObservation = { slotNumber: SlotNumber; blocks: L2Block[]; tipsAtPrune: L2Tips }; const prunePromises: Promise[] = nodes.map( (node, idx) => new Promise(resolve => { diff --git a/yarn-project/end-to-end/src/e2e_l1_publisher/e2e_l1_publisher.test.ts b/yarn-project/end-to-end/src/e2e_l1_publisher/e2e_l1_publisher.test.ts index 863e08e69bbc..b6a7da0e593c 100644 --- a/yarn-project/end-to-end/src/e2e_l1_publisher/e2e_l1_publisher.test.ts +++ b/yarn-project/end-to-end/src/e2e_l1_publisher/e2e_l1_publisher.test.ts @@ -319,7 +319,6 @@ describe('L1Publisher integration', () => { checkpointed: tipId, proven: tipId, finalized: tipId, - proposedCheckpoint: tipId, }; }, getBlockNumber(): Promise { diff --git a/yarn-project/prover-node/src/prover-node.ts b/yarn-project/prover-node/src/prover-node.ts index 296ec231a724..3c22c1c2a94d 100644 --- a/yarn-project/prover-node/src/prover-node.ts +++ b/yarn-project/prover-node/src/prover-node.ts @@ -268,12 +268,11 @@ export class ProverNode implements L2BlockStreamEventHandler, ProverNodeApi, Tra } const l1Constants = await this.getL1Constants(); - // Cap the catch-up at the two most recent epochs' worth of checkpoints. With at most one checkpoint per - // slot, an epoch spans at most `epochDuration` checkpoints, so two epochs is `2 * epochDuration`. When the - // cursor is much further behind (e.g. resyncing after a long time offline), fetching the whole gap could + // Cap the catch-up at the `(proofSubmissionEpochs + 1) * epochDuration` most recent checkpoints. + // When the cursor is much further behind (e.g. resyncing after a long time offline), fetching the whole gap could // load thousands of checkpoints we cannot act on: anything older than the last two epochs is already past // its proof-submission window, so we skip it and jump the cursor forward to the start of the capped range. - const maxCheckpoints = 2 * l1Constants.epochDuration; + const maxCheckpoints = (l1Constants.proofSubmissionEpochs + 1) * l1Constants.epochDuration; let from = CheckpointNumber(this.lastProcessedCheckpoint + 1); if (Number(targetCheckpoint - from) + 1 > maxCheckpoints) { const cappedFrom = CheckpointNumber(targetCheckpoint - maxCheckpoints + 1); diff --git a/yarn-project/pxe/src/block_synchronizer/block_stream_source.ts b/yarn-project/pxe/src/block_synchronizer/block_stream_source.ts index cbd6ed780cb2..8948ef621dae 100644 --- a/yarn-project/pxe/src/block_synchronizer/block_stream_source.ts +++ b/yarn-project/pxe/src/block_synchronizer/block_stream_source.ts @@ -11,10 +11,7 @@ export function blockStreamSourceFromAztecNode( node: AztecNode, ): Pick { return { - getL2Tips: async () => { - const tips = await node.getChainTips(); - return { ...tips, proposedCheckpoint: tips.checkpointed }; - }, + getL2Tips: () => node.getChainTips(), async getBlockData(query: BlockQuery): Promise { const response = await node.getBlock(query); diff --git a/yarn-project/sequencer-client/src/sequencer/automine/automine_sequencer.ts b/yarn-project/sequencer-client/src/sequencer/automine/automine_sequencer.ts index 243175965897..c50eeddc6562 100644 --- a/yarn-project/sequencer-client/src/sequencer/automine/automine_sequencer.ts +++ b/yarn-project/sequencer-client/src/sequencer/automine/automine_sequencer.ts @@ -409,7 +409,10 @@ export class AutomineSequencer { await this.deps.ethCheatCodes.setNextBlockTimestamp(slotBoundaryTs); } - const tips = await this.deps.l2BlockSource.getL2Tips(); + const [tips, proposedCheckpoint] = await Promise.all([ + this.deps.l2BlockSource.getL2Tips(), + this.deps.l2BlockSource.getProposedCheckpointData(), + ]); const syncedToBlockNumber = tips.proposed.number; // Ensure world state has processed the archiver's tip before forking. Without this, @@ -419,7 +422,8 @@ export class AutomineSequencer { await this.deps.worldState.syncImmediate(BlockNumber(syncedToBlockNumber)); const nextBlockNumber = BlockNumber(syncedToBlockNumber + 1); - const checkpointNumber = CheckpointNumber(tips.proposedCheckpoint.checkpoint.number + 1); + const parentCheckpointNumber = proposedCheckpoint?.checkpointNumber ?? tips.checkpointed.checkpoint.number; + const checkpointNumber = CheckpointNumber(parentCheckpointNumber + 1); const targetEpoch = getEpochAtSlot(SlotNumber(targetSlot), this.deps.l1Constants); this.log.verbose(`Building automine checkpoint`, { diff --git a/yarn-project/sequencer-client/src/sequencer/checkpoint_proposal_job.test.ts b/yarn-project/sequencer-client/src/sequencer/checkpoint_proposal_job.test.ts index f1da2c37dcee..89978ee6e45c 100644 --- a/yarn-project/sequencer-client/src/sequencer/checkpoint_proposal_job.test.ts +++ b/yarn-project/sequencer-client/src/sequencer/checkpoint_proposal_job.test.ts @@ -258,10 +258,6 @@ describe('CheckpointProposalJob', () => { block: { number: BlockNumber.ZERO, hash: 'block-hash' }, checkpoint: { number: CheckpointNumber.ZERO, hash: 'checkpointed-ckpt-hash' }, }, - proposedCheckpoint: { - block: { number: BlockNumber.ZERO, hash: 'block-hash' }, - checkpoint: { number: CheckpointNumber.ZERO, hash: 'proposed-ckpt-hash' }, - }, proven: { block: { number: BlockNumber.ZERO, hash: 'proven-hash' }, checkpoint: { number: CheckpointNumber.ZERO, hash: 'proven-ckpt-hash' }, @@ -1093,10 +1089,6 @@ describe('CheckpointProposalJob', () => { hash: opts.checkpointedHash ?? parentCheckpointHash, }, }, - proposedCheckpoint: { - block: { number: BlockNumber(1), hash: 'block-hash' }, - checkpoint: { number: CheckpointNumber(1), hash: parentCheckpointHash }, - }, proven: { block: { number: BlockNumber.ZERO, hash: 'proven-hash' }, checkpoint: { number: CheckpointNumber.ZERO, hash: 'proven-ckpt-hash' }, diff --git a/yarn-project/sequencer-client/src/sequencer/checkpoint_proposal_job.timing.test.ts b/yarn-project/sequencer-client/src/sequencer/checkpoint_proposal_job.timing.test.ts index 776929074d3c..2acf3e4cdf4c 100644 --- a/yarn-project/sequencer-client/src/sequencer/checkpoint_proposal_job.timing.test.ts +++ b/yarn-project/sequencer-client/src/sequencer/checkpoint_proposal_job.timing.test.ts @@ -458,10 +458,6 @@ describe('CheckpointProposalJob Timing Tests', () => { block: { number: BlockNumber.ZERO, hash: '' }, checkpoint: { number: CheckpointNumber(checkpointNumber - 1), hash: '' }, }, - proposedCheckpoint: { - block: { number: BlockNumber.ZERO, hash: '' }, - checkpoint: { number: CheckpointNumber(checkpointNumber - 1), hash: '' }, - }, proven: { block: { number: BlockNumber.ZERO, hash: '' }, checkpoint: { number: CheckpointNumber(0), hash: '' }, diff --git a/yarn-project/sequencer-client/src/sequencer/sequencer.test.ts b/yarn-project/sequencer-client/src/sequencer/sequencer.test.ts index 18e9117fa49e..816092341148 100644 --- a/yarn-project/sequencer-client/src/sequencer/sequencer.test.ts +++ b/yarn-project/sequencer-client/src/sequencer/sequencer.test.ts @@ -314,10 +314,6 @@ describe('sequencer', () => { getBlockNumber: mockFn().mockResolvedValue(lastBlockNumber), getL2Tips: mockFn().mockResolvedValue({ proposed: { number: lastBlockNumber, hash }, - proposedCheckpoint: { - block: { number: lastBlockNumber, hash }, - checkpoint: { number: CheckpointNumber.ZERO, hash: GENESIS_CHECKPOINT_HEADER_HASH.toString() }, - }, checkpointed: { block: { number: lastBlockNumber, hash }, checkpoint: { number: CheckpointNumber.ZERO, hash: GENESIS_CHECKPOINT_HEADER_HASH.toString() }, @@ -343,10 +339,6 @@ describe('sequencer', () => { getL1ToL2Messages: () => Promise.resolve(Array(NUMBER_OF_L1_L2_MESSAGES_PER_ROLLUP).fill(Fr.ZERO)), getL2Tips: mockFn().mockResolvedValue({ proposed: { number: lastBlockNumber, hash }, - proposedCheckpoint: { - block: { number: lastBlockNumber, hash }, - checkpoint: { number: CheckpointNumber.ZERO, hash: GENESIS_CHECKPOINT_HEADER_HASH.toString() }, - }, checkpointed: { block: { number: lastBlockNumber, hash }, checkpoint: { number: CheckpointNumber.ZERO, hash: GENESIS_CHECKPOINT_HEADER_HASH.toString() }, @@ -1285,9 +1277,8 @@ describe('sequencer', () => { await setupSingleTxBlock(); // Override to non-genesis state so checkSync doesn't take the genesis path. - // proposedCheckpoint is set with checkpoint number 1 > checkpointed tip 0, so hasProposedCheckpoint is true. + // The proposed checkpoint has number 1 > checkpointed tip 0, so hasProposedCheckpoint is true. const nonGenesisHash = Fr.random().toString(); - const proposedCheckpointHash = Fr.random().toString(); worldState.status.mockResolvedValue({ state: WorldStateRunningState.IDLE, syncSummary: { @@ -1300,10 +1291,6 @@ describe('sequencer', () => { } satisfies WorldStateSynchronizerStatus); const tipsWithBlock1 = { proposed: { number: BlockNumber(1), hash: nonGenesisHash }, - proposedCheckpoint: { - block: { number: BlockNumber(1), hash: nonGenesisHash }, - checkpoint: { number: CheckpointNumber(1), hash: proposedCheckpointHash }, - }, checkpointed: { block: { number: BlockNumber(1), hash: nonGenesisHash }, checkpoint: { number: CheckpointNumber.ZERO, hash: GENESIS_CHECKPOINT_HEADER_HASH.toString() }, @@ -1338,7 +1325,7 @@ describe('sequencer', () => { blockCount: 1, totalManaUsed: 0n, feeAssetPriceModifier: 0n, - } satisfies ProposedCheckpointData); + }); await sequencer.work(); @@ -1354,7 +1341,6 @@ describe('sequencer', () => { // Confirmed checkpoint is 1, pending is 2, proposed tip is in checkpoint 3. // So sequencer would try to build checkpoint 4, which exceeds the 1-deep pipeline limit. const nonGenesisHash = Fr.random().toString(); - const proposedCheckpointHash = Fr.random().toString(); const checkpointedHash = Fr.random().toString(); worldState.status.mockResolvedValue({ state: WorldStateRunningState.IDLE, @@ -1368,10 +1354,6 @@ describe('sequencer', () => { } satisfies WorldStateSynchronizerStatus); const tips = { proposed: { number: BlockNumber(3), hash: nonGenesisHash }, - proposedCheckpoint: { - block: { number: BlockNumber(2), hash: nonGenesisHash }, - checkpoint: { number: CheckpointNumber(2), hash: proposedCheckpointHash }, - }, checkpointed: { block: { number: BlockNumber(1), hash: nonGenesisHash }, checkpoint: { number: CheckpointNumber(1), hash: checkpointedHash }, @@ -1397,9 +1379,7 @@ describe('sequencer', () => { checkpointNumber: CheckpointNumber(3), indexWithinCheckpoint: IndexWithinCheckpoint(0), } satisfies BlockData); - l2BlockSource.getProposedCheckpointData.mockResolvedValue({ - checkpointNumber: CheckpointNumber(2), - } as any); + l2BlockSource.getProposedCheckpointData.mockResolvedValue({ checkpointNumber: CheckpointNumber(2) } as any); await sequencer.work(); @@ -1424,7 +1404,6 @@ describe('sequencer', () => { // Set up a pipelined parent (pending override = parentCheckpointNumber = 1). const nonGenesisHash = Fr.random().toString(); - const proposedCheckpointHash = Fr.random().toString(); worldState.status.mockResolvedValue({ state: WorldStateRunningState.IDLE, syncSummary: { @@ -1437,10 +1416,6 @@ describe('sequencer', () => { } satisfies WorldStateSynchronizerStatus); const tipsWithBlock1 = { proposed: { number: BlockNumber(1), hash: nonGenesisHash }, - proposedCheckpoint: { - block: { number: BlockNumber(1), hash: nonGenesisHash }, - checkpoint: { number: CheckpointNumber(1), hash: proposedCheckpointHash }, - }, checkpointed: { block: { number: BlockNumber(1), hash: nonGenesisHash }, checkpoint: { number: CheckpointNumber.ZERO, hash: GENESIS_CHECKPOINT_HEADER_HASH.toString() }, @@ -1475,7 +1450,7 @@ describe('sequencer', () => { blockCount: 1, totalManaUsed: 0n, feeAssetPriceModifier: 0n, - } satisfies ProposedCheckpointData); + }); await sequencer.work(); @@ -1510,18 +1485,17 @@ describe('sequencer', () => { describe('checkSync orphan-block guard', () => { // Mocks all sync sources so checkSync passes its earlier equality checks and reaches the orphan // guard, with the world-state tip at `blockNumber` (in `blockCheckpointNumber`) while the - // checkpointed and proposed-checkpoint tips sit at the given checkpoint numbers. + // checkpointed tip sits at `checkpointedCheckpointNumber`. The leading proposed checkpoint (if any) + // is supplied via `getProposedCheckpointData`. const setupSyncedToBlock = (opts: { blockNumber: BlockNumber; blockSlot: SlotNumber; blockCheckpointNumber: CheckpointNumber; checkpointedCheckpointNumber: CheckpointNumber; - proposedCheckpointTipNumber: CheckpointNumber; - proposedCheckpointData: ProposedCheckpointData | undefined; + proposedCheckpoint: ProposedCheckpointData | undefined; }) => { const hash = Fr.random().toString(); const checkpointHash = Fr.random().toString(); - const proposedCheckpointHash = Fr.random().toString(); worldState.status.mockResolvedValue({ state: WorldStateRunningState.IDLE, syncSummary: { @@ -1534,10 +1508,6 @@ describe('sequencer', () => { } satisfies WorldStateSynchronizerStatus); const tips = { proposed: { number: opts.blockNumber, hash }, - proposedCheckpoint: { - block: { number: opts.blockNumber, hash }, - checkpoint: { number: opts.proposedCheckpointTipNumber, hash: proposedCheckpointHash }, - }, checkpointed: { block: { number: opts.blockNumber, hash }, checkpoint: { number: opts.checkpointedCheckpointNumber, hash: checkpointHash }, @@ -1563,20 +1533,19 @@ describe('sequencer', () => { checkpointNumber: opts.blockCheckpointNumber, indexWithinCheckpoint: IndexWithinCheckpoint(0), } satisfies BlockData); - l2BlockSource.getProposedCheckpointData.mockResolvedValue(opts.proposedCheckpointData); + l2BlockSource.getProposedCheckpointData.mockResolvedValue(opts.proposedCheckpoint); }; it('returns undefined and logs debug while waiting for a matching proposed checkpoint', async () => { - // Local tip is a block at checkpoint 3, but the checkpointed and proposed-checkpoint tips are - // still at checkpoint 2 and no proposed checkpoint 3 exists: an orphan block-only tip whose - // enclosing checkpoint has not materialized into the archiver. + // Local tip is a block at checkpoint 3, but the checkpointed tip is still at checkpoint 2 and no + // proposed checkpoint 3 exists: an orphan block-only tip whose enclosing checkpoint has not + // materialized into the archiver. setupSyncedToBlock({ blockNumber: BlockNumber(3), blockSlot: SlotNumber(3), blockCheckpointNumber: CheckpointNumber(3), checkpointedCheckpointNumber: CheckpointNumber(2), - proposedCheckpointTipNumber: CheckpointNumber(2), - proposedCheckpointData: undefined, + proposedCheckpoint: undefined, }); const warnSpy = jest.spyOn(sequencer.getLogger(), 'warn'); const debugSpy = jest.spyOn(sequencer.getLogger(), 'debug'); @@ -1590,8 +1559,7 @@ describe('sequencer', () => { expect.objectContaining({ blockCheckpointNumber: CheckpointNumber(3), checkpointedCheckpointNumber: CheckpointNumber(2), - proposedCheckpointTipNumber: CheckpointNumber(2), - proposedCheckpointDataNumber: undefined, + proposedCheckpointTipNumber: undefined, }), ); }); @@ -1602,8 +1570,7 @@ describe('sequencer', () => { blockSlot: SlotNumber(3), blockCheckpointNumber: CheckpointNumber(3), checkpointedCheckpointNumber: CheckpointNumber(2), - proposedCheckpointTipNumber: CheckpointNumber(3), - proposedCheckpointData: { + proposedCheckpoint: { checkpointNumber: CheckpointNumber(3), header: CheckpointHeader.empty(), archive: AppendOnlyTreeSnapshot.empty(), @@ -1612,7 +1579,7 @@ describe('sequencer', () => { blockCount: 1, totalManaUsed: 0n, feeAssetPriceModifier: 0n, - } satisfies ProposedCheckpointData, + }, }); const result = await sequencer.checkSyncForTest({ ts: 1000n, slot: SlotNumber(2) }); diff --git a/yarn-project/sequencer-client/src/sequencer/sequencer.ts b/yarn-project/sequencer-client/src/sequencer/sequencer.ts index e25fb8f11834..e84f6dd0506a 100644 --- a/yarn-project/sequencer-client/src/sequencer/sequencer.ts +++ b/yarn-project/sequencer-client/src/sequencer/sequencer.ts @@ -798,9 +798,7 @@ export class Sequencer extends (EventEmitter as new () => TypedEventEmitter ({ proposed: t.proposed, checkpointed: t.checkpointed, proposedCheckpoint: t.proposedCheckpoint })), + this.l2BlockSource.getL2Tips().then(t => ({ proposed: t.proposed, checkpointed: t.checkpointed })), this.p2pClient.getStatus().then(p2p => p2p.syncedToL2Block), this.l1ToL2MessageSource.getL2Tips().then(t => ({ proposed: t.proposed, checkpointed: t.checkpointed })), this.l2BlockSource.getPendingChainValidationStatus(), @@ -845,16 +843,17 @@ export class Sequencer extends (EventEmitter as new () => TypedEventEmitter l2Tips.checkpointed.checkpoint.number && - (l2Tips.proposedCheckpoint.checkpoint.number !== blockData.checkpointNumber || - proposedCheckpointData?.checkpointNumber !== blockData.checkpointNumber) + proposedCheckpointData?.checkpointNumber !== blockData.checkpointNumber ) { const logCtx = { blockCheckpointNumber: blockData.checkpointNumber, checkpointedCheckpointNumber: l2Tips.checkpointed.checkpoint.number, - proposedCheckpointTipNumber: l2Tips.proposedCheckpoint.checkpoint.number, - proposedCheckpointDataNumber: proposedCheckpointData?.checkpointNumber, + proposedCheckpointTipNumber: proposedCheckpointData?.checkpointNumber, blockNumber: blockData.header.getBlockNumber(), blockSlot: blockData.header.getSlot(), syncedL2Slot, @@ -865,27 +864,7 @@ export class Sequencer extends (EventEmitter as new () => TypedEventEmitter l2Tips.checkpointed.checkpoint.number; - - // The l2Tips and proposedCheckpointData reads above come from independent archiver snapshots - // (a JS-side tips cache vs. a direct store read on `#proposedCheckpoints`). A concurrent archiver - // write that mutates both can be observed split, leaving us with `hasProposedCheckpoint=true` but - // no proposedCheckpointData (or one whose number doesn't match the tip). Refuse to proceed in that - // window — the next checkSync tick will see a coherent snapshot. - if ( - hasProposedCheckpoint && - (!proposedCheckpointData || - proposedCheckpointData.checkpointNumber !== l2Tips.proposedCheckpoint.checkpoint.number) - ) { - this.log.warn(`Sequencer sync check failed: inconsistent proposed-checkpoint state`, { - proposedCheckpointTipNumber: l2Tips.proposedCheckpoint.checkpoint.number, - checkpointedTipNumber: l2Tips.checkpointed.checkpoint.number, - proposedCheckpointDataNumber: proposedCheckpointData?.checkpointNumber, - syncedL2Slot, - ...args, - }); - return undefined; - } + const hasProposedCheckpoint = proposedCheckpointData !== undefined; // Check that the proposed checkpoint is indeed the parent of the checkpoint we'll be building // The checkpoint number to build is derived as blockData.checkpointNumber + 1 diff --git a/yarn-project/stdlib/src/block/l2_block_source.ts b/yarn-project/stdlib/src/block/l2_block_source.ts index fe3a6175cef4..34089220669c 100644 --- a/yarn-project/stdlib/src/block/l2_block_source.ts +++ b/yarn-project/stdlib/src/block/l2_block_source.ts @@ -241,7 +241,10 @@ export interface L2BlockSource { /** * Looks up a proposed (archiver-internal, not-yet-L1-confirmed) checkpoint. - * Returns the latest proposed entry when called with no args or `{ tag: 'proposed' }`. + * Returns the latest proposed entry when called with no args or `{ tag: 'proposed' }`; since a + * proposed entry can only be stored with a checkpoint number beyond the confirmed frontier (and is + * deleted on confirmation), the latest entry is always the leading one. Callers derive the proposed + * tip from the payload (checkpoint number, and last block `startBlock + blockCount - 1`). * With `{ number }` or `{ slot }`, returns the matching entry or undefined. * Never falls back to confirmed checkpoints. */ @@ -327,36 +330,29 @@ export interface L2BlockSourceEventEmitter extends L2BlockSource { } /** - * Identifier for L2 block tags. Internal counterpart to {@link BlockTag} that exposes - * the additional `proposedCheckpoint` value (used for the optimistic chain tip on the - * archiver side) and omits `latest` (which is an alias for `proposed` accepted only at - * the public RPC surface). + * Identifier for L2 block tags. Internal counterpart to {@link BlockTag} that omits `latest` + * (which is an alias for `proposed` accepted only at the public RPC surface). * * - proposed: Latest block proposed on L2. - * - proposedCheckpoint: Latest block in the most recent proposed checkpoint (archiver-internal). * - checkpointed: Latest block whose enclosing checkpoint has been published on L1. * - proven: Latest block whose enclosing checkpoint has been proven on L1. * - finalized: Latest block whose proving L1 transaction has reached L1 finality. - * - * TODO(palla): Remove `proposedCheckpoint` and unify with `proposed`. */ -export type L2BlockTag = 'proposed' | 'proposedCheckpoint' | 'checkpointed' | 'proven' | 'finalized'; +export type L2BlockTag = 'proposed' | 'checkpointed' | 'proven' | 'finalized'; /** Tips of the L2 chain. */ export type L2Tips = { proposed: L2BlockId; checkpointed: L2TipId; - proposedCheckpoint: L2TipId; proven: L2TipId; finalized: L2TipId; }; /** - * Tips of the L2 chain as tracked by a local provider (world-state, l2-tips-store). Omits - * `proposedCheckpoint`, which is degenerate in local stores (always equal to `checkpointed`) and - * is only meaningful on the archiver side via {@link L2BlockSource}. + * Tips of the L2 chain as tracked by a local provider (world-state, l2-tips-store). Identical to + * {@link L2Tips}; the alias is retained for call sites that document a local-only provenance. */ -export type LocalL2Tips = Omit; +export type LocalL2Tips = L2Tips; export const GENESIS_CHECKPOINT_HEADER_HASH = CheckpointHeader.empty().hash(); @@ -398,7 +394,6 @@ const L2TipIdSchema = z.object({ export const L2TipsSchema = z.object({ proposed: L2BlockIdSchema, checkpointed: L2TipIdSchema, - proposedCheckpoint: L2TipIdSchema, proven: L2TipIdSchema, finalized: L2TipIdSchema, }); diff --git a/yarn-project/stdlib/src/block/l2_block_stream/l2_block_stream.test.ts b/yarn-project/stdlib/src/block/l2_block_stream/l2_block_stream.test.ts index a123e9160682..28f7cb9d76cf 100644 --- a/yarn-project/stdlib/src/block/l2_block_stream/l2_block_stream.test.ts +++ b/yarn-project/stdlib/src/block/l2_block_stream/l2_block_stream.test.ts @@ -61,14 +61,7 @@ describe('L2BlockStream', () => { }); /** Sets the remote tips. All tips default to 0 except latest. */ - const setRemoteTips = ( - latest_: number, - checkpointed_?: number, - proven?: number, - finalized?: number, - proposedCheckpoint_?: number, - ) => { - proposedCheckpoint_ = proposedCheckpoint_ ?? 0; + const setRemoteTips = (latest_: number, checkpointed_?: number, proven?: number, finalized?: number) => { checkpointed_ = checkpointed_ ?? 0; proven = proven ?? 0; finalized = finalized ?? 0; @@ -77,7 +70,6 @@ describe('L2BlockStream', () => { blockSource.getL2Tips.mockResolvedValue({ proposed: { number: BlockNumber(latest), hash: makeHash(latest) }, checkpointed: makeTipId(checkpointed_), - proposedCheckpoint: makeTipId(proposedCheckpoint_), proven: makeTipId(proven), finalized: makeTipId(finalized), }); diff --git a/yarn-project/stdlib/src/interfaces/archiver.test.ts b/yarn-project/stdlib/src/interfaces/archiver.test.ts index db0f41c3468d..c1e2631cfad6 100644 --- a/yarn-project/stdlib/src/interfaces/archiver.test.ts +++ b/yarn-project/stdlib/src/interfaces/archiver.test.ts @@ -153,7 +153,6 @@ describe('ArchiverApiSchema', () => { expect(result).toEqual({ proposed: { number: 1, hash: `0x01` }, checkpointed: expectedTipId, - proposedCheckpoint: expectedTipId, proven: expectedTipId, finalized: expectedTipId, }); @@ -477,7 +476,6 @@ class MockArchiver implements ArchiverApi { return Promise.resolve({ proposed: { number: BlockNumber(1), hash: `0x01` }, checkpointed: tipId, - proposedCheckpoint: tipId, proven: tipId, finalized: tipId, }); diff --git a/yarn-project/stdlib/src/interfaces/aztec-node.test.ts b/yarn-project/stdlib/src/interfaces/aztec-node.test.ts index bd404091f483..fb23d9d56458 100644 --- a/yarn-project/stdlib/src/interfaces/aztec-node.test.ts +++ b/yarn-project/stdlib/src/interfaces/aztec-node.test.ts @@ -22,7 +22,7 @@ import { AztecAddress } from '../aztec-address/index.js'; import type { BlockData } from '../block/block_data.js'; import type { DataInBlock } from '../block/in_block.js'; import { BlockHash, type BlockParameter } from '../block/index.js'; -import type { CheckpointsQuery } from '../block/l2_block_source.js'; +import type { CheckpointsQuery, L2BlockTag, L2Tips } from '../block/l2_block_source.js'; import type { CheckpointData } from '../checkpoint/checkpoint_data.js'; import { type ContractClassPublic, @@ -64,7 +64,7 @@ import type { AllowedElement } from './allowed_element.js'; import { MAX_RPC_LEN } from './api_limit.js'; import { type AztecNode, AztecNodeApiSchema, type GetTxByHashOptions } from './aztec-node.js'; import type { BlockIncludeOptions, BlockResponse, BlocksIncludeOptions } from './block_response.js'; -import type { ChainTip, ChainTips } from './chain_tips.js'; +import type { CheckpointTag } from './chain_tips.js'; import type { CheckpointParameter } from './checkpoint_parameter.js'; import type { CheckpointIncludeOptions, CheckpointResponse } from './checkpoint_response.js'; import type { SequencerConfig } from './configs.js'; @@ -596,7 +596,7 @@ class MockAztecNode implements AztecNode { }); } - getChainTips(): Promise { + getChainTips(): Promise { const tipId = { block: { number: BlockNumber(1), hash: `0x01` }, checkpoint: { number: CheckpointNumber(1), hash: `0x01` }, @@ -765,10 +765,10 @@ class MockAztecNode implements AztecNode { getMaxPriorityFees(): Promise { return Promise.resolve(GasFees.empty()); } - getBlockNumber(_tip?: ChainTip): Promise { + getBlockNumber(_tip?: L2BlockTag): Promise { return Promise.resolve(BlockNumber(1)); } - getCheckpointNumber(_tip?: ChainTip): Promise { + getCheckpointNumber(_tip?: CheckpointTag): Promise { return Promise.resolve(CheckpointNumber(1)); } isReady(): Promise { diff --git a/yarn-project/stdlib/src/interfaces/aztec-node.ts b/yarn-project/stdlib/src/interfaces/aztec-node.ts index 99ee4d88a88c..62baf29ecb06 100644 --- a/yarn-project/stdlib/src/interfaces/aztec-node.ts +++ b/yarn-project/stdlib/src/interfaces/aztec-node.ts @@ -23,9 +23,15 @@ import { z } from 'zod'; import type { AztecAddress } from '../aztec-address/index.js'; import { type BlockData, BlockDataSchema } from '../block/block_data.js'; import { BlockHash } from '../block/block_hash.js'; -import { type BlockParameter, BlockParameterSchema } from '../block/block_parameter.js'; +import { type BlockParameter, BlockParameterSchema, BlockTagWithoutLatestSchema } from '../block/block_parameter.js'; import { type DataInBlock, dataInBlockSchemaFor } from '../block/in_block.js'; -import { type CheckpointsQuery, CheckpointsQuerySchema } from '../block/l2_block_source.js'; +import { + type CheckpointsQuery, + CheckpointsQuerySchema, + type L2BlockTag, + type L2Tips, + L2TipsSchema, +} from '../block/l2_block_source.js'; import { type CheckpointData, CheckpointDataSchema } from '../checkpoint/checkpoint_data.js'; import { type ContractClassPublic, @@ -80,7 +86,7 @@ import { type BlocksIncludeOptions, BlocksIncludeOptionsSchema, } from './block_response.js'; -import { type ChainTip, ChainTipSchema, type ChainTips, ChainTipsSchema } from './chain_tips.js'; +import { type CheckpointTag, CheckpointTagSchema } from './chain_tips.js'; import { type CheckpointParameter, CheckpointParameterSchema } from './checkpoint_parameter.js'; import { type CheckpointIncludeOptions, @@ -232,23 +238,20 @@ export interface AztecNode { ): Promise; /** - * Returns the block number at a given chain tip, or the latest proposed block number when + * Returns the block number at a given block tag, or the latest proposed block number when * `tip` is omitted. */ - getBlockNumber(tip?: ChainTip): Promise; + getBlockNumber(tip?: L2BlockTag): Promise; /** - * Returns the checkpoint number at a given chain tip, or the latest checkpoint number when - * `tip` is omitted. - * - * @remarks **Semantic foot-gun**: block-side `'proposed'` means "latest proposed block" (chain - * head), but checkpoint-side `'proposed'` means "latest confirmed checkpoint" — pre-L1-confirm - * checkpoints are not exposed over RPC. `'checkpointed'` on the checkpoint side is equivalent. + * Returns the checkpoint number at a given checkpoint tag, or the latest checkpointed number when + * `tip` is omitted. The proposed-but-unconfirmed checkpoint frontier is archiver-internal and not + * exposed over RPC, so `'proposed'` is not a valid checkpoint tag (see {@link CheckpointTag}). */ - getCheckpointNumber(tip?: ChainTip): Promise; + getCheckpointNumber(tip?: CheckpointTag): Promise; /** Returns the tips of the L2 chain. */ - getChainTips(): Promise; + getChainTips(): Promise; /** Returns the rollup constants for the current chain. */ getL1Constants(): Promise; @@ -622,11 +625,11 @@ export const AztecNodeApiSchema: ApiSchemaFor = { output: L2ToL1MembershipWitnessSchema.optional(), }), - getBlockNumber: z.function({ input: z.tuple([optional(ChainTipSchema)]), output: BlockNumberSchema }), + getBlockNumber: z.function({ input: z.tuple([optional(BlockTagWithoutLatestSchema)]), output: BlockNumberSchema }), - getCheckpointNumber: z.function({ input: z.tuple([optional(ChainTipSchema)]), output: CheckpointNumberSchema }), + getCheckpointNumber: z.function({ input: z.tuple([optional(CheckpointTagSchema)]), output: CheckpointNumberSchema }), - getChainTips: z.function({ input: z.tuple([]), output: ChainTipsSchema }), + getChainTips: z.function({ input: z.tuple([]), output: L2TipsSchema }), getL1Constants: z.function({ input: z.tuple([]), output: L1RollupConstantsSchema }), diff --git a/yarn-project/stdlib/src/interfaces/chain_tips.ts b/yarn-project/stdlib/src/interfaces/chain_tips.ts index fde42b09a136..45ccb6f313c7 100644 --- a/yarn-project/stdlib/src/interfaces/chain_tips.ts +++ b/yarn-project/stdlib/src/interfaces/chain_tips.ts @@ -1,24 +1,16 @@ import { z } from 'zod'; -import { type L2BlockTag, type L2Tips, L2TipsSchema } from '../block/l2_block_source.js'; - /** - * Public chain-tip selectors usable in RPC requests. - * Omits internal-only tags (e.g. `proposedCheckpoint`) from {@link L2BlockTag}. + * Public checkpoint-tip selectors usable in RPC requests. + * + * `'proposed'` is intentionally excluded: the proposed-but-unconfirmed checkpoint frontier is an + * archiver-internal pipelining concept, not part of the public chain-tip surface. Select the + * proposed *block* tip with a block tag (`L2BlockTag`) instead. */ -export type ChainTip = Exclude; +export type CheckpointTag = 'checkpointed' | 'proven' | 'finalized'; -export const ChainTipSchema = z.union([ - z.literal('proposed'), +export const CheckpointTagSchema = z.union([ z.literal('checkpointed'), z.literal('proven'), z.literal('finalized'), -]) satisfies z.ZodType; - -/** - * Tips of the L2 chain. - * Omits the sequencer-internal `proposedCheckpoint` from the public RPC surface. - */ -export type ChainTips = Omit; - -export const ChainTipsSchema = L2TipsSchema.omit({ proposedCheckpoint: true }); +]) satisfies z.ZodType; diff --git a/yarn-project/stdlib/src/interfaces/checkpoint_parameter.ts b/yarn-project/stdlib/src/interfaces/checkpoint_parameter.ts index 0c50a436d753..c7bd38738818 100644 --- a/yarn-project/stdlib/src/interfaces/checkpoint_parameter.ts +++ b/yarn-project/stdlib/src/interfaces/checkpoint_parameter.ts @@ -2,18 +2,18 @@ import { CheckpointNumberSchema, SlotNumberSchema } from '@aztec/foundation/bran import { z } from 'zod'; -import { ChainTipSchema } from './chain_tips.js'; +import { CheckpointTagSchema } from './chain_tips.js'; /** * Selector for a checkpoint in RPC calls. * * Accepts a numeric checkpoint number (or `{ number }`), a slot number (`{ slot }`), - * or a chain-tip name (e.g. `'proposed'`, `'proven'`). + * or a checkpoint-tip name (e.g. `'checkpointed'`, `'proven'`, `'finalized'`). */ export const CheckpointParameterSchema = z.union([ z.object({ number: CheckpointNumberSchema }).strict(), z.object({ slot: SlotNumberSchema }).strict(), - ChainTipSchema, + CheckpointTagSchema, CheckpointNumberSchema, ]); diff --git a/yarn-project/stdlib/src/interfaces/prover-node.test.ts b/yarn-project/stdlib/src/interfaces/prover-node.test.ts index 770412e760f9..a6a832ed72a5 100644 --- a/yarn-project/stdlib/src/interfaces/prover-node.test.ts +++ b/yarn-project/stdlib/src/interfaces/prover-node.test.ts @@ -45,7 +45,6 @@ describe('ProvingNodeApiSchema', () => { expect(result).toEqual({ proposed: { number: 1, hash: `0x01` }, checkpointed: expectedTipId, - proposedCheckpoint: expectedTipId, proven: expectedTipId, finalized: expectedTipId, }); @@ -76,7 +75,6 @@ class MockProverNode implements ProverNodeApi { return Promise.resolve({ proposed: { number: BlockNumber(1), hash: `0x01` }, checkpointed: tipId, - proposedCheckpoint: tipId, proven: tipId, finalized: tipId, }); diff --git a/yarn-project/stdlib/src/tests/factories.ts b/yarn-project/stdlib/src/tests/factories.ts index 0198fabd4bb4..298efb24082a 100644 --- a/yarn-project/stdlib/src/tests/factories.ts +++ b/yarn-project/stdlib/src/tests/factories.ts @@ -1745,10 +1745,6 @@ export function makeL2Tips( block: { number: bn, hash }, checkpoint: { number: cpn, hash: cph }, }, - proposedCheckpoint: { - block: { number: bn, hash }, - checkpoint: { number: cpn, hash: cph }, - }, proven: { block: { number: bn, hash }, checkpoint: { number: cpn, hash: cph }, diff --git a/yarn-project/txe/src/state_machine/archiver.ts b/yarn-project/txe/src/state_machine/archiver.ts index ab3b92562876..c91789f7d3bf 100644 --- a/yarn-project/txe/src/state_machine/archiver.ts +++ b/yarn-project/txe/src/state_machine/archiver.ts @@ -92,7 +92,6 @@ export class TXEArchiver extends ArchiverDataSourceBase { proven: tipId, finalized: tipId, checkpointed: tipId, - proposedCheckpoint: tipId, }; } From 49c510a2e935f6de613e46e34deeda05a3a546cc Mon Sep 17 00:00:00 2001 From: Santiago Palladino Date: Fri, 12 Jun 2026 10:10:06 -0300 Subject: [PATCH 03/14] fix(sequencer): wait for previous L1 block before publishing (#24037) Wait after the scheduled send time until the previous L1 block is observed before submitting. Alternative to #24035. Fixes A-1207 --- .../e2e_l1_publisher/e2e_l1_publisher.test.ts | 2 + .../end-to-end/src/e2e_synching.test.ts | 2 + yarn-project/foundation/src/config/env_var.ts | 2 + .../sequencer-client/src/publisher/config.ts | 23 +++- .../src/publisher/sequencer-publisher.test.ts | 110 +++++++++++++++++- .../src/publisher/sequencer-publisher.ts | 87 ++++++++++---- .../src/sequencer/checkpoint_proposal_job.ts | 4 +- 7 files changed, 202 insertions(+), 28 deletions(-) diff --git a/yarn-project/end-to-end/src/e2e_l1_publisher/e2e_l1_publisher.test.ts b/yarn-project/end-to-end/src/e2e_l1_publisher/e2e_l1_publisher.test.ts index b6a7da0e593c..e98867bc7f41 100644 --- a/yarn-project/end-to-end/src/e2e_l1_publisher/e2e_l1_publisher.test.ts +++ b/yarn-project/end-to-end/src/e2e_l1_publisher/e2e_l1_publisher.test.ts @@ -355,6 +355,8 @@ describe('L1Publisher integration', () => { l1ChainId: chainId, ethereumSlotDuration: config.ethereumSlotDuration, aztecSlotDuration: config.aztecSlotDuration, + sequencerPublisherPreviousL1BlockWaitTimeoutMs: config.sequencerPublisherPreviousL1BlockWaitTimeoutMs, + sequencerPublisherPreviousL1BlockWaitPollIntervalMs: config.sequencerPublisherPreviousL1BlockWaitPollIntervalMs, }, { blobClient, diff --git a/yarn-project/end-to-end/src/e2e_synching.test.ts b/yarn-project/end-to-end/src/e2e_synching.test.ts index 96047cc18ead..7dd54f7d3675 100644 --- a/yarn-project/end-to-end/src/e2e_synching.test.ts +++ b/yarn-project/end-to-end/src/e2e_synching.test.ts @@ -451,6 +451,8 @@ describe('e2e_synching', () => { l1ChainId: 31337, ethereumSlotDuration: ETHEREUM_SLOT_DURATION, aztecSlotDuration: AZTEC_SLOT_DURATION, + sequencerPublisherPreviousL1BlockWaitTimeoutMs: config.sequencerPublisherPreviousL1BlockWaitTimeoutMs, + sequencerPublisherPreviousL1BlockWaitPollIntervalMs: config.sequencerPublisherPreviousL1BlockWaitPollIntervalMs, }, { blobClient, diff --git a/yarn-project/foundation/src/config/env_var.ts b/yarn-project/foundation/src/config/env_var.ts index 3080a0fb9824..9d2fa2a35412 100644 --- a/yarn-project/foundation/src/config/env_var.ts +++ b/yarn-project/foundation/src/config/env_var.ts @@ -234,6 +234,8 @@ export type EnvVar = | 'SEQ_PUBLISHER_ADDRESSES' | 'SEQ_PUBLISHER_ALLOW_INVALID_STATES' | 'SEQ_PUBLISHER_FORWARDER_ADDRESS' + | 'SEQ_PUBLISHER_PREVIOUS_L1_BLOCK_WAIT_TIMEOUT_MS' + | 'SEQ_PUBLISHER_PREVIOUS_L1_BLOCK_WAIT_POLL_INTERVAL_MS' | 'PUBLISHER_FUNDING_THRESHOLD' | 'PUBLISHER_FUNDING_AMOUNT' | 'SEQ_POLLING_INTERVAL_MS' diff --git a/yarn-project/sequencer-client/src/publisher/config.ts b/yarn-project/sequencer-client/src/publisher/config.ts index fe934a2ede88..83eac988899e 100644 --- a/yarn-project/sequencer-client/src/publisher/config.ts +++ b/yarn-project/sequencer-client/src/publisher/config.ts @@ -1,7 +1,12 @@ import { type BlobClientConfig, blobClientConfigMapping } from '@aztec/blob-client/client/config'; import { type L1ReaderConfig, l1ReaderConfigMappings } from '@aztec/ethereum/l1-reader'; import { type L1TxUtilsConfig, l1TxUtilsConfigMappings } from '@aztec/ethereum/l1-tx-utils/config'; -import { type ConfigMappingsType, SecretValue, booleanConfigHelper } from '@aztec/foundation/config'; +import { + type ConfigMappingsType, + SecretValue, + booleanConfigHelper, + numberConfigHelper, +} from '@aztec/foundation/config'; import { EthAddress } from '@aztec/foundation/eth-address'; import { parseEther } from 'viem'; @@ -90,6 +95,10 @@ export type SequencerPublisherConfig = L1TxUtilsConfig & fishermanMode?: boolean; sequencerPublisherAllowInvalidStates?: boolean; sequencerPublisherForwarderAddress?: EthAddress; + /** How long to wait for the previous L1 block before sending scheduled publisher txs anyway. */ + sequencerPublisherPreviousL1BlockWaitTimeoutMs: number; + /** Poll interval while waiting for the previous L1 block before scheduled publisher txs. */ + sequencerPublisherPreviousL1BlockWaitPollIntervalMs: number; /** Store for failed L1 transaction inputs (test networks only). Format: gs://bucket/path */ l1TxFailedStore?: string; /** Min ETH balance below which a publisher gets funded. Undefined = funding disabled. */ @@ -170,6 +179,18 @@ export const sequencerPublisherConfigMappings: ConfigMappingsType EthAddress.fromString(val), }, + sequencerPublisherPreviousL1BlockWaitTimeoutMs: { + env: `SEQ_PUBLISHER_PREVIOUS_L1_BLOCK_WAIT_TIMEOUT_MS`, + description: + 'How long to wait for the previous L1 block before sending scheduled publisher txs anyway, in milliseconds.', + ...numberConfigHelper(8_000), + }, + sequencerPublisherPreviousL1BlockWaitPollIntervalMs: { + env: `SEQ_PUBLISHER_PREVIOUS_L1_BLOCK_WAIT_POLL_INTERVAL_MS`, + description: + 'Poll interval while waiting for the previous L1 block before scheduled publisher txs, in milliseconds.', + ...numberConfigHelper(500), + }, l1TxFailedStore: { env: 'L1_TX_FAILED_STORE', description: 'Store for failed L1 transaction inputs (test networks only). Format: gs://bucket/path', diff --git a/yarn-project/sequencer-client/src/publisher/sequencer-publisher.test.ts b/yarn-project/sequencer-client/src/publisher/sequencer-publisher.test.ts index 519fbdb6dfed..b8bc704fe5d3 100644 --- a/yarn-project/sequencer-client/src/publisher/sequencer-publisher.test.ts +++ b/yarn-project/sequencer-client/src/publisher/sequencer-publisher.test.ts @@ -41,7 +41,7 @@ import { } from 'viem'; import { privateKeyToAccount } from 'viem/accounts'; -import type { PublisherConfig, TxSenderConfig } from './config.js'; +import type { PublisherConfig, SequencerPublisherConfig, TxSenderConfig } from './config.js'; import type { SequencerPublisherMetrics } from './sequencer-publisher-metrics.js'; import { type Action, SequencerPublisher, compareActions } from './sequencer-publisher.js'; @@ -72,6 +72,7 @@ describe('SequencerPublisher', () => { let l1TxUtils: MockProxy; let l1Metrics: MockProxy; let forwardSpy: jest.SpiedFunction; + let dateProvider: TestDateProvider; let proposeTxHash: `0x${string}`; let proposeTxReceipt: GetTransactionReceiptReturnType; @@ -121,9 +122,12 @@ describe('SequencerPublisher', () => { l1RpcUrls: [`http://127.0.0.1:8545`], l1ChainId: 1, aztecSlotDuration: 36, + sequencerPublisherPreviousL1BlockWaitTimeoutMs: 8_000, + sequencerPublisherPreviousL1BlockWaitPollIntervalMs: 500, ...defaultL1TxUtilsConfig, } as unknown as TxSenderConfig & PublisherConfig & + SequencerPublisherConfig & Pick & L1TxUtilsConfig; @@ -149,6 +153,8 @@ describe('SequencerPublisher', () => { isEscapeHatchOpen: false, }); + dateProvider = new TestDateProvider(); + publisher = new SequencerPublisher(config, { blobClient, rollupContract: rollup, @@ -156,7 +162,7 @@ describe('SequencerPublisher', () => { epochCache, slashingProposerContract, governanceProposerContract, - dateProvider: new TestDateProvider(), + dateProvider, metrics: l1Metrics, lastActions: {}, }); @@ -346,7 +352,13 @@ describe('SequencerPublisher', () => { }); rotatingPublisher = new SequencerPublisher( - { ethereumSlotDuration: 12, aztecSlotDuration: 36, l1ChainId: 1 } as any, + { + ethereumSlotDuration: 12, + aztecSlotDuration: 36, + l1ChainId: 1, + sequencerPublisherPreviousL1BlockWaitTimeoutMs: 8_000, + sequencerPublisherPreviousL1BlockWaitPollIntervalMs: 500, + } as any, { blobClient, rollupContract: rollup, @@ -742,6 +754,98 @@ describe('SequencerPublisher', () => { } }); + it('waits for the previous L1 block before sending scheduled requests', async () => { + jest.useFakeTimers({ doNotFake: ['nextTick', 'queueMicrotask', 'setImmediate'] }); + try { + // EmptyL1RollupConstants has l1GenesisTime=0 and slotDuration=1s. With the publisher's + // ethereumSlotDuration=12s, slot 112 has submitAfter=100s. + jest.setSystemTime(new Date(100_000)); + const targetSlot = SlotNumber(112); + const sendSpy = jest.spyOn(publisher, 'sendRequests').mockResolvedValue(undefined); + l1TxUtils.getBlock + .mockResolvedValueOnce({ timestamp: 99n } as any) + .mockResolvedValueOnce({ timestamp: 100n } as any); + + const resultPromise = publisher.sendRequestsAt(targetSlot); + await jest.advanceTimersByTimeAsync(499); + + expect(sendSpy).not.toHaveBeenCalled(); + + await jest.advanceTimersByTimeAsync(1); + await jest.advanceTimersByTimeAsync(500); + await expect(resultPromise).resolves.toBeUndefined(); + + expect(sendSpy).toHaveBeenCalledWith(targetSlot); + expect(l1TxUtils.getBlock).toHaveBeenCalledTimes(2); + } finally { + jest.useRealTimers(); + } + }); + + it('sends scheduled requests if the previous L1 block wait times out', async () => { + jest.useFakeTimers({ doNotFake: ['nextTick', 'queueMicrotask', 'setImmediate'] }); + try { + jest.setSystemTime(new Date(100_000)); + const targetSlot = SlotNumber(112); + const sendSpy = jest.spyOn(publisher, 'sendRequests').mockResolvedValue(undefined); + l1TxUtils.getBlock.mockResolvedValue({ timestamp: 99n } as any); + + const resultPromise = publisher.sendRequestsAt(targetSlot); + await jest.advanceTimersByTimeAsync(8_000); + await jest.advanceTimersByTimeAsync(500); + + await expect(resultPromise).resolves.toBeUndefined(); + expect(sendSpy).toHaveBeenCalledWith(targetSlot); + expect(l1TxUtils.getBlock).toHaveBeenCalled(); + } finally { + jest.useRealTimers(); + } + }); + + it('uses configured previous L1 block wait timing for scheduled requests', async () => { + jest.useFakeTimers({ doNotFake: ['nextTick', 'queueMicrotask', 'setImmediate'] }); + try { + jest.setSystemTime(new Date(100_000)); + const configuredPublisher = new SequencerPublisher( + { + ethereumSlotDuration: 12, + aztecSlotDuration: 36, + l1ChainId: 1, + sequencerPublisherPreviousL1BlockWaitTimeoutMs: 1_000, + sequencerPublisherPreviousL1BlockWaitPollIntervalMs: 250, + } as any, + { + blobClient, + rollupContract: rollup, + l1TxUtils, + epochCache, + slashingProposerContract, + governanceProposerContract, + dateProvider, + metrics: l1Metrics, + lastActions: {}, + }, + ); + const targetSlot = SlotNumber(112); + const sendSpy = jest.spyOn(configuredPublisher, 'sendRequests').mockResolvedValue(undefined); + l1TxUtils.getBlock.mockResolvedValue({ timestamp: 99n } as any); + + const resultPromise = configuredPublisher.sendRequestsAt(targetSlot); + await jest.advanceTimersByTimeAsync(999); + + expect(sendSpy).not.toHaveBeenCalled(); + + await jest.advanceTimersByTimeAsync(1); + await jest.advanceTimersByTimeAsync(250); + await expect(resultPromise).resolves.toBeUndefined(); + + expect(sendSpy).toHaveBeenCalledWith(targetSlot); + expect(l1TxUtils.getBlock).toHaveBeenCalledTimes(4); + } finally { + jest.useRealTimers(); + } + }); + it('does not send requests if no valid requests are found', async () => { publisher.addRequest({ action: 'propose', diff --git a/yarn-project/sequencer-client/src/publisher/sequencer-publisher.ts b/yarn-project/sequencer-client/src/publisher/sequencer-publisher.ts index 738e83d5d5b5..7652016b2aec 100644 --- a/yarn-project/sequencer-client/src/publisher/sequencer-publisher.ts +++ b/yarn-project/sequencer-client/src/publisher/sequencer-publisher.ts @@ -177,6 +177,8 @@ export class SequencerPublisher { protected log: Logger; protected ethereumSlotDuration: bigint; protected aztecSlotDuration: bigint; + private readonly previousL1BlockWaitTimeoutMs: number; + private readonly previousL1BlockWaitPollIntervalMs: number; /** Date provider for wall-clock time. */ private readonly dateProvider: DateProvider; @@ -205,7 +207,13 @@ export class SequencerPublisher { protected requests: RequestWithExpiry[] = []; constructor( - private config: Pick & + private config: Pick< + SequencerPublisherConfig, + | 'fishermanMode' + | 'l1TxFailedStore' + | 'sequencerPublisherPreviousL1BlockWaitTimeoutMs' + | 'sequencerPublisherPreviousL1BlockWaitPollIntervalMs' + > & Pick & { l1ChainId: number }, deps: { telemetry?: TelemetryClient; @@ -225,6 +233,8 @@ export class SequencerPublisher { this.log = deps.log ?? createLogger('sequencer:publisher'); this.ethereumSlotDuration = BigInt(config.ethereumSlotDuration); this.aztecSlotDuration = BigInt(config.aztecSlotDuration); + this.previousL1BlockWaitTimeoutMs = config.sequencerPublisherPreviousL1BlockWaitTimeoutMs; + this.previousL1BlockWaitPollIntervalMs = config.sequencerPublisherPreviousL1BlockWaitPollIntervalMs; this.dateProvider = deps.dateProvider; this.epochCache = deps.epochCache; this.lastActions = deps.lastActions; @@ -617,34 +627,69 @@ export class SequencerPublisher { /* * Schedules sending all enqueued requests at (or after) the start of the given L2 slot. - * Sleeps until one L1 slot before the L2 slot boundary so the tx has a chance of being - * picked up by the first L1 block of the L2 slot. - * NB: there is a known correctness risk — being included in the L1 block right before the - * L2 slot starts would revert propose with HeaderLib__InvalidSlotNumber. - * Uses InterruptibleSleep so it can be cancelled via interrupt(). */ public async sendRequestsAt(targetSlot: SlotNumber): Promise { - const l1Constants = this.epochCache.getL1Constants(); - // Start of the target L2 slot, in ms (getTimestampForSlot returns seconds). - const startOfTargetSlotMs = Number(getTimestampForSlot(targetSlot, l1Constants)) * 1000; - // Aim to be in the mempool one L1 slot before the L2 slot starts, so we have a chance of - // being picked up by the first L1 block of the L2 slot. - const submitAfterMs = startOfTargetSlotMs - Number(this.ethereumSlotDuration) * 1000; + await this.waitForTargetSlot(targetSlot); if (this.interrupted) { return undefined; } - const sleepMs = submitAfterMs - this.dateProvider.now(); - if (sleepMs > 0) { - this.log.debug(`Sleeping ${sleepMs}ms before sending requests`, { - targetSlot, - submitAfterMs, - }); + + return this.sendRequests(targetSlot); + } + + /** + * Sleeps until one L1 slot before the L2 slot boundary, and then waits for that L1 block + * to be mined, so we don't risk being included in it. If that block never gets mined after + * a timeout, we assume it got skipped on L1, so we send the tx anyway. + */ + private async waitForTargetSlot(targetSlot: SlotNumber): Promise { + const l1Constants = this.epochCache.getL1Constants(); + const nowInSeconds = this.dateProvider.nowInSeconds(); + const startOfTargetSlotTs = getTimestampForSlot(targetSlot, l1Constants); + const previousL1BlockTs = startOfTargetSlotTs - this.ethereumSlotDuration; + const waitDeadlineTs = previousL1BlockTs + BigInt(this.previousL1BlockWaitTimeoutMs / 1000); + const logCtx = { targetSlot, startOfTargetSlotTs, nowInSeconds, previousL1BlockTs, waitDeadlineTs }; + + // Check if we are already past time + if (nowInSeconds >= startOfTargetSlotTs) { + this.log.verbose(`Target slot ${targetSlot} already started, sending requests immediately`, logCtx); + return; + } + + // Otherwise we wait + this.log.debug(`Waiting for slot ${targetSlot} before sending requests`, logCtx); + + // Wait until previous L1 block timestamp first + const sleepMs = (Number(previousL1BlockTs) - nowInSeconds) * 1000; + if (sleepMs > 0 && !this.interrupted) { + this.log.trace(`Sleeping ${sleepMs}ms before waiting for previous L1 block`, logCtx); await this.interruptibleSleep.sleep(sleepMs); } - if (this.interrupted) { - return undefined; + + // Then loop until we see the previous L1 block, so we know that we cannot be included in it. + // We time out after a while, once we are sure that that block is skipped in L1. + while (!this.interrupted) { + try { + const nowInSeconds = this.dateProvider.nowInSeconds(); + logCtx.nowInSeconds = nowInSeconds; + + if (nowInSeconds >= waitDeadlineTs) { + this.log.warn(`Timed out waiting for previous L1 block before sending requests, proceeding`, logCtx); + return; + } + + const latestBlockTs = await this.l1TxUtils.getBlock().then(b => b.timestamp); + if (latestBlockTs >= previousL1BlockTs) { + this.log.debug(`Previous L1 block mined, proceeding to send requests`, { ...logCtx, latestBlockTs }); + return; + } + this.log.trace(`Previous L1 block not mined yet, continuing to wait`, { ...logCtx, latestBlockTs }); + } catch (err) { + this.log.error(`Error while waiting for previous L1 block before sending requests; retrying`, err, logCtx); + } finally { + await this.interruptibleSleep.sleep(this.previousL1BlockWaitPollIntervalMs); + } } - return this.sendRequests(targetSlot); } private callbackBundledTransactions( diff --git a/yarn-project/sequencer-client/src/sequencer/checkpoint_proposal_job.ts b/yarn-project/sequencer-client/src/sequencer/checkpoint_proposal_job.ts index ed4824b62b7c..fbd82ed70f68 100644 --- a/yarn-project/sequencer-client/src/sequencer/checkpoint_proposal_job.ts +++ b/yarn-project/sequencer-client/src/sequencer/checkpoint_proposal_job.ts @@ -398,9 +398,7 @@ export class CheckpointProposalJob implements Traceable { } } - await this.publisher.enqueueProposeCheckpoint(checkpoint, attestations, attestationsSignature, { - txTimeoutAt, - }); + await this.publisher.enqueueProposeCheckpoint(checkpoint, attestations, attestationsSignature, { txTimeoutAt }); } /** From 15f769dd9e650989b2a9a352eb00cf296f902517 Mon Sep 17 00:00:00 2001 From: Santiago Palladino Date: Fri, 12 Jun 2026 12:02:44 -0300 Subject: [PATCH 04/14] test(e2e_fees): bridge fee juice from a dedicated L1 account (#24054) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## What `e2e_fees/account_init` → *"account pays its own fee › pays natively in the Fee Juice by bridging funds themselves"* flakes while preparing L1 fee juice (before the account deploy), failing on the bridge write: ``` ContractFunctionExecutionError: ... contract function "depositToAztecPublic" from: 0xf39Fd6e51aad88F6F4ce6aB8827279cffFb92266 Details: replacement transaction underpriced ``` CI: http://ci.aztec-labs.com/60b00f39a7a0e3cc ## Why The fee-juice bridge test harness does direct L1 writes (`mint` → `approve` → `depositToAztecPublic`) through a plain viem client with **no nonce manager**, so each write picks its nonce via `getTransactionCount('pending')`. That client uses Anvil account `0xf39f…92266` (mnemonic index 0) — which is **also the sequencer publisher account**. The publisher tracks its own nonce in `l1-tx-utils`, independently of the test client. When a test write and a publisher checkpoint tx land on the same nonce concurrently, the higher-priced publisher tx wins and the test's lower-priced tx is rejected. In the failed run the publisher was sending on account #0 at ~95–120 gwei: ``` node:l1-tx-utils Sent L1 transaction 0x9652… {nonce: 67, account: 0xf39f…92266, maxFeePerGas: "95.396…"} ``` while the test's `depositToAztecPublic` went out on the same account at nonce **60** / ~26 gwei → `replacement transaction underpriced`. The shared-account race is pre-existing but rarely hit; #24037 (changing publisher send timing) makes the overlap more likely, which surfaced it. ## Fix Send the harness's direct L1 writes from a dedicated mnemonic account (`L1_DIRECT_WRITE_ACCOUNT_INDEX = 1`) instead of the deployer/publisher account #0. Index 1 is unused by any L1-tx sender in these setups (publisher = 0, prover = 2, validators = 3+), so the test's writes no longer share a nonce stream with the publisher. Affects `FeesTest` and `ClientFlowsBenchmark`. --- .../client_flows/client_flows_benchmark.ts | 17 +++++++++++++++-- .../end-to-end/src/e2e_fees/fees_test.ts | 12 ++++++++++-- .../end-to-end/src/fixtures/fixtures.ts | 8 ++++++++ 3 files changed, 33 insertions(+), 4 deletions(-) diff --git a/yarn-project/end-to-end/src/bench/client_flows/client_flows_benchmark.ts b/yarn-project/end-to-end/src/bench/client_flows/client_flows_benchmark.ts index 5bcd3495fb0f..e00052407a00 100644 --- a/yarn-project/end-to-end/src/bench/client_flows/client_flows_benchmark.ts +++ b/yarn-project/end-to-end/src/bench/client_flows/client_flows_benchmark.ts @@ -28,7 +28,12 @@ import type { ContractInstanceWithAddress } from '@aztec/stdlib/contract'; import { Gas, GasSettings } from '@aztec/stdlib/gas'; import { deriveSigningKey } from '@aztec/stdlib/keys'; -import { AUTOMINE_E2E_OPTS, MNEMONIC, getPaddedMaxFeesPerGas } from '../../fixtures/fixtures.js'; +import { + AUTOMINE_E2E_OPTS, + L1_DIRECT_WRITE_ACCOUNT_INDEX, + MNEMONIC, + getPaddedMaxFeesPerGas, +} from '../../fixtures/fixtures.js'; import { type EndToEndContext, type SetupOptions, deployAccounts, setup, teardown } from '../../fixtures/setup.js'; import { mintTokensToPrivate } from '../../fixtures/token_utils.js'; import { setupSponsoredFPC } from '../../fixtures/utils.js'; @@ -244,7 +249,15 @@ export class ClientFlowsBenchmark { this.feeJuiceBridgeTestHarness = await FeeJuicePortalTestingHarnessFactory.create({ aztecNode: this.context.aztecNodeService, aztecNodeAdmin: this.context.aztecNodeService, - l1Client: this.context.deployL1ContractsValues.l1Client, + // Bridge from a dedicated L1 account so its direct writes don't race the sequencer publisher's + // txs on the deployer account (see L1_DIRECT_WRITE_ACCOUNT_INDEX). + l1Client: createExtendedL1Client( + this.context.config.l1RpcUrls, + MNEMONIC, + undefined, + undefined, + L1_DIRECT_WRITE_ACCOUNT_INDEX, + ), wallet: this.adminWallet, logger: this.logger, }); diff --git a/yarn-project/end-to-end/src/e2e_fees/fees_test.ts b/yarn-project/end-to-end/src/e2e_fees/fees_test.ts index d51bbf1ad861..9f401a1c5d1f 100644 --- a/yarn-project/end-to-end/src/e2e_fees/fees_test.ts +++ b/yarn-project/end-to-end/src/e2e_fees/fees_test.ts @@ -23,7 +23,7 @@ import type { AztecNodeAdmin } from '@aztec/stdlib/interfaces/client'; import { getContract } from 'viem'; -import { MNEMONIC, getPaddedMaxFeesPerGas } from '../fixtures/fixtures.js'; +import { L1_DIRECT_WRITE_ACCOUNT_INDEX, MNEMONIC, getPaddedMaxFeesPerGas } from '../fixtures/fixtures.js'; import { type EndToEndContext, type SetupOptions, @@ -230,7 +230,15 @@ export class FeesTest { this.feeJuiceBridgeTestHarness = await FeeJuicePortalTestingHarnessFactory.create({ aztecNode: this.context.aztecNodeService, aztecNodeAdmin: this.context.aztecNodeService, - l1Client: this.context.deployL1ContractsValues.l1Client, + // Bridge from a dedicated L1 account so its direct writes don't race the sequencer publisher's + // txs on the deployer account (see L1_DIRECT_WRITE_ACCOUNT_INDEX). + l1Client: createExtendedL1Client( + this.context.config.l1RpcUrls, + MNEMONIC, + undefined, + undefined, + L1_DIRECT_WRITE_ACCOUNT_INDEX, + ), wallet: this.wallet, logger: this.logger, }); diff --git a/yarn-project/end-to-end/src/fixtures/fixtures.ts b/yarn-project/end-to-end/src/fixtures/fixtures.ts index 45979253b366..edc92ab04023 100644 --- a/yarn-project/end-to-end/src/fixtures/fixtures.ts +++ b/yarn-project/end-to-end/src/fixtures/fixtures.ts @@ -109,6 +109,14 @@ export const TEST_PEER_CHECK_INTERVAL_MS = 1000; export const TEST_MAX_PENDING_TX_POOL_COUNT = 10_000; // Number of max pending TXs ~ 1.56GB export const MNEMONIC = 'test test test test test test test test test test test junk'; + +// Mnemonic account index for tests that issue direct L1 writes (e.g. bridging fee juice) while a +// sequencer is running. The deployer/sequencer publisher uses index 0, the prover index 2, and +// validators index 3+. Test-side viem writes and the publisher's l1-tx-utils track nonces +// independently, so sharing an account causes "replacement transaction underpriced" races; index 1 +// is otherwise unused, so issuing those writes from it keeps them off the publisher's nonce stream. +export const L1_DIRECT_WRITE_ACCOUNT_INDEX = 1; + export const privateKey = Buffer.from('ac0974bec39a17e36ba4a6b4d238ff944bacb478cbed5efcae784d7bf4f2ff80', 'hex'); export const privateKey2 = Buffer.from('59c6995e998f97a5a0044966f0945389dc9e86dae88c7a8412f4603b6b78690d', 'hex'); From 96d47a078ca83e64835f9e8ef823af1dcdbd1eef Mon Sep 17 00:00:00 2001 From: Santiago Palladino Date: Fri, 12 Jun 2026 12:09:30 -0300 Subject: [PATCH 05/14] fix(aztec-node): pipelining-aware slot and fee simulation in simulatePublicCalls (#24031) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Motivation `AztecNodeService.simulatePublicCalls` built globals for a fictional next block using "the L2 slot at the next L1 slot" with no pipelining offset and no L1 state overrides. Under proposer pipelining this simulates one slot too early and computes the mana min fee against the current L1 state, ignoring the gossiped proposed checkpoint that will land before the simulated block does — producing wrong L1-to-L2 message sets and wrong fees for wallet-side estimation. ## Approach The simulation now mirrors how the sequencer picks the build slot and fee inputs (`sequencer.ts` / `checkpoint_proposal_job.ts` / the checkpoint simulation overrides builder), split into two cases: - **Mid-checkpoint continuation** (proposed blocks past the last checkpoint proposal): every block in a checkpoint shares the same checkpoint-wide globals, so the next block's globals are copied verbatim from the latest proposed block header with only the block number bumped. No L1 reads, no L1-to-L2 message insertion. A missing header fails the request rather than falling through, since the fork already contains the ongoing checkpoint's messages and a fall-through would insert them twice. - **Opening a new checkpoint**: the target slot is the sequencer's formula (`next-L1-slot L2 slot + PROPOSER_PIPELINING_SLOT_OFFSET`), maxed with `proposedCheckpointSlot + 1` when a checkpoint proposal is pending L1. The same `SimulationOverridesPlan` the sequencer uses is always applied: parent archive/temp-log/fee-header overrides derived from the proposed checkpoint when pipelining, tips pinned to the rollback target when the pending chain is invalid, and tips pinned to the checkpointed tip otherwise (neutralizing prunes in fee computation). Both the target slot and the plan derive from a single `ProposedCheckpointData` read so they cannot disagree about the parent. The simulator trusts archiver tips verbatim — no staleness guards. Stale proposed blocks and checkpoints are the archiver's responsibility (orphan pruning and L1-authoritative eviction), so a stale tip mis-simulates only transiently. The logic moves out of `AztecNodeService` into a new `NodePublicCallsSimulator` class so the slot/globals selection is unit-testable without standing up a node. Supersedes #23389. Caching of the L1 fee queries introduced here is tracked separately in A-1208. **Structured for commit-by-commit review:** 1. `refactor(aztec-node)`: pure move of the existing `simulatePublicCalls` code (and its tests) into `NodePublicCallsSimulator` — no behavior change. 2. `fix(aztec-node)`: the actual behavior changes described above. 3. `refactor`: moves the checkpoint simulation overrides builder from sequencer-client to `@aztec/stdlib/checkpoint` so the node consumes it from a shared home. ## API changes - `GlobalVariableBuilder.buildGlobalVariables` is removed from the stdlib interface and its implementations (sequencer-client, TXE); slot selection is now the caller's job and only `buildCheckpointGlobalVariables` remains. - `buildCheckpointSimulationOverridesPlan` and `computePipelinedParentFeeHeader` move from sequencer-client internals to `@aztec/stdlib/checkpoint`, shared by the sequencer and the node simulator. - `AztecNodeService` constructor takes an optional `RollupContract`; environments that never see a proposed checkpoint or an invalid pending chain (TXE) may omit it, and the simulator fails loudly if those paths are reached without it. ## Changes - **aztec-node**: new `NodePublicCallsSimulator` with the two-case globals selection and always-on overrides plan; `simulatePublicCalls` reduced to a thin delegate. - **aztec-node (tests)**: 12 unit tests covering verbatim mid-checkpoint continuation, the missing-header guard, the pipelining offset, parent-slot/override derivation from the proposed checkpoint data, torn-snapshot fallback, invalid-pending-chain tips pinning, message insertion semantics, and the rollup-contract invariant. - **stdlib**: gains `checkpoint/simulation_overrides.ts` (moved from sequencer-client, with its unit tests); `buildGlobalVariables` removed from the `GlobalVariableBuilder` interface. - **sequencer-client**: imports the overrides builder from stdlib; `buildGlobalVariables` removed from the implementation. - **txe**: `buildGlobalVariables` removed; TXE node construction passes no rollup contract. Fixes A-1063 --- .../node_public_calls_simulator.test.ts | 457 ++++++++++++++++++ .../aztec-node/node_public_calls_simulator.ts | 383 +++++++++++++++ .../aztec-node/src/aztec-node/server.test.ts | 237 +-------- .../aztec-node/src/aztec-node/server.ts | 183 +------ .../global_variable_builder/global_builder.ts | 36 +- .../sequencer/checkpoint_proposal_job.test.ts | 218 --------- .../src/sequencer/checkpoint_proposal_job.ts | 2 +- .../src/sequencer/sequencer.test.ts | 1 - .../src/sequencer/sequencer.ts | 7 +- yarn-project/stdlib/src/checkpoint/index.ts | 1 + .../checkpoint/simulation_overrides.test.ts | 243 ++++++++++ .../src/checkpoint/simulation_overrides.ts} | 6 +- .../stdlib/src/tx/global_variable_builder.ts | 18 +- .../state_machine/global_variable_builder.ts | 18 +- yarn-project/txe/src/state_machine/index.ts | 1 + 15 files changed, 1130 insertions(+), 681 deletions(-) create mode 100644 yarn-project/aztec-node/src/aztec-node/node_public_calls_simulator.test.ts create mode 100644 yarn-project/aztec-node/src/aztec-node/node_public_calls_simulator.ts create mode 100644 yarn-project/stdlib/src/checkpoint/simulation_overrides.test.ts rename yarn-project/{sequencer-client/src/sequencer/chain_state_overrides.ts => stdlib/src/checkpoint/simulation_overrides.ts} (97%) diff --git a/yarn-project/aztec-node/src/aztec-node/node_public_calls_simulator.test.ts b/yarn-project/aztec-node/src/aztec-node/node_public_calls_simulator.test.ts new file mode 100644 index 000000000000..618d90163c1e --- /dev/null +++ b/yarn-project/aztec-node/src/aztec-node/node_public_calls_simulator.test.ts @@ -0,0 +1,457 @@ +import { L1ToL2MessagesNotReadyError } from '@aztec/archiver'; +import type { EpochCacheInterface } from '@aztec/epoch-cache'; +import { type FeeHeader, RollupContract } from '@aztec/ethereum/contracts'; +import { + BlockNumber, + CheckpointNumber, + EpochNumber, + IndexWithinCheckpoint, + SlotNumber, +} from '@aztec/foundation/branded-types'; +import { Fr } from '@aztec/foundation/curves/bn254'; +import { EthAddress } from '@aztec/foundation/eth-address'; +import { unfreeze } from '@aztec/foundation/types'; +import { PublicProcessor, PublicProcessorFactory } from '@aztec/simulator/server'; +import { AztecAddress } from '@aztec/stdlib/aztec-address'; +import { + type BlockData, + BlockHash, + type BlockQuery, + L2Block, + type L2BlockSource, + type L2Tips, + type ValidateCheckpointResult, +} from '@aztec/stdlib/block'; +import type { ProposedCheckpointData } from '@aztec/stdlib/checkpoint'; +import type { ContractDataSource } from '@aztec/stdlib/contract'; +import { GasFees } from '@aztec/stdlib/gas'; +import type { MerkleTreeWriteOperations, WorldStateSynchronizer } from '@aztec/stdlib/interfaces/server'; +import type { L1ToL2MessageSource } from '@aztec/stdlib/messaging'; +import { CheckpointHeader } from '@aztec/stdlib/rollup'; +import { mockTx } from '@aztec/stdlib/testing'; +import { AppendOnlyTreeSnapshot, MerkleTreeId } from '@aztec/stdlib/trees'; +import { + BlockHeader, + type CheckpointGlobalVariables, + type GlobalVariableBuilder, + GlobalVariables, + TxEffect, +} from '@aztec/stdlib/tx'; + +import { jest } from '@jest/globals'; +import { type MockProxy, mock } from 'jest-mock-extended'; + +import { NodePublicCallsSimulator } from './node_public_calls_simulator.js'; + +const CHAIN_ID = new Fr(12345); +const ROLLUP_VERSION = new Fr(1); +const ROLLUP_ADDRESS = EthAddress.random(); + +describe('NodePublicCallsSimulator', () => { + let blockSource: MockProxy; + let worldStateSynchronizer: MockProxy; + let l1ToL2MessageSource: MockProxy; + let contractDataSource: MockProxy; + let globalVariableBuilder: MockProxy; + let rollupContract: MockProxy; + let epochCache: MockProxy; + let merkleTreeFork: MockProxy; + + let simulator: NodePublicCallsSimulator; + + // Captures the globals the simulator builds for the next block by intercepting the processor it + // would run them through, so tests can assert on the result rather than on mock call counts. + let builtGlobals: GlobalVariables | undefined; + + const makeTips = (args: { + proposed: BlockNumber; + checkpointedBlock: BlockNumber; + checkpointed: CheckpointNumber; + proven?: CheckpointNumber; + }): L2Tips => ({ + proposed: { number: args.proposed, hash: '0x0' }, + checkpointed: { + block: { number: args.checkpointedBlock, hash: '0x0' }, + checkpoint: { number: args.checkpointed, hash: '0x0' }, + }, + proven: { + block: { number: BlockNumber.ZERO, hash: '0x0' }, + checkpoint: { number: args.proven ?? args.checkpointed, hash: '0x0' }, + }, + finalized: { + block: { number: BlockNumber.ZERO, hash: '0x0' }, + checkpoint: { number: args.proven ?? args.checkpointed, hash: '0x0' }, + }, + }); + + const makeBlockData = (blockNumber: BlockNumber, slotNumber: SlotNumber, gasFees = GasFees.empty()): BlockData => ({ + header: BlockHeader.empty({ + globalVariables: GlobalVariables.empty({ blockNumber, slotNumber, gasFees }), + }), + archive: L2Block.empty().archive, + blockHash: BlockHash.random(), + checkpointNumber: CheckpointNumber(1), + indexWithinCheckpoint: IndexWithinCheckpoint(0), + }); + + const mockNextL1Slot = (slot: SlotNumber) => { + epochCache.getEpochAndSlotInNextL1Slot.mockReturnValue({ + epoch: EpochNumber.ZERO, + slot, + ts: 0n, + nowSeconds: 0n, + }); + }; + + const checkpointGlobals = (slotNumber: SlotNumber): CheckpointGlobalVariables => ({ + chainId: CHAIN_ID, + version: ROLLUP_VERSION, + slotNumber, + timestamp: BigInt(slotNumber) * 72n, + coinbase: EthAddress.ZERO, + feeRecipient: AztecAddress.ZERO, + gasFees: GasFees.empty(), + }); + + const lowGasTx = () => + mockTx(0x10000, { + numberOfNonRevertiblePublicCallRequests: 0, + numberOfRevertiblePublicCallRequests: 0, + chainId: CHAIN_ID, + version: ROLLUP_VERSION, + }); + + beforeEach(() => { + builtGlobals = undefined; + + blockSource = mock(); + worldStateSynchronizer = mock(); + l1ToL2MessageSource = mock(); + contractDataSource = mock(); + globalVariableBuilder = mock(); + rollupContract = mock(); + epochCache = mock(); + merkleTreeFork = mock(); + + worldStateSynchronizer.syncImmediate.mockResolvedValue(BlockNumber.ZERO); + // The fork is an AsyncDisposable; provide the hook so `await using` does not throw. + (merkleTreeFork as unknown as { [Symbol.asyncDispose]: () => Promise })[Symbol.asyncDispose] = () => + Promise.resolve(); + worldStateSynchronizer.fork.mockResolvedValue(merkleTreeFork); + l1ToL2MessageSource.getL1ToL2Messages.mockResolvedValue([]); + blockSource.getPendingChainValidationStatus.mockResolvedValue({ valid: true }); + blockSource.getProposedCheckpointData.mockResolvedValue(undefined); + + globalVariableBuilder.buildCheckpointGlobalVariables.mockImplementation((_c, _f, slotNumber) => + Promise.resolve(checkpointGlobals(slotNumber)), + ); + + // Capture the globals passed to the public processor and short-circuit execution with a stub + // processor that echoes them back, so `simulate` returns an output reflecting the chosen globals. + jest + .spyOn(PublicProcessorFactory.prototype, 'create') + .mockImplementation((_fork, globalVariables: GlobalVariables) => { + builtGlobals = globalVariables; + const processedTx = { + revertReason: undefined, + globalVariables, + txEffect: TxEffect.empty(), + gasUsed: { totalGas: undefined, teardownGas: undefined, publicGas: undefined, billedGas: undefined }, + }; + return { + process: () => Promise.resolve([[processedTx], [], [], [], []]), + } as unknown as PublicProcessor; + }); + + simulator = new NodePublicCallsSimulator({ + blockSource, + worldStateSynchronizer, + l1ToL2MessageSource, + contractDataSource, + globalVariableBuilder, + rollupContract, + epochCache, + signatureContext: { chainId: CHAIN_ID.toNumber(), rollupAddress: ROLLUP_ADDRESS }, + config: { rpcSimulatePublicMaxGasLimit: 1e11, rpcSimulatePublicMaxDebugLogMemoryReads: 100 }, + }); + }); + + afterEach(() => { + jest.restoreAllMocks(); + }); + + it('rejects when the gas limit exceeds the maximum', async () => { + const tx = await lowGasTx(); + unfreeze(tx.data.constants.txContext.gasSettings.gasLimits).l2Gas = 1e12; + await expect(simulator.simulate(tx)).rejects.toThrow(/gas/i); + }); + + describe('continuing an in-progress checkpoint', () => { + // A proposed checkpoint (#2) terminates at block 5, but the latest proposed block (block 9) is + // ahead of it, so the next block continues the in-progress checkpoint built on top of the proposed one. + const setupMidCheckpoint = () => { + blockSource.getL2Tips.mockResolvedValue( + makeTips({ proposed: BlockNumber(9), checkpointedBlock: BlockNumber(3), checkpointed: CheckpointNumber(1) }), + ); + blockSource.getProposedCheckpointData.mockResolvedValue( + makeProposedCheckpointData({ checkpointNumber: CheckpointNumber(2), lastBlock: BlockNumber(5) }), + ); + }; + + it('copies the latest proposed header globals verbatim and bumps only the block number', async () => { + const tx = await lowGasTx(); + const headerSlot = SlotNumber(42); + const headerGasFees = new GasFees(0, 777); + setupMidCheckpoint(); + blockSource.getBlockData.mockImplementation((query: BlockQuery) => + Promise.resolve('number' in query ? makeBlockData(query.number, headerSlot, headerGasFees) : undefined), + ); + mockNextL1Slot(SlotNumber(100)); + + await simulator.simulate(tx); + + expect(builtGlobals).toBeDefined(); + expect(builtGlobals!.blockNumber).toEqual(BlockNumber(10)); + expect(builtGlobals!.slotNumber).toEqual(headerSlot); + expect(builtGlobals!.gasFees).toEqual(headerGasFees); + // No fresh globals built and no L1 reads for fees when continuing an in-progress checkpoint. + expect(globalVariableBuilder.buildCheckpointGlobalVariables).not.toHaveBeenCalled(); + expect(rollupContract.getManaTarget).not.toHaveBeenCalled(); + }); + + it('does not insert L1-to-L2 messages', async () => { + const tx = await lowGasTx(); + setupMidCheckpoint(); + blockSource.getBlockData.mockImplementation((query: BlockQuery) => + Promise.resolve('number' in query ? makeBlockData(query.number, SlotNumber(42)) : undefined), + ); + mockNextL1Slot(SlotNumber(100)); + + await simulator.simulate(tx); + + expect(l1ToL2MessageSource.getL1ToL2Messages).not.toHaveBeenCalled(); + expect(merkleTreeFork.appendLeaves).not.toHaveBeenCalled(); + }); + + it('fails with a retryable error when the latest proposed header is missing, without double-inserting messages', async () => { + const tx = await lowGasTx(); + setupMidCheckpoint(); + // Latest proposed block header is missing (torn snapshot). + blockSource.getBlockData.mockResolvedValue(undefined); + mockNextL1Slot(SlotNumber(100)); + + await expect(simulator.simulate(tx)).rejects.toThrow(); + + // Must not treat the next block as opening a new checkpoint and re-insert the ongoing checkpoint's messages. + expect(l1ToL2MessageSource.getL1ToL2Messages).not.toHaveBeenCalled(); + expect(merkleTreeFork.appendLeaves).not.toHaveBeenCalled(); + expect(globalVariableBuilder.buildCheckpointGlobalVariables).not.toHaveBeenCalled(); + }); + }); + + describe('opening a new checkpoint', () => { + // The latest proposed block (5) coincides with the proposed-checkpoint frontier, so the next + // block opens a new checkpoint. Tests that pipeline on a proposed checkpoint additionally mock + // `getProposedCheckpointData`; otherwise the frontier is the checkpointed tip (block 5). + const setupBoundary = (args?: { checkpointed?: CheckpointNumber }) => + makeTips({ + proposed: BlockNumber(5), + checkpointedBlock: BlockNumber(5), + checkpointed: args?.checkpointed ?? CheckpointNumber(1), + }); + + it('targets the next L1 slot plus the pipelining offset and pins tips to the checkpointed tip when idle', async () => { + const tx = await lowGasTx(); + blockSource.getL2Tips.mockResolvedValue(setupBoundary()); + blockSource.getBlockData.mockImplementation((query: BlockQuery) => + Promise.resolve('number' in query ? makeBlockData(query.number, SlotNumber(5)) : undefined), + ); + mockNextL1Slot(SlotNumber(20)); + + await simulator.simulate(tx); + + // Sequencer formula: nextL1Slot + PROPOSER_PIPELINING_SLOT_OFFSET (=1). + const [, , slotArg, plan] = globalVariableBuilder.buildCheckpointGlobalVariables.mock.calls[0]; + expect(slotArg).toEqual(SlotNumber(21)); + expect(builtGlobals!.blockNumber).toEqual(BlockNumber(6)); + // Idle: tips pinned to the checkpointed tip (number 1) for both pending and proven. + expect(plan?.chainTipsOverride).toEqual({ pending: CheckpointNumber(1), proven: CheckpointNumber(1) }); + }); + + it('inserts L1-to-L2 messages for the next checkpoint', async () => { + const tx = await lowGasTx(); + const messages = [Fr.fromString('0x1234'), Fr.fromString('0x5678')]; + blockSource.getL2Tips.mockResolvedValue(setupBoundary()); + blockSource.getBlockData.mockImplementation((query: BlockQuery) => + Promise.resolve('number' in query ? makeBlockData(query.number, SlotNumber(5)) : undefined), + ); + mockNextL1Slot(SlotNumber(20)); + l1ToL2MessageSource.getL1ToL2Messages.mockResolvedValue(messages); + + await simulator.simulate(tx); + + // targetCheckpoint = proposedCheckpoint.number + 1 + expect(l1ToL2MessageSource.getL1ToL2Messages).toHaveBeenCalledWith(CheckpointNumber(2)); + const [treeId, appended] = merkleTreeFork.appendLeaves.mock.calls[0]; + expect(treeId).toEqual(MerkleTreeId.L1_TO_L2_MESSAGE_TREE); + expect(appended.slice(0, 2)).toEqual(messages); + }); + + it('tolerates L1ToL2MessagesNotReadyError and simulates without messages', async () => { + const tx = await lowGasTx(); + blockSource.getL2Tips.mockResolvedValue(setupBoundary()); + blockSource.getBlockData.mockImplementation((query: BlockQuery) => + Promise.resolve('number' in query ? makeBlockData(query.number, SlotNumber(5)) : undefined), + ); + mockNextL1Slot(SlotNumber(20)); + l1ToL2MessageSource.getL1ToL2Messages.mockRejectedValue(new L1ToL2MessagesNotReadyError(CheckpointNumber(2), 0n)); + + await expect(simulator.simulate(tx)).resolves.toBeDefined(); + expect(merkleTreeFork.appendLeaves).not.toHaveBeenCalled(); + }); + + it('targets parentSlot + 1 and carries the parent overrides when pipelining on a proposed checkpoint', async () => { + const tx = await lowGasTx(); + const parentSlot = SlotNumber(30); + const parentArchiveRoot = Fr.fromString('0xabcabc'); + blockSource.getL2Tips.mockResolvedValue(setupBoundary({ checkpointed: CheckpointNumber(2) })); + // The parent slot must come from the proposed checkpoint data itself, not from a separate + // block-data read that can be torn from it — so leave block data unavailable here. + blockSource.getBlockData.mockResolvedValue(undefined); + // The next L1 slot is well behind the proposed parent's slot, so the proposed-checkpoint + 1 + // term must win the max(). + mockNextL1Slot(SlotNumber(5)); + + const proposedCheckpointData = makeProposedCheckpointData({ + checkpointNumber: CheckpointNumber(3), + lastBlock: BlockNumber(5), + slotNumber: parentSlot, + archiveRoot: parentArchiveRoot, + }); + blockSource.getProposedCheckpointData.mockResolvedValue(proposedCheckpointData); + + const grandparentFeeHeader = makeFeeHeader(); + rollupContract.getCheckpoint.mockResolvedValue({ feeHeader: grandparentFeeHeader } as any); + rollupContract.getManaTarget.mockResolvedValue(1000n); + const childFeeHeader = makeFeeHeader(); + jest.spyOn(RollupContract, 'computeChildFeeHeader').mockReturnValue(childFeeHeader); + + await simulator.simulate(tx); + + const [, , slotArg, plan] = globalVariableBuilder.buildCheckpointGlobalVariables.mock.calls[0]; + expect(slotArg).toEqual(SlotNumber(31)); + expect(plan?.pendingCheckpointState?.archive).toEqual(parentArchiveRoot); + expect(plan?.pendingCheckpointState?.slotNumber).toEqual(parentSlot); + expect(plan?.pendingCheckpointState?.feeHeader).toEqual(childFeeHeader); + expect(RollupContract.computeChildFeeHeader).toHaveBeenCalledWith( + grandparentFeeHeader, + proposedCheckpointData.totalManaUsed, + proposedCheckpointData.feeAssetPriceModifier, + 1000n, + ); + }); + + it('pins tips to firstInvalid - 1 when the pending chain is invalid', async () => { + const tx = await lowGasTx(); + blockSource.getL2Tips.mockResolvedValue(setupBoundary({ checkpointed: CheckpointNumber(5) })); + blockSource.getBlockData.mockImplementation((query: BlockQuery) => + Promise.resolve('number' in query ? makeBlockData(query.number, SlotNumber(5)) : undefined), + ); + mockNextL1Slot(SlotNumber(20)); + blockSource.getPendingChainValidationStatus.mockResolvedValue(makeInvalidStatus(CheckpointNumber(4))); + + await simulator.simulate(tx); + + const [, , , plan] = globalVariableBuilder.buildCheckpointGlobalVariables.mock.calls[0]; + // invalidateToPendingCheckpointNumber = firstInvalid (4) - 1 = 3. + expect(plan?.chainTipsOverride).toEqual({ pending: CheckpointNumber(3), proven: CheckpointNumber(3) }); + }); + + it('degrades to a pinned-tips plan when pipelining without a rollup contract', async () => { + const tx = await lowGasTx(); + simulator = makeSimulatorWithoutRollupContract(); + blockSource.getL2Tips.mockResolvedValue(setupBoundary({ checkpointed: CheckpointNumber(2) })); + mockNextL1Slot(SlotNumber(5)); + blockSource.getProposedCheckpointData.mockResolvedValue( + makeProposedCheckpointData({ + checkpointNumber: CheckpointNumber(3), + lastBlock: BlockNumber(5), + slotNumber: SlotNumber(30), + archiveRoot: Fr.fromString('0xabcabc'), + }), + ); + + await simulator.simulate(tx); + + const [, , , plan] = globalVariableBuilder.buildCheckpointGlobalVariables.mock.calls[0]; + // No rollup contract: pin tips to the checkpointed tip (2) without pipelining overrides. + expect(plan?.chainTipsOverride).toEqual({ pending: CheckpointNumber(2), proven: CheckpointNumber(2) }); + expect(plan?.pendingCheckpointState).toBeUndefined(); + }); + + it('simulates without a rollup contract when idle (the TXE shape)', async () => { + const tx = await lowGasTx(); + simulator = makeSimulatorWithoutRollupContract(); + blockSource.getL2Tips.mockResolvedValue(setupBoundary()); + mockNextL1Slot(SlotNumber(20)); + + await expect(simulator.simulate(tx)).resolves.toBeDefined(); + + const [, , , plan] = globalVariableBuilder.buildCheckpointGlobalVariables.mock.calls[0]; + expect(plan?.chainTipsOverride).toEqual({ pending: CheckpointNumber(1), proven: CheckpointNumber(1) }); + }); + + const makeSimulatorWithoutRollupContract = () => + new NodePublicCallsSimulator({ + blockSource, + worldStateSynchronizer, + l1ToL2MessageSource, + contractDataSource, + globalVariableBuilder, + epochCache, + signatureContext: { chainId: CHAIN_ID.toNumber(), rollupAddress: ROLLUP_ADDRESS }, + config: { rpcSimulatePublicMaxGasLimit: 1e11, rpcSimulatePublicMaxDebugLogMemoryReads: 100 }, + }); + }); +}); + +function makeFeeHeader(): FeeHeader { + return { excessMana: 0n, manaUsed: 0n, ethPerFeeAsset: 0n, congestionCost: 0n, proverCost: 0n }; +} + +function makeProposedCheckpointData(args: { + checkpointNumber: CheckpointNumber; + lastBlock: BlockNumber; + slotNumber?: SlotNumber; + archiveRoot?: Fr; +}): ProposedCheckpointData { + return { + checkpointNumber: args.checkpointNumber, + header: CheckpointHeader.empty({ slotNumber: args.slotNumber ?? SlotNumber(0) }), + startBlock: args.lastBlock, + blockCount: 1, + totalManaUsed: 555n, + feeAssetPriceModifier: 7n, + archive: new AppendOnlyTreeSnapshot(args.archiveRoot ?? Fr.ZERO, 0), + checkpointOutHash: Fr.fromString('0xfeed'), + }; +} + +function makeInvalidStatus(firstInvalid: CheckpointNumber): ValidateCheckpointResult { + return { + valid: false, + checkpoint: { + archive: Fr.random(), + lastArchive: Fr.random(), + slotNumber: SlotNumber(10), + checkpointNumber: firstInvalid, + timestamp: 0n, + }, + committee: [], + epoch: EpochNumber.ZERO, + seed: 0n, + attestors: [], + attestations: [], + reason: 'insufficient-attestations', + }; +} diff --git a/yarn-project/aztec-node/src/aztec-node/node_public_calls_simulator.ts b/yarn-project/aztec-node/src/aztec-node/node_public_calls_simulator.ts new file mode 100644 index 000000000000..60e4142aab26 --- /dev/null +++ b/yarn-project/aztec-node/src/aztec-node/node_public_calls_simulator.ts @@ -0,0 +1,383 @@ +import { L1ToL2MessagesNotReadyError } from '@aztec/archiver'; +import { PROPOSER_PIPELINING_SLOT_OFFSET } from '@aztec/epoch-cache'; +import type { EpochCacheInterface } from '@aztec/epoch-cache'; +import { + type RollupContract, + SimulationOverridesBuilder, + type SimulationOverridesPlan, +} from '@aztec/ethereum/contracts'; +import { BlockNumber, CheckpointNumber, SlotNumber } from '@aztec/foundation/branded-types'; +import { compactArray } from '@aztec/foundation/collection'; +import { Fr } from '@aztec/foundation/curves/bn254'; +import { EthAddress } from '@aztec/foundation/eth-address'; +import { BadRequestError } from '@aztec/foundation/json-rpc'; +import { type Logger, createLogger } from '@aztec/foundation/log'; +import { DateProvider } from '@aztec/foundation/timer'; +import { isErrorClass } from '@aztec/foundation/types'; +import { PublicContractsDB, PublicProcessorFactory } from '@aztec/simulator/server'; +import { CollectionLimitsConfig, PublicSimulatorConfig } from '@aztec/stdlib/avm'; +import { AztecAddress } from '@aztec/stdlib/aztec-address'; +import type { L2BlockSource, L2Tips } from '@aztec/stdlib/block'; +import { type ProposedCheckpointData, buildCheckpointSimulationOverridesPlan } from '@aztec/stdlib/checkpoint'; +import type { ContractDataSource } from '@aztec/stdlib/contract'; +import type { WorldStateSynchronizer } from '@aztec/stdlib/interfaces/server'; +import { type L1ToL2MessageSource, appendL1ToL2MessagesToTree } from '@aztec/stdlib/messaging'; +import type { CoordinationSignatureContext } from '@aztec/stdlib/p2p'; +import { + type GlobalVariableBuilder, + GlobalVariables, + PublicSimulationOutput, + type SimulationOverrides, + type Tx, +} from '@aztec/stdlib/tx'; +import { type TelemetryClient, getTelemetryClient } from '@aztec/telemetry-client'; + +import { applyPublicDataOverrides } from './public_data_overrides.js'; + +/** Config fields the simulator needs — a narrow subset of `AztecNodeConfig`. */ +export interface NodePublicCallsSimulatorConfig { + /** Maximum total gas limit accepted for an incoming simulation. */ + rpcSimulatePublicMaxGasLimit: number; + /** Maximum number of debug-log memory reads collected during simulation. */ + rpcSimulatePublicMaxDebugLogMemoryReads: number; +} + +/** Dependencies required to build a {@link NodePublicCallsSimulator}. */ +export interface NodePublicCallsSimulatorDeps { + blockSource: L2BlockSource; + worldStateSynchronizer: WorldStateSynchronizer; + l1ToL2MessageSource: L1ToL2MessageSource; + contractDataSource: ContractDataSource; + globalVariableBuilder: GlobalVariableBuilder; + /** + * Rollup contract used to build the fee-relevant L1 state overrides when opening a new checkpoint. + * Only needed when a proposed parent checkpoint exists (pipelining) or the pending chain is invalid; + * may be omitted in environments that never reach those states (e.g. TXE). When omitted, those paths + * degrade to a pinned-tips plan (non-pipelined fees) instead. + */ + rollupContract?: RollupContract; + epochCache: EpochCacheInterface; + signatureContext: CoordinationSignatureContext; + config: NodePublicCallsSimulatorConfig; + telemetry?: TelemetryClient; + log?: Logger; +} + +/** + * Simulates the public part of a transaction against a fresh world-state fork. + * + * Extracted from `AztecNodeService` so the slot/globals selection can be unit-tested without + * standing up the whole node, and to keep `server.ts` smaller. + * + * The simulator picks globals in one of two ways, mirroring how the sequencer builds the next block: + * - **When the next block continues an in-progress checkpoint** (the latest proposed block is ahead of + * the proposed-checkpoint frontier): every block in a checkpoint shares the same + * `CheckpointGlobalVariables`, so we copy the latest proposed block's globals verbatim and only + * bump the block number. No L1 calls, no L1-to-L2 message insertion. + * - **When the next block opens a new checkpoint** (the latest proposed block coincides with the + * proposed-checkpoint frontier): we compute fresh globals for the slot the next block will land in, + * applying the same `SimulationOverridesPlan` the sequencer applies so the simulated mana min fee + * matches what the sequencer will write into the block header. + */ +export class NodePublicCallsSimulator { + private readonly blockSource: L2BlockSource; + private readonly worldStateSynchronizer: WorldStateSynchronizer; + private readonly l1ToL2MessageSource: L1ToL2MessageSource; + private readonly contractDataSource: ContractDataSource; + private readonly globalVariableBuilder: GlobalVariableBuilder; + private readonly rollupContract: RollupContract | undefined; + private readonly epochCache: EpochCacheInterface; + private readonly signatureContext: CoordinationSignatureContext; + private readonly config: NodePublicCallsSimulatorConfig; + private readonly telemetry: TelemetryClient; + private readonly log: Logger; + + constructor(deps: NodePublicCallsSimulatorDeps) { + this.blockSource = deps.blockSource; + this.worldStateSynchronizer = deps.worldStateSynchronizer; + this.l1ToL2MessageSource = deps.l1ToL2MessageSource; + this.contractDataSource = deps.contractDataSource; + this.globalVariableBuilder = deps.globalVariableBuilder; + this.rollupContract = deps.rollupContract; + this.epochCache = deps.epochCache; + this.signatureContext = deps.signatureContext; + this.config = deps.config; + this.telemetry = deps.telemetry ?? getTelemetryClient(); + this.log = deps.log ?? createLogger('node:public-calls-simulator'); + } + + /** + * Simulates the public part of a transaction with the current state. + * @param tx - The transaction to simulate. + * @param skipFeeEnforcement - If true, fee enforcement is skipped. + * @param overrides - Optional pre-simulation overrides applied to the ephemeral fork and contract DB. + */ + public async simulate( + tx: Tx, + skipFeeEnforcement = false, + overrides?: SimulationOverrides, + ): Promise { + // Check total gas limit for simulation + const gasSettings = tx.data.constants.txContext.gasSettings; + const txGasLimit = gasSettings.gasLimits.l2Gas; + const teardownGasLimit = gasSettings.teardownGasLimits.l2Gas; + if (txGasLimit + teardownGasLimit > this.config.rpcSimulatePublicMaxGasLimit) { + throw new BadRequestError( + `Transaction total gas limit ${ + txGasLimit + teardownGasLimit + } (${txGasLimit} + ${teardownGasLimit}) exceeds maximum gas limit ${ + this.config.rpcSimulatePublicMaxGasLimit + } for simulation`, + ); + } + + const txHash = tx.getTxHash(); + const [l2Tips, proposedCheckpointData] = await Promise.all([ + this.blockSource.getL2Tips(), + this.blockSource.getProposedCheckpointData(), + ]); + const latestBlockNumber = l2Tips.proposed.number; + const blockNumber = BlockNumber.add(latestBlockNumber, 1); + + // Terminating block of the proposed-checkpoint frontier. `getProposedCheckpointData()` returns the + // leading proposed (not-yet-L1-confirmed) checkpoint, whose last block is `startBlock + blockCount + // - 1`; with no proposed checkpoint the frontier coincides with the checkpointed tip. + const proposedCheckpointLastBlock = proposedCheckpointData + ? BlockNumber.add(proposedCheckpointData.startBlock, proposedCheckpointData.blockCount - 1) + : l2Tips.checkpointed.block.number; + + // The next block continues the in-progress checkpoint when the latest proposed block is ahead of + // the proposed-checkpoint terminating block; it opens a new checkpoint when they coincide. + const atCheckpointBoundary = proposedCheckpointLastBlock === l2Tips.proposed.number; + + // `targetCheckpoint` is the checkpoint whose L1-to-L2 messages must be inserted into the fork + // before simulation. Only set when opening a new checkpoint, where the next block is its first block. + const { globalVariables: newGlobalVariables, targetCheckpoint } = atCheckpointBoundary + ? await this.buildGlobalVariablesForNewCheckpoint(l2Tips, proposedCheckpointData, blockNumber) + : { globalVariables: await this.copyGlobalVariablesFromLatestProposedBlock(latestBlockNumber, blockNumber) }; + + const publicProcessorFactory = new PublicProcessorFactory( + this.contractDataSource, + new DateProvider(), + this.telemetry, + this.log.getBindings(), + ); + + this.log.verbose(`Simulating public calls for tx ${txHash}`, { + globalVariables: newGlobalVariables.toInspect(), + txHash, + blockNumber, + atCheckpointBoundary, + }); + + // Ensure world-state has caught up with the latest block we loaded from the archiver + await this.worldStateSynchronizer.syncImmediate(latestBlockNumber); + + const nextCheckpointMessages = await this.getNextCheckpointMessages(targetCheckpoint); + + // Request a new fork of the world state at the latest block number, and apply any overrides and next checkpoint messages to it before simulation + await using merkleTreeFork = await this.worldStateSynchronizer.fork(latestBlockNumber); + + if (nextCheckpointMessages !== undefined) { + this.log.debug( + `Appending ${nextCheckpointMessages.length} L1-to-L2 messages to the world state tree for the next checkpoint`, + { checkpointNumber: targetCheckpoint }, + ); + await appendL1ToL2MessagesToTree(merkleTreeFork, nextCheckpointMessages); + } + + await applyPublicDataOverrides(merkleTreeFork, overrides?.publicStorage); + + const config = PublicSimulatorConfig.from({ + skipFeeEnforcement, + collectDebugLogs: true, + collectHints: false, + collectCallMetadata: true, + collectStatistics: false, + collectionLimits: CollectionLimitsConfig.from({ + maxDebugLogMemoryReads: this.config.rpcSimulatePublicMaxDebugLogMemoryReads, + }), + }); + + const contractsDB = new PublicContractsDB(this.contractDataSource, this.log.getBindings()); + if (overrides?.contracts) { + contractsDB.addContracts(Object.values(overrides.contracts).map(({ instance }) => instance)); + } + const processor = publicProcessorFactory.create(merkleTreeFork, newGlobalVariables, config, contractsDB); + + // REFACTOR: Consider merging ProcessReturnValues into ProcessedTx + const [processedTxs, failedTxs, _usedTxs, returns, debugLogs] = await processor.process([tx]); + // REFACTOR: Consider returning the error rather than throwing + if (failedTxs.length) { + this.log.warn(`Simulated tx ${txHash} fails: ${failedTxs[0].error}`, { txHash }); + throw failedTxs[0].error; + } + + const [processedTx] = processedTxs; + return new PublicSimulationOutput( + processedTx.revertReason, + processedTx.globalVariables, + processedTx.txEffect, + returns, + processedTx.gasUsed, + debugLogs, + ); + } + + /** + * Fetches the next checkpoint's L1-to-L2 messages to insert into the fork before simulation. Only set + * when opening a new checkpoint; when continuing an in-progress checkpoint the ongoing checkpoint's + * messages were already applied when its first block synced, so inserting here would double-count them + * — which is why a missing header for the latest proposed block throws rather than falling through to + * this path. A not-ready or failed fetch degrades to simulating without the messages rather than + * failing the request. + */ + private async getNextCheckpointMessages(targetCheckpoint: CheckpointNumber | undefined): Promise { + if (targetCheckpoint === undefined) { + return undefined; + } + try { + return await this.l1ToL2MessageSource.getL1ToL2Messages(targetCheckpoint); + } catch (err) { + if (isErrorClass(err, L1ToL2MessagesNotReadyError)) { + this.log.warn( + `L1-to-L2 messages for checkpoint ${targetCheckpoint} are not ready yet (simulating without them)`, + { checkpointNumber: targetCheckpoint }, + ); + } else { + this.log.error( + `Failed to get L1-to-L2 messages for checkpoint ${targetCheckpoint} (simulating without them)`, + err, + { checkpointNumber: targetCheckpoint }, + ); + } + return undefined; + } + } + + /** + * Continues an in-progress checkpoint: the next block extends the checkpoint the latest proposed + * block belongs to. Every block in a checkpoint shares the same `CheckpointGlobalVariables`, so the + * next block's globals are the latest proposed block's globals with only the block number bumped — + * including the proposer's real coinbase/feeRecipient. No L1 reads and no L1-to-L2 message insertion + * happen here. + * + * A missing header means the archiver reported a proposed tip via `getL2Tips` but no longer has its + * data (a torn snapshot). We throw a transient/retryable error rather than treating the next block as + * opening a new checkpoint: the fork at `latestBlockNumber` already contains the ongoing checkpoint's + * L1-to-L2 messages, so inserting the next checkpoint's messages would append them a second time. + */ + private async copyGlobalVariablesFromLatestProposedBlock( + latestBlockNumber: BlockNumber, + blockNumber: BlockNumber, + ): Promise { + const latestBlockData = await this.blockSource.getBlockData({ number: latestBlockNumber }); + if (!latestBlockData) { + throw new Error( + `Cannot simulate public calls: latest proposed block ${latestBlockNumber} has no header on this node ` + + `(torn archiver snapshot); retry`, + ); + } + return GlobalVariables.from({ ...latestBlockData.header.globalVariables, blockNumber }); + } + + /** + * Opens a new checkpoint: the next block is the first of a fresh checkpoint. Picks the slot the next + * block will land in, mirroring the sequencer, and builds the same `SimulationOverridesPlan` the + * sequencer applies so the simulated mana min fee matches what the sequencer will write into the + * block header. Coinbase and fee recipient stay zero (we cannot know the future proposer's payout + * addresses), unlike continuing an in-progress checkpoint which inherits the real ones from the + * proposed header. Returns the target checkpoint so the caller inserts that checkpoint's L1-to-L2 + * messages into the fork. + */ + private async buildGlobalVariablesForNewCheckpoint( + l2Tips: L2Tips, + proposedCheckpointData: ProposedCheckpointData | undefined, + blockNumber: BlockNumber, + ): Promise<{ globalVariables: GlobalVariables; targetCheckpoint: CheckpointNumber }> { + const checkpointedCheckpointNumber = l2Tips.checkpointed.checkpoint.number; + // The new checkpoint sits on top of the proposed one when pipelining, otherwise on the + // checkpointed tip. The target slot and the overrides plan both derive from the single + // `proposedCheckpointData` read, so they cannot disagree about the proposed parent. + const proposedCheckpointNumber = proposedCheckpointData?.checkpointNumber ?? checkpointedCheckpointNumber; + + const targetSlot = this.computeTargetSlot(proposedCheckpointData); + const plan = await this.buildSimulationOverridesPlan(proposedCheckpointData, checkpointedCheckpointNumber); + + const checkpointGlobalVariables = await this.globalVariableBuilder.buildCheckpointGlobalVariables( + EthAddress.ZERO, + AztecAddress.ZERO, + targetSlot, + plan, + ); + + return { + globalVariables: GlobalVariables.from({ blockNumber, ...checkpointGlobalVariables }), + targetCheckpoint: CheckpointNumber(proposedCheckpointNumber + 1), + }; + } + + /** + * Slot the next block will land in. The first term is the sequencer's exact formula + * (`getEpochAndSlotInNextL1Slot().slot + PROPOSER_PIPELINING_SLOT_OFFSET`). The `max` with + * `proposedCheckpointSlot + 1` is an RPC-side approximation of the next build: when a proposed + * checkpoint is gossiped before its L1 slot starts, the next build (once its wall clock arrives) + * will target `parentSlot + 1`. The sequencer never advances its own target past wall clock — it + * just declines to build — so this is a prediction of inclusion globals, not literal sequencer + * behavior. The parent slot comes from the proposed checkpoint header so the slot and the + * overrides plan cannot derive from different snapshots. + */ + private computeTargetSlot(proposedCheckpointData: ProposedCheckpointData | undefined): SlotNumber { + const slotFromNextL1Timestamp = + this.epochCache.getEpochAndSlotInNextL1Slot().slot + PROPOSER_PIPELINING_SLOT_OFFSET; + const slotAfterProposedCheckpoint = proposedCheckpointData + ? proposedCheckpointData.header.slotNumber + 1 + : undefined; + return SlotNumber(Math.max(...compactArray([slotFromNextL1Timestamp, slotAfterProposedCheckpoint]))); + } + + /** + * Builds the chain-state overrides plan the simulator passes to `buildCheckpointGlobalVariables`, + * mirroring the sequencer (which always pins tips to neutralize prunes). When pipelining, the plan + * carries the proposed parent's archive, temp-checkpoint-log cell, and locally-derived fee header. + * + * Both the pipelining and invalid-pending-chain paths need a rollup contract for the L1 fee reads. + * Environments that omit it (e.g. TXE, which never has a proposed checkpoint and whose pending chain + * is always valid) fall back to pinning both pending and proven tips to the checkpointed tip, which + * neutralizes prunes in fee computation at the cost of non-pipelined fees. + */ + private async buildSimulationOverridesPlan( + proposedCheckpointData: ProposedCheckpointData | undefined, + checkpointedCheckpointNumber: CheckpointNumber, + ): Promise { + const rollup = this.rollupContract; + if (rollup) { + if (proposedCheckpointData) { + return buildCheckpointSimulationOverridesPlan({ + checkpointNumber: CheckpointNumber(proposedCheckpointData.checkpointNumber + 1), + proposedCheckpointData, + checkpointedCheckpointNumber, + rollup, + signatureContext: this.signatureContext, + log: this.log, + }); + } + + const validationStatus = await this.blockSource.getPendingChainValidationStatus(); + if (!validationStatus.valid) { + return buildCheckpointSimulationOverridesPlan({ + checkpointNumber: CheckpointNumber(checkpointedCheckpointNumber + 1), + invalidateToPendingCheckpointNumber: CheckpointNumber(validationStatus.checkpoint.checkpointNumber - 1), + checkpointedCheckpointNumber, + rollup, + signatureContext: this.signatureContext, + log: this.log, + }); + } + } + + return new SimulationOverridesBuilder() + .withChainTips({ pending: checkpointedCheckpointNumber, proven: checkpointedCheckpointNumber }) + .build(); + } +} diff --git a/yarn-project/aztec-node/src/aztec-node/server.test.ts b/yarn-project/aztec-node/src/aztec-node/server.test.ts index 24d6df5b158f..6b7ae27bf658 100644 --- a/yarn-project/aztec-node/src/aztec-node/server.test.ts +++ b/yarn-project/aztec-node/src/aztec-node/server.test.ts @@ -242,6 +242,7 @@ describe('aztec node', () => { 12345, rollupVersion.toNumber(), globalVariablesBuilder, + rollupContract, feeProvider, epochCache, getPackageVersion(), @@ -709,220 +710,6 @@ describe('aztec node', () => { }); }); - describe('simulatePublicCalls', () => { - const mockNextL1Slot = (slot: SlotNumber) => { - jest.spyOn(epochCache, 'getEpochAndSlotInNextL1Slot').mockReturnValue({ - epoch: EpochNumber(0), - slot, - ts: 0n, - nowSeconds: BigInt(NOW_S), - }); - }; - - const makeSimulationBlockData = ( - blockNumber: BlockNumber, - slotNumber: SlotNumber, - checkpointNumber = CheckpointNumber(1), - ): BlockData => ({ - header: BlockHeader.empty({ - globalVariables: GlobalVariables.empty({ blockNumber, slotNumber }), - }), - archive: L2Block.empty().archive, - blockHash: BlockHash.random(), - checkpointNumber, - indexWithinCheckpoint: IndexWithinCheckpoint(0), - }); - - it('refuses to simulate public calls if the gas limit is too high', async () => { - const tx = await mockTxForRollup(0x10000); - unfreeze(tx.data.constants.txContext.gasSettings.gasLimits).l2Gas = 1e12; - await expect(node.simulatePublicCalls(tx)).rejects.toThrow(/gas/i); - }); - - it('uses the slot after the proposed checkpoint when it is later than the next L1 timestamp slot', async () => { - const tx = await mockTxForRollup(0x10000); - const checkpointNumber = CheckpointNumber(1); - const proposedCheckpointBlockNumber = BlockNumber(9); - const targetSlot = SlotNumber(10); - l2BlockSource.getL2Tips.mockResolvedValue(makeTips({ proposed: proposedCheckpointBlockNumber })); - l2BlockSource.getProposedCheckpointData.mockResolvedValue( - makeProposedCheckpoint({ - checkpointNumber, - blockNumber: proposedCheckpointBlockNumber, - slotNumber: SlotNumber(9), - }), - ); - mockNextL1Slot(SlotNumber(5)); - globalVariablesBuilder.buildCheckpointGlobalVariables.mockResolvedValue({ - chainId, - version: rollupVersion, - slotNumber: targetSlot, - timestamp: 0n, - coinbase: EthAddress.ZERO, - feeRecipient: AztecAddress.ZERO, - gasFees: GasFees.empty(), - }); - - await expect(node.simulatePublicCalls(tx)).rejects.toThrow(); - - // Slot is read from the proposed checkpoint payload header, so no block fetch is needed for it. - expect(globalVariablesBuilder.buildGlobalVariables).not.toHaveBeenCalled(); - expect(globalVariablesBuilder.buildCheckpointGlobalVariables).toHaveBeenCalledWith( - EthAddress.ZERO, - AztecAddress.ZERO, - targetSlot, - ); - }); - - it('uses the next L1 timestamp slot when it is later than the slot after the proposed checkpoint', async () => { - const tx = await mockTxForRollup(0x10000); - const checkpointNumber = CheckpointNumber(1); - const proposedCheckpointBlockNumber = BlockNumber(9); - const targetSlot = SlotNumber(12); - l2BlockSource.getL2Tips.mockResolvedValue(makeTips({ proposed: proposedCheckpointBlockNumber })); - l2BlockSource.getProposedCheckpointData.mockResolvedValue( - makeProposedCheckpoint({ - checkpointNumber, - blockNumber: proposedCheckpointBlockNumber, - slotNumber: SlotNumber(9), - }), - ); - mockNextL1Slot(targetSlot); - globalVariablesBuilder.buildCheckpointGlobalVariables.mockResolvedValue({ - chainId, - version: rollupVersion, - slotNumber: targetSlot, - timestamp: 0n, - coinbase: EthAddress.ZERO, - feeRecipient: AztecAddress.ZERO, - gasFees: GasFees.empty(), - }); - - await expect(node.simulatePublicCalls(tx)).rejects.toThrow(); - - expect(globalVariablesBuilder.buildGlobalVariables).not.toHaveBeenCalled(); - expect(globalVariablesBuilder.buildCheckpointGlobalVariables).toHaveBeenCalledWith( - EthAddress.ZERO, - AztecAddress.ZERO, - targetSlot, - ); - }); - - it('uses the latest proposed block slot when it is ahead of the proposed checkpoint', async () => { - const tx = await mockTxForRollup(0x10000); - const checkpointNumber = CheckpointNumber(1); - const proposedCheckpointBlockNumber = BlockNumber(9); - const latestProposedBlockNumber = BlockNumber(12); - const targetSlot = SlotNumber(12); - l2BlockSource.getL2Tips.mockResolvedValue(makeTips({ proposed: latestProposedBlockNumber })); - l2BlockSource.getProposedCheckpointData.mockResolvedValue( - makeProposedCheckpoint({ - checkpointNumber, - blockNumber: proposedCheckpointBlockNumber, - slotNumber: SlotNumber(9), - }), - ); - l2BlockSource.getBlockData.mockResolvedValue( - makeSimulationBlockData(latestProposedBlockNumber, targetSlot, checkpointNumber), - ); - mockNextL1Slot(SlotNumber(5)); - globalVariablesBuilder.buildCheckpointGlobalVariables.mockResolvedValue({ - chainId, - version: rollupVersion, - slotNumber: targetSlot, - timestamp: 0n, - coinbase: EthAddress.ZERO, - feeRecipient: AztecAddress.ZERO, - gasFees: GasFees.empty(), - }); - - await expect(node.simulatePublicCalls(tx)).rejects.toThrow(); - - // The latest proposed block is ahead of the proposed checkpoint, so its slot is fetched. - expect(l2BlockSource.getBlockData).toHaveBeenCalledWith({ number: latestProposedBlockNumber }); - expect(globalVariablesBuilder.buildGlobalVariables).not.toHaveBeenCalled(); - expect(globalVariablesBuilder.buildCheckpointGlobalVariables).toHaveBeenCalledWith( - EthAddress.ZERO, - AztecAddress.ZERO, - targetSlot, - ); - }); - - it('disregards missing proposed block slots and uses the next L1 timestamp slot', async () => { - const tx = await mockTxForRollup(0x10000); - const checkpointNumber = CheckpointNumber(1); - const proposedCheckpointBlockNumber = BlockNumber(9); - const latestProposedBlockNumber = BlockNumber(12); - const targetSlot = SlotNumber(13); - l2BlockSource.getL2Tips.mockResolvedValue(makeTips({ proposed: latestProposedBlockNumber })); - l2BlockSource.getProposedCheckpointData.mockResolvedValue( - makeProposedCheckpoint({ - checkpointNumber, - blockNumber: proposedCheckpointBlockNumber, - slotNumber: SlotNumber(9), - }), - ); - l2BlockSource.getBlockData.mockResolvedValue(undefined); - mockNextL1Slot(targetSlot); - globalVariablesBuilder.buildCheckpointGlobalVariables.mockResolvedValue({ - chainId, - version: rollupVersion, - slotNumber: targetSlot, - timestamp: 0n, - coinbase: EthAddress.ZERO, - feeRecipient: AztecAddress.ZERO, - gasFees: GasFees.empty(), - }); - - await expect(node.simulatePublicCalls(tx)).rejects.toThrow(); - - // Latest proposed block slot is unavailable; falls back to the next L1 timestamp slot. - expect(l2BlockSource.getBlockData).toHaveBeenCalledWith({ number: latestProposedBlockNumber }); - expect(globalVariablesBuilder.buildGlobalVariables).not.toHaveBeenCalled(); - expect(globalVariablesBuilder.buildCheckpointGlobalVariables).toHaveBeenCalledWith( - EthAddress.ZERO, - AztecAddress.ZERO, - targetSlot, - ); - }); - - it('treats slot zero as a valid proposed checkpoint slot', async () => { - const tx = await mockTxForRollup(0x10000); - const checkpointNumber = CheckpointNumber(0); - const proposedCheckpointBlockNumber = BlockNumber(0); - const targetSlot = SlotNumber(1); - // No proposed checkpoint leads the frontier; the proposed-checkpoint frontier falls back to the - // checkpointed tip (block 0, slot 0), whose slot is read via getBlockData. - l2BlockSource.getL2Tips.mockResolvedValue( - makeTips({ proposed: proposedCheckpointBlockNumber, checkpointed: checkpointNumber }), - ); - l2BlockSource.getProposedCheckpointData.mockResolvedValue(undefined); - l2BlockSource.getBlockData.mockResolvedValue( - makeSimulationBlockData(proposedCheckpointBlockNumber, SlotNumber(0), checkpointNumber), - ); - mockNextL1Slot(SlotNumber(0)); - globalVariablesBuilder.buildCheckpointGlobalVariables.mockResolvedValue({ - chainId, - version: rollupVersion, - slotNumber: targetSlot, - timestamp: 0n, - coinbase: EthAddress.ZERO, - feeRecipient: AztecAddress.ZERO, - gasFees: GasFees.empty(), - }); - - await expect(node.simulatePublicCalls(tx)).rejects.toThrow(); - - expect(l2BlockSource.getBlockData).toHaveBeenCalledWith({ number: proposedCheckpointBlockNumber }); - expect(globalVariablesBuilder.buildGlobalVariables).not.toHaveBeenCalled(); - expect(globalVariablesBuilder.buildCheckpointGlobalVariables).toHaveBeenCalledWith( - EthAddress.ZERO, - AztecAddress.ZERO, - targetSlot, - ); - }); - }); - describe('reloadKeystore', () => { it('throws BadRequestError if no file-based keystore directory is configured', async () => { // Default node has no keyStoreDirectory set @@ -985,6 +772,7 @@ describe('aztec node', () => { 12345, rollupVersion.toNumber(), globalVariablesBuilder, + undefined, feeProvider, epochCache, getPackageVersion(), @@ -1175,6 +963,7 @@ describe('aztec node', () => { 12345, rollupVersion.toNumber(), globalVariablesBuilder, + undefined, feeProvider, epochCache, getPackageVersion(), @@ -1246,6 +1035,7 @@ describe('aztec node', () => { 12345, rollupVersion.toNumber(), globalVariablesBuilder, + undefined, mock(), epochCache, getPackageVersion(), @@ -1299,6 +1089,7 @@ describe('aztec node', () => { 12345, rollupVersion.toNumber(), globalVariablesBuilder, + undefined, mock(), epochCache, getPackageVersion(), @@ -1411,24 +1202,6 @@ describe('aztec node', () => { }; } - /** Builds the payload of the atomic leading-proposed-checkpoint read (last block = startBlock). */ - function makeProposedCheckpoint(args: { - checkpointNumber: CheckpointNumber; - blockNumber: BlockNumber; - slotNumber: SlotNumber; - }): ProposedCheckpointData { - return { - checkpointNumber: args.checkpointNumber, - header: CheckpointHeader.random({ slotNumber: args.slotNumber }), - archive: AppendOnlyTreeSnapshot.empty(), - checkpointOutHash: Fr.ZERO, - startBlock: args.blockNumber, - blockCount: 1, - totalManaUsed: 0n, - feeAssetPriceModifier: 0n, - }; - } - describe('getCheckpoint', () => { /** Builds a minimal ProposedCheckpointData stub. */ function makeProposedCheckpointData( diff --git a/yarn-project/aztec-node/src/aztec-node/server.ts b/yarn-project/aztec-node/src/aztec-node/server.ts index 06ab33134232..f830e1f601fa 100644 --- a/yarn-project/aztec-node/src/aztec-node/server.ts +++ b/yarn-project/aztec-node/src/aztec-node/server.ts @@ -1,4 +1,4 @@ -import { Archiver, L1ToL2MessagesNotReadyError, createArchiver } from '@aztec/archiver'; +import { Archiver, createArchiver } from '@aztec/archiver'; import { BBCircuitVerifier, BatchChonkVerifier, QueuedIVCVerifier } from '@aztec/bb-prover'; import { TestCircuitVerifier } from '@aztec/bb-prover/test'; import { type BlobClientInterface, createBlobClientWithFileStores } from '@aztec/blob-client/client'; @@ -26,7 +26,6 @@ import { retryUntil } from '@aztec/foundation/retry'; import { count } from '@aztec/foundation/string'; import { DateProvider, Timer } from '@aztec/foundation/timer'; import { MembershipWitness, SiblingPath } from '@aztec/foundation/trees'; -import { isErrorClass } from '@aztec/foundation/types'; import { type KeyStore, KeystoreManager, loadKeystores, mergeKeystores } from '@aztec/node-keystore'; import { trySnapshotSync, uploadSnapshot } from '@aztec/node-lib/actions'; import { createForwarderL1TxUtilsFromSigners, createL1TxUtilsFromSigners } from '@aztec/node-lib/factories'; @@ -47,7 +46,6 @@ import { type SequencerPublisher, } from '@aztec/sequencer-client'; import { AutomineSequencer, createAutomineSequencer } from '@aztec/sequencer-client/automine'; -import { PublicContractsDB, PublicProcessorFactory } from '@aztec/simulator/server'; import { AttestationsBlockWatcher, AttestedInvalidProposalWatcher, @@ -59,7 +57,6 @@ import { createSlasher, } from '@aztec/slasher'; import { STANDARD_MULTI_CALL_ENTRYPOINT_ADDRESS } from '@aztec/standard-contracts/multi-call-entrypoint'; -import { CollectionLimitsConfig, PublicSimulatorConfig } from '@aztec/stdlib/avm'; import { AztecAddress } from '@aztec/stdlib/aztec-address'; import { type BlockData, @@ -119,12 +116,7 @@ import { } from '@aztec/stdlib/interfaces/server'; import type { DebugLogStore, LogResult, PrivateLogsQuery, PublicLogsQuery } from '@aztec/stdlib/logs'; import { InMemoryDebugLogStore, NullDebugLogStore } from '@aztec/stdlib/logs'; -import { - InboxLeaf, - type L1ToL2MessageSource, - type L2ToL1MembershipWitness, - appendL1ToL2MessagesToTree, -} from '@aztec/stdlib/messaging'; +import { InboxLeaf, type L1ToL2MessageSource, type L2ToL1MembershipWitness } from '@aztec/stdlib/messaging'; import type { CheckpointAttestation } from '@aztec/stdlib/p2p'; import type { Offense } from '@aztec/stdlib/slashing'; import type { NullifierLeafPreimage, PublicDataTreeLeafPreimage } from '@aztec/stdlib/trees'; @@ -134,7 +126,6 @@ import { type FeeProvider, type GetTxReceiptOptions, type GlobalVariableBuilder as GlobalVariableBuilderInterface, - GlobalVariables, type IndexedTxEffect, MinedTxReceipt, type MinedTxStatus, @@ -182,7 +173,7 @@ import { } from './block_response_helpers.js'; import { type AztecNodeConfig, createKeyStoreForValidator } from './config.js'; import { NodeMetrics } from './node_metrics.js'; -import { applyPublicDataOverrides } from './public_data_overrides.js'; +import { NodePublicCallsSimulator } from './node_public_calls_simulator.js'; /** * The aztec node. @@ -193,6 +184,7 @@ export class AztecNodeService implements AztecNode, AztecNodeAdmin, AztecNodeDeb private isUploadingSnapshot = false; // Saved minTxsPerBlock used by `pauseSequencer` to restore production-sequencer config on resume. private sequencerPausedMinTxsPerBlock: number | undefined; + private readonly nodePublicCallsSimulator: NodePublicCallsSimulator; public readonly tracer: Tracer; @@ -212,6 +204,7 @@ export class AztecNodeService implements AztecNode, AztecNodeAdmin, AztecNodeDeb protected readonly l1ChainId: number, protected readonly version: number, protected readonly globalVariableBuilder: GlobalVariableBuilderInterface, + protected readonly rollupContract: RollupContract | undefined, protected readonly feeProvider: FeeProvider, protected readonly epochCache: EpochCacheInterface, protected readonly packageVersion: string, @@ -228,6 +221,22 @@ export class AztecNodeService implements AztecNode, AztecNodeAdmin, AztecNodeDeb this.metrics = new NodeMetrics(telemetry, 'AztecNodeService'); this.tracer = telemetry.getTracer('AztecNodeService'); + // The node never represents a proposer's payout addresses, so the simulator zeroes coinbase and + // fee recipient. The signature context only needs chain id + rollup address (see signature_utils). + this.nodePublicCallsSimulator = new NodePublicCallsSimulator({ + blockSource: this.blockSource, + worldStateSynchronizer: this.worldStateSynchronizer, + l1ToL2MessageSource: this.l1ToL2MessageSource, + contractDataSource: this.contractDataSource, + globalVariableBuilder: this.globalVariableBuilder, + rollupContract: this.rollupContract, + epochCache: this.epochCache, + signatureContext: { chainId: this.l1ChainId, rollupAddress: this.config.rollupAddress }, + config: this.config, + telemetry: this.telemetry, + log: this.log.createChild('public-calls-simulator'), + }); + this.log.info(`Aztec Node version: ${this.packageVersion}`); this.log.info(`Aztec Node started on chain 0x${l1ChainId.toString(16)}`, pickL1ContractAddresses(config)); @@ -666,7 +675,7 @@ export class AztecNodeService implements AztecNode, AztecNodeAdmin, AztecNodeDeb slotDuration: Number(slotDuration), }; - const globalVariableBuilder = new GlobalVariableBuilder(dateProvider, publicClient, globalVariableBuilderConfig); + const globalVariableBuilder = new GlobalVariableBuilder(publicClient, globalVariableBuilderConfig); const feeProvider = new FeeProviderImpl(dateProvider, publicClient, globalVariableBuilderConfig); const proverOnly = config.enableProverNode && config.disableValidator; @@ -1023,6 +1032,7 @@ export class AztecNodeService implements AztecNode, AztecNodeAdmin, AztecNodeDeb ethereumChain.chainInfo.id, config.rollupVersion, globalVariableBuilder, + rollupContract, feeProvider, epochCache, packageVersion, @@ -1633,155 +1643,12 @@ export class AztecNodeService implements AztecNode, AztecNodeAdmin, AztecNodeDeb @trackSpan('AztecNodeService.simulatePublicCalls', (tx: Tx) => ({ [Attributes.TX_HASH]: tx.getTxHash().toString(), })) - public async simulatePublicCalls( + public simulatePublicCalls( tx: Tx, skipFeeEnforcement = false, overrides?: SimulationOverrides, ): Promise { - // Check total gas limit for simulation - const gasSettings = tx.data.constants.txContext.gasSettings; - const txGasLimit = gasSettings.gasLimits.l2Gas; - const teardownGasLimit = gasSettings.teardownGasLimits.l2Gas; - if (txGasLimit + teardownGasLimit > this.config.rpcSimulatePublicMaxGasLimit) { - throw new BadRequestError( - `Transaction total gas limit ${ - txGasLimit + teardownGasLimit - } (${txGasLimit} + ${teardownGasLimit}) exceeds maximum gas limit ${ - this.config.rpcSimulatePublicMaxGasLimit - } for simulation`, - ); - } - - const txHash = tx.getTxHash(); - const l2Tips = await this.blockSource.getL2Tips(); - const latestBlockNumber = l2Tips.proposed.number; - const blockNumber = BlockNumber.add(latestBlockNumber, 1); - - // If sequencer is not initialized, we just set these values to zero for simulation. - const coinbase = EthAddress.ZERO; - const feeRecipient = AztecAddress.ZERO; - - // Resolve the proposed-checkpoint frontier (latest proposed checkpoint, which leads the - // checkpointed tip, falling back to the checkpointed tip when none exists). The proposed payload - // carries its header slot, so no extra block fetch is needed to derive the slot. - const proposedCheckpoint = await this.blockSource.getProposedCheckpointData(); - const proposedCheckpointBlockNumber = proposedCheckpoint - ? BlockNumber(proposedCheckpoint.startBlock + proposedCheckpoint.blockCount - 1) - : l2Tips.checkpointed.block.number; - const proposedCheckpointNumber = proposedCheckpoint?.checkpointNumber ?? l2Tips.checkpointed.checkpoint.number; - - // Define the slot for simulation as the max of the next L1 timestamp slot, the slot after the proposed - // checkpoint, and the latest proposed block's slot. - const proposedCheckpointSlot = - proposedCheckpoint?.header.slotNumber ?? - (await this.blockSource.getBlockData({ number: proposedCheckpointBlockNumber }))?.header.getSlot(); - let slotAfterProposedCheckpoint: SlotNumber | undefined; - if (proposedCheckpointSlot !== undefined) { - slotAfterProposedCheckpoint = SlotNumber.fromBigInt(BigInt(proposedCheckpointSlot) + 1n); - } - - let latestProposedBlockSlot: SlotNumber | undefined; - if (l2Tips.proposed.number > proposedCheckpointBlockNumber) { - latestProposedBlockSlot = ( - await this.blockSource.getBlockData({ number: l2Tips.proposed.number }) - )?.header.getSlot(); - } - const slotFromNextL1Timestamp = this.epochCache.getEpochAndSlotInNextL1Slot().slot; - const targetSlot = SlotNumber( - Math.max(...compactArray([slotFromNextL1Timestamp, slotAfterProposedCheckpoint, latestProposedBlockSlot])), - ); - - const checkpointGlobalVariables = await this.globalVariableBuilder.buildCheckpointGlobalVariables( - coinbase, - feeRecipient, - targetSlot, - ); - const newGlobalVariables = GlobalVariables.from({ blockNumber, ...checkpointGlobalVariables }); - - const publicProcessorFactory = new PublicProcessorFactory( - this.contractDataSource, - new DateProvider(), - this.telemetry, - this.log.getBindings(), - ); - - this.log.verbose(`Simulating public calls for tx ${txHash}`, { - globalVariables: newGlobalVariables.toInspect(), - txHash, - blockNumber, - }); - - // Ensure world-state has caught up with the latest block we loaded from the archiver - await this.worldStateSynchronizer.syncImmediate(latestBlockNumber); - - // If we detect the next block would start a new checkpoint, then insert L1-to-L2 messages into - // the world state tree so simulation can take them into account. We detect if the next block would - // start a new checkpoint by checking if the proposed checkpoint's block number matches the latest block number, - // which means the next block would be the first block of the next checkpoint. - const targetCheckpoint = CheckpointNumber(proposedCheckpointNumber + 1); - const nextCheckpointMessages: Fr[] | undefined = - proposedCheckpointBlockNumber === l2Tips.proposed.number - ? await this.l1ToL2MessageSource.getL1ToL2Messages(targetCheckpoint).catch(err => { - if (isErrorClass(err, L1ToL2MessagesNotReadyError)) { - this.log.warn( - `L1-to-L2 messages for checkpoint ${targetCheckpoint} are not ready yet (simulating without them)`, - ); - } else { - this.log.error( - `Failed to get L1-to-L2 messages for checkpoint ${targetCheckpoint} (simulating without them)`, - err, - ); - } - return undefined; - }) - : undefined; - - // Request a new fork of the world state at the latest block number, and apply any overrides and next checkpoint messages to it before simulation - await using merkleTreeFork = await this.worldStateSynchronizer.fork(latestBlockNumber); - - if (nextCheckpointMessages !== undefined) { - this.log.debug( - `Appending ${nextCheckpointMessages.length} L1-to-L2 messages to the world state tree for the next checkpoint`, - { checkpointNumber: targetCheckpoint }, - ); - await appendL1ToL2MessagesToTree(merkleTreeFork, nextCheckpointMessages); - } - await applyPublicDataOverrides(merkleTreeFork, overrides?.publicStorage); - - const config = PublicSimulatorConfig.from({ - skipFeeEnforcement, - collectDebugLogs: true, - collectHints: false, - collectCallMetadata: true, - collectStatistics: false, - collectionLimits: CollectionLimitsConfig.from({ - maxDebugLogMemoryReads: this.config.rpcSimulatePublicMaxDebugLogMemoryReads, - }), - }); - - const contractsDB = new PublicContractsDB(this.contractDataSource, this.log.getBindings()); - if (overrides?.contracts) { - contractsDB.addContracts(Object.values(overrides.contracts).map(({ instance }) => instance)); - } - const processor = publicProcessorFactory.create(merkleTreeFork, newGlobalVariables, config, contractsDB); - - // REFACTOR: Consider merging ProcessReturnValues into ProcessedTx - const [processedTxs, failedTxs, _usedTxs, returns, debugLogs] = await processor.process([tx]); - // REFACTOR: Consider returning the error rather than throwing - if (failedTxs.length) { - this.log.warn(`Simulated tx ${txHash} fails: ${failedTxs[0].error}`, { txHash }); - throw failedTxs[0].error; - } - - const [processedTx] = processedTxs; - return new PublicSimulationOutput( - processedTx.revertReason, - processedTx.globalVariables, - processedTx.txEffect, - returns, - processedTx.gasUsed, - debugLogs, - ); + return this.nodePublicCallsSimulator.simulate(tx, skipFeeEnforcement, overrides); } public async isValidTx( diff --git a/yarn-project/sequencer-client/src/global_variable_builder/global_builder.ts b/yarn-project/sequencer-client/src/global_variable_builder/global_builder.ts index 6bba5543958e..3ec926f260ac 100644 --- a/yarn-project/sequencer-client/src/global_variable_builder/global_builder.ts +++ b/yarn-project/sequencer-client/src/global_variable_builder/global_builder.ts @@ -4,18 +4,16 @@ import { buildSimulationOverridesStateOverride, } from '@aztec/ethereum/contracts'; import type { ViemPublicClient } from '@aztec/ethereum/types'; -import { BlockNumber, SlotNumber } from '@aztec/foundation/branded-types'; +import type { SlotNumber } from '@aztec/foundation/branded-types'; import { Fr } from '@aztec/foundation/curves/bn254'; import type { EthAddress } from '@aztec/foundation/eth-address'; -import type { DateProvider } from '@aztec/foundation/timer'; import type { AztecAddress } from '@aztec/stdlib/aztec-address'; -import { type L1RollupConstants, getNextL1SlotTimestamp, getTimestampForSlot } from '@aztec/stdlib/epoch-helpers'; +import { type L1RollupConstants, getTimestampForSlot } from '@aztec/stdlib/epoch-helpers'; import { GasFees } from '@aztec/stdlib/gas'; import type { CheckpointGlobalVariables, GlobalVariableBuilder as GlobalVariableBuilderInterface, } from '@aztec/stdlib/tx'; -import { GlobalVariables } from '@aztec/stdlib/tx'; /** Configuration for the GlobalVariableBuilder (excludes L1 client config). */ export type GlobalVariableBuilderConfig = { @@ -29,7 +27,6 @@ export type GlobalVariableBuilderConfig = { */ export class GlobalVariableBuilder implements GlobalVariableBuilderInterface { private readonly rollupContract: RollupContract; - private readonly ethereumSlotDuration: number; private readonly aztecSlotDuration: number; private readonly l1GenesisTime: bigint; @@ -37,47 +34,18 @@ export class GlobalVariableBuilder implements GlobalVariableBuilderInterface { private version: Fr; constructor( - private readonly dateProvider: DateProvider, private readonly publicClient: ViemPublicClient, config: GlobalVariableBuilderConfig, ) { this.version = new Fr(config.rollupVersion); this.chainId = new Fr(this.publicClient.chain!.id); - this.ethereumSlotDuration = config.ethereumSlotDuration; this.aztecSlotDuration = config.slotDuration; this.l1GenesisTime = config.l1GenesisTime; this.rollupContract = new RollupContract(this.publicClient, config.rollupAddress); } - /** - * Simple builder of global variables. - * @param blockNumber - The block number to build global variables for. - * @param coinbase - The address to receive block reward. - * @param feeRecipient - The address to receive fees. - * @param slotNumber - The slot number to use for the global variables, if undefined it will be calculated. - * @returns The global variables for the given block number. - */ - public async buildGlobalVariables( - blockNumber: BlockNumber, - coinbase: EthAddress, - feeRecipient: AztecAddress, - maybeSlot?: SlotNumber, - ): Promise { - const slot: SlotNumber = - maybeSlot ?? - (await this.rollupContract.getSlotAt( - getNextL1SlotTimestamp(this.dateProvider.nowInSeconds(), { - l1GenesisTime: this.l1GenesisTime, - ethereumSlotDuration: this.ethereumSlotDuration, - }), - )); - - const checkpointGlobalVariables = await this.buildCheckpointGlobalVariables(coinbase, feeRecipient, slot); - return GlobalVariables.from({ blockNumber, ...checkpointGlobalVariables }); - } - /** Builds global variables that are constant throughout a checkpoint. */ public async buildCheckpointGlobalVariables( coinbase: EthAddress, diff --git a/yarn-project/sequencer-client/src/sequencer/checkpoint_proposal_job.test.ts b/yarn-project/sequencer-client/src/sequencer/checkpoint_proposal_job.test.ts index 89978ee6e45c..119b88234b77 100644 --- a/yarn-project/sequencer-client/src/sequencer/checkpoint_proposal_job.test.ts +++ b/yarn-project/sequencer-client/src/sequencer/checkpoint_proposal_job.test.ts @@ -1,5 +1,4 @@ import { EpochCache } from '@aztec/epoch-cache'; -import { type FeeHeader, RollupContract } from '@aztec/ethereum/contracts'; import { BlockNumber, CheckpointNumber, @@ -13,7 +12,6 @@ import { Fr } from '@aztec/foundation/curves/bn254'; import { TimeoutError } from '@aztec/foundation/error'; import { EthAddress } from '@aztec/foundation/eth-address'; import { Signature } from '@aztec/foundation/eth-signature'; -import { createLogger } from '@aztec/foundation/log'; import { promiseWithResolvers } from '@aztec/foundation/promise'; import { TestDateProvider } from '@aztec/foundation/timer'; import type { TypedEventEmitter } from '@aztec/foundation/types'; @@ -73,7 +71,6 @@ import { mockTxIterator, setupTxsAndBlock, } from '../test/utils.js'; -import { buildCheckpointSimulationOverridesPlan, computePipelinedParentFeeHeader } from './chain_state_overrides.js'; import { CheckpointProposalJob } from './checkpoint_proposal_job.js'; import type { CheckpointProposalJobMetricsRecorder } from './checkpoint_proposal_job_metrics.js'; import type { SequencerEvents } from './events.js'; @@ -801,221 +798,6 @@ describe('CheckpointProposalJob', () => { ); } - describe('computePipelinedParentFeeHeader', () => { - // Use checkpoint 3 so the grandparent (checkpoint 1) is valid - const pipelinedCheckpointNumber = CheckpointNumber(3); - - const pendingData: ProposedCheckpointData = { - checkpointNumber: CheckpointNumber(2), - header: CheckpointHeader.empty(), - archive: AppendOnlyTreeSnapshot.empty(), - checkpointOutHash: Fr.ZERO, - startBlock: BlockNumber(2), - blockCount: 1, - totalManaUsed: 5000n, - feeAssetPriceModifier: 100n, - }; - - const grandparentFeeHeader: FeeHeader = { - manaUsed: 3000n, - excessMana: 1000n, - ethPerFeeAsset: 500n, - congestionCost: 50n, - proverCost: 10n, - }; - - it('returns undefined when checkpoint number is below 2 (genesis grandparent)', async () => { - const result = await computePipelinedParentFeeHeader({ - checkpointNumber: CheckpointNumber(1), - proposedCheckpointData: pendingData, - rollup: publisher.rollupContract, - log: createLogger('test'), - }); - expect(result).toBeUndefined(); - }); - - function mockRollup(overrides: { grandparentCheckpoint?: any; manaTarget?: bigint }) { - const rollup = publisher.rollupContract; - jest.spyOn(rollup, 'getCheckpoint').mockResolvedValue(overrides.grandparentCheckpoint); - jest.spyOn(rollup, 'getManaTarget').mockResolvedValue(overrides.manaTarget ?? 10_000n); - } - - it('computes fee header from grandparent checkpoint', async () => { - const manaTarget = 10_000n; - - mockRollup({ grandparentCheckpoint: { feeHeader: grandparentFeeHeader }, manaTarget }); - - const result = await computePipelinedParentFeeHeader({ - checkpointNumber: pipelinedCheckpointNumber, - proposedCheckpointData: pendingData, - rollup: publisher.rollupContract, - log: createLogger('test'), - }); - - expect(result).toBeDefined(); - - const expected = RollupContract.computeChildFeeHeader( - grandparentFeeHeader, - pendingData.totalManaUsed, - pendingData.feeAssetPriceModifier, - manaTarget, - ); - expect(result).toEqual(expected); - }); - - it('throws when grandparent checkpoint is not found', async () => { - mockRollup({ grandparentCheckpoint: undefined }); - - await expect( - computePipelinedParentFeeHeader({ - checkpointNumber: pipelinedCheckpointNumber, - proposedCheckpointData: pendingData, - rollup: publisher.rollupContract, - log: createLogger('test'), - }), - ).rejects.toThrow(/Grandparent checkpoint or feeHeader missing/); - }); - - it('throws when grandparent checkpoint has no feeHeader', async () => { - mockRollup({ grandparentCheckpoint: { feeHeader: undefined } }); - - await expect( - computePipelinedParentFeeHeader({ - checkpointNumber: pipelinedCheckpointNumber, - proposedCheckpointData: pendingData, - rollup: publisher.rollupContract, - log: createLogger('test'), - }), - ).rejects.toThrow(/Grandparent checkpoint or feeHeader missing/); - }); - - it('propagates errors from rollup calls', async () => { - jest.spyOn(publisher.rollupContract, 'getCheckpoint').mockRejectedValue(new Error('rpc error')); - - await expect( - computePipelinedParentFeeHeader({ - checkpointNumber: pipelinedCheckpointNumber, - proposedCheckpointData: pendingData, - rollup: publisher.rollupContract, - log: createLogger('test'), - }), - ).rejects.toThrow(/rpc error/); - }); - }); - - describe('buildCheckpointSimulationOverridesPlan', () => { - const checkpointNumberUnderTest = CheckpointNumber(2); - - const grandparentFeeHeader: FeeHeader = { - manaUsed: 3000n, - excessMana: 1000n, - ethPerFeeAsset: 500n, - congestionCost: 50n, - proverCost: 10n, - }; - - function mockGrandparentFeeHeader() { - jest - .spyOn(publisher.rollupContract, 'getCheckpoint') - .mockResolvedValue({ feeHeader: grandparentFeeHeader } as any); - jest.spyOn(publisher.rollupContract, 'getManaTarget').mockResolvedValue(10_000n); - } - - function makeProposedParent(checkpointNumber: CheckpointNumber): ProposedCheckpointData { - return { - checkpointNumber, - header: CheckpointHeader.empty(), - archive: new AppendOnlyTreeSnapshot(Fr.random(), 1), - checkpointOutHash: Fr.random(), - startBlock: BlockNumber(1), - blockCount: 1, - totalManaUsed: 5000n, - feeAssetPriceModifier: 100n, - }; - } - - it('pins both pending and proven to the snapshot when no proposed/invalidate input is provided', async () => { - const plan = await buildCheckpointSimulationOverridesPlan({ - checkpointNumber: checkpointNumberUnderTest, - checkpointedCheckpointNumber: CheckpointNumber(4), - rollup: publisher.rollupContract, - signatureContext, - log: createLogger('test'), - }); - expect(plan?.chainTipsOverride?.pending).toEqual(CheckpointNumber(4)); - expect(plan?.chainTipsOverride?.proven).toEqual(CheckpointNumber(4)); - expect(plan?.pendingCheckpointState).toBeUndefined(); - }); - - it('overrides the full pending checkpoint cell from a pipelined parent', async () => { - mockGrandparentFeeHeader(); - const proposedData = makeProposedParent(CheckpointNumber(1)); - - const plan = await buildCheckpointSimulationOverridesPlan({ - checkpointNumber: checkpointNumberUnderTest, - proposedCheckpointData: proposedData, - checkpointedCheckpointNumber: CheckpointNumber(0), - rollup: publisher.rollupContract, - signatureContext, - log: createLogger('test'), - }); - - expect(plan?.chainTipsOverride?.pending).toEqual(CheckpointNumber(1)); - expect(plan?.chainTipsOverride?.proven).toEqual(CheckpointNumber(1)); - expect(plan?.pendingCheckpointState?.archive).toEqual(proposedData.archive.root); - expect(plan?.pendingCheckpointState?.slotNumber).toEqual(proposedData.header.slotNumber); - expect(plan?.pendingCheckpointState?.headerHash).toEqual(proposedData.header.hash()); - expect(plan?.pendingCheckpointState?.outHash).toEqual(proposedData.checkpointOutHash); - expect(plan?.pendingCheckpointState?.payloadDigest).toBeDefined(); - expect(plan?.pendingCheckpointState?.feeHeader).toBeDefined(); - }); - - it('throws when the pipelined parent does not match the expected parent checkpoint', async () => { - const proposedData = makeProposedParent(CheckpointNumber(5)); - - await expect( - buildCheckpointSimulationOverridesPlan({ - checkpointNumber: checkpointNumberUnderTest, - proposedCheckpointData: proposedData, - checkpointedCheckpointNumber: CheckpointNumber(0), - rollup: publisher.rollupContract, - signatureContext, - log: createLogger('test'), - }), - ).rejects.toThrow(/does not match expected parent/); - }); - - it('throws when both proposedCheckpointData and invalidateToPendingCheckpointNumber are provided', async () => { - const proposedData = makeProposedParent(CheckpointNumber(1)); - - await expect( - buildCheckpointSimulationOverridesPlan({ - checkpointNumber: checkpointNumberUnderTest, - proposedCheckpointData: proposedData, - invalidateToPendingCheckpointNumber: CheckpointNumber(0), - checkpointedCheckpointNumber: CheckpointNumber(0), - rollup: publisher.rollupContract, - signatureContext, - log: createLogger('test'), - }), - ).rejects.toThrow(/mutually exclusive/); - }); - - it('sets pending and proven from an invalidation rollback without archive/fee overrides', async () => { - const plan = await buildCheckpointSimulationOverridesPlan({ - checkpointNumber: checkpointNumberUnderTest, - invalidateToPendingCheckpointNumber: CheckpointNumber(0), - checkpointedCheckpointNumber: CheckpointNumber(2), - rollup: publisher.rollupContract, - signatureContext, - log: createLogger('test'), - }); - expect(plan?.chainTipsOverride?.pending).toEqual(CheckpointNumber(0)); - expect(plan?.chainTipsOverride?.proven).toEqual(CheckpointNumber(0)); - expect(plan?.pendingCheckpointState).toBeUndefined(); - }); - }); - describe('pipelining parent checkpoint validation', () => { const parentCheckpointHeader = CheckpointHeader.empty(); const parentCheckpointHash = parentCheckpointHeader.hash().toString(); diff --git a/yarn-project/sequencer-client/src/sequencer/checkpoint_proposal_job.ts b/yarn-project/sequencer-client/src/sequencer/checkpoint_proposal_job.ts index fbd82ed70f68..8c60a98e8e5d 100644 --- a/yarn-project/sequencer-client/src/sequencer/checkpoint_proposal_job.ts +++ b/yarn-project/sequencer-client/src/sequencer/checkpoint_proposal_job.ts @@ -37,6 +37,7 @@ import { import { type Checkpoint, type ProposedCheckpointData, + buildCheckpointSimulationOverridesPlan, getPreviousCheckpointOutHashes, validateCheckpoint, } from '@aztec/stdlib/checkpoint'; @@ -68,7 +69,6 @@ import { DutyAlreadySignedError, SlashingProtectionError } from '@aztec/validato import type { GlobalVariableBuilder } from '../global_variable_builder/global_builder.js'; import type { InvalidateCheckpointRequest, SequencerPublisher } from '../publisher/sequencer-publisher.js'; -import { buildCheckpointSimulationOverridesPlan } from './chain_state_overrides.js'; import type { CheckpointProposalJobMetricsRecorder } from './checkpoint_proposal_job_metrics.js'; import { CheckpointVoter } from './checkpoint_voter.js'; import { SequencerInterruptedError } from './errors.js'; diff --git a/yarn-project/sequencer-client/src/sequencer/sequencer.test.ts b/yarn-project/sequencer-client/src/sequencer/sequencer.test.ts index 816092341148..87caadf8b4c8 100644 --- a/yarn-project/sequencer-client/src/sequencer/sequencer.test.ts +++ b/yarn-project/sequencer-client/src/sequencer/sequencer.test.ts @@ -260,7 +260,6 @@ describe('sequencer', () => { rollupContract.getManaTarget.mockResolvedValue(10_000n); globalVariableBuilder = mock(); - globalVariableBuilder.buildGlobalVariables.mockResolvedValue(globalVariables); globalVariableBuilder.buildCheckpointGlobalVariables.mockResolvedValue(omit(globalVariables, 'blockNumber')); p2p = mock({ diff --git a/yarn-project/sequencer-client/src/sequencer/sequencer.ts b/yarn-project/sequencer-client/src/sequencer/sequencer.ts index e84f6dd0506a..0719c5e47830 100644 --- a/yarn-project/sequencer-client/src/sequencer/sequencer.ts +++ b/yarn-project/sequencer-client/src/sequencer/sequencer.ts @@ -18,7 +18,11 @@ import type { ProposedCheckpointSink, ValidateCheckpointResult, } from '@aztec/stdlib/block'; -import type { Checkpoint, ProposedCheckpointData } from '@aztec/stdlib/checkpoint'; +import { + type Checkpoint, + type ProposedCheckpointData, + buildCheckpointSimulationOverridesPlan, +} from '@aztec/stdlib/checkpoint'; import type { ChainConfig } from '@aztec/stdlib/config'; import { getEpochAtSlot } from '@aztec/stdlib/epoch-helpers'; import { @@ -45,7 +49,6 @@ import { DefaultSequencerConfig } from '../config.js'; import type { GlobalVariableBuilder } from '../global_variable_builder/global_builder.js'; import type { SequencerPublisherFactory } from '../publisher/sequencer-publisher-factory.js'; import type { InvalidateCheckpointRequest, SequencerPublisher } from '../publisher/sequencer-publisher.js'; -import { buildCheckpointSimulationOverridesPlan } from './chain_state_overrides.js'; import { CheckpointProposalJob } from './checkpoint_proposal_job.js'; import { CheckpointProposalJobMetrics } from './checkpoint_proposal_job_metrics.js'; import { CheckpointVoter } from './checkpoint_voter.js'; diff --git a/yarn-project/stdlib/src/checkpoint/index.ts b/yarn-project/stdlib/src/checkpoint/index.ts index cff0a7d04b2d..8a0432995659 100644 --- a/yarn-project/stdlib/src/checkpoint/index.ts +++ b/yarn-project/stdlib/src/checkpoint/index.ts @@ -5,4 +5,5 @@ export * from './checkpoint_reexecution_tracker.js'; export * from './digest.js'; export * from './previous_checkpoint_out_hashes.js'; export * from './published_checkpoint.js'; +export * from './simulation_overrides.js'; export * from './validate.js'; diff --git a/yarn-project/stdlib/src/checkpoint/simulation_overrides.test.ts b/yarn-project/stdlib/src/checkpoint/simulation_overrides.test.ts new file mode 100644 index 000000000000..0d30e9492eb1 --- /dev/null +++ b/yarn-project/stdlib/src/checkpoint/simulation_overrides.test.ts @@ -0,0 +1,243 @@ +import { type FeeHeader, RollupContract } from '@aztec/ethereum/contracts'; +import { BlockNumber, CheckpointNumber } from '@aztec/foundation/branded-types'; +import { Fr } from '@aztec/foundation/curves/bn254'; +import { EthAddress } from '@aztec/foundation/eth-address'; +import { createLogger } from '@aztec/foundation/log'; + +import { mock } from 'jest-mock-extended'; + +import type { CoordinationSignatureContext } from '../p2p/signature_utils.js'; +import { CheckpointHeader } from '../rollup/checkpoint_header.js'; +import { AppendOnlyTreeSnapshot } from '../trees/append_only_tree_snapshot.js'; +import type { ProposedCheckpointData } from './checkpoint_data.js'; +import { buildCheckpointSimulationOverridesPlan, computePipelinedParentFeeHeader } from './simulation_overrides.js'; + +describe('computePipelinedParentFeeHeader', () => { + let rollup: ReturnType>; + + beforeEach(() => { + rollup = mock(); + }); + + // Use checkpoint 3 so the grandparent (checkpoint 1) is valid + const pipelinedCheckpointNumber = CheckpointNumber(3); + + const pendingData: ProposedCheckpointData = { + checkpointNumber: CheckpointNumber(2), + header: CheckpointHeader.empty(), + archive: AppendOnlyTreeSnapshot.empty(), + checkpointOutHash: Fr.ZERO, + startBlock: BlockNumber(2), + blockCount: 1, + totalManaUsed: 5000n, + feeAssetPriceModifier: 100n, + }; + + const grandparentFeeHeader: FeeHeader = { + manaUsed: 3000n, + excessMana: 1000n, + ethPerFeeAsset: 500n, + congestionCost: 50n, + proverCost: 10n, + }; + + it('returns undefined when checkpoint number is below 2 (genesis grandparent)', async () => { + const result = await computePipelinedParentFeeHeader({ + checkpointNumber: CheckpointNumber(1), + proposedCheckpointData: pendingData, + rollup, + log: createLogger('test'), + }); + expect(result).toBeUndefined(); + }); + + function mockRollup(overrides: { grandparentCheckpoint?: any; manaTarget?: bigint }) { + rollup.getCheckpoint.mockResolvedValue(overrides.grandparentCheckpoint); + rollup.getManaTarget.mockResolvedValue(overrides.manaTarget ?? 10_000n); + } + + it('computes fee header from grandparent checkpoint', async () => { + const manaTarget = 10_000n; + + mockRollup({ grandparentCheckpoint: { feeHeader: grandparentFeeHeader }, manaTarget }); + + const result = await computePipelinedParentFeeHeader({ + checkpointNumber: pipelinedCheckpointNumber, + proposedCheckpointData: pendingData, + rollup, + log: createLogger('test'), + }); + + expect(result).toBeDefined(); + + const expected = RollupContract.computeChildFeeHeader( + grandparentFeeHeader, + pendingData.totalManaUsed, + pendingData.feeAssetPriceModifier, + manaTarget, + ); + expect(result).toEqual(expected); + }); + + it('throws when grandparent checkpoint is not found', async () => { + mockRollup({ grandparentCheckpoint: undefined }); + + await expect( + computePipelinedParentFeeHeader({ + checkpointNumber: pipelinedCheckpointNumber, + proposedCheckpointData: pendingData, + rollup, + log: createLogger('test'), + }), + ).rejects.toThrow(/Grandparent checkpoint or feeHeader missing/); + }); + + it('throws when grandparent checkpoint has no feeHeader', async () => { + mockRollup({ grandparentCheckpoint: { feeHeader: undefined } }); + + await expect( + computePipelinedParentFeeHeader({ + checkpointNumber: pipelinedCheckpointNumber, + proposedCheckpointData: pendingData, + rollup, + log: createLogger('test'), + }), + ).rejects.toThrow(/Grandparent checkpoint or feeHeader missing/); + }); + + it('propagates errors from rollup calls', async () => { + rollup.getCheckpoint.mockRejectedValue(new Error('rpc error')); + + await expect( + computePipelinedParentFeeHeader({ + checkpointNumber: pipelinedCheckpointNumber, + proposedCheckpointData: pendingData, + rollup, + log: createLogger('test'), + }), + ).rejects.toThrow(/rpc error/); + }); +}); + +describe('buildCheckpointSimulationOverridesPlan', () => { + let rollup: ReturnType>; + + const chainId = new Fr(12345); + const signatureContext: CoordinationSignatureContext = { + chainId: chainId.toNumber(), + rollupAddress: EthAddress.random(), + }; + + beforeEach(() => { + rollup = mock(); + }); + + const checkpointNumberUnderTest = CheckpointNumber(2); + + const grandparentFeeHeader: FeeHeader = { + manaUsed: 3000n, + excessMana: 1000n, + ethPerFeeAsset: 500n, + congestionCost: 50n, + proverCost: 10n, + }; + + function mockGrandparentFeeHeader() { + rollup.getCheckpoint.mockResolvedValue({ feeHeader: grandparentFeeHeader } as any); + rollup.getManaTarget.mockResolvedValue(10_000n); + } + + function makeProposedParent(checkpointNumber: CheckpointNumber): ProposedCheckpointData { + return { + checkpointNumber, + header: CheckpointHeader.empty(), + archive: new AppendOnlyTreeSnapshot(Fr.random(), 1), + checkpointOutHash: Fr.random(), + startBlock: BlockNumber(1), + blockCount: 1, + totalManaUsed: 5000n, + feeAssetPriceModifier: 100n, + }; + } + + it('pins both pending and proven to the snapshot when no proposed/invalidate input is provided', async () => { + const plan = await buildCheckpointSimulationOverridesPlan({ + checkpointNumber: checkpointNumberUnderTest, + checkpointedCheckpointNumber: CheckpointNumber(4), + rollup, + signatureContext, + log: createLogger('test'), + }); + expect(plan?.chainTipsOverride?.pending).toEqual(CheckpointNumber(4)); + expect(plan?.chainTipsOverride?.proven).toEqual(CheckpointNumber(4)); + expect(plan?.pendingCheckpointState).toBeUndefined(); + }); + + it('overrides the full pending checkpoint cell from a pipelined parent', async () => { + mockGrandparentFeeHeader(); + const proposedData = makeProposedParent(CheckpointNumber(1)); + + const plan = await buildCheckpointSimulationOverridesPlan({ + checkpointNumber: checkpointNumberUnderTest, + proposedCheckpointData: proposedData, + checkpointedCheckpointNumber: CheckpointNumber(0), + rollup, + signatureContext, + log: createLogger('test'), + }); + + expect(plan?.chainTipsOverride?.pending).toEqual(CheckpointNumber(1)); + expect(plan?.chainTipsOverride?.proven).toEqual(CheckpointNumber(1)); + expect(plan?.pendingCheckpointState?.archive).toEqual(proposedData.archive.root); + expect(plan?.pendingCheckpointState?.slotNumber).toEqual(proposedData.header.slotNumber); + expect(plan?.pendingCheckpointState?.headerHash).toEqual(proposedData.header.hash()); + expect(plan?.pendingCheckpointState?.outHash).toEqual(proposedData.checkpointOutHash); + expect(plan?.pendingCheckpointState?.payloadDigest).toBeDefined(); + expect(plan?.pendingCheckpointState?.feeHeader).toBeDefined(); + }); + + it('throws when the pipelined parent does not match the expected parent checkpoint', async () => { + const proposedData = makeProposedParent(CheckpointNumber(5)); + + await expect( + buildCheckpointSimulationOverridesPlan({ + checkpointNumber: checkpointNumberUnderTest, + proposedCheckpointData: proposedData, + checkpointedCheckpointNumber: CheckpointNumber(0), + rollup, + signatureContext, + log: createLogger('test'), + }), + ).rejects.toThrow(/does not match expected parent/); + }); + + it('throws when both proposedCheckpointData and invalidateToPendingCheckpointNumber are provided', async () => { + const proposedData = makeProposedParent(CheckpointNumber(1)); + + await expect( + buildCheckpointSimulationOverridesPlan({ + checkpointNumber: checkpointNumberUnderTest, + proposedCheckpointData: proposedData, + invalidateToPendingCheckpointNumber: CheckpointNumber(0), + checkpointedCheckpointNumber: CheckpointNumber(0), + rollup, + signatureContext, + log: createLogger('test'), + }), + ).rejects.toThrow(/mutually exclusive/); + }); + + it('sets pending and proven from an invalidation rollback without archive/fee overrides', async () => { + const plan = await buildCheckpointSimulationOverridesPlan({ + checkpointNumber: checkpointNumberUnderTest, + invalidateToPendingCheckpointNumber: CheckpointNumber(0), + checkpointedCheckpointNumber: CheckpointNumber(2), + rollup, + signatureContext, + log: createLogger('test'), + }); + expect(plan?.chainTipsOverride?.pending).toEqual(CheckpointNumber(0)); + expect(plan?.chainTipsOverride?.proven).toEqual(CheckpointNumber(0)); + expect(plan?.pendingCheckpointState).toBeUndefined(); + }); +}); diff --git a/yarn-project/sequencer-client/src/sequencer/chain_state_overrides.ts b/yarn-project/stdlib/src/checkpoint/simulation_overrides.ts similarity index 97% rename from yarn-project/sequencer-client/src/sequencer/chain_state_overrides.ts rename to yarn-project/stdlib/src/checkpoint/simulation_overrides.ts index 0230fa90559b..937413b1d518 100644 --- a/yarn-project/sequencer-client/src/sequencer/chain_state_overrides.ts +++ b/yarn-project/stdlib/src/checkpoint/simulation_overrides.ts @@ -6,8 +6,10 @@ import { } from '@aztec/ethereum/contracts'; import { CheckpointNumber } from '@aztec/foundation/branded-types'; import type { Logger } from '@aztec/foundation/log'; -import { type ProposedCheckpointData, computeCheckpointPayloadDigest } from '@aztec/stdlib/checkpoint'; -import type { CoordinationSignatureContext } from '@aztec/stdlib/p2p'; + +import type { CoordinationSignatureContext } from '../p2p/signature_utils.js'; +import type { ProposedCheckpointData } from './checkpoint_data.js'; +import { computeCheckpointPayloadDigest } from './digest.js'; type CheckpointSimulationOverridesPlanInput = { /** Target rollup contract. */ diff --git a/yarn-project/stdlib/src/tx/global_variable_builder.ts b/yarn-project/stdlib/src/tx/global_variable_builder.ts index ae585fe748dd..5bb7846ec83e 100644 --- a/yarn-project/stdlib/src/tx/global_variable_builder.ts +++ b/yarn-project/stdlib/src/tx/global_variable_builder.ts @@ -3,28 +3,12 @@ import type { EthAddress } from '@aztec/foundation/eth-address'; import type { SlotNumber } from '@aztec/foundation/schemas'; import type { AztecAddress } from '../aztec-address/index.js'; -import type { UInt32 } from '../types/index.js'; -import type { CheckpointGlobalVariables, GlobalVariables } from './global_variables.js'; +import type { CheckpointGlobalVariables } from './global_variables.js'; /** * Interface for building global variables for Aztec blocks. */ export interface GlobalVariableBuilder { - /** - * Builds global variables for a given block. - * @param blockNumber - The block number to build global variables for. - * @param coinbase - The address to receive block reward. - * @param feeRecipient - The address to receive fees. - * @param slotNumber - Optional. The slot number to use for the global variables. If undefined, it will be calculated. - * @returns A promise that resolves to the GlobalVariables for the given block number. - */ - buildGlobalVariables( - blockNumber: UInt32, - coinbase: EthAddress, - feeRecipient: AztecAddress, - slotNumber?: SlotNumber, - ): Promise; - /** Builds global variables that are constant throughout a checkpoint. */ buildCheckpointGlobalVariables( coinbase: EthAddress, diff --git a/yarn-project/txe/src/state_machine/global_variable_builder.ts b/yarn-project/txe/src/state_machine/global_variable_builder.ts index 03b904837064..c722168d5649 100644 --- a/yarn-project/txe/src/state_machine/global_variable_builder.ts +++ b/yarn-project/txe/src/state_machine/global_variable_builder.ts @@ -1,16 +1,11 @@ import type { SimulationOverridesPlan } from '@aztec/ethereum/contracts'; -import { BlockNumber, type SlotNumber } from '@aztec/foundation/branded-types'; +import type { SlotNumber } from '@aztec/foundation/branded-types'; import { times } from '@aztec/foundation/collection'; import type { EthAddress } from '@aztec/foundation/eth-address'; import type { AztecAddress } from '@aztec/stdlib/aztec-address'; import { FEE_ORACLE_LAG, GasFees } from '@aztec/stdlib/gas'; import { makeGlobalVariables } from '@aztec/stdlib/testing'; -import { - type CheckpointGlobalVariables, - type FeeProvider, - type GlobalVariableBuilder, - GlobalVariables, -} from '@aztec/stdlib/tx'; +import type { CheckpointGlobalVariables, FeeProvider, GlobalVariableBuilder } from '@aztec/stdlib/tx'; /** Simple FeeProvider for TXE that returns zero fees. */ export class TXEFeeProvider implements FeeProvider { @@ -24,15 +19,6 @@ export class TXEFeeProvider implements FeeProvider { } export class TXEGlobalVariablesBuilder implements GlobalVariableBuilder { - public buildGlobalVariables( - _blockNumber: BlockNumber, - _coinbase: EthAddress, - _feeRecipient: AztecAddress, - _slotNumber?: SlotNumber, - ): Promise { - return Promise.resolve(makeGlobalVariables()); - } - public buildCheckpointGlobalVariables( _coinbase: EthAddress, _feeRecipient: AztecAddress, diff --git a/yarn-project/txe/src/state_machine/index.ts b/yarn-project/txe/src/state_machine/index.ts index 7d3bd75e23c0..38d17b08a02e 100644 --- a/yarn-project/txe/src/state_machine/index.ts +++ b/yarn-project/txe/src/state_machine/index.ts @@ -58,6 +58,7 @@ export class TXEStateMachine { VERSION, CHAIN_ID, new TXEGlobalVariablesBuilder(), + undefined, new TXEFeeProvider(), new MockEpochCache(), PACKAGE_VERSION, From d979a26bc060068ecc33c0b1d88a8dac7e941bef Mon Sep 17 00:00:00 2001 From: Aztec Bot <49558828+AztecBot@users.noreply.github.com> Date: Fri, 12 Jun 2026 18:04:05 -0400 Subject: [PATCH 06/14] refactor: move registerContractFunctionSignatures to the node debug API (#24066) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fixes A-1220 ## Why `node.registerContractFunctionSignatures` was an always-on, unauthenticated write on the main `AztecNode` JSON-RPC surface. It populates a per-node, in-memory, non-gossiped selector→name map used only to name public-execution stack traces, so it has no business on a prod node's public API (anyone can spam it up to the cap, and you'd have to keep talking to the same node for it to mean anything). Decoding traces should be a client concern. ## What Relocate the method from the always-on `AztecNode` interface to the existing, gated `AztecNodeDebug` interface (served under the `nodeDebug_` namespace, only mounted when the node runs with debug endpoints enabled — always on in the in-process sandbox, off by default on standalone nodes). The node-side read path (`getDebugFunctionName` → AVM trace naming) is unchanged; only the RPC write moves. The PXE now calls it through an **optional** `nodeDebug` client and skips registration entirely when the node does not expose the debug API. PXE-side simulation errors keep named traces regardless, via the existing ABI-based enrichment in `pxe/src/error_enriching.ts`. - **stdlib**: removed `registerContractFunctionSignatures` from `AztecNode` + `AztecNodeApiSchema`; added it (and the length/count caps) to `AztecNodeDebug` + `AztecNodeDebugApiSchema`. - **aztec-node**: server impl unchanged — `AztecNodeService` already implements `AztecNodeDebug`, so the method now satisfies that interface and is served under `nodeDebug_`. - **pxe**: optional `nodeDebug?: AztecNodeDebug` on `PXECreateArgs`/constructor; both registration call sites become `this.nodeDebug?.registerContractFunctionSignatures(...)`. Threaded `options.nodeDebug` through all three `createPXE` entrypoints (option A — explicit injection). - **e2e**: wired `nodeDebug` in `end-to-end/src/fixtures/setup.ts` for both PXE-creation paths — `setup()` passes the in-process node service directly; `setupPXEAndGetWallet()` passes it via an `isAztecNodeDebug` type guard (one caller types its node loosely). - **tests**: moved the schema round-trip case from the node interface test to the debug interface test; retargeted the PXE registration tests to the `nodeDebug` mock; updated the `node_rpc_perf` benchmark. - **docs**: dropped the moved method from the generated node API reference (the generator only documents `node_`/`nodeAdmin_`), and added a migration note. ## Breaking change `node_registerContractFunctionSignatures` is removed from the main node RPC. Callers must use `nodeDebug_registerContractFunctionSignatures` against a debug-enabled node. See the migration note. ## Verification Local build could not be run here: the workspace is a cold checkout (noir submodule packages absent, no Rust/Docker), so `yarn install`/`yarn build` cannot bootstrap. The change is a contained TypeScript interface relocation + optional-client wiring; relying on CI for the full build/lint/test. A follow-up could wire `createAztecNodeDebugClient(nodeUrl)` into the URL-based PXE creators (bot/embedded wallet) if named traces are wanted against a debug-enabled remote node. --- *Created by [claudebox](https://claudebox.work/v2/sessions/b8d3c4fab6c0225a) · group: `slackbot`* --- .../docs/resources/migration_notes.md | 6 ++++++ .../generate_node_api_reference.ts | 2 +- .../end-to-end/src/bench/node_rpc_perf.test.ts | 3 ++- yarn-project/end-to-end/src/e2e_2_pxes.test.ts | 7 ++++--- yarn-project/end-to-end/src/e2e_synching.test.ts | 6 +++--- .../end-to-end/src/fixtures/e2e_prover_test.ts | 1 + yarn-project/end-to-end/src/fixtures/setup.ts | 10 +++++++++- .../pxe/src/entrypoints/client/bundle/utils.ts | 1 + .../pxe/src/entrypoints/client/lazy/utils.ts | 1 + .../pxe/src/entrypoints/pxe_creation_options.ts | 8 +++++++- yarn-project/pxe/src/entrypoints/server/utils.ts | 1 + yarn-project/pxe/src/pxe.test.ts | 13 ++++++++----- yarn-project/pxe/src/pxe.ts | 14 +++++++++++--- .../src/interfaces/aztec-node-debug.test.ts | 8 ++++++++ .../stdlib/src/interfaces/aztec-node-debug.ts | 15 +++++++++++++++ .../stdlib/src/interfaces/aztec-node.test.ts | 7 ------- yarn-project/stdlib/src/interfaces/aztec-node.ts | 14 -------------- 17 files changed, 78 insertions(+), 39 deletions(-) diff --git a/docs/docs-developers/docs/resources/migration_notes.md b/docs/docs-developers/docs/resources/migration_notes.md index da3a5640ec19..08bbdf23d881 100644 --- a/docs/docs-developers/docs/resources/migration_notes.md +++ b/docs/docs-developers/docs/resources/migration_notes.md @@ -93,6 +93,12 @@ All Aztec node JSON-RPC method prefixes have changed: If you call the node RPC directly (e.g. via `curl` or a custom client), update all method names accordingly. Clients created via `createAztecNodeClient`, `createAztecNodeAdminClient`, and `createAztecNodeDebugClient` are updated automatically. +### [Node RPC] `registerContractFunctionSignatures` moved to the debug API + +`registerContractFunctionSignatures` is no longer part of the main node JSON-RPC API (`aztec_` namespace). It is now a debug-only method exposed under the `aztecDebug_` namespace, which is only mounted when the node runs with debug endpoints enabled (`--node-debug`, always on in the in-process sandbox). This removes an unauthenticated write to node memory from prod-like nodes. + +Clients that registered public function signatures over `aztec_registerContractFunctionSignatures` should call `aztecDebug_registerContractFunctionSignatures` against a debug-enabled node instead. In the PXE, this is now driven by an optional debug client: pass a `nodeDebug` client (e.g. `createAztecNodeDebugClient(nodeUrl)`) when creating the PXE to keep named public-execution traces; when it is absent, signature registration is skipped. Client-side error enrichment from the contract ABI is unaffected. + ### [Aztec.nr] `get_pending_tagged_logs` oracle interface updated (oracle version 28) The `aztec_utl_getPendingTaggedLogs` oracle now takes an additional `provided_secrets` parameter of type `EphemeralArray`. This lets apps pass tagging secrets that PXE cannot derive on its own (e.g. handshake-derived secrets) alongside the secrets PXE manages internally. diff --git a/docs/scripts/node_api_reference_generation/generate_node_api_reference.ts b/docs/scripts/node_api_reference_generation/generate_node_api_reference.ts index 1b116f8dba1c..73ad56bbcdf1 100644 --- a/docs/scripts/node_api_reference_generation/generate_node_api_reference.ts +++ b/docs/scripts/node_api_reference_generation/generate_node_api_reference.ts @@ -606,7 +606,7 @@ const METHOD_GROUPS: { heading: string; namespace: string; methods: string[] }[] { heading: 'Debug operations', namespace: 'aztec', - methods: ['registerContractFunctionSignatures', 'getAllowedPublicSetup'], + methods: ['getAllowedPublicSetup'], }, { heading: 'Admin API', diff --git a/yarn-project/end-to-end/src/bench/node_rpc_perf.test.ts b/yarn-project/end-to-end/src/bench/node_rpc_perf.test.ts index c2186f40ec23..5509513ec15f 100644 --- a/yarn-project/end-to-end/src/bench/node_rpc_perf.test.ts +++ b/yarn-project/end-to-end/src/bench/node_rpc_perf.test.ts @@ -15,6 +15,7 @@ import { BlockNumber, EpochNumber } from '@aztec/foundation/branded-types'; import { Timer } from '@aztec/foundation/timer'; import { TokenContract } from '@aztec/noir-contracts.js/Token'; import { BlockHash } from '@aztec/stdlib/block'; +import type { AztecNodeDebug } from '@aztec/stdlib/interfaces/client'; import { SiloedTag, Tag } from '@aztec/stdlib/logs'; import { MerkleTreeId } from '@aztec/stdlib/trees'; import type { Tx, TxHash } from '@aztec/stdlib/tx'; @@ -98,7 +99,7 @@ describe('e2e_node_rpc_perf', () => { jest.setTimeout(10 * 60 * 1000); // 10 minutes let logger: Logger; - let aztecNode: AztecNode; + let aztecNode: AztecNode & AztecNodeDebug; let wallet: TestWallet; let ownerAddress: AztecAddress; let rollupCheatCodes: RollupCheatCodes; diff --git a/yarn-project/end-to-end/src/e2e_2_pxes.test.ts b/yarn-project/end-to-end/src/e2e_2_pxes.test.ts index 22274e82c59f..53a8c0757b37 100644 --- a/yarn-project/end-to-end/src/e2e_2_pxes.test.ts +++ b/yarn-project/end-to-end/src/e2e_2_pxes.test.ts @@ -6,6 +6,7 @@ import type { Logger } from '@aztec/aztec.js/log'; import type { AztecNode } from '@aztec/aztec.js/node'; import { TokenContract } from '@aztec/noir-contracts.js/Token'; import { ChildContract } from '@aztec/noir-test-contracts.js/Child'; +import type { AztecNodeDebug } from '@aztec/stdlib/interfaces/client'; import { expect, jest } from '@jest/globals'; @@ -19,7 +20,7 @@ const TIMEOUT = 300_000; describe('e2e_2_pxes', () => { jest.setTimeout(TIMEOUT); - let aztecNode: AztecNode; + let aztecNode: AztecNode & AztecNodeDebug; let walletA: TestWallet; let walletB: TestWallet; let accountAAddress: AztecAddress; @@ -30,12 +31,12 @@ describe('e2e_2_pxes', () => { let teardownB: () => Promise; async function setupSecondaryPXE( - node: AztecNode, + node: AztecNode & AztecNodeDebug, fundedAccounts: InitialAccountData[], accountIndex: number, pxeName: string, ) { - const { wallet, teardown } = await setupPXEAndGetWallet(node, {}, undefined, pxeName); + const { wallet, teardown } = await setupPXEAndGetWallet(node, node, {}, undefined, pxeName); const accountManager = await wallet.createSchnorrAccount( fundedAccounts[accountIndex].secret, fundedAccounts[accountIndex].salt, diff --git a/yarn-project/end-to-end/src/e2e_synching.test.ts b/yarn-project/end-to-end/src/e2e_synching.test.ts index 7dd54f7d3675..56b0231edf31 100644 --- a/yarn-project/end-to-end/src/e2e_synching.test.ts +++ b/yarn-project/end-to-end/src/e2e_synching.test.ts @@ -559,7 +559,7 @@ describe('e2e_synching', () => { const aztecNode = await AztecNodeService.createAndSync(opts.config!, {}, { genesis: opts.genesis }); const sequencer = aztecNode.getSequencer(); - const { wallet } = await setupPXEAndGetWallet(aztecNode!); + const { wallet } = await setupPXEAndGetWallet(aztecNode!, aztecNode!); variant.setWallet(wallet); const defaultAccountAddress = (await variant.deployAccounts(opts.initialFundedAccounts!.slice(0, 1)))[0]; @@ -700,7 +700,7 @@ describe('e2e_synching', () => { expect(await aztecNode.getBlockNumber()).toBeLessThan(blockBeforePrune); // We need to start the pxe after the re-org for now, because it won't handle it otherwise - const { wallet } = await setupPXEAndGetWallet(aztecNode!); + const { wallet } = await setupPXEAndGetWallet(aztecNode!, aztecNode!); variant.setWallet(wallet); const blockBefore = await aztecNode.getBlock(await aztecNode.getBlockNumber()); @@ -751,7 +751,7 @@ describe('e2e_synching', () => { const aztecNode = await AztecNodeService.createAndSync(opts.config!, {}, { genesis: opts.genesis }); const sequencer = aztecNode.getSequencer(); - const { wallet: newWallet } = await setupPXEAndGetWallet(aztecNode!); + const { wallet: newWallet } = await setupPXEAndGetWallet(aztecNode!, aztecNode!); variant.setWallet(newWallet); const blockBefore = await aztecNode.getBlock(await aztecNode.getBlockNumber()); diff --git a/yarn-project/end-to-end/src/fixtures/e2e_prover_test.ts b/yarn-project/end-to-end/src/fixtures/e2e_prover_test.ts index 05a6cb46064f..5ca51d41e32e 100644 --- a/yarn-project/end-to-end/src/fixtures/e2e_prover_test.ts +++ b/yarn-project/end-to-end/src/fixtures/e2e_prover_test.ts @@ -186,6 +186,7 @@ export class FullProverTest { this.logger.verbose(`Main setup completed, initializing full prover PXE, Node, and Prover Node`); const { wallet: provenWallet, teardown: provenTeardown } = await setupPXEAndGetWallet( this.aztecNode, + this.context.aztecNode, { proverEnabled: this.realProofs }, undefined, 'pxe-proven', diff --git a/yarn-project/end-to-end/src/fixtures/setup.ts b/yarn-project/end-to-end/src/fixtures/setup.ts index 9f67afe9c33d..a17bc2f352b0 100644 --- a/yarn-project/end-to-end/src/fixtures/setup.ts +++ b/yarn-project/end-to-end/src/fixtures/setup.ts @@ -119,6 +119,8 @@ export async function setupSharedBlobStorage(config: { dataDirectory?: string } /** * Sets up Private eXecution Environment (PXE) and returns the corresponding test wallet. * @param aztecNode - An instance of Aztec Node. + * @param nodeDebug - The node's debug API, used to register public function signatures for named traces; pass + * `undefined` when the node does not expose it. * @param opts - Partial configuration for the PXE. * @param logger - The logger to be used. * @param actor - Actor label to include in log output (e.g., 'pxe-test'). @@ -126,6 +128,7 @@ export async function setupSharedBlobStorage(config: { dataDirectory?: string } */ export async function setupPXEAndGetWallet( aztecNode: AztecNode, + nodeDebug: AztecNodeDebug | undefined, opts: Partial = {}, logger = getLogger(), actor?: string, @@ -146,7 +149,10 @@ export async function setupPXEAndGetWallet( const teardown = configuredDataDirectory ? () => Promise.resolve() : () => tryRmDir(PXEConfig.dataDirectory!); - const wallet = await TestWallet.create(aztecNode, PXEConfig, { loggerActorLabel: actor }); + const wallet = await TestWallet.create(aztecNode, PXEConfig, { + loggerActorLabel: actor, + nodeDebug, + }); return { wallet, @@ -609,6 +615,8 @@ export async function setup( pxeConfig.proverEnabled = !!pxeOpts.proverEnabled; const wallet = await TestWallet.create(aztecNodeService, pxeConfig, { loggerActorLabel: 'pxe-0', + // In-process node implements the debug API, so register public function signatures for named traces. + nodeDebug: aztecNodeService, ...opts.pxeCreationOptions, }); diff --git a/yarn-project/pxe/src/entrypoints/client/bundle/utils.ts b/yarn-project/pxe/src/entrypoints/client/bundle/utils.ts index 32626ee5bd13..52eba71c4192 100644 --- a/yarn-project/pxe/src/entrypoints/client/bundle/utils.ts +++ b/yarn-project/pxe/src/entrypoints/client/bundle/utils.ts @@ -63,6 +63,7 @@ export async function createPXE( const pxeLogger = loggers.pxe ?? createLogger('pxe:service', { actor }); const pxe = await PXE.create({ node: aztecNode, + nodeDebug: options.nodeDebug, store, proofCreator: prover, simulator, diff --git a/yarn-project/pxe/src/entrypoints/client/lazy/utils.ts b/yarn-project/pxe/src/entrypoints/client/lazy/utils.ts index d4e7853a6b1b..6fcaa4001df3 100644 --- a/yarn-project/pxe/src/entrypoints/client/lazy/utils.ts +++ b/yarn-project/pxe/src/entrypoints/client/lazy/utils.ts @@ -63,6 +63,7 @@ export async function createPXE( const pxeLogger = loggers.pxe ?? createLogger('pxe:service', { actor }); const pxe = await PXE.create({ node: aztecNode, + nodeDebug: options.nodeDebug, store, proofCreator: prover, simulator, diff --git a/yarn-project/pxe/src/entrypoints/pxe_creation_options.ts b/yarn-project/pxe/src/entrypoints/pxe_creation_options.ts index 189db991a14f..954901916dfc 100644 --- a/yarn-project/pxe/src/entrypoints/pxe_creation_options.ts +++ b/yarn-project/pxe/src/entrypoints/pxe_creation_options.ts @@ -2,7 +2,7 @@ import type { BBPrivateKernelProverOptions } from '@aztec/bb-prover/client'; import type { Logger } from '@aztec/foundation/log'; import type { AztecAsyncKVStore } from '@aztec/kv-store'; import type { CircuitSimulator } from '@aztec/simulator/client'; -import type { PrivateKernelProver } from '@aztec/stdlib/interfaces/client'; +import type { AztecNodeDebug, PrivateKernelProver } from '@aztec/stdlib/interfaces/client'; import type { ExecutionHooks } from '../hooks/index.js'; import type { PreloadedContractsProvider } from '../pxe.js'; @@ -18,6 +18,12 @@ export type PXECreationOptions = { hooks?: ExecutionHooks; /** Contracts to preload on startup. Replaces the default when set. */ preloadedContractsProvider?: PreloadedContractsProvider; + /** + * Optional debug API client for the same node. When provided, public function signatures are registered so the + * node can produce named public-execution stack traces. Only available on nodes started with debug endpoints + * enabled; omitted in prod-like setups. + */ + nodeDebug?: AztecNodeDebug; }; /** Checks if the given value implements the PrivateKernelProver interface via duck-typing. */ diff --git a/yarn-project/pxe/src/entrypoints/server/utils.ts b/yarn-project/pxe/src/entrypoints/server/utils.ts index 13a78bbaa284..04ca4d6183c4 100644 --- a/yarn-project/pxe/src/entrypoints/server/utils.ts +++ b/yarn-project/pxe/src/entrypoints/server/utils.ts @@ -73,6 +73,7 @@ export async function createPXE( const pxeLogger = loggers.pxe ?? createLogger('pxe:service', { actor }); const pxe = await PXE.create({ node: aztecNode, + nodeDebug: options.nodeDebug, store: options.store, proofCreator: prover, simulator, diff --git a/yarn-project/pxe/src/pxe.test.ts b/yarn-project/pxe/src/pxe.test.ts index 15dd843acea1..fad92c90669d 100644 --- a/yarn-project/pxe/src/pxe.test.ts +++ b/yarn-project/pxe/src/pxe.test.ts @@ -23,7 +23,7 @@ import { } from '@aztec/stdlib/block'; import { emptyChainConfig } from '@aztec/stdlib/config'; import { getContractClassFromArtifact } from '@aztec/stdlib/contract'; -import type { AztecNode, BlockResponse } from '@aztec/stdlib/interfaces/client'; +import type { AztecNode, AztecNodeDebug, BlockResponse } from '@aztec/stdlib/interfaces/client'; import { randomContractArtifact, randomContractInstanceWithAddress, @@ -43,10 +43,12 @@ describe('PXE', () => { let pxe: PXE; let kvStore: AztecLMDBStoreV2; let node: MockProxy; + let nodeDebug: MockProxy; beforeAll(async () => { kvStore = await openTmpStore('test'); node = mock(); + nodeDebug = mock(); const simulator = new WASMSimulator(); const kernelProver = new BBBundlePrivateKernelProver(simulator); const protocolContractsProvider = new BundledProtocolContractsProvider(); @@ -100,6 +102,7 @@ describe('PXE', () => { pxe = await PXE.create({ node, + nodeDebug, store: kvStore, proofCreator: kernelProver, simulator, @@ -202,11 +205,11 @@ describe('PXE', () => { it('does not call registerContractFunctionSignatures for contracts without public functions', async () => { const { artifact, instance } = await randomDeployedContract(); - node.registerContractFunctionSignatures.mockClear(); + nodeDebug.registerContractFunctionSignatures.mockClear(); await pxe.registerContract({ artifact, instance }); - expect(node.registerContractFunctionSignatures).not.toHaveBeenCalled(); + expect(nodeDebug.registerContractFunctionSignatures).not.toHaveBeenCalled(); }); it('calls registerContractFunctionSignatures for contracts with public functions', async () => { @@ -227,11 +230,11 @@ describe('PXE', () => { ]; const contractClass = await getContractClassFromArtifact(artifact); const instance = await randomContractInstanceWithAddress({ contractClassId: contractClass.id }); - node.registerContractFunctionSignatures.mockClear(); + nodeDebug.registerContractFunctionSignatures.mockClear(); await pxe.registerContract({ artifact, instance }); - expect(node.registerContractFunctionSignatures).toHaveBeenCalledWith(['my_public_fn()']); + expect(nodeDebug.registerContractFunctionSignatures).toHaveBeenCalledWith(['my_public_fn()']); }); // These tests are meant to quickly exercise PXE as a frontier API so we don't need to rely on slower E2E tests diff --git a/yarn-project/pxe/src/pxe.ts b/yarn-project/pxe/src/pxe.ts index 299b9f6b3cd5..154ec7e73a78 100644 --- a/yarn-project/pxe/src/pxe.ts +++ b/yarn-project/pxe/src/pxe.ts @@ -28,7 +28,7 @@ import { getContractClassFromArtifact, } from '@aztec/stdlib/contract'; import { SimulationError } from '@aztec/stdlib/errors'; -import type { AztecNode, PrivateKernelProver } from '@aztec/stdlib/interfaces/client'; +import type { AztecNode, AztecNodeDebug, PrivateKernelProver } from '@aztec/stdlib/interfaces/client'; import type { PrivateExecutionStep, PrivateKernelExecutionProofOutput, @@ -162,6 +162,11 @@ export type PreloadedContractsProvider = { export type PXECreateArgs = { /** The Aztec node to connect to. */ node: AztecNode; + /** + * Optional debug API client. When provided, public function signatures are registered with the node so that + * node-side public-execution stack traces can be named. Skipped when the node does not expose the debug API. + */ + nodeDebug?: AztecNodeDebug; /** The key-value store for persisting PXE state. */ store: AztecAsyncKVStore; /** The prover for generating private kernel proofs. */ @@ -187,6 +192,7 @@ export type PXECreateArgs = { export class PXE { private constructor( private node: AztecNode, + private nodeDebug: AztecNodeDebug | undefined, private db: AztecAsyncKVStore, private blockStateSynchronizer: BlockSynchronizer, private keyStore: KeyStore, @@ -224,6 +230,7 @@ export class PXE { */ public static async create({ node, + nodeDebug, store, proofCreator, simulator, @@ -302,6 +309,7 @@ export class PXE { const pxe = new PXE( node, + nodeDebug, store, synchronizer, keyStore, @@ -766,7 +774,7 @@ export class PXE { .filter(fn => fn.functionType === FunctionType.PUBLIC) .map(fn => decodeFunctionSignature(fn.name, fn.parameters)); if (publicFunctionSignatures.length > 0) { - await this.node.registerContractFunctionSignatures(publicFunctionSignatures); + await this.nodeDebug?.registerContractFunctionSignatures(publicFunctionSignatures); } } else { // Otherwise, make sure there is an artifact already registered for that class id @@ -815,7 +823,7 @@ export class PXE { .filter(fn => fn.functionType === FunctionType.PUBLIC) .map(fn => decodeFunctionSignature(fn.name, fn.parameters)); if (publicFunctionSignatures.length > 0) { - await this.node.registerContractFunctionSignatures(publicFunctionSignatures); + await this.nodeDebug?.registerContractFunctionSignatures(publicFunctionSignatures); } currentInstance.currentContractClassId = contractClass.id; diff --git a/yarn-project/stdlib/src/interfaces/aztec-node-debug.test.ts b/yarn-project/stdlib/src/interfaces/aztec-node-debug.test.ts index 2355115f8340..79b477526d93 100644 --- a/yarn-project/stdlib/src/interfaces/aztec-node-debug.test.ts +++ b/yarn-project/stdlib/src/interfaces/aztec-node-debug.test.ts @@ -32,6 +32,10 @@ describe('AztecNodeDebugApiSchema', () => { expect(await context.client.prove()).toEqual(7); expect(await context.client.prove(CheckpointNumber(3))).toEqual(3); }); + + it('registerContractFunctionSignatures', async () => { + await context.client.registerContractFunctionSignatures(['test()']); + }); }); class MockAztecNodeDebug implements AztecNodeDebug { @@ -42,4 +46,8 @@ class MockAztecNodeDebug implements AztecNodeDebug { prove(upToCheckpoint?: CheckpointNumber): Promise { return Promise.resolve(upToCheckpoint ?? CheckpointNumber(7)); } + + registerContractFunctionSignatures(_signatures: string[]): Promise { + return Promise.resolve(); + } } diff --git a/yarn-project/stdlib/src/interfaces/aztec-node-debug.ts b/yarn-project/stdlib/src/interfaces/aztec-node-debug.ts index ccaec0f4093c..b4927ce6eaa3 100644 --- a/yarn-project/stdlib/src/interfaces/aztec-node-debug.ts +++ b/yarn-project/stdlib/src/interfaces/aztec-node-debug.ts @@ -6,6 +6,9 @@ import { z } from 'zod'; import { type ApiSchemaFor, optional } from '../schemas/schemas.js'; import { type ComponentsVersions, getVersioningResponseHandler } from '../versioning/index.js'; +const MAX_SIGNATURES_PER_REGISTER_CALL = 100; +const MAX_SIGNATURE_LEN = 10000; + /** * Debug interface for Aztec node available in sandbox/local-network mode. */ @@ -33,11 +36,23 @@ export interface AztecNodeDebug { * @throws If no automine sequencer is running (only the automine sequencer supports synthetic proving). */ prove(upToCheckpoint?: CheckpointNumber): Promise; + + /** + * Registers public function signatures so the node can resolve selectors to names in public-execution stack + * traces. The mapping lives only in unpersisted node memory, is not gossiped, and is exposed here (rather than on + * the main node API) because it is a debug-only, unauthenticated write that should not be reachable on prod nodes. + * @param functionSignatures - Decoded `name(paramTypes)` signatures to register by selector. + */ + registerContractFunctionSignatures(functionSignatures: string[]): Promise; } export const AztecNodeDebugApiSchema: ApiSchemaFor = { mineBlock: z.function({ input: z.tuple([]), output: z.void() }), prove: z.function({ input: z.tuple([optional(CheckpointNumberSchema)]), output: CheckpointNumberSchema }), + registerContractFunctionSignatures: z.function({ + input: z.tuple([z.array(z.string().max(MAX_SIGNATURE_LEN)).max(MAX_SIGNATURES_PER_REGISTER_CALL)]), + output: z.void(), + }), }; export function createAztecNodeDebugClient( diff --git a/yarn-project/stdlib/src/interfaces/aztec-node.test.ts b/yarn-project/stdlib/src/interfaces/aztec-node.test.ts index fb23d9d56458..7e00574f58bd 100644 --- a/yarn-project/stdlib/src/interfaces/aztec-node.test.ts +++ b/yarn-project/stdlib/src/interfaces/aztec-node.test.ts @@ -307,10 +307,6 @@ describe('AztecNodeApiSchema', () => { expect(response).toEqual(Object.fromEntries(ProtocolContractsNames.map(name => [name, expect.any(AztecAddress)]))); }); - it('registerContractFunctionSignatures', async () => { - await context.client.registerContractFunctionSignatures(['test()']); - }); - it('getPrivateLogsByTags', async () => { const response = await context.client.getPrivateLogsByTags({ tags: [SiloedTag.random()] }); expect(response).toHaveLength(1); @@ -813,9 +809,6 @@ class MockAztecNode implements AztecNode { ); return Object.fromEntries(protocolContracts) as ProtocolContractAddresses; } - registerContractFunctionSignatures(_signatures: string[]): Promise { - return Promise.resolve(); - } getPrivateLogsByTags(query: PrivateLogsQuery): Promise { expect(Array.isArray(query.tags)).toBe(true); return Promise.resolve([query.tags.map(() => randomLogResult())]); diff --git a/yarn-project/stdlib/src/interfaces/aztec-node.ts b/yarn-project/stdlib/src/interfaces/aztec-node.ts index 62baf29ecb06..41328fc29545 100644 --- a/yarn-project/stdlib/src/interfaces/aztec-node.ts +++ b/yarn-project/stdlib/src/interfaces/aztec-node.ts @@ -385,12 +385,6 @@ export interface AztecNode { */ getProtocolContractAddresses(): Promise; - /** - * Registers contract function signatures for debugging purposes. - * @param functionSignatures - An array of function signatures to register by selector. - */ - registerContractFunctionSignatures(functionSignatures: string[]): Promise; - /** * Gets private logs matching the given tags. Returns one inner array per element of `query.tags`, in * input order. An empty inner array means no logs matched that tag. Set `query.includeEffects` to also @@ -569,9 +563,6 @@ export interface AztecNode { getProposalsForSlot(slot: SlotNumber): Promise; } -const MAX_SIGNATURES_PER_REGISTER_CALL = 100; -const MAX_SIGNATURE_LEN = 10000; - export const AztecNodeApiSchema: ApiSchemaFor = { getWorldStateSyncStatus: z.function({ input: z.tuple([]), output: WorldStateSyncStatusSchema }), @@ -694,11 +685,6 @@ export const AztecNodeApiSchema: ApiSchemaFor = { getProtocolContractAddresses: z.function({ input: z.tuple([]), output: ProtocolContractAddressesSchema }), - registerContractFunctionSignatures: z.function({ - input: z.tuple([z.array(z.string().max(MAX_SIGNATURE_LEN)).max(MAX_SIGNATURES_PER_REGISTER_CALL)]), - output: z.void(), - }), - getPrivateLogsByTags: z.function({ input: z.tuple([PrivateLogsQuerySchema]), output: z.array(z.array(LogResultSchema)), From f89fa28d8c380ca6d0ec4a8f616a8238be72a2d5 Mon Sep 17 00:00:00 2001 From: Aztec Bot <49558828+AztecBot@users.noreply.github.com> Date: Sun, 14 Jun 2026 13:08:52 -0400 Subject: [PATCH 07/14] =?UTF-8?q?fix:=20resolve=20v5-next=20=E2=86=92=20me?= =?UTF-8?q?rge-train/spartan-v5=20conflict=20(#24072)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Conflict resolution — review this PR Stacked on top of the raw merge PR #24071 (`cb/merge-train-spartan-v5-raw`). This PR is the reviewable diff that resolves the one conflict from merging `v5-next` into `merge-train/spartan-v5`. Land order when green: this PR → raw PR → `merge-train/spartan-v5`. ### Conflicts resolved (1) **`yarn-project/end-to-end/src/bench/client_flows/client_flows_benchmark.ts`** — import-list divergence: - Train side imported `L1_DIRECT_WRITE_ACCOUNT_INDEX` (from `fixtures.js`) and `deployAccounts` (from `setup.js`); v5-next had neither. - **Kept** `L1_DIRECT_WRITE_ACCOUNT_INDEX` — it is exported by `fixtures.ts` (`export const L1_DIRECT_WRITE_ACCOUNT_INDEX = 1`) and used in the file body (passed to the L1 client at the deployer-account index, ~line 255). - **Dropped** `deployAccounts` — it is no longer exported by the merged `setup.ts` (v5-next refactored it away) and is unused anywhere in the file body, so importing it would break the build. No fixtures/generated constants were involved; this is a pure import resolution. Full TS build is validated by CI on this PR. ### Parents - train `merge-train/spartan-v5` @ `d979a26` - base `v5-next` @ `5fd6187` --- *Created by [claudebox](https://claudebox.work/v2/sessions/0e582efb7d4723ed) · group: `slackbot`* --- .../src/bench/client_flows/client_flows_benchmark.ts | 5 ----- 1 file changed, 5 deletions(-) diff --git a/yarn-project/end-to-end/src/bench/client_flows/client_flows_benchmark.ts b/yarn-project/end-to-end/src/bench/client_flows/client_flows_benchmark.ts index 70b1b1189dd9..a2a8ce415f24 100644 --- a/yarn-project/end-to-end/src/bench/client_flows/client_flows_benchmark.ts +++ b/yarn-project/end-to-end/src/bench/client_flows/client_flows_benchmark.ts @@ -28,18 +28,13 @@ import type { ContractInstanceWithAddress } from '@aztec/stdlib/contract'; import { Gas, GasSettings } from '@aztec/stdlib/gas'; import { deriveSigningKey } from '@aztec/stdlib/keys'; -<<<<<<< HEAD import { AUTOMINE_E2E_OPTS, L1_DIRECT_WRITE_ACCOUNT_INDEX, MNEMONIC, getPaddedMaxFeesPerGas, } from '../../fixtures/fixtures.js'; -import { type EndToEndContext, type SetupOptions, deployAccounts, setup, teardown } from '../../fixtures/setup.js'; -======= -import { AUTOMINE_E2E_OPTS, MNEMONIC, getPaddedMaxFeesPerGas } from '../../fixtures/fixtures.js'; import { type EndToEndContext, type SetupOptions, setup, teardown } from '../../fixtures/setup.js'; ->>>>>>> origin/v5-next import { mintTokensToPrivate } from '../../fixtures/token_utils.js'; import { setupSponsoredFPC } from '../../fixtures/utils.js'; import { CrossChainTestHarness } from '../../shared/cross_chain_test_harness.js'; From 103dc95aa3ecb814aa18cd1e871bc46d9fd342fa Mon Sep 17 00:00:00 2001 From: PhilWindle <60546371+PhilWindle@users.noreply.github.com> Date: Mon, 15 Jun 2026 17:10:39 +0100 Subject: [PATCH 08/14] test(ci): mark e2e_epochs/epochs_mbps_redistribution as flaky (#24098) --- .test_patterns.yml | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/.test_patterns.yml b/.test_patterns.yml index 498fc75e22ad..3816c919c8e4 100644 --- a/.test_patterns.yml +++ b/.test_patterns.yml @@ -275,6 +275,13 @@ tests: owners: - *sean + # Asserts all late txs land in the checkpoint's last block, but the proposer seals the last block as + # soon as one tx is available and snapshots the mempool, so late txs still propagating over gossip + # spill into the next checkpoint. Flaky until the sequencer waits for the last block's tx window. + - regex: "src/e2e_epochs/epochs_mbps_redistribution.test.ts" + owners: + - *phil + # Blanket flake patterns for unstable p2p, epoch, and l1 tx utils tests # This is a temporary measure while team bandwidth is constrained # Replaced many specific patterns - see https://github.com/AztecProtocol/aztec-packages/pull/17962 for historical context From 092c30fac947c21f49b7d93fb4bb0c76b929606d Mon Sep 17 00:00:00 2001 From: Aztec Bot <49558828+AztecBot@users.noreply.github.com> Date: Mon, 15 Jun 2026 14:05:05 -0400 Subject: [PATCH 09/14] fix: isolate cross-chain L1 writes from publisher nonce (#24104) ## Summary - Keep the shared cross-chain rollup/inbox/outbox client on the deployer mnemonic account so L2->L1 message consumption signs with the expected recipient. - Let L1-writing cross-chain suites opt their harness into `L1_DIRECT_WRITE_ACCOUNT_INDEX` to avoid nonce races with the sequencer publisher under pipelining. - Apply that dedicated harness account to token bridge and L1->L2 message tests. - Fix Prettier formatting reported by CI. ## Verification - `git diff --check` - `./bootstrap.sh ci` (blocked locally: bootstrap reports `Unknown command: ci`; the container also has no Docker/Redis access) - `yarn-project/./bootstrap.sh format --check` (blocked locally: missing `yarn-project/node_modules/.bin/prettier`) --- .../cross_chain_messaging_test.ts | 19 ++++++++++++++----- .../l1_to_l2.parallel.test.ts | 3 ++- .../token_bridge_failure_cases.test.ts | 4 ++-- .../token_bridge_private.test.ts | 10 ++++++++-- .../token_bridge_public.test.ts | 10 ++++++++-- 5 files changed, 34 insertions(+), 12 deletions(-) diff --git a/yarn-project/end-to-end/src/e2e_cross_chain_messaging/cross_chain_messaging_test.ts b/yarn-project/end-to-end/src/e2e_cross_chain_messaging/cross_chain_messaging_test.ts index a6c22d2ddd79..409e8e430fcc 100644 --- a/yarn-project/end-to-end/src/e2e_cross_chain_messaging/cross_chain_messaging_test.ts +++ b/yarn-project/end-to-end/src/e2e_cross_chain_messaging/cross_chain_messaging_test.ts @@ -40,6 +40,7 @@ export class CrossChainMessagingTest { private setupOptions: SetupOptions; private deployL1ContractsArgs: Partial; private pxeOpts: Partial; + private l1HarnessAccountIndex?: number; logger: Logger; context!: EndToEndContext; aztecNode!: AztecNode; @@ -79,6 +80,7 @@ export class CrossChainMessagingTest { opts: SetupOptions = {}, deployL1ContractsArgs: Partial = {}, pxeOpts: Partial = {}, + l1HarnessAccountIndex?: number, ) { this.logger = createLogger(`e2e:e2e_cross_chain_messaging:${testName}`); this.setupOptions = opts; @@ -87,6 +89,7 @@ export class CrossChainMessagingTest { ...deployL1ContractsArgs, }; this.pxeOpts = pxeOpts; + this.l1HarnessAccountIndex = l1HarnessAccountIndex; this.requireEpochProven = opts.startProverNode ?? false; } @@ -177,18 +180,24 @@ export class CrossChainMessagingTest { await ensureAuthRegistryPublished(this.wallet, this.ownerAddress); - this.l1Client = createExtendedL1Client(this.aztecNodeConfig.l1RpcUrls, MNEMONIC); + const harnessL1Client = createExtendedL1Client( + this.aztecNodeConfig.l1RpcUrls, + MNEMONIC, + undefined, + undefined, + this.l1HarnessAccountIndex, + ); - const underlyingERC20Address = await deployL1Contract(this.l1Client, TestERC20Abi, TestERC20Bytecode, [ + const underlyingERC20Address = await deployL1Contract(harnessL1Client, TestERC20Abi, TestERC20Bytecode, [ 'Underlying', 'UND', - this.l1Client.account.address, + harnessL1Client.account.address, ]).then(({ address }) => address); this.logger.verbose(`Setting up cross chain harness...`); this.crossChainTestHarness = await CrossChainTestHarness.new( this.aztecNode, - this.l1Client, + harnessL1Client, this.wallet, this.ownerAddress, this.logger, @@ -222,7 +231,7 @@ export class CrossChainMessagingTest { this.ethAccount, tokenPortalAddress, crossChainContext.underlying, - l1Client, + harnessL1Client, pickL1ContractAddresses(this.aztecNodeConfig), this.wallet, this.ownerAddress, diff --git a/yarn-project/end-to-end/src/e2e_cross_chain_messaging/l1_to_l2.parallel.test.ts b/yarn-project/end-to-end/src/e2e_cross_chain_messaging/l1_to_l2.parallel.test.ts index 9d00b838a449..ae3a689173cb 100644 --- a/yarn-project/end-to-end/src/e2e_cross_chain_messaging/l1_to_l2.parallel.test.ts +++ b/yarn-project/end-to-end/src/e2e_cross_chain_messaging/l1_to_l2.parallel.test.ts @@ -14,7 +14,7 @@ import { ExecutionPayload } from '@aztec/stdlib/tx'; import { jest } from '@jest/globals'; -import { PIPELINING_SETUP_OPTS } from '../fixtures/fixtures.js'; +import { L1_DIRECT_WRITE_ACCOUNT_INDEX, PIPELINING_SETUP_OPTS } from '../fixtures/fixtures.js'; import { sendL1ToL2Message } from '../fixtures/l1_to_l2_messaging.js'; import type { CrossChainTestHarness } from '../shared/cross_chain_test_harness.js'; import { CrossChainMessagingTest } from './cross_chain_messaging_test.js'; @@ -57,6 +57,7 @@ describe('e2e_cross_chain_messaging l1_to_l2', () => { // (e.g. a missed checkpoint publish that prunes the pipelined proposed chain) doesn't // drop the wallet's in-flight tx via handlePrunedBlocks. { syncChainTip: 'checkpointed' }, + L1_DIRECT_WRITE_ACCOUNT_INDEX, ); await t.setup(); diff --git a/yarn-project/end-to-end/src/e2e_cross_chain_messaging/token_bridge_failure_cases.test.ts b/yarn-project/end-to-end/src/e2e_cross_chain_messaging/token_bridge_failure_cases.test.ts index 13d7837d4133..5ec2e47899ce 100644 --- a/yarn-project/end-to-end/src/e2e_cross_chain_messaging/token_bridge_failure_cases.test.ts +++ b/yarn-project/end-to-end/src/e2e_cross_chain_messaging/token_bridge_failure_cases.test.ts @@ -5,11 +5,11 @@ import { sha256ToField } from '@aztec/foundation/crypto/sha256'; import { toFunctionSelector } from 'viem'; -import { NO_L1_TO_L2_MSG_ERROR, PIPELINING_SETUP_OPTS } from '../fixtures/fixtures.js'; +import { L1_DIRECT_WRITE_ACCOUNT_INDEX, NO_L1_TO_L2_MSG_ERROR, PIPELINING_SETUP_OPTS } from '../fixtures/fixtures.js'; import { CrossChainMessagingTest } from './cross_chain_messaging_test.js'; describe('e2e_cross_chain_messaging token_bridge_failure_cases', () => { - const t = new CrossChainMessagingTest('token_bridge_failure_cases'); + const t = new CrossChainMessagingTest('token_bridge_failure_cases', {}, {}, {}, L1_DIRECT_WRITE_ACCOUNT_INDEX); let version: number = 1; let { crossChainTestHarness, ethAccount, l2Bridge, ownerAddress, user1Address, user2Address, rollup } = t; diff --git a/yarn-project/end-to-end/src/e2e_cross_chain_messaging/token_bridge_private.test.ts b/yarn-project/end-to-end/src/e2e_cross_chain_messaging/token_bridge_private.test.ts index d7fa4aebcd38..150779b6b6e7 100644 --- a/yarn-project/end-to-end/src/e2e_cross_chain_messaging/token_bridge_private.test.ts +++ b/yarn-project/end-to-end/src/e2e_cross_chain_messaging/token_bridge_private.test.ts @@ -8,7 +8,7 @@ import type { TokenBridgeContract } from '@aztec/noir-contracts.js/TokenBridge'; import { jest } from '@jest/globals'; -import { PIPELINING_SETUP_OPTS } from '../fixtures/fixtures.js'; +import { L1_DIRECT_WRITE_ACCOUNT_INDEX, PIPELINING_SETUP_OPTS } from '../fixtures/fixtures.js'; import type { CrossChainTestHarness } from '../shared/cross_chain_test_harness.js'; import type { TestWallet } from '../test-wallet/test_wallet.js'; import { CrossChainMessagingTest } from './cross_chain_messaging_test.js'; @@ -18,7 +18,13 @@ describe('e2e_cross_chain_messaging token_bridge_private', () => { // needs more than the default 300s per-test budget. jest.setTimeout(15 * 60 * 1000); - const t = new CrossChainMessagingTest('token_bridge_private', { startProverNode: true }); + const t = new CrossChainMessagingTest( + 'token_bridge_private', + { startProverNode: true }, + {}, + {}, + L1_DIRECT_WRITE_ACCOUNT_INDEX, + ); let crossChainTestHarness: CrossChainTestHarness; let ethAccount: EthAddress; diff --git a/yarn-project/end-to-end/src/e2e_cross_chain_messaging/token_bridge_public.test.ts b/yarn-project/end-to-end/src/e2e_cross_chain_messaging/token_bridge_public.test.ts index 6a08a69f112e..86f0ad96b336 100644 --- a/yarn-project/end-to-end/src/e2e_cross_chain_messaging/token_bridge_public.test.ts +++ b/yarn-project/end-to-end/src/e2e_cross_chain_messaging/token_bridge_public.test.ts @@ -3,7 +3,7 @@ import { retryUntil } from '@aztec/foundation/retry'; import { jest } from '@jest/globals'; -import { NO_L1_TO_L2_MSG_ERROR, PIPELINING_SETUP_OPTS } from '../fixtures/fixtures.js'; +import { L1_DIRECT_WRITE_ACCOUNT_INDEX, NO_L1_TO_L2_MSG_ERROR, PIPELINING_SETUP_OPTS } from '../fixtures/fixtures.js'; import { CrossChainMessagingTest } from './cross_chain_messaging_test.js'; describe('e2e_cross_chain_messaging token_bridge_public', () => { @@ -11,7 +11,13 @@ describe('e2e_cross_chain_messaging token_bridge_public', () => { // needs more than the default 300s per-test budget. jest.setTimeout(15 * 60 * 1000); - const t = new CrossChainMessagingTest('token_bridge_public', { startProverNode: true }); + const t = new CrossChainMessagingTest( + 'token_bridge_public', + { startProverNode: true }, + {}, + {}, + L1_DIRECT_WRITE_ACCOUNT_INDEX, + ); let { crossChainTestHarness, ethAccount, aztecNode, logger, ownerAddress, l2Bridge, l2Token, wallet, user2Address } = t; From b5830eeed7fb4489618dac0b9727f66ce931f1f2 Mon Sep 17 00:00:00 2001 From: Aztec Bot <49558828+AztecBot@users.noreply.github.com> Date: Mon, 15 Jun 2026 16:19:09 -0400 Subject: [PATCH 10/14] test(world-state): make delayed-close fork queue-cleanup wait deterministic (#24106) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Why the merge train was dequeued PR #24053 (`merge-train/spartan-v5` → `v5-next`) was dequeued from the merge queue because its `merge-queue-heavy` CI run failed. Tracing the failure: - CI3 merge_group run `27568575541` → shard `x4-full` (`ci-full-no-test-cache`) failed. - The failing test was `yarn-project/world-state/src/native/native_world_state.test.ts` (`code: 1`), specifically: > `NativeWorldState › Pending and Proven chain › does not fail when a delayed-close fork is destroyed by a reorg before its close fires` - Failing assertion (`native_world_state.test.ts:966`): ``` expect((ws as any).instance.queues.has(forkId)).toBe(false); // Expected: false, Received: true ``` The `warnSpy` assertion on the preceding line passed; only the queue-cleanup assertion failed. This test and the world-state instance/facade code it exercises are **not modified by this train** — the test already exists unchanged in `v5-next`. So this is a **pre-existing timing flake** that happened to fail in this grind and blocked the train. ## Root cause The delayed-close path is correct and always eventually cleans up: - `asyncDispose()` schedules `sleep(closeDelayMs).then(() => close())` (fire-and-forget). - `close()` → `doClose()` → `instance.call(DELETE_FORK)`, and the per-fork queue entry is deleted in `call()`'s `finally` regardless of whether the native side rejects with `Fork not found` (which it does here, because the reorg/unwind already destroyed the native fork). The test then waited a **fixed** `sleep(closeDelayMs * 3)` (3s) before asserting the queue was gone. Under a loaded merge-queue grind, the scheduled close (fired at +`closeDelayMs`) plus its async `DELETE_FORK` round-trip and `queue.stop()` can take longer than that fixed margin, so the queue entry was still present when the assertion ran — Received `true`. ## Fix Replace the racy fixed sleep with a deterministic wait on the exact asserted condition: ```ts await retryUntil(() => !(ws as any).instance.queues.has(forkId), 'delayed-close fork queue cleanup', 30, 0.1); ``` This polls until the queue entry is removed (the cleanup runs inside the same close path), with a 30s timeout. It preserves the regression intent: if cleanup genuinely stopped happening, `retryUntil` throws a clear `Timeout awaiting delayed-close fork queue cleanup` instead of a bare `toBe(false)`. The production code is unchanged; no test is skipped or weakened. The now-unused `sleep` import is swapped for `retryUntil`. ## Verification This is a test-only, single-statement timing change. I was unable to run `./bootstrap.sh ci` in this environment — the workspace has no `node_modules` and no barretenberg/native `world_state_napi` build artifacts, so exercising this native test would require a multi-hour C++ + TS build that isn't proportionate to a wait-deterministically change. The change is type-checked by inspection: `retryUntil(fn, name, timeoutSec, intervalSec)` matches the foundation signature, and the import path (`@aztec/foundation/retry`) is the one used across yarn-project. CI on this PR will validate it. --- *Created by [claudebox](https://claudebox.work/v2/sessions/6fb41d03fbbd7aaa) · group: `slackbot`* --- .../world-state/src/native/native_world_state.test.ts | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/yarn-project/world-state/src/native/native_world_state.test.ts b/yarn-project/world-state/src/native/native_world_state.test.ts index 5acafa7d67b6..8b36a4f17c50 100644 --- a/yarn-project/world-state/src/native/native_world_state.test.ts +++ b/yarn-project/world-state/src/native/native_world_state.test.ts @@ -14,7 +14,7 @@ import { timesAsync } from '@aztec/foundation/collection'; import { randomBytes } from '@aztec/foundation/crypto/random'; import { Fr } from '@aztec/foundation/curves/bn254'; import { EthAddress } from '@aztec/foundation/eth-address'; -import { sleep } from '@aztec/foundation/sleep'; +import { retryUntil } from '@aztec/foundation/retry'; import type { SiblingPath } from '@aztec/foundation/trees'; import { PublicDataWrite } from '@aztec/stdlib/avm'; import { L2Block } from '@aztec/stdlib/block'; @@ -960,7 +960,10 @@ describe('NativeWorldState', () => { await ws.unwindBlocks(BlockNumber.fromBigInt(2n)); await expect(delayedFork.getSiblingPath(MerkleTreeId.NULLIFIER_TREE, 0n)).rejects.toThrow('Fork not found'); - await sleep(closeDelayMs * 3); + // The delayed close fires after closeDelayMs and then asynchronously deletes the per-fork queue entry + // once its DELETE_FORK round-trip settles. Under a loaded CI grind that cleanup can take longer than a + // fixed sleep, so wait for it deterministically instead of racing closeDelayMs * 3. + await retryUntil(() => !(ws as any).instance.queues.has(forkId), 'delayed-close fork queue cleanup', 30, 0.1); expect(warnSpy).not.toHaveBeenCalled(); expect((ws as any).instance.queues.has(forkId)).toBe(false); From 93d3d0a2f8b483ad267575514241a5f9dd3eba24 Mon Sep 17 00:00:00 2001 From: Aztec Bot <49558828+AztecBot@users.noreply.github.com> Date: Tue, 16 Jun 2026 04:29:08 -0400 Subject: [PATCH 11/14] fix(kv-store): lazy-load skipped browser benchmark deps (#24108) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Summary - Investigated the `merge-train/spartan-v5` dequeue for train PR #24053. - Latest dequeue was from merge-group CI, not a merge conflict: #24053 is currently mergeable/clean on `merge-train/spartan-v5`. - The failing merge-group run was `27576008840` on `gh-readonly-queue/v5-next/pr-24053-fc2322d6...`, failing `ci/x3-full` via `yarn-project/kv-store/scripts/run_test.sh src/bench/sqlite-opfs-encrypted/map_bench.test.ts`. - Root cause: skipped-by-default browser benchmarks still imported SQLite/OPFS and shared benchmark dependencies at module load, so normal CI test discovery could fail before the `describe.skip` path. - Fix: lazy-load benchmark dependencies only when `VITE_BENCH=1` for the IndexedDB, SQLite-OPFS, and encrypted SQLite-OPFS map benchmarks. Failed dashboard log: http://ci.aztec-labs.com/dbe70cb905f2df69 ## Verification - `CI=1 PATH="$HOME/.cargo/bin:$PATH" scripts/run_test.sh src/bench/sqlite-opfs-encrypted/map_bench.test.ts` from `yarn-project/kv-store` passes with the benchmark skipped. - `CI=1 PATH="$HOME/.cargo/bin:$PATH" scripts/run_test.sh src/bench/sqlite-opfs/map_bench.test.ts` passes with the benchmark skipped. - `CI=1 PATH="$HOME/.cargo/bin:$PATH" scripts/run_test.sh src/bench/indexeddb/map_bench.test.ts` passes with the benchmark skipped. - `yarn format kv-store --check` passes. Notes: the requested root `./bootstrap.sh ci` is not a valid bootstrap command in this checkout (`Unknown command: ci`). I also tried the apparent CI equivalent, `./bootstrap.sh ci-full-no-test-cache`; it did not complete in this container because the local CI engine/tooling failed before reaching these tests. --- *Created by [claudebox](https://claudebox.work/v2/sessions/72862d04194777fa) · group: `slackbot`* --- .../kv-store/src/bench/indexeddb/map_bench.test.ts | 13 +++++++------ .../bench/sqlite-opfs-encrypted/map_bench.test.ts | 13 +++++++------ .../src/bench/sqlite-opfs/map_bench.test.ts | 13 +++++++------ 3 files changed, 21 insertions(+), 18 deletions(-) diff --git a/yarn-project/kv-store/src/bench/indexeddb/map_bench.test.ts b/yarn-project/kv-store/src/bench/indexeddb/map_bench.test.ts index 0fcf07afba4d..1e72320d1998 100644 --- a/yarn-project/kv-store/src/bench/indexeddb/map_bench.test.ts +++ b/yarn-project/kv-store/src/bench/indexeddb/map_bench.test.ts @@ -3,15 +3,16 @@ * Skipped by default; set `VITE_BENCH=1` to run (or invoke this file directly * via `yarn test:browser src/bench/indexeddb/map_bench.test.ts`). */ -import { createLogger } from '@aztec/foundation/log'; - -import { AztecIndexedDBStore } from '../../indexeddb/store.js'; -import { mockLogger } from '../../interfaces/utils.js'; -import { describeAztecMapBench } from '../shared_map_bench.js'; - const shouldRun = (import.meta as ImportMeta & { env?: { VITE_BENCH?: string } }).env?.VITE_BENCH === '1'; if (shouldRun) { + const [{ createLogger }, { AztecIndexedDBStore }, { mockLogger }, { describeAztecMapBench }] = await Promise.all([ + import('@aztec/foundation/log'), + import('../../indexeddb/store.js'), + import('../../interfaces/utils.js'), + import('../shared_map_bench.js'), + ]); + describeAztecMapBench( 'IndexedDB', () => AztecIndexedDBStore.open(mockLogger, undefined, true), diff --git a/yarn-project/kv-store/src/bench/sqlite-opfs-encrypted/map_bench.test.ts b/yarn-project/kv-store/src/bench/sqlite-opfs-encrypted/map_bench.test.ts index 04432e2f8123..cb9d90cace6d 100644 --- a/yarn-project/kv-store/src/bench/sqlite-opfs-encrypted/map_bench.test.ts +++ b/yarn-project/kv-store/src/bench/sqlite-opfs-encrypted/map_bench.test.ts @@ -9,15 +9,16 @@ * * Skipped by default; set VITE_BENCH=1 (and VITE_SQLITE_OPFS=1) to run. */ -import { createLogger } from '@aztec/foundation/log'; - -import { mockLogger } from '../../interfaces/utils.js'; -import { AztecSQLiteOPFSStore } from '../../sqlite-opfs/store.js'; -import { describeAztecMapBench } from '../shared_map_bench.js'; - const shouldRun = (import.meta as ImportMeta & { env?: { VITE_BENCH?: string } }).env?.VITE_BENCH === '1'; if (shouldRun) { + const [{ createLogger }, { mockLogger }, { AztecSQLiteOPFSStore }, { describeAztecMapBench }] = await Promise.all([ + import('@aztec/foundation/log'), + import('../../interfaces/utils.js'), + import('../../sqlite-opfs/store.js'), + import('../shared_map_bench.js'), + ]); + describeAztecMapBench( 'SQLite-OPFS (chacha20)', () => { diff --git a/yarn-project/kv-store/src/bench/sqlite-opfs/map_bench.test.ts b/yarn-project/kv-store/src/bench/sqlite-opfs/map_bench.test.ts index 4775b8dc1789..eee43649a656 100644 --- a/yarn-project/kv-store/src/bench/sqlite-opfs/map_bench.test.ts +++ b/yarn-project/kv-store/src/bench/sqlite-opfs/map_bench.test.ts @@ -3,15 +3,16 @@ * Skipped by default; set `VITE_BENCH=1` to run (or invoke this file directly * via `yarn test:browser src/bench/sqlite-opfs/map_bench.test.ts`). */ -import { createLogger } from '@aztec/foundation/log'; - -import { mockLogger } from '../../interfaces/utils.js'; -import { AztecSQLiteOPFSStore } from '../../sqlite-opfs/store.js'; -import { describeAztecMapBench } from '../shared_map_bench.js'; - const shouldRun = (import.meta as ImportMeta & { env?: { VITE_BENCH?: string } }).env?.VITE_BENCH === '1'; if (shouldRun) { + const [{ createLogger }, { mockLogger }, { AztecSQLiteOPFSStore }, { describeAztecMapBench }] = await Promise.all([ + import('@aztec/foundation/log'), + import('../../interfaces/utils.js'), + import('../../sqlite-opfs/store.js'), + import('../shared_map_bench.js'), + ]); + describeAztecMapBench( 'SQLite-OPFS', () => AztecSQLiteOPFSStore.open(mockLogger, undefined, true), From d9463da96974cafe7b38d93d5822772891c3c370 Mon Sep 17 00:00:00 2001 From: Santiago Palladino Date: Tue, 16 Jun 2026 08:04:58 -0300 Subject: [PATCH 12/14] feat(p2p): slash proposers exceeding max blocks per checkpoint (A-1166) (#24041) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Motivation When a node receives a block proposal whose index within its checkpoint exceeds the per-checkpoint block limit, it rejects the gossip message and penalizes the relaying peer. But an oversized checkpoint is produced by the proposer, not the relay: the proposer should be slashed, and the honest relay left alone. Because the proposal was rejected at ingress, it was never stored nor re-broadcast, so no other node could observe the evidence and a slash for this offense could never fire (slashing is social-consensus voting and needs enough validators to independently see the offense, as with equivocation). ## Approach Mirrors how equivocation slashing works today. - At gossip validation, only indices at or beyond the structural ceiling `MAX_ATTESTABLE_BLOCKS_PER_CHECKPOINT` (72) are rejected with a peer penalty. Indices in `[maxBlocksPerCheckpoint, 72)` are structurally valid proposer misbehavior and pass validation. Proposer/signature/slot checks still run first, so only the slot's legitimate proposer can get a proposal this far. - Oversized proposals ride the regular attestation-pool store path, so they are retained and re-broadcast as slashing evidence with storage and amplification bounded by the existing per-position and per-slot pool caps, and equivocation detection keeps working at oversized indices. The gossip handlers never process or attest to an oversized proposal: `isOversized` is computed once during validation and threaded through the validation-result metadata alongside `isEquivocated`. Pool-cap violations keep their relay penalty regardless of oversizedness — relays must respect the caps even when forwarding slashing evidence. - When the p2p service stores an oversized proposal it invokes a new oversized-proposal callback (the same pattern as the duplicate-proposal callback for equivocation, covering both the standalone block and the checkpoint-embedded gossip paths). The validator client handles it by emitting `BROADCASTED_INVALID_BLOCK_PROPOSAL` with the existing `slashBroadcastedInvalidBlockPenalty`, deduped per (proposer, slot): a signed block proposal at an illegal index is invalid in itself, no checkpoint proposal needed. The validator-client checkpoint-level `too_many_blocks_in_checkpoint` check stays as defense-in-depth, though it cannot fire for the gossip case since oversized proposals are never processed. - When `maxBlocksPerCheckpoint` is undefined (local/test setups), the oversized paths no-op and only the structural ceiling applies. ## Changes - **p2p**: `ProposalValidator.validateBlockIndexWithinCheckpoint` rejects only at the structural ceiling; `LibP2PService` threads `isOversized` through validation metadata, skips processing/attestation for oversized block and checkpoint proposals, and reports stored oversized proposals through a new `registerOversizedProposalCallback`. - **validator-client**: registers the oversized-proposal callback and emits the `BROADCASTED_INVALID_BLOCK_PROPOSAL` offense for the proposer, deduped per (proposer, slot). - **tests**: cover the new accept window, store/re-broadcast/no-process invariants on both gossip paths, duplicate and equivocating oversized proposals, pool-cap rejections, callback firing per stored proposal, offense emission and dedup, and undefined-config no-op. An e2e exercising the full slash flow is left as a follow-up. Fixes A-1166 --- yarn-project/p2p/src/client/interface.ts | 9 + yarn-project/p2p/src/client/p2p_client.ts | 5 + .../proposal_validator.test.ts | 24 +- .../proposal_validator/proposal_validator.ts | 31 +- .../p2p/src/services/dummy_service.ts | 6 + .../services/libp2p/libp2p_service.test.ts | 288 ++++++++++++++++++ .../p2p/src/services/libp2p/libp2p_service.ts | 93 ++++-- yarn-project/p2p/src/services/service.ts | 19 ++ .../txe/src/state_machine/dummy_p2p_client.ts | 5 + .../validator-client/src/proposal_handler.ts | 1 + .../validator-client/src/validator.test.ts | 20 ++ .../validator-client/src/validator.ts | 38 ++- 12 files changed, 502 insertions(+), 37 deletions(-) diff --git a/yarn-project/p2p/src/client/interface.ts b/yarn-project/p2p/src/client/interface.ts index 9d60457a1c21..2c541e4e4a69 100644 --- a/yarn-project/p2p/src/client/interface.ts +++ b/yarn-project/p2p/src/client/interface.ts @@ -13,6 +13,7 @@ import type { ReqRespSubProtocol, ReqRespSubProtocolHandler } from '../services/ import type { DuplicateAttestationInfo, DuplicateProposalInfo, + OversizedProposalInfo, P2PBlockReceivedCallback, P2PCheckpointAttestationCallback, P2PCheckpointReceivedCallback, @@ -97,6 +98,14 @@ export type P2P = P2PClient & { */ registerDuplicateProposalCallback(callback: (info: DuplicateProposalInfo) => void): void; + /** + * Registers a callback invoked when an oversized block proposal (index at or beyond the consensus + * per-checkpoint block limit) is stored and re-broadcast as slashing evidence. + * + * @param callback - Function called with info about the oversized proposal + */ + registerOversizedProposalCallback(callback: (info: OversizedProposalInfo) => void): void; + /** * Registers a callback invoked when a duplicate attestation is detected (equivocation). * A validator signing attestations for different proposals at the same slot. diff --git a/yarn-project/p2p/src/client/p2p_client.ts b/yarn-project/p2p/src/client/p2p_client.ts index 4d959a044e6b..ddd8c121993c 100644 --- a/yarn-project/p2p/src/client/p2p_client.ts +++ b/yarn-project/p2p/src/client/p2p_client.ts @@ -42,6 +42,7 @@ import { ReqRespSubProtocol, type ReqRespSubProtocolHandler } from '../services/ import type { DuplicateAttestationInfo, DuplicateProposalInfo, + OversizedProposalInfo, P2PBlockReceivedCallback, P2PCheckpointReceivedCallback, P2PService, @@ -414,6 +415,10 @@ export class P2PClient extends WithTracer implements P2P { this.p2pService.registerDuplicateProposalCallback(callback); } + public registerOversizedProposalCallback(callback: (info: OversizedProposalInfo) => void): void { + this.p2pService.registerOversizedProposalCallback(callback); + } + public registerDuplicateAttestationCallback(callback: (info: DuplicateAttestationInfo) => void): void { this.p2pService.registerDuplicateAttestationCallback(callback); } diff --git a/yarn-project/p2p/src/msg_validators/proposal_validator/proposal_validator.test.ts b/yarn-project/p2p/src/msg_validators/proposal_validator/proposal_validator.test.ts index 658c4bd0c156..8c053f27081b 100644 --- a/yarn-project/p2p/src/msg_validators/proposal_validator/proposal_validator.test.ts +++ b/yarn-project/p2p/src/msg_validators/proposal_validator/proposal_validator.test.ts @@ -554,13 +554,35 @@ describe('ProposalValidator', () => { epochCache.getProposerAttesterAddressInSlot.mockResolvedValue(signer.address); }); - it('rejects a block proposal whose indexWithinCheckpoint equals the cap (5 >= 5)', async () => { + it('accepts a block proposal whose indexWithinCheckpoint equals the consensus cap (5 >= 5) as slashing evidence', async () => { + // Over the consensus limit but below the hard attestable ceiling: structurally valid proposer + // misbehavior, so gossip validation accepts it for retention/re-broadcast rather than rejecting. const proposal = await makeBlockProposal({ blockHeader: makeBlockHeader(0, { slotNumber: currentSlot }), indexWithinCheckpoint: IndexWithinCheckpoint(5), signer, }); const result = await validator.validate(proposal); + expect(result).toEqual({ result: 'accept' }); + }); + + it('accepts a block proposal at an over-consensus index well within the attestable ceiling', async () => { + const proposal = await makeBlockProposal({ + blockHeader: makeBlockHeader(0, { slotNumber: currentSlot }), + indexWithinCheckpoint: IndexWithinCheckpoint(MAX_ATTESTABLE_BLOCKS_PER_CHECKPOINT - 1), + signer, + }); + const result = await validator.validate(proposal); + expect(result).toEqual({ result: 'accept' }); + }); + + it('rejects a block proposal at the hard attestable ceiling even with a lower consensus cap', async () => { + const proposal = await makeBlockProposal({ + blockHeader: makeBlockHeader(0, { slotNumber: currentSlot }), + indexWithinCheckpoint: IndexWithinCheckpoint(MAX_ATTESTABLE_BLOCKS_PER_CHECKPOINT), + signer, + }); + const result = await validator.validate(proposal); expect(result).toEqual({ result: 'reject', severity: PeerErrorSeverity.MidToleranceError }); }); diff --git a/yarn-project/p2p/src/msg_validators/proposal_validator/proposal_validator.ts b/yarn-project/p2p/src/msg_validators/proposal_validator/proposal_validator.ts index ad351727909e..281565e09795 100644 --- a/yarn-project/p2p/src/msg_validators/proposal_validator/proposal_validator.ts +++ b/yarn-project/p2p/src/msg_validators/proposal_validator/proposal_validator.ts @@ -19,7 +19,6 @@ export class ProposalValidator { private logger: Logger; private txsPermitted: boolean; private maxTxsPerBlock?: number; - private maxBlocksPerCheckpoint?: number; private skipSlotValidation: boolean; private signatureContext: CoordinationSignatureContext; private clockDisparityMs: number; @@ -41,7 +40,6 @@ export class ProposalValidator { this.timetable = timetable; this.txsPermitted = opts.txsPermitted; this.maxTxsPerBlock = opts.maxTxsPerBlock; - this.maxBlocksPerCheckpoint = opts.maxBlocksPerCheckpoint; this.skipSlotValidation = opts.skipSlotValidation ?? false; this.signatureContext = opts.signatureContext; this.clockDisparityMs = opts.clockDisparityMs; @@ -107,8 +105,12 @@ export class ProposalValidator { return { result: 'reject', severity: PeerErrorSeverity.MidToleranceError }; } - // A block proposal whose index lands beyond the checkpoint block limit can never be part of a - // checkpoint we would attest to, so reject it immediately at ingress. + // A block proposal whose index lands at or beyond the hard attestable ceiling is structurally + // impossible garbage, so reject it immediately at ingress. Indices in + // `[maxBlocksPerCheckpoint, MAX_ATTESTABLE_BLOCKS_PER_CHECKPOINT)` are over the consensus limit + // but structurally valid proposer misbehavior; they pass gossip validation here so the offending + // proposal can be retained and re-broadcast as slashing evidence (handled downstream in the p2p + // service), rather than penalizing the relaying peer. if ('indexWithinCheckpoint' in proposal) { const indexResult = this.validateBlockIndexWithinCheckpoint(proposal); if (indexResult.result !== 'accept') { @@ -126,20 +128,21 @@ export class ProposalValidator { } /** - * Rejects a block proposal whose index within its checkpoint lands at or beyond the per-checkpoint - * block limit. `MAX_ATTESTABLE_BLOCKS_PER_CHECKPOINT` is a hard ceiling applied even when - * `maxBlocksPerCheckpoint` is unset; a lower configured value tightens it further. - * `indexWithinCheckpoint` is 0-based, so a ceiling of 72 rejects the 73rd block. Applies to standalone + * Rejects a block proposal whose index within its checkpoint lands at or beyond the hard attestable + * ceiling `MAX_ATTESTABLE_BLOCKS_PER_CHECKPOINT` (a structurally impossible index that can never be + * part of any checkpoint we could attest to). `indexWithinCheckpoint` is 0-based, so a ceiling of 72 + * rejects the 73rd block. + * + * Indices in `[maxBlocksPerCheckpoint, MAX_ATTESTABLE_BLOCKS_PER_CHECKPOINT)` are over the configured + * consensus limit but still structurally valid: they are proposer misbehavior, not relaying-peer + * fault, so they are *not* rejected here. The p2p service retains and re-broadcasts the first such + * proposal per (slot, proposer) as slashing evidence and skips processing it. Applies to standalone * block proposals and to the terminal block embedded in a checkpoint proposal. */ public validateBlockIndexWithinCheckpoint(proposal: BlockProposal): ValidationResult { - const maxBlocksPerCheckpoint = Math.min( - this.maxBlocksPerCheckpoint ?? MAX_ATTESTABLE_BLOCKS_PER_CHECKPOINT, - MAX_ATTESTABLE_BLOCKS_PER_CHECKPOINT, - ); - if (proposal.indexWithinCheckpoint >= maxBlocksPerCheckpoint) { + if (proposal.indexWithinCheckpoint >= MAX_ATTESTABLE_BLOCKS_PER_CHECKPOINT) { this.logger.warn( - `Penalizing peer for proposal with indexWithinCheckpoint ${proposal.indexWithinCheckpoint} >= max ${maxBlocksPerCheckpoint}`, + `Penalizing peer for proposal with indexWithinCheckpoint ${proposal.indexWithinCheckpoint} >= attestable ceiling ${MAX_ATTESTABLE_BLOCKS_PER_CHECKPOINT}`, ); return { result: 'reject', severity: PeerErrorSeverity.MidToleranceError }; } diff --git a/yarn-project/p2p/src/services/dummy_service.ts b/yarn-project/p2p/src/services/dummy_service.ts index 7e132b28ef95..efbce631650c 100644 --- a/yarn-project/p2p/src/services/dummy_service.ts +++ b/yarn-project/p2p/src/services/dummy_service.ts @@ -28,6 +28,7 @@ import { type P2PCheckpointReceivedCallback, type P2PDuplicateAttestationCallback, type P2PDuplicateProposalCallback, + type P2POversizedProposalCallback, type P2PService, type PeerDiscoveryService, PeerDiscoveryState, @@ -98,6 +99,11 @@ export class DummyP2PService implements P2PService { */ public registerDuplicateProposalCallback(_callback: P2PDuplicateProposalCallback): void {} + /** + * Register a callback for when an oversized proposal is stored + */ + public registerOversizedProposalCallback(_callback: P2POversizedProposalCallback): void {} + /** * Register a callback for when a duplicate attestation is detected */ diff --git a/yarn-project/p2p/src/services/libp2p/libp2p_service.test.ts b/yarn-project/p2p/src/services/libp2p/libp2p_service.test.ts index b688db763ad7..3c163f0fe551 100644 --- a/yarn-project/p2p/src/services/libp2p/libp2p_service.test.ts +++ b/yarn-project/p2p/src/services/libp2p/libp2p_service.test.ts @@ -1227,6 +1227,294 @@ describe('LibP2PService', () => { }); }); + describe('oversized proposals (over the consensus maxBlocksPerCheckpoint)', () => { + let attestationPool: AttestationPool; + let mockTxPool: MockProxy; + let mockEpochCache: MockProxy; + let signer: Secp256k1Signer; + let blockReceivedCallback: jest.Mock; + let allNodesCheckpointReceivedCallback: jest.Mock; + let validatorCheckpointReceivedCallback: jest.Mock; + let duplicateProposalCallback: jest.Mock; + let oversizedProposalCallback: jest.Mock; + + const targetSlot = SlotNumber(100); + const nextSlot = SlotNumber(101); + const MAX_BLOCKS = 5; + // 0-based index 5 is the 6th block, one over the consensus cap of 5, but below the hard ceiling. + const oversizedIndex = IndexWithinCheckpoint(MAX_BLOCKS); + const secondOversizedIndex = IndexWithinCheckpoint(MAX_BLOCKS + 1); + + beforeEach(() => { + signer = Secp256k1Signer.random(); + attestationPool = new AttestationPool(openTmpStore(true)); + mockTxPool = mock(); + mockTxPool.protectTxs.mockResolvedValue([]); + + mockEpochCache = mock(); + mockEpochCache.getProposerAttesterAddressInSlot.mockResolvedValue(signer.address); + mockEpochCache.getTargetAndNextSlot.mockReturnValue({ targetSlot, nextSlot }); + mockEpochCache.getTargetSlot.mockReturnValue(targetSlot); + + mockPeerManager = mock(); + reportMessageValidationResultSpy = jest.fn(); + + service = createTestLibP2PServiceWithPools( + mockPeerManager, + reportMessageValidationResultSpy, + attestationPool, + mockTxPool, + mockEpochCache, + { maxBlocksPerCheckpoint: MAX_BLOCKS }, + ); + + blockReceivedCallback = jest.fn().mockImplementation(() => Promise.resolve(true)); + allNodesCheckpointReceivedCallback = jest.fn().mockImplementation(() => Promise.resolve([])); + validatorCheckpointReceivedCallback = jest.fn().mockImplementation(() => Promise.resolve([])); + duplicateProposalCallback = jest.fn(); + oversizedProposalCallback = jest.fn(); + service.registerBlockReceivedCallback(blockReceivedCallback as any); + service.registerAllNodesCheckpointReceivedCallback(allNodesCheckpointReceivedCallback as any); + service.registerValidatorCheckpointReceivedCallback(validatorCheckpointReceivedCallback as any); + service.registerDuplicateProposalCallback(duplicateProposalCallback); + service.registerOversizedProposalCallback(oversizedProposalCallback); + }); + + it('oversized block proposal: accepts (re-broadcast), stores as evidence, does NOT process, no penalty', async () => { + const proposal = await makeBlockProposal({ + signer, + blockHeader: makeBlockHeader(1, { slotNumber: targetSlot }), + indexWithinCheckpoint: oversizedIndex, + archiveRoot: Fr.random(), + }); + + await service.processBlockFromPeer(proposal.toBuffer(), 'msg-1', mockPeerId); + + // Accepted for re-broadcast as slashing evidence + expect(reportMessageValidationResultSpy).toHaveBeenCalledWith('msg-1', MOCK_PEER_ID, TopicValidatorResult.Accept); + // But never processed/attested, and the relaying peer is not penalized + expect(blockReceivedCallback).not.toHaveBeenCalled(); + expect(mockPeerManager.penalizePeer).not.toHaveBeenCalled(); + // Stored in the pool as evidence and reported for slashing + const stored = await attestationPool.getBlockProposalByArchive(proposal.archive.toString()); + expect(stored).toBeDefined(); + expect(oversizedProposalCallback).toHaveBeenCalledWith({ slot: targetSlot, proposer: signer.address }); + }); + + it('duplicate oversized block proposal: ignores, no penalty', async () => { + const proposal = await makeBlockProposal({ + signer, + blockHeader: makeBlockHeader(1, { slotNumber: targetSlot }), + indexWithinCheckpoint: oversizedIndex, + archiveRoot: Fr.random(), + }); + await service.processBlockFromPeer(proposal.toBuffer(), 'msg-1', mockPeerId); + + reportMessageValidationResultSpy.mockClear(); + blockReceivedCallback.mockClear(); + mockPeerManager.penalizePeer.mockClear(); + oversizedProposalCallback.mockClear(); + + await service.processBlockFromPeer(proposal.toBuffer(), 'msg-2', mockPeerId); + + expect(reportMessageValidationResultSpy).toHaveBeenCalledWith('msg-2', MOCK_PEER_ID, TopicValidatorResult.Ignore); + expect(blockReceivedCallback).not.toHaveBeenCalled(); + expect(mockPeerManager.penalizePeer).not.toHaveBeenCalled(); + // Nothing new was stored, so the oversized callback does not fire again + expect(oversizedProposalCallback).not.toHaveBeenCalled(); + }); + + it('equivocating oversized block proposals: accepts and fires the duplicate-proposal callback, no penalty', async () => { + const first = await makeBlockProposal({ + signer, + blockHeader: makeBlockHeader(1, { slotNumber: targetSlot }), + indexWithinCheckpoint: oversizedIndex, + archiveRoot: Fr.random(), + }); + await service.processBlockFromPeer(first.toBuffer(), 'msg-1', mockPeerId); + + reportMessageValidationResultSpy.mockClear(); + blockReceivedCallback.mockClear(); + mockPeerManager.penalizePeer.mockClear(); + + const second = await makeBlockProposal({ + signer, + blockHeader: makeBlockHeader(1, { slotNumber: targetSlot }), + indexWithinCheckpoint: oversizedIndex, + archiveRoot: Fr.random(), + }); + await service.processBlockFromPeer(second.toBuffer(), 'msg-2', mockPeerId); + + // Equivocation at an illegal index is still equivocation: re-broadcast and flag the proposer, + // but never process the proposal or penalize the relaying peer + expect(reportMessageValidationResultSpy).toHaveBeenCalledWith('msg-2', MOCK_PEER_ID, TopicValidatorResult.Accept); + expect(duplicateProposalCallback).toHaveBeenCalledWith({ + slot: targetSlot, + proposer: signer.address, + type: 'block', + }); + // The oversized callback fires for every stored oversized proposal, alongside equivocation + expect(oversizedProposalCallback).toHaveBeenCalledTimes(2); + expect(blockReceivedCallback).not.toHaveBeenCalled(); + expect(mockPeerManager.penalizePeer).not.toHaveBeenCalled(); + }); + + it('distinct oversized block proposals at different indices: each accepted and stored, no penalty', async () => { + const first = await makeBlockProposal({ + signer, + blockHeader: makeBlockHeader(1, { slotNumber: targetSlot }), + indexWithinCheckpoint: oversizedIndex, + archiveRoot: Fr.random(), + }); + await service.processBlockFromPeer(first.toBuffer(), 'msg-1', mockPeerId); + + const second = await makeBlockProposal({ + signer, + blockHeader: makeBlockHeader(2, { slotNumber: targetSlot }), + indexWithinCheckpoint: secondOversizedIndex, + archiveRoot: Fr.random(), + }); + await service.processBlockFromPeer(second.toBuffer(), 'msg-2', mockPeerId); + + expect(reportMessageValidationResultSpy).toHaveBeenCalledWith('msg-2', MOCK_PEER_ID, TopicValidatorResult.Accept); + expect(blockReceivedCallback).not.toHaveBeenCalled(); + expect(mockPeerManager.penalizePeer).not.toHaveBeenCalled(); + const stored = await attestationPool.getBlockProposalByArchive(second.archive.toString()); + expect(stored).toBeDefined(); + // Both stored proposals are reported; the consumer dedups per (proposer, slot) + expect(oversizedProposalCallback).toHaveBeenCalledTimes(2); + }); + + it('oversized checkpoint terminal block: accepts checkpoint (re-broadcast), processes neither block nor checkpoint, no penalty', async () => { + const proposal = await makeCheckpointProposal({ + signer, + checkpointHeader: makeCheckpointHeader(1, { slotNumber: targetSlot }), + lastBlock: { + blockHeader: makeBlockHeader(1, { slotNumber: targetSlot }), + indexWithinCheckpoint: oversizedIndex, + }, + archiveRoot: Fr.random(), + }); + + await service.handleGossipedCheckpointProposal(proposal.toBuffer(), 'msg-1', mockPeerId); + + // Checkpoint accepted for re-broadcast (carries the same signed evidence) + expect(reportMessageValidationResultSpy).toHaveBeenCalledWith('msg-1', MOCK_PEER_ID, TopicValidatorResult.Accept); + // Neither the embedded block nor the checkpoint is processed/attested + expect(blockReceivedCallback).not.toHaveBeenCalled(); + expect(allNodesCheckpointReceivedCallback).not.toHaveBeenCalled(); + expect(validatorCheckpointReceivedCallback).not.toHaveBeenCalled(); + expect(mockPeerManager.penalizePeer).not.toHaveBeenCalled(); + // The oversized terminal block is retained as evidence and reported for slashing + const storedBlock = await attestationPool.getBlockProposalByArchive( + proposal.getBlockProposal()!.archive.toString(), + ); + expect(storedBlock).toBeDefined(); + expect(oversizedProposalCallback).toHaveBeenCalledWith({ slot: targetSlot, proposer: signer.address }); + }); + + it('duplicate oversized checkpoint: ignores, processes neither, no penalty', async () => { + const proposal = await makeCheckpointProposal({ + signer, + checkpointHeader: makeCheckpointHeader(1, { slotNumber: targetSlot }), + lastBlock: { + blockHeader: makeBlockHeader(1, { slotNumber: targetSlot }), + indexWithinCheckpoint: oversizedIndex, + }, + archiveRoot: Fr.random(), + }); + await service.handleGossipedCheckpointProposal(proposal.toBuffer(), 'msg-1', mockPeerId); + + reportMessageValidationResultSpy.mockClear(); + blockReceivedCallback.mockClear(); + allNodesCheckpointReceivedCallback.mockClear(); + validatorCheckpointReceivedCallback.mockClear(); + mockPeerManager.penalizePeer.mockClear(); + oversizedProposalCallback.mockClear(); + + await service.handleGossipedCheckpointProposal(proposal.toBuffer(), 'msg-2', mockPeerId); + + // The terminal block is an exact duplicate, so nothing new was retained: ignored, not + // re-broadcast, not penalized, nothing processed, not reported again + expect(reportMessageValidationResultSpy).toHaveBeenCalledWith('msg-2', MOCK_PEER_ID, TopicValidatorResult.Ignore); + expect(blockReceivedCallback).not.toHaveBeenCalled(); + expect(allNodesCheckpointReceivedCallback).not.toHaveBeenCalled(); + expect(validatorCheckpointReceivedCallback).not.toHaveBeenCalled(); + expect(mockPeerManager.penalizePeer).not.toHaveBeenCalled(); + expect(oversizedProposalCallback).not.toHaveBeenCalled(); + }); + + it('oversized block proposal at the per-position cap: rejects and penalizes the relaying peer', async () => { + // Fill the (slot, index) position to its cap directly in the pool. + for (let i = 0; i < 2; i++) { + const existing = await makeBlockProposal({ + signer, + blockHeader: makeBlockHeader(1, { slotNumber: targetSlot }), + indexWithinCheckpoint: oversizedIndex, + archiveRoot: Fr.random(), + }); + const { added } = await attestationPool.tryAddBlockProposal(existing); + expect(added).toBe(true); + } + + const third = await makeBlockProposal({ + signer, + blockHeader: makeBlockHeader(1, { slotNumber: targetSlot }), + indexWithinCheckpoint: oversizedIndex, + archiveRoot: Fr.random(), + }); + await service.processBlockFromPeer(third.toBuffer(), 'msg-1', mockPeerId); + + // A peer should not relay more than the cap per slot+index, even for an oversized proposal: reject + // and penalize the relaying peer, do not re-broadcast, do not process. + expect(reportMessageValidationResultSpy).toHaveBeenCalledWith('msg-1', MOCK_PEER_ID, TopicValidatorResult.Reject); + expect(mockPeerManager.penalizePeer).toHaveBeenCalledWith(mockPeerId, PeerErrorSeverity.HighToleranceError); + expect(blockReceivedCallback).not.toHaveBeenCalled(); + const stored = await attestationPool.getBlockProposalByArchive(third.archive.toString()); + expect(stored).toBeUndefined(); + expect(oversizedProposalCallback).not.toHaveBeenCalled(); + }); + + it('oversized checkpoint at the per-slot checkpoint cap: rejects and penalizes the relaying peer', async () => { + // Fill the slot's checkpoint-proposal cap directly in the pool (e.g. a proposer that equivocated + // two checkpoints before sending an oversized one). + for (let i = 0; i < 2; i++) { + const existing = await makeCheckpointProposal({ + signer, + checkpointHeader: makeCheckpointHeader(1, { slotNumber: targetSlot }), + archiveRoot: Fr.random(), + }); + const { added } = await attestationPool.tryAddCheckpointProposal(existing.toCore()); + expect(added).toBe(true); + } + + const oversized = await makeCheckpointProposal({ + signer, + checkpointHeader: makeCheckpointHeader(1, { slotNumber: targetSlot }), + lastBlock: { + blockHeader: makeBlockHeader(1, { slotNumber: targetSlot }), + indexWithinCheckpoint: oversizedIndex, + }, + archiveRoot: Fr.random(), + }); + await service.handleGossipedCheckpointProposal(oversized.toBuffer(), 'msg-1', mockPeerId); + + // The per-slot cap is about checkpoint proposals, not block proposals: reject and penalize the + // relaying peer. The oversized terminal block was added before the cap check, so it is still + // retained as evidence and reported for slashing. + expect(reportMessageValidationResultSpy).toHaveBeenCalledWith('msg-1', MOCK_PEER_ID, TopicValidatorResult.Reject); + expect(mockPeerManager.penalizePeer).toHaveBeenCalledWith(mockPeerId, PeerErrorSeverity.HighToleranceError); + expect(blockReceivedCallback).not.toHaveBeenCalled(); + expect(allNodesCheckpointReceivedCallback).not.toHaveBeenCalled(); + expect(validatorCheckpointReceivedCallback).not.toHaveBeenCalled(); + const storedBlock = await attestationPool.getBlockProposalByArchive( + oversized.getBlockProposal()!.archive.toString(), + ); + expect(storedBlock).toBeDefined(); + expect(oversizedProposalCallback).toHaveBeenCalledWith({ slot: targetSlot, proposer: signer.address }); + }); + }); + // Regression for A-1013 describe('validateAndStoreCheckpointAttestation', () => { let attestationPool: AttestationPool; diff --git a/yarn-project/p2p/src/services/libp2p/libp2p_service.ts b/yarn-project/p2p/src/services/libp2p/libp2p_service.ts index 92c6b35ff95e..63dccf33be04 100644 --- a/yarn-project/p2p/src/services/libp2p/libp2p_service.ts +++ b/yarn-project/p2p/src/services/libp2p/libp2p_service.ts @@ -114,6 +114,7 @@ import type { P2PCheckpointAttestationCallback, P2PCheckpointReceivedCallback, P2PDuplicateAttestationCallback, + P2POversizedProposalCallback, P2PService, PeerDiscoveryService, } from '../service.js'; @@ -172,6 +173,9 @@ export class LibP2PService extends WithTracer implements P2PService { type: 'checkpoint' | 'block'; }) => void; + /** Callback invoked when an oversized block proposal is stored as slashing evidence (triggers slashing). */ + private oversizedProposalCallback?: P2POversizedProposalCallback; + /** Callback invoked when a duplicate attestation is detected (triggers slashing). */ private duplicateAttestationCallback?: P2PDuplicateAttestationCallback; @@ -762,6 +766,13 @@ export class LibP2PService extends WithTracer implements P2PService { this.duplicateProposalCallback = callback; } + /** + * Registers a callback to be invoked when an oversized block proposal is stored as slashing evidence. + */ + public registerOversizedProposalCallback(callback: P2POversizedProposalCallback): void { + this.oversizedProposalCallback = callback; + } + /** * Registers a callback to be invoked when a duplicate attestation is detected. * A validator signing attestations for different proposals at the same slot. @@ -1243,16 +1254,17 @@ export class LibP2PService extends WithTracer implements P2PService { const { result, obj: block, - metadata: { isEquivocated } = {}, - } = await this.validateReceivedMessage( + metadata: { isEquivocated, isOversized } = {}, + } = await this.validateReceivedMessage( () => this.validateAndStoreBlockProposal(source, BlockProposal.fromBuffer(payloadData)), msgId, source, TopicType.block_proposal, ); - // If not accepted or equivocated, return - if (result !== TopicValidatorResult.Accept || !block || isEquivocated) { + // If not accepted, equivocated, or oversized, return. Oversized proposals are re-broadcast as + // slashing evidence but never attested or processed. + if (result !== TopicValidatorResult.Accept || !block || isEquivocated || isOversized) { return; } @@ -1267,7 +1279,7 @@ export class LibP2PService extends WithTracer implements P2PService { protected async validateAndStoreBlockProposal( peerId: PeerId, block: BlockProposal, - ): Promise> { + ): Promise> { const validationResult = await this.blockProposalValidator.validate(block); if (validationResult.result === 'reject') { @@ -1282,6 +1294,12 @@ export class LibP2PService extends WithTracer implements P2PService { // Try to add the proposal: this handles existence check, cap check, and adding in one call const { added, alreadyExists, count } = await this.mempools.attestationPool.tryAddBlockProposal(block); const isEquivocated = count !== undefined && count > 1; + // An oversized proposal (index at or beyond the consensus per-checkpoint limit) is structurally valid + // proposer misbehavior: it is stored and re-broadcast as slashing evidence but never processed or + // attested to. No-ops when maxBlocksPerCheckpoint is unset (local/test). + const isOversized = + this.config.maxBlocksPerCheckpoint !== undefined && + block.indexWithinCheckpoint >= this.config.maxBlocksPerCheckpoint; // Duplicate proposal received, no need to re-broadcast if (alreadyExists) { @@ -1291,7 +1309,7 @@ export class LibP2PService extends WithTracer implements P2PService { proposer: block.getSender()?.toString(), source: peerId.toString(), }); - return { result: TopicValidatorResult.Ignore, obj: block, metadata: { isEquivocated } }; + return { result: TopicValidatorResult.Ignore, obj: block, metadata: { isEquivocated, isOversized } }; } // Too many blocks received for this slot and index, penalize peer and do not re-broadcast @@ -1305,11 +1323,27 @@ export class LibP2PService extends WithTracer implements P2PService { }); return { result: TopicValidatorResult.Reject, - metadata: { isEquivocated }, + metadata: { isEquivocated, isOversized }, severity: PeerErrorSeverity.HighToleranceError, }; } + // The proposal was stored: if oversized, invoke the oversized callback so the proposer can be + // slashed. Fired alongside (not instead of) equivocation detection below. + if (isOversized) { + const proposer = block.getSender(); + if (proposer) { + this.logger.warn(`Detected oversized block proposal at slot ${block.slotNumber}`, { + ...block.toBlockInfo(), + indexWithinCheckpoint: block.indexWithinCheckpoint, + maxBlocksPerCheckpoint: this.config.maxBlocksPerCheckpoint, + source: peerId.toString(), + proposer: proposer.toString(), + }); + this.oversizedProposalCallback?.({ slot: block.slotNumber, proposer }); + } + } + // If this was a duplicate proposal, do not process it, but do invoke the duplicate callback, // and do re-broadcast it so other nodes in the network know to slash the proposer if (isEquivocated) { @@ -1323,11 +1357,11 @@ export class LibP2PService extends WithTracer implements P2PService { if (proposer && count === 2) { this.duplicateProposalCallback?.({ slot: block.slotNumber, proposer, type: 'block' }); } - return { result: TopicValidatorResult.Accept, obj: block, metadata: { isEquivocated } }; + return { result: TopicValidatorResult.Accept, obj: block, metadata: { isEquivocated, isOversized } }; } // Otherwise, we're good to go! - return { result: TopicValidatorResult.Accept, obj: block }; + return { result: TopicValidatorResult.Accept, obj: block, metadata: { isEquivocated: false, isOversized } }; } // REFACTOR(palla): This method should be moved to the p2p_client or to a separate component, @@ -1367,17 +1401,21 @@ export class LibP2PService extends WithTracer implements P2PService { const { result, obj: checkpoint, - metadata: { isEquivocated, processBlock } = {}, - } = await this.validateReceivedMessage( + metadata: { isEquivocated, processBlock, isOversized } = {}, + } = await this.validateReceivedMessage< + CheckpointProposal, + { isEquivocated: boolean; processBlock: boolean; isOversized: boolean } + >( () => this.validateAndStoreCheckpointProposal(source, CheckpointProposal.fromBuffer(payloadData)), msgId, source, TopicType.checkpoint_proposal, ); - // Process checkpoint proposal if valid and not equivocated. + // An oversized checkpoint is re-broadcast as slashing evidence but never attested or processed. + // Process checkpoint proposal if valid and neither equivocated nor oversized. const processCheckpointFn = () => - result === TopicValidatorResult.Accept && checkpoint && !isEquivocated + result === TopicValidatorResult.Accept && checkpoint && !isEquivocated && !isOversized ? this.processValidCheckpointProposal(checkpoint.toCore(), source) : Promise.resolve(); @@ -1414,7 +1452,12 @@ export class LibP2PService extends WithTracer implements P2PService { protected async validateAndStoreCheckpointProposal( peerId: PeerId, checkpoint: CheckpointProposal, - ): Promise> { + ): Promise< + ReceivedMessageValidationResult< + CheckpointProposal, + { isEquivocated: boolean; processBlock: boolean; isOversized: boolean } + > + > { const validationResult = await this.checkpointProposalValidator.validate(checkpoint); if (validationResult.result === 'reject') { @@ -1429,13 +1472,16 @@ export class LibP2PService extends WithTracer implements P2PService { // Extract and try to add the block proposal first if present const blockProposal = checkpoint.getBlockProposal(); let processBlock = false; + let isOversized = false; if (blockProposal) { this.logger.debug(`Validating block proposal from propagated checkpoint`, { [Attributes.SLOT_NUMBER]: checkpoint.slotNumber.toString(), [Attributes.P2P_ID]: peerId.toString(), }); const blockProposalResult = await this.validateAndStoreBlockProposal(peerId, blockProposal); - const { obj, metadata: { isEquivocated } = {} } = blockProposalResult; + const { obj, metadata: { isEquivocated, isOversized: blockIsOversized } = {} } = blockProposalResult; + isOversized = blockIsOversized ?? false; + if (blockProposalResult.result === TopicValidatorResult.Reject || !obj || isEquivocated) { this.logger.debug(`Rejecting checkpoint due to invalid last block proposal`, { [Attributes.SLOT_NUMBER]: checkpoint.slotNumber.toString(), @@ -1448,7 +1494,8 @@ export class LibP2PService extends WithTracer implements P2PService { severity: 'severity' in blockProposalResult ? blockProposalResult.severity : PeerErrorSeverity.MidToleranceError, }; - } else if (blockProposalResult.result === TopicValidatorResult.Accept && obj && !isEquivocated) { + } else if (blockProposalResult.result === TopicValidatorResult.Accept && obj && !isEquivocated && !isOversized) { + // An oversized terminal block is re-broadcast as slashing evidence but never processed. processBlock = true; } } @@ -1468,11 +1515,11 @@ export class LibP2PService extends WithTracer implements P2PService { return { result: TopicValidatorResult.Ignore, obj: checkpoint, - metadata: { isEquivocated, processBlock }, + metadata: { isEquivocated, processBlock, isOversized }, }; } - // Too many checkpoint proposals received for this slot, penalize peer and do not re-broadcast + // Too many checkpoint proposals received for this slot, penalize peer and do not re-broadcast. // Note: We still return the checkpoint obj so the lastBlock can be processed if valid if (!added) { this.logger.warn(`Penalizing peer for checkpoint proposal exceeding per-slot cap`, { @@ -1483,7 +1530,7 @@ export class LibP2PService extends WithTracer implements P2PService { return { result: TopicValidatorResult.Reject, obj: checkpoint, - metadata: { isEquivocated, processBlock }, + metadata: { isEquivocated, processBlock, isOversized }, severity: PeerErrorSeverity.HighToleranceError, }; } @@ -1504,12 +1551,16 @@ export class LibP2PService extends WithTracer implements P2PService { return { result: TopicValidatorResult.Accept, obj: checkpoint, - metadata: { isEquivocated, processBlock }, + metadata: { isEquivocated, processBlock, isOversized }, }; } // Otherwise, we're good to go! - return { result: TopicValidatorResult.Accept, obj: checkpoint, metadata: { processBlock, isEquivocated } }; + return { + result: TopicValidatorResult.Accept, + obj: checkpoint, + metadata: { processBlock, isEquivocated, isOversized }, + }; } /** diff --git a/yarn-project/p2p/src/services/service.ts b/yarn-project/p2p/src/services/service.ts index a2f550da5259..2288272821fb 100644 --- a/yarn-project/p2p/src/services/service.ts +++ b/yarn-project/p2p/src/services/service.ts @@ -58,6 +58,19 @@ export type DuplicateProposalInfo = { */ export type P2PDuplicateProposalCallback = (info: DuplicateProposalInfo) => void; +/** Minimal info passed to the oversized proposal callback. */ +export type OversizedProposalInfo = { + slot: SlotNumber; + proposer: EthAddress; +}; + +/** + * Callback for when a block proposal whose index lands at or beyond the consensus per-checkpoint block + * limit is stored and re-broadcast as slashing evidence. May fire multiple times per (slot, proposer) + * if the proposer signed several oversized proposals; consumers are expected to dedup. + */ +export type P2POversizedProposalCallback = (info: OversizedProposalInfo) => void; + /** Minimal info passed to the duplicate attestation callback. */ export type DuplicateAttestationInfo = { slot: SlotNumber; @@ -108,6 +121,12 @@ export interface P2PService { */ registerDuplicateProposalCallback(callback: P2PDuplicateProposalCallback): void; + /** + * Registers a callback invoked when an oversized block proposal (index at or beyond the consensus + * per-checkpoint block limit) is stored and re-broadcast as slashing evidence. + */ + registerOversizedProposalCallback(callback: P2POversizedProposalCallback): void; + /** * Registers a callback invoked when a duplicate attestation is detected (equivocation). * A validator signing attestations for different proposals at the same slot. diff --git a/yarn-project/txe/src/state_machine/dummy_p2p_client.ts b/yarn-project/txe/src/state_machine/dummy_p2p_client.ts index 6772df1acc7c..f492fa5b65d3 100644 --- a/yarn-project/txe/src/state_machine/dummy_p2p_client.ts +++ b/yarn-project/txe/src/state_machine/dummy_p2p_client.ts @@ -9,6 +9,7 @@ import type { P2PConfig, P2PDuplicateAttestationCallback, P2PDuplicateProposalCallback, + P2POversizedProposalCallback, P2PSyncState, PeerId, ReqRespSubProtocol, @@ -237,6 +238,10 @@ export class DummyP2P implements P2P { throw new Error('DummyP2P does not implement "registerDuplicateProposalCallback"'); } + public registerOversizedProposalCallback(_callback: P2POversizedProposalCallback): void { + throw new Error('DummyP2P does not implement "registerOversizedProposalCallback"'); + } + public registerDuplicateAttestationCallback(_callback: P2PDuplicateAttestationCallback): void { throw new Error('DummyP2P does not implement "registerDuplicateAttestationCallback"'); } diff --git a/yarn-project/validator-client/src/proposal_handler.ts b/yarn-project/validator-client/src/proposal_handler.ts index 996268c7b6ab..3dc82c5c8111 100644 --- a/yarn-project/validator-client/src/proposal_handler.ts +++ b/yarn-project/validator-client/src/proposal_handler.ts @@ -968,6 +968,7 @@ export class ProposalHandler { }; } + // Note this condition should never trigger, since we dont process block proposals that exceed indexWithinCheckpoint const maxBlocksPerCheckpoint = this.config.maxBlocksPerCheckpoint; if (maxBlocksPerCheckpoint !== undefined && blocks.length > maxBlocksPerCheckpoint) { this.log.warn(`Checkpoint proposal exceeds maxBlocksPerCheckpoint`, { diff --git a/yarn-project/validator-client/src/validator.test.ts b/yarn-project/validator-client/src/validator.test.ts index 354685f501b0..a3ac867605d6 100644 --- a/yarn-project/validator-client/src/validator.test.ts +++ b/yarn-project/validator-client/src/validator.test.ts @@ -966,6 +966,26 @@ describe('ValidatorClient', () => { expect(validatorClient.hasInvalidProposals(proposal.slotNumber)).toBe(true); }); + it('emits invalid block proposal offense for oversized proposals, deduped per proposer and slot', async () => { + await validatorClient.registerHandlers(); + const oversizedProposalCallback = p2pClient.registerOversizedProposalCallback.mock.calls[0][0]; + const emitSpy = jest.spyOn(validatorClient, 'emit'); + + const info = { slot: proposal.slotNumber, proposer: proposal.getSender()! }; + oversizedProposalCallback(info); + oversizedProposalCallback(info); + + expect(emitSpy).toHaveBeenCalledTimes(1); + expect(emitSpy).toHaveBeenCalledWith(WANT_TO_SLASH_EVENT, [ + { + validator: info.proposer, + amount: config.slashBroadcastedInvalidBlockPenalty, + offenseType: OffenseType.BROADCASTED_INVALID_BLOCK_PROPOSAL, + epochOrSlot: BigInt(proposal.slotNumber), + }, + ]); + }); + it('records proposal equivocation and emits clear event', async () => { await validatorClient.registerHandlers(); const duplicateProposalCallback = p2pClient.registerDuplicateProposalCallback.mock.calls[0][0]; diff --git a/yarn-project/validator-client/src/validator.ts b/yarn-project/validator-client/src/validator.ts index 732c2cd28beb..8a081f5dc86b 100644 --- a/yarn-project/validator-client/src/validator.ts +++ b/yarn-project/validator-client/src/validator.ts @@ -11,7 +11,7 @@ import { RunningPromise } from '@aztec/foundation/running-promise'; import { sleep } from '@aztec/foundation/sleep'; import { DateProvider } from '@aztec/foundation/timer'; import type { KeystoreManager } from '@aztec/node-keystore'; -import type { DuplicateAttestationInfo, DuplicateProposalInfo, P2P, PeerId } from '@aztec/p2p'; +import type { DuplicateAttestationInfo, DuplicateProposalInfo, OversizedProposalInfo, P2P, PeerId } from '@aztec/p2p'; import { AuthRequest, AuthResponse, BlockProposalValidator, ReqRespSubProtocol } from '@aztec/p2p'; import { OffenseType, @@ -133,6 +133,7 @@ export class ValidatorClient extends (EventEmitter as new () => WatcherEmitter) private proposersOfInvalidBlocks = FifoSet.withLimit(MAX_PROPOSERS_OF_INVALID_BLOCKS); private slotsWithInvalidProposals = FifoSet.withLimit(MAX_TRACKED_INVALID_PROPOSAL_SLOTS); private invalidCheckpointProposalOffenseKeys = FifoSet.withLimit(MAX_TRACKED_INVALID_CHECKPOINT_PROPOSALS); + private oversizedProposalOffenseKeys = FifoSet.withLimit(MAX_TRACKED_INVALID_CHECKPOINT_PROPOSALS); private badAttestationOffenseKeys = FifoSet.withLimit(MAX_TRACKED_BAD_ATTESTATIONS); private slotsWithProposalEquivocation = FifoSet.withLimit(MAX_TRACKED_INVALID_PROPOSAL_SLOTS); @@ -436,6 +437,11 @@ export class ValidatorClient extends (EventEmitter as new () => WatcherEmitter) this.handleDuplicateProposal(info); }); + // Oversized proposal handler - triggers slashing for proposals beyond the per-checkpoint block limit + this.p2pClient.registerOversizedProposalCallback((info: OversizedProposalInfo) => { + this.handleOversizedProposal(info); + }); + // Duplicate attestation handler - triggers slashing for attestation equivocation this.p2pClient.registerDuplicateAttestationCallback((info: DuplicateAttestationInfo) => { this.handleDuplicateAttestation(info); @@ -849,6 +855,36 @@ export class ValidatorClient extends (EventEmitter as new () => WatcherEmitter) ]); } + /** + * Handle detection of an oversized block proposal: one whose index within its checkpoint lands at or + * beyond the consensus per-checkpoint block limit. A single signed proposal at an illegal index is + * self-contained evidence, so emit an invalid-block-proposal slash event for the proposer, deduped per + * (proposer, slot) since the p2p layer reports every oversized proposal it stores. + */ + private handleOversizedProposal(info: OversizedProposalInfo): void { + const { slot, proposer } = info; + const offenseType = OffenseType.BROADCASTED_INVALID_BLOCK_PROPOSAL; + if (!this.oversizedProposalOffenseKeys.addIfAbsent(`${proposer.toString()}:${offenseType}:${slot}`)) { + return; + } + + this.log.info(`Detected oversized block proposal offense from ${proposer.toString()} at slot ${slot}`, { + proposer: proposer.toString(), + slot, + amount: this.config.slashBroadcastedInvalidBlockPenalty, + offenseType: getOffenseTypeName(offenseType), + }); + + this.emit(WANT_TO_SLASH_EVENT, [ + { + validator: proposer, + amount: this.config.slashBroadcastedInvalidBlockPenalty, + offenseType, + epochOrSlot: BigInt(slot), + }, + ]); + } + /** * Handle detection of a duplicate proposal (equivocation). * Emits a slash event when a proposer sends multiple proposals for the same position. From 4d11f2ca65aaeb94bfbe0f924f54a5a83d5368f0 Mon Sep 17 00:00:00 2001 From: Santiago Palladino Date: Tue, 16 Jun 2026 08:25:30 -0300 Subject: [PATCH 13/14] fix(stdlib): fix race conditions in L2BlockStream (#24042) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Human-written summary Fixes three potential issues in the block-stream: 1. We snapshot the source tips, then download blocks until the proposed tip and pass them to the consumer, and then emit the checkpointed and proven tips. If there's a prune while we're downloading blocks, we may not be able to fetch blobs until the proposed tip. And worse, we may be then emitting checkpointed and proven events that reference blocks _we could never pass onto the consumer_. This PR fixes it by aborting all events if we fail to collect all the blocks we expected, and waiting until the next iteration to heal. 2. When we detect a prune, we walk back to find the common ancestor. But that walk back may take time. During that walk back, the checkpointed and proven tips may have shifted, so we re-fetch them to advertise the proper latest tips. 3. When walking back to find the common ancestor during a prune, we compare block hashes of source vs consumer. If the consumer didn't see a block for whatever reason, but that block was also removed from the source, then both block hashes compared are undefined, and we stop the walk-back, instead of continuing until finding an actual existing common ancestor. 4. When walking back, if the consumer for some reason did not have the block hashes available, a walk back to genesis would be triggered. We now cap the walk back to the local finalized chain tip. ## Motivation This PR hardens the block-mode `L2BlockStream` against three pre-existing hazards. It is split out of #24034 (the `chain-proposed`/`tips-only` feature work) so the correctness fixes can be reviewed on their own — #24034's `chain-proposed` emission depends on the pass-completion gate introduced here, so this PR sits below it in the stack. The three hazards, all latent in block mode today: - **Tier cursors advance from an incomplete download pass.** The download loop broke on an empty `getBlocks` result and end-of-pass reconciliation still ran, so `chain-checkpointed`/`chain-proven`/ `chain-finalized` could reference blocks the consumer never received via `blocks-added` — exactly the transient-inconsistency class #24007 set out to remove. - **`chain-pruned` carries stale clamp tips.** The prune event's `checkpointed`/`proven` payload came from the pass snapshot, which on a stale pass describes the chain *before* the prune. p2p feeds `event.checkpointed.checkpoint` into `isEpochPrune`, whose contract is "the checkpoint id **after** the prune", and the result drives the irreversible `deleteAllTxs` mempool wipe — a misclassification risk. - **The walk-back stops above the true divergence.** `areBlockHashesEqualAt` ended with `localHash === sourceHash`, so a height missing on both sides (`undefined === undefined`) counted as agreement. With a store that has gaps in its hash index (p2p keeps none below its `startingBlock`), a gap meeting a source-pruned height stopped the walk above the divergence — an under-deep prune that left old-fork state in place with no later re-detection. ## Approach - **Pass atomicity.** Extract the block download loop into `downloadBlocks()`, which returns whether the download plan completed; tier reconciliation is skipped for the pass when it did not. Two staleness terminations: an empty `getBlocks` below the target, and a delivered proposed block whose hash differs from the snapshot's (a mid-pass same-height fork swap). Intentional skips — the loop never running because `startingBlock`/caught-up put the cursor past the proposed tip — are trivially complete and still reconcile, preserving the A-1061 fix. - **Fresh reorg payload.** On walk-back divergence, re-read `getL2Tips` and drive the `chain-pruned` clamp tips, the #13471 prune-target clamp, the download target, and tier reconciliation from the re-read rather than the pre-prune snapshot. The pass does not abort if the re-read's proposed tip moved on (a busy chain advances every pass and prune emission would otherwise starve). - **Walk-back missing-local-hash rule.** A missing **local** hash now compares unequal regardless of the source side, so the walk continues to a recorded matching height or genesis (block 0 always resolves via the store's `initialBlockHash`). Prunes become over-deep-only, which consumers tolerate. The `skipFinalized` AbortError path is unchanged. - The walk-back hash cache is seeded with the **proposed tip only**. Seeding the tier tips would let a snapshot entry that went stale (a reorg between the snapshot and the walk) fake agreement at a reorged height and stop the walk early; the re-read appends only its proposed tip after the walk is over. **Behavior change worth calling out:** a *permanently* missing source range previously advanced tier cursors past undelivered blocks (masking the source bug). With pass atomicity it now wedges loudly — a warn each pass, p2p stays SYNCHING — instead of silently diverging. Loud failure is the better behavior. ## Changes - `stdlib/src/block/l2_block_stream/l2_block_stream.ts`: `downloadBlocks()` extraction + completion gate; post-divergence `getL2Tips` re-read driving the prune payload/target/download/reconciliation; missing local-hash → unequal; proposed-only cache seeding. - `stdlib/src/block/l2_block_stream/l2_block_stream.test.ts`: pass-atomicity terminations (empty `getBlocks`, delivered-hash mismatch, A-1061 fast-forward still reconciles), prune-payload freshness, re-read download target + reconciliation, and the walk-back both-undefined / stale tier-seed regressions. --- .../e2e_token_bridge_tutorial_test.test.ts | 18 +- yarn-project/p2p/src/client/p2p_client.ts | 2 + .../prover-node/src/checkpoint-store.test.ts | 35 +- .../prover-node/src/checkpoint-store.ts | 17 +- .../prover-node/src/prover-node.test.ts | 126 ++- yarn-project/prover-node/src/prover-node.ts | 94 +- .../block_synchronizer.test.ts | 190 +++- .../block_synchronizer/block_synchronizer.ts | 46 +- .../src/block/l2_block_stream/interfaces.ts | 19 +- .../l2_block_stream/l2_block_stream.test.ts | 900 +++++++++++++++++- .../block/l2_block_stream/l2_block_stream.ts | 222 ++++- .../l2_block_stream/l2_tips_store_base.ts | 45 +- .../block/test/l2_tips_store_test_suite.ts | 65 ++ .../telemetry-client/src/wrappers/index.ts | 1 - .../src/wrappers/l2_block_stream.ts | 38 - .../txe/esbuild/stubs/telemetry_stub.ts | 1 - .../server_world_state_synchronizer.ts | 9 + 17 files changed, 1606 insertions(+), 222 deletions(-) delete mode 100644 yarn-project/telemetry-client/src/wrappers/l2_block_stream.ts diff --git a/yarn-project/end-to-end/src/composed/e2e_token_bridge_tutorial_test.test.ts b/yarn-project/end-to-end/src/composed/e2e_token_bridge_tutorial_test.test.ts index e5947a5dc11c..f398e8f38ece 100644 --- a/yarn-project/end-to-end/src/composed/e2e_token_bridge_tutorial_test.test.ts +++ b/yarn-project/end-to-end/src/composed/e2e_token_bridge_tutorial_test.test.ts @@ -71,7 +71,9 @@ async function addMinter(l1TokenContract: EthAddress, l1TokenHandler: EthAddress abi: TestERC20Abi, client: l1Client, }); - await contract.write.addMinter([l1TokenHandler.toString()]); + await l1Client.waitForTransactionReceipt({ + hash: await contract.write.addMinter([l1TokenHandler.toString()]), + }); } // To run these tests against a local network: @@ -138,10 +140,16 @@ describe('e2e_cross_chain_messaging token_bridge_tutorial_test', () => { await l2TokenContract.methods.set_minter(l2BridgeContract.address, true).send({ from: ownerAztecAddress }); // Initialize L1 portal contract - await l1Portal.write.initialize( - [l1ContractAddresses.registryAddress.toString(), l1TokenContract.toString(), l2BridgeContract.address.toString()], - {}, - ); + await l1Client.waitForTransactionReceipt({ + hash: await l1Portal.write.initialize( + [ + l1ContractAddresses.registryAddress.toString(), + l1TokenContract.toString(), + l2BridgeContract.address.toString(), + ], + {}, + ), + }); logger.info('L1 portal contract initialized'); const l1PortalManager = new L1TokenPortalManager( diff --git a/yarn-project/p2p/src/client/p2p_client.ts b/yarn-project/p2p/src/client/p2p_client.ts index ddd8c121993c..60b2529981e3 100644 --- a/yarn-project/p2p/src/client/p2p_client.ts +++ b/yarn-project/p2p/src/client/p2p_client.ts @@ -179,6 +179,8 @@ export class P2PClient extends WithTracer implements P2P { break; case 'chain-checkpointed': break; + case 'chain-proposed': + break; default: { const _: never = event; break; diff --git a/yarn-project/prover-node/src/checkpoint-store.test.ts b/yarn-project/prover-node/src/checkpoint-store.test.ts index 7c7e9bc841f1..4e5c7eff30a3 100644 --- a/yarn-project/prover-node/src/checkpoint-store.test.ts +++ b/yarn-project/prover-node/src/checkpoint-store.test.ts @@ -1,6 +1,6 @@ import { ARCHIVE_HEIGHT } from '@aztec/constants'; import { makeTuple } from '@aztec/foundation/array'; -import { CheckpointNumber, EpochNumber, SlotNumber } from '@aztec/foundation/branded-types'; +import { BlockNumber, CheckpointNumber, EpochNumber, SlotNumber } from '@aztec/foundation/branded-types'; import { timesAsync } from '@aztec/foundation/collection'; import { Fr } from '@aztec/foundation/curves/bn254'; import type { L2BlockSource } from '@aztec/stdlib/block'; @@ -56,7 +56,7 @@ describe('CheckpointStore', () => { const first = await store.addOrUpdate(cp, makeRegisterData()); expect(first.isPruned()).toBe(false); - store.markPrunedAfter(CheckpointNumber(0)); + store.markPrunedAboveBlock(BlockNumber(0)); expect(first.isPruned()).toBe(true); // Re-adding the identical checkpoint (same archive root) reuses the existing prover. @@ -81,7 +81,7 @@ describe('CheckpointStore', () => { // After the predecessor is pruned, the replacement is accepted and keys to a distinct // prover (different archive root → different content id). - store.markPrunedAfter(CheckpointNumber(0)); + store.markPrunedAboveBlock(BlockNumber(0)); expect(proverA.isPruned()).toBe(true); const proverB = await store.addOrUpdate(b, makeRegisterData()); expect(proverB).not.toBe(proverA); @@ -89,16 +89,31 @@ describe('CheckpointStore', () => { expect(stubs.length).toBe(2); }); - it('markPrunedAfter marks every prover above the threshold and returns them', async () => { - const cps = await timesAsync(4, i => Checkpoint.random(CheckpointNumber(i + 1), { numBlocks: 1 })); + it('markPrunedAboveBlock marks every prover holding a block above the target and returns them', async () => { + // Four single-block checkpoints occupying blocks 1..4 (one block each). Pruning to block 2 orphans the + // checkpoints whose last block is above 2 — checkpoints 3 and 4 — and leaves 1 and 2 canonical. + const cps = await timesAsync(4, i => + Checkpoint.random(CheckpointNumber(i + 1), { numBlocks: 1, startBlockNumber: i + 1 }), + ); for (const cp of cps) { await store.addOrUpdate(cp, makeRegisterData()); } - const affected = store.markPrunedAfter(CheckpointNumber(2)); + const affected = store.markPrunedAboveBlock(BlockNumber(2)); expect(affected.map(p => p.checkpoint.number)).toEqual([3, 4]); expect(store.listCanonical().map(p => p.checkpoint.number)).toEqual([1, 2]); }); + it('markPrunedAboveBlock marks a checkpoint whose block range straddles the target (partially orphaned)', async () => { + // A single checkpoint spanning blocks 5..8. A prune to block 6 lands mid-checkpoint: the checkpoint is partially + // orphaned (blocks 7, 8 are gone) and must be marked, since its last block (8) is above the target. + const cp = await Checkpoint.random(CheckpointNumber(1), { numBlocks: 4, startBlockNumber: 5 }); + await store.addOrUpdate(cp, makeRegisterData()); + + const affected = store.markPrunedAboveBlock(BlockNumber(6)); + expect(affected.map(p => p.checkpoint.number)).toEqual([1]); + expect(store.listCanonical()).toEqual([]); + }); + it('reapExpired drops canonical provers whose epoch is ≤ expiredEpoch', async () => { // With epochDuration=1 each checkpoint's slot is also its epoch number. const cps = await Promise.all([ @@ -117,7 +132,7 @@ describe('CheckpointStore', () => { it('reapExpired leaves pruned provers in place', async () => { const cp = await Checkpoint.random(CheckpointNumber(1), { numBlocks: 1, slotNumber: SlotNumber(1) }); await store.addOrUpdate(cp, makeRegisterData()); - store.markPrunedAfter(CheckpointNumber(0)); + store.markPrunedAboveBlock(BlockNumber(0)); store.reapExpired(EpochNumber(10)); expect(store.listAll().map(p => p.checkpoint.number)).toEqual([1]); }); @@ -132,7 +147,7 @@ describe('CheckpointStore', () => { await store.addOrUpdate(cp, makeRegisterData()); } // Prune everything above checkpoint 0 ⇒ all three flip to pruned. - store.markPrunedAfter(CheckpointNumber(0)); + store.markPrunedAboveBlock(BlockNumber(0)); blockSource.getSyncedL2SlotNumber.mockResolvedValue(SlotNumber(3)); await store.triggerSlotWatcherTick(); @@ -160,7 +175,7 @@ describe('CheckpointStore', () => { it('slot watcher no-ops when getSyncedL2SlotNumber returns undefined', async () => { const cp = await Checkpoint.random(CheckpointNumber(1), { numBlocks: 1, slotNumber: SlotNumber(1) }); await store.addOrUpdate(cp, makeRegisterData()); - store.markPrunedAfter(CheckpointNumber(0)); + store.markPrunedAboveBlock(BlockNumber(0)); blockSource.getSyncedL2SlotNumber.mockResolvedValue(undefined); await store.triggerSlotWatcherTick(); @@ -174,7 +189,7 @@ describe('CheckpointStore', () => { it('slot watcher swallows getSyncedL2SlotNumber errors instead of crashing the tick', async () => { const cp = await Checkpoint.random(CheckpointNumber(1), { numBlocks: 1, slotNumber: SlotNumber(1) }); await store.addOrUpdate(cp, makeRegisterData()); - store.markPrunedAfter(CheckpointNumber(0)); + store.markPrunedAboveBlock(BlockNumber(0)); blockSource.getSyncedL2SlotNumber.mockRejectedValue(new Error('archiver unavailable')); await expect(store.triggerSlotWatcherTick()).resolves.toBeUndefined(); diff --git a/yarn-project/prover-node/src/checkpoint-store.ts b/yarn-project/prover-node/src/checkpoint-store.ts index 79e80a1d7efa..4006041c6950 100644 --- a/yarn-project/prover-node/src/checkpoint-store.ts +++ b/yarn-project/prover-node/src/checkpoint-store.ts @@ -1,4 +1,4 @@ -import type { CheckpointNumber, EpochNumber, SlotNumber } from '@aztec/foundation/branded-types'; +import type { BlockNumber, CheckpointNumber, EpochNumber, SlotNumber } from '@aztec/foundation/branded-types'; import { type Logger, type LoggerBindings, createLogger } from '@aztec/foundation/log'; import { RunningPromise } from '@aztec/foundation/promise'; import type { L2BlockSource } from '@aztec/stdlib/block'; @@ -97,14 +97,19 @@ export class CheckpointStore { } /** - * Marks every canonical prover whose checkpoint number is strictly greater than - * `prunedNumber` as pruned. Sub-tree work keeps running so a re-add of the same - * content can pick it up. Returns the affected provers. + * Marks every canonical prover that holds a block above the prune target as pruned. A checkpoint is orphaned by a + * prune to block `targetBlockNumber` iff its last block sits above the target — including a checkpoint whose range + * straddles the target (partially orphaned), which block-range marking catches without boundary ambiguity. Keying + * off the surviving block number (rather than a checkpoint number) is correct even when the source has already + * re-checkpointed past the divergence: the prune event reports the highest surviving block, which by construction + * survives on the source, whereas the source's current checkpointed tip can sit above the prune target. + * Sub-tree work keeps running so a re-add of the same content can pick it up. Returns the affected provers. */ - public markPrunedAfter(prunedNumber: CheckpointNumber): CheckpointProver[] { + public markPrunedAboveBlock(targetBlockNumber: BlockNumber): CheckpointProver[] { const affected: CheckpointProver[] = []; for (const prover of this.provers.values()) { - if (prover.checkpoint.number > prunedNumber && !prover.isPruned()) { + const lastBlockNumber = prover.checkpoint.blocks.at(-1)!.number; + if (lastBlockNumber > targetBlockNumber && !prover.isPruned()) { prover.markPruned(); affected.push(prover); } diff --git a/yarn-project/prover-node/src/prover-node.test.ts b/yarn-project/prover-node/src/prover-node.test.ts index 1725c0c13f3e..cb282dc6d376 100644 --- a/yarn-project/prover-node/src/prover-node.test.ts +++ b/yarn-project/prover-node/src/prover-node.test.ts @@ -124,7 +124,7 @@ describe('ProverNode', () => { expect(proverNode.getLastProcessedCheckpoint()).toEqual(CheckpointNumber(100)); }); - it('dispatches chain-pruned through markPrunedAfter and notifies the session manager only when affected', async () => { + it('dispatches chain-pruned through markPrunedAboveBlock and notifies the session manager only when affected', async () => { // No registered checkpoints — nothing to prune. await proverNode.handleBlockStreamEvent({ type: 'chain-pruned', @@ -134,9 +134,12 @@ describe('ProverNode', () => { }); expect(sessionManager.onPrune).not.toHaveBeenCalled(); - // Register a checkpoint, then prune. + // Register a checkpoint (cp 2 at block 2), then prune to block 1. The checkpoint's only block (2) is above the + // prune target, so it is marked pruned and its epoch (2) is reported. setupNotFullyProven(); await proverNode.handleBlockStreamEvent(mineCheckpoint(makeCheckpoint(2, 2, 2))); + // The prune target (block 1) resolves to checkpoint 1, clamping the cursor to checkpoint 0. + l2BlockSource.getBlockData.mockResolvedValue({ checkpointNumber: CheckpointNumber(1) } as any); await proverNode.handleBlockStreamEvent({ type: 'chain-pruned', @@ -145,6 +148,71 @@ describe('ProverNode', () => { proven: makeTipId(1), }); expect(sessionManager.onPrune).toHaveBeenCalledWith([EpochNumber(2)]); + expect(proverNode.getLastProcessedCheckpoint()).toEqual(CheckpointNumber(0)); + }); + + it('marks an orphaned checkpoint and reprocesses its same-number rebuild despite an inflated checkpointed tip', async () => { + // Regression for keying the prune off event.checkpointed (the source's CURRENT checkpointed tip) rather than + // event.block (the prune target). Register checkpoint 3 (block 3, epoch 3). A reorg drops block 3, but by the time + // the prune is observed the source has already re-checkpointed a replacement at the SAME number 3, so the event's + // checkpointed tip still reports number 3 — above the real prune target. Keying off that inflated number would + // (a) leave the orphaned prover canonical and (b) clamp the cursor to 3, permanently skipping the rebuilt + // checkpoint. Keying off the prune-target block (2) marks the orphan and clamps the cursor below 3 so the rebuild + // reprocesses. + setupNotFullyProven(); + const original = makeCheckpoint(3, 3, 3, Fr.random()); + await proverNode.handleBlockStreamEvent(mineCheckpoint(original)); + expect(proverNode.getLastProcessedCheckpoint()).toEqual(CheckpointNumber(3)); + const originalProver = proverNode.getCheckpointStore().listAll()[0]; + + // The prune target is block 2 (in checkpoint 2), but the event's checkpointed tip is inflated to the rebuilt 3. + // getBlockData also feeds collectRegisterData when the rebuild re-registers, so it carries a header too. + l2BlockSource.getBlockData.mockResolvedValue({ + checkpointNumber: CheckpointNumber(2), + header: { lastArchive: { root: Fr.ZERO } }, + } as any); + await proverNode.handleBlockStreamEvent({ + type: 'chain-pruned', + block: { number: BlockNumber(2), hash: '0x02' }, + checkpointed: makeTipId(3), + proven: makeTipId(2), + }); + + // The orphaned prover for checkpoint 3 is marked pruned, and the cursor was clamped below 3. + expect(originalProver.isPruned()).toBe(true); + expect(proverNode.getLastProcessedCheckpoint()).toEqual(CheckpointNumber(1)); + + // The rebuilt checkpoint 3 (distinct archive root) is now served by the source. A fresh chain-checkpointed(3) + // re-registers it because the cursor sits below 3. + sessionManager.onCheckpointAdded.mockClear(); + const rebuilt = makeCheckpoint(3, 3, 3, Fr.random()); + await proverNode.handleBlockStreamEvent(mineCheckpoint(rebuilt)); + expect(sessionManager.onCheckpointAdded).toHaveBeenCalledWith(EpochNumber(3)); + expect(proverNode.getCheckpointStore().getByCheckpoint(rebuilt)).toBeDefined(); + expect(proverNode.getLastProcessedCheckpoint()).toEqual(CheckpointNumber(3)); + }); + + it('throws on a prune whose target block data is missing, leaving provers and cursor untouched for retry', async () => { + // The cursor floor is resolved before any prover is marked, so a missing-data prune throws without side effects + // and the next pass retries the whole handler (the tips cursor only advances on success). + setupNotFullyProven(); + await proverNode.handleBlockStreamEvent(mineCheckpoint(makeCheckpoint(3, 3, 3))); + expect(proverNode.getLastProcessedCheckpoint()).toEqual(CheckpointNumber(3)); + const registeredProver = proverNode.getCheckpointStore().listAll()[0]; + + l2BlockSource.getBlockData.mockResolvedValue(undefined); + await expect( + proverNode.handleBlockStreamEvent({ + type: 'chain-pruned', + block: { number: BlockNumber(2), hash: '0x02' }, + checkpointed: makeTipId(3), + proven: makeTipId(2), + }), + ).rejects.toThrow(/No block data found for prune target/); + + expect(registeredProver.isPruned()).toBe(false); + expect(sessionManager.onPrune).not.toHaveBeenCalled(); + expect(proverNode.getLastProcessedCheckpoint()).toEqual(CheckpointNumber(3)); }); it('dispatches chain-proven to publishingService.onChainProven', async () => { @@ -307,17 +375,18 @@ describe('ProverNode', () => { await expect(proverNode.getJobs()).resolves.toEqual([]); }); - // ---------------- handleBlockStreamEvent: blocks-added is a no-op + still triggers expiry ---------------- + // ---------------- handleBlockStreamEvent: chain-proposed is a no-op + still triggers expiry ---------------- - it("'blocks-added' invokes no event handler but still runs the expiry sweep", async () => { + it("'chain-proposed' invokes no event handler but still runs the expiry sweep", async () => { // latestSlot=4 ⇒ epochs 0..2 expire. l2BlockSource.getSyncedL2SlotNumber.mockResolvedValue(SlotNumber(4)); l2BlockSource.getCheckpointsData.mockResolvedValue([]); const reapSpy = jest.spyOn(proverNode.getCheckpointStore(), 'reapExpired'); - // Use a real (random) L2Block so the tips-store handler doesn't choke on an empty array. - const block = await L2Block.random(BlockNumber(1)); - await proverNode.handleBlockStreamEvent({ type: 'blocks-added', blocks: [block] }); + await proverNode.handleBlockStreamEvent({ + type: 'chain-proposed', + block: { number: BlockNumber(1), hash: '0x01' }, + }); // No checkpoint, prune, or proven handler should have fired. expect(sessionManager.onCheckpointAdded).not.toHaveBeenCalled(); @@ -325,6 +394,8 @@ describe('ProverNode', () => { expect(publishingService.onChainProven).not.toHaveBeenCalled(); // But the expiry sweep ran. expect(reapSpy.mock.calls.map(([e]) => Number(e))).toEqual([0, 1, 2]); + // The tips store recorded the proposed tip (it is the walk-back history in tips-only mode). + expect(await proverNode.getTipsStore().getL2BlockHash(1)).toEqual('0x01'); }); // ---------------- checkEpochExpiry: latestEpoch < offset is a no-op ---------------- @@ -380,8 +451,7 @@ describe('ProverNode', () => { await proverNode.handleBlockStreamEvent(mineCheckpoint(makeCheckpoint(2, 7, 7))); expect(proverNode.getCheckpointStore().listAll().length).toBe(2); - // Pruning above checkpoint 0 marks both as pruned — onPrune must receive [EpochNumber(3)], - // not [3, 3]. + // Pruning to block 0 (genesis) marks both as pruned — onPrune must receive [EpochNumber(3)], not [3, 3]. sessionManager.onPrune.mockClear(); await proverNode.handleBlockStreamEvent({ type: 'chain-pruned', @@ -468,27 +538,25 @@ describe('ProverNode', () => { }); }); - // ---------------- computeStartupState branches ---------------- + // ---------------- resolveLastFullyProvenEpoch branches ---------------- - describe('computeStartupState', () => { - it('returns starting block 1 and no fully-proven epoch when nothing is proven', async () => { + describe('resolveLastFullyProvenEpoch', () => { + it('returns no fully-proven epoch when nothing is proven', async () => { l2BlockSource.getBlockNumber.mockResolvedValue(undefined); - await expect(proverNode.callComputeStartupState()).resolves.toEqual({ - startingBlock: BlockNumber(1), + await expect(proverNode.callResolveLastFullyProvenEpoch()).resolves.toEqual({ lastFullyProvenEpoch: undefined, }); }); - it('returns provenBlock+1 and no fully-proven epoch when the proven block has no archiver header', async () => { + it('returns no fully-proven epoch when the proven block has no archiver header', async () => { l2BlockSource.getBlockNumber.mockResolvedValue(BlockNumber(5)); l2BlockSource.getBlockData.mockResolvedValue(undefined); - await expect(proverNode.callComputeStartupState()).resolves.toEqual({ - startingBlock: BlockNumber(6), + await expect(proverNode.callResolveLastFullyProvenEpoch()).resolves.toEqual({ lastFullyProvenEpoch: undefined, }); }); - it('returns provenBlock+1 and provenEpoch when the proven block is the last of its epoch', async () => { + it('returns provenEpoch when the proven block is the last of its epoch', async () => { // epochDuration=1: slot 5 ⇒ epoch 5; next slot 6 ⇒ epoch 6 > 5 ⇒ last of epoch. l2BlockSource.getBlockNumber.mockResolvedValue(BlockNumber(5)); l2BlockSource.getBlockData.mockImplementation((q: any) => { @@ -500,13 +568,12 @@ describe('ProverNode', () => { } return Promise.resolve(undefined); }); - await expect(proverNode.callComputeStartupState()).resolves.toEqual({ - startingBlock: BlockNumber(6), + await expect(proverNode.callResolveLastFullyProvenEpoch()).resolves.toEqual({ lastFullyProvenEpoch: EpochNumber(5), }); }); - it('returns provenBlock+1 and provenEpoch via the isEpochComplete fallback when there is no next-block header', async () => { + it('returns provenEpoch via the isEpochComplete fallback when there is no next-block header', async () => { l2BlockSource.getBlockNumber.mockResolvedValue(BlockNumber(5)); l2BlockSource.getBlockData.mockImplementation((q: any) => { if (q.number === 5) { @@ -515,13 +582,12 @@ describe('ProverNode', () => { return Promise.resolve(undefined); }); l2BlockSource.isEpochComplete.mockResolvedValue(true); - await expect(proverNode.callComputeStartupState()).resolves.toEqual({ - startingBlock: BlockNumber(6), + await expect(proverNode.callResolveLastFullyProvenEpoch()).resolves.toEqual({ lastFullyProvenEpoch: EpochNumber(5), }); }); - it("returns the partially-proven epoch's first block and provenEpoch-1 when proven is mid-epoch", async () => { + it('returns provenEpoch-1 when proven is mid-epoch', async () => { // epochDuration=2: slot 5 ⇒ epoch 2; next slot 5 ⇒ same epoch ⇒ mid-epoch. const l1ConstantsTwo = { ...EmptyL1RollupConstants, epochDuration: 2, proofSubmissionEpochs: 1 }; l2BlockSource.getL1Constants.mockResolvedValue(l1ConstantsTwo); @@ -535,10 +601,8 @@ describe('ProverNode', () => { } return Promise.resolve(undefined); }); - l2BlockSource.getCheckpointsData.mockResolvedValue([{ startBlock: BlockNumber(3) } as any]); - await expect(proverNode.callComputeStartupState()).resolves.toEqual({ - startingBlock: BlockNumber(3), + await expect(proverNode.callResolveLastFullyProvenEpoch()).resolves.toEqual({ lastFullyProvenEpoch: EpochNumber(1), }); }); @@ -557,10 +621,8 @@ describe('ProverNode', () => { } return Promise.resolve(undefined); }); - l2BlockSource.getCheckpointsData.mockResolvedValue([{ startBlock: BlockNumber(1) } as any]); - await expect(proverNode.callComputeStartupState()).resolves.toEqual({ - startingBlock: BlockNumber(1), + await expect(proverNode.callResolveLastFullyProvenEpoch()).resolves.toEqual({ lastFullyProvenEpoch: undefined, }); }); @@ -677,8 +739,8 @@ class TestProverNode extends ProverNode { // ---------------- direct access for unit tests ---------------- - public callComputeStartupState() { - return this.computeStartupState(); + public callResolveLastFullyProvenEpoch() { + return this.resolveLastFullyProvenEpoch(); } public callIsEpochFullyProven(epoch: EpochNumber, l1Constants: { epochDuration: number }) { diff --git a/yarn-project/prover-node/src/prover-node.ts b/yarn-project/prover-node/src/prover-node.ts index 3c22c1c2a94d..dd30e810c428 100644 --- a/yarn-project/prover-node/src/prover-node.ts +++ b/yarn-project/prover-node/src/prover-node.ts @@ -12,6 +12,7 @@ import { getLastSiblingPath } from '@aztec/prover-client/helpers'; import { ChonkCache } from '@aztec/prover-client/orchestrator'; import { PublicProcessorFactory } from '@aztec/simulator/server'; import { + type L2BlockId, type L2BlockSource, L2BlockStream, type L2BlockStreamEvent, @@ -234,14 +235,22 @@ export class ProverNode implements L2BlockStreamEventHandler, ProverNodeApi, Tra await this.processCheckpointJump(event.checkpoint.number); break; case 'chain-pruned': - await this.handlePruneEvent(event.checkpointed.checkpoint); + await this.handlePruneEvent(event.block); break; case 'chain-proven': this.publishingService?.onChainProven(BlockNumber(event.block.number)); break; + // The proposed tip drives only the tips store's walk-back history (recorded below); the prover-node + // tracks checkpoints, not proposed blocks. `blocks-added` is never emitted in tips-only mode, and + // `chain-finalized` carries nothing the prover-node acts on. + case 'chain-proposed': case 'chain-finalized': case 'blocks-added': break; + default: { + const _: never = event; + break; + } } // Expiry is driven by the archiver's latest synced L2 slot await this.checkEpochExpiry(); @@ -329,6 +338,14 @@ export class ProverNode implements L2BlockStreamEventHandler, ProverNodeApi, Tra const registerData = await this.collectRegisterData(checkpoint, published.attestations); await this.checkpointStore.addOrUpdate(checkpoint, registerData); await this.sessionManager?.onCheckpointAdded(epochNumber); + + // Tips-only mode delivers no blocks, so record one witness per checkpointed block: a reorg into the checkpoint's + // range then prunes at the true divergence instead of the nearest sparse tip anchor. + await this.tipsStore.recordBlockHashes( + await Promise.all( + checkpoint.blocks.map(async block => ({ number: block.number, hash: (await block.header.hash()).toString() })), + ), + ); } /** @@ -357,15 +374,41 @@ export class ProverNode implements L2BlockStreamEventHandler, ProverNodeApi, Tra }; } - /** Mark every prover above the prune threshold as pruned and notify the session manager. */ - private async handlePruneEvent(prunedCheckpoint: { number: CheckpointNumber; hash: string }) { - this.log.warn(`Chain pruned to checkpoint ${prunedCheckpoint.number}`, { prunedCheckpoint }); - // Clamp the catch-up cursor down to the (post-prune) checkpointed tip so reprocessing resumes from the - // first checkpoint above the prune target rather than from a stale, now-orphaned cursor. - if (this.lastProcessedCheckpoint > prunedCheckpoint.number) { - this.lastProcessedCheckpoint = prunedCheckpoint.number; + /** + * Marks every prover orphaned by the prune as pruned, clamps the catch-up cursor below the prune target's + * checkpoint, and notifies the session manager. Keyed off the prune target block (the highest surviving block) + * rather than the source's checkpointed tip, which can sit above the target after a re-checkpoint and would leave + * orphaned provers canonical. Throws (rather than warning) if the cursor floor cannot be resolved, so the pass + * fails and the prune is retried next iteration. + */ + private async handlePruneEvent(prunedToBlock: L2BlockId) { + this.log.warn(`Chain pruned to block ${prunedToBlock.number}`, { prunedToBlock }); + + // Resolve the cursor floor BEFORE marking provers: markPrunedAboveBlock returns only newly-marked provers, so a + // throw after marking would leave a retry pass with nothing to act on. Resolving first means a throw leaves + // everything untouched and the next pass retries the whole handler (the tips cursor only advances on success). + let cursorFloor: CheckpointNumber; + if (prunedToBlock.number === 0) { + cursorFloor = CheckpointNumber.ZERO; + } else { + const targetData = await this.l2BlockSource.getBlockData({ number: prunedToBlock.number }); + if (targetData === undefined) { + throw new Error( + `No block data found for prune target block ${prunedToBlock.number}; cannot clamp checkpoint cursor`, + ); + } + // Clamp to `cpAtTarget - 1`: a mid-checkpoint target leaves that checkpoint partially orphaned and it must be + // reprocessed. Over-clamping merely re-registers a checkpoint (at-least-once by design — A-1041); under-clamping + // would permanently skip a rebuilt same-number checkpoint. + cursorFloor = CheckpointNumber(Math.max(0, Number(targetData.checkpointNumber) - 1)); + } + + const affected = this.checkpointStore.markPrunedAboveBlock(prunedToBlock.number); + + if (this.lastProcessedCheckpoint > cursorFloor) { + this.lastProcessedCheckpoint = cursorFloor; } - const affected = this.checkpointStore.markPrunedAfter(prunedCheckpoint.number); + if (affected.length === 0) { return; } @@ -476,12 +519,12 @@ export class ProverNode implements L2BlockStreamEventHandler, ProverNodeApi, Tra // Now that the store + manager exist, arm the live-state observable gauges. this.jobMetrics.observeState(this.checkpointStore, this.sessionManager); - const { startingBlock, lastFullyProvenEpoch } = await this.computeStartupState(); + const { lastFullyProvenEpoch } = await this.resolveLastFullyProvenEpoch(); this.lastExpiredEpoch = lastFullyProvenEpoch; this.lastProcessedCheckpoint = await this.computeStartingCheckpoint(lastFullyProvenEpoch); this.blockStream = new L2BlockStream(this.l2BlockSource, this.tipsStore, this, this.log, { pollIntervalMS: this.config.proverNodePollingIntervalMs, - startingBlock, + tipsOnly: true, }); this.blockStream.start(); @@ -630,38 +673,27 @@ export class ProverNode implements L2BlockStreamEventHandler, ProverNodeApi, Tra } /** - * Resolves the L2BlockStream's starting block and the last fully-proven epoch in one - * pass. The starting block is the first block of the next unproven epoch (or the start - * of the partially-proven epoch if the proven tip falls mid-epoch). The fully-proven - * epoch is `provenEpoch` when the proven tip is the last block of its epoch, otherwise - * `provenEpoch - 1`, or `undefined` if no block is proven yet. + * Resolves the last fully-proven epoch from L1 proven state, used to seed the catch-up cursor (via + * `computeStartingCheckpoint`) and `lastExpiredEpoch`. The fully-proven epoch is `provenEpoch` when the + * proven tip is the last block of its epoch, otherwise `provenEpoch - 1`, or `undefined` if no block is + * proven yet (so a restart reprocesses the partially-proven epoch rather than trusting a stale tip). */ - protected async computeStartupState(): Promise<{ - startingBlock: BlockNumber; - lastFullyProvenEpoch: EpochNumber | undefined; - }> { + protected async resolveLastFullyProvenEpoch(): Promise<{ lastFullyProvenEpoch: EpochNumber | undefined }> { const provenBlockNumber = await this.l2BlockSource.getBlockNumber({ tag: 'proven' }); if (!provenBlockNumber || provenBlockNumber <= 0) { - return { startingBlock: BlockNumber(1), lastFullyProvenEpoch: undefined }; + return { lastFullyProvenEpoch: undefined }; } const l1Constants = await this.getL1Constants(); const provenHeader = (await this.l2BlockSource.getBlockData({ number: BlockNumber(provenBlockNumber) }))?.header; if (!provenHeader) { - return { startingBlock: BlockNumber(provenBlockNumber + 1), lastFullyProvenEpoch: undefined }; + return { lastFullyProvenEpoch: undefined }; } const provenEpoch = getEpochAtSlot(provenHeader.getSlot(), l1Constants); if (await this.isProvenBlockLastOfItsEpoch(BlockNumber(provenBlockNumber), provenEpoch, l1Constants)) { - return { startingBlock: BlockNumber(provenBlockNumber + 1), lastFullyProvenEpoch: provenEpoch }; + return { lastFullyProvenEpoch: provenEpoch }; } - const epochCheckpoints = await this.l2BlockSource.getCheckpointsData({ epoch: provenEpoch }); - const firstBlockOfEpoch = - epochCheckpoints.length > 0 ? epochCheckpoints[0].startBlock : BlockNumber(provenBlockNumber); - this.log.info( - `Starting L2BlockStream at block ${firstBlockOfEpoch} (start of partially-proven epoch ${provenEpoch})`, - { provenBlockNumber, provenEpoch, firstBlockOfEpoch }, - ); const lastFullyProvenEpoch = provenEpoch > 0 ? EpochNumber(provenEpoch - 1) : undefined; - return { startingBlock: firstBlockOfEpoch, lastFullyProvenEpoch }; + return { lastFullyProvenEpoch }; } /** diff --git a/yarn-project/pxe/src/block_synchronizer/block_synchronizer.test.ts b/yarn-project/pxe/src/block_synchronizer/block_synchronizer.test.ts index c15898ba2109..9f8783e6881d 100644 --- a/yarn-project/pxe/src/block_synchronizer/block_synchronizer.test.ts +++ b/yarn-project/pxe/src/block_synchronizer/block_synchronizer.test.ts @@ -1,5 +1,4 @@ import { BlockNumber, CheckpointNumber } from '@aztec/foundation/branded-types'; -import { timesParallel } from '@aztec/foundation/collection'; import { Fr } from '@aztec/foundation/curves/bn254'; import type { AztecAsyncKVStore } from '@aztec/kv-store'; import { openTmpStore } from '@aztec/kv-store/lmdb-v2'; @@ -14,6 +13,7 @@ import { L2Block, type L2BlockId, type L2BlockStream, + type L2BlockStreamEvent, makeL2BlockId, makeL2CheckpointId, } from '@aztec/stdlib/block'; @@ -113,9 +113,36 @@ describe('BlockSynchronizer', () => { synchronizer = createSynchronizer(); }); - it('sets header from latest block', async () => { + // Builds the BlockData the node returns from getBlockData for a block (chain-proposed/checkpointed handlers + // fetch the tip header by hash through this path). + const blockData = async (block: L2Block): Promise => ({ + header: block.header, + archive: block.archive, + blockHash: await block.hash(), + checkpointNumber: block.checkpointNumber, + indexWithinCheckpoint: block.indexWithinCheckpoint, + }); + + // Mocks node.getBlockData to serve the given block only when queried by its own hash (mirrors the by-hash + // fetch the chain-proposed handler performs); any other query resolves undefined. + const serveBlockDataByHash = async (block: L2Block) => { + const data = await blockData(block); + aztecNode.getBlockData.mockImplementation(param => + Promise.resolve(param instanceof BlockHash && param.equals(data.blockHash) ? data : undefined), + ); + return data; + }; + + // Emits a chain-proposed tip event for the given block (the tip event PXE anchors on in proposed mode). + const proposedEvent = async (block: L2Block): Promise => ({ + type: 'chain-proposed', + block: makeL2BlockId(block.number, (await block.hash()).toString()), + }); + + it('sets header from the proposed tip', async () => { const block = await L2Block.random(BlockNumber(1)); - await synchronizer.handleBlockStreamEvent({ type: 'blocks-added', blocks: [block] }); + await serveBlockDataByHash(block); + await synchronizer.handleBlockStreamEvent(await proposedEvent(block)); const obtainedHeader = await anchorBlockStore.getBlockHeader(); expect(obtainedHeader.equals(block.header)).toBe(true); @@ -128,10 +155,10 @@ describe('BlockSynchronizer', () => { Promise.resolve(param instanceof BlockHash && param.equals(reorgResponse.hash) ? reorgResponse : undefined), ); - await synchronizer.handleBlockStreamEvent({ - type: 'blocks-added', - blocks: await timesParallel(5, i => L2Block.random(BlockNumber(i))), - }); + // Anchor sits above the prune target so the prune guard lets the rollback through. + const anchorBlock = await L2Block.random(BlockNumber(4)); + await anchorBlockStore.setHeader(anchorBlock.header); + await synchronizer.handleBlockStreamEvent({ type: 'chain-pruned', block: makeL2BlockId(reorgBlock.number, reorgResponse.hash.toString()), @@ -344,25 +371,56 @@ describe('BlockSynchronizer', () => { }); describe('syncChainTip config', () => { - it('updates anchor on blocks-added when syncChainTip is proposed (default)', async () => { + it('updates anchor on chain-proposed when syncChainTip is proposed (default)', async () => { synchronizer = createSynchronizer({ syncChainTip: 'proposed' }); const block = await L2Block.random(BlockNumber(1)); - await synchronizer.handleBlockStreamEvent({ type: 'blocks-added', blocks: [block] }); + await serveBlockDataByHash(block); + await synchronizer.handleBlockStreamEvent(await proposedEvent(block)); const obtainedHeader = await anchorBlockStore.getBlockHeader(); expect(obtainedHeader.equals(block.header)).toBe(true); }); - it('does not update anchor on blocks-added when syncChainTip is checkpointed', async () => { + it('throws and keeps the cursor retryable on chain-proposed when the block is missing by hash', async () => { + synchronizer = createSynchronizer({ syncChainTip: 'proposed' }); + + const initialBlock = await L2Block.random(BlockNumber(0)); + await anchorBlockStore.setHeader(initialBlock.header); + + const proposedBlock = await L2Block.random(BlockNumber(1)); + const proposedHash = await proposedBlock.hash(); + const event: L2BlockStreamEvent = { + type: 'chain-proposed', + block: makeL2BlockId(proposedBlock.number, proposedHash.toString()), + }; + + // The node cannot return the proposed block's data (node inconsistency). The handler must throw rather than + // warn-and-skip, so the tips-store cursor below it never advances and the next delivery can retry. + aztecNode.getBlockData.mockResolvedValue(undefined); + await expect(synchronizer.handleBlockStreamEvent(event)).rejects.toThrow(/not found/); + + // Anchor is left untouched and the proposed cursor did NOT advance: a quiet chain re-emits the same event. + expect((await anchorBlockStore.getBlockHeader()).equals(initialBlock.header)).toBe(true); + expect((await tipsStore.getL2Tips()).proposed.number).toBe(0); + + // The block becomes available; re-delivering the same event now lands the anchor and advances the cursor. + await serveBlockDataByHash(proposedBlock); + await synchronizer.handleBlockStreamEvent(event); + expect((await anchorBlockStore.getBlockHeader()).equals(proposedBlock.header)).toBe(true); + expect((await tipsStore.getL2Tips()).proposed.number).toBe(1); + }); + + it('does not update anchor on chain-proposed when syncChainTip is checkpointed', async () => { synchronizer = createSynchronizer({ syncChainTip: 'checkpointed' }); // First set a known anchor const initialBlock = await L2Block.random(BlockNumber(0)); await anchorBlockStore.setHeader(initialBlock.header); - // blocks-added should NOT update the anchor + // chain-proposed should NOT update the anchor in checkpointed mode const newBlock = await L2Block.random(BlockNumber(1)); - await synchronizer.handleBlockStreamEvent({ type: 'blocks-added', blocks: [newBlock] }); + await serveBlockDataByHash(newBlock); + await synchronizer.handleBlockStreamEvent(await proposedEvent(newBlock)); const obtainedHeader = await anchorBlockStore.getBlockHeader(); expect(obtainedHeader.equals(initialBlock.header)).toBe(true); @@ -425,9 +483,10 @@ describe('BlockSynchronizer', () => { it('does not update anchor on chain-checkpointed when syncChainTip is proposed', async () => { synchronizer = createSynchronizer({ syncChainTip: 'proposed' }); - // Set initial anchor via blocks-added + // Set initial anchor via the proposed tip const initialBlock = await L2Block.random(BlockNumber(1)); - await synchronizer.handleBlockStreamEvent({ type: 'blocks-added', blocks: [initialBlock] }); + await serveBlockDataByHash(initialBlock); + await synchronizer.handleBlockStreamEvent(await proposedEvent(initialBlock)); await synchronizer.handleBlockStreamEvent({ type: 'chain-checkpointed', @@ -508,4 +567,107 @@ describe('BlockSynchronizer', () => { expect(obtainedHeader.equals(anchorBlock.header)).toBe(true); }); }); + + // These exercise a real (non-mocked) L2BlockStream wired to the mocked node, so they cover the tips-only + // wiring end to end: the stream polls getChainTips, emits chain-proposed, and the synchronizer anchors by + // fetching the tip header by hash — never downloading block payloads via getBlocks. The local store is pre-seeded + // at block 1 (anchor + proposed tip) so the walk-back terminates there without touching genesis, and the source + // advances the proposed tip to block 2. + describe('tips-only stream sync', () => { + let realSynchronizer: BlockSynchronizer; + let block1: L2Block; + let block2: L2Block; + let block2Hash: BlockHash; + + // The L2Tips snapshot the stream reads: proposed at block 2, every confirmed tier still at block 1. + const tipsAtBlock2 = async () => { + const tip1 = { + block: makeL2BlockId(block1.number, (await block1.hash()).toString()), + checkpoint: makeL2CheckpointId(CheckpointNumber(1), Fr.random().toString()), + }; + return { + proposed: makeL2BlockId(block2.number, block2Hash.toString()), + checkpointed: tip1, + proven: tip1, + finalized: tip1, + }; + }; + + beforeEach(async () => { + block1 = await L2Block.random(BlockNumber(1)); + block2 = await L2Block.random(BlockNumber(2)); + block2Hash = await block2.hash(); + const block1Hash = await block1.hash(); + const block2Data = await blockData(block2); + + // Pre-seed the local store at block 1: the anchor header and the proposed-tip walk-back history. + await anchorBlockStore.setHeader(block1.header); + await tipsStore.handleBlockStreamEvent({ + type: 'chain-proposed', + block: makeL2BlockId(block1.number, block1Hash.toString()), + }); + + // The stream's source adapter resolves the walk-back hash for block 1 via node.getBlock({ number: 1 }), + // while the synchronizer fetches the proposed tip header for block 2 via node.getBlockData(block2Hash). + aztecNode.getChainTips.mockResolvedValue(await tipsAtBlock2()); + const block1Response = await blockResponse(block1); + getBlock.mockImplementation(param => + Promise.resolve( + typeof param === 'object' && 'number' in param && param.number === block1.number ? block1Response : undefined, + ), + ); + aztecNode.getBlockData.mockImplementation(param => + Promise.resolve(param instanceof BlockHash && param.equals(block2Hash) ? block2Data : undefined), + ); + + realSynchronizer = new BlockSynchronizer( + aztecNode, + store, + anchorBlockStore, + noteStore, + privateEventStore, + tipsStore, + contractSyncService, + { syncChainTip: 'proposed' }, + ); + }); + + afterEach(async () => { + await realSynchronizer.stop(); + }); + + it('anchors on the proposed tip via a by-hash fetch without downloading blocks', async () => { + await realSynchronizer.sync(); + + const obtainedHeader = await anchorBlockStore.getBlockHeader(); + expect(obtainedHeader.getBlockNumber()).toBe(2); + expect(obtainedHeader.equals(block2.header)).toBe(true); + expect(aztecNode.getBlocks).not.toHaveBeenCalled(); + }); + + it('re-emits the proposed tip on the next sync when the first anchor update throws', async () => { + // The anchor write throws on its first attempt to block 2 only. Because the tips store is advanced AFTER the + // anchor update, the throw leaves the proposed cursor at block 1, so the next sync re-emits chain-proposed for + // block 2 and the anchor lands. (The stream's work() loop logs the handler throw rather than rejecting sync().) + const originalSetHeader = anchorBlockStore.setHeader.bind(anchorBlockStore); + let firstAnchorToBlock2 = true; + anchorBlockStore.setHeader = async header => { + if (header.getBlockNumber() === 2 && firstAnchorToBlock2) { + firstAnchorToBlock2 = false; + throw new Error('transient anchor write failure'); + } + await originalSetHeader(header); + }; + + // First sync: the anchor write throws, so the anchor stays at block 1 and the proposed cursor does not advance. + await realSynchronizer.sync(); + expect((await anchorBlockStore.getBlockHeader()).getBlockNumber()).toBe(1); + expect((await tipsStore.getL2Tips()).proposed.number).toBe(1); + + // Second sync: chain-proposed is re-emitted for block 2, the anchor write succeeds, and the cursor advances. + await realSynchronizer.sync(); + expect((await anchorBlockStore.getBlockHeader()).equals(block2.header)).toBe(true); + expect((await tipsStore.getL2Tips()).proposed.number).toBe(2); + }); + }); }); diff --git a/yarn-project/pxe/src/block_synchronizer/block_synchronizer.ts b/yarn-project/pxe/src/block_synchronizer/block_synchronizer.ts index 3bfdc6141def..839344854f52 100644 --- a/yarn-project/pxe/src/block_synchronizer/block_synchronizer.ts +++ b/yarn-project/pxe/src/block_synchronizer/block_synchronizer.ts @@ -37,21 +37,20 @@ export class BlockSynchronizer implements L2BlockStreamEventHandler { bindings?: LoggerBindings, ) { this.log = createLogger('pxe:block_synchronizer', bindings); - this.blockStream = this.createBlockStream(config); + this.blockStream = this.createBlockStream(); this.eventQueue.start(); } - protected createBlockStream(config: Partial): L2BlockStream { + protected createBlockStream(): L2BlockStream { return new L2BlockStream( blockStreamSourceFromAztecNode(this.node), this.l2TipsStore, this, createLogger('pxe:block_stream', this.log.getBindings()), { - batchSize: config.l2BlockBatchSize, - // Skipping finalized blocks makes us sync much faster - we only need to download blocks other than the latest one - // in order to detect reorgs, and there can be no reorgs on finalized block, making this safe. - skipFinalized: true, + // PXE only tracks chain tips (it anchors on tip events and never replays payloads), so the stream runs + // in tips-only mode: no block downloads, just the tip events that move the anchor and detect reorgs. + tipsOnly: true, }, ); } @@ -62,16 +61,30 @@ export class BlockSynchronizer implements L2BlockStreamEventHandler { } private async doHandleBlockStreamEvent(event: L2BlockStreamEvent): Promise { - await this.l2TipsStore.handleBlockStreamEvent(event); - switch (event.type) { - case 'blocks-added': { + case 'chain-proposed': { if (this.config.syncChainTip === undefined || this.config.syncChainTip === 'proposed') { - const lastBlock = event.blocks.at(-1)!; - await this.updateAnchorBlockHeader(lastBlock.header); + // Fetch the proposed tip header by hash. By-hash is safer than by-number against a same-height reorg. + const block = await this.node.getBlockData(BlockHash.fromString(event.block.hash)); + if (!block) { + // The node served a proposed tip whose block data it cannot return — a node inconsistency, since the + // stream events and the block data come from the same node (same reasoning as the chain-pruned throw + // below). Throwing here propagates before the tips-store cursor advances, so the cursor stays put and the + // next sync re-emits chain-proposed (at-least-once). Were we to warn-and-skip, the cursor would advance and + // a quiet chain would never re-emit, leaving the anchor stale indefinitely. + throw new Error( + `Block header for proposed block ${event.block.number} and hash ${event.block.hash} not found. This ` + + `likely indicates a bug in the node, as we receive block stream events and fetch block headers from ` + + `the same node.`, + ); + } + await this.updateAnchorBlockHeader(block.header); } break; } + // Never emitted in tips-only mode; PXE anchors on chain-proposed instead. Kept for union exhaustiveness. + case 'blocks-added': + break; case 'chain-checkpointed': { if (this.config.syncChainTip === 'checkpointed') { // Fetch the checkpointed tip header by hash. By-hash is safer than by-number against a @@ -118,7 +131,7 @@ export class BlockSynchronizer implements L2BlockStreamEventHandler { `Ignoring prune event to block ${event.block.number} greater than current anchor block ${currentAnchorBlockNumber}`, { pruneEvent: event, currentAnchorBlockHeader: currentAnchorBlockHeader.toInspect() }, ); - return; + break; } this.log.warn(`Pruning data after block ${event.block.number} due to reorg`, { pruneBlock: event.block }); @@ -143,7 +156,16 @@ export class BlockSynchronizer implements L2BlockStreamEventHandler { }); break; } + default: { + const _: never = event; + break; + } } + + // Advance the tips store cursor only after the anchor update / prune rollback above succeeds. If that work + // throws, the cursor stays put and the stream re-emits this event on the next sync pass (at-least-once), + // matching the prover-node's handle-first/tips-last ordering. The SerialQueue makes this reorder safe. + await this.l2TipsStore.handleBlockStreamEvent(event); } /** Updates the anchor block header to the target block */ diff --git a/yarn-project/stdlib/src/block/l2_block_stream/interfaces.ts b/yarn-project/stdlib/src/block/l2_block_stream/interfaces.ts index 6ed09b019739..a9caf3fa03a7 100644 --- a/yarn-project/stdlib/src/block/l2_block_stream/interfaces.ts +++ b/yarn-project/stdlib/src/block/l2_block_stream/interfaces.ts @@ -45,6 +45,16 @@ export type L2BlockStreamEvent = type: 'blocks-added'; blocks: L2Block[]; } + | /** + * Reports the new proposed tip of the chain. Emitted once per sync pass when the source's proposed tip differs + * from the pre-pass local one (downloads, a prune, or a thin tip movement). Carries only the block id; in block + * mode the corresponding payloads arrive via preceding `blocks-added` events, while in tips-only mode this is the + * sole signal that the proposed tip moved. Consumers that only track the proposed tip can ignore `blocks-added` + * entirely and anchor on this event instead. + */ { + type: 'chain-proposed'; + block: L2BlockId; + } | /** * Reports a new checkpointed tip. Emitted at most once per sync pass when the source's checkpointed tip * leads the local one. Carries only the block + checkpoint ids; consumers that need the full checkpoint @@ -80,4 +90,11 @@ export type L2BlockStreamEvent = export type L2TipsStore = L2BlockStreamEventHandler & L2TipsProvider & - Pick; + Pick & { + /** + * Records `(number → hash)` witnesses into the walk-back hash index without moving any tip cursor. Consumers that + * materialize per-height state should record a witness for each height they materialize, so a reorg below the + * nearest sparse anchor does not produce an over-deep prune event. See {@link L2TipsStoreBase.recordBlockHashes}. + */ + recordBlockHashes(blocks: L2BlockId[]): Promise; + }; diff --git a/yarn-project/stdlib/src/block/l2_block_stream/l2_block_stream.test.ts b/yarn-project/stdlib/src/block/l2_block_stream/l2_block_stream.test.ts index 28f7cb9d76cf..73d2eca060ae 100644 --- a/yarn-project/stdlib/src/block/l2_block_stream/l2_block_stream.test.ts +++ b/yarn-project/stdlib/src/block/l2_block_stream/l2_block_stream.test.ts @@ -27,12 +27,15 @@ describe('L2BlockStream', () => { const makeHash = (number: number) => new Fr(number).toString(); + // `hash` is a shared function reference (not a per-call closure) so two makeBlock(n) objects compare equal under + // toEqual; called as a method it hashes `this.number`. The completion check in downloadBlocks calls it. const makeBlock = (number: number) => ({ number: BlockNumber(number), checkpointNumber: CheckpointNumber(number), indexWithinCheckpoint: 0, - }) as L2Block; + hash: blockHashFromNumber, + }) as unknown as L2Block; const makeBlockData = (number: number, checkpointNum: number): BlockData => ({ @@ -113,6 +116,7 @@ describe('L2BlockStream', () => { await blockStream.work(); expect(handler.events).toEqual([ { type: 'blocks-added', blocks: times(5, i => makeBlock(i + 1)) }, + { type: 'chain-proposed', block: makeBlockId(5) }, ] satisfies L2BlockStreamEvent[]); }); @@ -124,6 +128,7 @@ describe('L2BlockStream', () => { expect(blockSource.getBlocks).toHaveBeenCalledWith({ from: BlockNumber(11), limit: 5 }); expect(handler.events).toEqual([ { type: 'blocks-added', blocks: times(5, i => makeBlock(i + 11)) }, + { type: 'chain-proposed', block: makeBlockId(15) }, ] satisfies L2BlockStreamEvent[]); }); @@ -132,13 +137,14 @@ describe('L2BlockStream', () => { await blockStream.work(); expect(blockSource.getBlocks).toHaveBeenCalledTimes(5); - expect(handler.callCount).toEqual(5); + expect(handler.callCount).toEqual(6); expect(handler.events).toEqual([ { type: 'blocks-added', blocks: times(10, i => makeBlock(i + 1)) }, { type: 'blocks-added', blocks: times(10, i => makeBlock(i + 11)) }, { type: 'blocks-added', blocks: times(10, i => makeBlock(i + 21)) }, { type: 'blocks-added', blocks: times(10, i => makeBlock(i + 31)) }, { type: 'blocks-added', blocks: times(5, i => makeBlock(i + 41)) }, + { type: 'chain-proposed', block: makeBlockId(45) }, ] satisfies L2BlockStreamEvent[]); }); @@ -162,8 +168,9 @@ describe('L2BlockStream', () => { handler.throwing = false; await blockStream.work(); - expect(handler.callCount).toEqual(6); - expect(handler.events).toHaveLength(5); + // 5 blocks-added batches + 1 chain-proposed, plus the 1 throwing call from the first pass. + expect(handler.callCount).toEqual(7); + expect(handler.events).toHaveLength(6); }); it('handles a reorg and requests blocks from new tip', async () => { @@ -179,6 +186,7 @@ describe('L2BlockStream', () => { expect(handler.events).toEqual([ { type: 'chain-pruned', block: makeBlockId(36), checkpointed: makeTipId(0), proven: makeTipId(0) }, { type: 'blocks-added', blocks: times(9, i => makeBlock(i + 37)) }, + { type: 'chain-proposed', block: makeBlockId(45) }, ] satisfies L2BlockStreamEvent[]); }); @@ -191,6 +199,7 @@ describe('L2BlockStream', () => { await blockStream.work(); expect(handler.events).toEqual([ { type: 'blocks-added', blocks: times(5, i => makeBlock(i + 41)) }, + { type: 'chain-proposed', block: makeBlockId(45) }, { type: 'chain-proven', block: makeBlockId(40), checkpoint: makeCheckpointId(40) }, { type: 'chain-finalized', block: makeBlockId(35), checkpoint: makeCheckpointId(35) }, ] satisfies L2BlockStreamEvent[]); @@ -205,6 +214,7 @@ describe('L2BlockStream', () => { expect(handler.events).toEqual([ { type: 'blocks-added', blocks: times(5, i => makeBlock(i + 1)) }, + { type: 'chain-proposed', block: makeBlockId(5) }, checkpointedEvent(5), ]); // No checkpoint payloads are fetched anymore. @@ -217,9 +227,10 @@ describe('L2BlockStream', () => { await blockStream.work(); - // Download all 5 blocks, then a single checkpointed event for checkpoint 3. + // Download all 5 blocks, then chain-proposed for the proposed tip (5), then a single checkpointed event for 3. expect(handler.events).toEqual([ { type: 'blocks-added', blocks: times(5, i => makeBlock(i + 1)) }, + { type: 'chain-proposed', block: makeBlockId(5) }, checkpointedEvent(3), ]); }); @@ -347,17 +358,20 @@ describe('L2BlockStream', () => { await blockStream.work(); expect(handler.events).toEqual([ { type: 'blocks-added', blocks: times(6, i => makeBlock(i + 30)) }, + { type: 'chain-proposed', block: makeBlockId(35) }, checkpointedEvent(30), { type: 'chain-proven', block: makeBlockId(25), checkpoint: makeCheckpointId(25) }, { type: 'chain-finalized', block: makeBlockId(10), checkpoint: makeCheckpointId(10) }, ]); handler.clearEvents(); - // And then we reorg + // And then we reorg. The prune drops the proposed tip to 25, so chain-proposed re-fires the new tip (the + // pre-pass baseline was 35). setRemoteTips(25, 25, 25, 10); await blockStream.work(); expect(handler.events).toEqual([ { type: 'chain-pruned', block: makeBlockId(25), checkpointed: makeTipId(25), proven: makeTipId(25) }, + { type: 'chain-proposed', block: makeBlockId(25) }, ]); }); @@ -375,6 +389,7 @@ describe('L2BlockStream', () => { await blockStream.work(); expect(handler.events).toEqual([ { type: 'chain-pruned', block: makeBlockId(6), checkpointed: makeTipId(5), proven: makeTipId(0) }, + { type: 'chain-proposed', block: makeBlockId(6) }, ]); handler.clearEvents(); @@ -499,7 +514,11 @@ describe('L2BlockStream', () => { await blockStream.work(); - expect(handler.events).toEqual([{ type: 'blocks-added', blocks: times(6, i => makeBlock(i + 1)) }]); + // chain-proposed still fires (it is not a checkpoint event); only chain-checkpointed is suppressed. + expect(handler.events).toEqual([ + { type: 'blocks-added', blocks: times(6, i => makeBlock(i + 1)) }, + { type: 'chain-proposed', block: makeBlockId(6) }, + ]); }); it('still emits prune events but no checkpoint events', async () => { @@ -519,6 +538,7 @@ describe('L2BlockStream', () => { expect(handler.events).toEqual([ { type: 'chain-pruned', block: makeBlockId(3), checkpointed: makeTipId(3), proven: makeTipId(0) }, + { type: 'chain-proposed', block: makeBlockId(3) }, ]); }); @@ -529,6 +549,7 @@ describe('L2BlockStream', () => { expect(handler.events).toEqual([ { type: 'blocks-added', blocks: times(9, i => makeBlock(i + 1)) }, + { type: 'chain-proposed', block: makeBlockId(9) }, { type: 'chain-proven', block: makeBlockId(6), checkpoint: makeCheckpointId(6) }, { type: 'chain-finalized', block: makeBlockId(3), checkpoint: makeCheckpointId(3) }, ]); @@ -561,6 +582,7 @@ describe('L2BlockStream', () => { // Instead of fetching the next local block (6), we skip ahead to the latest finalized (35) and go from there. expect(handler.events).toEqual([ { type: 'blocks-added', blocks: times(6, i => makeBlock(i + 35)) }, + { type: 'chain-proposed', block: makeBlockId(40) }, { type: 'chain-proven', block: makeBlockId(38), checkpoint: makeCheckpointId(38) }, { type: 'chain-finalized', block: makeBlockId(35), checkpoint: makeCheckpointId(35) }, ] satisfies L2BlockStreamEvent[]); @@ -575,9 +597,11 @@ describe('L2BlockStream', () => { await blockStream.work(); - // proven and finalized tips already match the source on (number, hash), so only new blocks are emitted. + // proven and finalized tips already match the source on (number, hash), so only new blocks + the proposed tip + // are emitted. expect(handler.events).toEqual([ { type: 'blocks-added', blocks: times(2, i => makeBlock(i + 39)) }, + { type: 'chain-proposed', block: makeBlockId(40) }, ] satisfies L2BlockStreamEvent[]); }); }); @@ -600,9 +624,12 @@ describe('L2BlockStream', () => { await blockStream.work(); - // All 5 blocks are synced and no checkpoint events are emitted. - expect(handler.events).toEqual([{ type: 'blocks-added', blocks: times(5, i => makeBlock(i + 1)) }]); - expect(handler.events.every(e => e.type === 'blocks-added')).toBe(true); + // All 5 blocks are synced and no checkpoint events are emitted (chain-proposed is not a checkpoint event). + expect(handler.events).toEqual([ + { type: 'blocks-added', blocks: times(5, i => makeBlock(i + 1)) }, + { type: 'chain-proposed', block: makeBlockId(5) }, + ]); + expect(handler.events.some(e => e.type === 'chain-checkpointed')).toBe(false); }); it('surfaces a loud error when checkpoint emission is enabled without a checkpointed tip', async () => { @@ -619,8 +646,859 @@ describe('L2BlockStream', () => { ); }); }); + + describe('tipsOnly mode', () => { + let localData: TestL2TipsMemoryStore; + let handler: TestL2BlockStreamEventHandler; + let blockStream: TestL2BlockStream; + + beforeEach(() => { + localData = new TestL2TipsMemoryStore(); + handler = new TestL2BlockStreamEventHandler(localData); + blockStream = new TestL2BlockStream(blockSource, localData, handler, undefined, { tipsOnly: true }); + }); + + it('emits all tier events from the snapshot with zero getBlocks calls on a fresh store', async () => { + setRemoteTips(9, 9, 6, 3); + + await blockStream.work(); + + // No blocks-added, no download: only the proposed tip + the three tier events, in highest-to-lowest order. + expect(handler.events).toEqual([ + { type: 'chain-proposed', block: makeBlockId(9) }, + checkpointedEvent(9), + { type: 'chain-proven', block: makeBlockId(6), checkpoint: makeCheckpointId(6) }, + { type: 'chain-finalized', block: makeBlockId(3), checkpoint: makeCheckpointId(3) }, + ]); + expect(blockSource.getBlocks).not.toHaveBeenCalled(); + }); + + it('does not re-emit chain-proposed once the local proposed tip matches the source', async () => { + setRemoteTips(9, 9, 6, 3); + await blockStream.work(); + handler.clearEvents(); + + // Same snapshot: nothing moved, so no events at all. + await blockStream.work(); + expect(handler.events).toEqual([]); + expect(blockSource.getBlocks).not.toHaveBeenCalled(); + }); + + it('prunes to the highest recorded matching height via sparse history (over-deep is expected)', async () => { + // Poll 1 records a proposed anchor at block 5; poll 2 advances it to block 10. No blocks-added ever runs, so + // heights 6-9 are never recorded: the only sparse anchors are 5 and 10 (plus genesis). + setRemoteTips(5); + await blockStream.work(); + setRemoteTips(10); + await blockStream.work(); + handler.clearEvents(); + + // The source reorgs: blocks 4+ changed, the new fork ends at proposed=4. Blocks 1-3 keep their hashes (common + // ancestor); block 4 onward differs, and the source no longer serves 5-10. + blockSource.getBlockData.mockImplementation(query => { + if (!('number' in query)) { + return Promise.resolve(undefined); + } + if (query.number > 4) { + return Promise.resolve(undefined); + } + // Blocks 1-3 are the common ancestor (same hash); block 4 is on the new fork (different hash). + const checkpointNum = query.number; + const data = + query.number === 4 + ? ({ header: { hash: () => Promise.resolve(new BlockHash(new Fr(4000))) } } as unknown as BlockData) + : makeBlockData(query.number, checkpointNum); + return Promise.resolve(data); + }); + blockSource.getL2Tips.mockResolvedValue({ + proposed: { number: BlockNumber(4), hash: new Fr(4000).toString() }, + checkpointed: makeTipId(4), + proven: makeTipId(0), + finalized: makeTipId(0), + }); + + await blockStream.work(); + + // The walk-back cannot stop at the recorded anchor 5 or 10 (the source dropped them) and the heights between + // are unrecorded, so it walks down to genesis: the prune target is over-deep (block 0) but never ABOVE the + // true divergence (block 3). Over-deep prunes are tolerated by construction. + const pruneEvents = handler.events.filter(e => e.type === 'chain-pruned'); + expect(pruneEvents).toHaveLength(1); + expect(pruneEvents[0]).toMatchObject({ type: 'chain-pruned' }); + expect( + (pruneEvents[0] as Extract).block.number, + ).toBeLessThanOrEqual(3); + expect(blockSource.getBlocks).not.toHaveBeenCalled(); + }); + + it('throws on construction when combined with startingBlock, batchSize, or skipFinalized', () => { + expect( + () => new TestL2BlockStream(blockSource, localData, handler, undefined, { tipsOnly: true, startingBlock: 3 }), + ).toThrow(/tipsOnly is incompatible/); + expect( + () => new TestL2BlockStream(blockSource, localData, handler, undefined, { tipsOnly: true, batchSize: 10 }), + ).toThrow(/tipsOnly is incompatible/); + expect( + () => + new TestL2BlockStream(blockSource, localData, handler, undefined, { tipsOnly: true, skipFinalized: true }), + ).toThrow(/tipsOnly is incompatible/); + }); + }); + + describe('walk-back missing-local-hash regression', () => { + // A missing LOCAL hash must compare UNEQUAL, so the walk-back continues past sparse gaps to the true divergence + // (or genesis) and the prune target never lands ABOVE the divergence. Uses the memory store for genuinely sparse + // history (heights never written have no hash). + let localData: TestL2TipsMemoryStore; + let handler: TestL2BlockStreamEventHandler; + + beforeEach(() => { + localData = new TestL2TipsMemoryStore(); + handler = new TestL2BlockStreamEventHandler(localData); + }); + + it('walks past both-undefined heights when the source proposed tip dropped below the local tip', async () => { + // Block-mode sync to proposed=10 (records dense hashes 1-10). + const blockStream = new TestL2BlockStream(blockSource, localData, handler, undefined, { + batchSize: 10, + ignoreCheckpoints: true, + }); + setRemoteTips(10); + await blockStream.work(); + handler.clearEvents(); + + // Now drop a gap into the local history: delete the hash for block 7 (simulating sparse history), so the + // walk-back hits a both-undefined height inside (sourceProposed, localProposed]. + const store = localData as unknown as { blockHashes: Map }; + store.blockHashes.delete(7); + + // Source reorgs below the local proposed tip: blocks 4+ changed, new proposed=6 on a fork. Block 4 onward has + // new hashes; the source no longer serves blocks above 6. True divergence is after block 3. + blockSource.getBlockData.mockImplementation(query => { + if (!('number' in query) || query.number > 6) { + return Promise.resolve(undefined); + } + const forked = query.number >= 4; + return Promise.resolve({ + header: { hash: () => Promise.resolve(new BlockHash(new Fr(forked ? query.number + 1000 : query.number))) }, + } as unknown as BlockData); + }); + blockSource.getL2Tips.mockResolvedValue({ + proposed: { number: BlockNumber(6), hash: new Fr(1006).toString() }, + checkpointed: makeTipId(0), + proven: makeTipId(0), + finalized: makeTipId(0), + }); + + await blockStream.work(); + + // The walk continues past the missing height (7) and the forked heights; the prune target lands at or below the + // true divergence (block 3), never above it. + const pruneEvents = handler.events.filter( + (e): e is Extract => e.type === 'chain-pruned', + ); + expect(pruneEvents).toHaveLength(1); + expect(pruneEvents[0].block.number).toBeLessThanOrEqual(3); + }); + + // Seeding the walk-back cache with a stale tier tip poisons the walk: the snapshot's checkpointed tip sits at a + // reorged height carrying the OLD-fork hash, so it equals the local old-fork hash there and fakes agreement, + // stopping the walk ABOVE the true divergence (an under-deep prune). Only the proposed tip may seed the cache. + it('does not stop the walk-back at a stale tier-tip seed when the source reorged after the snapshot', async () => { + // Block-mode sync to proposed=10 (records dense old-fork hashes 1-10). + const blockStream = new TestL2BlockStream(blockSource, localData, handler, undefined, { + batchSize: 10, + ignoreCheckpoints: true, + }); + setRemoteTips(10); + await blockStream.work(); + handler.clearEvents(); + + // The source reorged after the snapshot. Live getBlockData reflects the post-reorg chain: heights 6-11 carry + // new-fork hashes, heights <= 5 keep the old (shared) hashes. True divergence is after block 5. + blockSource.getBlockData.mockImplementation(query => { + if (!('number' in query) || query.number > 11) { + return Promise.resolve(undefined); + } + const forked = query.number >= 6; + return Promise.resolve({ + header: { hash: () => Promise.resolve(new BlockHash(new Fr(forked ? query.number + 1000 : query.number))) }, + } as unknown as BlockData); + }); + + // FIRST getL2Tips (the pass snapshot) reports proposed=11 (old-fork hash, so the walk's first comparison at 10 + // misses the cache) and checkpointed=8 with the OLD-fork hash — the stale tier seed that, if cached, equals the + // local old-fork hash at the reorged height 8 and stops the walk there. The post-divergence re-read returns the + // fresh post-reorg tips (proposed=6 on the new fork). + let getTipsCall = 0; + blockSource.getL2Tips.mockImplementation(() => { + getTipsCall++; + return Promise.resolve( + getTipsCall === 1 + ? { + proposed: { number: BlockNumber(11), hash: makeHash(11) }, + checkpointed: makeTipId(8), + proven: makeTipId(8), + finalized: makeTipId(8), + } + : { + proposed: { number: BlockNumber(6), hash: new Fr(1006).toString() }, + checkpointed: makeTipId(0), + proven: makeTipId(0), + finalized: makeTipId(0), + }, + ); + }); + + await blockStream.work(); + + // The walk must reach the true divergence at block 5, NOT stop at the poisoned tier seed (block 8). + const pruneEvents = handler.events.filter( + (e): e is Extract => e.type === 'chain-pruned', + ); + expect(pruneEvents).toHaveLength(1); + expect(pruneEvents[0].block.number).toBeLessThanOrEqual(5); + }); + }); + + describe('block-mode chain-proposed', () => { + let localData: TestL2BlockStreamLocalDataProvider; + let handler: TestL2BlockStreamEventHandler; + let blockStream: TestL2BlockStream; + + beforeEach(() => { + localData = new TestL2BlockStreamLocalDataProvider(); + handler = new TestL2BlockStreamEventHandler(); + blockStream = new TestL2BlockStream(blockSource, localData, handler, undefined, { batchSize: 10 }); + }); + + it('emits chain-proposed after blocks-added carrying the snapshot proposed tip', async () => { + setRemoteTips(5, 5); + + await blockStream.work(); + + expect(handler.events).toEqual([ + { type: 'blocks-added', blocks: times(5, i => makeBlock(i + 1)) }, + { type: 'chain-proposed', block: makeBlockId(5) }, + checkpointedEvent(5), + ]); + }); + + it('does not emit chain-proposed when the proposed tip did not change', async () => { + setRemoteTips(5, 5); + localData.setProposed(5); + localData.setCheckpointed(5, 5); + + await blockStream.work(); + + expect(handler.events.filter(e => e.type === 'chain-proposed')).toEqual([]); + }); + + // Pre-pass baseline regression: on a prune-then-download pass the post-prune local proposed tip catches up to the + // source, so comparing against a post-prune re-read would suppress chain-proposed exactly when the tip moved most. + // Against the pre-pass baseline it fires. + it('emits chain-proposed on a prune-then-download pass against the pre-pass baseline', async () => { + const store = new TestL2TipsMemoryStore(); + const storeHandler = new TestL2BlockStreamEventHandler(store); + const stream = new TestL2BlockStream(blockSource, store, storeHandler, undefined, { + batchSize: 10, + ignoreCheckpoints: true, + }); + + // Sync to proposed=40 on the old fork. + setRemoteTips(40); + await stream.work(); + handler.clearEvents(); + storeHandler.clearEvents(); + + // Reorg: blocks 37-40 changed on the source (new fork), new proposed=45. The store still holds 37-40 with the + // old hashes, so the walk-back detects divergence after block 36 and prunes, then downloads 37-45. + blockSource.getBlockData.mockImplementation(query => { + if (!('number' in query) || query.number > 45) { + return Promise.resolve(undefined); + } + const forked = query.number >= 37; + return Promise.resolve({ + header: { hash: () => Promise.resolve(new BlockHash(new Fr(forked ? query.number + 1000 : query.number))) }, + } as unknown as BlockData); + }); + blockSource.getBlocks.mockImplementation((query: BlocksQuery) => + 'from' in query + ? Promise.resolve( + compactArray( + times(query.limit, i => { + const n = query.from + i; + if (n > 45) { + return undefined; + } + // Forked blocks (37+) carry new hashes so the completion check passes against the new snapshot. + return n >= 37 ? makeForkedBlock(n) : makeBlock(n); + }), + ), + ) + : Promise.resolve([]), + ); + blockSource.getL2Tips.mockResolvedValue({ + proposed: { number: BlockNumber(45), hash: new Fr(1045).toString() }, + checkpointed: makeTipId(0), + proven: makeTipId(0), + finalized: makeTipId(0), + }); + + await stream.work(); + + // chain-proposed fires (the tip moved 40 -> 45) even though the downloads pushed the local proposed to 45 by + // the time reconciliation runs. + const proposedEvents = storeHandler.events.filter(e => e.type === 'chain-proposed'); + expect(proposedEvents).toEqual([ + { type: 'chain-proposed', block: { number: BlockNumber(45), hash: new Fr(1045).toString() } }, + ]); + const pruneEvents = storeHandler.events.filter(e => e.type === 'chain-pruned'); + expect(pruneEvents).toHaveLength(1); + }); + }); + + describe('pass atomicity (block mode)', () => { + let localData: TestL2TipsMemoryStore; + let handler: TestL2BlockStreamEventHandler; + let blockStream: TestL2BlockStream; + + beforeEach(() => { + localData = new TestL2TipsMemoryStore(); + handler = new TestL2BlockStreamEventHandler(localData); + blockStream = new TestL2BlockStream(blockSource, localData, handler, undefined, { + batchSize: 10, + ignoreCheckpoints: true, + }); + }); + + // (a) Empty getBlocks mid-plan: the source advertises a proposed tip it cannot deliver. No tier events fire; the + // next pass (with a deliverable tip) reconciles. + it('skips reconciliation when getBlocks returns empty below the target, then recovers next pass', async () => { + // Source advertises proposed=10 and finalized=8, but getBlocks delivers nothing (blocks not actually available). + blockSource.getL2Tips.mockResolvedValue({ + proposed: { number: BlockNumber(10), hash: makeHash(10) }, + checkpointed: makeTipId(0), + proven: makeTipId(0), + finalized: makeTipId(8), + }); + blockSource.getBlocks.mockResolvedValue([]); + + await blockStream.work(); + + // No blocks delivered ⇒ plan incomplete ⇒ no finalized (or any tier) event this pass. + expect(handler.events.filter(e => e.type === 'chain-finalized')).toEqual([]); + expect(handler.events.filter(e => e.type === 'chain-proposed')).toEqual([]); + + // Next pass the source can deliver: reconciliation runs and the tiers catch up. + blockSource.getBlocks.mockImplementation((query: BlocksQuery) => + 'from' in query + ? Promise.resolve( + compactArray(times(query.limit, i => (query.from + i > 10 ? undefined : makeBlock(query.from + i)))), + ) + : Promise.resolve([]), + ); + latest = 10; + handler.clearEvents(); + + await blockStream.work(); + expect(handler.events.filter(e => e.type === 'chain-finalized')).toEqual([ + { type: 'chain-finalized', block: makeBlockId(8), checkpoint: makeCheckpointId(8) }, + ]); + }); + + // (b) Loop completes but the last block's hash differs from the snapshot proposed hash: a same-height fork swap + // happened mid-pass. No tier events. + it('skips reconciliation when the delivered proposed-block hash differs from the snapshot', async () => { + // Snapshot proposed hash is makeHash(5), but getBlocks delivers a block 5 carrying a different (forked) hash. + blockSource.getL2Tips.mockResolvedValue({ + proposed: { number: BlockNumber(5), hash: makeHash(5) }, + checkpointed: makeTipId(0), + proven: makeTipId(0), + finalized: makeTipId(3), + }); + blockSource.getBlocks.mockResolvedValue(times(5, i => makeForkedBlock(i + 1))); + + await blockStream.work(); + + // The delivered block-5 hash (forked) != snapshot proposed hash ⇒ plan incomplete ⇒ no tier events. + expect(handler.events.filter(e => e.type === 'chain-finalized')).toEqual([]); + expect(handler.events.filter(e => e.type === 'chain-proposed')).toEqual([]); + // blocks-added still emitted (it only populates hash history). + expect(handler.events.filter(e => e.type === 'blocks-added')).toHaveLength(1); + }); + + // (c) startingBlock fast-forward past the tip still reconciles (A-1061 regression): the loop never runs, the plan + // is trivially complete, and the snapshot tiers are emitted. + it('reconciles when startingBlock fast-forwards past the proposed tip (A-1061)', async () => { + const freshStore = new TestL2TipsMemoryStore(); + const freshHandler = new TestL2BlockStreamEventHandler(freshStore); + const stream = new TestL2BlockStream(blockSource, freshStore, freshHandler, undefined, { + startingBlock: 40, + ignoreCheckpoints: true, + }); + setRemoteTips(35, 0, 25, 10); + + await stream.work(); + + // The download loop never runs (startingBlock 40 > proposed 35), yet proven/finalized still reconcile. + expect(freshHandler.events.filter(e => e.type === 'chain-proven')).toEqual([ + { type: 'chain-proven', block: makeBlockId(25), checkpoint: makeCheckpointId(25) }, + ]); + expect(freshHandler.events.filter(e => e.type === 'chain-finalized')).toEqual([ + { type: 'chain-finalized', block: makeBlockId(10), checkpoint: makeCheckpointId(10) }, + ]); + }); + }); + + describe('prune payload freshness', () => { + // The walk-back detection uses live getBlockData reads; the prune event's clamp payload must come from a re-read + // taken AFTER divergence is detected, not from the (now stale) pass snapshot. + let localData: TestL2TipsMemoryStore; + let handler: TestL2BlockStreamEventHandler; + + beforeEach(() => { + localData = new TestL2TipsMemoryStore(); + handler = new TestL2BlockStreamEventHandler(localData); + }); + + it('carries the re-read checkpointed/proven tips on the prune event, not the snapshot ones', async () => { + const blockStream = new TestL2BlockStream(blockSource, localData, handler, undefined, { batchSize: 10 }); + // Sync to proposed=10, checkpointed=10, proven=10, finalized=0. + setRemoteTips(10, 10, 10, 0); + await blockStream.work(); + handler.clearEvents(); + + // The reorg drops the source to proposed=6 (blocks 7-10 gone; 1-6 unchanged). Between the pass snapshot and the + // walk-back the source's confirmed tips move: the FIRST getL2Tips (snapshot) still reports the proven tip behind + // at block 4; the SECOND (post-divergence re-read) reports it caught up to 6. The stream must clamp the prune + // event from the re-read, not the snapshot, so p2p's isEpochPrune sees the checkpoint id AFTER the prune. + let getTipsCall = 0; + blockSource.getL2Tips.mockImplementation(() => { + getTipsCall++; + if (getTipsCall === 1) { + // Pass snapshot: proven still lagging at block 4 (valid: proven <= checkpointed <= proposed). + return Promise.resolve({ + proposed: { number: BlockNumber(6), hash: makeHash(6) }, + checkpointed: makeTipId(6), + proven: makeTipId(4), + finalized: makeTipId(0), + }); + } + // Post-divergence re-read: the confirmed post-prune chain (proven caught up to 6). + return Promise.resolve({ + proposed: { number: BlockNumber(6), hash: makeHash(6) }, + checkpointed: makeTipId(6), + proven: makeTipId(6), + finalized: makeTipId(0), + }); + }); + latest = 6; + + await blockStream.work(); + + const pruneEvent = handler.events.find( + (e): e is Extract => e.type === 'chain-pruned', + ); + expect(pruneEvent).toBeDefined(); + // Clamp payload comes from the re-read (proven=6), NOT the stale snapshot (proven=4). + expect(pruneEvent!.checkpointed).toEqual(makeTipId(6)); + expect(pruneEvent!.proven).toEqual(makeTipId(6)); + }); + + // The re-read drives more than the prune clamp: the download loop must run THROUGH the re-read proposed tip and + // tier reconciliation must use the re-read tiers. First snapshot proposed=5/proven=4, re-read proposed=8/proven=6. + it('downloads through the re-read proposed tip and reconciles from the re-read tiers', async () => { + const store = new TestL2TipsMemoryStore(); + const storeHandler = new TestL2BlockStreamEventHandler(store); + const blockStream = new TestL2BlockStream(blockSource, store, storeHandler, undefined, { + batchSize: 10, + ignoreCheckpoints: true, + }); + + // Sync the local store to proposed=5 on the old fork (dense hashes 1-5). + setRemoteTips(5); + await blockStream.work(); + storeHandler.clearEvents(); + blockSource.getBlocks.mockClear(); + + // Reorg after block 3: heights 1-3 keep their old (shared) hashes, heights 4+ are on the new fork. The new fork + // extends to proposed=8. + blockSource.getBlockData.mockImplementation(query => { + if (!('number' in query) || query.number > 8) { + return Promise.resolve(undefined); + } + const forked = query.number >= 4; + return Promise.resolve({ + header: { hash: () => Promise.resolve(new BlockHash(new Fr(forked ? query.number + 1000 : query.number))) }, + } as unknown as BlockData); + }); + blockSource.getBlocks.mockImplementation((query: BlocksQuery) => + 'from' in query + ? Promise.resolve( + compactArray( + times(query.limit, i => { + const n = query.from + i; + return n > 8 ? undefined : n >= 4 ? makeForkedBlock(n) : makeBlock(n); + }), + ), + ) + : Promise.resolve([]), + ); + + // FIRST snapshot: a same-height fork already swapped block 5 (forked hash, != the local old-fork hash there), so + // the walk detects divergence and walks down to block 3 before re-reading; proven still lags at 4. The re-read + // reports the post-reorg chain: proposed=8 (new-fork hash), proven=6. + let getTipsCall = 0; + blockSource.getL2Tips.mockImplementation(() => { + getTipsCall++; + return Promise.resolve( + getTipsCall === 1 + ? { + proposed: { number: BlockNumber(5), hash: new Fr(1005).toString() }, + checkpointed: makeTipId(5), + proven: makeTipId(4), + finalized: makeTipId(0), + } + : { + proposed: { number: BlockNumber(8), hash: new Fr(1008).toString() }, + checkpointed: { + block: { number: BlockNumber(8), hash: new Fr(1008).toString() }, + checkpoint: makeCheckpointId(8), + }, + proven: { + block: { number: BlockNumber(6), hash: new Fr(1006).toString() }, + checkpoint: makeCheckpointId(6), + }, + finalized: makeTipId(0), + }, + ); + }); + + await blockStream.work(); + + // The download loop must run through the re-read proposed tip (8), not the stale snapshot tip (5): the highest + // requested block reaches 8. + const requestedThrough = Math.max( + ...blockSource.getBlocks.mock.calls.map(([q]) => ('from' in q ? q.from + q.limit - 1 : 0)), + ); + expect(requestedThrough).toBeGreaterThanOrEqual(8); + + // Tier reconciliation uses the re-read tiers: chain-proven carries the re-read proven tip (6), not the snapshot's + // lagging tip (4). + const provenEvents = storeHandler.events.filter( + (e): e is Extract => e.type === 'chain-proven', + ); + expect(provenEvents).toHaveLength(1); + expect(provenEvents[0].block.number).toBe(6); + }); + }); + + describe('walk-back floor and source coherence', () => { + let localData: TestL2BlockStreamLocalDataProvider; + let handler: TestL2BlockStreamEventHandler; + + beforeEach(() => { + localData = new TestL2BlockStreamLocalDataProvider(); + handler = new TestL2BlockStreamEventHandler(); + }); + + it('stops the walk-back at the local finalized tip instead of pruning deeper', async () => { + // Local synced to 10 with finalized at 5. The source disagrees on every height from 3 up — a fork reaching + // below our finalized tip, which no legitimate reorg can produce (finalized means the proof tx is itself + // L1-final). Its proposed tip is 8 with blocks 3-8 carrying forked hashes. + localData.setProposed(10); + localData.setFinalized(5); + blockSource.getL2Tips.mockResolvedValue({ + proposed: { number: BlockNumber(8), hash: new Fr(1008).toString() }, + checkpointed: makeTipId(0), + proven: makeTipId(0), + finalized: makeTipId(0), + }); + blockSource.getBlockData.mockImplementation(query => { + if (!('number' in query) || query.number > 8) { + return Promise.resolve(undefined); + } + const forked = query.number >= 3; + return Promise.resolve({ + header: { hash: () => Promise.resolve(new BlockHash(new Fr(forked ? query.number + 1000 : query.number))) }, + } as unknown as BlockData); + }); + + const log = mock(); + const blockStream = new TestL2BlockStream(blockSource, localData, handler, log, { batchSize: 10 }); + await blockStream.work(); + + // The walk stops at the floor: the prune target is the finalized tip (5), never the true divergence (2). + const pruneEvents = handler.events.filter( + (e): e is Extract => e.type === 'chain-pruned', + ); + expect(pruneEvents).toHaveLength(1); + expect(pruneEvents[0].block.number).toBe(5); + expect(log.warn).toHaveBeenCalledWith(expect.stringContaining('stopping the walk-back'), expect.anything()); + }); + + it('aborts the pass when the source has no data below its own proposed tip, then recovers', async () => { + // Local synced to 10; the source advertises proposed=12 but cannot serve block 10 (e.g. mid-unwind on the + // source, or a transient read failure). Treating the unreadable height as divergence would walk the prune + // deeper on phantom evidence, so the pass must be skipped instead. + localData.setProposed(10); + setRemoteTips(12); + blockSource.getBlockData.mockResolvedValue(undefined); + + const log = mock(); + const blockStream = new TestL2BlockStream(blockSource, localData, handler, log, { batchSize: 10 }); + await blockStream.work(); + + expect(handler.events).toEqual([]); + expect(log.warn).toHaveBeenCalledWith( + expect.stringContaining('no data for a block at or below its proposed tip'), + expect.anything(), + ); + + // Next pass the source serves data again: the stream catches up normally with no prune. + blockSource.getBlockData.mockImplementation(query => + Promise.resolve( + 'number' in query && query.number <= 12 ? makeBlockData(query.number, query.number) : undefined, + ), + ); + await blockStream.work(); + expect(handler.events.filter(e => e.type === 'chain-pruned')).toEqual([]); + expect(handler.events.filter(e => e.type === 'blocks-added')).toHaveLength(1); + }); + + it('still allows a legitimate prune to genesis when nothing is finalized', async () => { + // Local synced to 10 with finalized still at 0 (nothing proven or finalized yet, e.g. a young chain). The + // source forked from genesis: blocks 1-4 carry forked hashes. With no finalized floor, the walk may legally + // reach block 0 and the prune-to-genesis goes through. + localData.setProposed(10); + blockSource.getL2Tips.mockResolvedValue({ + proposed: { number: BlockNumber(4), hash: new Fr(1004).toString() }, + checkpointed: makeTipId(0), + proven: makeTipId(0), + finalized: makeTipId(0), + }); + blockSource.getBlockData.mockImplementation(query => { + if (!('number' in query) || query.number > 4) { + return Promise.resolve(undefined); + } + const forked = query.number >= 1; + return Promise.resolve({ + header: { hash: () => Promise.resolve(new BlockHash(new Fr(forked ? query.number + 1000 : query.number))) }, + } as unknown as BlockData); + }); + + const blockStream = new TestL2BlockStream(blockSource, localData, handler, undefined, { batchSize: 10 }); + await blockStream.work(); + + const pruneEvents = handler.events.filter( + (e): e is Extract => e.type === 'chain-pruned', + ); + expect(pruneEvents).toHaveLength(1); + expect(pruneEvents[0].block.number).toBe(0); + }); + }); + + describe('event ordering within a pass', () => { + let localData: TestL2TipsMemoryStore; + let handler: TestL2BlockStreamEventHandler; + + beforeEach(() => { + localData = new TestL2TipsMemoryStore(); + handler = new TestL2BlockStreamEventHandler(localData); + }); + + it('emits pruned, then blocks, then proposed, then checkpointed, proven, finalized', async () => { + const blockStream = new TestL2BlockStream(blockSource, localData, handler, undefined, { batchSize: 10 }); + // Sync to proposed=9 on the old fork. + setRemoteTips(9, 9, 6, 3); + await blockStream.work(); + handler.clearEvents(); + + // Reorg: blocks 7-9 changed (fork), new proposed=12, checkpointed=12, proven=10, finalized=8. + blockSource.getBlockData.mockImplementation(query => { + if (!('number' in query) || query.number > 12) { + return Promise.resolve(undefined); + } + const forked = query.number >= 7; + return Promise.resolve({ + header: { hash: () => Promise.resolve(new BlockHash(new Fr(forked ? query.number + 1000 : query.number))) }, + } as unknown as BlockData); + }); + blockSource.getBlocks.mockImplementation((query: BlocksQuery) => + 'from' in query + ? Promise.resolve( + compactArray( + times(query.limit, i => { + const n = query.from + i; + return n > 12 ? undefined : n >= 7 ? makeForkedBlock(n) : makeBlock(n); + }), + ), + ) + : Promise.resolve([]), + ); + blockSource.getL2Tips.mockResolvedValue({ + proposed: { number: BlockNumber(12), hash: new Fr(1012).toString() }, + checkpointed: { + block: { number: BlockNumber(12), hash: new Fr(1012).toString() }, + checkpoint: makeCheckpointId(12), + }, + proven: { block: { number: BlockNumber(10), hash: new Fr(1010).toString() }, checkpoint: makeCheckpointId(10) }, + finalized: { + block: { number: BlockNumber(8), hash: new Fr(1008).toString() }, + checkpoint: makeCheckpointId(8), + }, + }); + + await blockStream.work(); + + // Assert the exact ordered sequence of distinct event types (collapsing the possibly-batched blocks-added run). + // An indexOf comparison would pass vacuously if any event were missing (-1 < x), so compare the full sequence. + const order = collapseConsecutive(handler.events.map(e => e.type)); + expect(order).toEqual([ + 'chain-pruned', + 'blocks-added', + 'chain-proposed', + 'chain-checkpointed', + 'chain-proven', + 'chain-finalized', + ]); + }); + + // Prune-only pass (tips-only): handleChainPruned writes the proposed tag, then chain-proposed re-fires the same + // tip (intended double-notify). The store handles it idempotently. + it('re-fires chain-proposed on a prune-only pass (idempotent double-notify)', async () => { + const store = new TestL2TipsMemoryStore(); + const storeHandler = new TestL2BlockStreamEventHandler(store); + const stream = new TestL2BlockStream(blockSource, store, storeHandler, undefined, { tipsOnly: true }); + + // Tips-only sync to proposed=9. + setRemoteTips(9, 9, 0, 0); + await stream.work(); + storeHandler.clearEvents(); + + // Reorg drops to proposed=6 (still serves blocks 1-6 with matching hashes, so the walk stops at 6). + setRemoteTips(6, 6, 0, 0); + await stream.work(); + + // The prune must precede the (re-fired) proposed event. Filter to those two types and assert the exact ordered + // sequence: an indexOf comparison would pass vacuously if chain-pruned were missing (-1 < its index). + const order = storeHandler.events.map(e => e.type).filter(t => t === 'chain-pruned' || t === 'chain-proposed'); + expect(order).toEqual(['chain-pruned', 'chain-proposed']); + const proposed = storeHandler.events.find( + (e): e is Extract => e.type === 'chain-proposed', + ); + // chain-proposed carries the post-prune tip (6), matching what handleChainPruned already wrote. + expect(proposed!.block).toEqual(makeBlockId(6)); + // The store ends with proposed at 6 regardless of the double write. + expect((await store.getL2Tips()).proposed).toEqual(makeBlockId(6)); + }); + }); + + describe('sparse anchors vs finalized hash deletion', () => { + it('never deletes a hash a live tag points at as the finalized tip advances past sparse anchors', async () => { + const store = new TestL2TipsMemoryStore(); + const handler = new TestL2BlockStreamEventHandler(store); + const stream = new TestL2BlockStream(blockSource, store, handler, undefined, { tipsOnly: true }); + + // Record sparse proposed anchors at 3, then 7, then 12 via tips-only polls. + for (const tip of [3, 7, 12]) { + setRemoteTips(tip, tip, 0, 0); + await stream.work(); + } + + // Drive the finalized tip forward to 7 (past the sparse anchor at 3, up to the anchor at 7). + setRemoteTips(12, 12, 12, 7); + await stream.work(); + + // The finalized handler deletes hashes below the lowest live tip (finalized=7), but never the live tips + // themselves. getL2Tips must keep resolving every tier. + const tips = await store.getL2Tips(); + expect(tips.proposed).toEqual(makeBlockId(12)); + expect(tips.finalized.block).toEqual(makeBlockId(7)); + // The finalized tip's own hash survives deletion (a live tag points at it). + expect(await store.getL2BlockHash(BlockNumber(7))).toEqual(makeHash(7)); + // The anchor at 3 (below finalized) is pruned. + expect(await store.getL2BlockHash(BlockNumber(3))).toBeUndefined(); + }); + }); + + describe('tips-only prune depth vs recorded witnesses', () => { + // Shrunk version of the over-deep example: synced sparse anchors at 8 and 10 (the "80 and 100"), reorg to 9 (the + // "90"). The walk-back rolls back to the nearest recorded hash at or below the true divergence, so without a + // witness at 9 it overshoots to the older anchor at 8; recording a witness at 9 lands the prune exactly there. + const syncSparseAnchors = async (store: TestL2TipsMemoryStore, handler: TestL2BlockStreamEventHandler) => { + const stream = new TestL2BlockStream(blockSource, store, handler, undefined, { tipsOnly: true }); + for (const tip of [8, 10]) { + setRemoteTips(tip); + await stream.work(); + } + handler.clearEvents(); + return stream; + }; + + it('prunes over-deep to the older anchor when no witness covers the divergence', async () => { + const store = new TestL2TipsMemoryStore(); + const handler = new TestL2BlockStreamEventHandler(store); + const stream = await syncSparseAnchors(store, handler); + + // Reorg drops the proposed tip to 9: the source still serves blocks 1-9 (matching hashes) but no longer serves + // 10. With sparse history (anchors only at 8 and 10), the walk-back finds no recorded hash at 9 and overshoots + // to the anchor at 8. + setRemoteTips(9); + await stream.work(); + + const prune = handler.events.find( + (e): e is Extract => e.type === 'chain-pruned', + ); + expect(prune!.block.number).toBe(8); + }); + + it('prunes exactly to the divergence when a witness covers it', async () => { + const store = new TestL2TipsMemoryStore(); + const handler = new TestL2BlockStreamEventHandler(store); + const stream = await syncSparseAnchors(store, handler); + + // A consumer that materialized per-height state at 9 records a witness there before the reorg. + await store.recordBlockHashes([{ number: BlockNumber(9), hash: makeHash(9) }]); + + // Same reorg to 9: now the walk-back finds the recorded hash at 9 (matching the source) and stops there. + setRemoteTips(9); + await stream.work(); + + const prune = handler.events.find( + (e): e is Extract => e.type === 'chain-pruned', + ); + expect(prune!.block.number).toBe(9); + }); + }); }); +/** Collapses runs of identical adjacent items into one, so a batched event run (e.g. blocks-added) counts once. */ +function collapseConsecutive(items: T[]): T[] { + return items.filter((item, i) => i === 0 || item !== items[i - 1]); +} + +/** Shared block-hash function: hashes `this.number`. A single reference so makeBlock(n) objects compare equal. */ +function blockHashFromNumber(this: { number: number }): Promise { + return Promise.resolve(new BlockHash(new Fr(this.number))); +} + +/** Shared forked block-hash function: hashes `this.number + 1000`, simulating a same-height fork swap. */ +function forkedBlockHashFromNumber(this: { number: number }): Promise { + return Promise.resolve(new BlockHash(new Fr(this.number + 1000))); +} + +/** A block whose hash is forked (number + 1000), used to simulate same-height fork swaps. */ +function makeForkedBlock(number: number) { + return { + number: BlockNumber(number), + checkpointNumber: CheckpointNumber(number), + indexWithinCheckpoint: 0, + hash: forkedBlockHashFromNumber, + } as unknown as L2Block; +} + /** Builds a checkpoint id from a plain number, isolated so the branded-type lint rule sees no BlockNumber flow. */ function makeTipCheckpointId(checkpointNumber: number) { return { number: CheckpointNumber(checkpointNumber), hash: new Fr(checkpointNumber).toString() }; diff --git a/yarn-project/stdlib/src/block/l2_block_stream/l2_block_stream.ts b/yarn-project/stdlib/src/block/l2_block_stream/l2_block_stream.ts index 33f648c03694..67e9dca0bb29 100644 --- a/yarn-project/stdlib/src/block/l2_block_stream/l2_block_stream.ts +++ b/yarn-project/stdlib/src/block/l2_block_stream/l2_block_stream.ts @@ -3,7 +3,14 @@ import { AbortError } from '@aztec/foundation/error'; import { createLogger } from '@aztec/foundation/log'; import { RunningPromise } from '@aztec/foundation/running-promise'; -import { type L2BlockId, type L2BlockSource, type L2TipId, makeL2BlockId } from '../l2_block_source.js'; +import type { L2Block } from '../l2_block.js'; +import { + type L2BlockId, + type L2BlockSource, + type L2TipId, + type LocalL2Tips, + makeL2BlockId, +} from '../l2_block_source.js'; import type { L2BlockStreamEvent, L2BlockStreamEventHandler, @@ -33,8 +40,21 @@ export class L2BlockStream { skipFinalized?: boolean; /** When true, checkpoint events will not be emitted. Blocks are still fetched but only blocks-added events are emitted. */ ignoreCheckpoints?: boolean; + /** + * When true, the block download loop is skipped entirely: `getBlocks` is never called and `blocks-added` is + * never emitted. Only the tip events (`chain-proposed`/`chain-checkpointed`/`chain-proven`/`chain-finalized`) + * and `chain-pruned` are emitted, driven by the `getL2Tips` snapshot. For consumers that track tips but never + * consume block payloads. + */ + tipsOnly?: boolean; } = {}, ) { + if (opts.tipsOnly && (opts.startingBlock !== undefined || opts.batchSize !== undefined || opts.skipFinalized)) { + throw new Error( + 'tipsOnly is incompatible with startingBlock, batchSize, and skipFinalized: all three are ' + + 'block-download options and there is no download loop in tips-only mode.', + ); + } // Note that RunningPromise is in stopped state by default. This promise won't run until someone invokes `start`, // which makes it run periodically, or `sync`, which triggers it once. // Users of L2BlockStream decide what mode to run it in (_periodically_ vs _manually triggered_). @@ -68,7 +88,8 @@ export class L2BlockStream { protected async work() { try { - const sourceTips = await this.l2BlockSource.getL2Tips(); + // The source tips snapshot is the plan for this pass; it is re-read after the walk-back if a divergence is found. + let sourceTips = await this.l2BlockSource.getL2Tips(); const localTips = await this.localData.getL2Tips(); this.log.trace(`Running L2 block stream`, { sourceTips, localTips }); @@ -79,16 +100,26 @@ export class L2BlockStream { ); } - // Check if there was a reorg and emit a chain-pruned event if so. + // Baseline for the chain-proposed event; captured before the local store mutates during the pass. + const prePassProposed = localTips.proposed; + + // Walk back to find a reorg, floored at the local finalized tip (a legitimate reorg can never reach it, since + // finalized means the proving tx is itself L1-finalized). Seed the cache with ONLY the proposed tip: a stale + // tier seed at a reorged height equals the local old-fork hash, faking agreement and stopping the walk above the + // true divergence (an under-deep prune no later pass re-detects), whereas a stale proposed seed only masks the + // tip for one pass. let latestBlockNumber = localTips.proposed.number; const sourceCache = new BlockHashCache([sourceTips.proposed]); - while (!(await this.areBlockHashesEqualAt(latestBlockNumber, { sourceCache }))) { + const walkFloor = localTips.finalized.block.number; + while ( + !(await this.areBlockHashesEqualAt(latestBlockNumber, { + sourceCache, + sourceProposed: sourceTips.proposed.number, + })) + ) { if (latestBlockNumber === 0) { - // We walked all the way back to genesis and the hashes still differ. This means the - // local store and the source disagree on the genesis block itself — typically because - // they were configured with different `genesisTimestamp`/prefilled state. Continuing - // would underflow into negative block numbers and surface as "block hash not found - // for -1" further down. Fail loudly with a meaningful error instead. + // Walked back to genesis and the hashes still differ: the two sides disagree on block 0 itself (usually + // different genesisTimestamp/prefilled state). Fail loudly rather than underflow into negative heights. this.log.error(`Genesis block hash mismatch between local store and source`, { localBlockHash: await this.localData.getL2BlockHash(BlockNumber.ZERO), sourceBlockHash: sourceCache.get(0) ?? (await this.getBlockHashFromSource(BlockNumber.ZERO)), @@ -99,11 +130,30 @@ export class L2BlockStream { '(e.g. genesisTimestamp or prefilled public data).', ); } + if (latestBlockNumber <= walkFloor) { + // A mismatch at or below the finalized tip cannot be a reorg (it contradicts L1-finalized state), so stop + // here and prune at most to the finalized tip rather than pruning finalized state on non-reorg evidence. + this.log.warn(`Block hash mismatch at or below the local finalized tip; stopping the walk-back here`, { + blockNumber: latestBlockNumber, + finalizedBlockNumber: walkFloor, + localBlockHash: await this.localData.getL2BlockHash(latestBlockNumber), + sourceBlockHash: + sourceCache.get(latestBlockNumber) ?? (await this.getBlockHashFromSource(latestBlockNumber)), + }); + break; + } latestBlockNumber--; } let pruned = false; if (latestBlockNumber < localTips.proposed.number) { + // Re-read the source tips after the (possibly slow) walk-back so the prune event carries fresh checkpointed + // and proven tips, and the prune-target clamp, download plan, and tier reconciliation track the post-prune + // source chain. Append only the re-read proposed tip: it is a fresh entry that cannot poison the (already + // finished) walk and serves the prune-event hash lookup below. + sourceTips = await this.l2BlockSource.getL2Tips(); + sourceCache.add(sourceTips.proposed); + latestBlockNumber = BlockNumber(Math.min(latestBlockNumber, sourceTips.proposed.number)); // see #13471 const hash = sourceCache.get(latestBlockNumber) ?? (await this.getBlockHashFromSource(latestBlockNumber)); if (latestBlockNumber !== 0 && !hash) { @@ -121,47 +171,20 @@ export class L2BlockStream { pruned = true; } - // The post-prune cursor: the highest block number both sides agree on. Block downloads resume from here. - let nextBlockNumber = latestBlockNumber + 1; - - // If we are just starting from a fresh local store, fast-forward the download cursor to the configured - // starting block so we skip the history the consumer doesn't care about. - const startingBlock = this.opts.startingBlock !== undefined ? BlockNumber(this.opts.startingBlock) : undefined; - if (latestBlockNumber === 0 && startingBlock !== undefined) { - nextBlockNumber = Math.max(startingBlock, 1); - } - - if (this.opts.skipFinalized) { - // When skipping finalized blocks we need to provide reliable reorg detection while fetching as few blocks as - // possible. Finalized blocks cannot be reorged by definition, so we can skip most of them. We do need the very - // last finalized block however in order to guarantee that we will eventually find a block in which our local - // store matches the source. If the last finalized block is behind our local tip, there is nothing to skip. - nextBlockNumber = Math.max(sourceTips.finalized.block.number, nextBlockNumber); - } - - // Only log this entry once (for sanity) - if (!this.hasStarted) { - this.log.verbose(`Starting sync from block number ${nextBlockNumber - 1}`); - this.hasStarted = true; + // Pass atomicity: a prune mid-download leaves the source unable to serve the planned blocks, so the snapshot's + // tier tips may reference blocks the consumer never saw — skip tier reconciliation in that case. Tips-only mode + // has no download plan (the snapshot is one atomic getL2Tips read), so it always reconciles. + if (!this.opts.tipsOnly && !(await this.downloadBlocks(latestBlockNumber, sourceTips))) { + return; } - // Download every block up to the source's proposed tip, batched by `batchSize`. - while (nextBlockNumber <= sourceTips.proposed.number) { - const limit = Math.min(this.opts.batchSize ?? 50, sourceTips.proposed.number - nextBlockNumber + 1); - this.log.trace(`Requesting blocks from ${nextBlockNumber} limit ${limit}`); - const blocks = await this.l2BlockSource.getBlocks({ from: BlockNumber(nextBlockNumber), limit }); - if (blocks.length === 0) { - break; - } - await this.emitEvent({ type: 'blocks-added', blocks }); - nextBlockNumber = blocks.at(-1)!.number + 1; + // End-of-pass reconciliation: chain-proposed fires against the pre-pass baseline (a post-prune re-read would + // equal the source tip and suppress it), then the tiers highest-to-lowest so the finalized <= proven <= + // checkpointed <= proposed invariant holds mid-pass. + if (this.blockTipDiffers(prePassProposed, sourceTips.proposed)) { + await this.emitEvent({ type: 'chain-proposed', block: sourceTips.proposed }); } - // End-of-pass tier reconciliation. For each tier, emit a single event iff the source tip differs from the - // local one. All three source tips come from the SAME `sourceTips` snapshot, so no extra source fetches are - // needed. We re-read the local tips after a prune because the prune handler has already clamped the local - // cursors back; the `localTips` snapshot taken before the prune would be stale and would mis-drive the tier - // comparison (emitting events relative to cursors that no longer exist). const reconcileTips = pruned ? await this.localData.getL2Tips() : localTips; if (!this.opts.ignoreCheckpoints && this.tipDiffers(reconcileTips.checkpointed?.block, sourceTips.checkpointed)) { await this.emitEvent({ @@ -195,6 +218,74 @@ export class L2BlockStream { } } + /** + * Downloads every block from the post-prune cursor through the source's proposed tip, emitting `blocks-added` + * events. The return value gates tier-cursor advancement (pass atomicity): tier tips may only be reconciled when + * the plan that backs them ran to the proposed tip. + * @returns `true` if the plan completed (caught up, or delivered the proposed tip with a matching hash); `false` if + * the source no longer has a promised block, or served a fork at the proposed height mid-pass. + */ + private async downloadBlocks(latestBlockNumber: BlockNumber, sourceTips: LocalL2Tips): Promise { + // The post-prune cursor: the highest block number both sides agree on. Block downloads resume from here. + let nextBlockNumber = latestBlockNumber + 1; + + // From a fresh local store, fast-forward past history the consumer doesn't care about. + const startingBlock = this.opts.startingBlock !== undefined ? BlockNumber(this.opts.startingBlock) : undefined; + if (latestBlockNumber === 0 && startingBlock !== undefined) { + nextBlockNumber = Math.max(startingBlock, 1); + } + + if (this.opts.skipFinalized) { + // Finalized blocks cannot be reorged, so skip them — but keep the last finalized block as the guaranteed point + // where local and source agree, the floor the walk-back terminates against. + nextBlockNumber = Math.max(sourceTips.finalized.block.number, nextBlockNumber); + } + + if (!this.hasStarted) { + this.log.verbose(`Starting sync from block number ${nextBlockNumber - 1}`); + this.hasStarted = true; + } + + let lastDeliveredBlock: L2Block | undefined; + + while (nextBlockNumber <= sourceTips.proposed.number) { + const limit = Math.min(this.opts.batchSize ?? 50, sourceTips.proposed.number - nextBlockNumber + 1); + this.log.trace(`Requesting blocks from ${nextBlockNumber} limit ${limit}`); + const blocks = await this.l2BlockSource.getBlocks({ from: BlockNumber(nextBlockNumber), limit }); + if (blocks.length === 0) { + // The source no longer has a block the snapshot promised: the snapshot is provably stale, so report the plan + // incomplete and skip reconciliation this pass. + this.log.warn(`Block source returned no blocks for a promised range; skipping reconciliation this pass`, { + from: nextBlockNumber, + limit, + sourceProposed: sourceTips.proposed.number, + }); + return false; + } + await this.emitEvent({ type: 'blocks-added', blocks }); + lastDeliveredBlock = blocks.at(-1)!; + nextBlockNumber = lastDeliveredBlock.number + 1; + } + + if (lastDeliveredBlock === undefined) { + // Loop never ran: caught up before the plan started, or startingBlock past the tip (A-1061). Trivially complete. + return true; + } + + // Complete iff the block delivered at the proposed height carries the snapshot's proposed hash; a different hash + // means a same-height fork swap happened mid-pass, so the snapshot is stale. + const deliveredHash = (await lastDeliveredBlock.hash()).toString(); + if (deliveredHash !== sourceTips.proposed.hash) { + this.log.warn(`Delivered proposed-block hash differs from snapshot; skipping reconciliation this pass`, { + blockNumber: lastDeliveredBlock.number, + deliveredHash, + snapshotHash: sourceTips.proposed.hash, + }); + return false; + } + return true; + } + /** * Returns whether the source tip differs from the local one and therefore warrants a tier event. Compares block * number and, when both hashes are known, block hash. The hash comparison is skipped when the local hash is @@ -202,24 +293,39 @@ export class L2BlockStream { * and comparing against an undefined hash would re-emit the event on every poll. */ private tipDiffers(localBlock: LocalL2BlockId | undefined, sourceTip: L2TipId): boolean { + return this.blockTipDiffers(localBlock, sourceTip.block); + } + + /** + * Block-only variant of {@link tipDiffers} for the proposed tip (an {@link L2BlockId}, with no checkpoint). Compares + * block number and, when the local hash is known, block hash. The hash comparison is skipped when the local hash is + * undefined: world-state reports `undefined` for its proposed hash, and a strict comparison would re-emit + * `chain-proposed` on every poll for it. ({@link L2TipsStoreBase} consumers always carry a hash, so the leniency is + * inert for them.) + */ + private blockTipDiffers(localBlock: LocalL2BlockId | undefined, sourceBlock: L2BlockId): boolean { if (localBlock === undefined) { return true; } - if (sourceTip.block.number !== localBlock.number) { + if (sourceBlock.number !== localBlock.number) { return true; } if (localBlock.hash === undefined) { return false; } - return sourceTip.block.hash !== localBlock.hash; + return sourceBlock.hash !== localBlock.hash; } /** * Returns whether the source and local agree on the block hash at a given height. * @param blockNumber - The block number to test. - * @param args - A cache of data already requested from source, to avoid re-requesting it. + * @param args - A cache of data already requested from source (to avoid re-requesting it) and the source's + * advertised proposed tip from the pass snapshot (to detect an incoherent source). */ - private async areBlockHashesEqualAt(blockNumber: BlockNumber, args: { sourceCache: BlockHashCache }) { + private async areBlockHashesEqualAt( + blockNumber: BlockNumber, + args: { sourceCache: BlockHashCache; sourceProposed: BlockNumber }, + ) { const localBlockHash = await this.localData.getL2BlockHash(blockNumber); if (!localBlockHash && this.opts.skipFinalized) { // Failing to find a block hash when skipping finalized blocks can be highly problematic as we'd potentially need @@ -229,12 +335,28 @@ export class L2BlockStream { this.log.error(`No local block hash for block number ${blockNumber}`); throw new AbortError(); } + if (!localBlockHash) { + // A missing local hash compares UNEQUAL: treating both-undefined as equal would stop the walk above the true + // divergence (an under-deep prune no later pass re-detects). Over-deep is the safe direction, and block 0 always + // resolves via the store's initialBlockHash so the walk always terminates. + this.log.trace(`No local block hash for block number ${blockNumber}; treating as unequal`); + return false; + } const sourceBlockHashFromCache = args.sourceCache.get(blockNumber); const sourceBlockHash = args.sourceCache.get(blockNumber) ?? (await this.getBlockHashFromSource(blockNumber)); if (!sourceBlockHashFromCache && sourceBlockHash) { args.sourceCache.add({ number: blockNumber, hash: sourceBlockHash }); } + if (!sourceBlockHash && blockNumber !== 0 && blockNumber <= args.sourceProposed) { + // No source data at or below the source's own proposed tip: the source contradicts itself (mid-reorg unwind or + // a transient read failure), so skip this pass. + this.log.warn(`Source has no data for a block at or below its proposed tip; skipping this sync pass`, { + blockNumber, + sourceProposed: args.sourceProposed, + }); + throw new AbortError(); + } this.log.trace(`Comparing block hashes for block ${blockNumber}`, { localBlockHash, sourceBlockHash }); return localBlockHash === sourceBlockHash; diff --git a/yarn-project/stdlib/src/block/l2_block_stream/l2_tips_store_base.ts b/yarn-project/stdlib/src/block/l2_block_stream/l2_tips_store_base.ts index 4a2a317e18bb..8992cf1e96bc 100644 --- a/yarn-project/stdlib/src/block/l2_block_stream/l2_tips_store_base.ts +++ b/yarn-project/stdlib/src/block/l2_block_stream/l2_tips_store_base.ts @@ -76,11 +76,30 @@ export abstract class L2TipsStoreBase implements L2BlockStreamEventHandler, L2Bl }); } + /** + * Records `(number → hash)` witnesses into the block-hash index without moving any tip cursor. In tips-only mode + * the hash history is sparse (one anchor per tip-moving poll), so a consumer that materializes per-height state + * should witness those heights or its prunes are over-deep by the gap to the nearest anchor. Witnesses are compared + * against the source, not trusted, so they cannot cause under-deep prunes; this is always safe to call. + */ + public async recordBlockHashes(blocks: L2BlockId[]): Promise { + await this.runInTransaction(async () => { + for (const block of blocks) { + if (block.hash) { + await this.setBlockHash(block.number, block.hash); + } + } + }); + } + public async handleBlockStreamEvent(event: L2BlockStreamEvent): Promise { switch (event.type) { case 'blocks-added': await this.handleBlocksAdded(event); break; + case 'chain-proposed': + await this.handleChainProposed(event); + break; case 'chain-checkpointed': await this.handleChainCheckpointed(event); break; @@ -148,6 +167,15 @@ export abstract class L2TipsStoreBase implements L2BlockStreamEventHandler, L2Bl }); } + private async handleChainProposed(event: L2BlockStreamEvent): Promise { + if (event.type !== 'chain-proposed') { + return; + } + // Records the proposed tip into the block-hash index the walk-back reads. In tips-only mode this is the sole + // writer of proposed-tip history, leaving one sparse anchor per tip-moving pass for reorg detection. + await this.runInTransaction(() => this.saveTag('proposed', event.block)); + } + private async handleChainCheckpointed(event: L2BlockStreamEvent): Promise { if (event.type !== 'chain-checkpointed') { return; @@ -163,18 +191,15 @@ export abstract class L2TipsStoreBase implements L2BlockStreamEventHandler, L2Bl return; } await this.runInTransaction(async () => { - // A prune is a rollback: the proposed tip moves to the prune target unconditionally, but - // checkpoint-bearing cursors may only move backward. Forward-advancing them onto an - // uncheckpointed block leaves them on a block with no recorded checkpoint id, which getCheckpointId - // would then throw on. + // A prune is a rollback: the proposed tip moves to the prune target unconditionally, but checkpoint-bearing + // cursors may only move backward (advancing one onto an uncheckpointed block would leave it without a recorded + // checkpoint id, which getCheckpointId throws on). await this.saveTag('proposed', event.block); - // Clamp each checkpoint-bearing cursor down to its OWN source tip when it leads it. Clamping the proven - // cursor onto the checkpointed tip would transiently report unproven blocks as proven (the source's proven - // tip can sit below its checkpointed tip after a proof-tx reorg), until the corrective chain-proven event - // lands at the end of the same sync iteration. The event carries a valid (block, id) pair for each - // boundary, so the clamped cursor always resolves to a recorded id. The source guarantees proven <= - // checkpointed, so clamping each cursor to its own tip preserves the local proven <= checkpointed invariant. + // Clamp each checkpoint-bearing cursor down to its OWN source tip when it leads it; the event carries a valid + // (block, id) pair for each. Clamping the proven cursor onto the (possibly higher) checkpointed tip instead + // would transiently report unproven blocks as proven. The source guarantees proven <= checkpointed, so per-tip + // clamping preserves the local invariant. for (const { tag, sourceTip } of [ { tag: 'checkpointed', sourceTip: event.checkpointed }, { tag: 'proven', sourceTip: event.proven }, diff --git a/yarn-project/stdlib/src/block/test/l2_tips_store_test_suite.ts b/yarn-project/stdlib/src/block/test/l2_tips_store_test_suite.ts index e8a928e5fb3d..e1e7a8729551 100644 --- a/yarn-project/stdlib/src/block/test/l2_tips_store_test_suite.ts +++ b/yarn-project/stdlib/src/block/test/l2_tips_store_test_suite.ts @@ -127,6 +127,71 @@ export function testL2TipsStore(makeTipsStore: () => Promise) { expect(await tipsStore.getL2BlockHash(3)).toEqual(blockHashes.get(3)); }); + it('records the proposed tip number and hash from a chain-proposed event', async () => { + // chain-proposed is the sole writer of proposed-tip history in tips-only mode: no preceding blocks-added. + await tipsStore.handleBlockStreamEvent({ + type: 'chain-proposed', + block: { number: BlockNumber(5), hash: new Fr(500).toString() }, + }); + + const tips = await tipsStore.getL2Tips(); + expect(tips.proposed).toEqual({ number: BlockNumber(5), hash: new Fr(500).toString() }); + // The hash is recorded in the same index the walk-back reads, so it resolves as a sparse anchor. + expect(await tipsStore.getL2BlockHash(5)).toEqual(new Fr(500).toString()); + }); + + it('serves sparse proposed anchors from chain-proposed events for the walk-back', async () => { + // Two tip-moving polls in tips-only mode, with no blocks in between (sparse history). + await tipsStore.handleBlockStreamEvent({ + type: 'chain-proposed', + block: { number: BlockNumber(4), hash: new Fr(400).toString() }, + }); + await tipsStore.handleBlockStreamEvent({ + type: 'chain-proposed', + block: { number: BlockNumber(9), hash: new Fr(900).toString() }, + }); + + // Both recorded heights resolve; the heights between them were never seen and stay undefined. + expect(await tipsStore.getL2BlockHash(4)).toEqual(new Fr(400).toString()); + expect(await tipsStore.getL2BlockHash(9)).toEqual(new Fr(900).toString()); + expect(await tipsStore.getL2BlockHash(6)).toBeUndefined(); + + const tips = await tipsStore.getL2Tips(); + expect(tips.proposed).toEqual({ number: BlockNumber(9), hash: new Fr(900).toString() }); + }); + + it('records block-hash witnesses without moving any tip cursor', async () => { + // Establish a proposed tip at 9 in tips-only fashion (sole writer of proposed history). + await tipsStore.handleBlockStreamEvent({ + type: 'chain-proposed', + block: { number: BlockNumber(9), hash: new Fr(900).toString() }, + }); + + // Record sparse witnesses at heights below the tip (e.g. a consumer materializing per-height state). + await tipsStore.recordBlockHashes([ + { number: BlockNumber(4), hash: new Fr(404).toString() }, + { number: BlockNumber(6), hash: new Fr(606).toString() }, + ]); + + // Each recorded height resolves; an unrecorded gap stays undefined. + expect(await tipsStore.getL2BlockHash(4)).toEqual(new Fr(404).toString()); + expect(await tipsStore.getL2BlockHash(6)).toEqual(new Fr(606).toString()); + expect(await tipsStore.getL2BlockHash(5)).toBeUndefined(); + + // No tip cursor moved: the proposed tip is still 9, and every other tier is at genesis. + const tips = await tipsStore.getL2Tips(); + expect(tips.proposed).toEqual({ number: BlockNumber(9), hash: new Fr(900).toString() }); + expect(tips.checkpointed.block).toEqual(makeTip(0)); + expect(tips.proven.block).toEqual(makeTip(0)); + expect(tips.finalized.block).toEqual(makeTip(0)); + }); + + it('skips witnesses with no hash and tolerates an empty list', async () => { + await tipsStore.recordBlockHashes([]); + await tipsStore.recordBlockHashes([{ number: BlockNumber(3), hash: '' }]); + expect(await tipsStore.getL2BlockHash(3)).toBeUndefined(); + }); + it('checkpoints all proposed blocks', async () => { // Propose blocks 1-5 const blocks = await Promise.all(times(5, i => makeBlock(i + 1))); diff --git a/yarn-project/telemetry-client/src/wrappers/index.ts b/yarn-project/telemetry-client/src/wrappers/index.ts index ea9c9632436f..c0feffbcb6e4 100644 --- a/yarn-project/telemetry-client/src/wrappers/index.ts +++ b/yarn-project/telemetry-client/src/wrappers/index.ts @@ -1,3 +1,2 @@ -export * from './l2_block_stream.js'; export * from './fetch.js'; export * from './json_rpc_server.js'; diff --git a/yarn-project/telemetry-client/src/wrappers/l2_block_stream.ts b/yarn-project/telemetry-client/src/wrappers/l2_block_stream.ts deleted file mode 100644 index c6f49966cdb5..000000000000 --- a/yarn-project/telemetry-client/src/wrappers/l2_block_stream.ts +++ /dev/null @@ -1,38 +0,0 @@ -import { BlockNumber } from '@aztec/foundation/branded-types'; -import { createLogger } from '@aztec/foundation/log'; -import { - type L2BlockSource, - L2BlockStream, - type L2BlockStreamEventHandler, - type L2BlockStreamLocalDataProvider, -} from '@aztec/stdlib/block'; -import { type Traceable, type Tracer, trackSpan } from '@aztec/telemetry-client'; - -/** Extends an L2BlockStream with a tracer to create a new trace per iteration. */ -export class TraceableL2BlockStream extends L2BlockStream implements Traceable { - constructor( - l2BlockSource: Pick, - localData: L2BlockStreamLocalDataProvider, - handler: L2BlockStreamEventHandler, - public readonly tracer: Tracer, - private readonly name: string = 'L2BlockStream', - log = createLogger('types:block_stream'), - opts: { - proven?: boolean; - pollIntervalMS?: number; - batchSize?: number; - startingBlock?: BlockNumber; - } = {}, - ) { - super(l2BlockSource, localData, handler, log, opts); - } - - // We need to use a non-arrow function to be able to access `this` - // See https://www.typescriptlang.org/docs/handbook/2/functions.html#declaring-this-in-a-function - @trackSpan(function () { - return `${this!.name}.work`; - }) - override work() { - return super.work(); - } -} diff --git a/yarn-project/txe/esbuild/stubs/telemetry_stub.ts b/yarn-project/txe/esbuild/stubs/telemetry_stub.ts index 3ccd3129ae10..398e7b58139e 100644 --- a/yarn-project/txe/esbuild/stubs/telemetry_stub.ts +++ b/yarn-project/txe/esbuild/stubs/telemetry_stub.ts @@ -11,7 +11,6 @@ export * from '../../../telemetry-client/dest/start.js'; export * from '../../../telemetry-client/dest/otel_propagation.js'; export * from '../../../telemetry-client/dest/prom_otel_adapter.js'; export * from '../../../telemetry-client/dest/wrappers/fetch.js'; -export * from '../../../telemetry-client/dest/wrappers/l2_block_stream.js'; type MetricDefinition = { name: string; description: string; valueType: ValueType }; diff --git a/yarn-project/world-state/src/synchronizer/server_world_state_synchronizer.ts b/yarn-project/world-state/src/synchronizer/server_world_state_synchronizer.ts index 429f511a288d..356aef8ecc9f 100644 --- a/yarn-project/world-state/src/synchronizer/server_world_state_synchronizer.ts +++ b/yarn-project/world-state/src/synchronizer/server_world_state_synchronizer.ts @@ -308,6 +308,15 @@ export class ServerWorldStateSynchronizer case 'chain-finalized': await this.handleChainFinalized(event.block.number); break; + // World state runs in block mode with ignoreCheckpoints: it tracks tips via blocks-added/pruned/proven/finalized + // and ignores the thin tip events (it never anchors on them). + case 'chain-proposed': + case 'chain-checkpointed': + break; + default: { + const _: never = event; + break; + } } } From 566d37a133e2e100a88de0414ecd4da0a29f0f0d Mon Sep 17 00:00:00 2001 From: Aztec Bot <49558828+AztecBot@users.noreply.github.com> Date: Tue, 16 Jun 2026 09:13:45 -0400 Subject: [PATCH 14/14] test(prover-node): make checkpoint store pruning test deterministic (#24130) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Summary - Fixes the merge-queue failure on PR #24053 by making `CheckpointStore`'s pruning test use explicit slots for its synthetic checkpoints. - The failed CI run was merge-group run `27616214374`; `CI3` job `81652669415` failed in `x9-full` (`http://ci.aztec-labs.com/11cc1d3bfd3986cf`). - The failing test was `yarn-project/scripts/run_test.sh prover-node/src/checkpoint-store.test.ts`, log `http://ci.aztec-labs.com/0e7cfabe067f8bed`, where `markPrunedAboveBlock marks every prover holding a block above the target and returns them` hit the stricter same-slot canonical checkpoint guard while building random test data. ## Verification - `git diff --check` - `./bootstrap.sh ci` attempted, but this branch's root bootstrap has no `ci` target and the container has no Docker socket. - `./bootstrap.sh ci-full-no-test-cache` attempted to match the failed merge-queue leg, but local bootstrap failed before project tests because the test engine exited and interrupted the parallel build. - Focused Yarn/Jest verification is blocked in this checkout because `yarn-project` portal dependencies under `noir/packages/*` are not generated; `noir/bootstrap.sh` is blocked by a crates.io DNS/proxy failure while installing `just`. --- *Created by [claudebox](https://claudebox.work/v2/sessions/46e781f113c5bc56) · group: `slackbot`* --- yarn-project/prover-node/src/checkpoint-store.test.ts | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/yarn-project/prover-node/src/checkpoint-store.test.ts b/yarn-project/prover-node/src/checkpoint-store.test.ts index 4e5c7eff30a3..0c9c98e8de4f 100644 --- a/yarn-project/prover-node/src/checkpoint-store.test.ts +++ b/yarn-project/prover-node/src/checkpoint-store.test.ts @@ -93,7 +93,11 @@ describe('CheckpointStore', () => { // Four single-block checkpoints occupying blocks 1..4 (one block each). Pruning to block 2 orphans the // checkpoints whose last block is above 2 — checkpoints 3 and 4 — and leaves 1 and 2 canonical. const cps = await timesAsync(4, i => - Checkpoint.random(CheckpointNumber(i + 1), { numBlocks: 1, startBlockNumber: i + 1 }), + Checkpoint.random(CheckpointNumber(i + 1), { + numBlocks: 1, + startBlockNumber: i + 1, + slotNumber: SlotNumber(i + 1), + }), ); for (const cp of cps) { await store.addOrUpdate(cp, makeRegisterData());