fix(archiver): prune blocks without proposed checkpoint by end of build slot (#23606)

When the previous proposer sent some block proposals but failed to send
the corresponding checkpoint proposal, the current proposer would assume
there was no proposed checkpoint to build on top of, but would still use
the proposed blocks as chain tip. This meant a failed `canPropose` check
against the Rollup contract as soon as it started its slot, since the
proposed blocks from the previous proposer meant the proposer had a
wrong chain tip.

To fix, the sequencer is now aware that there may be proposed blocks
without the corresponding checkpoints, and it can't start building until
that's resolved. Also, the archiver now prunes proposed blocks without a
checkpoint when the corresponding _build_ slot is over.

---

## Motivation

Under proposer pipelining a node can receive and reexecute the
block-only proposals for a checkpoint before (or without ever) receiving
the enclosing proposed checkpoint. This leaves the local tip one
checkpoint ahead of the checkpointed tip with no proposed checkpoint
backing it. A sequencer that then builds the next checkpoint on top of
that orphan tip forks the chain off a parent no other node can follow,
which was the root cause behind the sentinel CI flake.

## Approach

Two complementary defenses. The sequencer's `checkSync` refuses to
proceed when the synced block's checkpoint is ahead of the checkpointed
tip and no matching proposed checkpoint exists, holding the line during
the window before cleanup. The archiver adds a wall-clock orphan prune
that, shortly after a block's build slot ends, removes a block-only tip
whose checkpoint was never proposed, restoring liveness even while L1 is
quiet.

## Changes

- **sequencer-client**: `checkSync` rejects syncing onto a proposed
block with no matching proposed-checkpoint tip/data, logging a
descriptive warning.
- **archiver**: new `pruneOrphanProposedBlocks` on the L1 synchronizer,
run from `Archiver.sync()` after the inbound queue drains and before L1
sync; prunes after `start(blockSlot) + grace` using the epoch-cache
pipelining offset and emits `L2PruneUncheckpointed`. The existing
L1-sync prune is preserved (shared prune/emit helper).
- **archiver/stdlib/foundation config**: new
`orphanProposedBlockPruneGraceSeconds` in `ArchiverSpecificConfig`,
archiver config mappings
(`ARCHIVER_ORPHAN_PROPOSED_BLOCK_PRUNE_GRACE_SECONDS`),
`mapArchiverConfig`, the synchronizer/archiver config types, and a new
`EnvVar`.
- **aztec-node**: defaults the grace window from `blockDurationMs /
1000` when unset, falling back to `MIN_EXECUTION_TIME`; the archiver
factory also defaults to `MIN_EXECUTION_TIME`.
- **sequencer-client (tests)**: orphan tip returns `undefined` and
warns; matching proposed checkpoint proceeds.
- **archiver (tests)**: no prune before grace; prune + event after
grace; no prune when a matching proposed checkpoint exists; queued
proposed checkpoint is processed before the prune.

Author: spalladino

yarn-project/archiver/README.md

@@ -26,12 +26,13 @@ The archiver runs a periodic sync loop with two phases:
 
 ```
 sync()
-├── processQueuedBlocks()       # Handle blocks pushed via addBlock()
+├── processQueuedBlocks()         # Handle blocks pushed via addBlock()
+├── pruneOrphanProposedBlocks()   # Wall-clock prune of orphan block-only tips
 └── syncFromL1()
-    ├── handleL1ToL2Messages()  # Sync messages from Inbox contract
-    ├── handleCheckpoints()     # Sync checkpoints from Rollup contract
+    ├── handleL1ToL2Messages()    # Sync messages from Inbox contract
+    ├── handleCheckpoints()       # Sync checkpoints from Rollup contract
     ├── pruneUncheckpointedBlocks()  # Prune provisional blocks from expired slots
-    ├── handleEpochPrune()      # Proactive unwind before proof window expires
+    ├── handleEpochPrune()        # Proactive unwind before proof window expires
     └── checkForNewCheckpointsBeforeL1SyncPoint()  # Handle L1 reorg edge case
 ```
 
@@ -100,8 +101,9 @@ Queued blocks are processed at the start of each sync iteration. This allows the
 Blocks added via `addBlock()` are considered "provisional" until they appear in an L1 checkpoint. These provisional blocks may need to be reconciled when:
 - **Checkpoint mismatch**: A checkpoint lands on L1 with different blocks than stored locally (e.g., a different proposer won the slot)
 - **Slot expiration**: An L2 slot ends without any checkpoint being mined on L1
+- **Orphan proposed block**: Under proposer pipelining, a proposer can broadcast a block-only proposal but never the matching `CheckpointProposal` (e.g. it crashes before assembling the checkpoint). The provisional block then has no proposed checkpoint backing it.
 
-When `handleCheckpoints()` processes incoming checkpoints, it compares archive roots of local blocks against the checkpoint's blocks. If they differ, local blocks are pruned and replaced with the checkpoint's blocks. After checkpoint sync, `pruneUncheckpointedBlocks()` removes any remaining provisional blocks from slots that have ended. Both cases emit `L2PruneUncheckpointed`.
+When `handleCheckpoints()` processes incoming checkpoints, it compares archive roots of local blocks against the checkpoint's blocks. If they differ, local blocks are pruned and replaced with the checkpoint's blocks. After checkpoint sync, `pruneUncheckpointedBlocks()` removes any remaining provisional blocks from slots that have ended. Independently, `pruneOrphanProposedBlocks()` runs on wall-clock time (so it fires during quiet L1 periods) and removes a block-only tip once its build slot ended without a matching proposed checkpoint, plus a grace window configured via `orphanProposedBlockPruneGraceSeconds`. All three cases emit `L2PruneUncheckpointed`.
 
 ### Querying Block Data
 

yarn-project/archiver/src/archiver-misc.test.ts

@@ -8,6 +8,7 @@ import { BlockNumber, CheckpointNumber, EpochNumber, SlotNumber } from '@aztec/f
 import { Buffer32 } from '@aztec/foundation/buffer';
 import { Fr } from '@aztec/foundation/curves/bn254';
 import { EthAddress } from '@aztec/foundation/eth-address';
+import { DateProvider } from '@aztec/foundation/timer';
 import { openTmpStore } from '@aztec/kv-store/lmdb-v2';
 import type { L2Tips } from '@aztec/stdlib/block';
 import type { CheckpointData } from '@aztec/stdlib/checkpoint';
@@ -57,6 +58,7 @@ describe('Archiver misc', () => {
     const rollupContract = mock<RollupContract>();
     const epochCache = mock<EpochCache>();
     epochCache.getCommitteeForEpoch.mockResolvedValue({ committee: [] as EthAddress[] } as EpochCommitteeInfo);
+    epochCache.pipeliningOffset.mockReturnValue(0);
 
     const tracer = getTelemetryClient().getTracer('');
     const instrumentation = mock<ArchiverInstrumentation>({ isEnabled: () => true, tracer });
@@ -78,7 +80,12 @@ describe('Archiver misc', () => {
         slashingProposerAddress: EthAddress.random(),
       },
       archiverStore,
-      { pollingIntervalMs: 1000, batchSize: 1000, maxAllowedEthClientDriftSeconds: 300 },
+      {
+        pollingIntervalMs: 1000,
+        batchSize: 1000,
+        maxAllowedEthClientDriftSeconds: 300,
+        orphanProposedBlockPruneGraceSeconds: 2,
+      },
       blobClient,
       instrumentation,
       l1Constants,
@@ -87,6 +94,8 @@ describe('Archiver misc', () => {
       initialHeader,
       initialBlockHash,
       l2TipsCache,
+      epochCache,
+      new DateProvider(),
     );
   });
 

yarn-project/archiver/src/archiver-store.test.ts

@@ -12,6 +12,7 @@ import {
 import { Buffer32 } from '@aztec/foundation/buffer';
 import { Fr } from '@aztec/foundation/curves/bn254';
 import { EthAddress } from '@aztec/foundation/eth-address';
+import { DateProvider } from '@aztec/foundation/timer';
 import { openTmpStore } from '@aztec/kv-store/lmdb-v2';
 import { L2Block } from '@aztec/stdlib/block';
 import type { L1RollupConstants } from '@aztec/stdlib/epoch-helpers';
@@ -63,6 +64,7 @@ describe('Archiver Store', () => {
     blobClient = mock<BlobClientInterface>();
     epochCache = mock<EpochCache>();
     epochCache.getCommitteeForEpoch.mockResolvedValue({ committee: [] as EthAddress[] } as EpochCommitteeInfo);
+    epochCache.pipeliningOffset.mockReturnValue(0);
 
     const rollupContract = mock<RollupContract>();
     Object.defineProperty(rollupContract, 'address', { value: rollupAddress.toString(), writable: true });
@@ -98,6 +100,7 @@ describe('Archiver Store', () => {
       batchSize: 1000,
       maxAllowedEthClientDriftSeconds: 300,
       ethereumAllowNoDebugHosts: true,
+      orphanProposedBlockPruneGraceSeconds: 2,
     };
 
     const events = new EventEmitter() as ArchiverEmitter;
@@ -120,6 +123,8 @@ describe('Archiver Store', () => {
       initialHeader,
       initialBlockHash,
       l2TipsCache,
+      epochCache,
+      new DateProvider(),
     );
   });
 

yarn-project/archiver/src/archiver-sync.test.ts

@@ -20,6 +20,7 @@ import type { ProposedCheckpointInput } from '@aztec/stdlib/checkpoint';
 import type { L1RollupConstants } from '@aztec/stdlib/epoch-helpers';
 import { computeInHashFromL1ToL2Messages } from '@aztec/stdlib/messaging';
 import { CheckpointHeader } from '@aztec/stdlib/rollup';
+import { mockCheckpointAndMessages } from '@aztec/stdlib/testing';
 import { BlockHeader } from '@aztec/stdlib/tx';
 import { getTelemetryClient } from '@aztec/telemetry-client';
 
@@ -92,6 +93,9 @@ describe('Archiver Sync', () => {
     // Create epoch cache mock (separate from fake)
     epochCache = mock<EpochCache>();
     epochCache.getCommitteeForEpoch.mockResolvedValue({ committee: [] as EthAddress[] } as EpochCommitteeInfo);
+    // Default to no pipelining offset; the orphan-prune tests below override this. Keeps the prune
+    // deadline well ahead of wall-clock time for the other tests so it never fires spuriously.
+    epochCache.pipeliningOffset.mockReturnValue(0);
 
     // Create instrumentation mock
     const tracer = getTelemetryClient().getTracer('');
@@ -118,6 +122,7 @@ describe('Archiver Sync', () => {
       maxAllowedEthClientDriftSeconds: 300,
       ethereumAllowNoDebugHosts: true,
       skipHistoricalLogsCheck: true,
+      orphanProposedBlockPruneGraceSeconds: 2,
     };
 
     // Create event emitter shared by archiver and synchronizer
@@ -162,6 +167,8 @@ describe('Archiver Sync', () => {
       initialHeader,
       initialBlockHash,
       l2TipsCache,
+      epochCache,
+      dateProvider,
     );
   });
 
@@ -2143,4 +2150,157 @@ describe('Archiver Sync', () => {
       expect(tips.proposedCheckpoint.block.number).toEqual(tips.checkpointed.block.number);
     }, 15_000);
   });
+
+  describe('pruning orphan proposed blocks', () => {
+    let pruneSpy: jest.Mock;
+
+    // Slot the orphan block targets. With slotDuration=24, slot S starts at l1GenesisTime + S*24.
+    const orphanSlot = SlotNumber(1);
+    // Grace period configured for these tests (see the `config` object above).
+    const graceSeconds = 2;
+
+    beforeEach(() => {
+      pruneSpy = jest.fn();
+      archiver.events.on(L2BlockSourceEvents.L2PruneUncheckpointed, pruneSpy);
+      // Normal proposer pipelining: a block targeting slot S is built during slot S-1, so its proposed
+      // checkpoint is expected by the start of slot S.
+      epochCache.pipeliningOffset.mockReturnValue(1);
+    });
+
+    afterEach(() => {
+      archiver.events.off(L2BlockSourceEvents.L2PruneUncheckpointed, pruneSpy);
+    });
+
+    // Wall-clock time (seconds) at which the orphan tip becomes prunable: start(orphanSlot) + grace.
+    const pruneDeadline = () => now + Number(orphanSlot) * l1Constants.slotDuration + graceSeconds;
+    const pruneDeadlineForSlot = (slot: SlotNumber) => now + Number(slot) * l1Constants.slotDuration + graceSeconds;
+
+    // Syncs checkpoint 1 (slot 0), then writes uncheckpointed blocks for slot 1 (checkpoint 2) straight
+    // into the store as a block-only tip with no matching proposed checkpoint. L1 is held at slot 1 so
+    // the L1-sync prune (which only fires once the build slot has ended on L1) stays out of the way.
+    const setupOrphanTip = async () => {
+      const { checkpoint: cp1 } = await fake.addCheckpoint(CheckpointNumber(1), {
+        l1BlockNumber: 1n,
+        messagesL1BlockNumber: 1n,
+        numL1ToL2Messages: 3,
+        slotNumber: SlotNumber(0),
+      });
+      const cp1Archive = cp1.blocks.at(-1)!.archive;
+      fake.setL1BlockNumber(1n);
+      await archiver.syncImmediate();
+      expect(await archiver.getCheckpointNumber()).toEqual(CheckpointNumber(1));
+
+      const lastBlockInCp1 = cp1.blocks.at(-1)!.number;
+      const provisionalBlocks = await fake.makeBlocks(CheckpointNumber(2), {
+        l1BlockNumber: 2n,
+        previousArchive: cp1Archive,
+        slotNumber: orphanSlot,
+      });
+      for (const block of provisionalBlocks) {
+        await archiver.addBlock(block);
+      }
+
+      // Hold L1 at slot 1 so the slot has not ended from L1's perspective.
+      fake.setL1BlockNumber(2n);
+      return { lastBlockInCp1, lastProvisional: provisionalBlocks.at(-1)!.number, provisionalBlocks };
+    };
+
+    const makeProposedCheckpoint = (lastBlockInCp1: BlockNumber, blockCount: number): ProposedCheckpointInput => ({
+      checkpointNumber: CheckpointNumber(2),
+      header: CheckpointHeader.empty({ slotNumber: orphanSlot }),
+      startBlock: BlockNumber(lastBlockInCp1 + 1),
+      blockCount,
+      totalManaUsed: 0n,
+      feeAssetPriceModifier: 0n,
+    });
+
+    it('does not prune before the grace window elapses', async () => {
+      const { lastProvisional } = await setupOrphanTip();
+
+      dateProvider.setTime((pruneDeadline() - 1) * 1000);
+      await archiver.syncImmediate();
+
+      expect(pruneSpy).not.toHaveBeenCalled();
+      expect(await archiver.getBlockNumber()).toEqual(lastProvisional);
+    }, 15_000);
+
+    it('prunes the orphan tip once the grace window elapses', async () => {
+      const { lastBlockInCp1, provisionalBlocks } = await setupOrphanTip();
+
+      dateProvider.setTime((pruneDeadline() + 1) * 1000);
+      await archiver.syncImmediate();
+
+      expect(pruneSpy).toHaveBeenCalledWith(
+        expect.objectContaining({
+          type: L2BlockSourceEvents.L2PruneUncheckpointed,
+          slotNumber: orphanSlot,
+          blocks: expect.arrayContaining(provisionalBlocks.map(b => expect.objectContaining({ number: b.number }))),
+        }),
+      );
+      expect(await archiver.getBlockNumber()).toEqual(lastBlockInCp1);
+      expect(await archiver.getCheckpointNumber()).toEqual(CheckpointNumber(1));
+    }, 15_000);
+
+    it('does not prune when a matching proposed checkpoint exists', async () => {
+      const { lastBlockInCp1, lastProvisional, provisionalBlocks } = await setupOrphanTip();
+
+      await archiver.addProposedCheckpoint(makeProposedCheckpoint(lastBlockInCp1, provisionalBlocks.length));
+
+      dateProvider.setTime((pruneDeadline() + 100) * 1000);
+      await archiver.syncImmediate();
+
+      expect(pruneSpy).not.toHaveBeenCalled();
+      expect(await archiver.getBlockNumber()).toEqual(lastProvisional);
+      expect(await archiverStore.blocks.getLastProposedCheckpoint()).toBeDefined();
+    }, 15_000);
+
+    it('processes a queued proposed checkpoint before pruning, sparing the tip', async () => {
+      const { lastBlockInCp1, lastProvisional, provisionalBlocks } = await setupOrphanTip();
+
+      // Past the grace window: without the matching checkpoint the next sync would prune the tip.
+      dateProvider.setTime((pruneDeadline() + 100) * 1000);
+
+      // Queue the proposed checkpoint. The triggered sync drains the inbound queue (storing the
+      // checkpoint) before running the orphan prune, so the prune sees it and stands down. If the
+      // order were reversed, this sync would prune the tip before storing the checkpoint.
+      await archiver.addProposedCheckpoint(makeProposedCheckpoint(lastBlockInCp1, provisionalBlocks.length));
+      await archiver.syncImmediate();
+
+      expect(pruneSpy).not.toHaveBeenCalled();
+      expect(await archiver.getBlockNumber()).toEqual(lastProvisional);
+      expect(await archiverStore.blocks.getLastProposedCheckpoint()).toBeDefined();
+    }, 15_000);
+
+    it('prunes only the orphan suffix after a covered pending checkpoint', async () => {
+      const { lastBlockInCp1, provisionalBlocks: checkpointTwoBlocks } = await setupOrphanTip();
+
+      await archiver.addProposedCheckpoint(makeProposedCheckpoint(lastBlockInCp1, checkpointTwoBlocks.length));
+
+      const orphanSuffixSlot = SlotNumber(orphanSlot + 1);
+      const { checkpoint: orphanSuffixCheckpoint } = await mockCheckpointAndMessages(CheckpointNumber(3), {
+        startBlockNumber: BlockNumber(checkpointTwoBlocks.at(-1)!.number + 1),
+        numBlocks: 1,
+        previousArchive: checkpointTwoBlocks.at(-1)!.archive,
+        slotNumber: orphanSuffixSlot,
+      });
+      const orphanSuffixBlocks = orphanSuffixCheckpoint.blocks;
+      for (const block of orphanSuffixBlocks) {
+        await archiver.addBlock(block);
+      }
+
+      dateProvider.setTime((pruneDeadlineForSlot(orphanSuffixSlot) + 1) * 1000);
+      await archiver.syncImmediate();
+
+      expect(pruneSpy).toHaveBeenCalledWith(
+        expect.objectContaining({
+          type: L2BlockSourceEvents.L2PruneUncheckpointed,
+          slotNumber: orphanSuffixSlot,
+          blocks: expect.arrayContaining(orphanSuffixBlocks.map(b => expect.objectContaining({ number: b.number }))),
+        }),
+      );
+      expect(await archiver.getBlockNumber()).toEqual(checkpointTwoBlocks.at(-1)!.number);
+      expect(await archiverStore.blocks.getProposedCheckpointByNumber(CheckpointNumber(2))).toBeDefined();
+      expect(await archiverStore.blocks.getProposedCheckpointByNumber(CheckpointNumber(3))).toBeUndefined();
+    }, 15_000);
+  });
 });