fix(validator): sync world state before forking in checkpoint proposal validation (#24694)

## Problem

On a live mainnet node, checkpoint-proposal validation could throw an
uncaught error out of the
gossipsub message handler whenever it ran while the world-state
synchronizer trailed the archiver
(block source) by a block:

```
ERROR world-state:database Call CREATE_FORK failed: Error: Unable to initialize from future
  block: 117860 unfinalizedBlockHeight: 117859. Tree name: NullifierTree
ERROR p2p:libp2p_service Error handling gossipsub message: Error: Unable to initialize from
  future block: 117860 unfinalizedBlockHeight: 117859. Tree name: NullifierTree
```

The archiver already had block N (so the block lookups inside checkpoint
validation succeeded), but
world state's `unfinalizedBlockHeight` was still N-1. When
`validateCheckpointProposal` forked world
state at the parent block before it had been applied, the raw tree error
escaped as an uncaught
ERROR-level gossipsub message. The node lost that checkpoint
validation/attestation for the round and
spammed error logs; it self-healed the next tick, but it should not
happen.

## Root cause

The two world-state fork sites were asymmetric. The block re-execution
path (`reexecuteTransactions`)
syncs world state *before* forking. The checkpoint validation path
(`validateCheckpointProposal`)
forked via `checkpointsBuilder.getFork` *without* a preceding sync. The
checkpoint path does sync the
block source (archiver) earlier, but that does not advance the world
state, so the gap remained.

## Fix

- Move the sync-before-fork invariant into
`FullNodeCheckpointsBuilder.getFork`: it now calls
`worldState.syncImmediate(blockNumber, blockHash)` before
`worldState.fork(blockNumber)`, so the
single fork helper owns the invariant rather than each caller.
`syncImmediate` blocks until world
state reaches the block, or throws a typed error if it genuinely cannot,
instead of leaking the raw
  tree error.
- In `validateCheckpointProposal`, look up the parent block's hash from
the block source and pass it
through so the sync re-syncs on a world-state reorg. Wrap the fork in a
try/catch that maps any
failure to a clean, non-slashable `world_state_not_synced` result, so a
sync/fork failure never
  surfaces as an uncaught ERROR-level gossipsub message.
- As a second, independent guard, check that the forked world state's
archive root matches the
proposal's `lastArchiveRoot` before rebuilding the checkpoint. A
mismatch means world state forked
from a different chain than the proposal was built on, so we fail fast
with a clean, non-slashable
`initial_archive_mismatch` result — mirroring the block-proposal
re-execution check — instead of a
  confusing downstream header/archive mismatch.

Both new reasons record an `unvalidated` outcome and are non-slashable,
matching the other
"we couldn't validate right now" / local-state-divergence reasons.

Other checkpoint-builder fork sites were reviewed and need no change:
the proposer's
`checkpoint_proposal_job` forks at its own already-synced,
coherence-checked world-state tip, and the
block-proposal re-execution path already syncs before forking and
performs the equivalent
archive-root check.

## Tests

- `checkpoint_builder.test.ts`: `getFork` syncs world state to the block
before forking, and
  propagates a sync failure without forking.
- `proposal_handler.test.ts`: forks at the parent block number passing
its block hash; a fork failure
returns `world_state_not_synced`; and a fork whose archive root diverges
from the proposal returns
  `initial_archive_mismatch`.

## Notes

The same bug exists on the v4 line — the v4 site is the same function,
forking via
`this.worldState.fork(parentBlockNumber)` directly without a preceding
`syncImmediate`. This fix
should be ported there as well.

Fixes A-1421

(cherry picked from commit b7a7d1b)

Author: spalladino

yarn-project/stdlib/src/interfaces/block-builder.ts

@@ -2,6 +2,7 @@ import type { BlockNumber, CheckpointNumber } from '@aztec/foundation/branded-ty
 import type { Fr } from '@aztec/foundation/curves/bn254';
 import type { LoggerBindings } from '@aztec/foundation/log';
 
+import type { BlockHash } from '../block/block_hash.js';
 import type { L2Block } from '../block/l2_block.js';
 import type { ChainConfig, SequencerConfig } from '../config/chain-config.js';
 import type { L1RollupConstants } from '../epoch-helpers/index.js';
@@ -143,7 +144,11 @@ export interface ICheckpointBlockBuilder {
 
 /** Interface for creating checkpoint builders. */
 export interface ICheckpointsBuilder {
-  getFork(blockNumber: BlockNumber): Promise<MerkleTreeWriteOperations>;
+  /**
+   * Syncs world state to `blockNumber` and returns a fork of it at that block. When `blockHash` is
+   * provided it is verified against the synced block, triggering a resync on mismatch (reorg detection).
+   */
+  getFork(blockNumber: BlockNumber, blockHash?: BlockHash): Promise<MerkleTreeWriteOperations>;
 
   startCheckpoint(
     checkpointNumber: CheckpointNumber,

yarn-project/validator-client/src/checkpoint_builder.test.ts

@@ -14,7 +14,7 @@ import { TestDateProvider } from '@aztec/foundation/timer';
 import type { LightweightCheckpointBuilder } from '@aztec/prover-client/light';
 import type { AvmSimulator, PublicContractsDB, PublicProcessor } from '@aztec/simulator/server';
 import { AztecAddress } from '@aztec/stdlib/aztec-address';
-import { L2Block } from '@aztec/stdlib/block';
+import { BlockHash, L2Block } from '@aztec/stdlib/block';
 import type { ContractDataSource } from '@aztec/stdlib/contract';
 import { Gas, GasFees } from '@aztec/stdlib/gas';
 import {
@@ -24,6 +24,7 @@ import {
   type MerkleTreeWriteOperations,
   type PublicProcessorLimits,
   type PublicProcessorValidator,
+  type WorldStateSynchronizer,
 } from '@aztec/stdlib/interfaces/server';
 import {
   type CheckpointGlobalVariables,
@@ -37,7 +38,7 @@ import type { TelemetryClient } from '@aztec/telemetry-client';
 import { describe, expect, it, jest } from '@jest/globals';
 import { type MockProxy, mock } from 'jest-mock-extended';
 
-import { CheckpointBuilder } from './checkpoint_builder.js';
+import { CheckpointBuilder, FullNodeCheckpointsBuilder } from './checkpoint_builder.js';
 
 describe('CheckpointBuilder', () => {
   let checkpointBuilder: TestCheckpointBuilder;
@@ -789,3 +790,52 @@ describe('CheckpointBuilder', () => {
     });
   });
 });
+
+describe('FullNodeCheckpointsBuilder', () => {
+  let worldState: MockProxy<WorldStateSynchronizer>;
+  let builder: FullNodeCheckpointsBuilder;
+
+  const blockNumber = BlockNumber(5);
+
+  beforeEach(() => {
+    worldState = mock<WorldStateSynchronizer>();
+    const telemetryClient = mock<TelemetryClient>();
+    telemetryClient.getMeter.mockReturnValue(mock());
+    telemetryClient.getTracer.mockReturnValue(mock());
+
+    builder = new FullNodeCheckpointsBuilder(
+      { l1GenesisTime: 0n, slotDuration: 24, l1ChainId: 1, rollupVersion: 1, rollupManaLimit: 200_000_000 },
+      worldState,
+      mock<ContractDataSource>(),
+      new TestDateProvider(),
+      telemetryClient,
+    );
+  });
+
+  describe('getFork', () => {
+    it('syncs world state to the block (with its hash) before forking', async () => {
+      const forkResult = mock<MerkleTreeWriteOperations>();
+      worldState.fork.mockResolvedValue(forkResult);
+      const blockHash = BlockHash.random();
+
+      const result = await builder.getFork(blockNumber, blockHash);
+
+      expect(result).toBe(forkResult);
+      // The block hash is relayed to syncImmediate for reorg detection.
+      expect(worldState.syncImmediate).toHaveBeenCalledWith(blockNumber, blockHash);
+      expect(worldState.fork).toHaveBeenCalledWith(blockNumber);
+      // Syncing must precede the fork, otherwise the fork can hit a block the trees have not applied yet
+      // and throw a raw "initialize from future block" tree error.
+      expect(worldState.syncImmediate.mock.invocationCallOrder[0]).toBeLessThan(
+        worldState.fork.mock.invocationCallOrder[0],
+      );
+    });
+
+    it('propagates a sync failure without forking', async () => {
+      worldState.syncImmediate.mockRejectedValue(new Error('Unable to initialize from future block'));
+
+      await expect(builder.getFork(blockNumber)).rejects.toThrow('Unable to initialize from future block');
+      expect(worldState.fork).not.toHaveBeenCalled();
+    });
+  });
+});

yarn-project/validator-client/src/checkpoint_builder.ts

@@ -15,7 +15,7 @@ import {
   PublicProcessor,
   createPublicTxSimulatorForBlockBuilding,
 } from '@aztec/simulator/server';
-import { L2Block } from '@aztec/stdlib/block';
+import { type BlockHash, L2Block } from '@aztec/stdlib/block';
 import { Checkpoint } from '@aztec/stdlib/checkpoint';
 import type { ContractDataSource } from '@aztec/stdlib/contract';
 import type { L1RollupConstants } from '@aztec/stdlib/epoch-helpers';
@@ -419,8 +419,17 @@ export class FullNodeCheckpointsBuilder implements ICheckpointsBuilder {
     );
   }
 
-  /** Returns a fork of the world state at the given block number. */
-  getFork(blockNumber: BlockNumber): Promise<MerkleTreeWriteOperations> {
+  /**
+   * Syncs world state to the given block number and returns a fork of it at that block.
+   *
+   * Syncing first is required: the block source (archiver) can already hold a block while world state
+   * still trails it, and forking a not-yet-applied block throws a raw "initialize from future block"
+   * tree error. syncImmediate blocks until world state reaches the block, or throws a typed error if it
+   * genuinely cannot. When `blockHash` is provided it is verified against the synced block, triggering a
+   * resync on mismatch (reorg detection).
+   */
+  async getFork(blockNumber: BlockNumber, blockHash?: BlockHash): Promise<MerkleTreeWriteOperations> {
+    await this.worldState.syncImmediate(blockNumber, blockHash);
     return this.worldState.fork(blockNumber);
   }
 }

yarn-project/validator-client/src/proposal_handler.test.ts

@@ -10,6 +10,7 @@ import { TestDateProvider } from '@aztec/foundation/timer';
 import { type FieldsOf, unfreeze } from '@aztec/foundation/types';
 import type { P2P } from '@aztec/p2p';
 import type { BlockProposalValidator } from '@aztec/p2p/msg_validators';
+import { BlockHash } from '@aztec/stdlib/block';
 import type { BlockData, L2Block, L2BlockSink, L2BlockSource } from '@aztec/stdlib/block';
 import { type Checkpoint, CheckpointReexecutionTracker, type ProposedCheckpointData } from '@aztec/stdlib/checkpoint';
 import type { L1RollupConstants } from '@aztec/stdlib/epoch-helpers';
@@ -491,8 +492,12 @@ describe('ProposalHandler checkpoint validation', () => {
     let mockCheckpointBuilder: MockProxy<CheckpointBuilder>;
     let mockDispose: jest.Mock;
 
-    /** Sets up mocks so the handler passes all early checks and reaches the checkpoint rebuild path. */
-    function setupDeepValidationMocks(computedCheckpoint: Partial<Checkpoint>) {
+    /**
+     * Sets up mocks so the handler passes all early checks and reaches the checkpoint rebuild path.
+     * `forkArchiveRoot` is the archive root the forked world state reports; it must match the proposal
+     * header's lastArchiveRoot (default Fr.ZERO, as CheckpointHeader.empty) to pass the fork archive check.
+     */
+    function setupDeepValidationMocks(computedCheckpoint: Partial<Checkpoint>, forkArchiveRoot: Fr = Fr.ZERO) {
       // Block with matching archive so the early archive check passes
       const block = {
         archive: new AppendOnlyTreeSnapshot(archiveRoot, 1),
@@ -505,7 +510,10 @@ describe('ProposalHandler checkpoint validation', () => {
       blockSource.getBlocksForSlot.mockResolvedValue([block]);
 
       mockDispose = jest.fn();
-      checkpointsBuilder.getFork.mockResolvedValue({ [Symbol.asyncDispose]: mockDispose } as any);
+      checkpointsBuilder.getFork.mockResolvedValue({
+        [Symbol.asyncDispose]: mockDispose,
+        getTreeInfo: () => Promise.resolve({ root: forkArchiveRoot.toBuffer() }),
+      } as any);
 
       mockCheckpointBuilder = mock<CheckpointBuilder>();
       mockCheckpointBuilder.completeCheckpoint.mockResolvedValue({
@@ -618,14 +626,17 @@ describe('ProposalHandler checkpoint validation', () => {
         toBlobFields: () => [],
       } as unknown as L2Block;
 
-      setupDeepValidationMocks({
-        header,
-        archive: new AppendOnlyTreeSnapshot(archiveRoot, 1),
-        blocks: [minimalBlock],
-        number: CheckpointNumber(1),
-        slot: SlotNumber(1),
-        toBlobFields: () => [],
-      });
+      setupDeepValidationMocks(
+        {
+          header,
+          archive: new AppendOnlyTreeSnapshot(archiveRoot, 1),
+          blocks: [minimalBlock],
+          number: CheckpointNumber(1),
+          slot: SlotNumber(1),
+          toBlobFields: () => [],
+        },
+        lastArchiveRoot,
+      );
 
       const proposal = await makeProposal({ archiveRoot, checkpointHeader: header });
       const result = await handler.handleCheckpointProposal(proposal, proposalInfo);
@@ -640,6 +651,63 @@ describe('ProposalHandler checkpoint validation', () => {
       await handler.handleCheckpointProposal(proposal, proposalInfo);
       expect(mockDispose).toHaveBeenCalled();
     });
+
+    // Parent of the single block (number 1) used across the deep-validation setup.
+    const parentBlockNumber = BlockNumber(0);
+
+    // getFork syncs world state to the parent block before forking (see FullNodeCheckpointsBuilder tests).
+    // This asserts the caller forks at the parent block number (one before the checkpoint's first block),
+    // passing the parent's block hash (looked up from the block source) for reorg detection.
+    it('forks at the parent block number, passing its block hash', async () => {
+      const parentBlockHash = BlockHash.random();
+      setupDeepValidationMocks({ header: makeHeader({ totalManaUsed: new Fr(999) }) });
+      blockSource.getBlockData.mockResolvedValue({
+        header: makeBlockHeader(),
+        blockHash: parentBlockHash,
+      } as BlockData);
+
+      const proposal = await makeProposal({ archiveRoot, checkpointHeader: makeHeader() });
+      await handler.handleCheckpointProposal(proposal, proposalInfo);
+
+      expect(checkpointsBuilder.getFork).toHaveBeenCalledWith(parentBlockNumber, parentBlockHash);
+    });
+
+    // If world state forked from a different chain than the proposal was built on (e.g. a reorg), the fork's
+    // archive root will not match the checkpoint's expected starting archive. Fail fast before rebuilding.
+    it('returns initial_archive_mismatch when the fork archive does not match the last archive', async () => {
+      // Fork reports a different archive root than the proposal header's lastArchiveRoot (Fr.ZERO).
+      setupDeepValidationMocks({ header: makeHeader() }, Fr.random());
+
+      const proposal = await makeProposal({ archiveRoot, checkpointHeader: makeHeader() });
+      const result = await handler.handleCheckpointProposal(proposal, proposalInfo);
+
+      expect(result).toEqual({
+        isValid: false,
+        reason: 'initial_archive_mismatch',
+        checkpointNumber: CheckpointNumber(1),
+      });
+      expect(mockDispose).toHaveBeenCalled();
+    });
+
+    // Regression: on a live node the archiver held block N (so getBlocksForSlot succeeds) while world state
+    // still trailed at N-1, so forking the parent threw "Unable to initialize from future block" and the raw
+    // tree error escaped as an uncaught gossipsub error. Validation must map any fork failure to a clean
+    // result instead of letting it escape.
+    it('returns world_state_not_synced when forking the parent block fails', async () => {
+      setupDeepValidationMocks({ header: makeHeader() });
+      checkpointsBuilder.getFork.mockRejectedValue(
+        new Error('Unable to initialize from future block: 1 unfinalizedBlockHeight: 0. Tree name: NullifierTree'),
+      );
+
+      const proposal = await makeProposal({ archiveRoot, checkpointHeader: makeHeader() });
+      const result = await handler.handleCheckpointProposal(proposal, proposalInfo);
+
+      expect(result).toEqual({
+        isValid: false,
+        reason: 'world_state_not_synced',
+        checkpointNumber: CheckpointNumber(1),
+      });
+    });
   });
 
   // Regression for A-1218: during a reorg the archiver can still hold a stale block at the proposal's

yarn-project/validator-client/src/proposal_handler.ts

@@ -25,7 +25,12 @@ import type { CheckpointReexecutionTracker, ReexecutionOutcome } from '@aztec/st
 import { getPreviousCheckpointOutHashes, validateCheckpoint } from '@aztec/stdlib/checkpoint';
 import { getEpochAtSlot } from '@aztec/stdlib/epoch-helpers';
 import { Gas } from '@aztec/stdlib/gas';
-import type { ITxProvider, ValidatorClientFullConfig, WorldStateSynchronizer } from '@aztec/stdlib/interfaces/server';
+import type {
+  ITxProvider,
+  MerkleTreeWriteOperations,
+  ValidatorClientFullConfig,
+  WorldStateSynchronizer,
+} from '@aztec/stdlib/interfaces/server';
 import {
   type L1ToL2MessageSource,
   accumulateCheckpointOutHashes,
@@ -91,10 +96,12 @@ export type CheckpointProposalValidationFailureReason =
   | 'invalid_fee_asset_price_modifier'
   | 'last_block_not_found'
   | 'block_fetch_error'
+  | 'world_state_not_synced'
   | 'checkpoint_already_published'
   | 'no_blocks_for_slot'
   | 'last_block_archive_mismatch'
   | 'too_many_blocks_in_checkpoint'
+  | 'initial_archive_mismatch'
   | 'checkpoint_header_mismatch'
   | 'archive_mismatch'
   | 'out_hash_mismatch'
@@ -115,6 +122,8 @@ const CHECKPOINT_VALIDATION_REASON_TO_OUTCOME: Record<
   checkpoint_already_published: undefined,
   last_block_not_found: 'unvalidated',
   block_fetch_error: 'unvalidated',
+  world_state_not_synced: 'unvalidated',
+  initial_archive_mismatch: 'unvalidated',
   no_blocks_for_slot: 'unvalidated',
   last_block_archive_mismatch: 'invalid',
   too_many_blocks_in_checkpoint: 'invalid',
@@ -186,6 +195,9 @@ export const SLASHABLE_CHECKPOINT_PROPOSAL_VALIDATION_RESULT: Record<
   ['invalid_signature']: false,
   ['last_block_not_found']: false,
   ['block_fetch_error']: false,
+  ['world_state_not_synced']: false,
+  // A reorg / divergent local chain, not a proposer offense (mirrors the block path's initial_state_mismatch).
+  ['initial_archive_mismatch']: false,
   ['checkpoint_already_published']: false,
 };
 
@@ -1161,9 +1173,41 @@ export class ProposalHandler {
       log: this.log,
     });
 
-    // Fork world state at the block before the first block
+    // Fork world state at the block before the first block. getFork syncs world state to the parent block
+    // first (see its doc): the block source (archiver) can already hold the block while world state still
+    // trails it by one, and forking a not-yet-applied block throws a raw tree error that would otherwise
+    // escape as an uncaught gossipsub error. We pass the parent's expected block hash so the sync detects a
+    // world-state reorg (undefined for the genesis parent, where no block exists to pin). On failure we map
+    // to a clean validation result rather than letting it escape.
     const parentBlockNumber = BlockNumber(firstBlock.number - 1);
-    await using fork = await this.checkpointsBuilder.getFork(parentBlockNumber);
+    let forkResult: MerkleTreeWriteOperations;
+    try {
+      const parentBlockHash = (await this.blockSource.getBlockData({ number: parentBlockNumber }))?.blockHash;
+      forkResult = await this.checkpointsBuilder.getFork(parentBlockNumber, parentBlockHash);
+    } catch (err) {
+      this.log.warn(`Failed to fork world state at block ${parentBlockNumber} for checkpoint proposal`, {
+        ...proposalInfo,
+        parentBlockNumber,
+        err,
+      });
+      return { isValid: false, reason: 'world_state_not_synced', checkpointNumber };
+    }
+    await using fork = forkResult;
+
+    // Verify the fork's archive root matches the checkpoint's expected starting archive (the archive after
+    // the parent block). A mismatch means world state forked from a different chain than the proposal was
+    // built on (e.g. a reorg), so recomputing the checkpoint against it would be meaningless. This mirrors
+    // the block-proposal re-execution check and fails fast with a clean, non-slashable result instead of a
+    // confusing downstream mismatch.
+    const forkArchiveRoot = new Fr((await fork.getTreeInfo(MerkleTreeId.ARCHIVE)).root);
+    if (!forkArchiveRoot.equals(proposal.checkpointHeader.lastArchiveRoot)) {
+      this.log.warn(`Fork archive root does not match checkpoint proposal's last archive`, {
+        ...proposalInfo,
+        forkArchiveRoot: forkArchiveRoot.toString(),
+        expectedLastArchiveRoot: proposal.checkpointHeader.lastArchiveRoot.toString(),
+      });
+      return { isValid: false, reason: 'initial_archive_mismatch', checkpointNumber };
+    }
 
     // Create checkpoint builder with all existing blocks
     const checkpointBuilder = await this.checkpointsBuilder.openCheckpoint(

yarn-project/validator-client/src/validator.test.ts

@@ -394,7 +394,12 @@ describe('ValidatorClient', () => {
       } as unknown as L2Block;
       const disposeFork = jest.fn();
       blockSource.getBlocksForSlot.mockResolvedValue([checkpointBlock]);
-      checkpointsBuilder.getFork.mockResolvedValue({ [Symbol.asyncDispose]: disposeFork } as any);
+      checkpointsBuilder.getFork.mockResolvedValue({
+        [Symbol.asyncDispose]: disposeFork,
+        // Match the proposal's expected starting archive so the fork archive check passes and validation
+        // reaches the header-mismatch offense under test.
+        getTreeInfo: () => Promise.resolve({ root: proposalHeader.lastArchiveRoot.toBuffer() }),
+      } as any);
       mockCheckpointBuilder.completeCheckpoint.mockResolvedValue({
         header: computedHeader,
         archive: new AppendOnlyTreeSnapshot(proposal.archive, blockNumber),