merge sync-queue-hashes

Author: lispc

AGENTS.md

@@ -48,7 +48,7 @@ Key hard-won rules:
 For continuously proving mainnet bundles in real time (poll-syncing the mainnet DB into the shadow DB), three silent starvation traps apply — see `tests/shadow-testing/docs/TROUBLESHOOTING.md` Trap 22/23:
 - Poll sync must also maintain `l2_block.chunk_hash` links (UPDATE, not INSERT) and `chunk.batch_hash`/`batch.bundle_hash` parent links, or tasks become invisible to the coordinator with no ERROR logged.
 - `sweep-stale-proving.sh` must reset `total_attempts`, not just `proving_status` — the coordinator skips tasks with `total_attempts >= 5`.
-- Bundles popping L1 messages enqueued after the Anvil fork block fail with `VerificationFailed(0x439cc0cd)`: run `tests/shadow-testing/scripts/sync-queue-hashes.py` (cron, 10 min) to mirror `messageRollingHashes` + `nextCrossDomainMessageIndex` from mainnet onto the fork.
+- Bundles popping L1 messages enqueued after the Anvil fork block fail with `VerificationFailed(0x439cc0cd)`: `sync-mainnet-db.py` poll mode runs `tests/shadow-testing/scripts/sync-queue-hashes.py` every cycle to mirror `messageRollingHashes` + `nextCrossDomainMessageIndex` from mainnet onto the fork.
 
 ### Sepolia Shadow Fork — Additional Rules
 

tests/shadow-testing/README.md

@@ -66,7 +66,7 @@ range), disable the relayer's local proposers and run the polling sync:
 
 ```bash
 python3 scripts/sync-mainnet-db.py --poll-interval 60   # DB rows + l2_block/parent links
-python3 scripts/sync-queue-hashes.py                    # L1 queue rolling hashes (cron every 10 min)
+python3 scripts/sync-queue-hashes.py                    # L1 queue rolling hashes (also runs inside sync-mainnet-db.py poll loop)
 scripts/sweep-stale-proving.sh                          # stale task sweeper (cron every 10 min)
 ```
 

tests/shadow-testing/configs/coordinator.json.template

@@ -3,7 +3,7 @@
         "provers_per_session": 1,
         "session_attempts": 5,
         "external_prover_threshold": 32,
-        "bundle_collection_time_sec": 180,
+        "bundle_collection_time_sec": 7200,
         "batch_collection_time_sec": 180,
         "chunk_collection_time_sec": 3600,
         "verifier": {

tests/shadow-testing/docs/GUIDE.md

@@ -1023,15 +1023,15 @@ Critical behavior of the sync (do not bypass):
 
 Watch item: the first bundle whose batches were committed on mainnet **after** the fork block exercises the relayer's commit path on Anvil (blob-carrying `commitBatches` tx). Keep `fusaka_timestamp: 2000000000` in the relayer config so Anvil accepts the blob sidecar.
 
-**L1 message queue follow-along (mandatory for long runs)**: bundles that pop L1 messages enqueued after the fork block fail finalization (`VerificationFailed` / `ErrorFinalizedIndexTooLarge`, Trap 23). Run `scripts/sync-queue-hashes.py` periodically — it copies `getMessageRollingHash(i)` from a mainnet RPC into the fork's `messageRollingHashes` mapping (slot 101) and aligns `nextCrossDomainMessageIndex` (slot 103) with mainnet.
+**L1 message queue follow-along (mandatory for long runs)**: bundles that pop L1 messages enqueued after the fork block fail finalization (`VerificationFailed` / `ErrorFinalizedIndexTooLarge`, Trap 23). `sync-mainnet-db.py` poll mode runs `scripts/sync-queue-hashes.py` every cycle — it copies `getMessageRollingHash(i)` from a mainnet RPC into the fork's `messageRollingHashes` mapping (slot 101) and aligns `nextCrossDomainMessageIndex` (slot 103) with mainnet.
 
 **Recommended automation** (what the 48-hour test ran with):
 
 | Cadence | Job | Purpose |
 |---------|-----|---------|
 | 60s loop | `sync-mainnet-db.py --poll-interval 60` | DB row sync + l2_block/parent-link repair |
 | 10 min cron | `scripts/sweep-stale-proving.sh` | Reset stale proving rows **and `total_attempts`** (attempt exhaustion silently starves tasks, Trap 22) |
-| 10 min cron | `scripts/sync-queue-hashes.py` | L1 queue rolling hashes + cursor follow mainnet (Trap 23) |
+| every poll cycle (in `sync-mainnet-db.py`) | `scripts/sync-queue-hashes.py` | L1 queue rolling hashes + cursor follow mainnet (Trap 23) |
 | 1 h cron | `scripts/monitor-catchup.py >> .work/catchup-metrics.log` | Hourly metrics snapshot for the final report |
 
 Final report: `SHADOW_REPORT_START=<ISO8601> python3 scripts/generate-catchup-report.py` — the env var windows the report to the current run (the metrics log accumulates across runs).

tests/shadow-testing/docs/TROUBLESHOOTING.md

@@ -261,7 +261,7 @@ Before executing a single command:
   3. Call the wrapper's `verify(bundleProof, publicInput)` directly with the metadata-derived publicInput. **If it succeeds**, proof/digests are fine and the mismatch is contract-side — ScrollChain computes `messageQueueHash` itself via `L1MessageQueueV2.getMessageRollingHash()`.
   4. Compare `getMessageRollingHash(i)` on fork vs mainnet (public RPC e.g. `https://ethereum-rpc.publicnode.com` works; Alchemy demo key is 429-rate-limited).
 - **Root cause**: `L1MessageQueueV2` keeps `mapping(uint256=>bytes32) messageRollingHashes` at **storage slot 101** (value = rolling hash with low 32 bits overwritten by enqueue timestamp; the getter clears those bits). Entries for messages enqueued after the fork block are zero on the fork, so the contract builds a publicInput whose `messageQueueHash` differs from the prover's → plonk rejects. Note the **off-by-one**: rh[i] is the hash *after* message i, so a bundle popping to index N needs rh[N-1]; DB `prev/post_l1_message_queue_hash` at popped_before=N equals `getMessageRollingHash(N-1)`.
-- **Fix**: `scripts/sync-queue-hashes.py` copies `getMessageRollingHash(i)` from a mainnet RPC into the fork via `anvil_setStorageAt(queue, keccak256(pad32(i)‖pad32(101)), hash)` and also bumps `nextCrossDomainMessageIndex` (slot 103 = 0x67) to mainnet's value (this is what clears the follow-up `ErrorFinalizedIndexTooLarge`). It is idempotent and runs on a 10-minute cron during catch-up mode. Run it manually after any Anvil restart/re-fork, and whenever a finalize reverts with either selector.
+- **Fix**: `scripts/sync-queue-hashes.py` copies `getMessageRollingHash(i)` from a mainnet RPC into the fork via `anvil_setStorageAt(queue, keccak256(pad32(i)‖pad32(101)), hash)` and also bumps `nextCrossDomainMessageIndex` (slot 103 = 0x67) to mainnet's value (this is what clears the follow-up `ErrorFinalizedIndexTooLarge`). It is idempotent and runs inside the `sync-mainnet-db.py` poll loop during catch-up mode (failures there only log a warning, never kill the DB sync). Run it manually after any Anvil restart/re-fork, and whenever a finalize reverts with either selector.
 - **Rule of thumb**: if finalization fails only for bundles whose `post_l1_message_queue_hash != prev_l1_message_queue_hash` (i.e. the batch pops messages), suspect this trap before touching proofs or the verifier wrapper.
 
 ## Step-by-Step Checklist

tests/shadow-testing/scripts/sync-mainnet-db.py

@@ -16,12 +16,14 @@
 """
 
 import argparse
+import importlib.util
 import json
 import logging
 import os
 import subprocess
 import time
 import urllib.request
+from pathlib import Path
 
 import psycopg2
 
@@ -31,6 +33,14 @@
 )
 log = logging.getLogger("sync-mainnet-db")
 
+# sync-queue-hashes.py (Trap 23) is loaded as a module so each poll cycle can
+# also mirror L1MessageQueueV2 rolling hashes onto the Anvil fork.
+_sqh_spec = importlib.util.spec_from_file_location(
+    "sync_queue_hashes", Path(__file__).parent / "sync-queue-hashes.py"
+)
+_sqh = importlib.util.module_from_spec(_sqh_spec)
+_sqh_spec.loader.exec_module(_sqh)
+
 SRC_DSN = os.environ.get(
     "MAINNET_DSN",
     "postgresql://mainnet_infra_team_read_only:AuexDUuaarskbG6tr9CH9gXsJqp4at67mddAbMrt@localhost:15432/mainnet_rollup",
@@ -581,6 +591,13 @@ def _ensure_cursors():
             finally:
                 src_cur.close()
                 dst_cur.close()
+            # Trap 23: mirror post-fork L1 queue rolling hashes onto the fork
+            # so bundles popping new L1 messages can be finalized. Failures
+            # here must never kill the DB sync loop.
+            try:
+                _sqh.run_once(log_fn=lambda m: log.info("queue-hash: %s", m))
+            except Exception as e:
+                log.warning("queue-hash sync failed: %s", e)
             time.sleep(interval)
     finally:
         src.close()

tests/shadow-testing/scripts/sync-queue-hashes.py

@@ -89,36 +89,38 @@ def set_storage(index, value32):
     )
 
 
-def main():
-    dry_run = "--dry-run" in sys.argv
+def run_once(dry_run=False, log_fn=print):
+    """One sync pass. Returns (synced_hashes, (old_next, new_next) or None)."""
     start = next_index(FORK_RPC) - 1  # last index guaranteed present on fork
-    print(f"fork nextCrossDomainMessageIndex-1 = {start}")
+    log_fn(f"fork nextCrossDomainMessageIndex-1 = {start}")
     synced = 0
     i = start + 1
     while synced < MAX_PER_RUN:
         try:
             mainnet_hash = rolling_hash(MAINNET_RPC, i)
         except Exception as e:
-            print(f"mainnet RPC error at index {i}: {e}; stopping")
+            log_fn(f"mainnet RPC error at index {i}: {e}; stopping")
             break
         if mainnet_hash == b"\x00" * 32:
             break  # not enqueued on mainnet yet
         fork_hash = rolling_hash(FORK_RPC, i)
         if fork_hash != mainnet_hash:
-            print(f"index {i}: fork={fork_hash.hex() or '0'} -> mainnet={mainnet_hash.hex()}")
+            log_fn(f"index {i}: fork={fork_hash.hex() or '0'} -> mainnet={mainnet_hash.hex()}")
             if not dry_run:
                 set_storage(i, mainnet_hash)
             synced += 1
         i += 1
-    print(f"done; {'would sync' if dry_run else 'synced'} {synced} rolling hashes")
+    log_fn(f"done; {'would sync' if dry_run else 'synced'} {synced} rolling hashes")
 
     # Keep the fork's enqueue cursor in line with mainnet: finalizing a bundle
     # that pops messages beyond the fork's nextCrossDomainMessageIndex reverts
     # with ErrorFinalizedIndexTooLarge (0x16465978). Slot 103 = 0x67.
+    bumped = None
     mainnet_next = next_index(MAINNET_RPC)
     fork_next = next_index(FORK_RPC)
     if mainnet_next > fork_next:
-        print(f"nextCrossDomainMessageIndex: fork={fork_next} -> mainnet={mainnet_next}")
+        log_fn(f"nextCrossDomainMessageIndex: fork={fork_next} -> mainnet={mainnet_next}")
+        bumped = (fork_next, mainnet_next)
         if not dry_run:
             subprocess.run(
                 [
@@ -129,6 +131,11 @@ def main():
                 check=True,
                 capture_output=True,
             )
+    return synced, bumped
+
+
+def main():
+    run_once(dry_run="--dry-run" in sys.argv)
 
 
 if __name__ == "__main__":