update

Author: lispc

AGENTS.md

@@ -43,6 +43,13 @@ Key hard-won rules:
 - **L1 messages**: If chunks contain L1 messages, prover needs `scroll_getL1MessagesInBlock` RPC support. Most chunks at current mainnet height do NOT contain L1 messages, so this is usually non-blocking.
 - **Anvil MUST fork Ethereum L1, NOT Scroll L2**: The ScrollChain proxy address `0xa13BAF47339d63B743e7Da8741db5456DAc1E556` is on **Ethereum mainnet** (chainId=1), not Scroll mainnet (chainId=534352). If you accidentally point Anvil at a Scroll L2 RPC (e.g., `scroll-mainnet.g.alchemy.com`), the proxy address will have no code or wrong code, and all contract interactions will fail. Always verify `eth_chainId` returns `1` after forking.
 
+### Real-Time Catch-Up Mode — Additional Rules
+
+For continuously proving mainnet bundles in real time (poll-syncing the mainnet DB into the shadow DB), three silent starvation traps apply — see `tests/shadow-testing/docs/TROUBLESHOOTING.md` Trap 22/23:
+- Poll sync must also maintain `l2_block.chunk_hash` links (UPDATE, not INSERT) and `chunk.batch_hash`/`batch.bundle_hash` parent links, or tasks become invisible to the coordinator with no ERROR logged.
+- `sweep-stale-proving.sh` must reset `total_attempts`, not just `proving_status` — the coordinator skips tasks with `total_attempts >= 5`.
+- Bundles popping L1 messages enqueued after the Anvil fork block fail with `VerificationFailed(0x439cc0cd)`: run `tests/shadow-testing/scripts/sync-queue-hashes.py` (cron, 10 min) to mirror `messageRollingHashes` + `nextCrossDomainMessageIndex` from mainnet onto the fork.
+
 ### Sepolia Shadow Fork — Additional Rules
 
 | Dimension | Mainnet | Sepolia | Trap |

tests/shadow-testing/README.md

@@ -59,6 +59,22 @@ cp configs/coordinator.json.template configs/coordinator.json
 #   - YOUR_SHADOW_DB_PASSWORD (in mainnet.json / sepolia.json db.dsn)
 ```
 
+## Real-Time Catch-Up Mode
+
+To follow mainnet's real bundle cadence (instead of replaying a fixed bundle
+range), disable the relayer's local proposers and run the polling sync:
+
+```bash
+python3 scripts/sync-mainnet-db.py --poll-interval 60   # DB rows + l2_block/parent links
+python3 scripts/sync-queue-hashes.py                    # L1 queue rolling hashes (cron every 10 min)
+scripts/sweep-stale-proving.sh                          # stale task sweeper (cron every 10 min)
+```
+
+See `docs/GUIDE.md` → "Real-Time Catch-Up Mode" for the full setup, the
+automation table, and the expected steady state. `docs/TROUBLESHOOTING.md`
+Traps 19–23 cover the failure modes this mode hits (fork-state desync, stale
+prover caches, silent task starvation, post-fork L1 queue hashes).
+
 ## How It Works
 
 The pipeline has three phases:

tests/shadow-testing/docs/GUIDE.md

@@ -1017,10 +1017,27 @@ Critical behavior of the sync (do not bypass):
   - `batch.index > boundary` → `rollup_status = 1` so the shadow relayer commits them on Anvil (requires `parentBatchHash == committedBatches[lastCommittedBatchIndex]`; keep Trap 19's boundary accurate).
   - `bundle` → always `rollup_status = 1`.
 - The boundary is queried from Anvil each poll cycle (`ANVIL_RPC` / `SCROLL_CHAIN` env vars to override), so it advances automatically as the shadow relayer commits new batches.
-- `ON CONFLICT DO NOTHING` everywhere: rows already advanced by the shadow relayer are never overwritten.
+- `ON CONFLICT DO NOTHING` everywhere: rows already advanced by the shadow relayer are never overwritten. **Consequence**: columns populated lazily on mainnet after the row is first copied stay stale in the shadow DB. The sync re-derives them every cycle instead:
+  - `sync_l2_blocks()` — links `l2_block.chunk_hash` for recent chunks (the blocks usually exist from baseline import but with NULL `chunk_hash`; an UPDATE, not an INSERT, is what fixes it). Missing this starves chunk task formatting (Trap 22).
+  - `sync_parent_links()` — re-derives `chunk.batch_hash` and `batch.bundle_hash` from parent index ranges (mainnet sets them at proposal time; rows copied before that keep NULL forever and are invisible to the coordinator's proof-status promotion, Trap 22).
 
 Watch item: the first bundle whose batches were committed on mainnet **after** the fork block exercises the relayer's commit path on Anvil (blob-carrying `commitBatches` tx). Keep `fusaka_timestamp: 2000000000` in the relayer config so Anvil accepts the blob sidecar.
 
+**L1 message queue follow-along (mandatory for long runs)**: bundles that pop L1 messages enqueued after the fork block fail finalization (`VerificationFailed` / `ErrorFinalizedIndexTooLarge`, Trap 23). Run `scripts/sync-queue-hashes.py` periodically — it copies `getMessageRollingHash(i)` from a mainnet RPC into the fork's `messageRollingHashes` mapping (slot 101) and aligns `nextCrossDomainMessageIndex` (slot 103) with mainnet.
+
+**Recommended automation** (what the 48-hour test ran with):
+
+| Cadence | Job | Purpose |
+|---------|-----|---------|
+| 60s loop | `sync-mainnet-db.py --poll-interval 60` | DB row sync + l2_block/parent-link repair |
+| 10 min cron | `scripts/sweep-stale-proving.sh` | Reset stale proving rows **and `total_attempts`** (attempt exhaustion silently starves tasks, Trap 22) |
+| 10 min cron | `scripts/sync-queue-hashes.py` | L1 queue rolling hashes + cursor follow mainnet (Trap 23) |
+| 1 h cron | `scripts/monitor-catchup.py >> .work/catchup-metrics.log` | Hourly metrics snapshot for the final report |
+
+Final report: `SHADOW_REPORT_START=<ISO8601> python3 scripts/generate-catchup-report.py` — the env var windows the report to the current run (the metrics log accumulates across runs).
+
+Expected steady state: mainnet produces ~1 bundle/hour; a 4-GPU fleet proves + finalizes a single-batch bundle in ~10 minutes, so once the initial backlog is cleared the pipeline idles most of the time — provers polling with `CoordinatorEmptyProofData` every ~20s is the normal idle state, not an error.
+
 
 ## Common DB Fixes
 
@@ -1064,3 +1081,8 @@ WHERE chunk_proofs_status != 0
 | `setup.sh` | One-command setup for PostgreSQL, coordinator, or prover |
 | `import-production-data.sh` | Export from production RDS and import to shadow DB |
 | `fetch-l2-blocks.py` | Fetch block headers from L2 RPC and populate `l2_block` table |
+| `sync-mainnet-db.py` | Poll-mode sync of chunk/batch/bundle rows from mainnet RDS, with `rollup_status` boundary fixup, `l2_block` linkage and parent-link (`batch_hash`/`bundle_hash`) repair every cycle |
+| `sync-queue-hashes.py` | Copy `L1MessageQueueV2` rolling hashes + `nextCrossDomainMessageIndex` from ETH mainnet into the Anvil fork (Trap 23) |
+| `sweep-stale-proving.sh` | Reset stale proving rows (chunk/batch 30 min, bundle 180 min) including `total_attempts` |
+| `monitor-catchup.py` | Append one hourly metrics snapshot to `.work/catchup-metrics.log` |
+| `generate-catchup-report.py` | Build the final catch-up report from the metrics log (`SHADOW_REPORT_START` windows it to the current run) |

tests/shadow-testing/docs/TROUBLESHOOTING.md

@@ -213,6 +213,57 @@ Before executing a single command:
 - **Cause**: The prover's local LevelDB (`<prover-workdir>/db`) caches proofs keyed by task. A proof generated under a different circuit/asset version (e.g. before an S3 asset refresh or code revert/restore) is replayed and rejected by the coordinator's shape check.
 - **Rule**: Fresh proofs (proofTime ~300s for bundles) from other provers succeed for the same task, so this is self-healing as long as one prover has a clean cache. To stop a prover from repeatedly burning attempts on a poisoned cache, stop it, delete `<prover-workdir>/db`, and restart. When switching circuit versions, wipe all prover local DBs as part of the reset ritual (same spirit as Trap 16).
 
+### Trap 22: Poll-Synced Chunks Missing `l2_block` Linkage → "failed to fetch block hashes of a chunk"
+
+- **Symptom**: In catch-up mode, after the initial backlog is proved, finalization stalls. Coordinator logs `format prover task failure ... failed to fetch block hashes of a chunk, chunk hash:0x... err:<nil>` on every dispatch attempt; provers spin on `CoordinatorEmptyProofData: get empty prover task` every ~20s; chunks pile up in `proving_status = 1` (with sweeps resetting stale `proving_status = 2` rows every 30 min).
+- **Cause**: `sync-mainnet-db.py` poll mode copies new chunk/batch/bundle rows but historically skipped `l2_block` (watermarking the 181 GB mainnet table by `MAX(number)` times out). New chunks therefore had no block rows linked via `chunk_hash`, and the coordinator cannot format chunk tasks without them. Subtlety: the block rows usually **already exist** in the shadow DB (imported by block number during baseline) but with `chunk_hash = NULL`, so a plain `INSERT ... ON CONFLICT (number) DO NOTHING` copy is a no-op — the linkage must be established with an UPDATE.
+- **Fix**: `sync_l2_blocks()` in `sync-mainnet-db.py` now runs every poll cycle: (1) `UPDATE l2_block SET chunk_hash = chunk.hash` for recent chunks (last 2000) whose blocks lack the link, then (2) copy any genuinely missing block rows from mainnet. One-off manual backfill if needed:
+  ```sql
+  UPDATE l2_block b SET chunk_hash = c.hash FROM chunk c
+  WHERE b.number BETWEEN c.start_block_number AND c.end_block_number
+    AND c.index > (SELECT COALESCE(MAX(index),0) - 2000 FROM chunk)
+    AND (b.chunk_hash IS NULL OR b.chunk_hash <> c.hash);
+  ```
+- **Diagnosis query**: chunks lacking block linkage —
+  ```sql
+  SELECT count(*) FROM chunk c
+  WHERE NOT EXISTS (SELECT 1 FROM l2_block b WHERE b.chunk_hash = c.hash);
+  ```
+- **Note**: `CoordinatorEmptyProofData` every 20s from an idle prover is the **normal** "no task available" poll response, not an error — chunk dispatch is serialized by parent-chunk proof dependency, so only 2-3 chunks prove concurrently even with 4 provers. It only indicates this trap when ALL provers spin AND the coordinator logs block-hash failures.
+- **Secondary damage — attempt exhaustion starvation**: the coordinator picks chunk tasks oldest-first but **skips tasks with `total_attempts >= 5`** (`chunk.go GetUnassignedChunk`: `total_attempts < maxAttempts`). Every failed dispatch during the outage burns one attempt, and `sweep-stale-proving.sh` historically reset only `proving_status`/`active_attempts`, not `total_attempts`. Result: the 44 backlog chunks hit 5/5 attempts and became permanently invisible — the coordinator silently leapfrogged to freshly-synced tip chunks while the backlog starved (symptom: only tip chunks prove, old pending chunks never get sessions, no ERROR logged). Fix: reset attempts after the root cause is repaired —
+  ```sql
+  UPDATE chunk SET total_attempts=0, active_attempts=0
+  WHERE proving_status=1 AND total_attempts>0;
+  ```
+  (same shape for `batch`/`bundle`). `sweep-stale-proving.sh` now resets `total_attempts = 0` on every swept row so future outages self-heal.
+- **Diagnosis query for starvation**: oldest pending chunk with maxed attempts —
+  ```sql
+  SELECT index, total_attempts, active_attempts FROM chunk
+  WHERE proving_status = 1 ORDER BY index LIMIT 5;
+  ```
+- **Tertiary damage — NULL parent links block proof-status promotion**: poll sync copies chunk/batch rows with `ON CONFLICT DO NOTHING`, so rows copied *before* mainnet's proposer assigned them keep `batch_hash` / `bundle_hash` NULL forever. The coordinator_cron promotes `batch.chunk_proofs_status = Ready` only when all chunks joined via `chunk.batch_hash` are verified (`collect_proof.go checkBatchAllChunkReady`), and likewise `bundle.batch_proofs_status` via `batch.bundle_hash`. NULL links → promotion never happens → the batch/bundle is silently invisible to task selection (oldest-first query filters on the status flag; no ERROR is ever logged). Symptom: chunks all verified but batch sessions never start; one stray batch/bundle proves while its older siblings starve. Fix: re-derive links from the parent rows' index ranges —
+  ```sql
+  UPDATE chunk c SET batch_hash = b.hash FROM batch b
+  WHERE c.index BETWEEN b.start_chunk_index AND b.end_chunk_index
+    AND (c.batch_hash IS NULL OR c.batch_hash = '' OR c.batch_hash <> b.hash);
+  UPDATE batch b SET bundle_hash = u.hash FROM bundle u
+  WHERE b.index BETWEEN u.start_batch_index AND u.end_batch_index
+    AND (b.bundle_hash IS NULL OR b.bundle_hash = '' OR b.bundle_hash <> u.hash);
+  ```
+  `sync-mainnet-db.py` runs both every poll cycle (`sync_parent_links()`, bounded to the recent 500 parents).
+
+### Trap 23: Bundles Popping Post-Fork L1 Messages → VerificationFailed (0x439cc0cd), then ErrorFinalizedIndexTooLarge (0x16465978)
+
+- **Symptom**: Most bundles finalize fine, but a bundle whose batches pop L1 messages enqueued **after the Anvil fork block** reverts on-chain with `VerificationFailed` (`0x439cc0cd`). After patching the queue hashes, it then reverts with `ErrorFinalizedIndexTooLarge` (`0x16465978`, appears as raw bytes `\x16FYx` in relayer logs).
+- **Diagnosis flow (all verified during the 2026-07-13 incident, bundle 18070)**:
+  1. Decode the bundle proof's `metadata.bundle_info` from the DB (`bundle.proof` is a JSON blob) and confirm every field matches mainnet DB values.
+  2. Recompute `keccak256(abi.encodePacked(protocolVersion, publicInput))` with the wrapper's packed layout (`chainId 8B | msgQueueHash 32B | numBatches 4B | prevStateRoot | prevBatchHash | postStateRoot | batchHash | withdrawRoot`) and confirm it equals `metadata.bundle_pi_hash`.
+  3. Call the wrapper's `verify(bundleProof, publicInput)` directly with the metadata-derived publicInput. **If it succeeds**, proof/digests are fine and the mismatch is contract-side — ScrollChain computes `messageQueueHash` itself via `L1MessageQueueV2.getMessageRollingHash()`.
+  4. Compare `getMessageRollingHash(i)` on fork vs mainnet (public RPC e.g. `https://ethereum-rpc.publicnode.com` works; Alchemy demo key is 429-rate-limited).
+- **Root cause**: `L1MessageQueueV2` keeps `mapping(uint256=>bytes32) messageRollingHashes` at **storage slot 101** (value = rolling hash with low 32 bits overwritten by enqueue timestamp; the getter clears those bits). Entries for messages enqueued after the fork block are zero on the fork, so the contract builds a publicInput whose `messageQueueHash` differs from the prover's → plonk rejects. Note the **off-by-one**: rh[i] is the hash *after* message i, so a bundle popping to index N needs rh[N-1]; DB `prev/post_l1_message_queue_hash` at popped_before=N equals `getMessageRollingHash(N-1)`.
+- **Fix**: `scripts/sync-queue-hashes.py` copies `getMessageRollingHash(i)` from a mainnet RPC into the fork via `anvil_setStorageAt(queue, keccak256(pad32(i)‖pad32(101)), hash)` and also bumps `nextCrossDomainMessageIndex` (slot 103 = 0x67) to mainnet's value (this is what clears the follow-up `ErrorFinalizedIndexTooLarge`). It is idempotent and runs on a 10-minute cron during catch-up mode. Run it manually after any Anvil restart/re-fork, and whenever a finalize reverts with either selector.
+- **Rule of thumb**: if finalization fails only for bundles whose `post_l1_message_queue_hash != prev_l1_message_queue_hash` (i.e. the batch pops messages), suspect this trap before touching proofs or the verifier wrapper.
+
 ## Step-by-Step Checklist
 
 ### Phase 0: Environment Validation
@@ -285,9 +336,12 @@ Before executing a single command:
 
 | Error / Symptom | Most Likely Cause | See |
 |-----------------|-------------------|-----|
-| `VerificationFailed(0x439cc0cd)` | Wrong verifier type, digest mismatch, **or wrong bundle withdrawRoot** | Trap 1, **Trap 10** |
+| `VerificationFailed(0x439cc0cd)` | Wrong verifier type, digest mismatch, **or wrong bundle withdrawRoot**; in catch-up mode: post-fork L1 queue hashes missing | Trap 1, **Trap 10**, Trap 23 |
 | `ErrorIncorrectBatchHash(0x2a1c1442)` | Sparse `committedBatches`, end batch hash is zero | Trap 4 |
-| `ErrorFinalizedIndexTooLarge(0x16465978)` | `nextUnfinalizedQueueIndex` too low or too high | Trap 5 |
+| `ErrorFinalizedIndexTooLarge(0x16465978)` | `nextUnfinalizedQueueIndex`/`nextCrossDomainMessageIndex` too low | Trap 5, Trap 20, Trap 23 |
+| Catch-up mode stalls: `failed to fetch block hashes of a chunk` | Poll-synced chunks lack `l2_block` linkage | Trap 22 |
+| Old pending chunks never get sessions, no ERROR logged | `total_attempts >= 5` starvation; sweep must reset `total_attempts` | Trap 22 |
+| Chunks verified but batch/bundle sessions never start | NULL `batch_hash`/`bundle_hash` parent links from poll sync | Trap 22 |
 | `record not found` (parent batch) | Parent batch not imported | Trap 6 |
 | `Out of gas: gas required exceeds allowance: 0` | Anvil gas estimation bug with fee caps | Trap 9 |
 | `Insufficient funds for gas * price + value` | Sender balance is 0 on Anvil | Trap 10 |

tests/shadow-testing/scripts/generate-catchup-report.py

@@ -34,6 +34,16 @@ def main():
             except json.JSONDecodeError:
                 continue
 
+    # Optional window filter: only count records at/after SHADOW_REPORT_START
+    # (ISO timestamp), so the report covers the current 48h test rather than
+    # older runs that share the same log file.
+    report_start = os.environ.get("SHADOW_REPORT_START")
+    if report_start:
+        cutoff = datetime.fromisoformat(report_start)
+        if cutoff.tzinfo is None:
+            cutoff = cutoff.replace(tzinfo=timezone.utc)
+        records = [r for r in records if datetime.fromisoformat(r["timestamp"]) >= cutoff]
+
     if not records:
         print("No metrics records found.")
         sys.exit(1)

tests/shadow-testing/scripts/sweep-stale-proving.sh

@@ -0,0 +1,28 @@
+#!/usr/bin/env bash
+# Sweep stale "proving" rows left behind when a prover dies mid-task.
+# The coordinator does not reliably time out chunk/batch tasks, and a single
+# stale row blocks the whole finalize queue (chunks -> batch -> bundle -> finalize).
+#
+# total_attempts is also reset: the coordinator skips tasks with
+# total_attempts >= 5 (see chunk.go GetUnassignedChunk), so an outage that
+# burns all attempts (e.g. Trap 22 block-hash failures) would permanently
+# starve the task even after proving_status is reset.
+#
+# Thresholds (generous vs. observed proof times):
+#   chunk  ~5 min  -> reset after 30 min
+#   batch  ~1 min  -> reset after 30 min
+#   bundle up to ~90 min (30-batch bundles) -> reset after 180 min
+#
+# Usage: ./sweep-stale-proving.sh [DSN]
+set -euo pipefail
+
+DSN="${1:-${SHADOW_DSN:-postgresql://postgres:shadow_pass@localhost:5433/shadow_rollup}}"
+
+psql "$DSN" -Atq <<'SQL'
+UPDATE chunk  SET proving_status = 1, active_attempts = 0, total_attempts = 0
+  WHERE proving_status = 2 AND updated_at < now() - interval '30 minutes';
+UPDATE batch  SET proving_status = 1, active_attempts = 0, total_attempts = 0
+  WHERE proving_status = 2 AND updated_at < now() - interval '30 minutes';
+UPDATE bundle SET proving_status = 1, active_attempts = 0, total_attempts = 0
+  WHERE proving_status = 2 AND updated_at < now() - interval '180 minutes';
+SQL