tests/shadow-testing: follow mode forks N hours back, adds catch-up phase

- 10-follow-up.sh: FOLLOW_FORK_HOURS_BACK (default 5) — fork at
  tip - hours*300 blocks so the run opens with a real backlog
  (committed-but-not-finalized at fork + everything produced since);
  lastFinalized/lastCommitted now read AT the fork block; 0 = old
  fork-at-tip behavior. follow-run.env records FORK_HOURS_BACK and
  MAINNET_BUNDLE_TIP_AT_START.
- generate-catchup-report.py: split report into catch-up phase (fork
  finalized index climbing to the recorded mainnet bundle tip) and
  steady-state phase; use the max_finalized_bundle watermark for
  finalized counts; backward-compatible with old env files.
- docs: follow README/GUIDE + AGENTS.md describe catch-up-then-follow.

Author: lispc

AGENTS.md

@@ -24,7 +24,7 @@ Follow the structured testing guide in [`docs/testing/openvm-upgrade-testing-gui
 
 For testing proof generation against **real mainnet production tasks** without interfering with the live system, use the **Shadow Coordinator** approach. This is significantly faster than a full shadow fork. There are two test modes, each in its own directory under `tests/shadow-testing/`:
 
-- **Follow mode (primary/default)** — `cd tests/shadow-testing/follow && make follow` forks the current ETH mainnet state and follows mainnet bundle production in real time (poll-sync → prove → finalize, default 48h window). This is the default acceptance test for prover/guest upgrades. Operations: `make follow-status`, `make follow-report`, `make follow-stop`, `make re-fork`.
+- **Follow mode (primary/default)** — `cd tests/shadow-testing/follow && make follow` forks the ETH mainnet state `FOLLOW_FORK_HOURS_BACK` hours in the past (default 5h; `0` = current tip), catches up the resulting backlog, then follows mainnet bundle production in real time (poll-sync → prove → finalize, default 48h window). This is the default acceptance test for prover/guest upgrades. Operations: `make follow-status`, `make follow-report`, `make follow-stop`, `make re-fork`.
 - **Snapshot replay mode (specialized)** — `cd tests/shadow-testing/snapshot`: fork a historical block, import a fixed bundle range, prove & finalize ~N bundles (`make all` / `make docker-all` / `make sepolia-all`). Use for incident reproduction, single-bundle debugging, Sepolia testing, targeted codec-migration checks.
 
 Shared details:

tests/shadow-testing/follow/GUIDE.md

@@ -1,6 +1,6 @@
 # Follow Mode Guide — Shadow Coordinator + Prover Testing
 
-This guide documents the **follow mode** of the shadow coordinator + local prover environment: fork the current ETH mainnet state and follow mainnet bundle production in real time. This is the **default acceptance test for prover/guest upgrades**.
+This guide documents the **follow mode** of the shadow coordinator + local prover environment: fork the ETH mainnet state from `FOLLOW_FORK_HOURS_BACK` hours in the past (default 5h), catch up the resulting backlog, then follow mainnet bundle production in real time. This is the **default acceptance test for prover/guest upgrades**.
 
 For a fixed historical bundle range (incident reproduction, single-bundle debugging, Sepolia testing, targeted codec-migration checks), see the [Snapshot Replay Mode Guide](../snapshot/GUIDE.md). Shared scripts live in [`../lib/`](../lib/), pitfalls in [`./TROUBLESHOOTING.md`](./TROUBLESHOOTING.md) (follow mode) and [`../docs/COMMON-TROUBLESHOOTING.md`](../docs/COMMON-TROUBLESHOOTING.md) (mode-independent). Runtime state (`.work/`) is shared by both modes at `tests/shadow-testing/.work` (i.e. `../.work` from here).
 
@@ -41,7 +41,7 @@ For a fixed historical bundle range (incident reproduction, single-bundle debugg
 
 # Follow Mode (Primary)
 
-Follow mode forks the **current** ETH mainnet state with Anvil, baseline-syncs the shadow DB from the mainnet read replica, then keeps following mainnet indefinitely — poll-syncing new chunk/batch/bundle rows, proving them with local GPU provers, and finalizing bundles on the fork — until a preset duration (`FOLLOW_RUN_HOURS`, default 48) or a failure. It answers the question a snapshot cannot: *can this prover fleet keep pace with real mainnet bundle production?*
+Follow mode forks the ETH mainnet state from **`FOLLOW_FORK_HOURS_BACK` hours in the past** (default 5; `0` = fork at the current tip) with Anvil, baseline-syncs the shadow DB from the mainnet read replica, then **catches up the backlog** (every bundle committed-but-not-finalized at the fork block, plus everything mainnet produced since) before **following** mainnet indefinitely — poll-syncing new chunk/batch/bundle rows, proving them with local GPU provers, and finalizing bundles on the fork — until a preset duration (`FOLLOW_RUN_HOURS`, default 48) or a failure. It answers the question a snapshot cannot: *can this prover fleet catch up a backlog and then keep pace with real mainnet bundle production?* The fork block is `tip - FOLLOW_FORK_HOURS_BACK*300` (12s L1 blocks); `lastFinalizedBatchIndex` and `lastCommittedBatchIndex` are read **at the fork block**, so the baseline window (fork-finalized → mainnet tip) is exactly the backlog. `make follow-report` splits metrics into catch-up and steady-state phases using `MAINNET_BUNDLE_TIP_AT_START` recorded in `.work/follow-run.env`.
 
 For a fixed historical bundle range (incident reproduction, single-bundle debugging, Sepolia, codec-migration checks), use the [Snapshot Replay Mode Guide](../snapshot/GUIDE.md).
 

tests/shadow-testing/follow/README.md

@@ -1,6 +1,6 @@
 # Follow Mode (Primary)
 
-Follow mode forks the **current** ETH mainnet state, baseline-syncs the shadow DB from the mainnet read replica, then follows mainnet indefinitely — poll-syncing new chunk/batch/bundle rows, proving them with local GPU provers, and finalizing on the fork — for a preset window (default 48h). It is the **default acceptance test for prover/guest upgrades**: it answers whether the prover fleet can keep pace with real mainnet bundle production.
+Follow mode forks the ETH mainnet state from **FOLLOW_FORK_HOURS_BACK hours in the past** (default 5h), baseline-syncs the shadow DB from the mainnet read replica, then first **catches up** the resulting backlog (all bundles committed-but-not-finalized at the fork block plus everything mainnet produced since) and then **follows** mainnet indefinitely — poll-syncing new chunk/batch/bundle rows, proving them with local GPU provers, and finalizing on the fork — for a preset window (default 48h). It is the **default acceptance test for prover/guest upgrades**: it answers whether the prover fleet can catch up a backlog *and* keep pace with real mainnet bundle production. `FOLLOW_FORK_HOURS_BACK=0` forks at the current tip (no backlog). The final report splits metrics into catch-up and steady-state phases.
 
 ## Quick Start
 

tests/shadow-testing/follow/scripts/10-follow-up.sh

@@ -1,10 +1,11 @@
 #!/usr/bin/env bash
 # 10-follow-up.sh — one-shot orchestrator for "follow mode" shadow testing.
 #
-# Brings up the full follow-mode stack against the CURRENT mainnet tip:
+# Brings up the full follow-mode stack against a RECENT mainnet state:
 #   a. sanity checks (postgres 5433, mainnet DSN, alchemy key, stale ports)
 #   b. baseline DB sync from the mainnet read replica
-#   c. Anvil fork at the current latest L1 block (via 01-setup-anvil.sh)
+#   c. Anvil fork at tip - FOLLOW_FORK_HOURS_BACK*300 blocks (default 5h back,
+#      via 01-setup-anvil.sh) so the run opens with a real backlog to catch up
 #   d. verifier wrapper deploy + registration (via 03-deploy-verifier.sh)
 #   e. initial L1 queue rolling-hash sync (Trap 23)
 #   f. coordinator_api + coordinator_cron
@@ -14,7 +15,8 @@
 #   j. records run metadata in .work/follow-run.env
 #
 # Idempotent-ish: a component whose pidfile exists and is alive is skipped.
-# Usage: ./10-follow-up.sh [--config configs/mainnet.json] [--dry-run]
+# Usage: ./10-follow-up.sh [--config configs/mainnet.json] [--dry-run] [--reset]
+# Env: FOLLOW_FORK_HOURS_BACK=0 forks at the current tip (no backlog).
 
 set -euo pipefail
 
@@ -31,6 +33,11 @@ WORK_DIR="${WORK_DIR:-${PROJECT_ROOT}/../.work}"
 # Default mirrors sync-mainnet-db.py; override via env if the tunnel differs.
 MAINNET_DSN="${MAINNET_DSN:-postgresql://mainnet_infra_team_read_only:AuexDUuaarskbG6tr9CH9gXsJqp4at67mddAbMrt@localhost:15432/mainnet_rollup}"
 FOLLOW_RUN_HOURS="${FOLLOW_RUN_HOURS:-48}"
+# Fork the L1 state this many hours in the PAST, so the run starts with a
+# real backlog (all bundles committed-but-not-finalized at the fork block,
+# plus everything mainnet produced since) to catch up before steady-state
+# following. 0 = fork at the current tip (old behavior, near-idle start).
+FOLLOW_FORK_HOURS_BACK="${FOLLOW_FORK_HOURS_BACK:-5}"
 SYNC_POLL_INTERVAL="${SYNC_POLL_INTERVAL:-60}"
 SWEEP_INTERVAL="${SWEEP_INTERVAL:-600}"
 MONITOR_INTERVAL="${MONITOR_INTERVAL:-3600}"
@@ -157,12 +164,33 @@ check_port_free 8391 coordinator-cron
 check_port_free 18545 anvil
 
 # ─── b. Baseline DB sync ─────────────────────────────────────────────────────
-# Window = mainnet's finalization lag, NOT history: L1 finalization is
-# sequential (each finalize tx chains prevBatchHash), so the shadow must prove
-# and finalize every committed-but-not-finalized bundle starting from the
-# fork's lastFinalizedBatchIndex. Anything older is already finalized on the
-# fork and is copied only as a parent reference (marked done post-baseline).
+# Window = everything NOT finalized at the fork block, up to the mainnet tip:
+# L1 finalization is sequential (each finalize tx chains prevBatchHash), so the
+# shadow must prove and finalize every committed-but-not-finalized bundle
+# starting from the fork's lastFinalizedBatchIndex. With FOLLOW_FORK_HOURS_BACK
+# > 0 this window covers the fork-back backlog PLUS everything mainnet produced
+# between the fork block and now — that backlog is the run's catch-up phase.
+# Anything older is already finalized on the fork and is copied only as a
+# parent reference (marked done post-baseline).
+#
+# Resolve the fork block FIRST (rather than passing "latest") so that (a) the
+# point is logged and reproducible, and (b) lastFinalized/lastCommitted below
+# are read AT the fork block, not at the tip — with a fork-back hours this
+# difference is exactly the backlog size. 300 blocks/hour = 12s L1 block time.
 log_info "=== b. Baseline DB sync ==="
+LATEST_BLOCK=$(cast block-number --rpc-url "$FORK_URL")
+if [[ "$FOLLOW_FORK_HOURS_BACK" -gt 0 ]]; then
+    FORK_BLOCK_NUMBER=$((LATEST_BLOCK - FOLLOW_FORK_HOURS_BACK * 300))
+else
+    FORK_BLOCK_NUMBER=$LATEST_BLOCK
+fi
+log_info "  fork block:        $FORK_BLOCK_NUMBER (latest $LATEST_BLOCK, ${FOLLOW_FORK_HOURS_BACK}h back)"
+LAST_FINALIZED=$(cast call "$SCROLL_CHAIN" "lastFinalizedBatchIndex()(uint256)" --rpc-url "$FORK_URL" --block "$FORK_BLOCK_NUMBER" | awk '{print $1}')
+MAINNET_TIP=$(psql "$MAINNET_DSN" -Atq -c "SELECT MAX(index) FROM batch" | tr -d ' ')
+MAINNET_BUNDLE_TIP=$(psql "$MAINNET_DSN" -Atq -c "SELECT MAX(index) FROM bundle" | tr -d ' ')
+BASELINE_START=$((LAST_FINALIZED - 1))
+log_info "  lastFinalizedBatchIndex@fork: $LAST_FINALIZED; mainnet batch tip: $MAINNET_TIP"
+log_info "  baseline window: ${BASELINE_START}..${MAINNET_TIP} ($((MAINNET_TIP - BASELINE_START)) batches = backlog + finalization lag)"
 if $RESET_DB; then
     # --reset: wipe task tables so the DB contains ONLY the finalization-lag
     # window. Without this, stale rows from earlier runs/imports (thousands of
@@ -198,11 +226,6 @@ if $RESET_DB; then
         rm -f "${WORK_DIR}/anvil-${CONFIG_NAME}.state.json" "${WORK_DIR}/verifier.env" "${WORK_DIR}/follow-run.env"
     fi
 fi
-LAST_FINALIZED=$(cast call "$SCROLL_CHAIN" "lastFinalizedBatchIndex()(uint256)" --rpc-url "$FORK_URL" | awk '{print $1}')
-MAINNET_TIP=$(psql "$MAINNET_DSN" -Atq -c "SELECT MAX(index) FROM batch" | tr -d ' ')
-BASELINE_START=$((LAST_FINALIZED - 1))
-log_info "  lastFinalizedBatchIndex: $LAST_FINALIZED; mainnet batch tip: $MAINNET_TIP"
-log_info "  baseline window: ${BASELINE_START}..${MAINNET_TIP} ($((MAINNET_TIP - BASELINE_START)) batches = finalization lag)"
 export MAINNET_DSN
 run_step python3 "${SCRIPT_DIR}/sync-mainnet-db.py" --init-from-batch "$BASELINE_START"
 # Boundary rows (<= lastFinalized) are already finalized on the fork: mark
@@ -214,7 +237,7 @@ run_step python3 "${SCRIPT_DIR}/sync-mainnet-db.py" --init-from-batch "$BASELINE
 # so the relayer re-commits/re-finalizes them in chain order. Proofs already
 # generated stay valid (proof content depends on L2 data, not L1 state).
 if ! $DRY_RUN; then
-    MISC_RAW=$(cast call "$SCROLL_CHAIN" 0x06582acb --rpc-url "$FORK_URL")
+    MISC_RAW=$(cast call "$SCROLL_CHAIN" 0x06582acb --rpc-url "$FORK_URL" --block "$FORK_BLOCK_NUMBER")
     FORK_COMMITTED=$((16#${MISC_RAW:2:64}))
     log_info "  fork lastCommittedBatchIndex: $FORK_COMMITTED"
     psql "$SHADOW_DSN" -Atq -c "
@@ -234,11 +257,10 @@ if ! $DRY_RUN; then
     log_ok "  rollup_status aligned to fork boundaries (finalized<=${LAST_FINALIZED}, committed<=${FORK_COMMITTED})"
 fi
 
-# ─── c. Anvil fork at the CURRENT latest block ───────────────────────────────
-log_info "=== c. Anvil fork at latest mainnet block ==="
-# Resolve the block number explicitly (rather than passing "latest") so the
-# fork point is logged and reproducible.
-FORK_BLOCK_NUMBER=$(cast block-number --rpc-url "$FORK_URL")
+# ─── c. Anvil fork at the chosen block ───────────────────────────────────────
+log_info "=== c. Anvil fork setup ==="
+# FORK_BLOCK_NUMBER was resolved in step b (tip, or tip - FOLLOW_FORK_HOURS_BACK
+# * 300). lastFinalized was read at that same block there.
 log_info "  fork block:        $FORK_BLOCK_NUMBER"
 log_info "  lastFinalized:     $LAST_FINALIZED (read from mainnet at fork block)"
 log_info "  nextQueueIndex:    $NEXT_QUEUE"
@@ -435,7 +457,10 @@ SHADOW_REPORT_START=${REPORT_START}
 FOLLOW_RUN_HOURS=${FOLLOW_RUN_HOURS}
 EXPECTED_PROTOCOL_VERSION=${EXPECTED_PROTOCOL_VERSION}
 FORK_BLOCK=${FORK_BLOCK_NUMBER}
+FORK_HOURS_BACK=${FOLLOW_FORK_HOURS_BACK}
 BASELINE_START_BATCH=${BASELINE_START}
+MAINNET_TIP_AT_START=${MAINNET_TIP}
+MAINNET_BUNDLE_TIP_AT_START=${MAINNET_BUNDLE_TIP}
 EOF
     log_ok "  wrote $FOLLOW_ENV (SHADOW_REPORT_START=$REPORT_START, FOLLOW_RUN_HOURS=$FOLLOW_RUN_HOURS)"
 fi

tests/shadow-testing/follow/scripts/generate-catchup-report.py

@@ -59,7 +59,51 @@ def main():
     anvil_end = records[-1].get("anvil_block", 0)
 
     bundles_created = last_summary.get("max_bundle", 0) - first_summary.get("max_bundle", 0)
-    bundles_finalized = last_summary.get("finalized_bundles", 0) - first_summary.get("finalized_bundles", 0)
+    # max_finalized_bundle is a monotonic watermark (shadow fork's finalized
+    # bundle index); finalized_bundles per record is an interval count, so the
+    # watermark delta is the robust total.
+    bundles_finalized = last_summary.get("max_finalized_bundle", 0) - first_summary.get("max_finalized_bundle", 0)
+
+    # Run metadata (written by 10-follow-up.sh) — used to split the report into
+    # a catch-up phase (fork finalized index climbing to the mainnet bundle tip
+    # recorded at run start) and a steady-state phase (everything after).
+    run_env = {}
+    env_path = os.environ.get(
+        "SHADOW_FOLLOW_ENV",
+        os.path.join(os.path.dirname(LOG_FILE), "follow-run.env"),
+    )
+    if os.path.exists(env_path):
+        for line in open(env_path):
+            if "=" in line:
+                k, v = line.strip().split("=", 1)
+                run_env[k] = v
+    bundle_tip_at_start = run_env.get("MAINNET_BUNDLE_TIP_AT_START")
+    bundle_tip_at_start = int(bundle_tip_at_start) if bundle_tip_at_start else None
+    fork_hours_back = run_env.get("FORK_HOURS_BACK")
+
+    catchup_end_idx = None
+    if bundle_tip_at_start is not None:
+        for i, r in enumerate(records):
+            if r["summary"].get("max_finalized_bundle", 0) >= bundle_tip_at_start:
+                catchup_end_idx = i
+                break
+
+    def phase_stats(recs):
+        """Return (hours, bundles_created_delta, bundles_finalized_sum, avg proof times)."""
+        if not recs:
+            return 0.0, 0, 0, None, None, None
+        hours = (datetime.fromisoformat(recs[-1]["timestamp"]) - datetime.fromisoformat(recs[0]["timestamp"])).total_seconds() / 3600
+        created = recs[-1]["summary"].get("max_bundle", 0) - recs[0]["summary"].get("max_bundle", 0)
+        finalized = recs[-1]["summary"].get("max_finalized_bundle", 0) - recs[0]["summary"].get("max_finalized_bundle", 0)
+
+        def mean(key):
+            vals = [r["summary"][key] for r in recs if r["summary"].get(key)]
+            return round(sum(vals) / len(vals)) if vals else None
+
+        return hours, created, finalized, mean("avg_chunk_proof_time"), mean("avg_batch_proof_time"), mean("avg_bundle_proof_time")
+
+    catchup_stats = phase_stats(records[: catchup_end_idx + 1]) if catchup_end_idx is not None else None
+    steady_stats = phase_stats(records[catchup_end_idx + 1 :]) if catchup_end_idx is not None else None
 
     # Pull average proof times from the latest record
     avg_chunk = last_summary.get("avg_chunk_proof_time")
@@ -92,8 +136,24 @@ def main():
     print("# 48-Hour Shadow Catch-Up Report")
     print(f"- Duration: {duration_hours:.2f} hours ({start.isoformat()} -> {end.isoformat()})")
     print(f"- Anvil blocks: {anvil_start} -> {anvil_end} ({anvil_end - anvil_start} blocks)")
+    if fork_hours_back is not None:
+        print(f"- Fork: {fork_hours_back}h back, mainnet bundle tip at start: {bundle_tip_at_start}")
     print()
-    print("## Bundle Throughput")
+    if catchup_stats is not None:
+        ch, cc, cf, c_chunk, c_batch, c_bundle = catchup_stats
+        sh, sc, sf, s_chunk, s_batch, s_bundle = steady_stats
+        print("## Catch-Up Phase (backlog at fork)")
+        print(f"- Duration to catch up: {ch:.2f} hours")
+        print(f"- Bundles finalized during catch-up: {cf}")
+        print(f"- Avg proof times (interval means): chunk {c_chunk}s / batch {c_batch}s / bundle {c_bundle}s")
+        print()
+        print("## Steady-State Phase (following mainnet)")
+        print(f"- Duration: {sh:.2f} hours")
+        print(f"- Bundles created on mainnet: {sc}")
+        print(f"- Bundles finalized on fork: {sf}")
+        print(f"- Avg proof times (interval means): chunk {s_chunk}s / batch {s_batch}s / bundle {s_bundle}s")
+        print()
+    print("## Bundle Throughput (whole window)")
     print(f"- Bundles created (max_bundle delta): {bundles_created}")
     print(f"- Bundles finalized: {bundles_finalized}")
     print(f"- Finalization / creation ratio: {(bundles_finalized / bundles_created * 100):.1f}%" if bundles_created else "- No bundles created")