shadow-testing: drain inherited blob gas on fresh fork; re-ensure isProver in follow-up (traps 28/29)

Author: lispc

tests/shadow-testing/follow/TROUBLESHOOTING.md

@@ -95,3 +95,17 @@ Three independent things must hold for the relayer to send its first on-fork com
 
 Also note: the relayer writes `rollup_status` **only** through its commit/finalize confirmation path (GORM `UPDATE … SET finalize_tx_hash, rollup_status`). If a batch/bundle shows `rollup_status = 5` with NULL `finalize_tx_hash` while `lastFinalizedBatchIndex` on the fork hasn't moved, do not trust the DB row — cross-check on-chain. `log_statement = 'mod'` on the shadow postgres is a cheap way to attribute every status write; it is asserted automatically by `02-prepare-db.sh` and `10-follow-up.sh` (re-applied on every setup, since `ALTER SYSTEM` lives in the container's data volume and is lost when the volume is recreated). Read the writes with `docker logs shadow-postgres` (or the postgres server log on a non-docker setup).
 
+
+### Trap 28: Post-Fusaka Fork + Anvil 1.0.0 → Blob Base Fee Explosion, Commits Fail "Insufficient funds" [follow mode]
+
+- **Symptom**: Right after a fresh fork, every `commitBatches` attempt fails with `estimateGasLimit failure ... Insufficient funds` (EOA balance is fine — verified 100 ETH). Batches sit at `rollup_status = 1` forever; `cast blob-base-fee` on the fork returns ~1e18 wei.
+- **Cause**: Anvil 1.0.0 predates the Fusaka fork. Forking post-Fusaka mainnet state inherits a large `excessBlobGas` (~1.8e8 when the mainnet blob market is congested), but Anvil prices blob gas with the **Dencun** update fraction (3338477), so the inherited excess maps to an astronomical blob base fee. Commit txs carry blob versioned hashes and become unaffordable.
+- **Fix (codified)**: `lib/01-setup-anvil.sh` now mines empty blocks right after `wait_for_anvil` until `cast blob-base-fee` drops below 1 gwei (excess decays ~1/8 per empty block; ~400 blocks suffice from 1.8e8). Manual equivalent: `cast rpc anvil_mine 400 --rpc-url http://localhost:18545`.
+- **Note**: Forking at tip does NOT dodge this — it only depends on the fork block's `excessBlobGas`, which is high whenever mainnet blob backlog is high. Long-term fix is upgrading Anvil to a Fusaka-aware release.
+
+### Trap 29: Same-Block Ordering — Manual `addProver` Mined After the Failing Finalize [follow mode]
+
+- **Symptom**: A `finalizeBundlePostEuclidV2` tx lands on-chain but reverts with `ErrorCallerIsNotProver` (0x7b263b17) even though `addProver` was sent first; the bundle ends up at `rollup_status = 7` (RollupFinalizeFailed), which `ProcessPendingBundles` **never retries**.
+- **Cause**: Anvil mines both txs in the same block and orders the finalize (txIndex 0) before the `addProver` (txIndex 1). The finalize legitimately reverts at execution time. (Also reachable via an Anvil state restore from a pre-addProver backup — see Trap 27.)
+- **Fix (codified)**: `10-follow-up.sh` step h now re-ensures `isProver(finalize_sender)` idempotently on every run, alongside the Trap-27 balance/sequencer checks. Manual recovery: impersonate the owner, `cast send … "addProver(address)" <finalize_eoa> --unlocked`, verify `isProver` = true, then `UPDATE bundle SET rollup_status = 1 WHERE index = <n>` so the relayer retries.
+- **Rule of thumb**: a bundle at `rollup_status = 7` is stranded by design (relayer logs it loudly). It always needs the manual `rollup_status = 1` reset after fixing the underlying cause.

tests/shadow-testing/follow/scripts/10-follow-up.sh

@@ -404,7 +404,19 @@ if ! $DRY_RUN; then
             || { cast rpc anvil_stopImpersonatingAccount "$OWNER" --rpc-url "$ANVIL_RPC" >/dev/null; log_error "addSequencer failed"; exit 1; }
         cast rpc anvil_stopImpersonatingAccount "$OWNER" --rpc-url "$ANVIL_RPC" >/dev/null
     fi
-    log_ok "  relayer EOAs funded (100 ETH), commit sender is sequencer"
+    # isProver gates finalizeBundlePostEuclidV2 (ErrorCallerIsNotProver
+    # 0x7b263b17). An Anvil state restore from a pre-addProver backup silently
+    # drops it, and a failed finalize then strands the bundle at
+    # rollup_status=7 (not retried). Re-ensure it here, idempotently.
+    if [[ "$(cast call "$SCROLL_CHAIN" "isProver(address)(bool)" "$FINALIZE_SENDER" --rpc-url "$ANVIL_RPC" 2>/dev/null)" != "true" ]]; then
+        log_warn "  authorizing $FINALIZE_SENDER as prover on the fork"
+        cast rpc anvil_impersonateAccount "$OWNER" --rpc-url "$ANVIL_RPC" >/dev/null
+        cast send --gas-limit 5000000 "$SCROLL_CHAIN" "addProver(address)" "$FINALIZE_SENDER" \
+            --from "$OWNER" --rpc-url "$ANVIL_RPC" --unlocked >/dev/null \
+            || { cast rpc anvil_stopImpersonatingAccount "$OWNER" --rpc-url "$ANVIL_RPC" >/dev/null; log_error "addProver failed"; exit 1; }
+        cast rpc anvil_stopImpersonatingAccount "$OWNER" --rpc-url "$ANVIL_RPC" >/dev/null
+    fi
+    log_ok "  relayer EOAs funded (100 ETH), commit sender is sequencer, finalize sender is prover"
 fi
 if pid_alive "${WORK_DIR}/relayer-${CONFIG_NAME}.pid"; then
     log_info "  relayer already running (pid $(cat "${WORK_DIR}/relayer-${CONFIG_NAME}.pid")), skipping"

tests/shadow-testing/lib/01-setup-anvil.sh

@@ -124,6 +124,25 @@ fi
 
 wait_for_anvil "$ANVIL_RPC"
 
+# ─── Step 1b: Drain inherited excess blob gas ────────────────────────────────
+# Anvil 1.0.0 predates Fusaka: when forking post-Fusaka mainnet state it keeps
+# the forked excessBlobGas but prices blob gas with the Dencun formula, so the
+# blob base fee explodes (~1e18 wei) and every commit tx fails with
+# "Insufficient funds". Mine empty blocks until the excess drains away
+# (excess decays by ~1/8 per empty block under Dencun rules).
+log_info "Draining inherited excess blob gas..."
+for _ in $(seq 1 8); do
+    blob_fee=$(cast blob-base-fee --rpc-url "$ANVIL_RPC" 2>/dev/null || echo 0)
+    # Stop once the blob base fee is below 1 gwei (1e9 wei).
+    if [[ -z "$blob_fee" || "$blob_fee" -lt 1000000000 ]]; then
+        break
+    fi
+    log_info "  blob base fee = $blob_fee wei; mining 100 empty blocks..."
+    cast rpc anvil_mine 100 --rpc-url "$ANVIL_RPC" >/dev/null 2>&1
+done
+blob_fee=$(cast blob-base-fee --rpc-url "$ANVIL_RPC" 2>/dev/null || echo unknown)
+log_ok "  blob base fee = $blob_fee wei"
+
 # ─── Step 2: Reset ScrollChain miscData ──────────────────────────────────────
 log_info "Resetting ScrollChain state..."
 log_info "  lastFinalizedBatchIndex → $LAST_FINALIZED"