shadow-testing: fix prover stack overflow by adding RUST_MIN_STACK

- Add export RUST_MIN_STACK=16777216 to 04-prover-up.sh
- Document the batch proof stack overflow issue in LESSONS_LEARNED.md
- Chunk proofs work with default stack, but batch/bundle proofs need 16MB

Author: lispc

tests/shadow-testing/docs/LESSONS_LEARNED.md

@@ -596,3 +596,123 @@ cast call 0xA0673eC0A48aa924f067F1274EcD281A10c5f19F \
 - [ ] Relayer started with `--config <path>` and `--min-codec-version 10`
 - [ ] Target bundles/batches reset to `rollup_status = 1`
 - [ ] Parent batch exists in shadow DB
+
+---
+
+## 2026-06-04: Shadow Testing with New zkvm-prover — Bundles 13450-13454
+
+### What Happened
+Attempted to prove bundles 13450-13454 using the local shadow prover (current build with OpenVM 1.6.0 / zkvm-prover `ed3b964`). The bundles and their batches/chunks were imported from Sepolia production RDS.
+
+### Lesson 1: Import Production Data, Then Clear Proofs and Reset Status
+
+**Wrong approach** (what was done initially):
+- Imported bundles with production proofs from RDS
+- Kept the proofs in the DB
+- Tried to assign bundle tasks directly to the prover
+- Coordinator failed with `ProofEnum deserialization` error because the imported proofs were generated by a different zkvm-prover version
+
+**Correct approach**:
+1. Import raw metadata (batch headers, chunk info, L2 blocks, state roots) from RDS
+2. **Clear all `proof` fields** after import:
+   ```sql
+   UPDATE chunk SET proof = NULL, proving_status = 1, ... WHERE ...;
+   UPDATE batch SET proof = NULL, proving_status = 1, chunk_proofs_status = 0, ... WHERE ...;
+   UPDATE bundle SET proof = NULL, proving_status = 1, batch_proofs_status = 1, ... WHERE ...;
+   ```
+3. Let the local prover regenerate all proofs from scratch
+4. The coordinator verifies the newly-generated proofs
+
+**Why**: Proofs are tied to the specific zkvm-prover / circuit version. The `StarkProof` struct uses `bincode_v1` serialization for `Proof<SC>`, and the binary format changes when the `openvm-stark-backend` or `openvm-sdk` versions change. Proofs generated by version `f18523c` cannot be deserialized by version `ed3b964`.
+
+### Lesson 2: Prover Config Must Use Valid Proof Types
+
+**Wrong config**:
+```json
+"supported_proof_types": [0, 1, 2, 3]
+```
+Type `0` = `ProofTypeUndefined`, which the coordinator rejects with `illegal proof type: 0`.
+
+**Correct config**:
+```json
+"supported_proof_types": [1, 2, 3]
+```
+- `1` = Chunk
+- `2` = Batch
+- `3` = Bundle
+
+### Lesson 3: Prover Binary Takes Only `--config` Flag
+
+**Wrong invocation**:
+```bash
+prover --config prover.json --http.addr 0.0.0.0 --http.port 10080
+```
+The prover binary does NOT accept `--http.addr` or `--http.port`. It only takes `--config <path>`.
+
+**Correct invocation**:
+```bash
+prover --config prover.json
+```
+The listener address goes in the config file:
+```json
+"health_listener_addr": "127.0.0.1:10080"
+```
+
+### Lesson 4: GPU OOM When Running Multiple Provers Concurrently
+
+**Symptom**: Running 4 provers (one per GPU) caused repeated CUDA OOM errors:
+```
+GPU allocation failed: OutOfMemory { requested: 4294967296, available: 2965372928 }
+thread 'tokio-rt-worker' panicked at ... cudaErrorMemoryAllocation: out of memory
+```
+
+**Root Cause**: Each prover allocates a large GPU memory pool for circuit proving. Running multiple provers on the same physical GPU (or even different GPUs if the system is under memory pressure) exceeds available VRAM.
+
+**Mitigation**:
+- Use the script's default of 2 GPUs (`GPUS="0,1"`) instead of 4
+- Or run provers sequentially on a single GPU
+- Monitor GPU memory with `nvidia-smi` during proving
+
+**Note**: With 1-2 provers on RTX 3090s, chunk proof generation takes ~700-800s. Batch and bundle proofs take longer.
+
+### Lesson 5: GORM `[]byte` Mapping for PostgreSQL `bytea` Works Correctly
+
+**Initial suspicion**: The `json.Unmarshal(batch.Proof, &message.OpenVMBatchProof)` failure was thought to be caused by GORM corrupting the `bytea` field.
+
+**Verification**: A standalone Go test using the exact same GORM model and PostgreSQL `bytea` type successfully read and unmarshaled all batch proofs (1,089,806 bytes each).
+
+**Conclusion**: GORM correctly handles PostgreSQL `bytea` fields. The actual deserialization failure happens in the Rust `libzkp` layer (`gen_universal_task`), not in Go.
+
+### Pre-Proving Checklist for New Guest Version Tests
+
+- [ ] Import raw data from RDS (exclude `proof` columns, or clear them after import)
+- [ ] Reset `proving_status` to 1 for all chunks, batches, bundles
+- [ ] Reset `chunk_proofs_status` to 0 for batches
+- [ ] Reset `batch_proofs_status` to 1 for bundles
+- [ ] Prover config has `"supported_proof_types": [1, 2, 3]` (NOT `[0, 1, 2, 3]`)
+- [ ] Prover launched with `--config <path>` only (no `--http.addr`/`--http.port`)
+- [ ] Coordinator asset hashes match prover circuit hashes
+- [ ] GPU count is appropriate for available VRAM (2 GPUs recommended for RTX 3090)
+- [ ] Shadow bypass is configured if testing finalize without real on-chain verification
+
+### 6. Prover Batch Proof Stack Overflow — Set `RUST_MIN_STACK`
+
+**Date**: 2026-06-04
+
+**Symptom**: After chunk proofs succeed, prover crashes during batch proof generation with:
+```
+thread 'tokio-rt-worker' has overflowed its stack
+fatal runtime error: stack overflow, aborting
+```
+Crash occurs in `gen_proof_universal` during aggregation keygen, right after `coset_lde_batch`.
+
+**Root Cause**: The default Rust thread stack size (2 MB) is insufficient for OpenVM 1.6.0 batch/bundle proof generation. Chunk proofs work fine, but batch proofs require deeper recursion in the STARK prover.
+
+**Fix**: Set `RUST_MIN_STACK=16777216` (16 MB) before starting the prover. The `zkvm-prover/Makefile` already sets this, but custom startup scripts must also export it:
+
+```bash
+export RUST_MIN_STACK=16777216
+CUDA_VISIBLE_DEVICES="$gpu_id" nohup "$PROVER_BIN" --config "$config_file" > "$log_file" 2>&1 &
+```
+
+**Lesson**: Always ensure `RUST_MIN_STACK` is exported in prover startup scripts, not just in the build Makefile.

tests/shadow-testing/scripts/04-prover-up.sh

@@ -90,6 +90,7 @@ EOF
 
     log_info "Starting prover on GPU $gpu_id..."
 
+    export RUST_MIN_STACK=16777216
     CUDA_VISIBLE_DEVICES="$gpu_id" nohup "$PROVER_BIN" \
         --config "$config_file" \
         > "$log_file" 2>&1 &