Disk-spill for proving 32M program (#466)

* Remove LDE spilling, reuse reconstruct_round1 for Rounds 2-4

* Remove no-op late trace re-spill

* Remove Merkle tree node spilling

* Revert "Remove Merkle tree node spilling"

This reverts commit e7e5769.

* Disk-spill composition poly and FRI layer Merkle trees

* Disk-spill FRI layer evaluations and composition poly evaluations

* Lazy-allocate pool buffers instead of pre-allocating at max size

* Bring back LDE spilling for Rounds 2-4 to reduce peak pool memory

* Free pool buffers between phases to reduce peak memory

* Spill LDE and free pool inside parallel closure to reduce peak memory

* Use k_commit=1 for Phase A/C to avoid multiple large pools alive simultaneously

* Drop trace mmap pages from page cache after extracting into pool buffers

* Restore main_commits_elapsed timing for instruments feature

* Add per-phase heap profiling to instruments

* Pre-allocate pool to max_lde_size and keep capacity between tables

* Spill aux traces to disk immediately after building

* Fix clippy unused_mut warnings when disk-spill is disabled

* Add 16M-128M sizes to bench_heap_profile.sh

* Default k_commit=4 for disk-spill Phase A/C to balance memory and parallelism

* Revert "Default k_commit=4 for disk-spill Phase A/C to balance memory and parallelism"

This reverts commit 8d0d6a7.

* Skip composition poly and FRI spills in Rounds 2-4 to reduce I/O overhead

* Remove max-rows CLI arg

* Remove heap profiling infrastructure (moved to separate PR)

* Use prover::prove for default blowup path

* Simplify MmapNodeBacking doc comment

* Clarify clone assert message for spilled MerkleTree

* Disable Clone for MerkleTree and FriLayer when disk-spill is enabled

* Simplify unsafe safety comment in node_get

* Add compile-time alignment assert for mmap node access

* Simplify spill_nodes_to_disk doc comment

* Fix safety comment for node byte cast

* Simplify safety comment

* Explain why tempfile is exclusively owned in safety comment

* Document _file field in MmapNodeBacking

* Document _file field, remove unused _len from EvalMmapBacking

* Document EvalMmapBacking struct

* Add safety comment to get_evaluation mmap access

* Add safety comments to spill_evaluation_to_disk

* Inline FriLayer::new back into push

* Document Round2EvalMmap struct

* Add safety comment to get_composition_eval mmap access

* Add safety comments to spill_evaluations_to_disk

* Add comments explaining advise_drop_cache calls

* Remove leftover spill comments

* Remove unnecessary &mut on round_2_result

* Simplify trace spill comment

* Clean up k_commit and pool allocation comments

* Remove leftover empty snap calls

* Simplify spilled_ldes comment

* Simplify Rounds 2-4 section comment

* Remove 'original flow' from comment

* Remove end-of-block comment

* Clarify why TableMmapBacking implements Clone

* Remove unnecessary Default impl for TableMmapBacking

* Fix safety comment in get_row

* Remove obvious comment in extract_columns_into

* Fix mmap safety comment in spill_to_disk

* Document TableMmapBacking fields

* Add safety comment and fix doc for advise_drop_cache

* Shorten advise_drop_cache safety comment

* Restore original halt timestamp comment

* Restore Phase 3 comment in prove_with_options

* Simplify spill_main_to_disk doc

* Simplify MmapBacking doc and document _file fields

* Fix safety comments in get_main and get_aux

* Inline safety comment for get_aux

* Simplify spill_main_from_pool doc

* Simplify write_pool_columns_to_mmap doc

* Fix mmap safety comment in write_pool_columns_to_mmap

* Fix generic type name in safety comment

* Fix formatting

* Add comment for TLS instruments pattern

* Extract round1_from_lde to deduplicate disk-spill Round1 construction

* Fix precomputed tree spill error propagation, remove dead spill code

* Remove trivial FriLayer::get_evaluation wrapper

* Remove trivial Round2 wrapper methods

* Document repr(transparent) requirement for mmap casts

* Assert mmap bounds in release builds

* Close mmap-backing file descriptors after creation

* Check spill byte counts for overflow

* Validate elem_size matches size_of in write_pool_columns_to_mmap

* Compare tables by field-element equality

* Propagate aux trace spill errors through ProvingError

* Advise page cache drop for aux mmap after extract

* Run disk-spill tests in CI

* Cache LDE in RAM instead of spilling to disk

* Fix non-parallel disk-spill build by declaring spill_iter mut

* Drop trace_lde from TableSubOps

* Add fib_iterative_24M program

* Remove fib_iterative_32M/64M/128M programs

* Revert unrelated fib_iterative_2M tweak

* Tighten FieldElement repr invariant doc

* Harden disk-spill Table safety and portability

* Add DiskSpill proving error variant

* Match instruments label to function name

* Drop heading from FieldElement doc

* Reword FieldElement doc without metaphor

* Drop 'silently' from FieldElement doc

* Make disk-spill opt-in for CLI

* Add clippy pass for disk-spill feature

* Fsync spill files before mmap

* Spill via MmapMut instead of write

* Make disk-spill default for CLI

* Spill memw_aligned and memw_register tables too

* Disk spilling based on heap usage (#557)

* Compile disk-spill types unconditionally

* Dispatch disk-spill via runtime StorageMode

* Remove disk-spill feature flag

* Auto-pick StorageMode from available RAM

* Add ProofOptions.max_ram_bytes cap

* Add tests for auto-storage selection

* Drop disk-spill feature from CI and Makefile

* Address bot review comments

* Honor max_ram_bytes during trace build

* Saturate peak estimate and tighten cap application

* Document estimator and trace-build-mode limits

* Estimate main_elements before trace-table build

* Warn when OS memory readout unavailable

* Include PAGE tables in main_elements estimate

* Replace silent wasm break with compile_error

* Guard wasm32 via target_arch not feature

* Log effective budget on auto disk-spill

* Guard crypto/crypto std on wasm32

* Simplify auto-disk-spill helpers

* Add streaming counter pass alongside prep

* Switch prove flow to streaming counter pass and analytical peak

* Remove PreparedTraceInputs and prepare_from_elf_and_logs

* Build peak_bytes from per-table allocations

* Print predicted peak bytes for threshold calibration

* Make table_parallelism pub

* Rework peak_bytes for chunk parallelism

* Add peak_bytes calibration test

* Add auto-spill threshold calibration script

* Bump auto-spill threshold to 90%

* Derive auto-spill log percent from constants

* Make disk-spill opt-in via feature flag

* Log predicted peak via log::info

* Warn on Ram fallback in select_storage_mode

* Assert aux upper-bound in count tests

* Default to Disk on unknown memory state

* Trim adverbs and em-dashes from docs

* Use saturating arithmetic in peak_bytes

* Gate mmap deps behind disk-spill feature

* Pass disk-spill feature to stark CI test

* Restore disk-spill position in CLI features

* Compile spill methods only with disk-spill

* Restore disk-spill position in crypto features

* Gate DiskSpill variant on disk-spill feature

* Move unused-warning suppressions to bindings

* Remove redundant multi_prove doc line

* Hide disk-spill plumbing when feature is off

* Gate StorageMode behind disk-spill feature

* Drop redundant cfg on Disk variant

* Drop unused TableLengths element accessors

* Gate count_table_lengths on disk-spill

* Inline derive_ops back into build_traces

* Drop redundant peak_bytes formulas from doc

* Trim auto_storage module doc to one line

* Drop effective_budget log; tighten visibility

* Trim verbose doc and inline comments

* Drop disk-spill feature comment in prover

* Return Err on commit overflow in count_table_lengths

* Note container limits in available_ram_bytes

* Add max_ram_bytes to ProofOptions in test

* Remove dead extract_columns_into

* Use checked_mul in spill_nodes_to_disk

* Require Copy on BaseType for spill_to_disk

* Materialize spilled data when serializing

* Gate wasm32 compile_error on disk-spill

* Assert mmap-fitting alignment instead of layout tautology

* Assert page_size is power of two

* Drop stale percentage from max_ram_bytes doc

* Tighten count_table_lengths comment

* Warn on Ram pick that may exceed cgroup limit

* Reserve spill file blocks via posix_fallocate

* Share reserve_file_blocks between spill paths

* Harden posix_fallocate against overflow and EOPNOTSUPP

* Compute decode rows analytically in count_table_lengths

* Build calibration CLI with disk-spill feature

* Replace Copy bound with SpillSafe marker on spill paths

* Guard Table::set against spilled tables

* Document SIGBUS risk on non-Linux temp filesystems

* Spill fixed and page tables in build_traces

* Suppress unused_mut warning without disk-spill

* Skip commit op materialization in count_table_lengths

* Remove unused spill_all_main_to_disk

* Make spill_nodes_to_disk idempotency check explicit

* Trim SpillSafe doc to layout invariants

* Loosen merkle node alignment to mmap page size

* Drop unnecessary mmap flush before make_read_only

* Assert max_rows is positive in padded_chunked_rows

* Fail fast when posix_fallocate is unsupported

* Tighten reserve_file_blocks doc

* Stream spilled MerkleTree nodes during serialize

* Document MerkleTree disk-spill Serialize impl

* Drop #[inline] from MerkleTree node accessors

* Tighten MmapNodesSeq doc comment

* Rephrase node_get SAFETY comment

* Refresh stale comments in spill_nodes_to_disk and spill_safe

* Stream spilled Table data through serialize_seq

* Make Table::set spill guard a hard assert

* Drop redundant node_size from MmapNodeBacking

* Account for halt padding in peak_bytes

* Drop stale max_ram_bytes advice from default-Disk warning

* Replace dead unwrap_or in cgroup-near-OOM warning

* Document why available.unwrap() is safe

* Note advise_drop_cache reliability per platform

* Extract spill_slice_to_mmap helper into mmap_util

* Make padded_chunked_rows zero-row guard a hard assert

* Note tmpfs caveat for spill files

* Test count_table_lengths against built Traces

* Trim spill_slice_to_mmap doc comment

* Gate aux_merkle on aux_cols > 0

* Revert halt_rows estimate to actual height

* Raise auto-Disk warning threshold to 75%

* Make reserve_file_blocks private

* Reorder mmap_util to lead with public API

* Restore owned buffers wording in coset_lde_full_expand doc

* Update FieldElement repr soundness doc to reference SpillSafe

* Restore derived PartialEq for non-spill Table builds

* Tighten SpillSafe module doc comment

* Feature-gate ProofOptions::max_ram_bytes behind disk-spill

* Tighten wasm32 disk-spill compile_error comment

* Drop typically from DiskSpill error doc

* Drop redundant advise_drop_cache inline comment

* Merge multi_prove and multi_prove_with_mode into one entry point

* Restore Domain/LdeTwiddles dedup rationale in multi_prove

* Restore domain_cache drop rationale comment

* Move multi_prove_ram from trait method to test_utils

* Use multi_prove_ram via use statement at call sites

* Trim TableMmapBacking doc comment

* Make TableMmapBacking and mmap_backing private

* Tighten Table Clone impl doc comment

* Drop streaming-serialize wrapper doc comments

* Drop Table PartialEq doc comment

* Tighten get_row/get assert comment

* Restore T1 transpose rationale in extract_columns doc

* Drop unnecessary inline on Table::get

* Drop is_spilled accessor, inline mmap_backing checks

* Use intra-doc links in spill_to_disk doc comment

* Use prelude size_of and import spill_slice_to_mmap

* Add use statements in mmap_util to drop fully-qualified paths

* Drop redundant doc comments on disk_spill_tests

* Drop redundant Test convenience suffix on multi_prove_ram doc

* Drop multi_prove_ram doc comment

* Drop disk-spill dep comment in stark Cargo.toml

* Rephrase unique_page_count doc comment

* Import HashSet in trace_builder

* Trim padded_chunked_rows doc comment

* Drop redundant TableLengths field doc comments

* Drop unused_mut rationale comment, gate HashSet import

* Drop stale multi_prove_inner reference in trace_builder spill comment

* Rephrase count_table_lengths doc comment

* Trim over-specific comments in count_table_lengths

* Drop redundant ECALL Commit comment

* Use building instead of materializing in commit count comment

* Restore ECALL Commit block-marker comment

* Merge from_elf_and_logs variants into one entry point

* Trim drift test module doc comment

* Trim disk_spill_tests module doc comment

* Drop redundant doc comments on disk_spill tests

* Use all_instructions_64 ELF for chunking disk-spill tests

* Use fib_iterative_372k in disk-spill scale test

* Trim peak_bytes_calibration_tests module doc

* Replace RSS calibration with jemalloc heap-only check

* Trim peak_bytes_calibration_tests module doc

* Run all prover tests under disk-spill feature in CI

* Extract AirTracePair alias to silence type_complexity

* Trim comments in prove_with_options_and_inputs storage mode block

* Import count_table_lengths in lib.rs

* Import table_parallelism in lib.rs

* Drop confusing Phase 5 comment before from_elf_and_logs

* Inline multi_prove arguments

* Replace ProofOptions::max_ram_bytes with FORCE_DISK_SPILL env var

* Drop env_logger init from disk_spill 372k test

* Reword calibrate_threshold doc to not leak internals

* Move peak_bytes calibration to integration test

* Compile Rust ELFs in Disk-spill tests job

* Use checked_next_power_of_two for commit_count

* Replace fib_iterative_24M with 32M

* Use traces_from_elf_and_logs_ram in keccak tests

* Use full bitwise for direct-indexing and security tests

* Filter prover CI step to disk-spill tests

* Move output assert before drop(result)

* Merge disk-spill prove and serde roundtrip tests

* Drop fib_iterative_372k disk-spill test

* Set FORCE_DISK_SPILL in CI, drop RAII guard and --test-threads=1

* Add make test-disk-spill target

Author: gabrielbosio

.github/workflows/pr_main.yaml

@@ -105,16 +105,18 @@ jobs:
   test:
     name: Test
     if: always()
-    needs: [test-executor, test-prover]
+    needs: [test-executor, test-prover, test-disk-spill]
     runs-on: ubuntu-latest
     steps:
       - name: Check results
         run: |
           executor="${{ needs.test-executor.result }}"
           prover="${{ needs.test-prover.result }}"
+          disk_spill="${{ needs.test-disk-spill.result }}"
 
           echo "test-executor: $executor"
           echo "test-prover: $prover"
+          echo "test-disk-spill: $disk_spill"
 
           # Allow "success" or "skipped" (skipped on merge queue pushes)
           if [[ "$executor" != "success" && "$executor" != "skipped" ]]; then
@@ -123,6 +125,68 @@ jobs:
           if [[ "$prover" != "success" && "$prover" != "skipped" ]]; then
             exit 1
           fi
+          if [[ "$disk_spill" != "success" && "$disk_spill" != "skipped" ]]; then
+            exit 1
+          fi
+
+  test-disk-spill:
+    name: Disk-spill tests
+    runs-on: ubuntu-latest
+    if: github.event_name != 'push' || github.actor != 'github-merge-queue[bot]'
+    steps:
+      - name: Checkout sources
+        uses: actions/checkout@v4
+
+      - name: Setup Rust Environment
+        uses: ./.github/actions/setup-rust
+
+      - name: Cache cargo build artifacts
+        uses: Swatinem/rust-cache@v2
+        with:
+          shared-key: "lambda-vm-disk-spill"
+          cache-all-crates: "true"
+
+      - name: Cache compiled ASM ELF artifacts
+        id: cache-asm-elfs
+        uses: actions/cache@v4
+        with:
+          path: executor/program_artifacts/asm
+          key: asm-elf-artifacts-${{ hashFiles('executor/programs/asm/**') }}
+
+      - name: Install clang and lld
+        if: steps.cache-asm-elfs.outputs.cache-hit != 'true'
+        run: sudo apt-get update && sudo apt-get install -y clang lld
+
+      - name: Compile ASM programs to ELF
+        if: steps.cache-asm-elfs.outputs.cache-hit != 'true'
+        run: |
+          make compile-programs-asm
+
+      - name: Cache compiled Rust ELF artifacts and build cache
+        id: cache-rust-elfs
+        uses: actions/cache@v4
+        with:
+          path: |
+            executor/program_artifacts/rust
+            executor/shared_target
+          key: rust-elf-artifacts-${{ hashFiles('executor/programs/rust/**', 'executor/programs/riscv64im-lambda-vm-elf.json', 'syscalls/**', 'Makefile') }}
+          restore-keys: |
+            rust-elf-artifacts-
+
+      - name: Compile Rust programs to ELF
+        if: steps.cache-rust-elfs.outputs.cache-hit != 'true'
+        run: |
+          make compile-programs-rust
+
+      - name: Run stark disk-spill tests
+        run: |
+          cargo test --release -p stark --features disk-spill disk_spill
+
+      - name: Run prover disk-spill tests
+        env:
+          FORCE_DISK_SPILL: "1"
+        run: |
+          cargo test --release -p lambda-vm-prover --features disk-spill -- disk_spill count_table_lengths
 
   build-prover-tests:
     name: Build prover tests