Skip to content

Commit 3f3c7ac

Browse files
committed
Fix bench_vs 5x projection inflation and add
--steps flag for nightly 1M/2M/4M/8M benchmarks
1 parent 0315a80 commit 3f3c7ac

3 files changed

Lines changed: 175 additions & 57 deletions

File tree

.github/workflows/bench-vs-nightly.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -43,14 +43,14 @@ jobs:
4343
- name: Run nightly benchmark
4444
run: |
4545
bash ./bench_vs/run.sh \
46-
-n 500000 1000000 1500000 2000000 \
46+
--steps 1000000 2000000 4000000 8000000 \
4747
--report-dir bench_vs_artifacts \
4848
--no-color
4949
5050
- name: Upload nightly benchmark artifact
5151
uses: actions/upload-artifact@v4
5252
with:
53-
name: bench-vs-nightly-${{ github.sha }}
53+
name: bench-vs-nightly-${{ github.run_number }}-${{ github.sha }}
5454
path: bench_vs_artifacts
5555
retention-days: 90
5656

bench_vs/README.md

Lines changed: 12 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,9 @@ Compares proving time for an identical u64 wrapping Fibonacci computation.
2929
# Custom series
3030
./bench_vs/run.sh -n 1000 50000
3131

32+
# Approximate workload steps (converted with 5 steps/iteration)
33+
./bench_vs/run.sh --steps 1000000 2000000 4000000 8000000
34+
3235
# Run only one prover
3336
./bench_vs/run.sh --lambda-only
3437
./bench_vs/run.sh --sp1-only
@@ -42,18 +45,21 @@ Only **proving time** is compared (wall-clock, no recursion/compression on eithe
4245
- **Lambda VM**: Generates RISC-V assembly at runtime, assembles to ELF, proves via the CLI.
4346
- **SP1 v6**: Compiles a Rust guest program to RISC-V, proves via `sp1-sdk` core mode.
4447

48+
The linear projection uses a common axis for both provers: target workload steps.
49+
When you pass `--steps`, that target is explicit. When you pass `-n`, the script
50+
approximates workload as `steps ~= 5 * n`. `SP1 cycles` are still reported, but
51+
only as telemetry and not as the regression axis.
52+
4553
## Output
4654

4755
```
4856
=== Summary ===
4957
Program: Fibonacci (u64 wrapping)
5058
51-
n Lambda VM SP1 v6 Ratio
52-
--- --------- ------ -----
53-
1000 13.3s 12.4s 0.9x
54-
10000 22.4s 12.9s 0.6x
55-
100000 116.4s 14.7s 0.1x
56-
300000 ... ... ...
59+
Target steps Iterations Lambda VM SP1 v6 SP1 cycles Ratio
60+
------------ ---------- --------- ------ ---------- -----
61+
1000000 200000 ...s ...s 1004794 ...
62+
2000000 400000 ...s ...s 2004794 ...
5763
5864
Green ratio = Lambda VM faster, Red = SP1 faster
5965
```

0 commit comments

Comments
 (0)