Skip to content

Commit 0b0e6d3

Browse files
MauroToscanogabrielbosiojotabulaciosnicole-graus
authored
Bench vs other vms (#365)
* bench vs others * Standarize guest * Fix decimals * Benchmark vs sp1 * Add lambda program * Remove stray blank line from .gitignore * Apply suggestion from @gabrielbosio Co-authored-by: Gabriel Bosio <38794644+gabrielbosio@users.noreply.github.com> * Add instruments * Fix syscall number * Add runtime private inputs, nightly CI workflow, and 500M step projection to bench_vs * save work * Fix bench_vs 5x projection inflation and add --steps flag for nightly 1M/2M/4M/8M benchmarks * Switch bench_vs projection to measured cycles and add --cycles CLI flag * Fix grep pipefail in bench_vs/run.sh by switching to sed --------- Co-authored-by: Gabriel Bosio <38794644+gabrielbosio@users.noreply.github.com> Co-authored-by: jotabulacios <jbulacios@fi.uba.ar> Co-authored-by: Nicole Graus <nicole.graus@lambdaclass.com>
1 parent 64bad0e commit 0b0e6d3

17 files changed

Lines changed: 7330 additions & 10 deletions

File tree

‎.github/workflows/bench-vs-nightly.yml‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
name: Bench Vs Nightly
2+
3+
on:
4+
schedule:
5+
# 03:00 America/Argentina/Buenos_Aires = 06:00 UTC
6+
- cron: "0 6 * * *"
7+
workflow_dispatch:
8+
9+
permissions:
10+
contents: read
11+
12+
concurrency:
13+
group: bench-vs-nightly-${{ github.ref }}
14+
cancel-in-progress: true
15+
16+
jobs:
17+
bench-vs:
18+
runs-on: [self-hosted, bench]
19+
timeout-minutes: 720
20+
steps:
21+
- name: Checkout
22+
uses: actions/checkout@v4
23+
24+
- name: Setup Rust Environment
25+
uses: ./.github/actions/setup-rust
26+
27+
- name: Add cargo to PATH
28+
run: echo "$HOME/.cargo/bin" >> "$GITHUB_PATH"
29+
30+
- name: Add SP1 to PATH
31+
run: echo "$HOME/.sp1/bin" >> "$GITHUB_PATH"
32+
33+
- name: Install SP1 toolchain
34+
run: |
35+
export PATH="$HOME/.cargo/bin:$HOME/.sp1/bin:$PATH"
36+
if ! cargo prove --version >/dev/null 2>&1; then
37+
curl -L https://sp1up.succinct.xyz | bash
38+
export PATH="$HOME/.sp1/bin:$PATH"
39+
sp1up
40+
fi
41+
cargo prove --version
42+
43+
- name: Run nightly benchmark
44+
run: |
45+
bash ./bench_vs/run.sh \
46+
--steps 1000000 2000000 4000000 8000000 \
47+
--report-dir bench_vs_artifacts \
48+
--no-color
49+
50+
- name: Upload nightly benchmark artifact
51+
uses: actions/upload-artifact@v4
52+
with:
53+
name: bench-vs-nightly-${{ github.run_number }}-${{ github.sha }}
54+
path: bench_vs_artifacts
55+
retention-days: 90
56+
57+
- name: Publish summary
58+
run: cat bench_vs_artifacts/summary.md >> "$GITHUB_STEP_SUMMARY"

‎bench_vs/README.md‎

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
# Lambda VM vs SP1 v6 Benchmark
2+
3+
Compares proving time for an identical u64 wrapping Fibonacci computation.
4+
5+
## Prerequisites
6+
7+
1. **Lambda VM CLI** (built from this repo):
8+
```bash
9+
cargo build --release -p cli
10+
```
11+
12+
2. **SP1 toolchain** (Succinct's prover):
13+
```bash
14+
curl -L https://sp1up.succinct.xyz | bash
15+
sp1up
16+
```
17+
18+
3. **Rust nightly** (for cross-compiling Lambda VM guest):
19+
```bash
20+
rustup toolchain install nightly
21+
```
22+
23+
## Usage
24+
25+
```bash
26+
# Default series: 1k, 10k, 100k, 300k iterations
27+
./bench_vs/run.sh
28+
29+
# Custom series
30+
./bench_vs/run.sh -n 1000 50000
31+
32+
# Approximate workload steps (converted with 5 steps/iteration)
33+
./bench_vs/run.sh --steps 1000000 2000000 4000000 8000000
34+
35+
# Project to a target cycle count
36+
./bench_vs/run.sh --target-cycles 500000000
37+
38+
# Run only one prover
39+
./bench_vs/run.sh --lambda-only
40+
./bench_vs/run.sh --sp1-only
41+
```
42+
43+
## What is measured
44+
45+
Both provers execute the same program: iterative Fibonacci with `u64::wrapping_add`.
46+
47+
The timing window on both sides is **end-to-end single-shot proving, with no
48+
verification and no recursion/compression**. Concretely:
49+
50+
| Phase | Lambda VM timer | SP1 v6 timer |
51+
|--------------------------------------------|:---------------:|:------------:|
52+
| Read ELF + input from disk | ❌ | ❌ |
53+
| Pre-pass execution to count cycles | ❌ | ❌ |
54+
| `setup` / verifying-key derivation | N/A (none) | ✅ |
55+
| ELF parse + guest execution (inside prove) | ✅ | ✅ |
56+
| Trace build | ✅ | ✅ |
57+
| AIR construction | ✅ | ✅ |
58+
| STARK prove (`core` mode) | ✅ | ✅ |
59+
| Proof serialization / write | ❌ | ❌ |
60+
| Verify | ❌ | ❌ |
61+
62+
Both sides run one extra execution pass **outside** the timer to report dynamic
63+
instruction counts (SP1's `execute(...)` / Lambda's executor pre-pass). This
64+
costs wall-clock time in the CI job but does not inflate the measured proving
65+
time, and the cost is symmetric between the two provers.
66+
67+
Lambda VM uses the default proof options from `prover::prove_with_inputs`
68+
(`GoldilocksCubicProofOptions::with_blowup(2)`, 50 FRI queries). SP1 v6 uses
69+
the `core` proof mode exposed by `sp1-sdk::ProverClient::from_env()`.
70+
71+
## Projection axis
72+
73+
The linear projection uses **measured cycles** per prover — Lambda's executor
74+
log count and SP1's `report.total_instruction_count()`. For Fibonacci the two
75+
values agree to within ~1% (both compile to the same inner loop shape on
76+
RISC-V). When cycle data is missing, the script falls back to the approximate
77+
`target_workload_steps ~= 5 * n` label that was passed on the command line.
78+
79+
## Output
80+
81+
```
82+
=== Summary ===
83+
Program: Fibonacci (u64 wrapping)
84+
85+
Target steps Iterations Lambda (s) Lambda cycles SP1 (s) SP1 cycles Ratio
86+
------------ ---------- ---------- ------------- ------- ---------- -----
87+
1000000 200000 ...s 1004794 ...s 1004794 ...
88+
2000000 400000 ...s 2004794 ...s 2004794 ...
89+
90+
Timing window covers single-shot end-to-end proving; SP1 includes setup; both exclude verification.
91+
Green ratio = Lambda VM faster, Red = SP1 faster
92+
```
93+
94+
With `--report-dir DIR` the script writes:
95+
- `results.tsv` — raw per-run data (`target_steps`, `iterations`, `lambda_time_s`, `lambda_axis_value`, `lambda_cycles`, `sp1_time_s`, `sp1_axis_value`, `sp1_cycles`, `ratio`).
96+
- `metrics.txt` — key=value pairs including `timing_window=setup_plus_end_to_end_prove_no_verify`.
97+
- `summary.md` — the same table plus linear projection to `TARGET_CYCLES` cycles.
98+
- `raw/` — stdout/stderr of every individual run.

‎bench_vs/lambda/fibonacci/.cargo/config.toml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
[target.riscv64im-lambda-vm-elf]
2+
rustflags = [
3+
"-C", "link-arg=-e",
4+
"-C", "link-arg=main",
5+
"-C", "passes=lower-atomic"
6+
]

‎bench_vs/lambda/fibonacci/Cargo.lock‎

Lines changed: 7 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎bench_vs/lambda/fibonacci/Cargo.toml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
[workspace]
2+
3+
[package]
4+
name = "fibonacci-bench"
5+
version = "0.1.0"
6+
edition = "2024"
7+
8+
[dependencies]

‎bench_vs/lambda/fibonacci/src/main.rs‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
#![no_std]
2+
#![no_main]
3+
4+
use core::arch::asm;
5+
use core::panic::PanicInfo;
6+
7+
const SYSCALL_GET_PRIVATE_INPUTS: u64 = 4;
8+
const SYSCALL_COMMIT: u64 = 64;
9+
const SYSCALL_HALT: u64 = 93;
10+
11+
#[panic_handler]
12+
fn panic(_info: &PanicInfo) -> ! {
13+
loop {}
14+
}
15+
16+
fn read_n() -> u64 {
17+
let mut input = [0u8; 12];
18+
19+
unsafe {
20+
asm!(
21+
"ecall",
22+
in("a0") input.as_mut_ptr(),
23+
in("a7") SYSCALL_GET_PRIVATE_INPUTS,
24+
);
25+
}
26+
27+
let mut n_bytes = [0u8; 8];
28+
n_bytes.copy_from_slice(&input[4..12]);
29+
u64::from_le_bytes(n_bytes)
30+
}
31+
32+
fn commit(bytes: &[u8]) {
33+
unsafe {
34+
asm!(
35+
"ecall",
36+
in("a0") 1u64,
37+
in("a1") bytes.as_ptr(),
38+
in("a2") bytes.len(),
39+
in("a7") SYSCALL_COMMIT,
40+
);
41+
}
42+
}
43+
44+
fn halt() -> ! {
45+
unsafe {
46+
asm!(
47+
"ecall",
48+
in("a0") 0u64,
49+
in("a7") SYSCALL_HALT,
50+
options(noreturn),
51+
);
52+
}
53+
}
54+
55+
#[unsafe(no_mangle)]
56+
pub fn main() -> ! {
57+
let n = read_n();
58+
let mut a: u64 = 0;
59+
let mut b: u64 = 1;
60+
for _ in 0..n {
61+
let c = a.wrapping_add(b);
62+
a = b;
63+
b = c;
64+
}
65+
66+
commit(&b.to_le_bytes());
67+
halt()
68+
}

0 commit comments

Comments
 (0)