Skip to content

Commit e9e9fbd

Browse files
committed
Switch bench_vs projection to measured cycles and add --cycles
CLI flag
1 parent 1288f27 commit e9e9fbd

4 files changed

Lines changed: 179 additions & 65 deletions

File tree

bench_vs/README.md

Lines changed: 46 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -32,34 +32,67 @@ Compares proving time for an identical u64 wrapping Fibonacci computation.
3232
# Approximate workload steps (converted with 5 steps/iteration)
3333
./bench_vs/run.sh --steps 1000000 2000000 4000000 8000000
3434

35+
# Project to a target cycle count
36+
./bench_vs/run.sh --target-cycles 500000000
37+
3538
# Run only one prover
3639
./bench_vs/run.sh --lambda-only
3740
./bench_vs/run.sh --sp1-only
3841
```
3942

40-
## What it measures
43+
## What is measured
4144

4245
Both provers execute the same program: iterative Fibonacci with `u64::wrapping_add`.
43-
Only **proving time** is compared (wall-clock, no recursion/compression on either side).
44-
45-
- **Lambda VM**: Generates RISC-V assembly at runtime, assembles to ELF, proves via the CLI.
46-
- **SP1 v6**: Compiles a Rust guest program to RISC-V, proves via `sp1-sdk` core mode.
4746

48-
The linear projection uses a common axis for both provers: target workload steps.
49-
When you pass `--steps`, that target is explicit. When you pass `-n`, the script
50-
approximates workload as `steps ~= 5 * n`. `SP1 cycles` are still reported, but
51-
only as telemetry and not as the regression axis.
47+
The timing window on both sides is **end-to-end single-shot proving, with no
48+
verification and no recursion/compression**. Concretely:
49+
50+
| Phase | Lambda VM timer | SP1 v6 timer |
51+
|--------------------------------------------|:---------------:|:------------:|
52+
| Read ELF + input from disk |||
53+
| Pre-pass execution to count cycles |||
54+
| `setup` / verifying-key derivation | N/A (none) ||
55+
| ELF parse + guest execution (inside prove) |||
56+
| Trace build |||
57+
| AIR construction |||
58+
| STARK prove (`core` mode) |||
59+
| Proof serialization / write |||
60+
| Verify |||
61+
62+
Both sides run one extra execution pass **outside** the timer to report dynamic
63+
instruction counts (SP1's `execute(...)` / Lambda's executor pre-pass). This
64+
costs wall-clock time in the CI job but does not inflate the measured proving
65+
time, and the cost is symmetric between the two provers.
66+
67+
Lambda VM uses the default proof options from `prover::prove_with_inputs`
68+
(`GoldilocksCubicProofOptions::with_blowup(2)`, 50 FRI queries). SP1 v6 uses
69+
the `core` proof mode exposed by `sp1-sdk::ProverClient::from_env()`.
70+
71+
## Projection axis
72+
73+
The linear projection uses **measured cycles** per prover — Lambda's executor
74+
log count and SP1's `report.total_instruction_count()`. For Fibonacci the two
75+
values agree to within ~1% (both compile to the same inner loop shape on
76+
RISC-V). When cycle data is missing, the script falls back to the approximate
77+
`target_workload_steps ~= 5 * n` label that was passed on the command line.
5278

5379
## Output
5480

5581
```
5682
=== Summary ===
5783
Program: Fibonacci (u64 wrapping)
5884
59-
Target steps Iterations Lambda VM SP1 v6 SP1 cycles Ratio
60-
------------ ---------- --------- ------ ---------- -----
61-
1000000 200000 ...s ...s 1004794 ...
62-
2000000 400000 ...s ...s 2004794 ...
85+
Target steps Iterations Lambda (s) Lambda cycles SP1 (s) SP1 cycles Ratio
86+
------------ ---------- ---------- ------------- ------- ---------- -----
87+
1000000 200000 ...s 1004794 ...s 1004794 ...
88+
2000000 400000 ...s 2004794 ...s 2004794 ...
6389
90+
Timing window covers single-shot end-to-end proving; SP1 includes setup; both exclude verification.
6491
Green ratio = Lambda VM faster, Red = SP1 faster
6592
```
93+
94+
With `--report-dir DIR` the script writes:
95+
- `results.tsv` — raw per-run data (`target_steps`, `iterations`, `lambda_time_s`, `lambda_axis_value`, `lambda_cycles`, `sp1_time_s`, `sp1_axis_value`, `sp1_cycles`, `ratio`).
96+
- `metrics.txt` — key=value pairs including `timing_window=setup_plus_end_to_end_prove_no_verify`.
97+
- `summary.md` — the same table plus linear projection to `TARGET_CYCLES` cycles.
98+
- `raw/` — stdout/stderr of every individual run.

0 commit comments

Comments
 (0)