Skip to content

Commit 0dae89a

Browse files
olwangclaude
andcommitted
docs(vm-jit): refresh stale baseline numbers + read-helper ABI scope
Second review pass (medium/low): - §3.2: clarify that new read helpers reuse the §7.2 fallback *proof* but NOT the ABI — current helpers (field_int/list_len/list_get_int) are Int-only and return a single i64, so non-Int and heap-returning reads (String, nested List/Struct, map values) are a real ABI design task (typed result / handle for heap results, extended handle domain) with its own §7.1 amendment + tests. - Refresh stale regression offenders in plan §0.2/§3.0 and README findings #1/#2 from baseline-20260620.json: list_sort 1.31 / map_int 1.19 / json 1.48 / dynamic_closure 1.66 / string_text 1.80 no longer match; current worst are tier-0 native_read_heap 2.59 / nested_struct_field 1.36 and native task_group_spawn 1.58 / bytes_scan 1.42. Added "re-read JSON before quoting". - Correct the "native near-Rust" framing with nat/rust (not reg/rust): scalar ~1.4x but read_heap ~13.3x. README table now carries an explicit nat/rust column; plan numbers match the committed baseline. - Mark plan status in progress (Phase 0 done; Set/task_group fixed) instead of "draft / not started". Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 423cc7b commit 0dae89a

2 files changed

Lines changed: 54 additions & 31 deletions

File tree

benchmarks/vm-jit/README.md

Lines changed: 19 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -111,15 +111,18 @@ speedup, **<1 = faster than the plain VM**).
111111

112112
1. **The native JIT codegen is excellent — it just almost never runs.** On the
113113
native-*eligible* kernels the native tier is **15–50× faster than the VM**
114-
and within ~1.4–2× of native Rust:
115-
116-
| kernel | reg_vm_ms | native_ms | rust_ms | nat/reg |
117-
|---|--:|--:|--:|--:|
118-
| `native_scalar_loop` | 153.5 | 3.20 | 2.41 | **0.02** |
119-
| `native_read_heap` | 115.5 | 6.77 | 0.61 | **0.06** |
120-
| `native_call_chain` | 196.3 | 5.33 | 5.56 | **0.03** |
121-
| `int_divmod_loop` | 179.3 | 4.20 | 1.96 | **0.02** |
122-
| `bool_logic_loop` | 347.2 | 6.70 | 3.21 | **0.02** |
114+
(`nat/reg` 0.02–0.07). Its distance from native Rust (`nat/rust`), though, is
115+
**opcode-dependent**: pure-scalar/arith native is near-Rust (~0.9–2.1×), but
116+
**read-heap native is ~13×** because each heap read crosses the host-helper
117+
call boundary (§7.1 of the Exec-Spec):
118+
119+
| kernel | reg_vm_ms | native_ms | rust_ms | nat/reg | nat/rust |
120+
|---|--:|--:|--:|--:|--:|
121+
| `native_scalar_loop` | 153.6 | 3.12 | 2.29 | **0.02** | 1.4× |
122+
| `native_read_heap` | 110.5 | 7.58 | 0.57 | **0.07** | **13.3×** |
123+
| `native_call_chain` | 217.8 | 5.31 | 5.65 | **0.02** | 0.9× |
124+
| `int_divmod_loop` | 168.0 | 4.13 | 1.95 | **0.02** | 2.1× |
125+
| `bool_logic_loop` | 321.7 | 6.26 | 2.98 | **0.02** | 2.1× |
123126

124127
So the problem is **not** codegen quality — it's **eligibility/coverage**.
125128
The moment a function does anything outside the pure-scalar/read-heap subset
@@ -129,11 +132,13 @@ speedup, **<1 = faster than the plain VM**).
129132

130133
2. **On the real (ineligible) kernels both JIT tiers do ~nothing**`jit/reg`
131134
and `nat/reg` hover at ~1.00, and frequently **>1.0 (the tier makes it
132-
slower)**: tier-0 regresses `string_text_processing` **1.80×** (the worst in
133-
the suite), `selfhost_mailbox` 1.21; native regresses `float` 1.23,
134-
`closure_alloc` 1.16, `list_closure` 1.09 — the tier translates part of the
135-
function, bails, and eats the overhead. (The exact offenders shift run to run;
136-
the pattern — tiers *regress* ineligible code — is stable.) A cheap early win:
135+
slower)**. In `baseline-20260620.json`: tier-0 (`jit/reg`) regresses
136+
`native_read_heap` **2.59×** and `nested_struct_field` 1.36; native (`nat/reg`)
137+
regresses `task_group_spawn` 1.58, `bytes_scan` 1.42, and
138+
`closure_alloc`/`option_result_chain`/`pipeline_chain` ~1.20 — the tier
139+
translates part of the function, bails, and eats the overhead. (The exact
140+
offenders shift run to run; re-read the JSON before quoting — the pattern,
141+
tiers *regress* ineligible code, is what's stable.) A cheap early win:
137142
**don't attempt the JIT on functions that will predictably bail.**
138143

139144
3. **`set_insert_contains` was pathological (reg/rust ≈ 1680×) — now FIXED

docs/planning/vm-jit-perf-plan.md

Lines changed: 35 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,10 @@ in the field (LuaJIT, V8 Sparkplug/Ignition, Wasmtime Winch/Wizard,
55
JavaScriptCore) — **without breaking the §2 parity invariant or §6 sandbox
66
guarantees** in [`docs/spec/RSScript_Execution_Spec_v0.1.md`](../spec/RSScript_Execution_Spec_v0.1.md).
77

8-
Status: **draft / not started.** Owner: TBD. Created 2026-06-19.
8+
Status: **in progress.** Phase 0 done (0.1 kernel suite + 0.2 baseline committed;
9+
0.3 profile and 0.4 metric remain), and the two runtime pathologies the baseline
10+
surfaced — hash-`Set` O(n²) and quadratic `task_group` — are **fixed**. Phases
11+
1–4 not started. Owner: TBD. Created 2026-06-19; updated 2026-06-19.
912

1013
---
1114

@@ -89,14 +92,18 @@ feeling, not a number. Establish where time actually goes before touching code.
8992
VM** (`nat/reg` 0.02–0.06). Native *codegen* is not the problem —
9093
**eligibility/coverage is.** Any heap write / string / collection /
9194
closure / suspend drops the whole function back to the interpreter.
92-
Caveat on "near-Rust": only **pure-scalar** native is near-Rust
93-
(`native_scalar_loop` 3.20 ms vs Rust 2.41, ~1.3×). **Read-heap** native
94-
is *not*`native_read_heap` is 6.77 ms vs Rust 0.61, **~11×** — because
95-
the host-helper call boundary (§7.1) is not free. Phase 3 must not assume
96-
every newly-eligible family lands at scalar-native speed.
95+
Caveat on "near-Rust" (`native_ms` vs `rust_ms`): only **pure-scalar**
96+
native is near-Rust (`native_scalar_loop` 3.12 ms vs Rust 2.29, **~1.**).
97+
**Read-heap** native is *not*`native_read_heap` is 7.58 ms vs Rust 0.57,
98+
**~13×** — because the host-helper call boundary (§7.1) is not free. Phase 3
99+
must not assume every newly-eligible family lands at scalar-native speed.
97100
- On the real (ineligible) kernels both tiers do ~nothing and often
98-
**regress** (native `list_sort` 1.31, `map_int` 1.19; tier-0 `json`
99-
1.48, `dynamic_closure` 1.66) — translate, bail, eat overhead.
101+
**regress** (tier-up costs more than it saves). Current `baseline-20260620.json`
102+
offenders: tier-0 (`jit/reg`) `native_read_heap` 2.59, `nested_struct_field`
103+
1.36; native (`nat/reg`) `task_group_spawn` 1.58, `bytes_scan` 1.42,
104+
`closure_alloc`/`option_result_chain`/`pipeline_chain` ~1.20 — translate,
105+
bail, eat overhead. (These shift run to run; re-read the JSON before
106+
quoting — the *pattern* is stable, the specific numbers are not.)
100107
- `set_insert_contains` was pathological at **1680× native Rust** — the
101108
`sorted_set_ops` kernel pinned it to the **hash-`Set`** (a `Vec` with a
102109
linear scan per op, O(n²)). **Fixed:** `Set` is now backed by the FNV
@@ -227,27 +234,38 @@ baseline beats it on the matrix *and* passes the full differential suite.
227234
**Re-weighted up by the Phase-0 baseline:** native is **15–50× faster than the
228235
VM** where it runs (`nat/reg` 0.02–0.06), so every opcode family this phase makes
229236
eligible converts a ~100–300× slowdown into something far closer to Rust. How
230-
much closer is opcode-dependent: pure-scalar native is ~1.3× Rust, but read-heap
231-
native is ~11× Rust (the §7.1 helper-call boundary; see §0.2) — so estimate each
237+
much closer is opcode-dependent: pure-scalar native is ~1.4× Rust, but read-heap
238+
native is ~13× Rust (the §7.1 helper-call boundary; see §0.2) — so estimate each
232239
family's ceiling from its helper traffic, not from the scalar number. Today
233240
native covers 20 scalar/read-heap opcodes with no OSR. Extend coverage and reduce
234241
the cliff.
235242

236243
- [ ] **3.0 Predict-and-skip bail (cheap, do first).** Before translating, cheaply
237244
predict whether a function will bail (contains an op family known to be
238-
unsupported) and skip native for it — the baseline shows native currently
239-
*regresses* `list_sort`/`map_int`/`closure_alloc` by translating then
240-
bailing. This is a guard, not new coverage, and stops the bleeding.
245+
unsupported) and skip native for it — the current `baseline-20260620.json`
246+
shows native *regressing* `bytes_scan` (1.42), `closure_alloc`/
247+
`option_result_chain`/`pipeline_chain` (~1.20) by translating then bailing.
248+
This is a guard, not new coverage, and stops the bleeding. (Refresh the
249+
example list from the JSON when implementing; the offenders drift per run.)
241250

242251
- [ ] **3.1 Coverage audit.** From the Phase-0 profile, list the highest-traffic
243252
opcodes that currently force a bail to the interpreter (likely list/map
244253
reads, string ops, struct writes). Rank by benchmark impact.
245254
- [ ] **3.2 Add host-helper ABIs — reads first.** Extend the **read-only** §7.1
246255
contract to the top read-side bail-causers (`list_get` for non-int elems,
247-
map lookup, more field reads). These inherit the existing
248-
immediate-bail-on-unsatisfiable-read contract and the §7.2 fallback proof
249-
unchanged. One opcode family per PR, each with a ledger entry (Exec-Spec
250-
Appendix C) and a force-deopt differential test.
256+
map lookup, more field reads). The §7.2 *fallback proof* carries over
257+
unchanged (these are still side-effect-free reads), **but the ABI does not**:
258+
today's helpers (`field_int`, `list_len`, `list_get_int`) are **Int-only and
259+
return a single `i64`**. A non-`Int` element or a heap-returning read (a
260+
`String`, a nested `List`/`Struct`, a map *value*) cannot be expressed as one
261+
`i64`, so this step is a real **ABI design task**, not a contract reuse:
262+
decide the typed-result representation (tagged return + out-param, or a
263+
result-into-the-handle-table scheme that hands native a *new* call-scoped
264+
handle for a heap result), extend rule 2's handle domain to cover
265+
helper-produced handles, and keep every unsatisfiable/wrong-type read on the
266+
immediate-bail path. One opcode family per PR, each with a ledger entry
267+
(Exec-Spec Appendix C) **plus a §7.1 ABI amendment** and a force-deopt
268+
differential test covering the new return shapes.
251269
- [ ] **3.2w Heap *writes* are a separate, harder track — do NOT fold into 3.2.**
252270
§7.1 is **reads only** and §7.2's fallback proof depends on the compiled
253271
subset being side-effect-free: a bail re-runs the function from the top, so

0 commit comments

Comments
 (0)