VM dynamic native loading + sqlx facade and sqlite benchmark by Haofei · Pull Request #1 · Haofei/rsscript

Haofei · 2026-06-07T06:00:31Z

Summary

Lets the register VM run native-backed packages by generating, building, and dlopen-ing a cdylib shim derived from each package's interface + bindings.rssbind.toml — no hard-coded host bindings, nothing per-package by hand.
Makes sqlx usable from plain RSS (facade split) and adds a VM vs compiled vs native sqlx+SQLite benchmark.

What's included

native-abi/ — shared crate: NativeValue, NativeInterpreterFn, the #[repr(C)] registry ABI, and a host-gated dlopen loader.
src/native_plugin/ — shim generator (recursive type coverage: Unit/String/Int/Float/Bool/Bytes/Path/List<T>/Option<T> and Result<T,String>) + loader (builds the shim with cargo, loads it, collects native deps transitively).
VM — resolves qualified user functions (Namespace.method bodies); mut native params now propagate via a List[result, mutated…] envelope written back to caller registers (generic — e.g. Rayon.sort_int sorts in place).
rss bench — --mode vm-internal/eval/release-internal accept package directories.
sqlx — split into rss-sqlx-ffi (native bindings + pooled sqlx crate) and the plain-RSS rss-sqlx facade an app imports without features: native.
benchmarks/sqlx-sqlite/ — workload run three ways with a runner script + README (documents that SQLite work dominates, so the modes land within noise).
Smoke tests — sqlx, sqlite (Path), rayon sum (List<Int>) and sort (mut write-back).

Verification

cargo build clean; VM + interpreter suites pass (87/0 after updating the now-obsolete Random.int "unsupported" test — it's a supported intrinsic now).
rss bench --mode vm-internal runs sqlx / sqlite / rayon packages via the dynamically loaded shim; Rayon.sort_int mutation verified to propagate.

🤖 Generated with Claude Code

Lets the register VM run native-backed packages by generating, building, and dlopen-ing a cdylib shim from each package's interface + bindings — no hard-coded host bindings. Also makes sqlx usable from plain RSS and adds a VM/compiled/native benchmark. - native-abi: shared crate with NativeValue + NativeInterpreterFn + the cdylib registry ABI and a host-gated dlopen loader. - native_plugin: shim generator (recursive type coverage: Unit/String/Int/ Float/Bool/Bytes/Path/List<T>/Option<T>/Result<T,String>) + loader that builds and loads the shim, with transitive native-dep collection. - VM: support qualified user functions; `mut` native params now propagate via a result+mutated envelope written back to caller registers (generic, e.g. Rayon.sort_int sorts in place). - rss bench: --mode vm-internal/eval/release-internal accept package dirs. - sqlx: split into rss-sqlx-ffi (native bindings + pooled sqlx crate) and a plain-RSS rss-sqlx facade an app imports without `features: native`. - benchmarks/sqlx-sqlite: query workload run three ways (native/compiled/VM) with a runner + README. - smoke tests: sqlx, sqlite (Path), rayon sum (List<Int>) and sort (mut). - Update Random.int VM test: it is now a supported intrinsic. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…age) Unifies the existing scattered seams (fixture corpus, VM<->compiled parity helpers, coverage tracking) into one spine: "one corpus, all stages, parity by default" for a two-backend language. - tests/common: shared engine -- run_vm_source, cached run_compiled_source, error_codes -- so any program can be checked across the VM and the compiled backend and asserted equal. - tests/corpus.rs (libtest-mimic, harness=false): one named trial per fixture. `.rss` + `.toml` sidecar declares execution (run on VM, and on compiled with parity asserted) or diagnostics (exact error-code set). Starter corpus covers arithmetic/string/list/option/result/match + a diagnostics fixture. - coverage::required_tags gate: fails if a required capability has no fixture, making "test all aspects" an enforced invariant. - tests/properties.rs (proptest): generative differential checking -- random integer arithmetic programs vs a Rust oracle (shared checked semantics). - Unit tests for the previously-untested native bridge: src/native_plugin shim_gen (type mapping, generated code, mut envelope) and native-abi. - CI: run native-abi crate tests via the test-runner manifest; bump the rsscript test timeout for the corpus's compiled builds. (The corpus + property suites run under the existing `cargo test -p rsscript`.) - Fix a pre-existing clippy::ptr_arg in rss_http_server_native (&Vec -> &[_]) so the workspace clippy gate isn't blocked by it. - tests/corpus/README.md documents how to add fixtures. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

There is only one execution engine (the register VM); "interpreter" implied a second one. The file tests VM `eval`/`eval_package` (source -> run) plus async and native-host-binding parity, so name it for what it does -- the eval-level companion to vm.rs's compile-level tests. - git mv interpreter.rs -> vm_eval.rs, with a module header clarifying its scope. - Rename the internal helper assert_interpreter_matches_backend* -> assert_vm_eval_matches_backend* (file-local, no external refs). - Update the path read by the coverage gate (execution_coverage.rs) and the corpus README reference. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The eval tests were one 4958-line file; the suite ran test binaries serially (cargo test), leaving most of a 24-core box idle. - Split vm_eval.rs: shared parity helpers + network test servers moved into tests/common; Part A (eval/async/native, 17 tests + the // parity: coverage manifest) stays in vm_eval.rs; the bulk parity suite (70 tests) moves to vm_eval_parity.rs. No tests lost (17 + 70 = 87) and the coverage gate still reads vm_eval.rs. - Adopt cargo-nextest: .config/nextest.toml + CI installs it and the test-runner manifest runs `cargo nextest run --workspace`, scheduling every test case across all cores in one pool. Measured ~97s vs ~236s for `cargo test` (2.4x). Note: splitting only helps wall-clock under nextest; plain cargo test runs binaries serially. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

assert_vm_eval_matches_backend_internal re-ran `cargo run` on a freshly generated crate every time (~10s each x70 tests). Cache its (stdout, stderr) keyed by source + args, like vm.rs already does, so warm reruns skip the generated-crate rebuild. Measured (24 cores, nextest): warm full rsscript suite ~36s vs ~100s cold and ~236s under `cargo test`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…eview-map counts Several tests were red on main under the current toolchain. Fixed the safe set: - Regenerate stale core-package-index.json and the VS Code grammar snapshot. - core/collections/deque.rssi was missing `struct Deque<T>`, so the type didn't resolve (RS0024); add it like set/sorted_map/persistent_map. - Allow `fresh` on `class` constructors (a freshly created class shell); only a `resource` can't be fresh. Unblocks the app-review-benchmark pass fixture. - Update review-map fixtures' stale function/region counts (the fixtures gained functions; unknown-region invariants are unchanged and still asserted). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…res) fresh-return-class-type.rss and fresh-result-class-type.rss are fail fixtures that assert `fresh` on a `class` is rejected (RS0603). Allowing it broke them, so revert: the intended behavior is that only struct/sum can be `fresh`. The app-review-benchmark pass fixture's use of fresh-on-class is a separate pre-existing fixture bug (needs a struct-vs-class decision). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

A named Copy arg in a receiver call to a runtime intrinsic / native fn (e.g. `names.insert(key: read 1, ...)` -> map_insert(map, key: &K, ...)) was lowered by value (`1`), producing Rust that doesn't compile. Honor the parameter effect (`&`/`&mut`) for Copy named args when the callee is a native target (runtime intrinsic via runtime_intrinsic_target, or a native binding); non-Copy named args and protocol/user calls keep their existing lowering. Fixes rust_lowering_allows_receiver_calls_for_{core_namespace,generic_map}. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…group by-value arg - RS0209 message was standardized to "match pattern X cannot match scrutinee type Y"; update the one test still asserting the old "variant ... scalar type" wording. - A user-declared `RetainedImageStore.store` resolves to UserFunction, not BuiltinFunction; fix the hir resolution test's stale expectation. - async-let lowers `fetch_user(id)` (Copy arg by value, matching the lowered fn signature), not `fetch_user(&id)`; update the task_group lowering assertion. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…ixtures - Parser: `read || { ... }` (an effect keyword immediately followed by a closure) was parsed as `read OR {block}` because binary parsing ran before the effect prefix; recognize the effect-annotated closure first. Unblocks the inline managed-closure fixtures (RS0801, review-map) and pass/fail fixture suites. - pass fixture app-review-benchmark: a `class` constructor can't be `fresh` (consistent with the fresh-*-class fail fixtures); drop `fresh`. - fail fixtures (builtin/fresh-branch/loop/return-retained-local): declared the `Image.load`/`RetainedImageStore.store` they reference, so the calls resolve and the intended RS0501/RS0601 fire instead of RS0206 unknown-callee. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Dropping fresh from the 2 class constructors reclassifies them Foldable, so review_required 35->33, foldable 9->11. Update the assertions. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Capability `category` strings were free-form, so the review surface couldn't reason about a stable set of powers. Add a canonical taxonomy: - src/capability.rs: CAPABILITY_CATEGORIES (`domain.action` names: fs/network/ database/env/process/compute/crypto/secret/identity/time/random/runtime) each with a coarse domain and default CapabilityRisk; predicates is_known_capability_category / capability_risk (unknown -> High). - review: a capability binding whose category isn't canonical is surfaced for review (same channel as an unbound native facade), keeping the taxonomy meaningful and reproducible. - tests: unit tests for the taxonomy + a gate asserting every shipped package's declared categories are canonical (keeps packages and taxonomy in sync). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

PackageReviewCapability now carries a `risk` (from the canonical taxonomy; unknown category -> high), so `pkg review` ranks powers by risk instead of treating every capability the same. Additive field; existing field-access tests unaffected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

`pkg review` now prints a "capabilities (by risk)" section listing the distinct powers a package requires, high-risk first, with provider and any "unrecognized category" note — the reviewer-facing view of the capability taxonomy. Example (rss-sqlx-ffi): capabilities (by risk): [high] database.write via SqlxFfi.execute (provider sqlx) [medium] database.read via SqlxFfi.query_strings (provider sqlx) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

`pkg diff` now reports the distinct capabilities added/removed between two package versions, high-risk first, and flags new high-risk powers as review reasons. This is the core "AI-generated code review" signal: surface capability escalation (and de-classification) instead of making a reviewer read the code. capability changes: + [high] unknown via SqlxFfi.query_strings # dropped binding -> unclassified - [medium] database.read via SqlxFfi.query_strings Additive PackageDiff.capability_changes field (JSON + human); unit-tested. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The review hash covered identity/risk/reasons/summary but not the individual capabilities, so a provider swap or re-classification that kept the same summary counts was invisible to the lock. Hash each capability's binding symbol, category, provider, service, action, and resource so any capability change (notably a supply-chain provider swap) changes review_hash and the lock catches it. Reproducible review now means capability-aware review. - test: a capability provider swap (same category/code) changes review_hash. - regenerate committed rsspkg.lock files (this also refreshed pre-existing stale interface/native hashes; no test verified the committed locks, so they had drifted). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Evidence built from invalid source (error diagnostics) could still be consumed as a passing gate input. Make that impossible to do silently: - CiGateOutput gains `valid_for_gating` (default true) + `gating_reason`. The gate sets it false and forces status=Fail when the bundle contains any error-severity diagnostic fact, regardless of capability reconciliation. - `reir collect --strict` refuses to emit a bundle that contains error diagnostics (fail closed at evidence-generation time). - tests: an error diagnostic makes the bundle invalid_for_gating + Fail. Together with the earlier `pkg metadata` fail-closed (3.1), review/REIR evidence from a half-broken AST no longer reads as authoritative. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

run.sh + README now also demonstrate, on the same before/after package: - reir report-pr reconciliation under examples/rss-policy.toml with SARIF output (the PR's new network.client is missing from the deployment grants -> gate exit 1, SARIF error); - rss pkg review --markdown (PR-facing render); - rss native audit (adapter risk facts, transitive = not_audited); - rss pkg metadata fail-closed on a parse error (no REIR written); - rss check --explain RS0015 --json (machine-readable diagnostic for agents). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…rification + matrix wiring Fix #1 — tier-0 is no longer slower than the interpreter: - Cache per-function JIT analysis `(supported, has_loop)` on `RegFunction` (a `Cell`, computed once) instead of a per-call HashMap lookup. - Only JIT functions that contain a back-edge (a loop), where the specializing executor's smaller dispatch pays off; straight-line functions called in hot loops stay on the interpreter. Result: function_call_hot_loop 1654->1605, match_option_loop 191->188 (were ~20% slower before). - `jit_force_all` mode keeps verification broad: the differential / parity tests JIT every supported function (not just loops) so the whole covered subset is checked; production (bench) uses the loop heuristic. Fix #2 — `benchmark/run-matrix.sh` now reports a `jit_ms` column and `jit/reg`. Also: the generative differential surfaced a real VM<->compiler gap — integer literals lower to untyped Rust (default i32) and can const-overflow at compile time. Documented in docs/jit-todo.md as a known bug to fix; the generator caps literal magnitudes so it tests VM<->JIT parity rather than re-finding it. Whole workspace 1176 passing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Telemetry (review #1): NativeState now records where native-tier attempts go — considered/translated/compiled/not_eligible/compile_failed/native calls/bails/arg_mismatch/tier_deferred plus compile-vs-run nanoseconds. `RSS_JIT_STATS=1` prints the summary after a native eval, so the next coverage win is measurable instead of guessed (try_native's many silent `None`s are now attributed). Benchmark matrix (review #2): run-matrix.sh builds with --features native-jit and adds native_ms + nat/reg columns (was jit-internal only). Also remove docs/jit-todo.md (superseded; design lives in code + the architecture spec). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…ument release fast path for package check #1 (List<T>.clone() etc.): `.clone()` on a builtin value type whose clone the checker resolves (List/Bytes/Buffer) typechecked via the Clone protocol but lowered to a dangling `List_clone`-style call -> rustc E0425. The lowerer now emits Rust's `.clone()` for these (they lower to Clone Rust types). Same frontend-accepts/ backend-fails class as the borrowed-match E0308 fix. Removes the need for manual copy helpers in source-shaped ports. Regression test in checker_lowering/stdlib. #2 (slow package check in debug): documented the release-binary fast path in README (Performance) and cross-referenced from BUGS.md RSS-11 — debug leaves the ~O(n^3) generic-substitution path unoptimized; release is usable. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…generic params, File.open effects - vm-jit: NativeModule::call now rejects an args slice whose length != the compiled function's n_params (the generated entry block reads exactly n_params words from args_ptr without bounding by n_args, so a short slice was an out-of-bounds read in a 'safe' API) — #1, the highest-severity finding. CompiledId now carries a per-module id so an id from another NativeModule is rejected instead of indexing the wrong table (#2). Tests: call_rejects_wrong_arg_count, call_rejects_id_from_another_module. - lowerer: builtin_generic_type_params mapped Result to [K,V] (Map's params); split so Result uses [T,E] — fixes silently-failing generic substitution for Result.map/map_error/and_then (#7). Unit-test guard added. - stdlib/fs/file.rssi: File.open/open_read/open_write now carry effects(native) so host file-resource creation is review-visible in the signature, consistent with the handle reads/writes (#6). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…acement, async-lowerability, review facts) check_await_placement_expr, expr_first_await (AST RS0411 path), and collect_body_facts_in_expr now recurse into match-arm guards and object/array literal children instead of treating them as leaves. 'let xs = [await load()]' is now correctly rejected (RS0411); review facts inside guards/object/array literals are now collected. First slice of the central-traversal fix. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…ntic passes Extends the central-traversal fix to check_try_error_types_expr, apply_expr_effects, check_expr_semantics_with_context, collect_expr_uses, collect_spawn_capture_idents, collect_closure_effect_accesses_expr, collect_await_operand_live_uses, and check_resource_pool_discards_expr: each now recurses into match-arm guards and object/array literal children (was: guards skipped, object/array treated as leaves). Also annotates the pre-existing supported intrinsic Bytes.to_string (parity gap). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…lectors Extends the central-traversal fix to the local-analysis collector passes: collect_expr_managed_closure_uses, collect_expr_resource_escapes, collect_retained_closure_captures_from_expr, collect_expr_take_handle_fields, collect_resource_escapes_in_expr, collect_hir_expr_effect_events, collect_closure_local_moved_uses_from_expr, and collect_expr_managed_closure_capture_names. Each now recurses into match-arm guards (where applicable) and object/array/map literal children instead of treating them as leaves, so effects, resource escapes, moved uses, retained captures, and take-handle fields inside collection/object literals are no longer silently dropped. Also regenerates schemas/core-package-index.json to include the existing Bytes.to_string interface fn (pre-existing snapshot drift). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Remove completed entries (now in git history) and keep only open/partial work: - Dropped: symbol identity (#1), class/static members (#2, type-associated consts + static methods), clone consistency (#6), Option ergonomics (#7), plus the already-checked namespace isolation / pattern matching / tuples / lower_name. - Narrowed: type aliases -> generic aliases only; default args -> construction helpers only (defaults done); closures -> named function values. - A header now records what's done so far; 8 items remain. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Second review pass (medium/low): - §3.2: clarify that new read helpers reuse the §7.2 fallback *proof* but NOT the ABI — current helpers (field_int/list_len/list_get_int) are Int-only and return a single i64, so non-Int and heap-returning reads (String, nested List/Struct, map values) are a real ABI design task (typed result / handle for heap results, extended handle domain) with its own §7.1 amendment + tests. - Refresh stale regression offenders in plan §0.2/§3.0 and README findings #1/#2 from baseline-20260620.json: list_sort 1.31 / map_int 1.19 / json 1.48 / dynamic_closure 1.66 / string_text 1.80 no longer match; current worst are tier-0 native_read_heap 2.59 / nested_struct_field 1.36 and native task_group_spawn 1.58 / bytes_scan 1.42. Added "re-read JSON before quoting". - Correct the "native near-Rust" framing with nat/rust (not reg/rust): scalar ~1.4x but read_heap ~13.3x. README table now carries an explicit nat/rust column; plan numbers match the committed baseline. - Mark plan status in progress (Phase 0 done; Set/task_group fixed) instead of "draft / not started". Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…ns (Pending #1) Run native_scalar_replace_options on the function before OSR loop detection/compilation, with a transformed->original instruction-index map so the OSR-exit resume_ip points at the original post-loop ip. Lets a hot loop that constructs/matches a non-escaping scalar Option inside an otherwise-ineligible (I/O-tangled, once-called) function run as an allocation-free native loop. Loop-internal Option is dead at the OSR boundary, so live-in/out stay the original loop-carried registers. OSR differential backend byte-identical on/off; 46.9x faster on osr_option_loop. First slice of OSR×J2/J3 composition per the plan. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…ing, Pending #1) Generalize the Option scalar-replacement to user sum/variant values: a non-escaping variant with a single scalar payload per arm dissolves into a tag register (arm index) + payload register (MakeVariant/MatchVariant/ UnwrapVariantValue -> LoadInt/compare/Move), so an inline variant loop in an I/O-tangled once-called function runs allocation-free native via OSR. Reuses the OSR ip-map / dead-at-boundary region gate. OSR differential byte-identical on/off; 41x faster on osr_variant_loop. Positive + escaping-negative acceptance tests. Per the plan, Pending #1. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…adening, Pending #1) Generalize the Option/variant scalar-replacement to user structs: a non-escaping flat struct with scalar fields dissolves into one register per field slot (MakeStruct/GetFieldSlot -> Move), so an inline struct loop in an I/O-tangled once-called function runs allocation-free native via OSR. Third region pass chained after Option+variant, composing ip-maps; nested struct fields bail (future). OSR differential byte-identical on/off; 69x faster on osr_struct_loop. Positive + escaping-negative tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…s (Pending #1) Run native_inline_leaf_calls inside the OSR loop region (now returning a transformed->original ip-map) before the Option/variant/struct passes, composing all ip-maps, so a value built/matched across CallKnown leaves called from the loop (e.g. variant_match_loop's make_shape/area) inlines into the loop body and dissolves to scalars -> allocation-free native. Bails if the OSR boundary maps into an inlined region. OSR differential byte-identical on/off; 65x faster on osr_inline_variant_loop. Positive + negative (non-inlinable leaf) tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…ing #1) Broaden closure inlining to capturing (scalar-capture) monomorphic param-handle closures via a new closure_capture host helper that materializes captures at the inline site, and compose it into the OSR region (ip-map remapped, boundary-into-inline bails). A captured-closure hot loop in an I/O-tangled once-called function inlines + OSRs to an allocation-free native loop. OSR differential byte-identical on/off; 14.1x faster on osr_closure_loop. Stored/polymorphic closures and multi-backedge loops (dynamic_closure_call) deferred. Pos + neg tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…s (Pending #1) Generalize detect_single_natural_loop from exactly-one-backedge to a reducible natural loop with one header (multiple backedges collapsed, internal forward branches allowed), keeping single-exit + contiguous + native-subset/dissolvable region requirements. Also make the OSR inline pass (native_inline_leaf_calls) region-scoped: only calls INSIDE the hot [header,exit) region must be inlinable; a non-inlinable pre-/post-loop helper (e.g. bench_size) is copied through and no longer vetoes OSR. Together these unlock OSR for the real variant_match_loop kernel (internal `if sel==N` reset + an out-of-loop bench_size call). Conservative: multi-exit / multi-header / non-contiguous / multi-entry => reject. OSR differential byte-identical on/off (33/0); 30.98x faster on variant_match_loop. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…nding #1) Relax closure inlining: the closure operand may be any native-readable handle (GetFieldSlot/ListGet result, not just a param), and the polymorphic inline cache now allows scalar-capturing callees (per-arm closure_capture materialization). Makes a stored, polymorphic, capturing closure loop (dynamic_closure_call) allocation-free native via OSR. OSR differential byte-identical on/off; 4.8x faster on dynamic_closure_call. Pos + neg (non-inlinable-callee / non-scalar-capture) tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…nding #1) Recursive struct dissolution: a non-escaping struct whose fields are scalars OR themselves dissolvable structs dissolves to one register per leaf scalar field (innermost-first), so a chained read a.b.c becomes register moves and a nested-struct loop runs allocation-free native via OSR. Conservative: heap/non-dissolvable field or escaping use bails. OSR differential byte-identical on/off; 124x faster on nested_struct_field. Positive + escaping-negative tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…dening, Pending #1) Each variant arm dissolves to a tag register plus one fresh scalar register per payload field (N>=1 per arm), so a variant whose arms carry several scalar fields runs allocation-free native via OSR. Conservative: heap/aggregate payload or escaping use bails. OSR differential byte- identical on/off; 41.7x faster on osr_multifield_variant_loop. Positive + escaping-negative tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…#4 shipped) Audit of the 9-item JIT follow-ups roadmap found the planning docs were stale: items #2/#3/#4/#5 were already shipped this session. Verified each green and corrected the docs to match reality. - jit_acceptance: add two nested-loop OSR regression tests (native_osr_nested_inner_loop_matches_interpreter + ..._with_dirty_outer_...) — both byte-parity with the interpreter and osr_entries > 0. The OSR pipeline is already multi-loop-aware (detect_natural_loops/select_osr_candidate_loop); detect_single_natural_loop is diagnostics-only, not the OSR gate. - vm-optimizing-jit-plan: J0.4 row → S0–S3 shipped (10 AllocatesResult helpers, mem_budget exact via Model-A refusal, §7.2 force-deopt-tested); S4 still future. - jit-followups-roadmap (new): #2 (deque-pop fusion, OSR in-region path), #3 (flat-list read inlining), #4 (nested-loop OSR), #5 (allocate-only heap writes) marked done; remaining = #1 (on-demand), #3-remainder flat-struct, string split-fold, and the hard precise-deopt main line #7→#6→#8. Runtime suite 403 passed (only red = known env-broken jit_perf_gate baseline); vm-jit lib 83/0. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…nsert Adds the `MapInsertHandleKeyInt` host helper and wires it through lowering so a hot `Map<String, Int>` insert loop runs natively (the first heap-key collection write — roadmap item #1's first slice). - vm-jit: new `host_helpers!` row + `HostHelpers` field + typed fn; generic HostCall codegen handles it with no new instruction. - rsscript: `rss_jit_map_insert_handle_key_int` resolves the key handle via the existing `heap_read_handle` and wraps it in `VmMapKey` — hashing/equality is the host's own canonical map-key semantics, never re-implemented in native — then writes through `with_journaled_map_write` so a later bail rolls it back (§7.2). The value is scalar `Int` (no aliasing). - translate.rs: `MapInsert` lowering now selects the helper by key/value type (Int-key/Float, Int-key/Int, heap-key/Int). Test: native_osr_map_insert_string_key_matches_interpreter (osr_entries>0, byte-identical to the interpreter). Note: a freshly-allocated key escaping into a live-in map is rejected by the allocate-only path, so the test uses a live-in String key param. Also corrects the roadmap: #8 (caller-aliased flat-list in-place writes via ListSetIntDirect) and #9 (task_group spawn/join OSR fusion) were mislabeled Pending — both already have shipped, tested slices, so they are in-progress. Verified: vm-jit 83/0, runtime 409 pass (only red = known env-broken perf gate), lib 276/0, differential 33/0, default clippy gate clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…ware list-growth veto Completes #1's heap-VALUE write axis (the key axis shipped earlier as MapInsertHandleKeyInt). A hot loop pushing heap values (e.g. String) onto a list now runs natively. The blocker was the OSR growth-admissibility veto, which refused ALL live-in ListPush. Key fact (verified): flat-buffer pinning is params-only — the TV2 flat classification gates on `reg < n_params`. A function-local (non-parameter) list is handle-accessed, never pinned, so growing it is safe; only a flat PARAM buffer would dangle on realloc. So: - `native_osr_growth_admissible` now takes `n_params` and admits a ListPush when its list is region-local OR a non-parameter register (handle-accessed). A parameter list stays vetoed (conservative). Threaded `n_params` through `osr_loop_region_is_native_subset` and the transform-candidate check. - `ListPushHandle` host helper: resolves the value handle via the existing `heap_read_handle` and appends through the journaled list write (§7.2 rollback). Wired through the table, registration, and ListPush lowering; added to the mem-charge predicate (list push grows a buffer the interpreter accounts). Two existing tests reflected the old blanket veto and are updated for the intended, safe relaxation (NOT regressions — the cross-backend differential stays 33/0 and byte-parity holds): a mutated local-list loop now safely OSRs, and the "reject growth-only setup loop" unit test becomes "accept non-param list growth". New test native_osr_list_push_handle_matches_interpreter (osr_entries>0, parity). Production hotness-gating still keeps short build loops off OSR. Verified: runtime 412 pass (only red = known env-broken perf gate), lib 276/0, differential 33/0, vm-jit 83/0, default clippy gate clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…dle) Adds the `SetInsertHandle` host helper and lowers `Set.insert` with a heap value to it, so a hot `Set<String>.insert` loop runs natively — the same proven pattern as the map-insert/list-push handle helpers. The value handle is resolved via `heap_read_handle` and wrapped in `VmMapKey` (a set is a map with Unit values), so hashing/equality is the host's own canonical key, never re-implemented in native; the write is journaled (§7.2 rollback). Test: native_osr_set_insert_string_matches_interpreter (osr_entries>0, byte-parity). Verified: runtime 415 pass (only red = known env-broken perf gate), differential 33/0, lib 276/0, vm-jit 83/0, default clippy gate clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…lpers Adds `SortedSetInsertHandle` (SortedSet<String>.insert) and `SortedMapInsertHandleKeyInt` (SortedMap<String,Int>.insert), the heap-value/ heap-key analogs of the Int versions, following the same proven pattern: resolve the handle via `heap_read_handle`, let the host's own `sorted_insert_vm` / `sorted_map_insert_in_place` do the ordering, journaled write (§7.2). With these, #1's heap-key/value collection writes now cover the main containers: Map, List, Set, SortedSet, SortedMap. Test: native_osr_sorted_set_and_map_string_insert_matches_interpreter (osr_entries>0, byte-parity for both). Verified: runtime 416 pass (only red = known env-broken perf gate), differential 33/0, lib 276/0, vm-jit 83/0, default clippy gate clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…andle) Adds the `FieldSetHandle` host helper and lowers `SetFieldSlot` with a heap value to it (both the whole-function and OSR-region lowering paths), so a hot loop setting a struct/variant field to a heap value (e.g. `h.name = s`) runs natively. Mirrors `FieldSetInt`: resolves the value handle, COW-rebuilds the struct with the field replaced, and publishes the new value (ReplacesInput + writeback to root). A scalar field at the slot is a shape mismatch and bails. The scalar-field-replacement analysis already disqualifies heap fields, so they route to this helper. With this, #1's heap-key/value write coverage spans the main containers AND struct fields: Map, List, Set, SortedSet, SortedMap inserts + struct FieldSetHandle. Test: native_osr_field_set_handle_matches_interpreter (osr_entries>0, byte-parity). Verified: runtime 417 pass (only red = known env-broken perf gate), differential 33/0, lib 276/0, vm-jit 83/0, default clippy gate clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…te axis complete Adds `DequePushBackHandle` + `DequePushFrontHandle` and lowers `DequePushBack`/ `DequePushFront` with a heap value to them (both lowering paths), so a hot `Deque<String>` push loop runs natively. Same proven pattern: resolve the value handle, journaled deque write (§7.2). This completes #1's heap-key/value collection-write coverage across every container plus struct fields: Map, List, Set, SortedSet, SortedMap inserts, struct FieldSetHandle (COW), and Deque front/back push — each with an OSR byte-parity acceptance test. Test: native_osr_deque_push_handle_matches_interpreter (osr_entries>0, byte-parity). Verified: runtime 418 pass (only red = known env-broken perf gate), differential 33/0, lib 276/0, vm-jit 83/0, default clippy gate clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…truction (deopt ABI) Completes the #7 two-armed Result frontier: a `Result<String,String>` built in a loop and READ AFTER it (live-after) with a HEAP payload now OSRs and reconstructs correctly at OSR-exit (previously declined — the deopt record couldn't carry a heap value). Two coordinated pieces: 1. Handle-carrying deopt ABI. `DeoptValue` gains a `Handle(i64)` variant; `decode_deopt_live` captures a `Handle` reg's heap-table index (was dropped). The live-after variant reconstruction resolves that index against the still-live JIT heap (`JitHostCallCtx::from_token(heap_tx.host_ctx()).heap_read_handle`, the same path host helpers use) to rebuild `Ok`/`Err`. All other deopt consumers (try_native restore, scalar-field writeback, option reconstruct, general OSR writeback) skip `Handle` (their old behavior, since `Handle` regs were previously absent from the live set). Recipe validation relaxes `scalar` → `scalar_or_handle` for payloads. 2. Runtime param-type seeding. A param used in-region ONLY as a dissolved payload `Move` (the unwrap/consumer is outside the loop) has no typed use for the inference to unify against, so the payload `Move` lowering declined (`ty[src]?`). `try_osr` now classifies each param by its live value and the translator seeds it into the inference — but ONLY for params NOT "typed elsewhere": a reg whose value reaches any in-region NON-`Move` instruction (a typed op or a collection helper like `MapInsert`/`ListPush`, which type it authoritatively — e.g. a heap map key as the Int handle-index) is tainted (back- propagated through `Move`s) and left to inference. This taint is essential: pre-seeding a heap collection key/value param `Handle` would conflict with its helper-spec Int type and wrongly decline the loop. Test: native_osr_two_armed_heap_result_live_after_reconstructs (osr_entries>0, byte-parity). Verified: runtime 422 pass (only red = known env-broken perf gate), differential 33/0, lib 276/0, vm-jit 83/0, default clippy clean — the deopt-ABI + inference changes (both high-blast-radius: every deopt / every OSR loop) caused NO regression. In particular the five #1 collection-write OSR loops still OSR (the taint gate protects them). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…silent mis-key bug) CORRECTNESS FIX. The #1 heap-key/value collection-write helpers were effectively UNREACHABLE in OSR: the type-inference forced collection key/value operands to `Int` (`native_set_ty(operand, NativeTy::Int)`), so the lowering's `handle_reg` test never fired and a heap key/value was mis-lowered to the Int helper — storing the operand's heap-table INDEX as a flat Int instead of the heap value. The original single-element / `len`-only acceptance tests could not see this (one repeated key → len 1 regardless), so it shipped silently; the differential didn't cover heap-keyed insert+lookup either. Discriminating tests (insert in a hot loop, then look the key back up) exposed it: `Map<String,Int>` returned -1 (lookup miss) vs the interpreter's 199. Fix: classify each param by its live value in `try_osr` (already threaded as `param_native_types`) and, in the OSR inference, type a heap collection key/value operand `Handle` (via a `heap_param` predicate) instead of the default `Int`. The Handle helpers (`MapInsertHandleKeyInt`, `SetInsertHandle`, `SortedSet/MapInsert*`) then fire correctly for heap operands; Int/Float operands are unchanged. The param seed is also simplified (its prior taint was only needed to dodge the now-removed Int-force conflict; the seed and the inference now agree, so all classified params are seeded fill-None) — this also types heap-value params for the unconstrained-value ops (`ListPush`/`MapInsert`-value/`DequePush`/ `SetFieldSlot`). The whole-function tier has no runtime param classification, so its `heap_param` is always-false (that path's latent same-bug is documented, left unchanged). A `List<String>` push loop now CORRECTLY DECLINES OSR (a Boxed/heap-element list's capacity growth is not natively accounted) and runs on the interpreter — previously it "OSR'd" only by the mis-typing that made it look like a flat-int list. Tests: native_osr_map_insert_string_key_lookup_matches_interpreter (DISCRIMINATING: insert+lookup, osr_entries>0, key correct); native_osr_list_push_handle now reads an element back (discriminating) and asserts interpreter parity (declines OSR, correct). Verified: runtime 423 pass (only red = known env-broken perf gate), differential 33/0, lib 276/0, vm-jit 83/0, clippy clean. The live-after heap-payload Result (04a24de) still OSRs; the heap map/set/sorted inserts now OSR with CORRECT keys/values. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…insert + read-back) Adds DISCRIMINATING regression guards that read a heap key/value back after building a collection — the kind of test that would have caught the #1 silent mis-key bug (a `len`-only check cannot). All confirm native OSR matches the interpreter: - native_osr_heap_set_and_sorted_map_discriminating: Set<String> insert+contains (two distinct keys present, a third absent) and SortedMap<String,Int> insert+lookup. - native_whole_function_heap_key_get_matches_interpreter / native_whole_function_heap_key_insert_matches_interpreter: confirm the whole-function tier handles heap-key Map ops correctly — it declines to compile them (compiled=0) and runs on the interpreter, so the same latent inference mis-typing is UNREACHABLE there. Verified: runtime 426 pass (only red = known env-broken perf gate). Map/List (3237704) plus Set/SortedMap heap-collection paths are now correctness-locked. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

… cleanups Address a code review of the cold-arm / native-heap-write work. HIGH (soundness): the interpreter deep-copies every non-`mut` heap parameter at the prologue (`DeepCopy`), giving the callee an isolated copy; native lowers `DeepCopy` to a Nop. That was sound when native was read-only, but native now performs in-place heap WRITES, so mutating a DeepCopy'd param's value (directly or via an alias) leaked to the caller while the interpreter mutated only the copy. Confirmed: a `read List<Int>` param aliased by `let mut ys = xs` and written via `List.set` in a hot loop left the caller's list mutated under native (`199999`) vs unchanged under the interpreter (`7`). Fix: a taint analysis (`native_deepcopy_param_unsoundly_mutated`) seeds every DeepCopy'd heap-param register, propagates the taint forward through `Move` and heap-extraction ops (alias edges), and DECLINES native (whole-function AND OSR paths) if a tainted heap container is mutated in place or passed as a `mut` call arg. Storing/returning a tainted value is deliberately NOT flagged: native types collapse all heap values to `Handle`, so flagging stores would wrongly decline the common, sound pattern of inserting a `read` string into a `mut` collection (verified: it would have regressed the #1 collection-write + selfhost-mailbox tests). Regression test: native_non_mut_heap_param_mutation_does_not_leak_to_caller. MEDIUM (robustness): jit_value_contains_list_rc tracked only `Managed` pointer identity, so a cyclic List/Map/Deque heap graph could recurse forever / stack overflow. Now records the pointer identity of every interior-mutable container (List/Deque/Map/Managed) in one visited set; a back-edge returns false. MEDIUM (perf): the OSR string-literal interner used an O(L) linear scan per literal (O(L²) total). Mirror the whole-function `HashMap<Rc<String>, i64>` interner for O(1)-amortized interning. LOW: remove two dead items (the always-false `heap_param` closure in the whole-function path; the unused `external_reads_touch` method) — native-jit build + default clippy now have 0 warnings. Reorder exec.rs so the `tick()` doc-comment and `#[inline]` attach to `tick()` instead of being stranded above `native_limits_unarmed`. Verified in dev container: lib 276/0, runtime 448/1 (sole red = known env-broken jit_perf_gate), vm-jit 83/0, differential 33/0, soak 7/0; 0 build/clippy warnings. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The previous DeepCopy soundness guard caught direct in-place mutation and mut-arg passing of a tainted heap param, but missed the STORE-AND-RELOAD shape: storing a non-`mut` `read List` param into a struct field launders the alias taint (the struct is not tainted, so a later `GetFieldSlot` reload is not either), and the subsequent `List.set` on the reload mutated the caller's original handle. Also observable without the reload: storing the tainted handle into a caller-visible aggregate that escapes. Fix: flag STORING a tainted value into caller-visible heap (SetFieldSlot / List.set/push / Set/SortedSet insert / Deque push / MapInsert value / MakeStruct/Variant/List/Map/Some) and RETURNING it. Naively flagging all tainted stores regressed the #1 string-key collection tests (a `read String` inserted into a `mut` collection is sound -- strings are immutable, sharing the `Rc` is unobservable) and the selfhost-mailbox test (inlined-leaf params of unknown kind). So the guard now keeps TWO taint sets: - broad `tainted` (all DeepCopy'd roots except proven-immutable String/Bytes, incl. inlined params) drives the in-place-mutation + mut-arg checks (never fire on an immutable value); - narrow `mut_tainted` (only roots PROVEN to be a mutable outer param) drives the store/return checks, which cannot distinguish String from a container at the native-type level otherwise. Heap-kind source (`immutable_leaf_params`, true = String/Bytes): the whole -function path reads the declared signature (native_declared_type_is_immutable_leaf); the OSR path reads the runtime param value in try_osr (VmValue::String/Bytes), plumbed through translate_osr_loop_profiled/_inner. Tests: native_store_reload_mutate_non_mut_heap_param_does_not_leak (struct field store -> reload -> mutate) alongside the existing direct-alias test. Residual: an inlined-leaf param's store/reload leak (no type info for inlined params) -- far narrower than the in-place case already closed; differential is the backstop. Verified in dev container: lib 276/0, runtime 449/1 (sole red = known env-broken jit_perf_gate), vm-jit 83/0, differential 33/0, soak 7/0; 0 build/clippy warnings. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Turn the real-project workloads from engagement-only into wall-time gated where stable enough (measured 2×9 iterations, release): - selfhost_mailbox_bench (mixed ring-buffer actor): 40.5/41.2ms, ~2% run-to-run — STABLE. Uses its existing committed baseline (native median 36.2ms; native beats the interpreter ~2.4x on this real workload). Now in DEFAULT_CASES, wall-time gated + engagement-asserted. (+15% vs the June-26 baseline is real native drift, within the 75% threshold — left visible rather than hidden by a refresh.) - selfhost_stdlib_reporter (collection-heavy): 1.31/1.44ms, ~10% — added a committed native_ms=1.45 baseline; wall-time gated (the generous 75% threshold absorbs the noise) + engagement-asserted. - selfhost_manifest_inspector: a one-shot tool, compiles NO native (~0.05ms) — wall time is pure noise, so it STAYS telemetry-only. Verified: clippy 0; release gate times mailbox (+15.3% OK) and stdlib (-2.4% OK); debug full runtime 453/0 (36s; mailbox adds ~6s in debug, timing skipped there). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Item #1 (unify native op metadata) — first and highest-value slice: the flat-list DIRECT op set (ListGet/Set Int/Float Direct + ListLenDirect + ListIsEmptyDirect) was hand-enumerated in multiple classification sites and had DRIFTED — the native-leaf eligibility set was missing ListSetFloatDirect/ListIsEmptyDirect (fixed reactively earlier this session). Establish one source of truth: - New canonical `JitInstr::is_flat_list_direct()` in vm-jit. - The boolean predicate sites now derive from it: `jit_function_scalar_leaf_callable` (rsscript tier.rs) and `native_scalar_leaf_callable` (vm-jit). Adding a new *Direct op now only needs updating the canonical method — those sites cannot drift again. - The cost model's `match` (profitability.rs) keeps EXPLICIT patterns on purpose: its exhaustiveness is the complementary drift guard (a new op forces a compile error there), which a `_ if is_flat_list_direct()` guard arm would have silently defeated. Pure classification consolidation — no generated code changes. Verified behavior-identical: build 0 warnings; clippy 0; vm-jit 83/0; lib 276/0; differential 33/0 (native-leaf composition parity holds). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Haofei and others added 16 commits June 6, 2026 23:00

Update app-review-benchmark review-map counts after fresh removal

9e6e855

Dropping fresh from the 2 class constructors reclassifies them Foldable, so review_required 35->33, foldable 9->11. Update the assertions. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Haofei merged commit 9877d81 into main Jun 7, 2026
1 of 2 checks passed

Haofei deleted the vm-native-dynamic-loading branch June 15, 2026 01:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

VM dynamic native loading + sqlx facade and sqlite benchmark#1

VM dynamic native loading + sqlx facade and sqlite benchmark#1
Haofei merged 16 commits into
mainfrom
vm-native-dynamic-loading

Haofei commented Jun 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

Haofei commented Jun 7, 2026

Summary

What's included

Verification

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant