selfhost Phase 4 (lexer perf): ~5000x VM slowdown is per-char dispatch, not string O(n^2)

Add lexer_perf_corpus benchmark (rss lexer on reg-VM vs native lex() over the
712KB corpus). Result: 79.5s vs 15.3ms (~5100x). Controlled experiment:
switching token/output building from String.concat (O(n^2)) to StringBuilder
(O(n)) moved nothing — the cost is ~6 intrinsic dispatches per source char
(List.get on List<Char> + Char.* peeks), i.e. VM value-representation +
intrinsic-dispatch overhead. Recorded as SH-022; feeds the parked perf roadmap
with real-workload evidence. Lexer keeps StringBuilder (correct/idiomatic).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Author: olwang

crates/rsscript/src/selfhost_parity.rs

@@ -254,6 +254,50 @@ fn lexer_parity_corpus() {
     );
 }
 
+/// Phase-4 perf probe (ignored; run with `--release -- --ignored --nocapture`):
+/// how much slower is the self-hosted rss lexer (on the reg-VM) than the native
+/// Rust `lex()` over the whole corpus? This is the "is the self-hosted tool
+/// slow?" macro-benchmark — a real workload, not a microkernel. Feeds the parked
+/// VM value-representation / intrinsic-dispatch perf work.
+#[test]
+#[ignore]
+fn lexer_perf_corpus() {
+    use std::time::Instant;
+    let root = workspace_root();
+    let files = collect_rss_files(&root);
+    let exe = compile_lexer().expect("rss lexer should compile");
+    let mut rust_ns: u128 = 0;
+    let mut rss_ns: u128 = 0;
+    let mut bytes: usize = 0;
+    let mut n_ok = 0usize;
+    for file in &files {
+        let rel = file.strip_prefix(&root).unwrap_or(file).display().to_string();
+        let Ok(source) = std::fs::read_to_string(file) else {
+            continue;
+        };
+        bytes += source.len();
+        let t0 = Instant::now();
+        let _ = lex(&rel, &source);
+        rust_ns += t0.elapsed().as_nanos();
+        let t1 = Instant::now();
+        if exe.eval_main_with_args([source.clone()]).is_ok() {
+            n_ok += 1;
+        }
+        rss_ns += t1.elapsed().as_nanos();
+    }
+    let rust_ms = rust_ns as f64 / 1e6;
+    let rss_ms = rss_ns as f64 / 1e6;
+    eprintln!(
+        "\n=== lexer_perf_corpus ===\n  files: {} (ran {n_ok})\n  bytes: {bytes}\n  \
+         Rust lex():   {rust_ms:.1} ms  ({:.1} MB/s)\n  rss lexer/VM: {rss_ms:.1} ms  \
+         ({:.1} MB/s)\n  slowdown (rss/Rust): {:.1}x\n",
+        files.len(),
+        bytes as f64 / 1e6 / (rust_ms / 1e3),
+        bytes as f64 / 1e6 / (rss_ms / 1e3),
+        rss_ms / rust_ms,
+    );
+}
+
 // ---------------------------------------------------------------------------
 // Phase 2 — parser recognition parity.
 //

docs/ledgers/rss-selfhost-ledger.md

@@ -536,3 +536,41 @@ gap is VM value-representation / intrinsic-dispatch cost (the next big lever).
   the writeup does not overclaim — Phase 2 delivered a self-hosted *recognizer*.
 - **Tests:** `crate::selfhost_parity::parser_parity_corpus`.
 - **Status:** decided.
+
+### SH-022 — self-hosted lexer is ~5000× slower on the VM; cost is per-char intrinsic/collection dispatch, NOT string building
+
+- **Tool:** self-hosted lexer (`selfhost/lexer.rss`) run on the reg-VM vs native
+  `crate::lexer::lex`, over the whole 545-file corpus (712 KB).
+- **Symptom (measured, release):**
+  | lexer | time | throughput |
+  |-------|------|-----------|
+  | native Rust `lex()` | 15.3 ms | 46.5 MB/s |
+  | rss lexer on reg-VM | **79.5 s** | ~0.009 MB/s |
+  → **~5100× slowdown** (~112 µs per source char).
+- **Controlled experiment:** rewrote the token/output string building from repeated
+  `String.concat` (O(n²)) to `StringBuilder` (O(n)). **No measurable change**
+  (5140× → 5195×, within noise), parity still 544/544. So string-building is NOT
+  the bottleneck (most tokens are short, so the quadratic term never dominates).
+- **Backend:** vm (also relevant to tier-0/native — this code is intrinsic-bound,
+  not native-eligible, cf. SH-001/SH-004).
+- **Root cause:** the main loop is O(n) (single `String.chars` + one pass), but does
+  ~6 intrinsic dispatches PER CHARACTER — `List.get` on a `List<Char>` (VmValue
+  boxing + refcount) plus `Char.to_code`/`is_whitespace`/`is_digit`/… and the
+  `code_at` peeks (c1,c2). The dominant cost is the VM's per-op value-representation
+  + intrinsic-dispatch overhead over ~712 K chars, exactly the lever flagged by
+  SH-001/SH-004/SH-011 and the parked perf roadmap. This is the first REAL-workload
+  profile that is unambiguously VM-dispatch-bound.
+- **Classification:** VM (value representation + intrinsic dispatch) + stdlib
+  (no char-cursor / native String-iteration intrinsic; `String.chars → List<Char>`
+  forces per-char boxed access).
+- **Decision:** the real lever is cheaper per-char access, NOT string building:
+  e.g. a native string byte/char cursor intrinsic (iterate without materializing a
+  boxed `List<Char>`), and lower per-intrinsic dispatch overhead. Cross-ref SH-006:
+  AOT is ~144× faster than the VM on such code, so an AOT-compiled self-hosted lexer
+  would land ~0.5 s (~30× native Rust) — the residual being the intrinsic *count*
+  per char, which the char-cursor intrinsic would also cut. Feeds
+  [[perf-refactor-roadmap]] / [[jit-collection-perf-measurement]] with real-workload
+  evidence (the trigger that work was waiting for).
+- **Tests / bench:** `crate::selfhost_parity::lexer_perf_corpus`
+  (`--release -- --ignored`).
+- **Status:** open (VM/stdlib lever identified; feeds perf roadmap).

selfhost/lexer.rss

@@ -8,6 +8,11 @@
 // rss constraints honored throughout (see ledger SH-016/SH-017):
 //  - no character literals: compare code points via Char.to_code.
 //  - statement-level expressions stay on ONE line.
+//
+// SH-022: string building uses StringBuilder (O(n)) not repeated String.concat
+// (O(n^2)) — `local` bindings require `features: local`.
+
+features: local
 
 // The 30 reserved keywords (crate::lexer::KEYWORDS). Contextual words
 // (await, native) and builtin constants (true false Ok Err Some None Unit)
@@ -41,39 +46,39 @@ fn code_at(chars: read List<Char>, n: read Int, i: read Int) -> Int {
 
 // Raw substring chars[from..to].
 fn slice(chars: read List<Char>, from: read Int, to: read Int) -> String {
-    let mut out = ""
+    local out = StringBuilder.new()
     let mut k = from
     while k < to {
         let c = List.get(list: read chars, index: read k)
-        out = String.concat(left: read out, right: read Char.to_string(value: read c))
+        StringBuilder.push(builder: mut out, value: read Char.to_string(value: read c))
         k = k + 1
     }
-    return out
+    return StringBuilder.finish(builder: take out)
 }
 
 // Escape a payload exactly like the harness: backslash->\\, \n, \t, \r.
 fn escape(s: read String) -> String {
     let cs = String.chars(value: read s)
     let m = List.len(list: read cs)
-    let mut out = ""
+    local out = StringBuilder.new()
     let mut k = 0
     while k < m {
         let c = List.get(list: read cs, index: read k)
         let code = Char.to_code(value: read c)
         if code == 92 {
-            out = String.concat(left: read out, right: read "\\\\")
+            StringBuilder.push(builder: mut out, value: read "\\\\")
         } else if code == 10 {
-            out = String.concat(left: read out, right: read "\\n")
+            StringBuilder.push(builder: mut out, value: read "\\n")
         } else if code == 9 {
-            out = String.concat(left: read out, right: read "\\t")
+            StringBuilder.push(builder: mut out, value: read "\\t")
         } else if code == 13 {
-            out = String.concat(left: read out, right: read "\\r")
+            StringBuilder.push(builder: mut out, value: read "\\r")
         } else {
-            out = String.concat(left: read out, right: read Char.to_string(value: read c))
+            StringBuilder.push(builder: mut out, value: read Char.to_string(value: read c))
         }
         k = k + 1
     }
-    return out
+    return StringBuilder.finish(builder: take out)
 }
 
 fn emit(kind: read String, payload: read String) -> Unit {