test/docs: strengthen self-host parity gates + refresh stale self-host docs

Follow-up review findings on the self-host harness:
- Add non-ignored negative-parser and positive-RS0005-checker smoke tests, so a
  tool degenerating to always-OK / always-CLEAN is caught without the ignored
  corpus gate (the accept-only / no-duplicate tiny samples could not).
- rss_dump_with now errors on a malformed non-empty dump line instead of silently
  dropping it (filter_map), so a stray/garbled line fails the harness.
- Refresh stale counts: plan table and ledger SH-018/SH-023 now read 556/556 (was
  544/545/546) and ~46x (was ~5100x); SH-023 note reflects the recover-past-
  malformed scanner (was "stops at first").
- FORMAT.md: add Char and Unknown to the KIND list; reconcile the "richest line"
  claim with lexer.rss's current 0:0:0 placeholder positions.

Verified: self-host lib tests + corpus gates (lexer/parser/checker 556x, perf) green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Author: olwang

crates/rsscript/src/selfhost_parity.rs

@@ -147,12 +147,17 @@ fn rss_dump_with(exe: &RegVmExecutable, source: &str) -> Result<Vec<CanonTok>, S
     let output = exe
         .eval_main_with_args([source.to_string()])
         .map_err(|e| format!("rss lexer failed to run: {e:?}"))?;
-    Ok(output
+    // Fail on any malformed non-empty dump line rather than silently dropping it
+    // (a stray debug line or a garbled token would otherwise vanish and let a
+    // broken lexer pass parity by emitting fewer/no tokens).
+    output
         .stdout
         .lines()
         .filter(|l| !l.is_empty())
-        .filter_map(parse_line)
-        .collect())
+        .map(|l| {
+            parse_line(l).ok_or_else(|| format!("rss lexer emitted a malformed dump line: {l:?}"))
+        })
+        .collect()
 }
 
 /// Convenience: compile + run once (used by the single-file smoke test).
@@ -417,6 +422,24 @@ fn parser_parity_tiny_sample() {
     compare_parse(oracle, actual, parse_position_tier()).unwrap_or_else(|msg| panic!("{msg}"));
 }
 
+/// Phase-2 NEGATIVE smoke (non-ignored): the rss parser must REJECT malformed
+/// source, matching the Rust oracle. The accept-only tiny sample above would
+/// still pass if the rss parser degenerated to always printing `OK`; this closes
+/// that gap without needing the (ignored) full-corpus gate.
+#[test]
+fn parser_rejects_malformed_source_smoke() {
+    let source = "fn main() -> Unit {\n    return Unit\n}\n\nfn\n";
+    let oracle = parse_oracle_error("parser-negative.rss", source);
+    assert!(
+        oracle.is_some(),
+        "oracle Rust parser must reject the malformed sample (else the smoke test proves nothing)"
+    );
+    let exe = compile_parser().expect("rss parser should compile");
+    let actual = run_parser(&exe, source).expect("rss parser should run");
+    // Recognition tier: both must reject (accept-vs-reject only).
+    compare_parse(oracle, actual, false).unwrap_or_else(|msg| panic!("{msg}"));
+}
+
 /// Phase-2 gate (ignored by default): the rss parser's accept/reject matches the
 /// Rust parser over the whole `.rss` corpus.
 #[test]
@@ -534,6 +557,27 @@ fn checker_parity_tiny_sample() {
     assert_eq!(oracle, actual, "checker parity diverged on tiny sample");
 }
 
+/// Phase-3 POSITIVE smoke (non-ignored): the rss checker must REPORT RS0005 for a
+/// duplicate declaration, matching the analyzer. The no-duplicate tiny sample
+/// above would still pass if the rss checker degenerated to always printing
+/// `CLEAN`; this closes that gap without the (ignored) full-corpus gate.
+#[test]
+fn checker_reports_rs0005_for_duplicate_declaration_smoke() {
+    let source =
+        "fn dup() -> Unit {\n    return Unit\n}\nfn dup() -> Unit {\n    return Unit\n}\n";
+    let oracle = checker_oracle_codes("checker-duplicate.rss", source);
+    assert!(
+        oracle.contains(&"RS0005".to_string()),
+        "oracle analyzer must report RS0005 for the duplicate declaration; got {oracle:?}"
+    );
+    let exe = compile_checker().expect("rss checker should compile");
+    let actual = run_checker(&exe, source).expect("rss checker should run");
+    assert_eq!(
+        oracle, actual,
+        "checker parity diverged on the duplicate-declaration smoke test"
+    );
+}
+
 /// Phase-3 gate (ignored by default): the rss checker's target-code diagnostics
 /// match the analyzer over the whole `.rss` corpus.
 #[test]

docs/ledgers/rss-selfhost-ledger.md

@@ -501,7 +501,7 @@ gap is VM value-representation / intrinsic-dispatch cost (the next big lever).
   emits `(*pos)` on read and as the assignment target for such a param, since
   `mut T` already lowers to `&mut T`. Non-Copy `mut` params keep their `&mut Struct`
   lowering and stay non-reassignable (only fields/elements are mutable).
-- **Tests:** `crate::selfhost_parity::lexer_parity_corpus` (tier 0, 544/544);
+- **Tests:** `crate::selfhost_parity::lexer_parity_corpus` (tier 0, 556/556);
   `tests/fixtures/pass/mut-scalar-writeback.rss` (Int + Bool write-back).
 - **Status:** fixed (scalar Copy `mut` params are reassignable with caller
   write-back; non-Copy `mut` params stay non-reassignable).
@@ -770,8 +770,8 @@ gap is VM value-representation / intrinsic-dispatch cost (the next big lever).
 
 - **Tool:** self-hosted checker (`selfhost/check.rss`) run on the reg-VM vs
   `crate::analyze_source` filtered to error-severity `RS0005`
-  (DUPLICATE_DECLARATION), over the whole 546-file corpus.
-- **Symptom (positive):** the checker reproduces RS0005 with **546/546** parity
+  (DUPLICATE_DECLARATION), over the whole 556-file corpus.
+- **Symptom (positive):** the checker reproduces RS0005 with **556/556** parity
   using ONLY top-level declaration structure — no statement/expression/pattern
   body parsing (confirms SH-021: RS0005 is decidable from declaration shape). It
   reuses the proven `selfhost/parser.rss` recognizer verbatim; the sole addition
@@ -801,10 +801,11 @@ gap is VM value-representation / intrinsic-dispatch cost (the next big lever).
   threaded cursors returned by value (SH-018). `Set<String>` (`Set<String>.new()`,
   `Set.contains<String>`, `Set.insert<String>`) worked as the duplicate detector;
   `features: local` was NOT needed (no `StringBuilder`/`local` bindings). The
-  scanner conservatively STOPS on the first malformed/unknown top-level item
-  (mirroring the recognizer), which can only under-report on syntactically broken
-  files — safe, since the analyzer emits RS0005 on exactly the 2 well-formed
-  fixtures and the other 544 files stay CLEAN (zero false positives).
+  scanner RECOVERS past an unrecognized top-level item (skips one token and keeps
+  scanning, mirroring the analyzer's recovery) rather than stopping at the first
+  one, so a later duplicate is not missed. Parity holds: the analyzer emits RS0005
+  on exactly the 2 well-formed duplicate fixtures and the other 554 files stay
+  CLEAN (zero false positives).
 - **Classification:** docs (records the analyzer's duplicate-symbol namespace
   rule and that RS0005 is a declaration-only property).
 - **Tests:** `crate::selfhost_parity::checker_parity_tiny_sample` and

docs/self-hosting-stress-test-plan.md

@@ -187,14 +187,16 @@ runs it on the reg-VM in-process, passing the corpus file's *content* as `argv[0
 
 | Phase | rss tool | Oracle | Corpus result |
 |-------|----------|--------|---------------|
-| 1 — lexer | `selfhost/lexer.rss` | `crate::lexer::lex` (canonical token dump) | **544/544 tier-0**, 0 run-failures |
-| 2 — parser | `selfhost/parser.rss` | `crate::syntax::parse_source_raw` (accept/reject) | **545/545 recognition** |
-| 3 — checker | `selfhost/check.rss` | `crate::analyze_source` (code `RS0005`) | **546/546** |
-| 4 — perf | lexer on VM vs native | wall-clock over corpus | **~5100× slower** (see SH-022) |
+| 1 — lexer | `selfhost/lexer.rss` | `crate::lexer::lex` (canonical token dump) | **556/556 tier-0**, 0 run-failures |
+| 2 — parser | `selfhost/parser.rss` | `crate::syntax::parse_source_raw` (accept/reject) | **556/556 recognition** |
+| 3 — checker | `selfhost/check.rss` | `crate::analyze_source` (code `RS0005`) | **556/556** |
+| 4 — perf | lexer on VM vs native | wall-clock over corpus | **~46× slower** (post-SH-022; was ~5100×) |
 
 Gates (all green): `cargo test -p rsscript --features native-jit --lib` (3 tiny-sample
-tests); the corpus gates run with `-- --ignored` (`lexer_parity_corpus`,
-`parser_parity_corpus`, `checker_parity_corpus`, `lexer_perf_corpus`).
+tests plus a non-ignored negative-parser and positive-`RS0005`-checker smoke test, so
+a degenerate always-`OK`/always-`CLEAN` tool is caught without the corpus gate); the
+corpus gates run with `-- --ignored` (`lexer_parity_corpus`, `parser_parity_corpus`,
+`checker_parity_corpus`, `lexer_perf_corpus`).
 
 ### Findings (ledger `SH-016` … `SH-023`)
 - **SH-016** — no character-literal syntax; `'` lexes to `?`, cascading a misleading

selfhost/FORMAT.md

@@ -10,17 +10,21 @@ tab-separated fields:
 - `line`, `col` — 1-indexed position of the token start (matches `Span.line` /
   `Span.column`). `len` — byte length of the token (matches `Span.length`).
 - `KIND` — one of the exact `TokenKind` variant names:
-  `Ident Number String InterpolatedString MultilineString Keyword Symbol Eof`.
-- `PAYLOAD` — the token's raw text (ident/number/keyword/symbol text, or the raw
-  content captured between string delimiters). For `Eof` the payload is empty.
+  `Ident Number String Char InterpolatedString MultilineString Keyword Symbol
+  Unknown Eof`. `Char` is a character literal `'c'`; `Unknown` is a source
+  character outside the lexical inventory (payload is that raw char).
+- `PAYLOAD` — the token's raw text (ident/number/keyword/symbol/char text, or the
+  raw content captured between string delimiters). For `Eof` the payload is empty.
 - PAYLOAD is escaped so a token is always exactly one line:
   `\` → `\\`, newline → `\n`, tab → `\t`, carriage return → `\r` (backslash first).
 
 ## Comparison tiers (env `RSS_SELFHOST_TIER`, default `0`)
 
-The rss lexer always emits the full richest line. The harness parses both dumps
+The rss lexer emits all three fields on every line; the harness parses both dumps
 into structured tokens and compares only the fields for the active tier, so a
-single rss implementation supports graded parity:
+single rss implementation supports graded parity. (The current `selfhost/lexer.rss`
+has not implemented span tracking yet, so it emits placeholder `0:0:0` positions
+and is exercised at tier 0 — see the note below.)
 
 - **tier 0** (default): compare `(KIND, PAYLOAD)` — pure tokenization logic.
 - **tier 1**: also compare `(line, col)` — span position arithmetic.