Skip to content

Commit a048310

Browse files
olwangclaude
andcommitted
fix(selfhost): move tk_col/tk_len into astdump.rss (parser.rss owns its own tk_col)
The shared scan.rss prelude is prepended to EVERY tool; parser.rss already declares its own `tk_col`, so adding one to scan.rss made parser.rss fail to compile (RS0005 duplicate declaration) — the AST corpus tests only compile astdump.rss so they didn't surface it. Relocate the two new accessors into astdump.rss (identical bodies; astdump already defines its own tk_text the same way). All 9 non-ignored selfhost_parity tests green. Also record the AST span milestone (all three tiers 619/619) in the ledger. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 629253d commit a048310

3 files changed

Lines changed: 59 additions & 22 deletions

File tree

docs/ledgers/rss-selfhost-ledger.md

Lines changed: 43 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1069,11 +1069,46 @@ gap is VM value-representation / intrinsic-dispatch cost (the next big lever).
10691069
now byte-exact at **all three tiers** (0 kind+payload, 1 +line/col, 2 +length) —
10701070
576 files, token-mismatches 0 each. The lexer span ladder is fully closed; the
10711071
additive `Tok.len` left parser/checker/astdump parity untouched.
1072-
- **AST spans (step 3, remaining "last phase"):** NOT done. Would need each
1073-
`astdump.rss` `emit` to carry the correct token's span and the oracle serializer
1074-
to append ` @L:C:N` per node (plus a tier-strip so tier-0 stays green). Feasible
1075-
but large + precise: node spans are mostly `tokens[start]` but not uniformly
1076-
(`Try` uses the `?` token, some binary the operator, ReceiverCall the receiver),
1077-
so it is an invasive per-node-type effort across ~150 emit sites — its own phase.
1078-
- **Status:** open. Step-2 milestone 2a and step-3 lexer spans are done and gated;
1079-
step-2 deeper checks and step-3 AST spans are the scoped remainder.
1072+
- **AST spans (step 3) — DONE, 2026-07-03:** `ast_parity_corpus` is byte-exact
1073+
**619/619 at all three tiers** (0 structure+payload, 1 `@line:col`, 2
1074+
`@line:col:len`), 0 run-failures each.
1075+
- **Discovered span rule (load-bearing):** every AST node's `Span` is ONE
1076+
representative token's `line:col:len` — never a multi-token range. So the
1077+
producer reproduces a node's span by emitting that ONE token's position and
1078+
length (`tk_len`), NOT by measuring the node's extent. Representative token
1079+
per node: *first (paren-trimmed) token* for the vast majority; **Binary → the
1080+
operator token**; **Try → the trailing `?`**; **ReceiverCall → the receiver
1081+
token** (`tokens[receiver_start]`, i.e. after the effect kw); **TypeRef → the
1082+
NAME token** (`name_index`, after prefix keywords read/mut/take/fresh/handle/
1083+
weak/noescape/owned); **tuple type/expr → the `(`**; decls (fn/type/sum/const/
1084+
alias/module/use) → the decl's FIRST token *including* `#`-attributes and
1085+
`pub` (parse_*'s `current()` at entry); **MatchArm → the arm's first token**;
1086+
**if-let's two synthetic arms + the synthetic protocol `Self: Managed` generic
1087+
→ the `if`/`fn` token** (`method.span`); interpolated-string desugar nodes
1088+
(call/effect/string/array) → the interp-string token, and its EMBEDDED exprs
1089+
self-reproduce because BOTH oracle and producer re-tokenize the fragment from
1090+
1:1; an empty named call-arg (`print(value:)`) → the arg's NAME token
1091+
(parse_call_args' `tokens[start]` Unknown fallback). Patterns (`pat-*`) and
1092+
pure structural labels (value/body/block/cond/then/else/callee-*/arg/
1093+
object-field/map-entry/key/derive/bound/effect-name/malformed-generic|param|
1094+
field|effect|arm/…) are NOT spanned on EITHER side.
1095+
- **Mechanism (mirrors the lexer ladder, no new invention):** the producer
1096+
ALWAYS emits the richest ` @line:col:len` via `emit_at`/`spanof`; the harness
1097+
(`run_astdump`) PROJECTS each line down to the active `RSS_SELFHOST_AST_TIER`
1098+
before the byte-exact compare (tier 0 drops the suffix, tier 1 keeps
1099+
`line:col`). No tier flag threads through the producer. The oracle side uses
1100+
the pre-added `sp`/`push_node` scaffolding, tier-gated by the same env var.
1101+
`expr_rep_tok` mirrors `emit_expr`'s dispatch to recover an expression's span
1102+
token without descending (used for expr-statement heads + single-expression
1103+
closure bodies — the latter was the dominant tier-1 failure, since
1104+
`noescape-callback-*` fixtures all pass `|| …`/`|x| …` callbacks whose
1105+
`Stmt::Expr` body head is spanned).
1106+
- **Commits:** 1d86a430 (oracle heads → push_node, tier 0 unchanged 619/619) /
1107+
629253d2 (producer spans + harness projection; tier 1 619/619; tier 2 619/619
1108+
verified on the same code — `:len` is just the rep token's own length, which
1109+
lexer parity already proved matches). No node types left blocked.
1110+
- **Gate:** default (env unset) stays tier 0, byte-exact 619/619 — the committed
1111+
gate. Tiers 1 and 2 are run via `RSS_SELFHOST_AST_TIER=1|2`.
1112+
- **Status:** the frontend-object AST parity ladder (structure → line:col
1113+
line:col:len) is CLOSED. Remaining self-host frontier is the semantic tier
1114+
(type-inference + borrow-checker codes; see the SH-026 step-2 tail).

selfhost/astdump.rss

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -77,6 +77,22 @@ fn tk_text(toks: read List<Tok>, i: read Int) -> String {
7777
return t.text
7878
}
7979

80+
// Span-field accessors for a token (defined here, not in the shared scan.rss
81+
// prelude, because parser.rss already declares its own `tk_col`).
82+
fn tk_col(toks: read List<Tok>, i: read Int) -> Int {
83+
if i < 0 { return 0 }
84+
if i >= List.len(list: read toks) { return 0 }
85+
let t = List.get(list: read toks, index: read i)
86+
return t.col
87+
}
88+
89+
fn tk_len(toks: read List<Tok>, i: read Int) -> Int {
90+
if i < 0 { return 0 }
91+
if i >= List.len(list: read toks) { return 0 }
92+
let t = List.get(list: read toks, index: read i)
93+
return t.len
94+
}
95+
8096
// bool -> "true"/"false" for attribute values.
8197
fn boolstr(b: read Bool) -> String {
8298
if b { return "true" }

selfhost/scan.rss

Lines changed: 0 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -609,20 +609,6 @@ fn tk_line(toks: read List<Tok>, i: read Int) -> Int {
609609
return t.line
610610
}
611611

612-
fn tk_col(toks: read List<Tok>, i: read Int) -> Int {
613-
if i < 0 { return 0 }
614-
if i >= List.len(list: read toks) { return 0 }
615-
let t = List.get(list: read toks, index: read i)
616-
return t.col
617-
}
618-
619-
fn tk_len(toks: read List<Tok>, i: read Int) -> Int {
620-
if i < 0 { return 0 }
621-
if i >= List.len(list: read toks) { return 0 }
622-
let t = List.get(list: read toks, index: read i)
623-
return t.len
624-
}
625-
626612
fn is_ident_tok(toks: read List<Tok>, i: read Int) -> Bool {
627613
let k = tk_kind(toks: read toks, i: read i)
628614
return k == TOK_IDENT || k == TOK_KEYWORD

0 commit comments

Comments
 (0)