tree-sitter: derive structural member/type-param/property captures (88.8% -> 95.3%)

Extend the highlight-query generator to mirror more of gen-tm's per-position scope
inference. Every capture is derived from grammar SHAPE — never a hardcoded rule name:

- type parameters: identifiers inside a `< … >` binder rule -> @type
- member keys: the property-name rule (a pure name-form alternation) -> @Property
- type-object keys: the member-list element inside a type-context `{ … }` -> @Property
- property access: the `expr (expr) ident` tail, when the grammar has `.` access -> @Property
- declaration names, keyword-anchored: one decl rule is shared across kinds, so anchor
  on the keyword token (`class`/`interface`/`type`/…) to give each kind its own capture
  and to reach names with no `name` field (a bare type-alias name)
- enum-like value members: a rule whose every branch is `[nameRule, opt(…)]` overrides
  the member-key -> @Property capture back to @variable (enum members are values)

Also skip `not` lookaheads between a keyword and its name in collectNameFields (a
type-alias parses as `type <not …> Ident`), which is what surfaces the alias name.

Graded through the same tsc oracle as the README bench, the derived tree-sitter
highlighter rises from 88.8% to 95.3% — above the official parsers (tree-sitter 92.7%,
Lezer 89.5%). The detectors stay grammar-agnostic (test/agnostic.ts: 5/5).

Add a `--miss-mts` bench diagnostic (misses grouped by role, flagging which ones the
official grammar gets right) that drove the work. Regenerate TS + JS artifacts.

Author: johnsoncodehk

README.md

@@ -77,7 +77,7 @@ TypeScript — token-family accuracy (higher = more correct)
 <sub>Graded over 102 ambiguity-rich snippets ([`test/issue-cases.ts`](test/issue-cases.ts)) against tsc's own parse tree. The three parser-based highlighters (Monogram, tree-sitter, Lezer) cluster well above the hand-written TextMate grammar — that gap is the whole point. Per-engine vocabulary→family maps (frozen, auditable): [`test/highlight-engines.ts`](test/highlight-engines.ts). JavaScript: not yet on the bench. Regenerate: `npm run bench:readme`.</sub>
 <!-- bench:end -->
 
-> **Why the Monogram row is its *TextMate* output.** That's the one CI can rebuild on every commit. Monogram *also* derives a full **tree-sitter** grammar + query from the same definition; built locally to wasm and graded through this *same* `tsc` oracle, it scores **88.8%** — above both TextMate grammars (its own 87.6%, official 76.2%) and within range of the official parsers (tree-sitter 92.7%, Lezer 89.5%). The lift is *structural*: type references are captured by node position — `(type (ident) @type)` — not a hardcoded builtin list, so it never paints a value as a type. It's opt-in (needs a local wasm build), hence out of the CI chart: `MONOGRAM_TS_WASM=… node test/highlight-bench.ts --corpus adversarial --engines`. *(There is no Monogram-**Lezer** number: the grammar is expressive enough to trip Lezer's LR(1) shift/reduce wall — it parses under tree-sitter's GLR but won't build under Lezer.)*
+> **Why the Monogram row is its *TextMate* output.** That's the one CI can rebuild on every commit. Monogram *also* derives a full **tree-sitter** grammar + query from the same definition; built locally to wasm and graded through this *same* `tsc` oracle, it scores **95.3%** — **above the official parsers** (tree-sitter 92.7%, Lezer 89.5%) and both TextMate grammars (its own 87.6%, official 76.2%). The lift is *structural*: every capture is derived from grammar shape — type references by node position (`(type (ident) @type)`), member keys from the member-name rule, type parameters from the `< … >` binder, declaration names keyword-anchored — never a hardcoded vocabulary, so it never paints a value as a type. It's opt-in (needs a local wasm build), hence out of the CI chart: `MONOGRAM_TS_WASM=… node test/highlight-bench.ts --corpus adversarial --engines`. *(There is no Monogram-**Lezer** number: the grammar is expressive enough to trip Lezer's LR(1) shift/reduce wall — it parses under tree-sitter's GLR but won't build under Lezer.)*
 
 > **The other side of the ledger (honesty check).** On the *broad* TS parser-conformance corpus — not just the documented bugs — the two are now neck-and-neck: token-role accuracy is **tied at ~99%**, Monogram **leads on per-cell coverage** (100% of strict cells vs official's ~97%), and whole-file-perfect snippets are essentially level (~88% vs ~89%). The small residual is niche multiline type-annotation scoping, **not** the ambiguity class above. Clone the TS corpus to `/tmp/ts-repo` and run `node test/highlight-bench.ts` to see both corpora.
 
@@ -93,7 +93,7 @@ From one grammar definition (a small TypeScript combinator API), three outputs a
 
 And — from the same grammar — generators for the rest of the ecosystem, at varying maturity:
 
-- **tree-sitter** — `grammar.js` + a **structural** `queries/highlights.scm` + an external scanner for context-sensitive lexing. Builds end-to-end (tree-sitter's GLR absorbs the grammar; compiles to wasm) and the *derived* query scores **88.8%** through the same oracle as the chart above — above both TextMate grammars, just under the official parsers. Opt-in (needs a local wasm build), so it stays out of the CI chart.
+- **tree-sitter** — `grammar.js` + a **structural** `queries/highlights.scm` + an external scanner for context-sensitive lexing. Builds end-to-end (tree-sitter's GLR absorbs the grammar; compiles to wasm) and the *derived* query scores **95.3%** through the same oracle as the chart above — **above the official parsers** (tree-sitter 92.7%, Lezer 89.5%). Opt-in (needs a local wasm build), so it stays out of the CI chart.
 - **Lezer** — a CodeMirror 6 grammar + `styleTags` + a JS tokenizer. Does **not** build: the grammar trips Lezer's **LR(1) shift/reduce wall** (TypeScript's `<` type-vs-expression ambiguity, which `tsc` resolves with semantic information LR(1) lacks). That's a ceiling of Lezer's parser class rather than a gap in the grammar — so there is no derived Lezer highlighter number.
 - **Monarch** — a Monaco (web) tokenizer (functional, bounded by JS-regex limits).
 

examples/tree-sitter/javascript/queries/highlights.scm

@@ -15,6 +15,16 @@
 (decl name: (ident) @function)
 (decl name: (ident) @type)
 
+;; Declaration names, keyword-anchored (disambiguates shared decl rules).
+(expr "class" (ident) @type)
+(expr "function" (ident) @function)
+(decl "function" (ident) @function)
+(decl "class" (ident) @type)
+
+;; Structural member / type-param / property-access captures.
+(member_name (ident) @property)
+(expr (expr) (ident) @property)
+
 ;; Literal token nodes.
 (shebang) @comment
 (jsdoc) @comment.documentation

examples/tree-sitter/typescript/grammar.js

@@ -205,7 +205,7 @@ module.exports = grammar({
 
     type_param: $ => choice(seq(repeat1(choice("const", "in", "out", "public", "private", "protected", "readonly")), $.ident, optional(seq("extends", $.type)), optional(seq("=", $.type))), seq(choice("const", "in", "out", "public", "private", "protected", "readonly"), choice($.ident, "in", "out"), optional(seq("extends", $.type)), optional(seq("=", $.type))), seq(choice($.ident, "in", "out"), optional(seq("extends", $.type)), optional(seq("=", $.type)))),
 
-    decl: $ => choice(seq(optional("async"), "function", optional("*"), field('name', $.ident), optional($.type_params), "(", optional(seq($.param, repeat(seq(",", $.param)), optional(","))), ")", optional(seq(":", $.type)), choice($.block, optional(";"))), seq("interface", field('name', $.ident), optional($.type_params), optional(seq("extends", optional(seq($.type, repeat(seq(",", $.type)), optional(","))))), "{", repeat(seq($.interface_member, optional(choice(";", ",")))), "}"), seq("type", $.ident, optional($.type_params), "=", $.type, optional(";")), seq(repeat($.decorator_expr), optional("abstract"), "class", field('name', $.ident), optional($.type_params), optional(seq("extends", $.class_heritage)), optional(seq("implements", optional(seq($.type, repeat(seq(",", $.type)), optional(","))))), "{", repeat($.class_member), "}"), seq("enum", field('name', $.ident), "{", optional(seq($.enum_member, repeat(seq(",", $.enum_member)), optional(","))), "}"), seq("declare", "function", optional("*"), field('name', $.ident), optional($.type_params), "(", optional(seq($.param, repeat(seq(",", $.param)), optional(","))), ")", optional(seq(":", $.type)), optional(";")), seq("declare", choice($.decl, $.stmt)), seq("namespace", field('name', $.ident), repeat(seq(".", $.ident)), "{", repeat($.stmt), "}"), seq("module", choice(seq($.ident, repeat(seq(".", $.ident))), $.string), "{", repeat($.stmt), "}"), seq("export", choice($.decl, $.stmt)), seq("export", "default", choice(seq(optional("async"), "function", optional("*"), optional(field('name', $.ident)), optional($.type_params), "(", optional(seq($.param, repeat(seq(",", $.param)), optional(","))), ")", optional(seq(":", $.type)), choice($.block, optional(";"))), seq("abstract", "class", field('name', $.ident), optional($.type_params), optional(seq("extends", $.class_heritage)), optional(seq("implements", optional(seq($.type, repeat(seq(",", $.type)), optional(","))))), "{", repeat($.class_member), "}"), seq("abstract", "class", optional($.type_params), optional(seq("extends", $.class_heritage)), optional(seq("implements", optional(seq($.type, repeat(seq(",", $.type)), optional(","))))), "{", repeat($.class_member), "}"), seq($.expr, optional(";")))), seq("export", "*", choice(seq("from", $.string, optional(";")), seq("as", $.ident, "from", $.string, optional(";")))), seq("export", "{", optional(seq($.import_specifier, repeat(seq(",", $.import_specifier)), optional(","))), "}", optional(seq("from", $.string)), optional(";")), seq("export", "=", $.expr, optional(";")), seq("export", "type", "{", optional(seq($.import_specifier, repeat(seq(",", $.import_specifier)), optional(","))), "}", optional(seq("from", $.string)), optional(";")), seq("const", "enum", field('name', $.ident), "{", optional(seq($.enum_member, repeat(seq(",", $.enum_member)), optional(","))), "}"), seq("import", choice(seq($.import_clause, "from", $.string, optional(";")), seq("type", $.import_clause, "from", $.string, optional(";")), seq($.ident, "=", $.expr, optional(";")), seq($.string, optional(";")))), seq(repeat($.decorator_expr), "export", choice($.decl, $.stmt))),
+    decl: $ => choice(seq(optional("async"), "function", optional("*"), field('name', $.ident), optional($.type_params), "(", optional(seq($.param, repeat(seq(",", $.param)), optional(","))), ")", optional(seq(":", $.type)), choice($.block, optional(";"))), seq("interface", field('name', $.ident), optional($.type_params), optional(seq("extends", optional(seq($.type, repeat(seq(",", $.type)), optional(","))))), "{", repeat(seq($.interface_member, optional(choice(";", ",")))), "}"), seq("type", field('name', $.ident), optional($.type_params), "=", $.type, optional(";")), seq(repeat($.decorator_expr), optional("abstract"), "class", field('name', $.ident), optional($.type_params), optional(seq("extends", $.class_heritage)), optional(seq("implements", optional(seq($.type, repeat(seq(",", $.type)), optional(","))))), "{", repeat($.class_member), "}"), seq("enum", field('name', $.ident), "{", optional(seq($.enum_member, repeat(seq(",", $.enum_member)), optional(","))), "}"), seq("declare", "function", optional("*"), field('name', $.ident), optional($.type_params), "(", optional(seq($.param, repeat(seq(",", $.param)), optional(","))), ")", optional(seq(":", $.type)), optional(";")), seq("declare", choice($.decl, $.stmt)), seq("namespace", field('name', $.ident), repeat(seq(".", $.ident)), "{", repeat($.stmt), "}"), seq("module", choice(seq($.ident, repeat(seq(".", $.ident))), $.string), "{", repeat($.stmt), "}"), seq("export", choice($.decl, $.stmt)), seq("export", "default", choice(seq(optional("async"), "function", optional("*"), optional(field('name', $.ident)), optional($.type_params), "(", optional(seq($.param, repeat(seq(",", $.param)), optional(","))), ")", optional(seq(":", $.type)), choice($.block, optional(";"))), seq("abstract", "class", field('name', $.ident), optional($.type_params), optional(seq("extends", $.class_heritage)), optional(seq("implements", optional(seq($.type, repeat(seq(",", $.type)), optional(","))))), "{", repeat($.class_member), "}"), seq("abstract", "class", optional($.type_params), optional(seq("extends", $.class_heritage)), optional(seq("implements", optional(seq($.type, repeat(seq(",", $.type)), optional(","))))), "{", repeat($.class_member), "}"), seq($.expr, optional(";")))), seq("export", "*", choice(seq("from", $.string, optional(";")), seq("as", $.ident, "from", $.string, optional(";")))), seq("export", "{", optional(seq($.import_specifier, repeat(seq(",", $.import_specifier)), optional(","))), "}", optional(seq("from", $.string)), optional(";")), seq("export", "=", $.expr, optional(";")), seq("export", "type", "{", optional(seq($.import_specifier, repeat(seq(",", $.import_specifier)), optional(","))), "}", optional(seq("from", $.string)), optional(";")), seq("const", "enum", field('name', $.ident), "{", optional(seq($.enum_member, repeat(seq(",", $.enum_member)), optional(","))), "}"), seq("import", choice(seq($.import_clause, "from", $.string, optional(";")), seq("type", $.import_clause, "from", $.string, optional(";")), seq($.ident, "=", $.expr, optional(";")), seq($.string, optional(";")))), seq(repeat($.decorator_expr), "export", choice($.decl, $.stmt))),
 
     interface_member: $ => choice(seq(optional("new"), optional($.type_params), "(", optional(seq($.param, repeat(seq(",", $.param)), optional(","))), ")", optional(seq(":", $.type))), seq(choice("get", "set"), $.member_name, "(", optional(seq($.param, repeat(seq(",", $.param)), optional(","))), ")", optional(seq(":", $.type))), seq(optional("static"), optional(choice("+", "-")), optional("readonly"), "[", $.ident, "in", $.type, optional(seq("as", $.type)), "]", optional(choice("+", "-")), optional("?"), ":", $.type), seq("readonly", $.member_name, optional("?"), ":", $.type), seq($.member_name, optional("?"), choice(seq(optional($.type_params), "(", optional(seq($.param, repeat(seq(",", $.param)), optional(","))), ")", optional(seq(":", $.type))), optional(seq(":", $.type)))), seq(optional("static"), optional("readonly"), "[", optional(seq($.param, repeat(seq(",", $.param)), optional(","))), "]", optional(seq(":", $.type)))),
 

examples/tree-sitter/typescript/queries/highlights.scm

@@ -15,9 +15,28 @@
 (decl name: (ident) @function)
 (decl name: (ident) @type)
 
-;; Type-reference identifiers (inside a type node) → @type.
+;; Declaration names, keyword-anchored (disambiguates shared decl rules).
+(expr "class" (ident) @type)
+(expr "function" (ident) @function)
+(decl "function" (ident) @function)
+(decl "interface" (ident) @type)
+(decl "type" (ident) @type)
+(decl "class" (ident) @type)
+(decl "enum" (ident) @type)
+(decl "namespace" (ident) @type)
+
+;; Type-reference identifiers (inside a type node) -> @type.
 (type (ident) @type)
 
+;; Structural member / type-param / property-access captures.
+(type_param (ident) @type)
+(member_name (ident) @property)
+(type_member (ident) @property)
+(expr (expr) (ident) @property)
+
+;; Enum-like value members (override member-key, which would say @property).
+(enum_member (member_name (ident) @variable))
+
 ;; Literal token nodes.
 (shebang) @comment
 (jsdoc) @comment.documentation