bench: measure Monogram's own tree-sitter through the tsc oracle

Add an opt-in `--engines` mode to highlight-bench that grades Monogram's derived
tree-sitter (loaded from a locally-built wasm via MONOGRAM_TS_WASM) with the same
family oracle the README chart already uses for the official engines. It stays out
of the auto-generated chart because building the wasm needs the wasi-sdk toolchain,
which CI does not run; the result (88.8%) is recorded as a hand-written note.

Also document the Lezer finding: Monogram's grammar trips Lezer's LR(1)
shift/reduce wall (the `<` type-vs-expression ambiguity tsc resolves with
semantics), so there is no derived Lezer highlighter number — a ceiling of Lezer's
parser class, not a gap in the grammar.

Author: johnsoncodehk

README.md

@@ -77,6 +77,8 @@ TypeScript — token-family accuracy (higher = more correct)
 <sub>Graded over 102 ambiguity-rich snippets ([`test/issue-cases.ts`](test/issue-cases.ts)) against tsc's own parse tree. The three parser-based highlighters (Monogram, tree-sitter, Lezer) cluster well above the hand-written TextMate grammar — that gap is the whole point. Per-engine vocabulary→family maps (frozen, auditable): [`test/highlight-engines.ts`](test/highlight-engines.ts). JavaScript: not yet on the bench. Regenerate: `npm run bench:readme`.</sub>
 <!-- bench:end -->
 
+> **Why the Monogram row is its *TextMate* output.** That's the one CI can rebuild on every commit. Monogram *also* derives a full **tree-sitter** grammar + query from the same definition; built locally to wasm and graded through this *same* `tsc` oracle, it scores **88.8%** — above both TextMate grammars (its own 87.6%, official 76.2%) and within range of the official parsers (tree-sitter 92.7%, Lezer 89.5%). The lift is *structural*: type references are captured by node position — `(type (ident) @type)` — not a hardcoded builtin list, so it never paints a value as a type. It's opt-in (needs a local wasm build), hence out of the CI chart: `MONOGRAM_TS_WASM=… node test/highlight-bench.ts --corpus adversarial --engines`. *(There is no Monogram-**Lezer** number: the grammar is expressive enough to trip Lezer's LR(1) shift/reduce wall — it parses under tree-sitter's GLR but won't build under Lezer.)*
+
 > **The other side of the ledger (honesty check).** On the *broad* TS parser-conformance corpus — not just the documented bugs — the two are now neck-and-neck: token-role accuracy is **tied at ~99%**, Monogram **leads on per-cell coverage** (100% of strict cells vs official's ~97%), and whole-file-perfect snippets are essentially level (~88% vs ~89%). The small residual is niche multiline type-annotation scoping, **not** the ambiguity class above. Clone the TS corpus to `/tmp/ts-repo` and run `node test/highlight-bench.ts` to see both corpora.
 
 ## What you get
@@ -89,10 +91,10 @@ From one grammar definition (a small TypeScript combinator API), three outputs a
 - **A VS Code language configuration** — `language-configuration.json` (comments, bracket pairs, auto-close/surround, folding) derived from the same tokens.
 - **CST node types** — a TypeScript discriminated union (keyed by rule) for typed tree consumers.
 
-And — from the same grammar — **first-pass generators** for the rest of the ecosystem. These are structurally valid and a strong starting point, but each carries known incomplete sections (marked in the output) that need hand-tuning before production:
+And — from the same grammar — generators for the rest of the ecosystem, at varying maturity:
 
-- **tree-sitter** — `grammar.js` + `queries/highlights.scm` + an external-scanner scaffold (template state machine stubbed).
-- **Lezer** — a CodeMirror 6 grammar + `styleTags` + a JS tokenizer (a handful of token regexes can't be expressed in Lezer's static model; marked `INCOMPLETE`).
+- **tree-sitter** — `grammar.js` + a **structural** `queries/highlights.scm` + an external scanner for context-sensitive lexing. Builds end-to-end (tree-sitter's GLR absorbs the grammar; compiles to wasm) and the *derived* query scores **88.8%** through the same oracle as the chart above — above both TextMate grammars, just under the official parsers. Opt-in (needs a local wasm build), so it stays out of the CI chart.
+- **Lezer** — a CodeMirror 6 grammar + `styleTags` + a JS tokenizer. Does **not** build: the grammar trips Lezer's **LR(1) shift/reduce wall** (TypeScript's `<` type-vs-expression ambiguity, which `tsc` resolves with semantic information LR(1) lacks). That's a ceiling of Lezer's parser class rather than a gap in the grammar — so there is no derived Lezer highlighter number.
 - **Monarch** — a Monaco (web) tokenizer (functional, bounded by JS-regex limits).
 
 ## The grammar is the source of truth

test/highlight-bench.ts

@@ -37,7 +37,7 @@ import onig from 'vscode-oniguruma';
 import { R, ROLE_SPEC, gradeScope, isCorrect, normScope, roleFamily, acceptableFamilies } from './scope-roles.ts';
 import type { RoleName, Verdict, Family } from './scope-roles.ts';
 import { tests as issueTests, multiLineTests as issueMultiLine } from './issue-cases.ts';
-import { scopeFamily, lezerFamilies, treesitterFamilies, loadTreeSitter, familyAt } from './highlight-engines.ts';
+import { scopeFamily, lezerFamilies, treesitterFamilies, loadTreeSitter, familyAt, loadMonogramTreeSitter, monogramTreesitterFamilies } from './highlight-engines.ts';
 import type { Span } from './highlight-engines.ts';
 
 const normScopeShort = (s: string): string => (s ? normScope(s) : '(none)');
@@ -589,13 +589,16 @@ const tmSpans = (grammar: vsctm.IGrammar, text: string): Span[] =>
   tmTokenize(grammar, text).map((t) => ({ start: t.start, end: t.end, family: scopeFamily(t.scope) }));
 
 // Grade each engine's token-FAMILY classification against the tsc oracle over `inputs`.
-async function engineFamilyScores(inputs: { name: string; text: string }[], tsOk: boolean): Promise<EngineScore[]> {
+// mtsOk = Monogram's own compiled tree-sitter is available (opt-in via MONOGRAM_TS_WASM;
+// kept OUT of the auto-chart since CI can't build that wasm — surfaced only by --engines).
+async function engineFamilyScores(inputs: { name: string; text: string }[], tsOk: boolean, mtsOk = false): Promise<EngineScore[]> {
   const engines: { name: string; spans: (t: string) => Span[] }[] = [
     { name: 'Monogram (TextMate, derived)', spans: (t) => tmSpans(monogramGrammar, t) },
     { name: 'official TextMate', spans: (t) => tmSpans(officialGrammar, t) },
     { name: 'official Lezer', spans: lezerFamilies },
   ];
   if (tsOk) engines.splice(2, 0, { name: 'official tree-sitter', spans: treesitterFamilies });
+  if (mtsOk) engines.push({ name: 'Monogram (tree-sitter, derived)', spans: monogramTreesitterFamilies });
 
   const acc: Record<string, { correct: number; total: number }> = {};
   for (const e of engines) acc[e.name] = { correct: 0, total: 0 };
@@ -725,10 +728,30 @@ if (WRITE_README) {
     ...issueMultiLine.map((t) => ({ name: t.label, text: t.lines.join('\n') })),
   ];
   const tsOk = await loadTreeSitter();
-  const scores = await engineFamilyScores(advInputs, tsOk);
+  const scores = await engineFamilyScores(advInputs, tsOk); // auto-chart: 4 engines (CI-reproducible)
   const graded = advInputs.filter((i) => {
     const sf = ts.createSourceFile('c.ts', i.text, ts.ScriptTarget.Latest, true, ts.ScriptKind.TS);
     return ((sf as any).parseDiagnostics?.length ?? 0) === 0;
   }).length;
   writeReadmeBlock(buildBenchMarkdown(scores, graded, tsOk));
 }
+
+// --engines: print per-engine family accuracy to the terminal (NOT the README).
+// Set MONOGRAM_TS_WASM (+ MONOGRAM_TS_QUERY) to also grade Monogram's OWN compiled
+// tree-sitter — the opt-in measurement that needs a locally-built wasm.
+if (argv.includes('--engines')) {
+  const advInputs = [
+    ...issueTests.map((t) => ({ name: t.label, text: t.input })),
+    ...issueMultiLine.map((t) => ({ name: t.label, text: t.lines.join('\n') })),
+  ];
+  const tsOk = await loadTreeSitter();
+  const mtsWasm = process.env.MONOGRAM_TS_WASM;
+  const mtsOk = mtsWasm
+    ? await loadMonogramTreeSitter(mtsWasm, process.env.MONOGRAM_TS_QUERY ?? 'examples/tree-sitter/typescript/queries/highlights.scm')
+    : false;
+  const scores = await engineFamilyScores(advInputs, tsOk, mtsOk);
+  console.log('\n── token-family accuracy vs tsc oracle (issue-cases corpus) ──');
+  for (const s of [...scores].sort((a, b) => b.correct / b.total - a.correct / a.total)) {
+    console.log(`  ${s.name.padEnd(32)} ${((s.correct / s.total) * 100).toFixed(1)}%  (${s.correct}/${s.total})`);
+  }
+}

test/highlight-engines.ts

@@ -127,14 +127,13 @@ export async function loadTreeSitter(): Promise<boolean> {
   }
 }
 
-export function treesitterFamilies(code: string): Span[] {
-  if (!tsParser || !tsQuery) throw new Error('call loadTreeSitter() first');
-  const tree = tsParser.parse(code);
-  // Per captured node keep the family from the HIGHEST pattern index — tree-sitter
-  // highlighting is "last matching pattern wins", which is why the TS additions'
-  // capitalized-identifier `@type` rule overrides the ecma base `@variable`.
+// Shared capture→family reduction. Per captured node keep the family from the
+// HIGHEST pattern index — tree-sitter highlighting is "last matching pattern wins"
+// (e.g. the TS additions' capitalized-identifier `@type` overrides the ecma base `@variable`).
+function tsFamiliesWith(parser: any, query: any, code: string): Span[] {
+  const tree = parser.parse(code);
   const best = new Map<string, { start: number; end: number; family: Family; pat: number }>();
-  for (const m of tsQuery.matches(tree.rootNode)) {
+  for (const m of query.matches(tree.rootNode)) {
     const pat = m.patternIndex ?? m.pattern ?? 0;
     for (const c of m.captures) {
       const fam = tsCaptureToFamily(c.name);
@@ -148,6 +147,36 @@ export function treesitterFamilies(code: string): Span[] {
   return [...best.values()].sort((a, b) => a.start - b.start || a.end - b.end);
 }
 
+export function treesitterFamilies(code: string): Span[] {
+  if (!tsParser || !tsQuery) throw new Error('call loadTreeSitter() first');
+  return tsFamiliesWith(tsParser, tsQuery, code);
+}
+
+// Monogram's OWN generated tree-sitter (compiled from examples/tree-sitter, loaded
+// from a prebuilt wasm). Gated behind explicit paths because building that wasm needs
+// the wasi-sdk toolchain — not something CI does — so it's a local/opt-in measurement.
+let mtsParser: any = null;
+let mtsQuery: any = null;
+export async function loadMonogramTreeSitter(wasmPath: string, queryPath: string): Promise<boolean> {
+  if (mtsParser) return true;
+  try {
+    const { Parser, Language, Query } = await import('web-tree-sitter');
+    await Parser.init();
+    const lang = await Language.load(wasmPath);
+    mtsParser = new Parser();
+    mtsParser.setLanguage(lang);
+    mtsQuery = new Query(lang, readFileSync(queryPath, 'utf8'));
+    return true;
+  } catch (e) {
+    console.error('monogram tree-sitter init failed:', (e as Error).message);
+    return false;
+  }
+}
+export function monogramTreesitterFamilies(code: string): Span[] {
+  if (!mtsParser || !mtsQuery) throw new Error('call loadMonogramTreeSitter() first');
+  return tsFamiliesWith(mtsParser, mtsQuery, code);
+}
+
 // ─── self-test: dump families for a diagnostic snippet across all engines ───────
 if (import.meta.url === `file://${process.argv[1]}`) {
   const code = 'const x: NS.Foo = bar(Baz); function g(p: number){ return p; } class C { m(): void {} } /* c */ const re = /ab/g;';