Remove the Lezer target entirely

Monogram's grammar is non-deterministic by design (backtracking recursive-descent +
Pratt: ordered choice, not() lookahead, speculative reparse). That maps to RUNTIME
exploration, which tree-sitter's GLR absorbs directly — the derived tree-sitter
highlighter scores 95.9%, above official. It does NOT map to Lezer's BUILD-TIME LR(1)
automaton: the optional-modifier-heavy rules blow up the LR table combinatorially
(ClassMember 716 choices, buildParser hangs), and the `<` type-vs-expression ambiguity
is a genuine shift/reduce conflict with no mechanical resolution. Getting it to build
would require a hand-authored, Lezer-specific grammar — which contradicts the "one
grammar, derive everything" thesis (the official @lezer/javascript is exactly that:
hand-written, not derived).

Rather than carry a generator that produces nothing usable, drop the whole target:
- delete src/gen-lezer.ts, examples/lezer/, test/lezer-smoke.ts, the CLI emission
- drop the official-Lezer comparison from the bench (engine + README chart row) and the
  @lezer/generator + @lezer/javascript + @lezer/highlight devDependencies
- update README / ROADMAP / docs; regenerate the now 3-engine chart

tree-sitter remains the parser-based highlighting target (Neovim, GitHub, Zed, Helix);
TextMate and Monarch are the other derived outputs.

Author: johnsoncodehk

.github/workflows/ci.yml

@@ -29,7 +29,7 @@ jobs:
       - name: Generate editor artifacts (must be in sync)
         run: |
           node src/cli.ts examples/typescript.ts
-          git diff --exit-code -- examples/typescript.tmLanguage.json examples/typescript.language-configuration.json examples/typescript.monarch.json examples/typescript.cst-types.ts examples/tree-sitter examples/lezer
+          git diff --exit-code -- examples/typescript.tmLanguage.json examples/typescript.language-configuration.json examples/typescript.monarch.json examples/typescript.cst-types.ts examples/tree-sitter
 
       # Self-contained suites (each exits non-zero on failure). The conformance /
       # coverage / bench tools are excluded here: they need the external TypeScript

README.md

@@ -1,6 +1,6 @@
 # Monogram
 
-Write a language's grammar **once**. Monogram runs that single definition as a real parser, proves it against the language's official conformance suite, and then **derives a syntax highlighter** — TextMate, tree-sitter, Lezer, Monarch — from the same proven grammar. The highlighter's correctness flows *down* from a parser-verified model instead of *up* from hand-tuned regex.
+Write a language's grammar **once**. Monogram runs that single definition as a real parser, proves it against the language's official conformance suite, and then **derives a syntax highlighter** — TextMate, tree-sitter, Monarch — from the same proven grammar. The highlighter's correctness flows *down* from a parser-verified model instead of *up* from hand-tuned regex.
 
 > *mono + grammar — one grammar definition, many derived artifacts.*
 
@@ -31,7 +31,7 @@ Take `typeof x < y`. A regex highlighter has to guess whether `<` opens a generi
 
 1. **Write the grammar, then prove it.** The grammar is executable. Monogram runs it as a recursive-descent + [Pratt](https://en.wikipedia.org/wiki/Operator-precedence_parser) (operator-precedence) parser over the TypeScript conformance suite and measures *bidirectionally* — it must **accept** every input `tsc` accepts **and reject** every input `tsc` rejects. Today: **100%** of valid single-file cases parse, and **97.8%** bidirectional agreement (see [Results](#results)).
 
-2. **Derive the highlighter from the proven grammar.** The TextMate grammar — and the tree-sitter / Lezer / Monarch ones — are generated from that same parser-validated definition, never hand-written. Their correctness is underwritten by the conformance run, not by regex tuning.
+2. **Derive the highlighter from the proven grammar.** The TextMate grammar — and the tree-sitter / Monarch ones — are generated from that same parser-validated definition, never hand-written. Their correctness is underwritten by the conformance run, not by regex tuning.
 
 The remaining 2.2% bidirectional gap is **over-acceptance** — the grammar is still too permissive on some *invalid* inputs (valid-code coverage is already 100%; nothing valid is missed). Most of what's left is code `tsc`'s parser rejects via *context-sensitive* rules a context-free grammar can't express — reserved-word placement, modifier combinations like `default abstract class`, `super` type-arguments — the kind of constraint a highlighter grammar was never meant to enforce. We push as close to 100% both ways as a pure grammar can; that's the asymptote, and the highlighter rides down to it for free.
 
@@ -56,7 +56,7 @@ The valid-code/bidirectional numbers are the grammar's correctness proof; the hi
 
 ### Head-to-head: highlighter correctness across ecosystems
 
-**Auto-generated** by [`test/highlight-bench.ts`](test/highlight-bench.ts) — Monogram's *derived* TextMate highlighter against the official hand-written grammars of all three ecosystems (TextMate, tree-sitter, Lezer), each graded by the same neutral `tsc` oracle:
+**Auto-generated** by [`test/highlight-bench.ts`](test/highlight-bench.ts) — Monogram's *derived* TextMate highlighter against the official hand-written grammars of two ecosystems (TextMate, tree-sitter), each graded by the same neutral `tsc` oracle:
 
 <!-- bench:start -->
 <!-- generated by `npm run bench:readme` — do not edit by hand -->
@@ -69,15 +69,14 @@ official hand-written grammar for its ecosystem; Monogram's is *derived* from it
 ```
 TypeScript — token-family accuracy (higher = more correct)
   official tree-sitter          ████████████████████░░  92.7%
-  official Lezer                ████████████████████░░  89.5%
   Monogram (TextMate, derived)  ███████████████████░░░  87.6%
   official TextMate             █████████████████░░░░░  76.2%
 ```
 
-<sub>Graded over 102 ambiguity-rich snippets ([`test/issue-cases.ts`](test/issue-cases.ts)) against tsc's own parse tree. The three parser-based highlighters (Monogram, tree-sitter, Lezer) cluster well above the hand-written TextMate grammar — that gap is the whole point. Per-engine vocabulary→family maps (frozen, auditable): [`test/highlight-engines.ts`](test/highlight-engines.ts). JavaScript: not yet on the bench. Regenerate: `npm run bench:readme`.</sub>
+<sub>Graded over 102 ambiguity-rich snippets ([`test/issue-cases.ts`](test/issue-cases.ts)) against tsc's own parse tree. The parser-derived highlighters (Monogram, official tree-sitter) sit well above the hand-written TextMate grammar — that gap is the whole point. Per-engine vocabulary→family maps (frozen, auditable): [`test/highlight-engines.ts`](test/highlight-engines.ts). JavaScript: not yet on the bench. Regenerate: `npm run bench:readme`.</sub>
 <!-- bench:end -->
 
-> **Why the Monogram row is its *TextMate* output.** That's the one CI can rebuild on every commit. Monogram *also* derives a full **tree-sitter** grammar + query from the same definition; built locally to wasm and graded through this *same* `tsc` oracle, it scores **95.9%** — **above the official parsers** (tree-sitter 92.7%, Lezer 89.5%) and both TextMate grammars (its own 87.6%, official 76.2%). The lift is *structural*: every capture is derived from grammar shape — type references by node position (`(type (ident) @type)`), member keys from the member-name rule, type parameters from the `< … >` binder, declaration names keyword-anchored — never a hardcoded vocabulary, so it never paints a value as a type. It's opt-in (needs a local wasm build), hence out of the CI chart: `MONOGRAM_TS_WASM=… node test/highlight-bench.ts --corpus adversarial --engines`. *(There is no Monogram-**Lezer** number: the optional-modifier-heavy rules blow up Lezer's LR table generation combinatorially, and behind that the `<`/type-vs-expression ambiguity is a genuine LR(1) shift/reduce conflict. tree-sitter's GLR absorbs both; Lezer's LR(1) does not.)*
+> **Why the Monogram row is its *TextMate* output.** That's the one CI can rebuild on every commit. Monogram *also* derives a full **tree-sitter** grammar + query from the same definition; built locally to wasm and graded through this *same* `tsc` oracle, it scores **95.9%** — **above official tree-sitter** (92.7%) and both TextMate grammars (its own 87.6%, official 76.2%). The lift is *structural*: every capture is derived from grammar shape — type references by node position (`(type (ident) @type)`), member keys from the member-name rule, type parameters from the `< … >` binder, declaration names keyword-anchored — never a hardcoded vocabulary, so it never paints a value as a type. It's opt-in (needs a local wasm build), hence out of the CI chart: `MONOGRAM_TS_WASM=… node test/highlight-bench.ts --corpus adversarial --engines`.
 
 > **The other side of the ledger (honesty check).** On the *broad* TS parser-conformance corpus — not just the documented bugs — the two are now neck-and-neck: token-role accuracy is **tied at ~99%**, Monogram **leads on per-cell coverage** (100% of strict cells vs official's ~97%), and whole-file-perfect snippets are essentially level (~88% vs ~89%). The small residual is niche multiline type-annotation scoping, **not** the ambiguity class above. Clone the TS corpus to `/tmp/ts-repo` and run `node test/highlight-bench.ts` to see both corpora.
 
@@ -93,8 +92,7 @@ From one grammar definition (a small TypeScript combinator API), three outputs a
 
 And — from the same grammar — generators for the rest of the ecosystem, at varying maturity:
 
-- **tree-sitter** — `grammar.js` + a **structural** `queries/highlights.scm` + an external scanner for context-sensitive lexing. Builds end-to-end (tree-sitter's GLR absorbs the grammar; compiles to wasm) and the *derived* query scores **95.9%** through the same oracle as the chart above — **above the official parsers** (tree-sitter 92.7%, Lezer 89.5%). Opt-in (needs a local wasm build), so it stays out of the CI chart.
-- **Lezer** — a CodeMirror 6 grammar + `styleTags` + a JS tokenizer. Does **not** build: the grammar trips Lezer's **LR(1) shift/reduce wall** (TypeScript's `<` type-vs-expression ambiguity, which `tsc` resolves with semantic information LR(1) lacks). That's a ceiling of Lezer's parser class rather than a gap in the grammar — so there is no derived Lezer highlighter number.
+- **tree-sitter** — `grammar.js` + a **structural** `queries/highlights.scm` + an external scanner for context-sensitive lexing. Builds end-to-end (tree-sitter's GLR absorbs the grammar; compiles to wasm) and the *derived* query scores **95.9%** through the same oracle as the chart above — **above official tree-sitter** (92.7%). Opt-in (needs a local wasm build), so it stays out of the CI chart.
 - **Monarch** — a Monaco (web) tokenizer (functional, bounded by JS-regex limits).
 
 ## The grammar is the source of truth
@@ -161,7 +159,7 @@ A new language is **one grammar file, proven the way TypeScript is** — by its
 2. **Prove it as a parser** against the language's *own* official test suite, measured **bidirectionally** (accept what the reference accepts, reject what it rejects). A grammar is "ready" when valid-code coverage is 100% and bidirectional agreement is high — that run is the correctness proof.
 3. **Bring the reference highlighter as the baseline.** Drop in the language's existing official TextMate grammar so coverage is *measured against the thing you're replacing*, not asserted.
 
-The highlighter, lexer, and CST types fall out of step 1 automatically (the tree-sitter / Lezer / Monarch generators give you a scaffold to finish); steps 2–3 are how the result earns trust. A new-language PR is reviewed on exactly two numbers: **parser conformance** and **highlighter coverage vs the official grammar**.
+The highlighter, lexer, and CST types fall out of step 1 automatically (the tree-sitter / Monarch generators give you a scaffold to finish); steps 2–3 are how the result earns trust. A new-language PR is reviewed on exactly two numbers: **parser conformance** and **highlighter coverage vs the official grammar**.
 
 > The conformance and highlighter *harnesses* are currently TypeScript-specific — they call `tsc`'s `parseDiagnostics` and read VS Code's bundled TS grammar. A contributor adapts those harness scripts to their reference compiler's diagnostic API; the engine and generators themselves are reused unchanged.
 
@@ -222,15 +220,14 @@ examples/typescript.ts                one grammar (TypeScript combinator API)
         ├─ src/gen-tm.ts ───────────▶ typescript.tmLanguage.json            (TextMate highlighter)
         ├─ src/gen-vscode-config.ts ▶ typescript.language-configuration.json (editor behavior)
         ├─ src/gen-treesitter.ts ───▶ tree-sitter/  (grammar.js + highlights.scm + scanner.c)
-        ├─ src/gen-lezer.ts ────────▶ lezer/        (grammar + styleTags + tokenizer)
         ├─ src/gen-monarch.ts ──────▶ typescript.monarch.json
         └─ src/gen-ast-types.ts ────▶ typescript.cst-types.ts
 
 shared  src/grammar-utils.ts          structural helpers used across stages
         src/api.ts, types.ts          the grammar's combinator + type surface
 ```
 
-Every highlighter target (TextMate, tree-sitter queries, Lezer styleTags, Monarch) is produced by the *same* structural scope-inference, retargeted per format — so highlighting stays consistent across ecosystems.
+Every highlighter target (TextMate, tree-sitter queries, Monarch) is produced by the *same* structural scope-inference, retargeted per format — so highlighting stays consistent across ecosystems.
 
 - **One grammar, many derived artifacts.** `gen-lexer` builds a tokenizer from the token definitions; `gen-parser` composes it and interprets the rules into a CST; `gen-tm` reads the same rule *shapes* to derive TextMate patterns; `gen-vscode-config` derives editor config from the same tokens and `scopes`. Shared structural primitives (`grammar-utils.ts`) keep them consistent.
 - **CST, not AST.** Keeping every token (punctuation, keywords) as a node is required for the highlighter and for lossless source reconstruction — roughly 2× the nodes of an AST, by design.

ROADMAP.md

@@ -10,12 +10,11 @@ All emitted by `node src/cli.ts examples/typescript.ts`. Highlighting for every
 | CST parser | `createParser` → CST | ✅ 94.9% conformance (3584/3776) |
 | TextMate | `.tmLanguage.json` | ✅ more correct than VS Code on its own bug ledger (`test/highlight-bench.ts`) |
 | VS Code language-config | `.language-configuration.json` | ✅ comments/folding/`/**` match official |
-| tree-sitter | `grammar.js` + `queries/highlights.scm` + `scanner.c` | 🟡 first pass — external C scanner is a scaffold |
-| Lezer | grammar + `styleTags` + JS tokenizer | 🟡 first pass — Pratt→precedence may need hand-tuning |
+| tree-sitter | `grammar.js` + `queries/highlights.scm` + `scanner.c` | ✅ derived highlighter **95.9%** via the same `tsc` oracle (beats official tree-sitter); builds to wasm. Opt-in (local wasm build). |
 | Monarch | Monaco tokenizer JSON | 🟡 first pass — JS-regex bounded (still beats Monaco's capitalization heuristic) |
 | CST node types | TS discriminated union | 🟡 structural — named-field accessors need grammar field labels |
 
-Remaining: finish the tree-sitter C scanner; hand-tune Lezer operator precedence; (optional) add field labels to the grammar DSL for richer AST types.
+Remaining: (optional) add field labels to the grammar DSL for richer AST types. **Lezer was dropped** — Monogram's non-deterministic (backtracking) grammar can't be mechanically derived into Lezer's build-time LR(1) automaton (combinatorial table blowup + unresolvable `<`-ambiguity conflicts); tree-sitter's runtime GLR absorbs it instead.
 
 ## Current State (v2.5)
 

docs/upstream-issues.md

@@ -45,7 +45,7 @@ When an issue is **not** solved, it gets exactly one verdict, and **⛔ TM-impos
 1. **🔧 Backlog** — reproduce it; if a parser-derived TM pattern *can* express the fix, it's backlog. Fix it in `examples/typescript.ts`, add a case to `test/test-issues.ts`, move it to Solved.
 2. **📦 Out of scope** — the construct belongs to a dialect/region we don't generate yet (TSX, JSDoc body).
 3. **🧠 Needs semantics** — the correct scope depends on resolving a symbol's *kind* (type/value/namespace) or type relationships. See [Proof A](#proof-a). Out of scope for every grammar, ours included.
-4. **⛔ TM-impossible** — *only* when you can show **(i)** our own generated `tmLanguage.json` cannot express it (a concrete failing pattern) **and (ii)** our tree-sitter or Lezer target (real parsers) *does* get it right. See [Proof B](#proof-b). File the failing-TM + passing-parser pair as the proof. **Currently 0 issues have cleared this bar.**
+4. **⛔ TM-impossible** — *only* when you can show **(i)** our own generated `tmLanguage.json` cannot express it (a concrete failing pattern) **and (ii)** our tree-sitter target (a real parser) *does* get it right. See [Proof B](#proof-b). File the failing-TM + passing-parser pair as the proof. **Currently 0 issues have cleared this bar.**
 
 ---
 
@@ -84,7 +84,7 @@ Theming / token-color config (`#1064`, `#991`), scope-name data & spelling fixes
 **A purely syntactic highlighter — TextMate grammar *or* CST parser — cannot resolve a symbol's *kind*.** Both see only the token stream, never the resolved program. Whether `Foo` in `Foo.Bar` is a namespace or a class, whether an `import` binds a type or a value, whether `MAX` is a constant — all require name resolution / type-checking against declarations that may be in another file. Monogram's parser is a *concrete syntax tree* builder; it has no symbol table either. So semantic-coloring issues are unsolvable by **every** layer here — the job belongs to a semantic-token provider (language server). This is *not* a TextMate-specific weakness; it is the boundary of syntax highlighting itself. (Monogram's win is making the *syntactic* layer exact, so only the genuinely-semantic residue is left for the server.)
 
 ### Proof B
-**A TextMate grammar is a line-oriented pushdown machine whose only inter-line memory is a finite stack of scope contexts**; each step is a single Oniguruma match (bounded even with `\g<>` subexpression recursion, which is per-line). Where correct highlighting requires context *not* captured by the enclosing-scope stack — an unbounded lookback to a token that is not an ancestor context — no TextMate grammar can express it, the official one or ours. Monogram's **tree-sitter and Lezer** targets are real incremental parsers holding the full tree, so they resolve such cases exactly; the generated TM target (like the official) can only approximate.
+**A TextMate grammar is a line-oriented pushdown machine whose only inter-line memory is a finite stack of scope contexts**; each step is a single Oniguruma match (bounded even with `\g<>` subexpression recursion, which is per-line). Where correct highlighting requires context *not* captured by the enclosing-scope stack — an unbounded lookback to a token that is not an ancestor context — no TextMate grammar can express it, the official one or ours. Monogram's **tree-sitter** target is a real incremental parser holding the full tree, so it resolves such cases exactly; the generated TM target (like the official) can only approximate.
 
 > **But beware over-claiming this.** The 50 solved cases include heavy multiline / whitespace / nested-generic scenarios long *assumed* "impossible for TextMate" — they are not; they are TM-*expressible* once a parser tells you the right pattern. So this bucket is narrow, and we keep it empty until an issue is *proven* into it (failing generated-TM + passing parser-target). Don't reach for "TM can't" when the honest answer is "we haven't derived the pattern yet."