Fix variance type-param regression; regenerate artifacts (3602/3776)

A type parameter named `in`/`out` preceded by a variance modifier
(`type T1<in in>`, `type T2<out out>`) failed to parse: the greedy
`many1(mod)` consumed both keywords, leaving no name and no backtrack.
TS treats the last `in`/`out` before the terminator as the name.

Add a single-modifier `[mod, name]` branch where the name may itself be
a contextual variance keyword, and let longest-match arbitrate against
the modifier-soup and bare-name branches. Verified no regressions across
the conformance suite.

Regenerate all derived targets (TextMate, tree-sitter, Lezer, Monarch)
from the merged grammar and update README conformance figures.

Author: johnsoncodehk

README.md

@@ -10,7 +10,7 @@ A TextMate grammar is a pile of regexes guessing at a language's structure. It's
 
 Monogram inverts the dependency:
 
-1. **Write the grammar, then prove it.** The grammar is executable. Monogram runs it as a recursive-descent + Pratt parser over the TypeScript conformance suite. Today it parses **95.0%** (3589 / 3776 files); the goal is **100% coverage of the official parser** — at which point the grammar is a *verified, complete model* of the language's syntax, not an approximation.
+1. **Write the grammar, then prove it.** The grammar is executable. Monogram runs it as a recursive-descent + Pratt parser over the TypeScript conformance suite. Today it parses **95.4%** (3602 / 3776 files); the goal is **100% coverage of the official parser** — at which point the grammar is a *verified, complete model* of the language's syntax, not an approximation.
 
 2. **Derive the highlighter from the proven grammar.** The TextMate grammar is generated from that same parser-validated grammar — never hand-written. Its correctness is underwritten by the parser conformance run, not by regex tuning.
 
@@ -37,7 +37,7 @@ And — from the same grammar — first-pass generators for the rest of the edit
 Validated on TypeScript (grammar: [`examples/typescript.ts`](examples/typescript.ts), 537 lines):
 
 ```
-Parser conformance   95.0%   3589 / 3776 files from the TypeScript conformance suite   (goal: 100%)
+Parser conformance   95.4%   3602 / 3776 files from the TypeScript conformance suite   (goal: 100%)
 Highlighter          99.3%   589 / 593 tokens match VS Code's official grammar
 Generated grammar    42 KB   vs the official hand-written 226 KB
 Engine               language-agnostic — no TypeScript-specific code
@@ -184,4 +184,4 @@ Every highlighter target (TextMate, tree-sitter queries, Lezer styleTags, Monarc
 | ANTLR | yes | — | — |
 | Langium | yes | Monarch (separate) | — |
 | ungrammar | AST types | — | — |
-| **Monogram** | **CST (95.0% → 100%)** | **auto-derived (99.3%)** | **yes** |
+| **Monogram** | **CST (95.4% → 100%)** | **auto-derived (99.3%)** | **yes** |

examples/lezer/tokens.js

@@ -9,7 +9,7 @@
 //
 // This module must also supply these tokens that exceeded Lezer's @tokens syntax
 // (straightforward longest-match scanners — INCOMPLETE, regex listed for porting):
-//   JSDoc: /\/\*\*[\s\S]*?\*\//  (skip)
+//   JSDoc: /\/\*\*(?!\/)[\s\S]*?\*\//  (skip)
 //   BlockComment: /\/\*[\s\S]*?\*\//  (skip)
 //   Ident: /(?:[a-zA-Z_$]|\\u[0-9a-fA-F]{4}|\\u\{[0-9a-fA-F]+\})(?:[a-zA-Z0-9_$]|\\u[0-9a-fA-F]{4}|\\u\{[0-9a-fA-F]+\})*/
 //   Number: /[0-9]+(_[0-9]+)*(?:\.[0-9]*(_[0-9]+)*)?(?:[eE][+-]?[0-9]+)?/

examples/lezer/typescript.grammar

@@ -27,7 +27,7 @@ Type {
   (Ident (kw_is Type)? | Type AngleL (Type (Comma Type)*)? AngleR | Type BracketL BracketR | Type Op_7c Type | Type Op_26 Type | Op_7c Type | Op_26 Type | kw_keyof Type | kw_typeof TypeofRef | kw_readonly Type | ParenL Type ParenR | TypeParams? ParenL (Param (Comma Param)*)? ParenR Arrow Type | kw_abstract? kw_new TypeParams? ParenL (Param (Comma Param)*)? ParenR Arrow Type | BracketL (Spread? (Ident Question? Colon)? Spread? Type Question? Comma?)* BracketR | BraceL (TypeMember (Semi | Comma)?)* BraceR | kw_asserts Ident (kw_is Type)? | Type kw_extends Type Question Type Colon Type | kw_infer Ident (kw_extends Type (Question Type Colon Type)?)? | String | Number | HexNumber | OctalNumber | BinaryNumber | BigInt | Op_2d (Number | BigInt) | kw_true | kw_false | kw_null | kw_undefined | kw_void | kw_this | kw_unique kw_symbol | kw_import ParenL Type ParenR | Template | Type BracketL Type BracketR | Type Dot Ident)
 }
 TypeMember {
-  (kw_new? TypeParams? ParenL (Param (Comma Param)*)? ParenR (Colon Type)? | (Op_2b | Op_2d)? kw_readonly? BracketL (Ident (kw_in Type (kw_as Type)? BracketR (Op_2b | Op_2d)? Question? Colon Type | Colon Type BracketR (Colon Type)?) | Expr BracketR Question? (TypeParams? ParenL (Param (Comma Param)*)? ParenR (Colon Type)? | (Colon Type)?)) | kw_readonly Ident Question? Colon Type | (Ident | Number | String | PrivateField) Question? (TypeParams? ParenL (Param (Comma Param)*)? ParenR (Colon Type)? | (Colon Type)?))
+  (kw_new? TypeParams? ParenL (Param (Comma Param)*)? ParenR (Colon Type)? | (Op_2b | Op_2d)? kw_readonly? BracketL (Ident (kw_in Type (kw_as Type)? BracketR (Op_2b | Op_2d)? Question? Colon Type | Colon Type BracketR (Colon Type)?) | Expr BracketR Question? (TypeParams? ParenL (Param (Comma Param)*)? ParenR (Colon Type)? | (Colon Type)?) | BracketR (Colon Type)?) | kw_readonly Ident Question? Colon Type | (Ident | Number | String | PrivateField) Question? (TypeParams? ParenL (Param (Comma Param)*)? ParenR (Colon Type)? | (Colon Type)?))
 }
 DecoratorExpr {
   Decorator (ParenL (Expr (Comma Expr)*)? ParenR | AngleL (Type (Comma Type)*)? AngleR ParenL (Expr (Comma Expr)*)? ParenR)?
@@ -48,7 +48,7 @@ Expr {
     kw_this |
     kw_super |
     Spread Expr |
-    Expr AngleL (Type (Comma Type)*)? AngleR (ParenL (Expr (Comma Expr)*)? ParenR | Template)? |
+    Expr AngleL (Type (Comma Type)*)? AngleR (ParenL (Expr (Comma Expr)*)? ParenR | Template | ) |
     Expr ParenL (Expr (Comma Expr)*)? ParenR |
     Expr Dot (Ident | PrivateField) |
     Expr QuestionDot (Ident | PrivateField | ParenL (Expr (Comma Expr)*)? ParenR | BracketL Expr BracketR | Template) |
@@ -145,7 +145,7 @@ TypeParams {
   AngleL (TypeParam (Comma TypeParam)*)? AngleR
 }
 TypeParam {
-  ((kw_const | kw_in | kw_out | kw_public | kw_private | kw_protected | kw_readonly)+ Ident (kw_extends Type)? (Op_3d Type)? | Ident (kw_extends Type)? (Op_3d Type)?)
+  ((kw_const | kw_in | kw_out | kw_public | kw_private | kw_protected | kw_readonly)+ Ident (kw_extends Type)? (Op_3d Type)? | (kw_const | kw_in | kw_out | kw_public | kw_private | kw_protected | kw_readonly) (Ident | kw_in | kw_out) (kw_extends Type)? (Op_3d Type)? | (Ident | kw_in | kw_out) (kw_extends Type)? (Op_3d Type)?)
 }
 Decl {
   (kw_async? kw_function Op_2a? Ident TypeParams? ParenL (Param (Comma Param)*)? ParenR (Colon Type)? (Block | Semi?) | kw_interface Ident TypeParams? (kw_extends (Type (Comma Type)*)?)? BraceL (InterfaceMember (Semi | Comma)?)* BraceR | kw_type Ident TypeParams? Op_3d Type Semi? | DecoratorExpr* kw_abstract? kw_class Ident TypeParams? (kw_extends ClassHeritage)? (kw_implements (Type (Comma Type)*)?)? BraceL ClassMember* BraceR | kw_enum Ident BraceL (EnumMember (Comma EnumMember)*)? BraceR | kw_declare kw_function Op_2a? Ident TypeParams? ParenL (Param (Comma Param)*)? ParenR (Colon Type)? Semi? | kw_declare (Decl | Stmt) | kw_namespace Ident BraceL Stmt* BraceR | kw_module (String | Ident) BraceL Stmt* BraceR | kw_export (Decl | Stmt) | kw_export kw_default (kw_async? kw_function Op_2a? Ident? TypeParams? ParenL (Param (Comma Param)*)? ParenR (Colon Type)? (Block | Semi?) | kw_abstract kw_class Ident TypeParams? (kw_extends ClassHeritage)? (kw_implements (Type (Comma Type)*)?)? BraceL ClassMember* BraceR | kw_abstract kw_class TypeParams? (kw_extends ClassHeritage)? (kw_implements (Type (Comma Type)*)?)? BraceL ClassMember* BraceR | Expr Semi?) | kw_export Op_2a (kw_from String Semi? | kw_as Ident kw_from String Semi?) | kw_export BraceL (ImportSpecifier (Comma ImportSpecifier)*)? BraceR (kw_from String)? Semi? | kw_export Op_3d Expr Semi? | kw_export kw_type BraceL (ImportSpecifier (Comma ImportSpecifier)*)? BraceR (kw_from String)? Semi? | kw_const kw_enum Ident BraceL (EnumMember (Comma EnumMember)*)? BraceR | kw_import (ImportClause kw_from String Semi? | kw_type ImportClause kw_from String Semi? | Ident Op_3d Expr Semi? | String Semi?) | DecoratorExpr* kw_export (Decl | Stmt))
@@ -248,7 +248,7 @@ Program {
 
 @tokens {
   Shebang { "#!"![\n]* }
-  // INCOMPLETE: JSDoc block span "\\/\\*\\*"…"\\*\\/" → external tokenizer.
+  // INCOMPLETE: JSDoc block span "\\/\\*\\*(?!\\/)"…"\\*\\/" → external tokenizer.
   TripleSlash { "/""/""/"$[ \t\n\r]*"<"![\n]* }
   LineComment { "/""/"![\n]* }
   // INCOMPLETE: BlockComment block span "\\/\\*"…"\\*\\/" → external tokenizer.
@@ -337,5 +337,5 @@ Program {
 
 // ── INCOMPLETE / hand-tuning notes ──
 // - Rule Expr: Pratt operators (op/prefix/postfix) mapped to precedence-marked binary/unary alternatives. Lezer is GLR + @precedence, not Pratt, so deeply overloaded operators (e.g. '<' as both comparison and type-args, '+' infix vs prefix) may need hand-tuning or the external tokenizer to fully disambiguate.
-// - Block-style token JSDoc (delimiters "\\/\\*\\*"…"\\*\\/") routed to the external tokenizer; Lezer single tokens can't longest-match a multi-char close delimiter.
+// - Block-style token JSDoc (delimiters "\\/\\*\\*(?!\\/)"…"\\*\\/") routed to the external tokenizer; Lezer single tokens can't longest-match a multi-char close delimiter.
 // - Block-style token BlockComment (delimiters "\\/\\*"…"\\*\\/") routed to the external tokenizer; Lezer single tokens can't longest-match a multi-char close delimiter.

examples/tree-sitter/grammar.js

@@ -36,7 +36,7 @@ module.exports = grammar({
 
     type: $ => choice(seq($.ident, optional(seq("is", $.type))), seq($.type, "<", optional(seq($.type, repeat(seq(",", $.type)), optional(","))), ">"), seq($.type, "[", "]"), seq($.type, "|", $.type), seq($.type, "&", $.type), seq("|", $.type), seq("&", $.type), seq("keyof", $.type), seq("typeof", $.typeof_ref), seq("readonly", $.type), seq("(", $.type, ")"), seq(optional($.type_params), "(", optional(seq($.param, repeat(seq(",", $.param)), optional(","))), ")", "=>", $.type), seq(optional("abstract"), "new", optional($.type_params), "(", optional(seq($.param, repeat(seq(",", $.param)), optional(","))), ")", "=>", $.type), seq("[", repeat(seq(optional("..."), optional(seq($.ident, optional("?"), ":")), optional("..."), $.type, optional("?"), optional(","))), "]"), seq("{", repeat(seq($.type_member, optional(choice(";", ",")))), "}"), seq("asserts", $.ident, optional(seq("is", $.type))), seq($.type, "extends", $.type, "?", $.type, ":", $.type), seq("infer", $.ident, optional(seq("extends", $.type, optional(seq("?", $.type, ":", $.type))))), $.string, $.number, $.hex_number, $.octal_number, $.binary_number, $.big_int, seq("-", choice($.number, $.big_int)), "true", "false", "null", "undefined", "void", "this", seq("unique", "symbol"), seq("import", "(", $.type, ")"), $.template, seq($.type, "[", $.type, "]"), seq($.type, ".", $.ident)),
 
-    type_member: $ => choice(seq(optional("new"), optional($.type_params), "(", optional(seq($.param, repeat(seq(",", $.param)), optional(","))), ")", optional(seq(":", $.type))), seq(optional(choice("+", "-")), optional("readonly"), "[", choice(seq($.ident, choice(seq("in", $.type, optional(seq("as", $.type)), "]", optional(choice("+", "-")), optional("?"), ":", $.type), seq(":", $.type, "]", optional(seq(":", $.type))))), seq($.expr, "]", optional("?"), choice(seq(optional($.type_params), "(", optional(seq($.param, repeat(seq(",", $.param)), optional(","))), ")", optional(seq(":", $.type))), optional(seq(":", $.type)))))), seq("readonly", $.ident, optional("?"), ":", $.type), seq(choice($.ident, $.number, $.string, $.private_field), optional("?"), choice(seq(optional($.type_params), "(", optional(seq($.param, repeat(seq(",", $.param)), optional(","))), ")", optional(seq(":", $.type))), optional(seq(":", $.type))))),
+    type_member: $ => choice(seq(optional("new"), optional($.type_params), "(", optional(seq($.param, repeat(seq(",", $.param)), optional(","))), ")", optional(seq(":", $.type))), seq(optional(choice("+", "-")), optional("readonly"), "[", choice(seq($.ident, choice(seq("in", $.type, optional(seq("as", $.type)), "]", optional(choice("+", "-")), optional("?"), ":", $.type), seq(":", $.type, "]", optional(seq(":", $.type))))), seq($.expr, "]", optional("?"), choice(seq(optional($.type_params), "(", optional(seq($.param, repeat(seq(",", $.param)), optional(","))), ")", optional(seq(":", $.type))), optional(seq(":", $.type)))), seq("]", optional(seq(":", $.type))))), seq("readonly", $.ident, optional("?"), ":", $.type), seq(choice($.ident, $.number, $.string, $.private_field), optional("?"), choice(seq(optional($.type_params), "(", optional(seq($.param, repeat(seq(",", $.param)), optional(","))), ")", optional(seq(":", $.type))), optional(seq(":", $.type))))),
 
     decorator_expr: $ => seq($.decorator, optional(choice(seq("(", optional(seq($.expr, repeat(seq(",", $.expr)), optional(","))), ")"), seq("<", optional(seq($.type, repeat(seq(",", $.type)), optional(","))), ">", "(", optional(seq($.expr, repeat(seq(",", $.expr)), optional(","))), ")")))),
 
@@ -72,7 +72,7 @@ module.exports = grammar({
       "this",
       "super",
       seq("...", $.expr),
-      prec.left(18, seq($.expr, "<", optional(seq($.type, repeat(seq(",", $.type)), optional(","))), ">", optional(choice(seq("(", optional(seq($.expr, repeat(seq(",", $.expr)), optional(","))), ")"), $.template)))),
+      prec.left(18, seq($.expr, "<", optional(seq($.type, repeat(seq(",", $.type)), optional(","))), ">", choice(seq("(", optional(seq($.expr, repeat(seq(",", $.expr)), optional(","))), ")"), $.template, blank()))),
       prec.left(18, seq($.expr, "(", optional(seq($.expr, repeat(seq(",", $.expr)), optional(","))), ")")),
       prec.left(18, seq($.expr, ".", choice($.ident, $.private_field))),
       prec.left(18, seq($.expr, "?.", choice($.ident, $.private_field, seq("(", optional(seq($.expr, repeat(seq(",", $.expr)), optional(","))), ")"), seq("[", $.expr, "]"), $.template))),
@@ -136,7 +136,7 @@ module.exports = grammar({
 
     type_params: $ => seq("<", optional(seq($.type_param, repeat(seq(",", $.type_param)), optional(","))), ">"),
 
-    type_param: $ => choice(seq(repeat1(choice("const", "in", "out", "public", "private", "protected", "readonly")), $.ident, optional(seq("extends", $.type)), optional(seq("=", $.type))), seq($.ident, optional(seq("extends", $.type)), optional(seq("=", $.type)))),
+    type_param: $ => choice(seq(repeat1(choice("const", "in", "out", "public", "private", "protected", "readonly")), $.ident, optional(seq("extends", $.type)), optional(seq("=", $.type))), seq(choice("const", "in", "out", "public", "private", "protected", "readonly"), choice($.ident, "in", "out"), optional(seq("extends", $.type)), optional(seq("=", $.type))), seq(choice($.ident, "in", "out"), optional(seq("extends", $.type)), optional(seq("=", $.type)))),
 
     decl: $ => choice(seq(optional("async"), "function", optional("*"), field('name', $.ident), optional($.type_params), "(", optional(seq($.param, repeat(seq(",", $.param)), optional(","))), ")", optional(seq(":", $.type)), choice($.block, optional(";"))), seq("interface", field('name', $.ident), optional($.type_params), optional(seq("extends", optional(seq($.type, repeat(seq(",", $.type)), optional(","))))), "{", repeat(seq($.interface_member, optional(choice(";", ",")))), "}"), seq("type", field('name', $.ident), optional($.type_params), "=", $.type, optional(";")), seq(repeat($.decorator_expr), optional("abstract"), "class", field('name', $.ident), optional($.type_params), optional(seq("extends", $.class_heritage)), optional(seq("implements", optional(seq($.type, repeat(seq(",", $.type)), optional(","))))), "{", repeat($.class_member), "}"), seq("enum", field('name', $.ident), "{", optional(seq($.enum_member, repeat(seq(",", $.enum_member)), optional(","))), "}"), seq("declare", "function", optional("*"), field('name', $.ident), optional($.type_params), "(", optional(seq($.param, repeat(seq(",", $.param)), optional(","))), ")", optional(seq(":", $.type)), optional(";")), seq("declare", choice($.decl, $.stmt)), seq("namespace", field('name', $.ident), "{", repeat($.stmt), "}"), seq("module", choice($.string, $.ident), "{", repeat($.stmt), "}"), seq("export", choice($.decl, $.stmt)), seq("export", "default", choice(seq(optional("async"), "function", optional("*"), optional(field('name', $.ident)), optional($.type_params), "(", optional(seq($.param, repeat(seq(",", $.param)), optional(","))), ")", optional(seq(":", $.type)), choice($.block, optional(";"))), seq("abstract", "class", field('name', $.ident), optional($.type_params), optional(seq("extends", $.class_heritage)), optional(seq("implements", optional(seq($.type, repeat(seq(",", $.type)), optional(","))))), "{", repeat($.class_member), "}"), seq("abstract", "class", optional($.type_params), optional(seq("extends", $.class_heritage)), optional(seq("implements", optional(seq($.type, repeat(seq(",", $.type)), optional(","))))), "{", repeat($.class_member), "}"), seq($.expr, optional(";")))), seq("export", "*", choice(seq("from", $.string, optional(";")), seq("as", $.ident, "from", $.string, optional(";")))), seq("export", "{", optional(seq($.import_specifier, repeat(seq(",", $.import_specifier)), optional(","))), "}", optional(seq("from", $.string)), optional(";")), seq("export", "=", $.expr, optional(";")), seq("export", "type", "{", optional(seq($.import_specifier, repeat(seq(",", $.import_specifier)), optional(","))), "}", optional(seq("from", $.string)), optional(";")), seq("const", "enum", field('name', $.ident), "{", optional(seq($.enum_member, repeat(seq(",", $.enum_member)), optional(","))), "}"), seq("import", choice(seq($.import_clause, "from", $.string, optional(";")), seq("type", $.import_clause, "from", $.string, optional(";")), seq($.ident, "=", $.expr, optional(";")), seq($.string, optional(";")))), seq(repeat($.decorator_expr), "export", choice($.decl, $.stmt))),
 
@@ -152,7 +152,7 @@ module.exports = grammar({
 
     shebang: $ => token(/^#![^\n]*/),
 
-    jsdoc: $ => token(/\/\*\*[\s\S]*?\*\//),
+    jsdoc: $ => token(/\/\*\*(?!\/)[\s\S]*?\*\//),
 
     triple_slash: $ => token(/\/\/\/\s*<[^\n]*/),
 

examples/typescript.monarch.json

@@ -185,7 +185,7 @@
         "white"
       ],
       [
-        "\\/\\*\\*",
+        "\\/\\*\\*(?!\\/)",
         {
           "token": "comment",
           "next": "@comment_slashstarstar"
@@ -251,7 +251,7 @@
         "white"
       ],
       [
-        "\\/\\*\\*",
+        "\\/\\*\\*(?!\\/)",
         {
           "token": "comment",
           "next": "@comment_slashstarstar"
@@ -317,7 +317,7 @@
         "white"
       ],
       [
-        "\\/\\*\\*",
+        "\\/\\*\\*(?!\\/)",
         {
           "token": "comment",
           "next": "@comment_slashstarstar"
@@ -477,7 +477,7 @@
         "white"
       ],
       [
-        "\\/\\*\\*",
+        "\\/\\*\\*(?!\\/)",
         {
           "token": "comment",
           "next": "@comment_slashstarstar"

examples/typescript.tmLanguage.json

@@ -411,7 +411,7 @@
     },
     "jsdoc": {
       "name": "comment.block.documentation.typescript",
-      "begin": "\\/\\*\\*",
+      "begin": "\\/\\*\\*(?!\\/)",
       "end": "\\*\\/",
       "contentName": "meta.embedded.block.jsdoc"
     },

examples/typescript.ts

@@ -312,15 +312,17 @@ const Stmt = rule($ => [
 
 const TypeParam = rule($ => {
   // TS parses any modifier soup before a type-param name (variance `in`/`out`,
-  // `const`, even bogus `public`), then reports invalid ones post-parse. `many1`
-  // requires a name AFTER the modifiers, so a param NAMED like a modifier (`<out>`,
-  // `<in = any>`) falls through to the bare-name branch (longest-match: when a
-  // name is present the modifier branch wins because it consumes more).
+  // `const`, even bogus `public`), then reports invalid ones post-parse. A param
+  // can also be NAMED like a modifier — `<in>`, `<out = any>`, and even the
+  // variance-modified `<in in>` / `<out out>` (first `in`/`out` is the modifier,
+  // second is the name). Longest-match picks among:
   const tail = [opt('extends', Type), opt('=', Type)];
   const mod = alt('const', 'in', 'out', 'public', 'private', 'protected', 'readonly');
+  const name = alt(Ident, 'in', 'out');  // a name may itself be a contextual variance keyword
   return [
-    [many1(mod), Ident, ...tail],
-    [Ident, ...tail],
+    [many1(mod), Ident, ...tail],  // modifier soup + real-ident name: `<const in T>`, `<in T>`
+    [mod, name, ...tail],          // single modifier + in/out-named param: `<in in>`, `<out out>`
+    [name, ...tail],               // bare name, incl. `<in>`, `<out>`: `<T>`, `<in = any>`
   ];
 });