Close the last 2 valid-code conformance gaps → 100% valid-code coverage

Both remaining REAL gaps (valid TS that tsc parses cleanly but we failed)
are fixed. Valid single-file coverage is now 3376/3376 (zero gaps);
bidirectional agreement with tsc is 96.7% (the rest is over-acceptance).

1. Postfix closes the access tail (parserComputedPropertyName29).
   `[e] = id++` ⏎ `[e2]` was mis-parsed as `(id++)[e2]`, but member access
   can't attach to a postfix result (an UpdateExpression isn't a
   MemberExpression). After a true postfix binds, no access-tail LED
   (`.`/`[`/`(`/`?.`/`<T>`/tagged-template) may follow — binary/ternary
   ops still do. Tails are detected structurally (last item isn't a
   self-operand; connector isn't a word-keyword), so the engine stays
   language-agnostic. TS non-null `!` moves from prec-postfix to an
   access-tail LED `[$, '!']`, since it IS part of the LHS chain
   (`x!.y`, `x!()` are valid). Bonus: `a--.x`, `a++ ++`, `foo ++ ++` are
   now correctly rejected (they were over-accepts).

2. Mixfix operand re-bind for left-recursive rules (inferTypesWithExtends1).
   `(T extends infer U extends number ? 1 : 0)` failed: inside parens, an
   `infer …extends T` over-consumed the conditional's `?`. The re-bind that
   already fixed this for expressions lived only in parsePratt, but Type is
   left-recursive (parseLeftRec). Factor the mixfix detection into mixfixOf()
   and reuse matchMixfixLed from parseLeftRec via a contMixfix map.

Add test/conformance-matrix.ts (the bidirectional confusion matrix vs tsc's
parseDiagnostics — the real metric) and test/verify-rejects.ts (checks we
reject at tsc's first-error offset). Regenerate artifacts; reframe the
README around valid-code coverage + bidirectional agreement.

Author: johnsoncodehk

README.md

@@ -10,7 +10,7 @@ A TextMate grammar is a pile of regexes guessing at a language's structure. It's
 
 Monogram inverts the dependency:
 
-1. **Write the grammar, then prove it.** The grammar is executable. Monogram runs it as a recursive-descent + Pratt parser over the TypeScript conformance suite. Today it parses **95.4%** (3602 / 3776 files); the goal is **100% coverage of the official parser** — at which point the grammar is a *verified, complete model* of the language's syntax, not an approximation.
+1. **Write the grammar, then prove it.** The grammar is executable. Monogram runs it as a recursive-descent + Pratt parser over the TypeScript conformance suite and measures conformance *bidirectionally* — it must accept what `tsc` accepts **and** reject what `tsc` rejects. Today it parses **100% of valid single-file cases** (3376 / 3376 — zero valid-code gaps) and agrees with `tsc` **bidirectionally on 96.7%** (3544 / 3664); the remaining gap is *over-acceptance* (the grammar is still too permissive on some invalid inputs), and the goal is 100% both ways — a *verified, complete model* of the language's syntax, not an approximation.
 
 2. **Derive the highlighter from the proven grammar.** The TextMate grammar is generated from that same parser-validated grammar — never hand-written. Its correctness is underwritten by the parser conformance run, not by regex tuning.
 
@@ -37,7 +37,8 @@ And — from the same grammar — first-pass generators for the rest of the edit
 Validated on TypeScript (grammar: [`examples/typescript.ts`](examples/typescript.ts), 537 lines):
 
 ```
-Parser conformance   95.4%   3602 / 3776 files from the TypeScript conformance suite   (goal: 100%)
+Valid-code coverage  100%    3376 / 3376 valid single-file conformance cases parse   (zero gaps)
+Bidirectional        96.7%   3544 / 3664 — also rejects what tsc rejects   (goal: 100%, gap = over-acceptance)
 Highlighter          99.3%   589 / 593 tokens match VS Code's official grammar
 Generated grammar    42 KB   vs the official hand-written 226 KB
 Engine               language-agnostic — no TypeScript-specific code
@@ -184,4 +185,4 @@ Every highlighter target (TextMate, tree-sitter queries, Lezer styleTags, Monarc
 | ANTLR | yes | — | — |
 | Langium | yes | Monarch (separate) | — |
 | ungrammar | AST types | — | — |
-| **Monogram** | **CST (95.4% → 100%)** | **auto-derived (99.3%)** | **yes** |
+| **Monogram** | **CST (100% valid / 96.7% bidir)** | **auto-derived (99.3%)** | **yes** |

examples/lezer/highlight.js

@@ -45,8 +45,8 @@ export const highlighting = styleTags({
   "Semi": t.punctuation,
   "Op_25/Op_2a/Op_2a2a/Op_2b/Op_2d/Op_2f": t.arithmeticOperator,
   "QuestionDot": t.derefOperator,
-  "Op_253d/Op_26263d/Op_263d/Op_2a2a3d/Op_2a3d/Op_2b3d/Op_2d3d/Op_2f3d/Op_3c3c3d/Op_3d/Op_3e3e3d/Op_3e3e3e3d/Op_3f3f3d/Op_5e3d/Op_7c3d/Op_7c7c3d": t.definitionOperator,
   "Op_21/Op_2626/Op_3f3f/Op_7c7c/Op_7e": t.logicOperator,
+  "Op_253d/Op_26263d/Op_263d/Op_2a2a3d/Op_2a3d/Op_2b3d/Op_2d3d/Op_2f3d/Op_3c3c3d/Op_3d/Op_3e3e3d/Op_3e3e3e3d/Op_3f3f3d/Op_5e3d/Op_7c3d/Op_7c7c3d": t.definitionOperator,
   "Op_213d/Op_213d3d/Op_3d3d/Op_3d3d3d": t.compareOperator,
   "Op_2b2b/Op_2d2d": t.operator,
 });

examples/lezer/typescript.grammar

@@ -53,6 +53,7 @@ Expr {
     Expr Dot (Ident | PrivateField) |
     Expr QuestionDot (Ident | PrivateField | ParenL (Expr (Comma Expr)*)? ParenR | BracketL Expr BracketR | Template) |
     Expr BracketL Expr BracketR |
+    Expr Op_21 |
     Expr Question Expr Colon Expr |
     Expr kw_as Type |
     Expr kw_instanceof Expr |
@@ -94,7 +95,7 @@ Expr {
     Expr !l13 Op_2a2a Expr |
     !l14 (Op_21 | Op_7e | Op_2b | Op_2d | kw_typeof | kw_void | kw_delete | kw_await | kw_yield) Expr |
     !l15 (Op_2b2b | Op_2d2d) Expr |
-    Expr !l16 (Op_2b2b | Op_2d2d | Op_21)
+    Expr !l16 (Op_2b2b | Op_2d2d)
 }
 Prop {
   (Spread Expr | (kw_public | kw_private | kw_protected)? (kw_get | kw_set) MemberName ParenL ((Param (Comma Param)*)?)? ParenR (Colon Type)? Block | kw_async? Op_2a? MemberName TypeParams? ParenL (Param (Comma Param)*)? ParenR (Colon Type)? Block | Template (Colon Expr | ParenL (Param (Comma Param)*)? ParenR (Colon Type)? Block) | MemberName Colon Expr | BracketL Expr (Comma Expr)* BracketR Colon Expr | Ident (Op_3d Expr | Question (Colon Expr)? | ))

examples/tree-sitter/grammar.js

@@ -59,7 +59,7 @@ module.exports = grammar({
       prec.right(14, seq($.expr, field('operator', "**"), $.expr)),
       prec.right(15, seq(field('operator', choice("!", "~", "+", "-", "typeof", "void", "delete", "await", "yield")), $.expr)),
       prec.right(16, seq(field('operator', choice("++", "--")), $.expr)),
-      prec.left(17, seq($.expr, field('operator', choice("++", "--", "!")))),
+      prec.left(17, seq($.expr, field('operator', choice("++", "--")))),
       $.ident,
       $.number,
       $.string,
@@ -77,6 +77,7 @@ module.exports = grammar({
       prec.left(18, seq($.expr, ".", choice($.ident, $.private_field))),
       prec.left(18, seq($.expr, "?.", choice($.ident, $.private_field, seq("(", optional(seq($.expr, repeat(seq(",", $.expr)), optional(","))), ")"), seq("[", $.expr, "]"), $.template))),
       prec.left(18, seq($.expr, "[", $.expr, "]")),
+      prec.left(18, seq($.expr, "!")),
       prec.left(18, seq($.expr, "?", $.expr, ":", $.expr)),
       prec.left(18, seq($.expr, "as", $.type)),
       prec.left(18, seq($.expr, "instanceof", $.expr)),

examples/typescript.monarch.json

@@ -1011,7 +1011,7 @@
         }
       ],
       [
-        ">>>=|\\*\\*=|<<=|>>=|\\?\\?=|\\|\\|=|&&=|===|!==|>>>|=>|\\+=|-=|\\*=|%=|&=|\\|=|\\^=|\\?\\?|\\|\\||&&|==|!=|<=|>=|<<|>>|\\*\\*|\\+\\+|--|\\||&|-|\\+|\\*|=|!|\\^|%|~",
+        ">>>=|\\*\\*=|<<=|>>=|\\?\\?=|\\|\\|=|&&=|===|!==|>>>|=>|\\+=|-=|\\*=|%=|&=|\\|=|\\^=|\\?\\?|\\|\\||&&|==|!=|<=|>=|<<|>>|\\*\\*|\\+\\+|--|\\||&|-|\\+|!|\\*|=|\\^|%|~",
         {
           "token": "operator",
           "switchTo": "@root"
@@ -1230,7 +1230,7 @@
         "delimiter.square"
       ],
       [
-        ">>>=|\\*\\*=|<<=|>>=|\\?\\?=|\\|\\|=|&&=|===|!==|>>>|=>|\\+=|-=|\\*=|%=|&=|\\|=|\\^=|\\?\\?|\\|\\||&&|==|!=|<=|>=|<<|>>|\\*\\*|\\+\\+|--|\\||&|-|\\+|\\*|=|!|\\^|%|~",
+        ">>>=|\\*\\*=|<<=|>>=|\\?\\?=|\\|\\|=|&&=|===|!==|>>>|=>|\\+=|-=|\\*=|%=|&=|\\|=|\\^=|\\?\\?|\\|\\||&&|==|!=|<=|>=|<<|>>|\\*\\*|\\+\\+|--|\\||&|-|\\+|!|\\*|=|\\^|%|~",
         "operator"
       ],
       [

examples/typescript.ts

@@ -178,6 +178,7 @@ const Expr = rule($ => [
   // optional chaining: ?.x | ?.#x | ?.(args) | ?.[i] | ?.`…`
   [$, '?.', alt(Ident, PrivateField, ['(', sep($, ','), ')'], ['[', $, ']'], Template)],
   [$, '[', $, ']'],
+  [$, '!'],   // TS non-null assertion — a LHS-chain tail (access can follow: `x!.y`, `x!()`), unlike update `++`/`--`
   [$, '?', $, ':', $],
   [$, 'as', Type],
   [$, 'instanceof', $],
@@ -484,7 +485,7 @@ export default defineGrammar({
     right('**'),
     right(prefix('!', '~', '+', '-', 'typeof', 'void', 'delete', 'await', 'yield')),
     right(prefix('++', '--')),
-    left(postfix('++', '--', '!')),
+    left(postfix('++', '--')),
   ],
 
   rules: {

src/gen-parser.ts

@@ -200,16 +200,55 @@ export function createParser(grammar: CstGrammar) {
     return !!e && e.type === 'ref' && e.name === ruleName;
   }
   type MixfixInfo = { openLit: string; sepLit: string };
+  // A continuation `<lit L1> $self <lit L2> …` whose inner `$self` operand can
+  // over-consume the `L2` the operator needs (e.g. ternary `? $ : $`, conditional
+  // type `extends $ ? $ : $`). The re-bind retries that operand capped.
+  function mixfixOf(items: RuleExpr[], ruleName: string): MixfixInfo | null {
+    if (items.length >= 4
+      && items[0]?.type === 'literal'
+      && selfRefName(items[1], ruleName)
+      && items[2]?.type === 'literal') {
+      return { openLit: items[0].value, sepLit: items[2].value };
+    }
+    return null;
+  }
   const ledMixfix = new Map<object, MixfixInfo>();
+  for (const [ruleName, { leds }] of prattClassified.entries()) {
+    for (const led of leds) {
+      const info = mixfixOf(led.items, ruleName);
+      if (info) ledMixfix.set(led, info);
+    }
+  }
+  // Same re-bind for left-recursive (non-Pratt) rules like `Type`, whose
+  // continuations carry the implicit leading `$`, so they are already stripped.
+  const contMixfix = new Map<object, MixfixInfo>();
+  for (const [ruleName, { continuations }] of leftRecClassified.entries()) {
+    for (const cont of continuations) {
+      const info = mixfixOf(cont, ruleName);
+      if (info) contMixfix.set(cont, info);
+    }
+  }
+
+  // ── Access-tail LEDs (closed under a postfix operator) ──
+  // A postfix operator (`a++`) turns its operand into an "update expression" that
+  // member-access tails can no longer attach to: `a++[b]`, `a++.c`, `a++()`,
+  // `a++<T>()`, `` a++`x` `` are all ill-formed (member access needs a primary, not
+  // a postfix result). So once a postfix binds, such tails must NOT continue — the
+  // bracketed term belongs to whatever follows (e.g. the next class field after an
+  // ASI). Detected structurally, language-agnostically: an access tail is a non-op
+  // LED that is "closed" (its last item is not a fresh same-rule operand, unlike a
+  // binary/ternary `… in $` / `? $ : $`) AND whose connector is a punctuator, not a
+  // word-operator — so `as`/`satisfies`/`in`/`instanceof`/`?:` still bind after `a++`.
+  const accessTailLeds = new Set<object>();
   for (const [ruleName, { leds }] of prattClassified.entries()) {
     for (const led of leds) {
       const it = led.items;
-      if (it.length >= 4
-        && it[0]?.type === 'literal'
-        && selfRefName(it[1], ruleName)
-        && it[2]?.type === 'literal') {
-        ledMixfix.set(led, { openLit: it[0].value, sepLit: it[2].value });
-      }
+      if (it.length === 0) continue;
+      if (it[0].type === 'op' || it[0].type === 'postfix') continue;   // operator LEDs, not tails
+      const last = it[it.length - 1];
+      const lastIsOperand = selfRefName(last, ruleName);                // open binary/ternary operand
+      const wordConnector = it[0].type === 'literal' && /^[A-Za-z]/.test(it[0].value);
+      if (!lastIsOperand && !wordConnector) accessTailLeds.add(led);
     }
   }
 
@@ -523,7 +562,16 @@ export function createParser(grammar: CstGrammar) {
         const contSaved = pos;
         for (const cont of continuations) {
           pos = contSaved;
-          const children = matchSeq(cont);
+          let children = matchSeq(cont);
+          // Mixfix operand re-bind (same fix parsePratt uses): a continuation of the
+          // shape `<lit> $self <lit> …` (e.g. the conditional type `extends $ ? $ : $`)
+          // can have its inner operand over-consume the separator the operator needs
+          // (an `infer U extends T ? …` swallowing the conditional's `?`). Retry it
+          // capped so the operand stops before that separator.
+          if (children === null) {
+            const mix = contMixfix.get(cont);
+            if (mix) { pos = contSaved; children = matchMixfixLed({ items: cont }, rule.name, mix); }
+          }
           if (children !== null) {
             node = {
               kind: 'node',
@@ -587,6 +635,10 @@ export function createParser(grammar: CstGrammar) {
 
       if (!lhs) { pos = saved; return null; }
 
+      // Once a postfix operator binds (`a++`), the operand is an update expression
+      // that access tails (`[…]`, `.x`, `(…)`, `<T>`, tagged template) can't extend.
+      let tailClosed = false;
+
       // LED loop
       while (true) {
         const tok = peek();
@@ -599,6 +651,7 @@ export function createParser(grammar: CstGrammar) {
         for (const led of leds) {
           if (led.items[0]?.type === 'op' || led.items[0]?.type === 'postfix') continue;
           if (maxBp <= minBp) continue;
+          if (tailClosed && accessTailLeds.has(led)) continue;   // no access tail after a postfix
           // Skip a LED whose connector is excluded in this context (e.g. `in` under
           // a no-`in` for-head) — it rebinds to the enclosing rule instead.
           if (suppressCur && led.items[0]?.type === 'literal' && suppressCur.has(led.items[0].value)) continue;
@@ -632,10 +685,13 @@ export function createParser(grammar: CstGrammar) {
         const info = opTable.get(tokKey);
         if (info && info.lbp > minBp) {
           if (info.position === 'postfix') {
-            pos++;
-            const opLeaf: CstLeaf = { kind: 'leaf', tokenType: '$operator', text: tok.text, offset: tok.offset, end: tok.offset + tok.text.length };
-            lhs = { kind: 'node', rule: rule.name, children: [lhs, opLeaf], offset: lhs.offset, end: opLeaf.end };
-            matched = true;
+            if (!tailClosed) {                                   // can't postfix an update expr (`a++ --`)
+              pos++;
+              const opLeaf: CstLeaf = { kind: 'leaf', tokenType: '$operator', text: tok.text, offset: tok.offset, end: tok.offset + tok.text.length };
+              lhs = { kind: 'node', rule: rule.name, children: [lhs, opLeaf], offset: lhs.offset, end: opLeaf.end };
+              tailClosed = true;
+              matched = true;
+            }
           } else {
             pos++;
             const opLeaf: CstLeaf = { kind: 'leaf', tokenType: '$operator', text: tok.text, offset: tok.offset, end: tok.offset + tok.text.length };

test/conformance-matrix.ts

@@ -0,0 +1,48 @@
+// Bidirectional conformance: a real parser must ACCEPT what TS accepts AND REJECT
+// what TS rejects. The snapshot's "pass = didn't throw" only measures acceptance —
+// the easy half — so it counts an over-accept (invalid code we wave through) as a
+// success. This computes the full confusion matrix against TS's own parseDiagnostics
+// over single-file conformance cases (multi-`@filename` excluded — not one program).
+import { createParser } from '../src/gen-parser.ts';
+import { readdirSync, readFileSync } from 'fs';
+import { join } from 'path';
+import ts from 'typescript';
+
+const grammar = (await import('../examples/typescript.ts')).default;
+const { parse } = createParser(grammar);
+const base = '/tmp/ts-repo/tests/cases/conformance';
+
+function walk(d: string): string[] {
+  let o: string[] = [];
+  for (const e of readdirSync(d, { withFileTypes: true })) {
+    const f = join(d, e.name);
+    if (e.isDirectory()) o = o.concat(walk(f));
+    else if (e.name.endsWith('.ts') && !e.name.endsWith('.d.ts')) o.push(f);
+  }
+  return o;
+}
+const isMulti = (t: string) => /^\s*\/\/\s*@filename:/im.test(t);
+
+let TP = 0, FN = 0, FP = 0, TN = 0;
+const fns: string[] = [], fps: string[] = [];
+for (const f of walk(base)) {
+  const code = readFileSync(f, 'utf8');
+  if (isMulti(code)) continue;
+  const sf = ts.createSourceFile('t.ts', code, ts.ScriptTarget.Latest, true, ts.ScriptKind.TS);
+  const tsAccept = ((sf as any).parseDiagnostics?.length ?? 0) === 0;
+  let weAccept = true;
+  try { parse(code); } catch { weAccept = false; }
+  if (tsAccept && weAccept) TP++;
+  else if (tsAccept && !weAccept) { FN++; fns.push(f.replace(base + '/', '')); }
+  else if (!tsAccept && weAccept) { FP++; fps.push(f.replace(base + '/', '')); }
+  else TN++;
+}
+const total = TP + FN + FP + TN;
+console.log(`Single-file conformance cases: ${total}\n`);
+console.log('                     WE accept        WE reject');
+console.log(`  TS accept (valid)  ${String(TP).padStart(6)} (correct)  ${String(FN).padStart(4)} (MISS — valid-code gap)`);
+console.log(`  TS reject (error)  ${String(FP).padStart(6)} (over-accept)${String(TN).padStart(4)} (correct reject)`);
+console.log('');
+console.log(`  Valid-code coverage : ${(TP / (TP + FN) * 100).toFixed(2)}%  (${TP}/${TP + FN})  ← parses every valid file when FN=0`);
+console.log(`  Bidirectional agree : ${((TP + TN) / total * 100).toFixed(2)}%  (${TP + TN}/${total})`);
+if (fns.length) console.log(`\n  MISSED valid (real gaps):\n${fns.map(x => '    - ' + x).join('\n')}`);

test/verify-rejects.ts

@@ -0,0 +1,71 @@
+// Rigor check for the "error-test" bucket: failing a file that TS also rejects
+// only counts as *correct* if we reject for the RIGHT reason — i.e. we reach at
+// least as far as TS's first syntax error before bailing. If our parser chokes
+// EARLIER than TS's first diagnostic, the code before that point is valid TS we
+// silently can't parse — a hidden gap hiding behind the "intentional error" label.
+import { createParser } from '../src/gen-parser.ts';
+import { readdir } from 'fs/promises';
+import { readFileSync } from 'fs';
+import { join } from 'path';
+import ts from 'typescript';
+
+const grammar = (await import('../examples/typescript.ts')).default;
+const { parse } = createParser(grammar);
+const baseDir = '/tmp/ts-repo/tests/cases/conformance';
+const SLACK = 8; // chars of tolerance (token boundary / trivia differences)
+
+async function allTsFiles(dir: string): Promise<string[]> {
+  const out: string[] = [];
+  for (const e of await readdir(dir, { withFileTypes: true })) {
+    const full = join(dir, e.name);
+    if (e.isDirectory()) out.push(...await allTsFiles(full));
+    else if (e.name.endsWith('.ts') && !e.name.endsWith('.d.ts')) out.push(full);
+  }
+  return out;
+}
+
+const isMulti = (t: string) => /^\s*\/\/\s*@filename:/im.test(t);
+// How far our parser actually reached: prefer the `farthest` backtracking mark,
+// else the primary error offset.
+function ourReach(msg: string): number | null {
+  const far = msg.match(/farthest: offset (\d+)/);
+  if (far) return +far[1];
+  const at = msg.match(/offset (\d+)/);
+  return at ? +at[1] : null;
+}
+
+const files = (await allTsFiles(baseDir)).sort();
+let agree = 0, early = 0, unknown = 0;
+const earlies: { file: string; ourReach: number; tsFirst: number; ctx: string }[] = [];
+
+for (const file of files) {
+  const code = readFileSync(file, 'utf-8');
+  if (isMulti(code)) continue;                 // single-file only (clean comparison)
+  let msg = '';
+  try { parse(code); continue; } catch (e: any) { msg = e.message; }  // only files we FAIL
+
+  const sf = ts.createSourceFile('t.ts', code, ts.ScriptTarget.Latest, true, ts.ScriptKind.TS);
+  const diags = (sf as any).parseDiagnostics ?? [];
+  if (diags.length === 0) continue;            // that's a REAL gap, handled elsewhere
+
+  const tsFirst = Math.min(...diags.map((d: any) => d.start ?? Infinity));
+  const reach = ourReach(msg);
+  if (reach == null) { unknown++; continue; }
+
+  if (reach >= tsFirst - SLACK) {
+    agree++;                                    // we got to (or past) TS's first error → right reason
+  } else {
+    early++;                                    // we bailed on valid code BEFORE the error
+    earlies.push({ file: file.replace(baseDir + '/', ''), ourReach: reach, tsFirst, ctx: JSON.stringify(code.slice(Math.max(0, reach - 40), reach + 15)) });
+  }
+}
+
+console.log(`Single-file error-tests we fail: ${agree + early + unknown}`);
+console.log(`  AGREE (reach >= TS first error - ${SLACK}) : ${agree}  ← rejected for the right reason`);
+console.log(`  EARLY (bail before TS's error)            : ${early}  ← hidden gap: valid code we can't parse`);
+console.log(`  UNKNOWN (no offset in our error)          : ${unknown}`);
+if (earlies.length) {
+  console.log(`\n===== EARLY (hidden gaps) =====`);
+  earlies.sort((a, b) => (a.tsFirst - a.ourReach) - (b.tsFirst - b.ourReach));
+  for (const e of earlies) console.log(`  ${e.file}\n      ours@${e.ourReach} vs TS@${e.tsFirst} (gap ${e.tsFirst - e.ourReach})  near ${e.ctx}`);
+}