You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
test-issues: lock in 10 verified backlog issues (solved 39→49)
Verified 10 ledger-backlog issues are already correct in the generated TextMate
grammar and added regression cases (310 checks pass, was 288): #1021 regex `v`
flag, #788 optional chaining, #1025 for-of/in without spaces, #815 method named
`new`, #881 `override` modifier, #1066 triple-slash directive, #992 type named
`type`, #995 `as keyof typeof`, #994 default type-arg value, #1027 nested `>>`.
Monogram now solves 49 of the 106 open official issues (~46%). Ledger + README
updated; backlog trimmed to ~7, with #891 (binding named `from` mis-scoped as a
keyword) and #1071 (contested import.meta) documented as the honest residual gaps.
Copy file name to clipboardExpand all lines: README.md
+3-3Lines changed: 3 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -35,7 +35,7 @@ Take `typeof x < y`. A regex highlighter has to guess whether `<` opens a generi
35
35
36
36
The remaining 2.2% bidirectional gap is **over-acceptance** — the grammar is still too permissive on some *invalid* inputs (valid-code coverage is already 100%; nothing valid is missed). Most of what's left is code `tsc`'s parser rejects via *context-sensitive* rules a context-free grammar can't express — reserved-word placement, modifier combinations like `default abstract class`, `super` type-arguments — the kind of constraint a highlighter grammar was never meant to enforce. We push as close to 100% both ways as a pure grammar can; that's the asymptote, and the highlighter rides down to it for free.
37
37
38
-
That's the categorical part: a highlighter derived from a parser-proven grammar isn't *a better hand-written grammar* — it's playing a different game. You can't out-regex it, because its correctness comes from a dimension hand-written grammars never operate in. The evidence is concrete — [`test/test-issues.ts`](test/test-issues.ts) replays **39 real bugs** from the official grammar's issue tracker (the `typeof x < y` ambiguities, regex-after-keyword cases, `as`-casts inside `<>`), all **288** token checks pass, and **all 39 are still open upstream** — Monogram already fixes ~37% of the 106 open official issues, because those failure modes are *structurally precluded* by a parser rather than patched one regex at a time. The [**upstream issue ledger**](docs/upstream-issues.md) tracks exactly which issues we solve and gives an honest verdict on each one we don't (backlog / out-of-scope / needs-semantics / proven-TM-impossible).
38
+
That's the categorical part: a highlighter derived from a parser-proven grammar isn't *a better hand-written grammar* — it's playing a different game. You can't out-regex it, because its correctness comes from a dimension hand-written grammars never operate in. The evidence is concrete — [`test/test-issues.ts`](test/test-issues.ts) replays **49 real bugs** from the official grammar's issue tracker (the `typeof x < y` ambiguities, regex-after-keyword cases, `as`-casts inside `<>`, nested `>>`), all **310** token checks pass, and **all 49 are still open upstream** — Monogram already fixes ~46% of the 106 open official issues, because those failure modes are *structurally precluded* by a parser rather than patched one regex at a time. The [**upstream issue ledger**](docs/upstream-issues.md) tracks exactly which issues we solve and gives an honest verdict on each one we don't (backlog / out-of-scope / needs-semantics / proven-TM-impossible).
39
39
40
40
## Results
41
41
@@ -45,7 +45,7 @@ Measured against the TypeScript compiler's own conformance suite (single-file ca
Copy file name to clipboardExpand all lines: docs/upstream-issues.md
+14-12Lines changed: 14 additions & 12 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -8,31 +8,31 @@ A maintained accounting of [microsoft/TypeScript-TmLanguage](https://github.com/
8
8
9
9
| Verdict | Count | Meaning |
10
10
|---|---:|---|
11
-
| ✅ **Solved**|**39**| parser-derived TM gets it right; replayed green in [`test/test-issues.ts`](../test/test-issues.ts) (288 checks). **All 39 are still open upstream.**|
12
-
| 🔧 Backlog (our domain) |~17| TM-expressible + parser-derivable; not done yet. Some likely already correct, just untested. |
11
+
| ✅ **Solved**|**49**| parser-derived TM gets it right; replayed green in [`test/test-issues.ts`](../test/test-issues.ts) (310 checks). **All 49 are still open upstream.**|
12
+
| 🔧 Backlog (our domain) |~7| TM-expressible + parser-derivable; not done yet, or a real residual gap. |
13
13
| 📦 Out of scope (now) | 8 + 16 | TSX dialect (8); JSDoc *embedded-content* highlighting (16) — needs the embedded-grammar pipeline. |
14
14
| 🧠 Needs semantics |~6 | type/symbol resolution — **no syntax highlighter (TM or parser) can do this**; the language server's job. |
15
15
| 🛠 Not a grammar concern |~20 | theming, semantic-token config, scope-name data, tooling, IDE support. |
16
16
| ⛔ Proven TM-impossible | 0 confirmed | none yet proven (requires a failing generated-TM + a passing parser-target — see discipline). |
17
17
18
-
The headline: **Monogram already solves 39 of the 106 still-open official issues (~37%)** — and they are exactly the regex-vs-division / generic-vs-comparison / multiline-context class that a hand-written grammar perpetually loses to.
18
+
The headline: **Monogram already solves 49 of the 106 still-open official issues (~46%)** — exactly the regex-vs-division / generic-vs-comparison / multiline-context class that a hand-written grammar perpetually loses to.
19
19
20
20
---
21
21
22
-
## ✅ Solved (39)
22
+
## ✅ Solved (49)
23
23
24
24
All replayed in [`test/test-issues.ts`](../test/test-issues.ts) and all still open upstream. Grouped by the ambiguity class — every one is something a regex grammar cannot reliably decide but a parser settles for free.
25
25
26
26
**Regex literal vs. division `/`** — the lexer uses parser context (what the previous token *is*) to decide:
27
-
`#853` (after `throw`/`void`/`typeof`/`await`/`yield`), `#804` (`/[a\-b]/` char class), `#1024` (regex on a new line), `#883` (`for (const x of /re/.exec…)`), `#1055` (backtick inside a regex), `#1063` (`\cJ`/`\cj` control-char escapes).
27
+
`#853` (after `throw`/`void`/`typeof`/`await`/`yield`), `#804` (`/[a\-b]/` char class), `#1024` (regex on a new line), `#883` (`for (const x of /re/.exec…)`), `#1055` (backtick inside a regex), `#1063` (`\cJ`/`\cj` control-char escapes), `#1021` (the `v` unicode-sets flag).
28
28
29
29
**Generic type-args `<…>` vs. less-than `<`** — the parser knows when `<` opens type arguments:
30
-
`#1050`, `#978` (`typeof x < y`), `#859` (`as` cast inside `<>`), `#884` (`0 < x && 1 > 0`), `#904` (`bad < obj.two`), `#1020` (`new Map<…>` with no parens), `#855` (`<` with a `/* comment */` inside the args).
30
+
`#1050`, `#978` (`typeof x < y`), `#859` (`as` cast inside `<>`), `#884` (`0 < x && 1 > 0`), `#904` (`bad < obj.two`), `#1020` (`new Map<…>` with no parens), `#855` (`<` with a `/* comment */` inside the args), `#1027` (nested `>>` closes two arg-lists, not a shift), `#994` (default type-arg *value* is colored), `#995` (paren-wrapped `as keyof typeof`), `#992` (a type *named*`type`).
31
31
32
32
**Multiline / whitespace-fragile generics, types, unions** — structure, not column position, decides; a later construct never "breaks":
**Member / call scoping & control flow:**`#736` (method call → `entity.name.function`), `#770` (call parens are punctuation), `#869` (`x in obj ? a : b` ternary).
35
+
**Member / call scoping, keywords-as-names, control flow:**`#736` (method call → `entity.name.function`), `#770` (call parens are punctuation), `#869` (`x in obj ? a : b` ternary), `#788` (optional chaining `?.`), `#1025` (`for…of`/`in` with no surrounding space), `#815` (a method *named*`new`), `#881` (`override` modifier), `#1066` (triple-slash `/// <reference>` directive).
36
36
37
37
> Each maps to one or more cases in `test/test-issues.ts`; grep the issue number there for the exact input + expected scopes.
38
38
@@ -49,13 +49,15 @@ When an issue is **not** solved, it gets exactly one verdict, and **⛔ TM-impos
49
49
50
50
---
51
51
52
-
## 🔧 Backlog — our domain, solvable, not yet (≈17)
52
+
## 🔧 Backlog — our domain, not yet solved (≈7)
53
53
54
-
TM-expressible parse/lex disambiguation, squarely in Monogram's wheelhouse. Several are probably already correct in the generated grammar and only need a test case to confirm and move to Solved:
54
+
TM-expressible parse/lex disambiguationin Monogram's wheelhouse, still open after the verification pass (10 candidates that *were* here are now confirmed-correct and moved to Solved):
55
55
56
-
`#1071``import.meta.dirname` (`import` mis-keyworded), `#1048` multiline `extends` in a generic with split parens, `#1027` color inside angle brackets, `#1025``for-of`/`in` with no surrounding space, `#1021` regex `v` flag *(our `Regex` token already lists `v` — likely solved, verify)*, `#995``(obj as keyof typeof X)` paren-wrapped assertion, `#994` default template argument color, `#992` cast on a variable named `type`, `#891` default import named `from`*(we parse `import type from "…"` — likely solved)*, `#881``override` modifier, `#815` method named `new`, `#857` multiple `export default` overloads, `#497``import` declaration / semicolon scoping, `#810` trailing newline in a line comment, `#788` optional chaining *(supported — verify)*, `#1039``String.raw` escape tokens, `#1066` triple-slash reference directive.
56
+
-`#891` default import *named*`from` (`import from from "x"`) — **confirmed gap:** the binding `from` is mis-scoped as `keyword.control.import` instead of a variable, because `from` is in the keyword scope-map regardless of position. Needs the import-clause name to override the keyword scope. (Doesn't *break* the rest, unlike upstream — but the binding color is wrong.)
57
+
-`#1071``import.meta.dirname` — **contested:** we scope `import` as `keyword.control.import` (defensible — it *is* the import meta-property keyword) and `meta`/`dirname` as properties, with no breakage; the reporter wants `import` non-keyworded. Left until the intended scope is settled.
58
+
-`#1048` multiline `extends` in a generic with split parens; `#857` multiple `export default` overloads; `#497``import` declaration / semicolon scoping; `#810` trailing newline scope in a line comment; `#1039``String.raw` escape-token visibility.
57
59
58
-
> Action: each should be run through the generated TM (the `test/test-issues.ts`harness) to confirm pass/fail before claiming. Listing here ≠ "broken in Monogram" — it means "not yet verified-and-tested."
60
+
> Action: run each through `test/test-issues.ts` to confirm before claiming. Listing here ≠ "breaks in Monogram" — it means "not yet correct-and-tested."
**A TextMate grammar is a line-oriented pushdown machine whose only inter-line memory is a finite stack of scope contexts**; each step is a single Oniguruma match (bounded even with `\g<>` subexpression recursion, which is per-line). Where correct highlighting requires context *not* captured by the enclosing-scope stack — an unbounded lookback to a token that is not an ancestor context — no TextMate grammar can express it, the official one or ours. Monogram's **tree-sitter and Lezer** targets are real incremental parsers holding the full tree, so they resolve such cases exactly; the generated TM target (like the official) can only approximate.
83
85
84
-
> **But beware over-claiming this.** The 39 solved cases include heavy multiline / whitespace / nested-generic scenarios long *assumed* "impossible for TextMate" — they are not; they are TM-*expressible* once a parser tells you the right pattern. So this bucket is narrow, and we keep it empty until an issue is *proven* into it (failing generated-TM + passing parser-target). Don't reach for "TM can't" when the honest answer is "we haven't derived the pattern yet."
86
+
> **But beware over-claiming this.** The 49 solved cases include heavy multiline / whitespace / nested-generic scenarios long *assumed* "impossible for TextMate" — they are not; they are TM-*expressible* once a parser tells you the right pattern. So this bucket is narrow, and we keep it empty until an issue is *proven* into it (failing generated-TM + passing parser-target). Don't reach for "TM can't" when the honest answer is "we haven't derived the pattern yet."
0 commit comments