|
| 1 | +# Upstream issue ledger |
| 2 | + |
| 3 | +A maintained accounting of [microsoft/TypeScript-TmLanguage](https://github.com/microsoft/TypeScript-TmLanguage/issues) issues against Monogram: what we solve, what we don't, and — for what we don't — *why*, with an honest verdict per issue. |
| 4 | + |
| 5 | +**The honest frame.** Monogram **generates** a TextMate grammar, so anything genuinely impossible in the TextMate model is impossible in *our* TM output too. The advantage is not magic regex power — it is that we derive the TM patterns from a parser proven against the conformance suite, so we get *right* the disambiguation that is **TM-expressible but infeasible to hand-write**. That means an unsolved issue is usually one of: *(a)* in our backlog (TM-expressible, parser-derivable, not done yet), *(b)* out of our current grammar scope (TSX, JSDoc-content), *(c)* not a grammar's job at all (semantic tokens, theming, tooling), or *(d)* genuinely TM-format-impossible. Only **(d)** is "TM can't" — and we don't claim it without proof (see [the discipline](#classifying-an-unsolved-issue)). |
| 6 | + |
| 7 | +**As of 2026-05-29** — 106 open upstream issues. |
| 8 | + |
| 9 | +| Verdict | Count | Meaning | |
| 10 | +|---|---:|---| |
| 11 | +| ✅ **Solved** | **39** | parser-derived TM gets it right; replayed green in [`test/test-issues.ts`](../test/test-issues.ts) (288 checks). **All 39 are still open upstream.** | |
| 12 | +| 🔧 Backlog (our domain) | ~17 | TM-expressible + parser-derivable; not done yet. Some likely already correct, just untested. | |
| 13 | +| 📦 Out of scope (now) | 8 + 16 | TSX dialect (8); JSDoc *embedded-content* highlighting (16) — needs the embedded-grammar pipeline. | |
| 14 | +| 🧠 Needs semantics | ~6 | type/symbol resolution — **no syntax highlighter (TM or parser) can do this**; the language server's job. | |
| 15 | +| 🛠 Not a grammar concern | ~20 | theming, semantic-token config, scope-name data, tooling, IDE support. | |
| 16 | +| ⛔ Proven TM-impossible | 0 confirmed | none yet proven (requires a failing generated-TM + a passing parser-target — see discipline). | |
| 17 | + |
| 18 | +The headline: **Monogram already solves 39 of the 106 still-open official issues (~37%)** — and they are exactly the regex-vs-division / generic-vs-comparison / multiline-context class that a hand-written grammar perpetually loses to. |
| 19 | + |
| 20 | +--- |
| 21 | + |
| 22 | +## ✅ Solved (39) |
| 23 | + |
| 24 | +All replayed in [`test/test-issues.ts`](../test/test-issues.ts) and all still open upstream. Grouped by the ambiguity class — every one is something a regex grammar cannot reliably decide but a parser settles for free. |
| 25 | + |
| 26 | +**Regex literal vs. division `/`** — the lexer uses parser context (what the previous token *is*) to decide: |
| 27 | +`#853` (after `throw`/`void`/`typeof`/`await`/`yield`), `#804` (`/[a\-b]/` char class), `#1024` (regex on a new line), `#883` (`for (const x of /re/.exec…)`), `#1055` (backtick inside a regex), `#1063` (`\cJ`/`\cj` control-char escapes). |
| 28 | + |
| 29 | +**Generic type-args `<…>` vs. less-than `<`** — the parser knows when `<` opens type arguments: |
| 30 | +`#1050`, `#978` (`typeof x < y`), `#859` (`as` cast inside `<>`), `#884` (`0 < x && 1 > 0`), `#904` (`bad < obj.two`), `#1020` (`new Map<…>` with no parens), `#855` (`<` with a `/* comment */` inside the args). |
| 31 | + |
| 32 | +**Multiline / whitespace-fragile generics, types, unions** — structure, not column position, decides; a later construct never "breaks": |
| 33 | +`#1002`, `#1014`, `#1019`, `#1028`, `#1035`, `#1040`, `#1041`, `#1043`, `#1051`, `#1053`, `#1056`, `#1059`, `#819`, `#876`, `#889`, `#890`, `#894`, `#896`, `#911`, `#973`, `#981`, `#983`, `#873`. |
| 34 | + |
| 35 | +**Member / call scoping & control flow:** `#736` (method call → `entity.name.function`), `#770` (call parens are punctuation), `#869` (`x in obj ? a : b` ternary). |
| 36 | + |
| 37 | +> Each maps to one or more cases in `test/test-issues.ts`; grep the issue number there for the exact input + expected scopes. |
| 38 | +
|
| 39 | +--- |
| 40 | + |
| 41 | +## Classifying an unsolved issue |
| 42 | + |
| 43 | +When an issue is **not** solved, it gets exactly one verdict, and **⛔ TM-impossible requires evidence** — you may not assert it from intuition: |
| 44 | + |
| 45 | +1. **🔧 Backlog** — reproduce it; if a parser-derived TM pattern *can* express the fix, it's backlog. Fix it in `examples/typescript.ts`, add a case to `test/test-issues.ts`, move it to Solved. |
| 46 | +2. **📦 Out of scope** — the construct belongs to a dialect/region we don't generate yet (TSX, JSDoc body). |
| 47 | +3. **🧠 Needs semantics** — the correct scope depends on resolving a symbol's *kind* (type/value/namespace) or type relationships. See [Proof A](#proof-a). Out of scope for every grammar, ours included. |
| 48 | +4. **⛔ TM-impossible** — *only* when you can show **(i)** our own generated `tmLanguage.json` cannot express it (a concrete failing pattern) **and (ii)** our tree-sitter or Lezer target (real parsers) *does* get it right. See [Proof B](#proof-b). File the failing-TM + passing-parser pair as the proof. **Currently 0 issues have cleared this bar.** |
| 49 | + |
| 50 | +--- |
| 51 | + |
| 52 | +## 🔧 Backlog — our domain, solvable, not yet (≈17) |
| 53 | + |
| 54 | +TM-expressible parse/lex disambiguation, squarely in Monogram's wheelhouse. Several are probably already correct in the generated grammar and only need a test case to confirm and move to Solved: |
| 55 | + |
| 56 | +`#1071` `import.meta.dirname` (`import` mis-keyworded), `#1048` multiline `extends` in a generic with split parens, `#1027` color inside angle brackets, `#1025` `for-of`/`in` with no surrounding space, `#1021` regex `v` flag *(our `Regex` token already lists `v` — likely solved, verify)*, `#995` `(obj as keyof typeof X)` paren-wrapped assertion, `#994` default template argument color, `#992` cast on a variable named `type`, `#891` default import named `from` *(we parse `import type from "…"` — likely solved)*, `#881` `override` modifier, `#815` method named `new`, `#857` multiple `export default` overloads, `#497` `import` declaration / semicolon scoping, `#810` trailing newline in a line comment, `#788` optional chaining *(supported — verify)*, `#1039` `String.raw` escape tokens, `#1066` triple-slash reference directive. |
| 57 | + |
| 58 | +> Action: each should be run through the generated TM (the `test/test-issues.ts` harness) to confirm pass/fail before claiming. Listing here ≠ "broken in Monogram" — it means "not yet verified-and-tested." |
| 59 | +
|
| 60 | +## 📦 Out of current scope |
| 61 | + |
| 62 | +**TSX dialect (8)** — Monogram's grammar is TypeScript, not TSX. TSX layers the JSX dialect on top; a TSX grammar is future work using the same approach. Notably several are *generic-default-in-TSX* (`#1042`, `#990`, `#967`, `#979`, `#1033`) — the TS-side disambiguation Monogram already does; only the TSX wrapper is missing. Also `#1047` (Vue), `#908`, `#627`. |
| 63 | + |
| 64 | +**JSDoc embedded content (16)** — JSDoc is an `embed` region. Monogram marks it (`embed: 'jsdoc'` → TextMate `contentName`) but does not yet *generate* a JSDoc sub-grammar, so JSDoc-body highlighting is unsolved until the embedded-grammar pipeline (the ["Embedded languages" design goal](../README.md#embedded-languages)) is built. This is a *scope* gap, not a TM limitation — JSDoc highlighting is perfectly expressible in TM. Issues: `#1049 #1046 #1037 #1031 #1029 #1023 #993 #986 #965 #910 #877 #860 #856 #799 #693` (and JSDoc-flavored `#900`). |
| 65 | + |
| 66 | +## 🧠 Needs semantics — no syntax highlighter can do these |
| 67 | + |
| 68 | +Resolving these requires knowing a symbol's declared kind or type — see [Proof A](#proof-a). VS Code already covers them via the **semantic-token provider** (the TS language server), which is the correct layer; a grammar (TM *or* parser) is the wrong tool. `#939` (distinguish `import` *types* from values), `#645` (color `UPPER_CASED` as constant — a convention, not syntax), `#1038` (Symbol suggestion), parts of `#997`/`#879` (which scope a name *should* get). |
| 69 | + |
| 70 | +## 🛠 Not a grammar concern (≈20) |
| 71 | + |
| 72 | +Theming / token-color config (`#1064`, `#991`), scope-name data & spelling fixes (`#1034`, `#1031`, `#799`, `#879`, `#630`), and tooling / meta / IDE-support (`#1045` package-lock, `#742`, `#714`, `#462` Emacs, `#256` TextMate 2, `#693` example-body embedding, `#781` HTML comments, `#915` unicode identifiers, `#950`/`#988`/`#989`/`#1003`/`#1052`/`#691` "this file breaks" reports needing triage). Several "Other" reports are unreduced repros — they move to Backlog or Needs-semantics once minimized. |
| 73 | + |
| 74 | +--- |
| 75 | + |
| 76 | +## The impossibility proofs |
| 77 | + |
| 78 | +### Proof A |
| 79 | +**A purely syntactic highlighter — TextMate grammar *or* CST parser — cannot resolve a symbol's *kind*.** Both see only the token stream, never the resolved program. Whether `Foo` in `Foo.Bar` is a namespace or a class, whether an `import` binds a type or a value, whether `MAX` is a constant — all require name resolution / type-checking against declarations that may be in another file. Monogram's parser is a *concrete syntax tree* builder; it has no symbol table either. So semantic-coloring issues are unsolvable by **every** layer here — the job belongs to a semantic-token provider (language server). This is *not* a TextMate-specific weakness; it is the boundary of syntax highlighting itself. (Monogram's win is making the *syntactic* layer exact, so only the genuinely-semantic residue is left for the server.) |
| 80 | + |
| 81 | +### Proof B |
| 82 | +**A TextMate grammar is a line-oriented pushdown machine whose only inter-line memory is a finite stack of scope contexts**; each step is a single Oniguruma match (bounded even with `\g<>` subexpression recursion, which is per-line). Where correct highlighting requires context *not* captured by the enclosing-scope stack — an unbounded lookback to a token that is not an ancestor context — no TextMate grammar can express it, the official one or ours. Monogram's **tree-sitter and Lezer** targets are real incremental parsers holding the full tree, so they resolve such cases exactly; the generated TM target (like the official) can only approximate. |
| 83 | + |
| 84 | +> **But beware over-claiming this.** The 39 solved cases include heavy multiline / whitespace / nested-generic scenarios long *assumed* "impossible for TextMate" — they are not; they are TM-*expressible* once a parser tells you the right pattern. So this bucket is narrow, and we keep it empty until an issue is *proven* into it (failing generated-TM + passing parser-target). Don't reach for "TM can't" when the honest answer is "we haven't derived the pattern yet." |
| 85 | +
|
| 86 | +--- |
| 87 | + |
| 88 | +## Maintenance protocol |
| 89 | + |
| 90 | +- **Refresh the universe:** `gh issue list --repo microsoft/TypeScript-TmLanguage --state open --limit 300 --json number,title`. |
| 91 | +- **Solved set is the source of truth:** the issue numbers in `test/test-issues.ts` (`grep -oE '#[0-9]+' test/test-issues.ts | sort -u`). Solving an issue = add its repro case there (it must pass against the *generated* `tmLanguage.json`), then move it to [Solved](#-solved-39). |
| 92 | +- **Every unsolved issue carries one verdict** from [the discipline](#classifying-an-unsolved-issue). "⛔ TM-impossible" is the only verdict that needs attached evidence. |
| 93 | +- Counts above are a point-in-time snapshot; re-run the cross-reference (solved ∩ open) when updating. |
0 commit comments