You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Restructure README "How it measures up": one map over both scoreboards, data-only coverage region
The README presented the same "Monogram vs official highlighter" comparison as two
numbers that read as a contradiction — the coverage table (broad-corpus agreement, e.g.
TypeScript 99.2% vs 99.3%, a near-tie) sat 30 lines apart from the issue ledger (filed
bugs, e.g. 26/26 vs 8/26, a blowout), with nothing telling the reader they measure
different things. Plus the coverage table's preamble was a 7-line block of nested
parentheticals, and the load-bearing maintained-vs-dead-bundle context was buried in a
small-print <sub>.
- Co-locate both scoreboards under one `## How it measures up` section, led by a short
MAP: the broad table is the easy-token floor; the filed-bug ledger is the ambiguous
frontier; a near-tie up top next to a lopsided ledger is the expected shape, not a
contradiction. (Renames `## Comparison`; updates the one in-doc anchor that pointed at it.)
- Root-cause the dense preamble: the narrative was trapped INSIDE the generated coverage
region. `coverage-table.ts` now emits DATA only (the table); the agree/covered/Highlighter
legend is hand-written prose above the markers, so it can be reworded without a regen.
- Promote the maintained/unmaintained context out of <sub> into body text under the table.
- Trim the redundant example list from the ceiling's opening (the map already lists them).
Numbers are unchanged (regenerated identical); the issue ledger region is byte-for-byte
untouched. Hand-written prose carries no regenerated figure, so nothing can drift stale.
Copy file name to clipboardExpand all lines: README.md
+31-24Lines changed: 31 additions & 24 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -11,26 +11,6 @@ Write a language's grammar **once**, as an executable definition. Monogram runs
11
11
-**HTML** ([`html.ts`](html.ts)) — the engine reaching *past token streams into markup*; ~95 lines, validated against [`parse5`](https://github.com/inikulin/parse5).
12
12
-**Vue** ([`vue.ts`](vue.ts)) — a dialect of `html.ts`: SFC blocks that embed Monogram's own TS/JS/CSS, plus directives and `{{ }}` interpolation.
13
13
14
-
<!-- coverage:start -->
15
-
Per-grammar comparison vs the **official parser** as the neutral oracle (`node test/coverage-table.ts --write`).
16
-
17
-
**Parser** — Monogram's parser vs the official parser (`test/src-coverage.ts`). **agree** is the closeness number: Monogram and the official parser return the same verdict on each corpus file (both accept / both reject; **structural parse-tree equality** for HTML via parse5). **covered** is the share of the official parser's branches the corpus actually exercises — a blind-spot gauge; Monogram's behaviour on the uncovered remainder is untested, so read `agree` as "on the `covered` portion." For the non-HTML grammars `agree` is accept/reject, *not* tree-equality; their parse-**structure** correctness is exercised instead by the **Highlighter** axis below, whose token roles are read off the parse tree. (Each adapter's detailed output also prints a coverage-weighted branch-alignment %, which is more lenient than `agree`.)
18
-
19
-
**Highlighter** — Monogram's derived TextMate grammar vs the official one, both graded against the parser's token roles (`test/scope-gap.ts`); the [vscode#203212](https://github.com/microsoft/vscode/issues/203212) comparison.
20
-
21
-
| Grammar | Parser — agree · covered | Highlighter — Monogram vs official |
22
-
|---|---|---|
23
-
| TypeScript | 97.1% · 76.4% | 99.2% vs 99.3% |
24
-
| JavaScript | 96.3% · 65.5% | 99.0% vs 83.6% |
25
-
| JSX | 97.1% · 52.5% | 94.3% vs 94.3% |
26
-
| TSX | 96.7% · 65.7% | 95.6% vs 95.4% |
27
-
| HTML | 95.3% · 49.3% | 100.0% vs 98.8% |
28
-
| YAML | 100.0% · 73.9% | 100.0% vs 99.5% |
29
-
| Vue | — | 98.8% vs 98.0% |
30
-
<!-- coverage:end -->
31
-
32
-
<sub>**Which “official” grammar each row compares against:** HTML’s is the unmaintained [`textmate/html.tmbundle`](https://github.com/textmate/html.tmbundle) — the #203212 case Monogram targets. YAML’s is the maintained [RedCMD/YAML-Syntax-Highlighter](https://github.com/RedCMD/YAML-Syntax-Highlighter) that VS Code switched to ([microsoft/vscode#232244](https://github.com/microsoft/vscode/pull/232244)) — so YAML’s gap is Monogram vs a *maintained* grammar, not a dead bundle. JS/TS use Microsoft’s maintained [TypeScript-TmLanguage](https://github.com/microsoft/TypeScript-TmLanguage).</sub>
33
-
34
14
## Quick start
35
15
36
16
Requires Node 24+ (runs `.ts` directly — no build step, no `tsx`).
@@ -60,9 +40,36 @@ Take `typeof x < y`. A regex highlighter has to guess whether `<` opens a generi
60
40
61
41
That single source reaches across grammars, too: an embedded snippet runs *another Monogram grammar* — a `<script>` body is highlighted by Monogram's own JavaScript, so `<script>const x = 1 < 2</script>` colours `<` as a JS operator, the same ambiguity resolved *inside* the embed. Where VS Code's embeds fray — two independently-written grammars meeting with nothing checking the seam — Monogram owns both sides, so self-verifying that seam becomes possible (a design goal beyond today's standard `contentName` injection).
62
42
63
-
## Comparison
43
+
## How it measures up
44
+
45
+
Two numbers answer two different questions — read them together, not against each other:
46
+
47
+
-**Broad agreement** (the table just below) — over a whole corpus, does the derived grammar match the official **parser**'s accept/reject and the official **highlighter**'s token roles? This is dominated by the easy bulk of unambiguous tokens: the floor every grammar clears, not the interesting part.
48
+
-**The filed bugs** (the ledger under it) — on the exact cases reported against the hand-written official grammar, does the *derived* one fix them? This strips the easy bulk away and shows only the ambiguous frontier — generic-`<`-vs-less-than, regex-vs-division, whitespace-fragile multiline generics — where a parser-derived grammar pulls away from hand-tuned regex.
49
+
50
+
So the two aren't in tension: a near-tie in the broad table can sit right next to a lopsided ledger — the broad average dilutes the difference with easy tokens, while the ledger zooms in on the hard cases it buries.
51
+
52
+
### Broad agreement vs the official grammar
53
+
54
+
**Parser** (Monogram vs the official parser, [`test/src-coverage.ts`](test/src-coverage.ts)) — **agree** = the same accept/reject verdict on each corpus file (for HTML, full **parse-tree equality** via parse5); **covered** = how much of the official parser's own branches the corpus exercises, so read `agree` as "on the covered portion." (For the non-HTML grammars `agree` is accept/reject; their parse-*tree* correctness is exercised by the Highlighter axis, whose roles are read off the tree.) **Highlighter** (Monogram's derived TextMate grammar vs the official one, [`test/scope-gap.ts`](test/scope-gap.ts)) — both graded against the parser's per-token roles, the [vscode#203212](https://github.com/microsoft/vscode/issues/203212) comparison.
55
+
56
+
<!-- coverage:start -->
57
+
| Grammar | Parser — agree · covered | Highlighter — Monogram vs official |
58
+
|---|---|---|
59
+
| TypeScript | 97.1% · 76.4% | 99.2% vs 99.3% |
60
+
| JavaScript | 96.3% · 65.5% | 99.0% vs 83.6% |
61
+
| JSX | 97.1% · 52.5% | 94.3% vs 94.3% |
62
+
| TSX | 96.7% · 65.7% | 95.6% vs 95.4% |
63
+
| HTML | 95.3% · 49.3% | 100.0% vs 98.8% |
64
+
| YAML | 100.0% · 73.9% | 100.0% vs 99.5% |
65
+
| Vue | — | 98.8% vs 98.0% |
66
+
<!-- coverage:end -->
67
+
68
+
Measured against the *maintained* official grammar where it matters, not a dead bundle: JS/TS use Microsoft's maintained [TypeScript-TmLanguage](https://github.com/microsoft/TypeScript-TmLanguage); YAML uses the maintained [RedCMD/YAML-Syntax-Highlighter](https://github.com/RedCMD/YAML-Syntax-Highlighter) that VS Code switched to ([microsoft/vscode#232244](https://github.com/microsoft/vscode/pull/232244)); only HTML's baseline is the unmaintained [textmate/html.tmbundle](https://github.com/textmate/html.tmbundle) — the #203212 case Monogram targets.
69
+
70
+
### Every bug filed against the official grammar
64
71
65
-
The same question, every language at once: take the bugs reported against each *hand-written* official grammar and ask whether the *derived* grammar solves them. Which does **only** the official solve, which does **only** Monogram solve — and which do **both** still get wrong (the shared frontier neither reaches today)?
72
+
Take the bugs reported against each *hand-written* official grammar and ask whether the *derived* grammar solves them— and which **both** still get wrong (the shared frontier neither reaches today).
66
73
67
74
<!-- issues:start -->
68
75
<!-- generated by `npm run bench:issues` — do not edit by hand -->
@@ -216,7 +223,7 @@ _No asymmetries — both grammars handle all 8 filed bugs below._
216
223
217
224
### The ceiling — and the bar for claiming it
218
225
219
-
Deriving from a proven parser wins the disambiguation that is*TextMate-expressible but infeasible to hand-write* — regex-vs-division, generic-vs-comparison, whitespace-fragile multiline generics — the **only-Monogram** column. The **both-miss** cases are ones neither grammar gets *today* — not, by default, ones TextMate *can't*.
226
+
The **only-Monogram** wins above are all disambiguations that are*TextMate-expressible but infeasible to hand-write* — a parser supplies the pattern a human can't reliably guess. The **both-miss** cases are ones neither grammar gets *today* — not, by default, ones TextMate *can't*.
220
227
221
228
"TextMate can't express X" is not a guess or an assertion; it is a claim to be **proven from the model**. TextMate is a line-oriented matcher whose only cross-line memory is a finite stack of scope contexts, so a proof exhibits an X whose correct highlighting provably needs memory that model lacks — unbounded lookback to a token that is not an enclosing context. A failed *attempt* to derive a pattern is not such a proof: a cleverer pattern may exist, and most "impossible for TextMate" folklore is exactly this error — the multiline / nested-generic cases turn out TM-expressible once a parser supplies the pattern, which is why the derived grammar gets them right. Where a construct provably exceeds the model, Monogram's **tree-sitter** target — a real parser over the whole tree — resolves it.
222
229
@@ -338,7 +345,7 @@ The lexer, CST types, and all three highlighters fall out of step 1; a *dialect*
338
345
339
346
## Known differences from the official highlighter
340
347
341
-
A handful of token patterns are scoped differently from VS Code's official TypeScript grammar — all intentional, and in some Monogram is arguably *more* correct (these are *deliberate divergences*, distinct from the bug-class fixes the [ledger](#comparison) measures):
348
+
A handful of token patterns are scoped differently from VS Code's official TypeScript grammar — all intentional, and in some Monogram is arguably *more* correct (these are *deliberate divergences*, distinct from the bug-class fixes the [ledger](#every-bug-filed-against-the-official-grammar) measures):
342
349
343
350
| Token | Monogram | Official | Why we keep ours |
md+="Per-grammar comparison vs the **official parser** as the neutral oracle (`node test/coverage-table.ts --write`).\n\n**Parser** — Monogram's parser vs the official parser (`test/src-coverage.ts`). **agree** is the closeness number: Monogram and the official parser return the same verdict on each corpus file (both accept / both reject; **structural parse-tree equality** for HTML via parse5). **covered** is the share of the official parser's branches the corpus actually exercises — a blind-spot gauge; Monogram's behaviour on the uncovered remainder is untested, so read `agree` as \"on the `covered` portion.\" For the non-HTML grammars `agree` is accept/reject, *not* tree-equality; their parse-**structure** correctness is exercised instead by the **Highlighter** axis below, whose token roles are read off the parse tree. (Each adapter's detailed output also prints a coverage-weighted branch-alignment %, which is more lenient than `agree`.)\n\n**Highlighter** — Monogram's derived TextMate grammar vs the official one, both graded against the parser's token roles (`test/scope-gap.ts`); the [vscode#203212](https://github.com/microsoft/vscode/issues/203212) comparison.\n\n";
66
-
md+='| Grammar | Parser — agree · covered | Highlighter — Monogram vs official |\n|---|---|---|\n';
65
+
// The legend (agree / covered / Highlighter) is hand-written prose in the README, above the
66
+
// markers — this region is DATA only, so the narrative can be reworded without a regen.
67
+
md+='| Grammar | Parser — agree · covered | Highlighter — Monogram vs official |\n|---|---|---|\n';
0 commit comments