|
| 1 | +# Monogram |
| 2 | + |
| 3 | +Define a language's grammar **once**. Monogram validates that grammar by running it as a real parser against the language's official conformance suite — then derives a **TextMate syntax highlighter** from the same, proven grammar. The highlighter's correctness flows down from a parser-verified model instead of up from regex guesswork. |
| 4 | + |
| 5 | +> *mono + grammar: one grammar, many derived artifacts.* |
| 6 | +
|
| 7 | +## The idea |
| 8 | + |
| 9 | +A TextMate grammar is a pile of regexes guessing at a language's structure. It's written by hand, independently of any parser, and it's perpetually wrong at the edges — VS Code's official TypeScript grammar carries [100+ open issues](https://github.com/microsoft/TypeScript-TmLanguage/issues) for exactly this reason. Everyone who tries to fix it is competing on the same losing axis: *who can hand-write better regexes.* |
| 10 | + |
| 11 | +Monogram inverts the dependency: |
| 12 | + |
| 13 | +1. **Write the grammar, then prove it.** The grammar is executable. Monogram runs it as a recursive-descent + Pratt parser over the TypeScript conformance suite. Today it parses **94.8%** (3579 / 3776 files); the goal is **100% coverage of the official parser** — at which point the grammar is a *verified, complete model* of the language's syntax, not an approximation. |
| 14 | + |
| 15 | +2. **Derive the highlighter from the proven grammar.** The TextMate grammar is generated from that same parser-validated grammar — never hand-written. Its correctness is underwritten by the parser conformance run, not by regex tuning. |
| 16 | + |
| 17 | +That's the whole point, and it's a categorical advantage, not an incremental one: a highlighter derived from a parser-complete grammar isn't *a better hand-written grammar* — it's playing a different game. You cannot out-regex it, because its correctness comes from a dimension hand-written grammars never operate in. Push the grammar to 100% parser coverage and the highlighter comes along for free, correct by construction. |
| 18 | + |
| 19 | +## What you get |
| 20 | + |
| 21 | +From one grammar definition (a small TypeScript combinator API): |
| 22 | + |
| 23 | +- **A CST parser** — recursive descent + Pratt operator precedence, producing a full-fidelity concrete syntax tree where every token is a node. |
| 24 | +- **A TextMate grammar** — a `.tmLanguage.json` for editor syntax highlighting, derived from the same rules the parser runs on. |
| 25 | + |
| 26 | +## Results |
| 27 | + |
| 28 | +Validated on TypeScript (grammar: [`examples/typescript.ts`](examples/typescript.ts), 521 lines): |
| 29 | + |
| 30 | +``` |
| 31 | +Parser conformance 94.8% 3579 / 3776 files from the TypeScript conformance suite (goal: 100%) |
| 32 | +Highlighter 99.3% 589 / 593 tokens match VS Code's official grammar |
| 33 | +Generated grammar 42 KB vs the official hand-written 226 KB |
| 34 | +Engine language-agnostic — no TypeScript-specific code |
| 35 | +``` |
| 36 | + |
| 37 | +The parser number is the one that matters: it's the grammar's correctness proof. Most remaining failures are intentional *error* tests that the TypeScript compiler also rejects. The highlighter accuracy is what that proof buys you — and the four remaining differences are deliberate (see [Known differences](#known-differences-from-the-official-highlighter)). |
| 38 | + |
| 39 | +## The grammar is the source of truth |
| 40 | + |
| 41 | +A grammar is a TypeScript module: tokens, operator precedence, and rules built from small combinators. A self-contained mini-example: |
| 42 | + |
| 43 | +```ts |
| 44 | +import { token, rule, defineGrammar, left, op, sep } from './src/api.ts'; |
| 45 | + |
| 46 | +const Ident = token(/[a-zA-Z_$][a-zA-Z0-9_$]*/, { identifier: true }); |
| 47 | +const Number = token(/[0-9]+(\.[0-9]+)?/); |
| 48 | + |
| 49 | +const Expr = rule($ => [ |
| 50 | + Ident, |
| 51 | + Number, |
| 52 | + [$, op, $], // binary operators (precedence declared below) |
| 53 | + [$, '(', sep(Expr, ','), ')'], // call: foo(a, b) |
| 54 | + [$, '.', Ident], // member: obj.name |
| 55 | +]); |
| 56 | + |
| 57 | +export default defineGrammar({ |
| 58 | + name: 'mini', |
| 59 | + tokens: { Ident, Number }, |
| 60 | + prec: [ left('+', '-'), left('*', '/') ], |
| 61 | + rules: { Expr }, |
| 62 | + entry: Expr, |
| 63 | +}); |
| 64 | +``` |
| 65 | + |
| 66 | +The parser uses these rules to build a CST. The highlighter reads the same rule **shapes** and infers scopes — with no manual scope assignment: |
| 67 | + |
| 68 | +- `foo(x)` → `foo` is `entity.name.function` (from the `$ '(' …` call form) |
| 69 | +- `obj.name` → `name` is `entity.other.property` (from the `$ '.' Ident` form) |
| 70 | +- `'class' Ident` → `Ident` is `entity.name.type` (from declaration structure) |
| 71 | +- `':' Type` → enter type-annotation highlighting (from the `type` rule flag) |
| 72 | +- `Expr '<' Type '>' '('` → a generic call, not a comparison (from rule structure) |
| 73 | + |
| 74 | +## A language-agnostic engine |
| 75 | + |
| 76 | +Nothing in the engine knows about TypeScript. Everything language-specific lives in the grammar — keywords, which token is the identifier, template-literal delimiters, and the regex-vs-division lexer ambiguity are all *declared per token*: |
| 77 | + |
| 78 | +```ts |
| 79 | +const Template = token(/`…`/, { template: { open: '`', interpOpen: '${', interpClose: '}' } }); |
| 80 | +const Regex = token(/\/…\//, { |
| 81 | + regex: true, |
| 82 | + regexContext: { |
| 83 | + divisionAfterTypes: ['Ident', 'Number', 'String', 'Template'], |
| 84 | + divisionAfterTexts: [')', ']', 'this', 'true', /* … */], |
| 85 | + regexAfterTexts: ['return', 'typeof', 'instanceof', /* … */], |
| 86 | + }, |
| 87 | +}); |
| 88 | +``` |
| 89 | + |
| 90 | +[`test/agnostic.ts`](test/agnostic.ts) proves it: the same engine parses a toy grammar whose identifier token is named `Word`, with no templates and no regex. Supporting a new language means writing a new grammar file, not changing the engine. |
| 91 | + |
| 92 | +## Embedded languages, without the broken seams |
| 93 | + |
| 94 | +Editors highlight embedded snippets — CSS in a template string, a regex literal, SQL in a query, JSDoc in a comment — by handing the region to another language's grammar at the boundary. In VS Code that only works if the host grammar and the embedded grammar, written independently by different plugin authors, *both* implement the boundary correctly. Nothing checks that the two halves agree, so embedded highlighting is flaky exactly at the seams. |
| 95 | + |
| 96 | +Monogram declares embedding points in the grammar itself (a token's `embed` annotation). When the languages on both sides are Monogram grammars, one system owns the whole boundary: it can generate the host and the embedded grammar together and exercise the seam in a single integrated self-test, so the boundary is *verified* rather than left to two strangers happening to agree. The cross-language robustness problem dissolves for the same reason the highlighter does — one source of truth, checked end to end. |
| 97 | + |
| 98 | +## Usage |
| 99 | + |
| 100 | +Requires Node 24+ (runs `.ts` directly — no build step, no `tsx`). |
| 101 | + |
| 102 | +```bash |
| 103 | +npm install |
| 104 | + |
| 105 | +# generate the TextMate grammar from the grammar definition |
| 106 | +node src/cli.ts examples/typescript.ts # → examples/typescript.tmLanguage.json |
| 107 | +``` |
| 108 | + |
| 109 | +Parse some source into a CST: |
| 110 | + |
| 111 | +```ts |
| 112 | +import { createParser } from './src/gen-parser.ts'; |
| 113 | +import grammar from './examples/typescript.ts'; |
| 114 | + |
| 115 | +const { parse } = createParser(grammar); |
| 116 | +const cst = parse('const x = f(a, b)'); // → concrete syntax tree |
| 117 | +``` |
| 118 | + |
| 119 | +Tests: |
| 120 | + |
| 121 | +```bash |
| 122 | +node test/sanity-check.ts # quick smoke test |
| 123 | +node test/run-conformance.ts # parser vs the TypeScript conformance suite — the correctness proof |
| 124 | +node test/coverage.ts # highlighter vs VS Code's official grammar |
| 125 | +node test/agnostic.ts # proves the engine is language-agnostic |
| 126 | +``` |
| 127 | + |
| 128 | +## Known differences from the official highlighter |
| 129 | + |
| 130 | +On the comparison sample, **4 tokens** are scoped differently from VS Code's official TypeScript grammar. All are intentional — in some, Monogram is arguably *more* correct: |
| 131 | + |
| 132 | +| Token | Monogram | Official | Why we keep ours | |
| 133 | +|---|---|---|---| |
| 134 | +| `console` in `console.log` | `support.variable` | `variable.other.object` | We highlight built-in globals (`console`, `window`, …) distinctly — a deliberate, common choice. | |
| 135 | +| `transform` (a function parameter) | `variable.parameter` | `entity.name.function` | It **is** a parameter. Official's heuristic mis-reads `name: (…) => T` as a function definition; we're more correct. | |
| 136 | +| `error` (the method in `console.error(…)`) | `entity.name.function` | `variable.other` | We scope a called method as a function name — arguably more informative. | |
| 137 | + |
| 138 | +> Built-in class names in **type** position (e.g. `Error` in `extends Error`) now correctly emit `entity.name.type`, matching official; in **value** position (`new Error()`) they remain `support.class`, also matching official. |
| 139 | +
|
| 140 | +Matching the official grammar *exactly* would, in cases like `transform`, make the output worse. The metric counts these as differences, not defects. |
| 141 | + |
| 142 | +## Architecture |
| 143 | + |
| 144 | +``` |
| 145 | +examples/typescript.ts one grammar (TypeScript combinator API) |
| 146 | + │ |
| 147 | + ├─ src/gen-parser.ts ───▶ CST parser (recursive descent + Pratt + packrat memo) |
| 148 | + │ run against the conformance suite = the grammar's proof |
| 149 | + │ |
| 150 | + └─ src/gen-tm.ts ───────▶ typescript.tmLanguage.json (TextMate highlighter) |
| 151 | +
|
| 152 | +shared src/grammar-utils.ts structural helpers used by both |
| 153 | + src/api.ts, types.ts the grammar's combinator + type surface |
| 154 | +``` |
| 155 | + |
| 156 | +- **One grammar, two consumers.** `gen-parser` interprets the rules to parse; `gen-tm` reads the same rule *shapes* to derive TextMate patterns. They share structural primitives (`grammar-utils.ts`) — e.g. a single keyword/punctuation predicate — so they classify tokens identically. |
| 157 | +- **CST, not AST.** The parser keeps every token (punctuation, keywords) as a node — required for the highlighter and for lossless source reconstruction. Roughly 2× the nodes of an AST, by design. |
| 158 | +- **Both stages are language-agnostic.** All language specifics live in the grammar; the engine is a generic, reusable runtime. |
| 159 | + |
| 160 | +## Prior art |
| 161 | + |
| 162 | +| Tool | Parser | Highlighting | Single source | |
| 163 | +|------|:---:|:---:|:---:| |
| 164 | +| TextMate grammars | — | manual regex | — | |
| 165 | +| tree-sitter | yes | queries (separate) | — | |
| 166 | +| ANTLR | yes | — | — | |
| 167 | +| Langium | yes | Monarch (separate) | — | |
| 168 | +| ungrammar | AST types | — | — | |
| 169 | +| **Monogram** | **CST (94.8% → 100%)** | **auto-derived (99.3%)** | **yes** | |
0 commit comments