Skip to content

Commit 3165780

Browse files
committed
README: add source-size replacement, issues-solved, and a contribution guide
Three gaps the README had: 1. Source replacement: one 628-line grammar (32 KB) replaces the official 3331-line hand-written TextMate YAML (140 KB) — 5x less — and also yields a parser. 2. Issues solved with less complexity: test/test-issues.ts replays 39 real TypeScript-TmLanguage tracker bugs (288 token checks), all pass, because the regex-ambiguity failure modes are structurally precluded by a real parser. 3. 'Adding a language' section: the contribution workflow is grammar-first and test-driven — write the grammar, PROVE it bidirectionally against the language's own conformance suite, and bring the official TextMate grammar as the coverage baseline. A new-language PR is judged on parser conformance + highlighter coverage.
1 parent 48556a3 commit 3165780

1 file changed

Lines changed: 14 additions & 1 deletion

File tree

README.md

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -40,10 +40,13 @@ Validated on TypeScript (grammar: [`examples/typescript.ts`](examples/typescript
4040
Valid-code coverage 100% 3376 / 3376 valid single-file conformance cases parse (zero gaps)
4141
Bidirectional 97.8% 3585 / 3664 — also rejects what tsc rejects (gap = over-acceptance; most is TS semantic-only rejection)
4242
Highlighter 99.3% 589 / 593 tokens match VS Code's official grammar
43-
Generated grammar 42 KB vs the official hand-written 226 KB
43+
Official-grammar bugs 39 real TypeScript-TmLanguage issues replayed (288 token checks) — all pass
44+
Source size 628 ln one grammar file (32 KB) replaces 3331 lines of hand-written TextMate YAML (140 KB) — 5× less, and it also yields the parser
4445
Engine language-agnostic — no TypeScript-specific code
4546
```
4647

48+
Read those last two lines together — they're the whole argument. **One 628-line grammar replaces the official 3331-line hand-written TextMate YAML at a fifth the size** *and* throws in a conformance-proven parser the official grammar never had. **Less to maintain, and demonstrably more correct:** [`test/test-issues.ts`](test/test-issues.ts) replays **39 real bugs** from the [official grammar's issue tracker](https://github.com/microsoft/TypeScript-TmLanguage/issues) — the `typeof x < y` generic-vs-comparison ambiguities, regex-after-keyword cases, `as`-cast-in-`<>` confusions — and all 288 token checks pass, because those failure modes are *structurally* precluded by a real parser rather than patched one regex at a time.
49+
4750
The parser number is the one that matters: it's the grammar's correctness proof. Most remaining failures are intentional *error* tests that the TypeScript compiler also rejects. The highlighter accuracy is what that proof buys you — and the four remaining differences are deliberate (see [Known differences](#known-differences-from-the-official-highlighter)).
4851

4952
## The grammar is the source of truth
@@ -99,6 +102,16 @@ const Regex = token(/\/…\//, {
99102

100103
[`test/agnostic.ts`](test/agnostic.ts) proves it: the same engine parses a toy grammar whose identifier token is named `Word`, with no templates and no regex. Supporting a new language means writing a new grammar file, not changing the engine.
101104

105+
## Adding a language
106+
107+
A new language is **one grammar file, proven the way TypeScript is** — by its own parser conformance, not by eyeballing colors. The contribution workflow is grammar-first and test-driven:
108+
109+
1. **Write the grammar** with the combinator API ([`src/api.ts`](src/api.ts)) — tokens, operator precedence, rules. Everything language-specific lives here; you never touch the engine.
110+
2. **Prove it as a parser** against the language's *own* official test suite. Point the conformance harness at that corpus and measure **bidirectionally** — accept what the reference accepts **and** reject what it rejects (`tsc`'s `parseDiagnostics` for TypeScript; the equivalent for another language). The grammar is "ready" when valid-code coverage hits 100% and bidirectional agreement is high. That run *is* the correctness proof — the same standard TypeScript is held to here, not a lower one.
111+
3. **Bring the reference highlighter as the comparison baseline.** Drop in the language's existing official TextMate grammar so [`test/coverage.ts`](test/coverage.ts) can diff the *derived* highlighter against a real-world target. Coverage is then *measured against the thing you're replacing*, not asserted — and every divergence is either a win to keep or a gap to close, on the record.
112+
113+
The highlighter, lexer, tree-sitter / Lezer / Monarch grammars, and CST types all fall out of step 1 for free; steps 2–3 are how the contribution *earns trust*. So a new-language PR is reviewed on exactly two numbers: its **parser conformance** (the proof) and its **highlighter coverage vs the official grammar** (the payoff) — the same two this README leads with for TypeScript.
114+
102115
## Embedded languages, without the broken seams
103116

104117
Editors highlight embedded snippets — CSS in a template string, a regex literal, SQL in a query, JSDoc in a comment — by handing the region to another language's grammar at the boundary. In VS Code that only works if the host grammar and the embedded grammar, written independently by different plugin authors, *both* implement the boundary correctly. Nothing checks that the two halves agree, so embedded highlighting is flaky exactly at the seams.

0 commit comments

Comments
 (0)