Skip to content

Commit 15afd7d

Browse files
johnsoncodehkjohnsoncodehk
committed
Generate a VS Code language configuration from the grammar
gen-vscode-config.ts derives a language-configuration.json (comments, bracket pairs, auto-closing/surrounding pairs, colorized brackets, autoCloseBefore, folding markers, word pattern, indentation, and comment-continuation onEnter rules) from the grammar's tokens and scopes. cli.ts emits it next to the TextMate grammar; CI's gen-sync check covers both. Comments, folding, the /** */ pair, brackets and auto-close all match VS Code's hand-written TypeScript config; the remaining differences are control-flow indentation that isn't derivable from a grammar. A 'string' token flag lets string delimiters be derived rather than hardcoded, so the generator stays language-agnostic (test/agnostic.ts guards it).
1 parent 699ec0e commit 15afd7d

9 files changed

Lines changed: 410 additions & 10 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -26,10 +26,10 @@ jobs:
2626

2727
# Regenerate the TextMate grammar and fail if it drifts from the committed file
2828
# (i.e. someone edited the grammar but forgot to regenerate).
29-
- name: Generate highlighter (must be in sync)
29+
- name: Generate editor artifacts (must be in sync)
3030
run: |
3131
node src/cli.ts examples/typescript.ts
32-
git diff --exit-code -- examples/typescript.tmLanguage.json
32+
git diff --exit-code -- examples/typescript.tmLanguage.json examples/typescript.language-configuration.json
3333
3434
# Self-contained suites (each exits non-zero on failure). The conformance /
3535
# coverage / bench tools are excluded here: they need the external TypeScript

‎README.md‎

Lines changed: 11 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ From one grammar definition (a small TypeScript combinator API):
2323
- **A lexer** — tokenizes source straight from the grammar's token definitions; usable on its own (`createLexer(grammar).tokenize`).
2424
- **A CST parser** — recursive descent + Pratt operator precedence on top of the lexer, producing a full-fidelity concrete syntax tree where every token is a node.
2525
- **A TextMate grammar** — a `.tmLanguage.json` for editor syntax highlighting, derived from the same rules.
26+
- **A VS Code language configuration** — a `language-configuration.json` (comments, bracket pairs, auto-closing/surrounding pairs, folding, indentation) for editor behavior.
2627

2728
## Results
2829

@@ -143,20 +144,22 @@ Matching the official grammar *exactly* would, in cases like `transform`, make t
143144
## Architecture
144145

145146
```
146-
examples/typescript.ts one grammar (TypeScript combinator API)
147+
examples/typescript.ts one grammar (TypeScript combinator API)
147148
│
148-
├─ src/gen-lexer.ts ───▶ lexer → tokens (standalone: createLexer)
149+
├─ src/gen-lexer.ts ───────▶ lexer → tokens (standalone: createLexer)
149150
│ ▲ composed by
150-
├─ src/gen-parser.ts ───▶ CST parser (recursive descent + Pratt + packrat memo;
151-
│ run against the conformance suite = the grammar's proof)
151+
├─ src/gen-parser.ts ───────▶ CST parser (recursive descent + Pratt + packrat memo;
152+
│ run against the conformance suite = the grammar's proof)
152153
│
153-
└─ src/gen-tm.ts ───────▶ typescript.tmLanguage.json (TextMate highlighter)
154+
├─ src/gen-tm.ts ───────────▶ typescript.tmLanguage.json (TextMate highlighter)
155+
│
156+
└─ src/gen-vscode-config.ts ▶ typescript.language-configuration.json (editor behavior)
154157
155-
shared src/grammar-utils.ts structural helpers used across stages
156-
src/api.ts, types.ts the grammar's combinator + type surface
158+
shared src/grammar-utils.ts structural helpers used across stages
159+
src/api.ts, types.ts the grammar's combinator + type surface
157160
```
158161

159-
- **One grammar, three derived stages.** `gen-lexer` builds a tokenizer from the token definitions + lexer hints; `gen-parser` composes that lexer and interprets the rules to build a CST; `gen-tm` reads the same rule *shapes* to derive TextMate patterns. Shared structural primitives (`grammar-utils.ts`) — e.g. one keyword/punctuation predicate — keep them classifying tokens identically.
162+
- **One grammar, many derived artifacts.** `gen-lexer` builds a tokenizer from the token definitions + lexer hints; `gen-parser` composes that lexer and interprets the rules to build a CST; `gen-tm` reads the same rule *shapes* to derive TextMate patterns; `gen-vscode-config` derives the editor config (comments, brackets, auto-close) from the same tokens and `scopes`. Shared structural primitives (`grammar-utils.ts`) — e.g. one keyword/punctuation predicate — keep them consistent.
160163
- **CST, not AST.** The parser keeps every token (punctuation, keywords) as a node — required for the highlighter and for lossless source reconstruction. Roughly 2× the nodes of an AST, by design.
161164
- **Every stage is language-agnostic.** All language specifics live in the grammar; lexer, parser and generator are generic, reusable runtimes.
162165

‎examples/typescript.language-configuration.json‎

Lines changed: 179 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,179 @@
1+
{
2+
"comments": {
3+
"lineComment": "//",
4+
"blockComment": [
5+
"/*",
6+
"*/"
7+
]
8+
},
9+
"brackets": [
10+
[
11+
"${",
12+
"}"
13+
],
14+
[
15+
"(",
16+
")"
17+
],
18+
[
19+
"{",
20+
"}"
21+
],
22+
[
23+
"[",
24+
"]"
25+
]
26+
],
27+
"colorizedBracketPairs": [
28+
[
29+
"(",
30+
")"
31+
],
32+
[
33+
"{",
34+
"}"
35+
],
36+
[
37+
"[",
38+
"]"
39+
],
40+
[
41+
"<",
42+
">"
43+
]
44+
],
45+
"autoClosingPairs": [
46+
{
47+
"open": "${",
48+
"close": "}"
49+
},
50+
{
51+
"open": "(",
52+
"close": ")"
53+
},
54+
{
55+
"open": "{",
56+
"close": "}"
57+
},
58+
{
59+
"open": "[",
60+
"close": "]"
61+
},
62+
{
63+
"open": "\"",
64+
"close": "\"",
65+
"notIn": [
66+
"string",
67+
"comment"
68+
]
69+
},
70+
{
71+
"open": "'",
72+
"close": "'",
73+
"notIn": [
74+
"string",
75+
"comment"
76+
]
77+
},
78+
{
79+
"open": "`",
80+
"close": "`",
81+
"notIn": [
82+
"string",
83+
"comment"
84+
]
85+
},
86+
{
87+
"open": "/**",
88+
"close": " */",
89+
"notIn": [
90+
"string"
91+
]
92+
}
93+
],
94+
"surroundingPairs": [
95+
[
96+
"${",
97+
"}"
98+
],
99+
[
100+
"(",
101+
")"
102+
],
103+
[
104+
"{",
105+
"}"
106+
],
107+
[
108+
"[",
109+
"]"
110+
],
111+
[
112+
"\"",
113+
"\""
114+
],
115+
[
116+
"'",
117+
"'"
118+
],
119+
[
120+
"`",
121+
"`"
122+
],
123+
[
124+
"<",
125+
">"
126+
]
127+
],
128+
"autoCloseBefore": "})]>=.;,:` \n\t",
129+
"folding": {
130+
"markers": {
131+
"start": "^\\s*//\\s*#?region\\b",
132+
"end": "^\\s*//\\s*#?endregion\\b"
133+
}
134+
},
135+
"wordPattern": "(?:[a-zA-Z_$]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})(?:[a-zA-Z0-9_$]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})*",
136+
"indentationRules": {
137+
"decreaseIndentPattern": "^\\s*[)}\\]].*$",
138+
"increaseIndentPattern": "^.*(\\([^)]*|\\{[^}]*|\\[[^\\]]*)$"
139+
},
140+
"onEnterRules": [
141+
{
142+
"beforeText": "^\\s*/\\*\\*(?!/)([^*]|\\*(?!/))*$",
143+
"afterText": "^\\s*\\*/$",
144+
"action": {
145+
"indent": "indentOutdent",
146+
"appendText": " * "
147+
}
148+
},
149+
{
150+
"beforeText": "^\\s*/\\*\\*(?!/)([^*]|\\*(?!/))*$",
151+
"action": {
152+
"indent": "none",
153+
"appendText": " * "
154+
}
155+
},
156+
{
157+
"beforeText": "^(\\t|[ ])*\\*([ ]([^*]|\\*(?!/))*)?$",
158+
"action": {
159+
"indent": "none",
160+
"appendText": "* "
161+
}
162+
},
163+
{
164+
"beforeText": "^(\\t|[ ])*[ ]\\*/\\s*$",
165+
"action": {
166+
"indent": "none",
167+
"removeText": 1
168+
}
169+
},
170+
{
171+
"beforeText": "(?<!\\\\|\\w:)//\\s*\\S",
172+
"afterText": "^(?!\\s*$).+",
173+
"action": {
174+
"indent": "none",
175+
"appendText": "// "
176+
}
177+
}
178+
]
179+
}

‎examples/typescript.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ const BinaryNumber = token(/0[bB][01]+(_[01]+)*/, { s
1414
const BigInt_ = token(/[0-9]+(_[0-9]+)*n/, { scope: 'constant.numeric.bigint' });
1515
const Number_ = token(/[0-9]+(_[0-9]+)*(?:\.[0-9]*(_[0-9]+)*)?(?:[eE][+-]?[0-9]+)?/);
1616
const String_ = token(/"(?:[^"\\]|\\[\s\S])*"|'(?:[^'\\]|\\[\s\S])*'/, {
17+
string: true,
1718
escape: /\\(?:[nrtbfv0'"\\]|x[0-9a-fA-F]{2}|u[0-9a-fA-F]{4}|u\{[0-9a-fA-F]+\})/,
1819
});
1920
const Template = token(/`(?:[^`\\$]|\\.|\$(?!\{))*`/, {

‎src/api.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ interface TokenOptions {
1616
divisionAfterTexts?: string[];
1717
regexAfterTexts?: string[];
1818
};
19+
string?: boolean;
1920
}
2021

2122
export class TokenRef {
@@ -284,6 +285,7 @@ export function defineGrammar(config: GrammarConfig): CstGrammar & { name: strin
284285
divisionAfterTexts: tok.opts.regexContext.divisionAfterTexts ?? [],
285286
regexAfterTexts: tok.opts.regexContext.regexAfterTexts ?? [],
286287
},
288+
string: tok.opts.string,
287289
};
288290
});
289291

‎src/cli.ts‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
import { writeFileSync } from 'node:fs';
22
import { basename, dirname, join, resolve } from 'node:path';
33
import { generateTmLanguage } from './gen-tm.ts';
4+
import { generateLanguageConfig } from './gen-vscode-config.ts';
45
import type { CstGrammar, RuleExpr } from './types.ts';
56

67
const file = process.argv[2];
@@ -46,6 +47,12 @@ const outPath = join(dirname(file), `${langName}.tmLanguage.json`);
4647
writeFileSync(outPath, JSON.stringify(tm, null, 2) + '\n');
4748
console.log(`\n→ Generated ${outPath}`);
4849

50+
// Generate VS Code language configuration (editor behaviors)
51+
const langConfig = generateLanguageConfig(grammar);
52+
const cfgPath = join(dirname(file), `${langName}.language-configuration.json`);
53+
writeFileSync(cfgPath, JSON.stringify(langConfig, null, 2) + '\n');
54+
console.log(`→ Generated ${cfgPath}`);
55+
4956
function formatExpr(expr: RuleExpr): string {
5057
switch (expr.type) {
5158
case 'literal': return `'${expr.value}'`;

0 commit comments

Comments
 (0)