Skip to content

Commit 00d9cff

Browse files
committed
tree-sitter: capture destructuring rename keys (95.3% -> 95.9%)
A `{ key: binding }` destructuring property names a property, but a blanket `(binding_property (ident) …)` capture would also hit a shorthand `{ a }`, where the identifier IS the binding (that mistake cost 14 tokens in an earlier attempt). Detect rules with a `[ identifier, ':', ruleRef ]` branch and anchor the @Property capture on the trailing element rule, so shorthand bindings stay @variable. Skip the self-referential `Stmt: Ident ':' Stmt` shape — a labeled statement is a jump target, not a destructuring key. This closes the last cleanly-structural gap: 95.9% vs official tree-sitter 92.7%. The ~22 remaining misses are parser-level (the `<`/`>` comparison-vs-generic ambiguity and template tails, which the official grammars also miss) or niche edge positions (template-literal types, `typeof X` in a type, tuple element labels) where a query pattern would mis-paint conditional-type idents. Regenerate TS + JS artifacts.
1 parent e9ca26c commit 00d9cff

4 files changed

Lines changed: 32 additions & 2 deletions

File tree

README.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -77,7 +77,7 @@ TypeScript — token-family accuracy (higher = more correct)
7777
<sub>Graded over 102 ambiguity-rich snippets ([`test/issue-cases.ts`](test/issue-cases.ts)) against tsc's own parse tree. The three parser-based highlighters (Monogram, tree-sitter, Lezer) cluster well above the hand-written TextMate grammar — that gap is the whole point. Per-engine vocabulary→family maps (frozen, auditable): [`test/highlight-engines.ts`](test/highlight-engines.ts). JavaScript: not yet on the bench. Regenerate: `npm run bench:readme`.</sub>
7878
<!-- bench:end -->
7979

80-
> **Why the Monogram row is its *TextMate* output.** That's the one CI can rebuild on every commit. Monogram *also* derives a full **tree-sitter** grammar + query from the same definition; built locally to wasm and graded through this *same* `tsc` oracle, it scores **95.3%** — **above the official parsers** (tree-sitter 92.7%, Lezer 89.5%) and both TextMate grammars (its own 87.6%, official 76.2%). The lift is *structural*: every capture is derived from grammar shape — type references by node position (`(type (ident) @type)`), member keys from the member-name rule, type parameters from the `< … >` binder, declaration names keyword-anchored — never a hardcoded vocabulary, so it never paints a value as a type. It's opt-in (needs a local wasm build), hence out of the CI chart: `MONOGRAM_TS_WASM=… node test/highlight-bench.ts --corpus adversarial --engines`. *(There is no Monogram-**Lezer** number: the grammar is expressive enough to trip Lezer's LR(1) shift/reduce wall — it parses under tree-sitter's GLR but won't build under Lezer.)*
80+
> **Why the Monogram row is its *TextMate* output.** That's the one CI can rebuild on every commit. Monogram *also* derives a full **tree-sitter** grammar + query from the same definition; built locally to wasm and graded through this *same* `tsc` oracle, it scores **95.9%** — **above the official parsers** (tree-sitter 92.7%, Lezer 89.5%) and both TextMate grammars (its own 87.6%, official 76.2%). The lift is *structural*: every capture is derived from grammar shape — type references by node position (`(type (ident) @type)`), member keys from the member-name rule, type parameters from the `< … >` binder, declaration names keyword-anchored — never a hardcoded vocabulary, so it never paints a value as a type. It's opt-in (needs a local wasm build), hence out of the CI chart: `MONOGRAM_TS_WASM=… node test/highlight-bench.ts --corpus adversarial --engines`. *(There is no Monogram-**Lezer** number: the grammar is expressive enough to trip Lezer's LR(1) shift/reduce wall — it parses under tree-sitter's GLR but won't build under Lezer.)*
8181

8282
> **The other side of the ledger (honesty check).** On the *broad* TS parser-conformance corpus — not just the documented bugs — the two are now neck-and-neck: token-role accuracy is **tied at ~99%**, Monogram **leads on per-cell coverage** (100% of strict cells vs official's ~97%), and whole-file-perfect snippets are essentially level (~88% vs ~89%). The small residual is niche multiline type-annotation scoping, **not** the ambiguity class above. Clone the TS corpus to `/tmp/ts-repo` and run `node test/highlight-bench.ts` to see both corpora.
8383
@@ -93,7 +93,7 @@ From one grammar definition (a small TypeScript combinator API), three outputs a
9393

9494
And — from the same grammar — generators for the rest of the ecosystem, at varying maturity:
9595

96-
- **tree-sitter**`grammar.js` + a **structural** `queries/highlights.scm` + an external scanner for context-sensitive lexing. Builds end-to-end (tree-sitter's GLR absorbs the grammar; compiles to wasm) and the *derived* query scores **95.3%** through the same oracle as the chart above — **above the official parsers** (tree-sitter 92.7%, Lezer 89.5%). Opt-in (needs a local wasm build), so it stays out of the CI chart.
96+
- **tree-sitter**`grammar.js` + a **structural** `queries/highlights.scm` + an external scanner for context-sensitive lexing. Builds end-to-end (tree-sitter's GLR absorbs the grammar; compiles to wasm) and the *derived* query scores **95.9%** through the same oracle as the chart above — **above the official parsers** (tree-sitter 92.7%, Lezer 89.5%). Opt-in (needs a local wasm build), so it stays out of the CI chart.
9797
- **Lezer** — a CodeMirror 6 grammar + `styleTags` + a JS tokenizer. Does **not** build: the grammar trips Lezer's **LR(1) shift/reduce wall** (TypeScript's `<` type-vs-expression ambiguity, which `tsc` resolves with semantic information LR(1) lacks). That's a ceiling of Lezer's parser class rather than a gap in the grammar — so there is no derived Lezer highlighter number.
9898
- **Monarch** — a Monaco (web) tokenizer (functional, bounded by JS-regex limits).
9999

examples/tree-sitter/javascript/queries/highlights.scm

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@
2424
;; Structural member / type-param / property-access captures.
2525
(member_name (ident) @property)
2626
(expr (expr) (ident) @property)
27+
(binding_property (ident) @property (binding_element))
2728

2829
;; Literal token nodes.
2930
(shebang) @comment

examples/tree-sitter/typescript/queries/highlights.scm

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@
3333
(member_name (ident) @property)
3434
(type_member (ident) @property)
3535
(expr (expr) (ident) @property)
36+
(binding_property (ident) @property (binding_element))
3637

3738
;; Enum-like value members (override member-key, which would say @property).
3839
(enum_member (member_name (ident) @variable))

src/gen-treesitter.ts

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1001,6 +1001,11 @@ function buildHighlightsScm(
10011001
const exprRuleName = [...ctx.prattRules][0];
10021002
const exprSnake = exprRuleName ? snake(exprRuleName) : null;
10031003
if (exprSnake && hasPropertyAccess(grammar, identName)) struct.push(`(${exprSnake} (${exprSnake}) (${identSnake}) @property)`);
1004+
// destructuring rename key `{ key: binding }` — the key, anchored on the trailing element
1005+
for (const [n, el] of bindingKeyRules(grammar, identName)) {
1006+
const s = snake(n), es = snake(el);
1007+
if (s && es) struct.push(`(${s} (${identSnake}) @property (${es}))`);
1008+
}
10041009
if (struct.length > 0) {
10051010
out.push(';; Structural member / type-param / property-access captures.');
10061011
out.push(...struct);
@@ -1324,6 +1329,29 @@ function hasPropertyAccess(grammar: CstGrammar, identName: string): boolean {
13241329
return found;
13251330
}
13261331

1332+
/** Rules with a `[ identifier, ':', ruleRef ]` branch — a `key: binding` property whose
1333+
* KEY is a property name. The trailing rule ref distinguishes it from a shorthand
1334+
* binding (`{ a }`, where the identifier IS the binding), so the capture is anchored
1335+
* on that element. Returns rule → the element-rule that must follow the key. */
1336+
function bindingKeyRules(grammar: CstGrammar, identName: string): Map<string, string> {
1337+
const out = new Map<string, string>();
1338+
for (const r of grammar.rules) {
1339+
for (const b of ruleBranches(r.body)) {
1340+
for (let i = 0; i + 2 < b.length; i++) {
1341+
if (b[i].type === 'ref' && (b[i] as { name: string }).name === identName
1342+
&& b[i + 1].type === 'literal' && (b[i + 1] as { value: string }).value === ':'
1343+
&& b[i + 2].type === 'ref') {
1344+
const element = (b[i + 2] as { name: string }).name;
1345+
// skip a self-referential value (`Stmt: Ident ':' Stmt` is a labeled statement,
1346+
// a jump target — not a destructuring key whose value is a sub-element rule).
1347+
if (element !== r.name) out.set(r.name, element);
1348+
}
1349+
}
1350+
}
1351+
}
1352+
return out;
1353+
}
1354+
13271355
// ════════════════════════════════════════════════════════════════════════════
13281356
// 3. src/scanner.c — external scanner
13291357
//

0 commit comments

Comments
 (0)