Skip to content

Commit 2876b9c

Browse files
johnsoncodehkjohnsoncodehk
committed
Fix 5 highlighter gaps the bench found (parser-corpus misses 593 → 159)
Five gen-tm.ts fixes, each diagnosed and verified in an isolated worktree, then serial-merged. All live in the highlighter generator (the parser is untouched, so conformance stays FN=0 / FP=79). - Unicode identifiers: widen the emitted Oniguruma identifier class with \p{L}… so non-ASCII names (Ω, Cyrillic А) are scoped like the lexer already accepts (33→0). - Angle-bracket prefix casts <Type>expr: emit a type-cast pattern so the inner type reaches #type-inner instead of value member-access scopes (type.ref 285→139). - Method signatures: handle optional `f2?()` and inject #method-signature into type literals, so signature names read entity.name.function (110→0). - `constructor` as an identifier: drop it from the flat keyword match (its keyword role is owned by #constructor-declaration) and fix that context's body-less end (20→0). public/private as property names, and Symbol/o, left as-is (unsafe / oracle-strictness). - Multiline generic args with comments (#876): zero-width type-alias `while` lookahead + comment includes in #type-inner (#876 fixed; #819 async-arrow generic left). Verified: conformance 100% valid (FN=0, FP=79); test-issues 318/0; tsc clean; scope-roles 31/31; js-coverage 98.4% (unchanged). Parser corpus: monogram misses 593→159, now 100% of strict cells (37/37, ahead of official's 36/37) and token-role tied at 99.0% (was 96.3%). Adversarial bug ledger: monogram 38→39/49, "worse than official" 2→1. README chart + balance note refreshed.
1 parent c1230af commit 2876b9c

4 files changed

Lines changed: 290 additions & 55 deletions

File tree

‎README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -67,7 +67,7 @@ highlighter from its conformance-proven parser; the official one is hand-written
6767

6868
```
6969
TypeScript
70-
Monogram █████████████████░░░░░ 78% (38/49)
70+
Monogram ██████████████████░░░░ 80% (39/49)
7171
official █████████░░░░░░░░░░░░░ 39% (19/49)
7272
JavaScript
7373
pending — JS not yet on the neutral-oracle bench (see ROADMAP)
@@ -76,7 +76,7 @@ JavaScript
7676
<sub>TypeScript = 49 oracle-adjudicable open [`microsoft/TypeScript-TmLanguage`](https://github.com/microsoft/TypeScript-TmLanguage/issues) issues ([`test/issue-cases.ts`](test/issue-cases.ts)) — 21 of Monogram's wins are fixes the official grammar gets *structurally* wrong. Per-issue breakdown: `node test/highlight-bench.ts`. Regenerate: `npm run bench:readme`.</sub>
7777
<!-- bench:end -->
7878

79-
> **The other side of the ledger (honesty check).** On the *broad* TS parser-conformance corpus — not just the documented bugs — raw token-role accuracy flips: the official grammar's decade of hardening leads (~99% vs ~96%). Monogram's gap there is specific backlog (Unicode identifiers, qualified type names), **not** the ambiguity class above. Clone the TS corpus to `/tmp/ts-repo` and run `node test/highlight-bench.ts` to see both corpora.
79+
> **The other side of the ledger (honesty check).** On the *broad* TS parser-conformance corpus — not just the documented bugs — the two are now neck-and-neck: token-role accuracy is **tied at ~99%**, and Monogram leads on per-cell coverage (100% of strict cells vs official's ~97%). The official grammar still edges ahead on whole-file-perfect snippets (~89% vs ~83%); the residual is Monogram's remaining qualified-type-name backlog, **not** the ambiguity class above. Clone the TS corpus to `/tmp/ts-repo` and run `node test/highlight-bench.ts` to see both corpora.
8080
8181
## What you get
8282

‎examples/javascript.tmLanguage.json‎

Lines changed: 18 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -242,7 +242,7 @@
242242
},
243243
"ident": {
244244
"name": "variable.other.javascript",
245-
"match": "(?:[a-zA-Z_$]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})(?:[a-zA-Z0-9_$]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})*"
245+
"match": "(?:[a-zA-Z_$\\p{L}\\p{Nl}]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})(?:[a-zA-Z0-9_$\\p{L}\\p{Nl}\\p{Nd}\\p{Mn}\\p{Mc}\\p{Pc}]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})*"
246246
},
247247
"hexnumber": {
248248
"name": "constant.numeric.hex.javascript",
@@ -425,7 +425,7 @@
425425
]
426426
},
427427
"declaration-param-name": {
428-
"match": "(?<=[,(])\\s*(\\.\\.\\.)?\\s*((?:[a-zA-Z_$]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})(?:[a-zA-Z0-9_$]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})*)(?=\\s*[,)=])",
428+
"match": "(?<=[,(])\\s*(\\.\\.\\.)?\\s*((?:[a-zA-Z_$\\p{L}\\p{Nl}]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})(?:[a-zA-Z0-9_$\\p{L}\\p{Nl}\\p{Nd}\\p{Mn}\\p{Mc}\\p{Pc}]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})*)(?=\\s*[,)=])",
429429
"captures": {
430430
"1": {
431431
"name": "keyword.operator.spread.javascript"
@@ -463,7 +463,7 @@
463463
},
464464
"function-declaration": {
465465
"name": "meta.function.javascript",
466-
"begin": "\\b(function)\\s*(\\*)?\\s*((?:[a-zA-Z_$]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})(?:[a-zA-Z0-9_$]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})*)",
466+
"begin": "\\b(function)\\s*(\\*)?\\s*((?:[a-zA-Z_$\\p{L}\\p{Nl}]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})(?:[a-zA-Z0-9_$\\p{L}\\p{Nl}\\p{Nd}\\p{Mn}\\p{Mc}\\p{Pc}]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})*)",
467467
"beginCaptures": {
468468
"1": {
469469
"name": "storage.type.function.javascript"
@@ -490,7 +490,7 @@
490490
},
491491
"class-declaration": {
492492
"name": "meta.class.javascript",
493-
"begin": "\\b(class)\\s+((?:[a-zA-Z_$]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})(?:[a-zA-Z0-9_$]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})*)",
493+
"begin": "\\b(class)\\s+((?:[a-zA-Z_$\\p{L}\\p{Nl}]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})(?:[a-zA-Z0-9_$\\p{L}\\p{Nl}\\p{Nd}\\p{Mn}\\p{Mc}\\p{Pc}]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})*)",
494494
"beginCaptures": {
495495
"1": {
496496
"name": "storage.type.class.javascript"
@@ -517,7 +517,7 @@
517517
"name": "storage.type.function.javascript"
518518
}
519519
},
520-
"end": "(?<=\\})",
520+
"end": "(?<=\\})|(?=[;}])",
521521
"patterns": [
522522
{
523523
"include": "#declaration-params"
@@ -532,10 +532,13 @@
532532
},
533533
"method-signature": {
534534
"name": "meta.method.javascript",
535-
"begin": "\\b((?:[a-zA-Z_$]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})(?:[a-zA-Z0-9_$]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})*)\\s*(?=[<(])",
535+
"begin": "\\b((?:[a-zA-Z_$\\p{L}\\p{Nl}]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})(?:[a-zA-Z0-9_$\\p{L}\\p{Nl}\\p{Nd}\\p{Mn}\\p{Mc}\\p{Pc}]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})*)(\\?)?\\s*(?=[<(])",
536536
"beginCaptures": {
537537
"1": {
538538
"name": "entity.name.function.javascript"
539+
},
540+
"2": {
541+
"name": "keyword.operator.optional.javascript"
539542
}
540543
},
541544
"end": "(?<=\\})|(?=[;\\}])",
@@ -552,7 +555,7 @@
552555
]
553556
},
554557
"extends-definition": {
555-
"match": "\\b(extends)\\s+((?:[a-zA-Z_$]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})(?:[a-zA-Z0-9_$]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})*)",
558+
"match": "\\b(extends)\\s+((?:[a-zA-Z_$\\p{L}\\p{Nl}]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})(?:[a-zA-Z0-9_$\\p{L}\\p{Nl}\\p{Nd}\\p{Mn}\\p{Mc}\\p{Pc}]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})*)",
556559
"captures": {
557560
"1": {
558561
"name": "keyword.other.extends.javascript"
@@ -574,15 +577,15 @@
574577
}
575578
},
576579
"function-call": {
577-
"match": "((?:[a-zA-Z_$]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})(?:[a-zA-Z0-9_$]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})*)(?=\\s*\\()",
580+
"match": "((?:[a-zA-Z_$\\p{L}\\p{Nl}]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})(?:[a-zA-Z0-9_$\\p{L}\\p{Nl}\\p{Nd}\\p{Mn}\\p{Mc}\\p{Pc}]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})*)(?=\\s*\\()",
578581
"captures": {
579582
"1": {
580583
"name": "entity.name.function.javascript"
581584
}
582585
}
583586
},
584587
"object-method-key": {
585-
"match": "((?:[a-zA-Z_$]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})(?:[a-zA-Z0-9_$]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})*)(?=\\s*:\\s*(?:\\w+\\s+)?[\\(])",
588+
"match": "((?:[a-zA-Z_$\\p{L}\\p{Nl}]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})(?:[a-zA-Z0-9_$\\p{L}\\p{Nl}\\p{Nd}\\p{Mn}\\p{Mc}\\p{Pc}]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})*)(?=\\s*:\\s*(?:\\w+\\s+)?[\\(])",
586589
"captures": {
587590
"1": {
588591
"name": "entity.name.function.javascript"
@@ -600,13 +603,13 @@
600603
"end": "(?=[()}\\],=;])",
601604
"patterns": [
602605
{
603-
"match": "(?:[a-zA-Z_$]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})(?:[a-zA-Z0-9_$]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})*",
606+
"match": "(?:[a-zA-Z_$\\p{L}\\p{Nl}]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})(?:[a-zA-Z0-9_$\\p{L}\\p{Nl}\\p{Nd}\\p{Mn}\\p{Mc}\\p{Pc}]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})*",
604607
"name": "entity.name.function.javascript"
605608
}
606609
]
607610
},
608611
"method-call": {
609-
"match": "(?<=[\\w$\\]\\)])(\\.)\\s*((?:[a-zA-Z_$]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})(?:[a-zA-Z0-9_$]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})*)(?=\\s*\\()",
612+
"match": "(?<=[\\w$\\]\\)])(\\.)\\s*((?:[a-zA-Z_$\\p{L}\\p{Nl}]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})(?:[a-zA-Z0-9_$\\p{L}\\p{Nl}\\p{Nd}\\p{Mn}\\p{Mc}\\p{Pc}]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})*)(?=\\s*\\()",
610613
"captures": {
611614
"1": {
612615
"name": "punctuation.accessor.javascript"
@@ -628,7 +631,7 @@
628631
}
629632
},
630633
"property-access": {
631-
"match": "(?<=[\\w$\\]\\)])(\\.)\\s*((?:[a-zA-Z_$]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})(?:[a-zA-Z0-9_$]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})*)",
634+
"match": "(?<=[\\w$\\]\\)])(\\.)\\s*((?:[a-zA-Z_$\\p{L}\\p{Nl}]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})(?:[a-zA-Z0-9_$\\p{L}\\p{Nl}\\p{Nd}\\p{Mn}\\p{Mc}\\p{Pc}]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})*)",
632635
"captures": {
633636
"1": {
634637
"name": "punctuation.accessor.javascript"
@@ -639,7 +642,7 @@
639642
}
640643
},
641644
"arrow-parameter": {
642-
"match": "\\b((?:[a-zA-Z_$]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})(?:[a-zA-Z0-9_$]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})*)\\s*(?==>)",
645+
"match": "\\b((?:[a-zA-Z_$\\p{L}\\p{Nl}]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})(?:[a-zA-Z0-9_$\\p{L}\\p{Nl}\\p{Nd}\\p{Mn}\\p{Mc}\\p{Pc}]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})*)\\s*(?==>)",
643646
"captures": {
644647
"1": {
645648
"name": "variable.parameter.javascript"
@@ -662,7 +665,7 @@
662665
},
663666
"patterns": [
664667
{
665-
"match": "(?<=[,(])\\s*(\\.\\.\\.)?\\s*((?:[a-zA-Z_$]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})(?:[a-zA-Z0-9_$]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})*)(?=\\s*[,:)?=])",
668+
"match": "(?<=[,(])\\s*(\\.\\.\\.)?\\s*((?:[a-zA-Z_$\\p{L}\\p{Nl}]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})(?:[a-zA-Z0-9_$\\p{L}\\p{Nl}\\p{Nd}\\p{Mn}\\p{Mc}\\p{Pc}]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})*)(?=\\s*[,:)?=])",
666669
"captures": {
667670
"1": {
668671
"name": "keyword.operator.spread.javascript"
@@ -752,7 +755,7 @@
752755
"name": "keyword.other.javascript"
753756
},
754757
"scope-storage-type-function": {
755-
"match": "\\b(function|constructor)\\b",
758+
"match": "\\b(function)\\b",
756759
"name": "storage.type.function.javascript"
757760
},
758761
"scope-storage-type-property": {

0 commit comments

Comments
 (0)