Commit f4ceb13
committed
Emit a specialized lexer: per-charCode dispatch baked from the token IR
emitLexer(grammar) specializes tokenize() for token-stream grammars: one
switch(charCode) whose cases hold exactly that char's candidate token
regexes (declaration order) and a longest-first punct compare-chain with
the literal's int baked at each leaf. Token regexes stay V8 regexes (the
dispatch was the measured cost, not the matching). Regex-vs-division
context, paren-head tracking and the template state machine are baked:
prevIsValue is four int-table loads, the '('/')'/'!' bookkeeping exists
only on those punct branches, scanTemplateSpan/identTextValid are
emitted with their config inlined. Grammars using markup/indent/newline
fall back to the createLexer import (emission unchanged for yaml/html).
New gate test/emit-lexer-verify.ts: emitted stream ≡ createLexer over
the 5,695-file conformance corpus — every field including k/t and the
stamp flags, plus identical error messages (5,624 same + 71 same-throw,
0 diffs).
Gates: full 18,805-file corpus byte-identical, conformance 5386/5659
unchanged, 26/26 check.
Bench: whole parse +57~77%; vs ts.createSourceFile the aggregate is now
1.24x (parse-only) / 0.94x (setParentNodes:true) — below parity on the
latter, with fixSignatureCaching at 0.69x even against parse-only.1 parent 26bd41c commit f4ceb13
3 files changed
Lines changed: 496 additions & 13 deletions
0 commit comments