Skip to content

Commit f4ceb13

Browse files
committed
Emit a specialized lexer: per-charCode dispatch baked from the token IR
emitLexer(grammar) specializes tokenize() for token-stream grammars: one switch(charCode) whose cases hold exactly that char's candidate token regexes (declaration order) and a longest-first punct compare-chain with the literal's int baked at each leaf. Token regexes stay V8 regexes (the dispatch was the measured cost, not the matching). Regex-vs-division context, paren-head tracking and the template state machine are baked: prevIsValue is four int-table loads, the '('/')'/'!' bookkeeping exists only on those punct branches, scanTemplateSpan/identTextValid are emitted with their config inlined. Grammars using markup/indent/newline fall back to the createLexer import (emission unchanged for yaml/html). New gate test/emit-lexer-verify.ts: emitted stream ≡ createLexer over the 5,695-file conformance corpus — every field including k/t and the stamp flags, plus identical error messages (5,624 same + 71 same-throw, 0 diffs). Gates: full 18,805-file corpus byte-identical, conformance 5386/5659 unchanged, 26/26 check. Bench: whole parse +57~77%; vs ts.createSourceFile the aggregate is now 1.24x (parse-only) / 0.94x (setParentNodes:true) — below parity on the latter, with fixSignatureCaching at 0.69x even against parse-only.
1 parent 26bd41c commit f4ceb13

3 files changed

Lines changed: 496 additions & 13 deletions

File tree

0 commit comments

Comments
 (0)