Skip to content

Commit 53bff58

Browse files
committed
docs: record how the windowed re-lex is verified (#45)
The audit (B2) noted the windowed re-lex (resync / findRestart) has no gen-lexer counterpart and emit-lexer-verify only checks a full lex, implying it is untested. It is verified transitively: incremental-verify / exhaustive-edits compare an edited parse — whose tokens come from the windowed re-lex — to a fresh FULL parse, byte-identical, so a wrong windowed token changes the tree and fails there. Record that coverage chain at the lexer core so it is not mistaken for a gap.
1 parent fd16358 commit 53bff58

1 file changed

Lines changed: 7 additions & 0 deletions

File tree

src/emit-lexer.ts

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -266,6 +266,13 @@ export function emitLexer(grammar: CstGrammar, st: LexerSymtab): string | null {
266266
emit(` lexCore(source, 0, -1, 0, -1, 0, 0);`);
267267
emit(` return tokN;`);
268268
emit(`}`);
269+
// Verification of the WINDOWED path (issue #45 B2): emit-lexer-verify only exercises a FULL
270+
// lex (emit ≡ createLexer), and gen-lexer has no windowed counterpart to diff against — but the
271+
// windowed re-lex IS independently checked at the tree level. incremental-verify / exhaustive-
272+
// edits compare an edited parse (whose tokens come from this windowed re-lex) to a FRESH FULL
273+
// parse of the same text, byte-identical: a wrong windowed token would change the tree (or its
274+
// newlineBefore/commentBefore-driven shape) and fail there. So the oracle is the fresh full
275+
// parse, applied transitively through the parser.
269276
emit(`// The lexer core, parameterized for WINDOWED re-lexing: start at startPos with`);
270277
emit(`// the previous token's (k, t) as the regex-context seed (-1 = none / file start)`);
271278
emit(`// and EMPTY template/paren stacks (the caller restarts only at depth-0 safe`);

0 commit comments

Comments
 (0)