Summary

• extract the streaming bounded SQL lexer from the relation-site state machine into one package-private module
• preserve the exact PostgreSQL, DuckDB, BigQuery, and Dremio lexical profiles, UTF-16 offsets, embedded-region barriers, quote/comment behavior, one-token pushback, and 16,384-lexeme ceiling
• keep query-site keyword, comment-cursor, region, and resource semantics local to the consumer
• translate generic lexer resource evidence through an exhaustive package-owned map
• add direct boundary tests without exposing tokens or lexer APIs from the package

This is a zero-semantics prerequisite for the separate bounded CTE layout/visibility recognizer. The recognizers will initially use separate streaming traversals over the same lexical implementation; any traversal fusion remains benchmark-driven.

Performance

An initial broader helper extraction caused a reproducible Vite SSR namespace-call regression on the hot path. The boundary was narrowed before commit.

Stable means at the exact head are back at the PR #186 baseline:

• exact 10 KiB statement: about 0.56–0.59 ms
• 1,000 classified aliases: about 0.36–0.38 ms
• 1,000 authenticated USING columns: about 0.21 ms

The lexer remains streaming and does not allocate a token tape.

Verification

• 1,543 tests passed plus 1 expected failure
• changed coverage: 97.05% statements, 95.79% branches, 100% functions, 97.04% lines
• bounded lexer coverage: 98.92% statements/lines, 98.78% branches, 100% functions
• repository coverage: 95.34% statements, 92.05% branches, 96.15% functions, 95.33% lines
• source, test, loose-optional, and demo typechecks pass
• repository oxlint, test-integrity, and diff checks pass
• browser, package, worker-placement, demo, and benchmark gates pass
• 20,000 deterministic differential lexer comparisons passed across dialects, masked regions, subranges, UTF-16, comments, quotes, punctuation, and pushback
• independent SQL/API and concurrency/performance reviewers approved exact commit ed079df0f68aadaf8957a7ca5e6c497e91b74860

Part of #169.

Summary by cubic

Refactored vNext to share a streaming bounded SQL lexer and the embedded-region lookup, and switched query-site to use them. Behavior is unchanged across PostgreSQL, DuckDB, BigQuery, and Dremio; this unblocks the bounded CTE recognizer in #169.

• Refactors
  • Moved the lexer into package-private src/vnext/bounded-sql-lexer.ts.
  • Centralized findSqlEmbeddedRegionAtOrAfter in src/vnext/source.ts and reused it in the lexer and query-site (with tests).
  • Preserves lexical profiles, UTF-16 offsets, embedded-region barriers, quote/comment rules, one-token pushback, and the 16,384-lexeme cap.
  • Kept consumer-specific semantics in query-site; mapped lexer resource signals to local query-site resources.
  • Removed duplicated lexer and region-lookup code from src/vnext/query-site.ts and wired it to the shared modules.

^{Written for commit 76190f9. Summary will update on new commits.}