refactor(vnext): share bounded lexical primitives

[Light2Dark]

Summary

• extract dialect lexical profiles and shared UTF-16 identifier, quote, block-comment, and dollar-quote scanners from the statement index
• preserve statement-index behavior while adding explicit upper bounds needed by the parser-independent query-site recognizer
• make bounded dollar-quote opener/closer classification prefix-deterministic and keep partial close search bounded
• add direct boundary regressions for quotes, block comments, dollar quotes, UTF-16 identifiers, and SQL whitespace

This is a private prerequisite for ADR 0005 step 3; it adds no public package export.

Validation

• pnpm run typecheck (strict and loose optional-property modes)
• pnpm exec oxlint
• pnpm run test:integrity
• 1,272 Node tests passed + 1 expected failure
• 7 Chromium tests passed
• packed-package smoke passed
• worker-placement/runtime evidence passed
• demo build passed
• changed coverage: 98.51% statements, 96.75% branches, 100% functions, 98.50% lines
• three independent exact-head adversarial approvals at 652b108

Review history

Adversarial review found a dollar-opener read beyond an explicit scan limit. The final head guards the opener before inspecting a suffix and includes bounded-prefix equivalence coverage. No additional Copilot request will be made after the one request for this PR.

---

Summary by cubic

Shares SQL lexical primitives and adds explicit scan limits to prevent over-reads while keeping statement-index behavior. Also preserves fast, bounded ASCII word scans and fixes dollar-quote opener detection.

• Refactors

  • Moved scanners for quotes, block comments, dollar quotes, UTF‑16 identifiers, and SQL whitespace into src/vnext/lexical.ts with explicit limit handling; preserved fast ASCII word scanning in statement-index.ts for performance.
  • Centralized dialect profiles and helpers; statement-index.ts now imports these and drops duplicate logic.

• Bug Fixes

  • Dollar-quote opener classification is now prefix-bounded; no suffix reads beyond the scan limit.
  • Quote and block comment scanners respect limits, including line-break and backslash cases.
  • Dollar-quote tags over 256 chars return delimiterTooLong instead of scanning unbounded.

^{Written for commit 3600269. Summary will update on new commits.}

[github-actions]

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	98.28% (🎯 91%)	400 / 407
🔵	Statements	98.28% (🎯 91%)	402 / 409
🔵	Functions	100% (🎯 90%)	46 / 46
🔵	Branches	96.58% (🎯 85%)	368 / 381

File Coverage

File	Stmts	Branches	Functions	Lines	Uncovered Lines
Changed Files
src/vnext/lexical.ts	99.17%	98.29%	100%	99.15%	71
src/vnext/statement-index.ts	97.91%	95.83%	100%	97.91%	177, 560, 741, 762, 850, 901

Generated in workflow #714 for commit 3600269 by the Vitest Coverage Report Action

[cubic-dev-ai]

No issues found across 3 files

_{Re-trigger cubic}