feat(vnext): add dialect-aware statement index (#174)

## Summary

- add an internal synchronous parser-free statement partition and
binary-search cursor lookup
- preserve explicit half-open ownership for trivia, terminators, empty
slots, EOF, and left/right affinity
- add bounded PostgreSQL, DuckDB, BigQuery, and Dremio lexical profiles
with opaque fail-closed suffixes
- keep incremental reuse, session attachment, parser semantics, and
public APIs out of this slice

## Safety and performance

- cap materialized slots at 10,000 and collapse the remainder into one
opaque slot
- bound dollar delimiters and retain no statement substrings
- conservatively handle Unicode dollar tags and prefix boundaries
- fail closed for BigQuery procedural bodies, custom delimiters, and
PostgreSQL BEGIN ATOMIC bodies
- 16 MiB scans under a 128 MiB heap: about 44 ms ASCII, 61 ms Latin, 47
ms astral

## Evidence

- exact head: 56b57ea
- focused statement-index tests: 82 passed
- full suite: 795 passed, 1 governed expected failure
- changed coverage: 98.76% statements, 96.97% branches, 100% functions,
98.75% lines
- strict typecheck, non-mutating lint, test integrity, demo build,
packed-package smoke, and Chromium passed
- lexical/correctness reviewer: APPROVE exact head, no
blocker/high/medium findings
- performance/API reviewer: APPROVE exact head, no blocker/high/medium
findings

## Public API

No vNext public exports or generated root declarations are added. The
full scanner remains the correctness oracle for a later independently
reviewed incremental slice.

<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Adds an internal, dialect-aware SQL statement index for vNext. It
partitions `analysisText` into exact half-open slots with binary-search
cursor lookup and bounded resources, with no public API changes.

- **New Features**
- Exact slot partitioning and ownership: terminators associate left;
trailing trivia moves to the next slot; explicit empty slots are
preserved.
- Dialect profiles for PostgreSQL, DuckDB, BigQuery, and Dremio;
protects strings/comments (e.g., dollar-quoting, E-strings,
triple/raw/backtick, nested comments) and fails closed for custom
delimiters, BigQuery procedural blocks, and PostgreSQL BEGIN ATOMIC.
- Left/right-affinity lookup via binary search with explicit EOF
behavior for terminated vs open statements.
- Bounded resources: up to 10,000 materialized slots; bounded
dollar-quote tag length; no statement substring copies; frozen partition
data.
- Reports unterminated strings/comments without exposing internal
semicolons; operates on `analysisText` so UTF-16 offsets match the
original text under the current masking transform; dialect behavior is
keyed by internal profile identity, not a caller-controlled ID.

<sup>Written for commit 2ca3c25.
Summary will update on new commits.</sup>

<a
href="https://cubic.dev/pr/marimo-team/codemirror-sql/pull/174?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>

<!-- End of auto-generated description by cubic. -->

Author: Light2Dark

docs/adr/0001-language-service-and-session.md

@@ -202,6 +202,20 @@ document offsets and statement-relative offsets use distinct branded types.
 Statement lookup uses explicit cursor affinity at boundaries and EOF. It must
 not reintroduce the v0.x inclusive-end behavior.
 
+The first statement-index implementation is an internal synchronous full-scan
+oracle. Exact slot extents partition the analysis text, terminators associate
+left, and trivia after a terminator associates with the following slot. Empty
+documents and terminal delimiters retain explicit empty slots. Lookup requires
+left or right affinity and uses binary search.
+
+The index does not contain statement text, line numbers, inferred kinds, parser
+validity, or AST data. It materializes at most 10,000 slots and collapses an
+unscanned remainder into an opaque slot on resource limits or unsupported
+delimiter/procedural constructs. Opaque slots cannot be passed to later parsing
+as exact source. Dialect lexical behavior uses internal configuration identity,
+never a caller-controlled dialect ID. Incremental reuse and session attachment
+remain a separate change tested against the full-scan oracle.
+
 ## Request outcomes
 
 Every session feature request settles with the same top-level union:

docs/vnext/source-coordinates.md

@@ -44,3 +44,7 @@ functions remain internal. This slice intentionally does not publish a source
 transformer or source-map SPI. Generated or reordered source requires a
 versioned segment-map design and evidence from real consumers before becoming
 public.
+
+The internal [statement index](./statement-index.md) scans `analysisText` and
+uses its length-preserving offsets without publishing analysis-coordinate
+ranges.

docs/vnext/statement-index.md

@@ -0,0 +1,79 @@
+# vNext Statement Index
+
+Status: internal full-scan correctness oracle
+
+The statement index is a synchronous, parser-free partition of
+`analysisText`. It does not classify, parse, validate, or copy statements, and
+it has no public `/vnext` export yet.
+
+Each exact slot contains:
+
+- An `extent` that participates in a contiguous partition of the complete text.
+- A `source` range that includes leading and trailing trivia but excludes the
+  terminator.
+- An optional one-code-unit `terminator`, owned by the slot on its left.
+- `hasCode`, which distinguishes code from whitespace/comment-only slots.
+- A lexical end state for incomplete quoted strings or block comments.
+
+An empty document has one empty slot. A document ending exactly in a semicolon
+has an explicit trailing empty slot; trailing trivia instead forms the final
+trivia-only slot. Consecutive semicolons retain their empty slots.
+
+The scanner returns an opaque suffix instead of guessed boundaries when it
+encounters an unsupported custom delimiter, an unsupported BigQuery procedural
+body, or a resource limit. Opaque slots omit `source`, `terminator`, and
+`hasCode`, so later layers cannot accidentally parse them as exact statements.
+At most 10,000 slots are materialized; a semicolon-dense remainder collapses
+into one opaque slot.
+
+## Cursor affinity
+
+Point lookup always requires `left` or `right` affinity. At a shared extent
+boundary, left selects the preceding slot and right selects the following slot.
+At position zero both select the first slot. At EOF after a terminator, left
+selects the terminated slot and right selects the explicit trailing empty slot.
+At EOF in an unterminated final statement, both select that statement.
+
+The lookup uses binary search. It deliberately does not implement a hidden
+"nearest code statement" fallback; completion, hover, gutter, and future
+run-current-statement commands need different policies.
+
+## Dialect profiles
+
+Lexical behavior is carried by immutable internal profile identity. It is never
+inferred from a caller-controlled dialect ID. This first oracle owns profiles
+for:
+
+- PostgreSQL: doubled quotes, `E'...'`, dollar-quoted strings, and nested block
+  comments, following the
+  [PostgreSQL lexical contract](https://www.postgresql.org/docs/current/sql-syntax-lexical.html).
+  Dollar-tag and literal-prefix boundaries conservatively treat every
+  non-ASCII code point as identifier-like, covering the engines' permissive
+  Unicode behavior without exposing internal semicolons. SQL routine bodies
+  introduced by `BEGIN ATOMIC` are opaque in this slice.
+- DuckDB: doubled quotes, escape strings, tagged dollar-quoted strings, and
+  nested comments. Dollar quoting follows
+  [DuckDB literal types](https://duckdb.org/docs/current/sql/data_types/literal_types).
+- BigQuery: single, double, triple, raw, bytes, and raw-bytes strings; backtick
+  identifiers; `#` comments; and non-nesting block comments, following the
+  [GoogleSQL lexical contract](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/lexical).
+  Procedural bodies, including labeled loops, are opaque in this slice.
+- Dremio: a compatibility profile limited to verified single-quoted strings,
+  double-quoted identifiers, and standard comments. It does not silently
+  inherit PostgreSQL extensions.
+
+Unterminated lexical constructs consume the remainder and report their opening
+offset. Regular BigQuery strings also fail closed at a line break, because only
+triple-quoted strings may span lines.
+
+## Complexity and sequencing
+
+A full build is linear in UTF-16 code units, retains only slot records and a
+bounded dollar delimiter, and creates no statement substrings. Point lookup is
+logarithmic in slot count. The scanner operates on `analysisText`; the current
+length-preserving source transform makes its analysis ranges valid at the same
+offsets in `originalText`.
+
+This implementation remains the correctness oracle. Incremental rescanning,
+change mapping, cache reuse, and session attachment belong to a later slice and
+must be tested against a fresh full build after arbitrary edits.