feat(vnext): add incremental statement indexing (#175)

## Summary

- update the dialect-aware statement index incrementally from trusted
analysis-coordinate edits while preserving the full scan as the oracle
- add a private lazy per-session index cache with atomic
invalidation/reuse rules
- replace structural dialect definitions with opaque built-in factory
handles, keeping lexical profiles and indexes out of the public API

## Correctness and safety

- conservative restart at the old slot at or left of the earliest edit
- convergence only at an exact terminated boundary after the final edit
- exact full-oracle fallback for inconsistent metadata or absent
convergence
- global 10,000-slot and opaque-suffix behavior preserved
- monotonic convergence cursor keeps resource-cap recovery linear
- no source text or history retained by the cache

## Validation

- 833 unit tests passed; 1 governed expected failure
- 33 incremental tests with deterministic differential sequences across
all four profiles
- 80,000 additional adversarial differential edits matched the full
oracle
- changed coverage: 98.78% statements, 97.00% branches, 100% functions,
98.77% lines
- typecheck, oxlint, integrity, browser, demo, and packed-package smoke
passed
- 1 MiB benchmark: full scan about 2.2-2.5 ms, middle edit about 0.028
ms, prefix shift about 0.11 ms
- resource-cap recovery about 1.35-1.46 ms after fixing the reviewed
quadratic path

## Review

Two independent exact-SHA reviewers approve
2e83d17 with no unresolved blocker,
high, or medium findings.

<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Adds incremental statement indexing with a lazy, per-session cache so
edits update indexes in microseconds, while a full scan remains the
correctness fallback. Replaces caller-defined dialect objects with
built-in opaque dialect handles that bind lexical behavior internally.

- New Features
- Incremental updates from trusted analysis-coordinate edits; reuse
unchanged prefix and reuse/shift suffix at exact terminated boundaries,
with full-oracle fallback.
- Private per-session statement-index cache: reused on context-only
updates (same profile) and same-text replacements; incrementally updated
on trusted identity-source changes; cleared on profile change or changed
replacements; no source text or history retained.
- Built-in dialect factories return frozen opaque singletons:
`duckdbDialect()`, `postgresDialect()`, `bigQueryDialect()`,
`dremioDialect()`; handles are validated, local to the package instance,
and don’t cross workers/processes.
- Bench and tests: `pnpm run bench:statement-index`; extensive unit,
randomized differential, and session cache tests.

- Migration
- Replace `defineSqlDialect(...)` and `SqlDialectDefinition` with
built-in handles and `SqlDialect`.
- Before: `createSqlLanguageService({ dialects: [defineSqlDialect({ id:
"duckdb", displayName: "DuckDB" })] })`
    - After: `createSqlLanguageService({ dialects: [duckdbDialect()] })`
- Keep using the handle’s `.id` string in document context: `{ dialect:
dialect.id }`.
- Do not copy/serialize/fabricate dialects; pass authentic handles from
the same `@marimo-team/codemirror-sql/vnext` instance (handles don’t
cross workers/processes).

<sup>Written for commit ce94176.
Summary will update on new commits.</sup>

<a
href="https://cubic.dev/pr/marimo-team/codemirror-sql/pull/175?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>

<!-- End of auto-generated description by cubic. -->

Author: Light2Dark

SQL_EDITOR_RESEARCH.md

@@ -708,8 +708,12 @@ Features could degrade honestly based on capabilities instead of treating
 A dialect should be more than CodeMirror highlighting and a
 `node-sql-parser` database name:
 
+The stable API should expose a package-owned opaque dialect handle; the
+following is the kind of internal bundle that handle resolves to, not a
+caller-constructed public object:
+
 ```ts
-interface SqlDialectDefinition {
+interface InternalSqlDialectBundle {
   id: string;
   codeMirrorDialect: SQLDialect;
   parserDialect: string;

docs/adr/0001-language-service-and-session.md

@@ -110,9 +110,17 @@ interface SqlDocumentContext {
 
 The service resolves dialect IDs through its immutable registry and rejects
 unknown or duplicate IDs. Duplicate registration is rejected even when the
-definitions appear equivalent. Dialect identity includes an internal
-configuration revision; calling a dialect factory again does not change the
-identity of an already registered dialect.
+definitions appear equivalent. The built-in dialect factories return frozen
+opaque singletons backed by package-owned `WeakMap` metadata; repeated calls to
+one factory return the same handle and internal configuration identity. A
+copied, serialized, fabricated, or different-package-instance handle is
+rejected.
+
+Dialect handles are in-process service configuration and do not cross cloning,
+worker, or provider boundaries. Serializable context contains only the
+registered handle's string ID. Another realm or package instance reconstructs
+configuration with its own built-in factory. IDs perform registry lookup but
+never infer lexical behavior.
 
 Hosts can add fields such as engine or SQL mode. Contexts must be
 structured-cloneable plain data. On open/update, the session creates and owns a
@@ -213,8 +221,15 @@ validity, or AST data. It materializes at most 10,000 slots and collapses an
 unscanned remainder into an opaque slot on resource limits or unsupported
 delimiter/procedural constructs. Opaque slots cannot be passed to later parsing
 as exact source. Dialect lexical behavior uses internal configuration identity,
-never a caller-controlled dialect ID. Incremental reuse and session attachment
-remain a separate change tested against the full-scan oracle.
+never a caller-controlled dialect ID.
+
+The full scanner remains the correctness oracle. Incremental indexing restarts
+conservatively at the old slot at or to the left of the earliest trusted
+analysis-coordinate change and converges only at an exact terminated boundary
+mapped to an unchanged old suffix. It reuses the unchanged prefix and either
+reuses or shifts the suffix. Inconsistent metadata falls back to the full
+oracle; absent convergence scans through EOF, and resource or
+unsupported-syntax opacity remains fail-closed.
 
 ## Request outcomes
 
@@ -409,6 +424,12 @@ SPI. Sessions create a new source snapshot for document updates and reuse the
 existing snapshot for context-only updates. Non-identity generated/reordered
 source remains deferred until a concrete consumer validates the segment model.
 
+Current session edits use identity sources, so validated original-document
+changes are also trusted analysis-coordinate changes. A future transformed
+source must provide its own validated analysis-coordinate changes for
+incremental statement indexing. Without them, changed analysis text
+invalidates the index and is rebuilt lazily.
+
 Edits crossing an ambiguous or unmapped boundary are rejected. Mapping failure
 is unavailable analysis, not invalid SQL.
 
@@ -498,6 +519,15 @@ dialect, parser/configuration, and template identities. Catalog changes do not
 invalidate parsing. Renderer, theme, focus, and cursor changes do not invalidate
 semantic artifacts.
 
+The current statement-index cache is private, lazy, and per session. It stores
+only the current index, document sequence, and lexical-profile identity.
+Context-only updates reuse it for the same profile; equal analysis text reuses
+it across a new document revision; trusted identity-source changes update it
+incrementally. A changed replacement, profile change, or transformed source
+without trusted analysis changes clears it for a later full build. Invalid
+updates do not mutate it, disposal clears it, and it retains no source text or
+revision history.
+
 In-flight deduplication uses the same complete key. Cancelling one consumer
 detaches it; shared underlying work is aborted only when no consumers remain.
 Caches are bounded by both item count and estimated retained bytes.

docs/vnext/session-primitives.md

@@ -16,23 +16,22 @@ contract and the internal immutable source-snapshot model.
 ```ts
 import {
   createSqlLanguageService,
-  defineSqlDialect,
+  duckdbDialect,
   type SqlDocumentContext,
 } from "@marimo-team/codemirror-sql/vnext";
 
 interface AppSqlContext extends SqlDocumentContext {
   readonly engine: string;
 }
 
+const dialect = duckdbDialect();
 const service = createSqlLanguageService<AppSqlContext>({
-  dialects: [
-    defineSqlDialect({ id: "duckdb", displayName: "DuckDB" }),
-  ],
+  dialects: [dialect],
 });
 
 const session = service.openDocument({
   text: "SELECT * FROM users",
-  context: { dialect: "duckdb", engine: "local" },
+  context: { dialect: dialect.id, engine: "local" },
 });
 
 const revision = session.update({
@@ -55,6 +54,20 @@ service.dispose();
 Use `{ kind: "replace", text }` for full replacement and
 `{ kind: "context", baseRevision, context }` for a context-only update.
 
+## Dialect registration
+
+`duckdbDialect()`, `postgresDialect()`, `bigQueryDialect()`, and
+`dremioDialect()` return frozen, opaque built-in handles. Each factory returns
+the same singleton on repeated calls. The service resolves a handle through
+package-owned runtime metadata, so a copied, serialized, fabricated, or
+different-package-instance handle is rejected.
+
+Handles are local service configuration, not transport data. Document context
+contains only the handle's serializable string `id`; a worker or separate
+package instance must call its own built-in factory rather than receive a
+handle through cloning or messaging. The ID selects a registered handle but
+does not itself infer lexical behavior.
+
 ## Contract
 
 - Every accepted update creates a new frozen, service-issued revision.
@@ -72,6 +85,15 @@ Use `{ kind: "replace", text }` for full replacement and
 - Synchronous public failures use `SqlSessionError` and a stable `code`; caught
   platform errors are not exposed as causes.
 
+The internal statement index is built lazily per session. Context-only updates
+reuse it when the lexical-profile identity is unchanged. A no-op document
+update or same-text replacement advances the public revision and document
+sequence but can reuse the index. Trusted incremental identity-source changes
+update a populated cache; a changed replacement, profile change, or changed
+transformed source without analysis-coordinate changes clears it for a later
+full build. Invalid updates leave both the session snapshot and cache
+unchanged, and disposal clears the cache.
+
 The safety envelope currently permits at most 10,000 changes in one update, a
 16 Mi-code-unit document, 1,000 registered dialects, and bounded context graph
 depth, size, properties, and string data. These are defensive walking-skeleton

docs/vnext/source-coordinates.md

@@ -48,3 +48,16 @@ public.
 The internal [statement index](./statement-index.md) scans `analysisText` and
 uses its length-preserving offsets without publishing analysis-coordinate
 ranges.
+
+Statement-index reuse depends on `analysisText` value and internal lexical
+profile identity, not source-object identity. A document update therefore
+still creates a fresh source snapshot and public revision even when equal
+analysis text permits reuse. The index does not retain either the old or
+current source text.
+
+Current sessions create identity sources, so their validated document changes
+are also trusted analysis-coordinate changes. The masking primitive is not yet
+attached to session updates. A future transformed-source pipeline must produce
+validated analysis-coordinate changes before it can use incremental indexing;
+otherwise a changed analysis document invalidates the cache and receives a
+fresh full build on demand.

docs/vnext/statement-index.md

@@ -1,6 +1,6 @@
 # vNext Statement Index
 
-Status: internal full-scan correctness oracle
+Status: internal full-scan oracle with incremental session reuse
 
 The statement index is a synchronous, parser-free partition of
 `analysisText`. It does not classify, parse, validate, or copy statements, and
@@ -41,8 +41,9 @@ run-current-statement commands need different policies.
 ## Dialect profiles
 
 Lexical behavior is carried by immutable internal profile identity. It is never
-inferred from a caller-controlled dialect ID. This first oracle owns profiles
-for:
+inferred from a caller-controlled dialect ID. Frozen built-in dialect
+singletons select package-owned profiles through private runtime metadata. This
+first oracle owns profiles for:
 
 - PostgreSQL: doubled quotes, `E'...'`, dollar-quoted strings, and nested block
   comments, following the
@@ -66,14 +67,48 @@ Unterminated lexical constructs consume the remainder and report their opening
 offset. Regular BigQuery strings also fail closed at a line break, because only
 triple-quoted strings may span lines.
 
-## Complexity and sequencing
+## Incremental updates
+
+The full build remains the correctness oracle. When a session already has an
+index and receives trusted ordered changes in analysis coordinates, the
+incremental path restarts conservatively at the beginning of the old slot at or
+to the left of the earliest change. Slot starts are safe checkpoints because
+lexical and prefix state is normal there.
+
+Scanning continues until an exact terminated boundary maps to the start of an
+unchanged old suffix after the final change. The unchanged prefix is retained;
+the suffix is reused directly when its offset is unchanged or copied with
+shifted frozen ranges when the net edit length changes. Convergence is rejected
+if the combined result would violate the slot limit or make a prior
+resource-limit suffix unsafe to reuse.
+
+Inconsistent change metadata falls back to a fresh full build. If no safe
+boundary converges, scanning continues through EOF. Unsupported procedural or
+custom-delimiter syntax and resource limits remain scanner-produced opaque
+suffixes; the incremental layer never guesses a boundary or creates a new
+opacity reason.
+
+## Session cache and complexity
+
+The index cache is private, lazy, and per session. It retains only the current
+index, document sequence, and lexical-profile identity, not source text or
+history. Context-only updates reuse it for the same profile. Equal analysis
+text reuses it across a new document revision. Trusted identity-source changes
+update it incrementally; changed replacements, profile changes, and transformed
+sources without trusted analysis-coordinate changes invalidate it. Disposal
+clears it.
 
 A full build is linear in UTF-16 code units, retains only slot records and a
 bounded dollar delimiter, and creates no statement substrings. Point lookup is
 logarithmic in slot count. The scanner operates on `analysisText`; the current
 length-preserving source transform makes its analysis ranges valid at the same
 offsets in `originalText`.
 
-This implementation remains the correctness oracle. Incremental rescanning,
-change mapping, cache reuse, and session attachment belong to a later slice and
-must be tested against a fresh full build after arbitrary edits.
+Incremental work is proportional to the rescanned region plus any shifted
+suffix records, with a worst case equal to a full linear scan. Randomized edit
+tests compare every incremental result with a fresh oracle build.
+
+`pnpm run bench:statement-index` measures full and incremental paths on a
+roughly 1 MiB, 1,000-statement document, including a local middle edit and a
+prefix insertion that shifts the reusable suffix. It also measures recovery
+from a resource-limited 10,000-slot index to guard the linear convergence path.

package.json

@@ -24,6 +24,7 @@
     "test:coverage:changed": "node ./scripts/changed-coverage.mjs",
     "test:browser": "vitest run --config vitest.browser.config.ts",
     "test:integrity": "node ./scripts/check-test-integrity.mjs",
+    "bench:statement-index": "vitest bench --run src/vnext/__tests__/statement-index.bench.ts",
     "test:package": "node ./scripts/clean.mjs && tsc && node ./scripts/package-smoke.mjs",
     "demo": "vite build",
     "build": "node ./scripts/clean.mjs && tsc",

scripts/package-smoke.mjs

@@ -174,11 +174,11 @@ try {
     join(temporaryDirectory, "vnext-consumer.mjs"),
     `import {
   createSqlLanguageService,
-  defineSqlDialect,
+  duckdbDialect,
 } from "@marimo-team/codemirror-sql/vnext";
 
 const service = createSqlLanguageService({
-  dialects: [defineSqlDialect({ displayName: "DuckDB", id: "duckdb" })],
+  dialects: [duckdbDialect()],
 });
 const session = service.openDocument({
   context: { dialect: "duckdb" },
@@ -208,7 +208,7 @@ import {
 } from "@marimo-team/codemirror-sql/dialects";
 import {
   createSqlLanguageService,
-  defineSqlDialect,
+  duckdbDialect,
   type SqlDocumentContext,
   type SqlTextRange,
 } from "@marimo-team/codemirror-sql/vnext";
@@ -225,7 +225,7 @@ const extensions: Extension[] = [
 ];
 const parser = new NodeSqlParser();
 const service = createSqlLanguageService<HostContext>({
-  dialects: [defineSqlDialect({ displayName: "DuckDB", id: "duckdb" })],
+  dialects: [duckdbDialect()],
 });
 const session = service.openDocument({
   context: { dialect: "duckdb", engine: "local" },
@@ -266,7 +266,10 @@ if (!dialects.BigQueryDialect || !dialects.DremioDialect || !dialects.DuckDBDial
 }
 if (
   typeof vnext.createSqlLanguageService !== "function" ||
-  typeof vnext.defineSqlDialect !== "function"
+  typeof vnext.bigQueryDialect !== "function" ||
+  typeof vnext.dremioDialect !== "function" ||
+  typeof vnext.duckdbDialect !== "function" ||
+  typeof vnext.postgresDialect !== "function"
 ) {
   throw new Error("vNext package exports are incomplete");
 }
@@ -281,7 +284,7 @@ if (!parseResult.success || !parseResult.ast) {
 }
 
 const service = vnext.createSqlLanguageService({
-  dialects: [vnext.defineSqlDialect({ displayName: "DuckDB", id: "duckdb" })],
+  dialects: [vnext.duckdbDialect()],
 });
 const session = service.openDocument({
   context: { dialect: "duckdb" },