feat(vnext): make embedded regions atomic session inputs (#184)

## Summary

- make embedded regions a public, atomic input to document open and
update transactions
- require every text mutation to carry the complete post-edit region set
while preserving omission and explicit-clear semantics for non-document
updates
- mask embedded regions before statement analysis, with separate source
sequencing and conservative cache reuse
- align runtime decoding with TypeScript structural typing by copying
only declared own data fields and leaving host metadata opaque
- migrate the provisional completion contract and packed consumers to
the real session transaction types

## Correctness and consumer behavior

- text, context, dialect, regions, revision, sequences, and statement
cache commit or roll back together
- stale revisions are rejected before candidate payload inspection;
disposal dominates hostile/reentrant failures
- region ranges use resulting-document UTF-16 coordinates and include
complete non-SQL delimiters such as marimo's `{df}`
- source snapshots and normalized regions are owned and frozen;
same-value source transactions still advance revision/source sequence
- incremental statement indexing remains limited to identity-to-identity
edits; changed masked analysis rebuilds lazily
- structurally richer marimo region/envelope values compile and run
without invoking or retaining extra host fields
- removed outer `kind` and contradictory document fields remain
explicitly forbidden in both types and runtime

## Validation

- `pnpm run typecheck`
- `pnpm exec oxlint`
- `pnpm run test:integrity`
- `pnpm test` — 1,264 passed, 1 expected failure
- `pnpm run test:browser` — 7 passed
- `pnpm run test:package` with typed and runtime nonempty embedded
regions
- `pnpm run test:worker-placement`
- `pnpm run demo`
- changed coverage — 98.70% statements, 97.42% branches, 100% functions,
98.69% lines
- pre-rebase review/fix/re-review loop across SQL/API, marimo
ergonomics, and concurrency/performance perspectives
- fresh exact-head approval from all three reviewers at
`ff35130197fcf69a10d42d195c8dd6324f6d5ed9`

Part of #169.

<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Make embedded regions a first-class, atomic session input in vNext.
Document edits now require the complete post-edit region set, masking
happens before analysis, and cache reuse is tied to a separate source
sequence. Part of #169.

- **New Features**
- Embedded regions are accepted on open and update; document mutations
must include the full resulting `embeddedRegions`. Region-only and
context-only updates are supported.
- Region ranges use resulting-document UTF-16 and include full non-SQL
delimiters (e.g., marimo `{df}`); masking is length-preserving and runs
before statement analysis.
- Unchanged text+regions reuse the immutable source and statement index
via a separate source sequence; only identity-to-identity edits reuse
incrementally. Changed masked analysis invalidates the cache unless
analysis text is equal.
- Inputs are structural: only declared own data fields are read; host
metadata and extras stay opaque. Context accessors and non-enumerable
properties are rejected. Stale revisions are rejected before payload
inspection; disposal dominates hostile/reentrant failures.
- Optional `document`/`embeddedRegions`/`context` fields: `undefined` is
treated as omission at runtime for compatibility with and without
`exactOptionalPropertyTypes`.

- **Migration**
  - Update `session.update` calls:
- Context-only: `{ baseRevision, context }` (no `kind`; `context` must
be defined by types).
- Document mutation: `{ baseRevision, document: {...}, embeddedRegions:
[...] }` (regions required).
    - Region-only: `{ baseRevision, embeddedRegions: [...] }`.
- Open with regions: `service.openDocument({ text, context,
embeddedRegions })`.
- Use `SqlEmbeddedRegion` and `SqlDocumentUpdate` from
`@marimo-team/codemirror-sql/vnext`; stop importing `SqlEmbeddedRegion`
from `./source`.

<sup>Written for commit db6eeee.
Summary will update on new commits.</sup>

<a
href="https://cubic.dev/pr/marimo-team/codemirror-sql/pull/184?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>

<!-- End of auto-generated description by cubic. -->

Author: Light2Dark

docs/adr/0001-language-service-and-session.md

@@ -163,7 +163,9 @@ region set for its resulting text. Without a document mutation, an explicit
 complete region set can change alone or together with context; omission
 preserves the current regions. An explicit empty set clears them. Opening a
 document accepts the same optional complete set, with omission or an empty set
-meaning identity source.
+meaning identity source. For compatibility with consumers regardless of their
+`exactOptionalPropertyTypes` setting, `undefined` on optional state fields has
+the same meaning as omission.
 
 A full-text replacement is available for simple consumers. Incremental changes
 are ordered, non-overlapping, absolute UTF-16 ranges in the current document.
@@ -463,11 +465,12 @@ set for document or region updates and reuse the existing snapshot only when
 text and regions are unchanged. Non-identity generated/reordered source remains
 deferred until a concrete consumer validates the segment model.
 
-Current session edits use identity sources, so validated original-document
-changes are also trusted analysis-coordinate changes. A future transformed
-source must provide its own validated analysis-coordinate changes for
-incremental statement indexing. Without them, changed analysis text
-invalidates the index and is rebuilt lazily.
+Identity-to-identity session edits can use validated original-document changes
+as trusted analysis-coordinate changes. Masked-source edits do not make that
+claim: unchanged analysis can reuse its index, while changed analysis
+invalidates the index and rebuilds lazily. A future transformed source must
+provide its own validated analysis-coordinate changes before incremental
+indexing can consume them.
 
 Edits crossing an ambiguous or unmapped boundary are rejected. Mapping failure
 is unavailable analysis, not invalid SQL.

docs/vnext/session-primitives.md

@@ -35,12 +35,12 @@ const session = service.openDocument({
 });
 
 const revision = session.update({
-  kind: "document",
   baseRevision: session.revision,
   document: {
     kind: "changes",
     changes: [{ from: 14, to: 19, insert: "customers" }],
   },
+  embeddedRegions: [],
 });
 
 if (session.isCurrent(revision)) {
@@ -52,7 +52,9 @@ service.dispose();
 ```
 
 Use `{ kind: "replace", text }` for full replacement and
-`{ kind: "context", baseRevision, context }` for a context-only update.
+`{ baseRevision, context }` for a context-only update. Every document mutation
+also supplies the complete embedded-region set for its resulting text. A
+region-only transaction can replace or clear that set without a fake text edit.
 
 ## Dialect registration
 
@@ -74,7 +76,12 @@ does not itself infer lexical behavior.
 - `baseRevision` is checked by identity and cannot come from another session.
 - Incremental ranges are ordered, non-overlapping, half-open UTF-16 offsets in
   the pre-update document.
-- Text and context are validated completely before the session snapshot changes.
+- Text, context, and embedded regions are validated completely before the
+  session snapshot changes.
+- Public envelopes, edits, changes, and regions are structural inputs. The
+  service copies only their declared own data fields; additional host metadata
+  remains opaque and is never invoked. Removed or contradictory discriminants
+  are explicitly forbidden rather than treating every object as exact.
 - Context is structured-cloned and recursively frozen. Accepted values are
   finite primitives, arrays, and string-keyed plain objects. Cycles and shared
   references are retained. Accessors, symbols, functions, class instances,
@@ -86,13 +93,15 @@ does not itself infer lexical behavior.
   platform errors are not exposed as causes.
 
 The internal statement index is built lazily per session. Context-only updates
-reuse it when the lexical-profile identity is unchanged. A no-op document
-update or same-text replacement advances the public revision and document
-sequence but can reuse the index. Trusted incremental identity-source changes
-update a populated cache; a changed replacement, profile change, or changed
-transformed source without analysis-coordinate changes clears it for a later
-full build. Invalid updates leave both the session snapshot and cache
-unchanged, and disposal clears the cache.
+reuse it when the lexical-profile identity is unchanged. Source-changing
+transactions use a separate source sequence, so region-only changes cannot
+reuse a stale index. A no-op document update or same-text replacement advances
+the public revision and document sequence but can reuse the index. Trusted
+incremental identity-source changes update a populated cache; a changed
+replacement, profile change, or changed masked source without verified
+analysis-coordinate changes clears it for a later full build. Invalid updates
+leave both the session snapshot and cache unchanged, and disposal clears the
+cache.
 
 The safety envelope currently permits at most 10,000 changes in one update, a
 16 Mi-code-unit document, 1,000 registered dialects, and bounded context graph

docs/vnext/source-coordinates.md

@@ -17,10 +17,10 @@ are valid. `SqlTextChange` extends this shape with `insert` and continues to use
 pre-update document coordinates.
 
 The service internally owns an immutable source snapshot with separately named
-`originalText` and `analysisText`. A document update creates a new identity
-snapshot; a context-only update reuses the same source object. This separation
-allows later analysis transforms without making providers depend on editor
-state or ambiguous coordinate spaces.
+`originalText` and `analysisText`. A source transaction builds the complete
+post-update masked snapshot; a context-only update reuses the same source
+object. This separation keeps providers independent of editor state and
+ambiguous coordinate spaces.
 
 ## Length-preserving masking
 
@@ -35,29 +35,52 @@ The first internal transform masks explicit embedded regions:
 - `analysisText.length` always equals `originalText.length`, so range mapping is
   identity-preserving while still returning a fresh validated range.
 
+`SqlEmbeddedRegion` is the public document-coordinate input:
+
+```ts
+const session = service.openDocument({
+  text: "SELECT * FROM {df}",
+  context,
+  embeddedRegions: [{ from: 14, to: 18, language: "python" }],
+});
+
+session.update({
+  baseRevision: session.revision,
+  document: {
+    kind: "changes",
+    changes: [{ from: 15, to: 17, insert: "next_df" }],
+  },
+  embeddedRegions: [{ from: 14, to: 23, language: "python" }],
+});
+```
+
+The half-open interval covers the complete non-SQL fragment, including its
+template delimiters. For marimo, `[14, 18)` masks all of `{df}`; masking only
+`df` would leave `{}` to be analyzed as SQL. A document transaction always
+supplies the complete region set in coordinates of the resulting text.
+
 At most 10,000 regions and 16 Mi UTF-16 code units are accepted. Masking uses
 bounded 64 Ki-code-unit chunks, including for newline-dense input, rather than
 a per-code-unit or per-newline array.
 
-The source snapshot, embedded-region model, masking factory, and mapping
-functions remain internal. This slice intentionally does not publish a source
-transformer or source-map SPI. Generated or reordered source requires a
-versioned segment-map design and evidence from real consumers before becoming
-public.
+The source snapshot, masking factory, and mapping functions remain internal.
+Only the document-facing `SqlEmbeddedRegion` input shape is public. This slice
+intentionally does not publish a source transformer or source-map SPI.
+Generated or reordered source requires a versioned segment-map design and
+evidence from real consumers before becoming public.
 
 The internal [statement index](./statement-index.md) scans `analysisText` and
 uses its length-preserving offsets without publishing analysis-coordinate
 ranges.
 
 Statement-index reuse depends on `analysisText` value and internal lexical
-profile identity, not source-object identity. A document update therefore
-still creates a fresh source snapshot and public revision even when equal
-analysis text permits reuse. The index does not retain either the old or
-current source text.
-
-Current sessions create identity sources, so their validated document changes
-are also trusted analysis-coordinate changes. The masking primitive is not yet
-attached to session updates. A future transformed-source pipeline must produce
-validated analysis-coordinate changes before it can use incremental indexing;
-otherwise a changed analysis document invalidates the cache and receives a
-fresh full build on demand.
+profile identity, not source-object identity. Every accepted source transaction
+creates a public revision, while an unchanged text-and-region value may reuse
+the immutable source snapshot and index. The index does not retain either the
+old or current source text.
+
+Sessions accept complete length-preserving embedded-region sets on open and
+source transactions. Original document changes are trusted as analysis changes
+only for identity-to-identity updates. A changed masked source invalidates the
+cache unless its analysis text is unchanged; it then receives a fresh full
+build on demand.

package.json

@@ -13,10 +13,11 @@
   },
   "scripts": {
     "dev": "vite",
-    "typecheck": "pnpm run typecheck:src && pnpm run typecheck:tests && pnpm run typecheck:vnext-tests && pnpm run typecheck:demo",
+    "typecheck": "pnpm run typecheck:src && pnpm run typecheck:tests && pnpm run typecheck:vnext-tests && pnpm run typecheck:vnext-tests:loose-optional && pnpm run typecheck:demo",
     "typecheck:src": "tsc --project tsconfig.json --noEmit",
     "typecheck:tests": "tsc --project tsconfig.tests.json",
     "typecheck:vnext-tests": "tsc --project tsconfig.vnext-tests.json",
+    "typecheck:vnext-tests:loose-optional": "tsc --project tsconfig.vnext-tests.loose-optional.json",
     "typecheck:demo": "tsc --project tsconfig.demo.json",
     "lint": "oxlint --fix",
     "test": "vitest run --config vitest.config.ts",

scripts/package-smoke.mjs

@@ -199,12 +199,13 @@ const service = createSqlLanguageService({
 });
 const session = service.openDocument({
   context: { dialect: "duckdb" },
-  text: "SELECT 1",
+  embeddedRegions: [{ from: 14, language: "python", to: 18 }],
+  text: "SELECT * FROM {df}",
 });
 session.update({
-  kind: "document",
+  embeddedRegions: [{ from: 14, language: "python", to: 23 }],
   baseRevision: session.revision,
-  document: { kind: "replace", text: "SELECT 2" },
+  document: { kind: "replace", text: "SELECT * FROM {next_df}" },
 });
 service.dispose();
 `,
@@ -227,6 +228,7 @@ import {
   createSqlLanguageService,
   duckdbDialect,
   type SqlDocumentContext,
+  type SqlEmbeddedRegion,
   type SqlTextRange,
 } from "@marimo-team/codemirror-sql/vnext";
 import commonKeywords from "@marimo-team/codemirror-sql/data/common-keywords.json" with { type: "json" };
@@ -244,15 +246,19 @@ const parser = new NodeSqlParser();
 const service = createSqlLanguageService<HostContext>({
   dialects: [duckdbDialect()],
 });
+const embeddedRegions: readonly SqlEmbeddedRegion[] = [
+  { from: 14, language: "python", to: 18 },
+];
 const session = service.openDocument({
   context: { dialect: "duckdb", engine: "local" },
-  text: "SELECT 1",
+  embeddedRegions,
+  text: "SELECT * FROM {df}",
 });
 const range: SqlTextRange = { from: 0, to: 6 };
 session.update({
-  kind: "document",
+  embeddedRegions: [{ from: 14, language: "python", to: 18 }],
   baseRevision: session.revision,
-  document: { kind: "changes", changes: [{ from: 7, insert: "2", to: 8 }] },
+  document: { kind: "changes", changes: [] },
 });
 
 void extensions;
@@ -305,13 +311,14 @@ const service = vnext.createSqlLanguageService({
 });
 const session = service.openDocument({
   context: { dialect: "duckdb" },
-  text: "SELECT 1",
+  embeddedRegions: [{ from: 14, language: "python", to: 18 }],
+  text: "SELECT * FROM {df}",
 });
 const originalRevision = session.revision;
 const updatedRevision = session.update({
-  kind: "document",
+  embeddedRegions: [{ from: 14, language: "python", to: 23 }],
   baseRevision: originalRevision,
-  document: { kind: "replace", text: "SELECT 2" },
+  document: { kind: "replace", text: "SELECT * FROM {next_df}" },
 });
 if (session.isCurrent(originalRevision) || !session.isCurrent(updatedRevision)) {
   throw new Error("The packaged vNext session violated revision identity");