Skip to content

Commit ffa8229

Browse files
authored
feat(vnext): coordinate catalog epochs (#193)
Part of #169. - Add a package-private provider/scope epoch gate so catalog responses and invalidations cannot publish against mixed catalog state. - Share one subscription per scope with two-phase membership activation, single-use authenticated captures, monotonic arbitration, and exact revision fan-out. - Bound callback storms, FIFO admissions, memberships, scopes, and reentrant provider cleanup with fail-closed quarantine. - Retire failed targets atomically and sever provider/session closure graphs before external cleanup or retained-handle disposal. - Record the lifecycle, ordering, and resource contracts in ADR 0005. <!-- This is an auto-generated description by cubic. --> --- ## Summary by cubic Introduces a catalog epoch coordinator that gates responses and invalidations per scope to prevent mixed catalog state and ensure deterministic revision fan‑out, and isolates hostile cleanup promises. Part of #169. - **New Features** - One subscription per scope with two‑phase membership and authenticated, single‑use epoch captures. - Monotonic arbitration and exact revision fan‑out across sessions sharing a scope. - Bounds for callback storms and cleanups, FIFO admissions, and fail‑closed quarantine for misbehavior. - Drains async provider/session cleanup settlement, isolates hostile cleanup promises, and atomically retires failed targets with reentrant‑safe cleanup. - Updated ADR‑0005 to document lifecycle and ordering; added extensive tests and a benchmark with `bench:catalog-coordinator`. - **Migration** - `SqlRelationCatalogProvider.subscribe` now returns a this‑free cleanup function `SqlCatalogSubscriptionCleanup` (may be async) instead of a `SqlDisposable`; update providers to return `() => void | PromiseLike<void>`. - `SqlDisposable.dispose` is now this‑free: change to `dispose(this: void): void` and avoid using `this` inside dispose. <sup>Written for commit 3d66bd4. Summary will update on new commits.</sup> <a href="https://cubic.dev/pr/marimo-team/codemirror-sql/pull/193?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. -->
1 parent b77db7c commit ffa8229

7 files changed

Lines changed: 3850 additions & 16 deletions

docs/adr/0005-parser-independent-relation-completion.md

Lines changed: 112 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -481,18 +481,108 @@ The service reference-counts one provider subscription per provider
481481
configuration and scope, then fans invalidation out to subscribed sessions.
482482
The installed callback closure supplies authenticated provider and scope
483483
identity; the untrusted invalidation payload therefore contains only an epoch.
484-
Subscription membership is installed atomically before a provider can call
485-
back synchronously. A newly joining session captures an already-observed epoch
486-
without a synthetic revision bump. Disposal removes membership before
487-
unsubscribing or running external cleanup. Fifty sessions sharing a scope do
488-
not create fifty provider subscriptions.
484+
Membership uses a two-phase prepare/commit protocol: the session stores its
485+
prepared lease before commit can invoke provider code. Commit installs the
486+
membership atomically before a provider can call back synchronously. Because a
487+
subscription callback is always a change event, a valid callback fired during
488+
`subscribe` advances that committed member after subscription installation
489+
succeeds; it is not reclassified as an initial snapshot. A later joining
490+
session captures an already-observed epoch without a synthetic revision bump.
491+
Disposal removes membership before unsubscribing or running external cleanup.
492+
Retirement also clears the membership's target and coordinator link before
493+
that cleanup, so a consumer retaining a disposed lease retains only inert,
494+
bounded membership metadata. The coordinator stores only the copied provider
495+
ID and captured subscription function; it does not retain the unused search
496+
function. Coordinator disposal revokes that subscription reference before
497+
external cleanup, so a consumer retaining the disposed coordinator handle does
498+
not retain the provider graph.
499+
Fifty sessions sharing a scope do not create fifty provider subscriptions.
500+
The coordinator reserves membership state and capacity before reading the
501+
revision target callback. A throwing or invalid callback getter rolls that
502+
reservation back with `invalid-target` unavailable evidence. Reentrant
503+
coordinator disposal instead reports `disposed` and leaves no reserved
504+
membership or captured callback behind.
489505

490506
Each installed subscription has a service-owned incarnation identity captured
491507
by its callback. The identity is revoked before external unsubscribe or
492508
cleanup, and every callback checks it before entering the serialized epoch
493509
gate. Cleanup from a retired incarnation is keyed to that identity and cannot
494510
remove, mutate, or notify a reentrantly installed replacement.
495511

512+
The callback closes over a revocable cell rather than the service, scope state,
513+
or sessions. Retirement clears the cell before external cleanup, so a provider
514+
that retains an old callback retains only an inert bounded object. Synchronous
515+
callbacks are decoded into frozen epochs while `subscribe` is running; raw
516+
payloads and premature audience snapshots are not buffered. After the returned
517+
cleanup closure is validated, the buffered epochs are staged together in FIFO
518+
order. Each accepted baseline or advance snapshots active membership and
519+
installs the epoch as one serialized step with no external call between those
520+
operations. A member joined during one event's dispatch therefore
521+
participates in a later pending event, but not the event already committed.
522+
A thrown subscription or malformed returned cleanup value discards that buffer
523+
and disables automatic invalidation for the live scope; explicit search
524+
remains available.
525+
The returned cleanup closure is itself untrusted: the service captures only a
526+
function, calls it with `this === undefined` at most once, and isolates
527+
malformed values, thrown cleanup, and rejected or hostile thenable results.
528+
Detached cleanup settlement retains no coordinator state. A closure avoids a
529+
structural TypeScript contract that would accept class instances or inherited
530+
methods which the hostile runtime boundary could not safely validate. Dropping
531+
the last owner removes the complete scope incarnation before cleanup. A later
532+
join creates a new unobserved incarnation and may attempt a fresh subscription.
533+
534+
Malformed, duplicate, stale, or token-conflicting invalidations do not mutate
535+
state and do not tear down an otherwise valid subscription. Every raw callback
536+
still consumes the callback reset-window allowance. Exceeding that allowance
537+
revokes the incarnation and disables automatic invalidation before decoding
538+
further payloads. The conservative window resets from a zero-delay timer
539+
scheduled by the first callback. It intentionally makes no claim about formal
540+
JavaScript task or turn boundaries, while still preventing a provider from
541+
evading the allowance with a microtask chain. If hostile payload inspection
542+
synchronously reenters the same callback, the nested callback consumes that
543+
allowance but is ignored fail-closed; the coordinator does not retain the raw
544+
payload or invert callback order.
545+
546+
An epoch capture authenticates the exact provider configuration, scope
547+
incarnation, membership, observed epoch, and notification sequence. An
548+
equal-current response is publishable only if no accepted invalidation or
549+
higher response changed that sequence after capture. Otherwise it is
550+
superseded even when its epoch equals the newly current epoch. A capture is a
551+
single-use work token: submission claims it before queue admission, and replay
552+
is discarded even when the first submission was overloaded.
553+
554+
Epoch commands use one bounded, non-recursive FIFO drain. Its hard admission
555+
limit counts all work accepted during that drain, including commands already
556+
processed, so a one-at-a-time reentrant chain cannot evade the bound. For every
557+
accepted change, active membership is snapshotted and the epoch is immediately
558+
installed before revision preparation or any external listener runs. All
559+
affected session revisions then change before dispatch. Reentrant commands run
560+
only after the current change has reached every still-active member, preserving
561+
event order. The core does not coalesce revision events. The CodeMirror adapter
562+
may coalesce the resulting UI refresh, as described below. Admission fails
563+
before epoch mutation:
564+
an overflowing provider callback revokes automatic invalidation for that
565+
incarnation, while an overflowing response observation settles with closed
566+
overload evidence.
567+
Invalid, retired, replayed, disposed, and overflowing response submissions
568+
return closed settlement evidence without invoking a callback. Only an admitted
569+
FIFO command invokes its completion callback, so rejection cannot recursively
570+
reenter the coordinator's drain.
571+
572+
A revision target prepares its state change and returns a listener-dispatch
573+
closure. Returning `null`, returning a non-function, or throwing retires that
574+
membership. Retirement detaches state immediately, but hostile provider cleanup
575+
is deferred until every remaining target has prepared. A higher response epoch
576+
installed before preparation remains observed; if its submitting membership is
577+
retired during preparation, its response settles as retired while other
578+
successfully prepared members still dispatch. Coordinator disposal takes
579+
precedence over that retired evidence. The outermost cleanup barrier admits and
580+
invokes at most 1,024 captured cleanups. If reentrant work attempts to admit a
581+
1,025th cleanup, the coordinator quarantines itself immediately: all cells and
582+
memberships become inert and all not-yet-invoked cleanup references are
583+
cleared without calling more provider code. Cleanup is never continued from a
584+
timer.
585+
496586
The session exposes a disposable revision-change subscription for
497587
service-originated changes:
498588

@@ -671,6 +761,12 @@ The initial checked limits are:
671761
| Components per search path | 4 |
672762
| Total catalog context | 16,384 UTF-16 units |
673763
| Configured relation catalog providers | 1 |
764+
| Live catalog scopes per service | 128 |
765+
| Catalog memberships per service | 1,024 |
766+
| Catalog memberships per scope | 256 |
767+
| Raw subscription callbacks per reset window | 256 |
768+
| Catalog epoch admissions per drain | 1,024 |
769+
| Provider cleanups per outer barrier | 1,024 |
674770
| Catalog results per search | 100 |
675771
| Completion results after composition | 100 |
676772
| Provider ID | 256 UTF-16 units |
@@ -815,19 +911,22 @@ Mixed cached assets fail the exact version check and retire the generation.
815911
2. Attach embedded regions to session open/update transactions atomically.
816912
3. Add the bounded partial-`SELECT` query-site recognizer.
817913
4. Add the bounded CTE frame and visibility recognizer.
818-
5. Add the service-owned catalog coordinator, subscriptions, cache,
819-
in-flight sharing, cancellation, and provider contract suite.
820-
6. Add the session completion method and deterministic composition.
821-
7. Add the separate CodeMirror adapter and packed/browser fixtures.
822-
8. Add the pinned packed/browser/runtime marimo fixture and 1/10/50-editor
914+
5. Add the service-owned scoped epoch/subscription coordinator and its hostile
915+
lifecycle contract suite. It does not invoke provider search or retain
916+
result pages.
917+
6. Add catalog search scheduling, cache, in-flight sharing, cancellation,
918+
retry gates, and availability observers over the proven epoch coordinator.
919+
7. Add the session completion method and deterministic composition.
920+
8. Add the separate CodeMirror adapter and packed/browser fixtures.
921+
9. Add the pinned packed/browser/runtime marimo fixture and 1/10/50-editor
823922
performance and leak evidence.
824-
9. Validate both the in-memory/notebook and hierarchical remote provider
923+
10. Validate both the in-memory/notebook and hierarchical remote provider
825924
shapes, then stabilize the declarations and public export surface.
826-
10. Design scoped parser semantics and protocol v2 against PostgreSQL and
925+
11. Design scoped parser semantics and protocol v2 against PostgreSQL and
827926
BigQuery corpora before any additional parser-derived scope feature
828927
consumes it.
829928

830-
Breaking refinement remains allowed through step 8. The provider and
929+
Breaking refinement remains allowed through step 9. The provider and
831930
completion types become stable only after the working vertical slice, reusable
832931
provider contract suite, two materially different provider shapes, marimo
833932
packed/browser integration, hostile decoding, cancellation, epoch and

package.json

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@
2727
"test:worker-placement": "node ./scripts/worker-placement.mjs",
2828
"test:integrity": "node ./scripts/check-test-integrity.mjs",
2929
"bench:catalog-boundary": "vitest bench --run src/vnext/__tests__/relation-catalog-boundary.bench.ts",
30+
"bench:catalog-coordinator": "vitest bench --run src/vnext/__tests__/relation-catalog-epoch-coordinator.bench.ts",
3031
"bench:local-relation-site": "vitest bench --run src/vnext/__tests__/local-relation-site.bench.ts",
3132
"bench:parser-adapter": "vitest bench --run src/vnext/__tests__/node-sql-parser-adapter.bench.ts",
3233
"bench:query-site": "vitest bench --run src/vnext/__tests__/query-site.bench.ts",

0 commit comments

Comments
 (0)