feat(vnext): index bounded CTE visibility

[Light2Dark]

Summary

• add a private, cursor-independent bounded CTE layout and visibility index for PostgreSQL, DuckDB, BigQuery, and Dremio grammar subsets
• preserve committed declarations separately from unfinished-body drafts, with correct nonrecursive order, nested shadowing, recursive withholding, duplicate blocking, and source-faithful insertion evidence
• use dialect-owned symmetric tri-state identifier comparison rather than generic folding or a second nullable comparison-key model
• preserve exact proven prefixes across lexer exhaustion and model cursor positions correctly at EOF, closing delimiters, barriers, and scope boundaries
• fail closed on hostile dialect callbacks/data, embedded regions, malformed syntax, unknown equivalence, and every checked resource boundary
• extend ADR 0005 and the provisional public relation-completion type contract for tri-state equality/prefix behavior and CTE-scope uncertainty

Bounds and performance

• active statement: 65,536 UTF-16 units
• shared lexical tokens: 16,384
• parenthesis/query depth: 128
• CTE frames: 256
• CTE declarations: 256
• identifier segment: 256 UTF-16 units

Representative local means:

• ordinary 10 KiB statement: ~0.37 ms
• 256 declarations with pairwise equivalence validation: ~3.0 ms
• 128 nested CTE frames: ~1.0 ms
• 256 sequential incomplete frames: ~0.15 ms
• cached 256-declaration projection: ~0.022 ms

Verification

• 1,587 tests passed, plus 1 governed expected failure
• changed runtime coverage: 96.96% statements, 95.88% branches, 100% functions, 96.94% lines
• all source, test, vNext exact/loose-optional, and demo typechecks pass
• oxlint and test-integrity gates pass
• package smoke, production build, demo build, and 7 Playwright browser tests pass
• three independent exact-head adversarial reviewers approved 8a1572369a171f49d3e23aeed4993aceb88a2cb2

Deferred follow-up

The bounded grammar currently authenticates SELECT query leaders. Dialect-owned support for PostgreSQL VALUES/data-modifying CTE bodies, DuckDB FROM-first queries, and additionally parenthesized BigQuery recursive terms remains an explicit follow-up before relation completion is feature-complete.

Part of #169.

---

Summary by cubic

Adds a bounded CTE layout and visibility index for vNext to improve relation completion across PostgreSQL, DuckDB, BigQuery, and Dremio, and now rejects invalid CTE cursor positions to avoid bogus suggestions. Introduces tri-state CTE identifier comparison and strict, fail-closed resource limits; part of #169.

• New Features

  • Private, cursor‑independent CTE index with correct order, shadowing, duplicate blocking, and recursive withholding.
  • Closed, dialect‑owned grammar per engine with a documented acceptance matrix; no cross‑dialect borrowing.
  • Tri‑state equality and prefix matching through the dialect runtime; preserves exact prefixes at EOF and scope boundaries.
  • Fast, bounded visibility via visibleSqlCtesAt; now rejects invalid cursor positions and the lexer reports limit hits with resourceAt, failing closed on malformed input.

• Migration

  • In SqlRelationCompletionDialectRuntime, replace cteIdentifiersEqual with compareCteIdentifiers returning "equal" | "distinct" | "unknown".
  • Implement cteIdentifierMatchesPrefix returning "match" | "no-match" | "unknown".

^{Written for commit 7302de4. Summary will update on new commits.}

[github-actions]

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	96.95% (🎯 91%)	637 / 657
🔵	Statements	96.97% (🎯 91%)	641 / 661
🔵	Functions	100% (🎯 90%)	49 / 49
🔵	Branches	95.89% (🎯 85%)	444 / 463

File Coverage

File	Stmts	Branches	Functions	Lines	Uncovered Lines
Changed Files
src/vnext/bounded-sql-lexer.ts	100%	100%	100%	100%
src/vnext/cte-layout.ts	96.51%	95.06%	100%	96.49%	328, 483, 497-499, 713, 808-810, 853, 1255, 1301, 1520-1523, 1641, 1689-1691
src/vnext/relation-completion-types.ts	0%	0%	0%	0%

Generated in workflow #727 for commit 7302de4 by the Vitest Coverage Report Action

[Copilot]

Pull request overview

Introduces a new vNext, bounded (resource-limited) CTE layout + visibility index to improve relation completion correctness across multiple SQL dialect subsets, and updates the dialect runtime contract to support tri-state CTE identifier equality/prefix matching.

Changes:

• Add a private, bounded CTE layout builder (analyzeSqlCteLayout) and visibility projection (visibleSqlCtesAt) with explicit “partial/unavailable” fail-closed behavior.
• Extend vNext relation-completion runtime types to use tri-state CTE identifier comparison and prefix matching.
• Enhance bounded SQL lexer limit reporting with resourceAt, plus add extensive tests/benchmarks and ADR updates.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
test/vnext-types/marimo-relation-completion.test-d.ts	Updates the d.ts contract test to the new tri-state CTE runtime APIs.
src/vnext/relation-completion-types.ts	Adds tri-state CTE comparison/prefix types and updates the dialect runtime interface accordingly.
src/vnext/cte-layout.ts	New bounded CTE layout + visibility implementation with resource limits and fail-closed behavior.
src/vnext/bounded-sql-lexer.ts	Adds resourceAt to pinpoint where lexer resource limits were hit.
src/vnext/tests/cte-layout.test.ts	Adds comprehensive unit tests for CTE layout/visibility correctness and boundary conditions.
src/vnext/tests/cte-layout.bench.ts	Adds vitest benchmarks for representative CTE layout workloads.
src/vnext/tests/bounded-sql-lexer.test.ts	Extends lexer tests to validate resourceAt behavior at resource boundaries.
docs/adr/0005-parser-independent-relation-completion.md	Updates ADR with the bounded grammar matrix, tri-state identifier comparison, and CTE visibility semantics.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[cubic-dev-ai]

All reported issues were addressed across 8 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>