feat(vnext): recognize bounded relation query sites by Light2Dark · Pull Request #186 · marimo-team/codemirror-sql

Light2Dark · 2026-07-24T23:00:11Z

Summary

add a private parser-independent partial SELECT relation-site recognizer
authenticate full relation paths and statement-relative UTF-16 replacement ranges
enforce dialect-owned identifier/path decoding and path depth under a global safety ceiling
recognize base FROM, qualified prefixes, aliases, joins, same-depth commas, nested queries, and dialect-owned NATURAL joins
authenticate the bounded USING(identifier [, identifier ...]) grammar before crossing a join constraint
deliberately fail closed on ON until a parser-backed or separately specified expression recognizer can prove its boundary
preserve frame-local nested-query recognition after proven relation transitions and valid USING clause exits

Safety and performance

active statement: 65,536 UTF-16 units
lexemes: 16,384
nesting: 128
path: dialect limit under global 32
decoded identifier segment: 256 UTF-16 units
exact 10 KiB statement: about 0.56 ms mean
1,000 classified aliases: about 0.36 ms mean
1,000 authenticated USING columns: about 0.21 ms mean
recognizer state remains bounded per query frame and per active USING constraint

Verification

1,537 tests passed plus 1 expected failure
changed coverage: 96.84% statements, 95.76% branches, 100% functions, 96.83% lines
repository coverage: 95.31% statements, 92.04% branches, 96.14% functions, 95.30% lines
source, test, loose-optional, and demo typechecks pass
repository oxlint, test-integrity, and diff checks pass
browser, package, worker-placement, demo, and benchmark gates pass
three independent adversarial reviewers approved exact commit 01cd8d29bf99a7c07f6a72ca746da328847c6dfb

Part of #169.

github-actions · 2026-07-24T23:01:18Z

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	96.83% (🎯 91%)	612 / 632
🔵	Statements	96.84% (🎯 91%)	614 / 634
🔵	Functions	100% (🎯 90%)	36 / 36
🔵	Branches	95.77% (🎯 85%)	657 / 686

File Coverage

File	Stmts	Branches	Functions	Lines	Uncovered Lines
Changed Files
src/vnext/query-site.ts	96.84%	95.77%	100%	96.83%	351-352, 620, 773-774, 800, 887, 1003, 1089, 1175, 1227, 1230, 1235, 1330, 1418, 1504-1507, 1545, 1553-1556

Generated in workflow #720 for commit 01cd8d2 by the Vitest Coverage Report Action

cubic-dev-ai

All reported issues were addressed

_{Reply with feedback, questions, or to request a fix.

Re-trigger cubic}

cubic-dev-ai

1 issue found across 3 files (changes from recent commits).

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="src/vnext/query-site.ts">

<violation number="1" location="src/vnext/query-site.ts:1388">
P2: Malformed `NATURAL LEFT(x)` sequences can now produce relation completions instead of failing closed because `(` clears compound NATURAL join prefixes. Preserve the function-call recovery only for non-compound prefixes; mark `natural-*` prefixes ambiguous when `JOIN` is absent.</violation>
</file>

_{Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic}

cubic-dev-ai · 2026-07-24T23:30:01Z

+        if (code === 40) {
+          punctuationFrame.joinPrefix = null;
+        } else {
+          markUnavailable(punctuationFrame, "ambiguous-query-site");
+        }


P2: Malformed NATURAL LEFT(x) sequences can now produce relation completions instead of failing closed because ( clears compound NATURAL join prefixes. Preserve the function-call recovery only for non-compound prefixes; mark natural-* prefixes ambiguous when JOIN is absent.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At src/vnext/query-site.ts, line 1388: <comment>Malformed `NATURAL LEFT(x)` sequences can now produce relation completions instead of failing closed because `(` clears compound NATURAL join prefixes. Preserve the function-call recovery only for non-compound prefixes; mark `natural-*` prefixes ambiguous when `JOIN` is absent.</comment> <file context> @@ -1253,12 +1381,15 @@ export function recognizeSqlRelationQuerySite( punctuationFrame.joinPrefix !== null ) { - punctuationFrame.joinPrefix = null; + if (code === 40) { + punctuationFrame.joinPrefix = null; + } else { </file context>

Suggested change

if (code === 40) {

punctuationFrame.joinPrefix = null;

} else {

markUnavailable(punctuationFrame, "ambiguous-query-site");

}

if (

code === 40 &&

!punctuationFrame.joinPrefix.startsWith("natural-")

) {

punctuationFrame.joinPrefix = null;

} else {

markUnavailable(punctuationFrame, "ambiguous-query-site");

}

cubic-dev-ai

All reported issues were addressed across 4 files (changes from recent commits).

_{Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic}

## Summary - extract the streaming bounded SQL lexer from the relation-site state machine into one package-private module - preserve the exact PostgreSQL, DuckDB, BigQuery, and Dremio lexical profiles, UTF-16 offsets, embedded-region barriers, quote/comment behavior, one-token pushback, and 16,384-lexeme ceiling - keep query-site keyword, comment-cursor, region, and resource semantics local to the consumer - translate generic lexer resource evidence through an exhaustive package-owned map - add direct boundary tests without exposing tokens or lexer APIs from the package This is a zero-semantics prerequisite for the separate bounded CTE layout/visibility recognizer. The recognizers will initially use separate streaming traversals over the same lexical implementation; any traversal fusion remains benchmark-driven. ## Performance An initial broader helper extraction caused a reproducible Vite SSR namespace-call regression on the hot path. The boundary was narrowed before commit. Stable means at the exact head are back at the PR #186 baseline: - exact 10 KiB statement: about 0.56–0.59 ms - 1,000 classified aliases: about 0.36–0.38 ms - 1,000 authenticated `USING` columns: about 0.21 ms The lexer remains streaming and does not allocate a token tape. ## Verification - 1,543 tests passed plus 1 expected failure - changed coverage: 97.05% statements, 95.79% branches, 100% functions, 97.04% lines - bounded lexer coverage: 98.92% statements/lines, 98.78% branches, 100% functions - repository coverage: 95.34% statements, 92.05% branches, 96.15% functions, 95.33% lines - source, test, loose-optional, and demo typechecks pass - repository oxlint, test-integrity, and diff checks pass - browser, package, worker-placement, demo, and benchmark gates pass - 20,000 deterministic differential lexer comparisons passed across dialects, masked regions, subranges, UTF-16, comments, quotes, punctuation, and pushback - independent SQL/API and concurrency/performance reviewers approved exact commit `ed079df0f68aadaf8957a7ca5e6c497e91b74860` Part of #169.  --- ## Summary by cubic Refactored vNext to share a streaming bounded SQL lexer and the embedded-region lookup, and switched `query-site` to use them. Behavior is unchanged across PostgreSQL, DuckDB, BigQuery, and Dremio; this unblocks the bounded CTE recognizer in #169. - **Refactors** - Moved the lexer into package-private `src/vnext/bounded-sql-lexer.ts`. - Centralized `findSqlEmbeddedRegionAtOrAfter` in `src/vnext/source.ts` and reused it in the lexer and `query-site` (with tests). - Preserves lexical profiles, UTF-16 offsets, embedded-region barriers, quote/comment rules, one-token pushback, and the 16,384-lexeme cap. - Kept consumer-specific semantics in `query-site`; mapped lexer resource signals to local `query-site` resources. - Removed duplicated lexer and region-lookup code from `src/vnext/query-site.ts` and wired it to the shared modules. <sup>Written for commit 76190f9. Summary will update on new commits.</sup> <a href="https://cubic.dev/pr/marimo-team/codemirror-sql/pull/187?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>

Light2Dark added 5 commits July 25, 2026 06:55

feat(vnext): recognize bounded relation query sites

e05f1ab

fix(vnext): recognize quoted relation aliases

b0e7978

fix(vnext): harden relation query recognition

bdb91f1

test(vnext): cover chained join modifiers

aba3f22

docs(vnext): clarify dialect path depth budget

31c873b

Light2Dark mentioned this pull request Jul 24, 2026

Roadmap: shared SQL language service architecture and correctness gaps #169

Open

test(vnext): use spell-safe identifier fixture

12c7ac3

cubic-dev-ai Bot reviewed Jul 24, 2026

View reviewed changes

Comment thread src/vnext/query-site.ts

Comment thread src/vnext/__tests__/query-site.bench.ts Outdated

fix(vnext): authenticate relation continuations

8fc0a97

cubic-dev-ai Bot reviewed Jul 24, 2026

View reviewed changes

fix(vnext): authenticate join continuations

eea3f34

cubic-dev-ai Bot reviewed Jul 24, 2026

View reviewed changes

Comment thread src/vnext/query-site.ts

fix(vnext): bound identifier callbacks

01cd8d2

Light2Dark merged commit 8347817 into dev-refactor Jul 25, 2026
9 checks passed

Light2Dark deleted the codex/vnext-query-site-recognizer branch July 25, 2026 00:11

Light2Dark mentioned this pull request Jul 25, 2026

refactor(vnext): share bounded SQL lexer #187

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(vnext): recognize bounded relation query sites - #186

feat(vnext): recognize bounded relation query sites#186
Light2Dark merged 9 commits into
dev-refactorfrom
codex/vnext-query-site-recognizer

Light2Dark commented Jul 24, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 24, 2026 •

edited

Loading

Uh oh!

cubic-dev-ai Bot left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment

Uh oh!

cubic-dev-ai Bot Jul 24, 2026

Uh oh!

cubic-dev-ai Bot left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

Light2Dark commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Safety and performance

Verification

Uh oh!

github-actions Bot commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Coverage Report

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Light2Dark commented Jul 24, 2026 •

edited

Loading

github-actions Bot commented Jul 24, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading