Skip to content

Commit fc01f90

Browse files
MagicalTuxclaude
MagicalTux
and
claude
committed
feat(eqp): render the SEARCH + LIST SUBQUERY for a seekable IN (SELECT) (B9a-seek)
A single non-correlated positive `col IN (SELECT …)` in the WHERE, where `col` is seekable (a secondary-index leading column, an INTEGER PRIMARY KEY referenced by name, or a `rowid` alias), now renders SQLite's per-candidate access plan byte-exact: SEARCH t USING INDEX tb (b=?) LIST SUBQUERY 1 SCAN u USING COVERING INDEX uy CREATE BLOOM FILTER The executor already seeks this shape — `fold_subquery_expr` evaluates the non-correlated subquery to a value list and `try_index_in` seeks per value; the gap was EQP-only. Closed by: - `in_select_folds_structurally` — the structural (non-running) half of `eval_foldable_in_select`, so `eqp_access` can plan the seek without evaluating the subquery (matching SQLite, which never runs it under EXPLAIN); - an `InSelect { negated: false }` arm in `placeholder_fold_where_inner` that folds a foldable IN to a placeholder `IN (0)` so the constraint collectors recognise the seek; - the LIST SUBQUERY gate now emits for the seek case too, but only when the rendered access line *is* the IN column's own seek (`(in_col=?)`, with an IPK referenced by name rendering as `(rowid=?)`). A competing equality/range seek on a different column declines rather than mis-render, keeping the "never emit a node into a non-matching plan" invariant. The `NOT IN` / unindexed-IN bare-SCAN case (B9a) is unchanged. Residual (a separate render node, not B9a-seek): a non-seekable / `NOT IN` whose subquery projects an *indexed* column — SQLite renders `USING INDEX <ix> FOR IN-OPERATOR` rather than `LIST SUBQUERY`+bloom; graphite still emits the bloom node (or, with a competing seek, the correct access line and no IN node). Correlated / compound bodies stay deferred. Verified vs sqlite3 3.50.4 (EQP + row parity: rowid, secondary index, empty list). `seek_scalar_subquery.rs`'s stale "IN (SELECT) stays SCAN" expectation is moved to `eqp_in_subquery.rs` as a now-rendered case. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 0744c0a commit fc01f90

4 files changed

Lines changed: 125 additions & 41 deletions

File tree

‎ROADMAP.md‎

Lines changed: 15 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1175,14 +1175,21 @@ temp-b-tree node (**B9d** subset). Details live in Track A's EQP paragraph and t
11751175
`eqp-derived-coroutine` / `planner-index-seeks` memories. Still open (rows already
11761176
correct for all of these — perf/EQP-fidelity, not correctness):
11771177

1178-
- **B9a-seek — positive `IN (SELECT …)` on an indexed/rowid column.** The `LIST
1179-
SUBQUERY` + `CREATE BLOOM FILTER` render for a non-correlated `NOT IN` / unindexed `IN`
1180-
is done (**B9a**); a positive `IN` on a *seekable* column, which SQLite serves with a
1181-
per-candidate `SEARCH t (col=?)`, still declines (graphite folds the `IN` in the
1182-
tree-walker and scans — no wrong node). Needs the executor to evaluate the
1183-
non-correlated subquery to a value list and seek per value (`find_in_constraint` /
1184-
`try_index_in` already seek a literal `IN` list), plus the outer-access EQP. A
1185-
compound / correlated body (`CORRELATED LIST SUBQUERY`) stays deferred.
1178+
- **B9a-seek — positive `IN (SELECT …)` on an indexed/rowid column. DONE.** A
1179+
non-correlated positive `IN` on a *seekable* column (secondary-index leading col,
1180+
`INTEGER PRIMARY KEY` by name, or `rowid` alias) now renders SQLite's per-candidate
1181+
`SEARCH t … (col=?)` followed by `LIST SUBQUERY 1` → the body plan → `CREATE BLOOM
1182+
FILTER`, byte-exact. The executor already folds the non-correlated subquery to a value
1183+
list and seeks per value (`fold_subquery_expr` → `try_index_in`); this closed the EQP
1184+
side by teaching `eqp_access`'s placeholder fold to recognise a structurally-foldable
1185+
`IN (SELECT)` (`in_select_folds_structurally`) and emitting the `LIST SUBQUERY` only
1186+
when the rendered access line *is* the IN column's seek (`(in_col=?)`), so a competing
1187+
equality/range seek on a different column declines rather than mis-render. Verified vs
1188+
sqlite3 3.50.4 (`eqp_in_subquery.rs`: EQP + row parity, rowid + secondary + empty list).
1189+
*Residual (separate, not B9a-seek):* a non-seekable / `NOT IN` whose subquery projects
1190+
an *indexed* column — SQLite renders `USING INDEX <ix> FOR IN-OPERATOR` rather than
1191+
`LIST SUBQUERY`+bloom; graphite still emits the bloom node there. Correlated / compound
1192+
bodies stay deferred (`CORRELATED LIST SUBQUERY`).
11861193
- **B9b — window-function EQP.** `… OVER (…)` renders `CO-ROUTINE (subquery-N)` over the
11871194
windowed input; the `(subquery-N)` label is codegen-order-fragile, so this needs a
11881195
deterministic-numbering model before it can be byte-exact.

‎src/exec/mod.rs‎

Lines changed: 76 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -789,6 +789,28 @@ impl Connection {
789789
&& self.compound_arms_computed(sel2)
790790
}
791791

792+
/// Whether an `IN (SELECT …)` candidate subquery is foldable to a value list
793+
/// *without running it* — the structural half of [`Self::eval_foldable_in_select`]
794+
/// (which additionally runs the body). Used to plan the `IN` seek in
795+
/// `eqp_access` without evaluating the subquery, mirroring the executor fold so
796+
/// the EQP and the seek agree. A bare-column candidate needs a resolvable single
797+
/// origin (for its affinity); a computed candidate needs every compound arm
798+
/// computed too.
799+
fn in_select_folds_structurally(&self, sel2: &Select) -> bool {
800+
if !self.vdbe_subquery_foldable(sel2) || sel2.columns.len() != 1 {
801+
return false;
802+
}
803+
let sql::ast::ResultColumn::Expr { expr, .. } = &sel2.columns[0] else {
804+
return false;
805+
};
806+
if is_bare_column_expr(expr) {
807+
self.subquery_column_origins(sel2)
808+
.is_some_and(|o| !o.is_empty())
809+
} else {
810+
self.compound_arms_computed(sel2)
811+
}
812+
}
813+
792814
fn eval_foldable_scalar(&self, sel2: &Select) -> Option<Value> {
793815
if !self.scalar_subquery_folds_structurally(sel2) {
794816
return None;
@@ -854,6 +876,25 @@ impl Connection {
854876
right: Box::new(subq_placeholder(right, changed)),
855877
}
856878
}
879+
// A positive `col IN (<foldable SELECT>)` seeks the `col` index per
880+
// candidate value; SQLite plans the `SEARCH` without evaluating the
881+
// subquery, so replace the candidate set with a single non-NULL
882+
// placeholder literal — the constraint collectors then recognize the `IN`
883+
// seek. The executor mirrors this by folding the subquery to its real
884+
// value list (`eval_foldable_in_select`) before its `try_index_in` seek.
885+
Expr::InSelect {
886+
expr,
887+
select,
888+
negated: false,
889+
} if self.in_select_folds_structurally(select) => {
890+
*changed = true;
891+
Expr::InList {
892+
expr: expr.clone(),
893+
list: alloc::vec![Expr::Literal(Literal::Integer(0))],
894+
negated: false,
895+
candidate_affinity: None,
896+
}
897+
}
857898
other => other.clone(),
858899
}
859900
}
@@ -14431,17 +14472,17 @@ impl Connection {
1443114472
}
1443214473
// A single non-correlated `[NOT] IN (SELECT …)` in the WHERE renders a
1443314474
// `LIST SUBQUERY 1` node (child = the body's plan, then a `CREATE BLOOM FILTER`
14434-
// sibling under it) after the scan. We render it only where the whole plan is
14435-
// provably byte-exact: graphite's access is a bare `SCAN {label}` — so there is
14436-
// no seek to diverge from SQLite's cost-model choice (an `… AND c=?` that
14437-
// graphite seeks makes the line a SEARCH and declines here, dodging the case
14438-
// where SQLite scans-plus-bloom where graphite would seek) — AND either the form
14439-
// is `NOT IN` (which never seeks the IN column) or the IN column is not seekable
14440-
// (so SQLite also scans; a positive `IN` on an indexed / rowid column, which
14441-
// SQLite SEARCHes per candidate, declines — that seek is roadmap B9a-seek).
14442-
if from.joins.is_empty()
14443-
&& single_scan_detail.as_deref() == Some(alloc::format!("SCAN {label}").as_str())
14444-
{
14475+
// sibling under it) after the access. It emits in two provably-byte-exact cases:
14476+
// - `NOT IN` / an IN column that is *not* seekable → graphite's access is a
14477+
// bare `SCAN {label}`, matching SQLite (which also scans);
14478+
// - a positive `IN` on a *seekable* (rowid / index-leading) column → the
14479+
// executor folds the subquery to a value list and seeks per candidate
14480+
// (`try_index_in`), and `eqp_access`'s placeholder fold renders the matching
14481+
// `SEARCH {label} … (col=?)` — but only when that access line *is* the IN
14482+
// column's seek (a competing equality/range on another column would make
14483+
// SQLite's cost-model choice diverge, so we require the rendered access to
14484+
// seek the IN column exactly — `(in_col=?)`).
14485+
if from.joins.is_empty() {
1444514486
if let Some((body, negated, operand)) = sel
1444614487
.where_clause
1444714488
.as_ref()
@@ -14450,14 +14491,36 @@ impl Connection {
1445014491
let operand_is_rowid = matches!(operand, Expr::Column { column, .. }
1445114492
if is_rowid_alias(column)
1445214493
&& !meta.columns.iter().any(|c| c.name.eq_ignore_ascii_case(column)));
14494+
let in_col_idx = col_index(operand, &meta.columns);
1445314495
let in_col_seekable = operand_is_rowid
14454-
|| col_index(operand, &meta.columns).is_some_and(|c| {
14496+
|| in_col_idx.is_some_and(|c| {
1445514497
meta.ipk == Some(c)
1445614498
|| self
1445714499
.indexes_of(&from.first.name)
1445814500
.is_ok_and(|ixs| ixs.iter().any(|i| i.cols.first() == Some(&c)))
1445914501
});
14460-
if (negated || !in_col_seekable) && self.eqp_scalar_bodies_renderable(&[body]) {
14502+
let bare_scan =
14503+
single_scan_detail.as_deref() == Some(alloc::format!("SCAN {label}").as_str());
14504+
// The seek-column render tag: a rowid / INTEGER-PRIMARY-KEY IN reads
14505+
// `(rowid=?)` (the IPK column renders as `rowid` in the access line even
14506+
// when referenced by its declared name), a secondary-index IN reads
14507+
// `(col=?)`.
14508+
let in_col_tag =
14509+
if operand_is_rowid || (in_col_idx.is_some() && in_col_idx == meta.ipk) {
14510+
Some(alloc::string::String::from("rowid"))
14511+
} else {
14512+
in_col_idx.map(|c| meta.columns[c].name.clone())
14513+
};
14514+
let seek_is_in_col = !negated
14515+
&& in_col_seekable
14516+
&& in_col_tag.as_deref().is_some_and(|nm| {
14517+
single_scan_detail.as_deref().is_some_and(|d| {
14518+
d.starts_with(alloc::format!("SEARCH {label}").as_str())
14519+
&& d.contains(alloc::format!("({nm}=?)").as_str())
14520+
})
14521+
});
14522+
let nonseek_case = (negated || !in_col_seekable) && bare_scan;
14523+
if (nonseek_case || seek_is_in_col) && self.eqp_scalar_bodies_renderable(&[body]) {
1446114524
let list_id = *next_id;
1446214525
*next_id += 1;
1446314526
out.push((list_id, parent, String::from("LIST SUBQUERY 1")));

‎tests/eqp_in_subquery.rs‎

Lines changed: 25 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -2,13 +2,17 @@
22
//! `LIST SUBQUERY 1` node (its body plan + a `CREATE BLOOM FILTER` child) after the
33
//! table access, matching SQLite. graphite used to render just the bare access.
44
//!
5-
//! Rendered only where the whole plan is provably byte-exact: graphite's access is a
6-
//! bare `SCAN` (so there is no seek to diverge from SQLite's cost-model choice), and
7-
//! either the form is `NOT IN` (which never seeks the IN column) or the IN column is
8-
//! not seekable (so SQLite also scans). A *positive* `IN` on an indexed / rowid column
9-
//! — which SQLite serves with a per-candidate `SEARCH` — declines (that seek is a
10-
//! separate follow-up), as do a correlated / compound / cross-position subquery.
11-
//! Verified vs the sqlite3 3.50.4 CLI.
5+
//! Two provably-byte-exact cases render:
6+
//! - `NOT IN` / an IN column that is not seekable → graphite's access stays a bare
7+
//! `SCAN` (so there is no seek to diverge from SQLite's cost-model choice), and
8+
//! SQLite also scans;
9+
//! - a *positive* `IN` on a seekable (rowid / index-leading) column → the executor
10+
//! folds the subquery to a value list and seeks per candidate (`try_index_in`), and
11+
//! the access renders the matching `SEARCH … (col=?)` in lockstep — but only when
12+
//! that access line *is* the IN column's seek (B9a-seek).
13+
//!
14+
//! A correlated / compound-body / cross-position subquery declines to the prior bare
15+
//! `SCAN t`. Verified vs the sqlite3 3.50.4 CLI.
1216
1317
#![cfg(feature = "std")]
1418

@@ -63,21 +67,25 @@ fn in_subquery_list_subquery_matches_sqlite() {
6367
// Positive IN on an UNINDEXED column → SQLite also scans.
6468
"SELECT * FROM t WHERE d IN (SELECT y FROM u)",
6569
"SELECT d FROM t WHERE d IN (SELECT y FROM u)",
70+
// B9a-seek: positive IN on a seekable column → SQLite SEARCHes per candidate,
71+
// and graphite now renders that SEARCH + LIST SUBQUERY + bloom in lockstep.
72+
"SELECT * FROM t WHERE b IN (SELECT y FROM u)", // secondary-index seek
73+
"SELECT * FROM t WHERE a IN (SELECT x FROM u)", // INTEGER PRIMARY KEY (by name)
74+
"SELECT * FROM t WHERE rowid IN (SELECT x FROM u)", // rowid alias
75+
"SELECT * FROM t WHERE b IN (SELECT y FROM u WHERE x>0)", // filtered body
76+
"SELECT * FROM t WHERE b IN (SELECT y FROM u) AND c=5", // no competing seek (c unindexed)
6677
] {
6778
assert_eq!(plan("sqlite3", BASE, q), plan(g, BASE, q), "plan for {q}");
6879
}
6980
}
7081

7182
#[test]
7283
fn in_subquery_out_of_subset_declines_to_bare_scan() {
73-
// These render a SEARCH / different node shape in sqlite that graphite doesn't
74-
// reproduce yet; graphite must keep its prior bare `SCAN t` (no LIST SUBQUERY /
75-
// bloom node emitted into a non-matching plan).
84+
// These render a different node shape in sqlite that graphite doesn't reproduce
85+
// yet; graphite must keep its prior bare `SCAN t` (no LIST SUBQUERY / bloom node
86+
// emitted into a non-matching plan).
7687
let g = env!("CARGO_BIN_EXE_graphitesql");
7788
for q in [
78-
"SELECT * FROM t WHERE b IN (SELECT y FROM u)", // positive IN on indexed col → SQLite SEARCHes
79-
"SELECT * FROM t WHERE a IN (SELECT x FROM u)", // rowid seek
80-
"SELECT * FROM t WHERE rowid IN (SELECT x FROM u)",
8189
"SELECT * FROM t WHERE b IN (SELECT y FROM u WHERE u.y=t.a)", // correlated
8290
"SELECT * FROM t WHERE b NOT IN (SELECT y FROM u UNION SELECT x FROM u)", // compound body
8391
"SELECT * FROM t WHERE d IN (SELECT y FROM u) AND (SELECT count(*) FROM w)>0", // cross-position
@@ -101,6 +109,10 @@ fn in_subquery_rows_unaffected() {
101109
"SELECT a FROM t WHERE b NOT IN (SELECT y FROM u) ORDER BY a",
102110
"SELECT a FROM t WHERE d IN (SELECT y FROM u) ORDER BY a",
103111
"SELECT count(*) FROM t WHERE c NOT IN (SELECT x FROM u)",
112+
// B9a-seek row parity: the per-candidate SEARCH must return the same rows.
113+
"SELECT a FROM t WHERE b IN (SELECT y FROM u) ORDER BY a", // secondary-index seek
114+
"SELECT a FROM t WHERE a IN (SELECT x FROM u) ORDER BY a", // rowid seek
115+
"SELECT a FROM t WHERE b IN (SELECT y FROM u WHERE x<0) ORDER BY a", // empty list
104116
] {
105117
assert_eq!(rows("sqlite3", &base, q), rows(g, &base, q), "rows for {q}");
106118
}

‎tests/seek_scalar_subquery.rs‎

Lines changed: 9 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -8,10 +8,12 @@
88
//! evaluates it, so even `b = (SELECT 1/0)` plans a `SEARCH`). Equality and range,
99
//! secondary index and INTEGER PRIMARY KEY.
1010
//!
11-
//! A *correlated* body, `EXISTS`, `IN (SELECT)`, and a bare-column subquery
11+
//! A *correlated* body, `EXISTS`, and a bare-column subquery
1212
//! (`(SELECT x FROM u)` — folding it would drop the column's affinity) do NOT seek —
1313
//! the outer table stays a `SCAN`, and the full `WHERE` is re-applied so rows are
14-
//! exact regardless. Verified vs the sqlite3 3.50.4 CLI.
14+
//! exact regardless. (A non-correlated positive `IN (SELECT …)` on a seekable column
15+
//! *does* seek — that is B9a-seek, covered by `eqp_in_subquery.rs`.) Verified vs the
16+
//! sqlite3 3.50.4 CLI.
1517
1618
#![cfg(feature = "std")]
1719

@@ -75,14 +77,14 @@ fn scalar_subquery_operand_seeks_like_sqlite() {
7577

7678
#[test]
7779
fn non_seekable_subquery_shapes_stay_scan() {
78-
// A correlated / EXISTS / IN (SELECT) / bare-column subquery must not seek the
79-
// outer table — graphite keeps the SCAN (results still correct via the WHERE
80-
// re-apply). (SQLite renders extra CORRELATED / LIST SUBQUERY nodes graphite does
81-
// not model, so we assert only that the outer access is not a SEARCH.)
80+
// A correlated / EXISTS / bare-column subquery must not seek the outer table —
81+
// graphite keeps the SCAN (results still correct via the WHERE re-apply). (SQLite
82+
// renders extra CORRELATED nodes graphite does not model, so we assert only that
83+
// the outer access is not a SEARCH.) A positive `IN (SELECT …)` on a seekable
84+
// column *does* seek now (B9a-seek); that is asserted in `eqp_in_subquery.rs`.
8285
let g = env!("CARGO_BIN_EXE_graphitesql");
8386
for q in [
8487
"SELECT * FROM t WHERE b=(SELECT y FROM u WHERE x=t.a)", // correlated
85-
"SELECT * FROM t WHERE b IN (SELECT x FROM u)", // IN (SELECT)
8688
"SELECT * FROM t WHERE EXISTS(SELECT 1 FROM u WHERE x=b)", // correlated EXISTS
8789
"SELECT * FROM t WHERE b=(SELECT x FROM u)", // bare-column projection
8890
] {

0 commit comments

Comments
 (0)