feat(eqp): render the FOR IN-OPERATOR node for an indexed IN-subquery

When a scanned outer's WHERE has a single `col [NOT] IN (SELECT <c> FROM <table>
[ORDER BY …])` whose subquery projects one plain *indexed* column, SQLite
evaluates the IN by iterating that column's index rather than materializing the
result, and renders a single `… FOR IN-OPERATOR` plan node in place of the
`LIST SUBQUERY` / `CREATE BLOOM FILTER` subtree. graphite now matches: a
secondary index leading with the column renders `USING INDEX <name> FOR
IN-OPERATOR`; the rowid / INTEGER PRIMARY KEY renders `USING ROWID SEARCH ON
TABLE <table> FOR IN-OPERATOR`.

The new `in_operator_index_node` gates on a bare-`SCAN` outer plus a simple body
(no WHERE/GROUP/HAVING/DISTINCT/LIMIT/OFFSET/join/compound/CTE/window and a
single plain-column projection). When two or more plain indexes lead with the
column, which one SQLite iterates is a cost-model tiebreak unverifiable against
the stat1-only oracle, so those keep the `LIST SUBQUERY` form. This is an
EXPLAIN-QUERY-PLAN-only change; executed results are unaffected. Verified
byte-for-byte vs sqlite3 3.50.4 (`tests/eqp_in_operator.rs`).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: MagicalTux

ROADMAP.md

@@ -1244,14 +1244,18 @@ correct for all of these — perf/EQP-fidelity, not correctness):
   when the rendered access line *is* the IN column's seek (`(in_col=?)`), so a competing
   equality/range seek on a different column declines rather than mis-render. Verified vs
   sqlite3 3.50.4 (`eqp_in_subquery.rs`: EQP + row parity, rowid + secondary + empty list).
-  *Residual (separate, not B9a-seek):* a non-seekable / `NOT IN` whose subquery projects
-  an *indexed* column — SQLite renders `USING INDEX <ix> FOR IN-OPERATOR` rather than
-  `LIST SUBQUERY`+bloom; graphite still emits the bloom node there. Correlated / compound
-  bodies stay deferred (`CORRELATED LIST SUBQUERY`).
-
-With B9a-seek shipped, the only open EQP-fidelity threads are the `FOR IN-OPERATOR`
-render residual above and **B9b** (window EQP), now confirmed deferred by design —
-see below.
+  *`FOR IN-OPERATOR` (DONE 2026-07-05):* a scanned outer whose `[NOT] IN
+  (SELECT <col> FROM <table> [ORDER BY …])` projects a single *indexed* plain column now
+  renders SQLite's `USING INDEX <ix> FOR IN-OPERATOR` (secondary index) / `USING ROWID
+  SEARCH ON TABLE <t> FOR IN-OPERATOR` (rowid / INTEGER PRIMARY KEY) in place of the
+  `LIST SUBQUERY`+bloom subtree (`in_operator_index_node`, gated on a bare-`SCAN` outer +
+  a simple, WHERE/LIMIT/DISTINCT/GROUP/join/expression-free body). When two+ plain indexes
+  lead with the column the choice is a cost-model tiebreak we don't reproduce → it stays on
+  `LIST SUBQUERY` (`eqp_in_operator.rs`). Correlated / compound bodies stay deferred
+  (`CORRELATED LIST SUBQUERY`).
+
+With B9a-seek and `FOR IN-OPERATOR` shipped, the only open EQP-fidelity thread is **B9b**
+(window EQP), now confirmed deferred by design — see below.
 
 **Blocked / deferred by design:**
 - **B9h — cost-model single-table index *choice*.** SQLite prefers, among indexes
@@ -1496,10 +1500,12 @@ reasonable order:
 1. **B5b-2 — live storage cursors on the VDBE.** The largest remaining VDBE piece;
    it turns the materialized inner join into a seek-driven one and is the
    prerequisite for streaming correlated subqueries and windows. Perf/coverage,
-   parity-gated, low risk. **B5b-2a landed** (INNER-join inner rowid seek on an
-   INTEGER PRIMARY KEY — live `TableCursor::seek`, no inner materialization);
-   remaining sub-steps (in-interpreter seek opcodes, secondary-index / WITHOUT
-   ROWID seeks, N-table chains) tracked under §4's B5b-2 entry.
+   parity-gated, low risk. **B5b-2a–2d landed** (INNER *and* LEFT inner rowid seek on an
+   INTEGER PRIMARY KEY, compound-`ON` conjunct seek, and an N-table left-deep chain of
+   ipk seeks — live `TableCursor::seek`, no inner materialization); the remaining
+   sub-step is the in-interpreter `OpenRead`/`SeekRowid` opcodes (moving the seek into
+   bytecode — an architectural refactor with no user-visible behavior change), plus the
+   affinity-blocked secondary-index / `WITHOUT ROWID` seeks, tracked under §4's B5b-2 entry.
 2. **B5c-2 — correlated subqueries on the VDBE**, once B5b-2 lands the live-cursor
    machinery.
 3. **C9a → C9d — the concurrency story** (persistent read locks in `src/pager/`,
@@ -1512,10 +1518,13 @@ reasonable order:
    prepare pass for the lazy-validation gaps.
 6. **`EXPLAIN QUERY PLAN` fidelity (Track B) — essentially closed.** The whole B9
    cluster shipped in 2026-07 (B9a incl. the seekable-`IN` render, B9c–B9g, the B9d
-   subset). What remains is deferred by design: the cost-model index-choice items
-   (**B9h**, **B9j**), and **B9b** window EQP — whose co-routine body is itself the B9h
-   index choice (see §4). The lone open non-blocked residual is the `FOR IN-OPERATOR`
-   render node for a `NOT IN`/unindexed subquery over an indexed column.
+   subset), and the **`FOR IN-OPERATOR`** render node landed 2026-07-05 (a scanned
+   outer whose `[NOT] IN (SELECT <indexed-col> FROM <table>)` iterates the subquery
+   column's index — `USING INDEX <name>` / `USING ROWID SEARCH ON TABLE <t>` — instead
+   of a `LIST SUBQUERY`; an ambiguous multi-index choice stays on `LIST SUBQUERY`, a
+   cost-model tiebreak deliberately not chased). What remains is deferred by design:
+   the cost-model index-choice items (**B9h**, **B9j**), and **B9b** window EQP — whose
+   co-routine body is itself the B9h index choice (see §4).
 
 **Deferred / blocked** (documented in §4): **B1b** join reordering and **B4**
 `sqlite_stat4` (diverge from / unverifiable against the stat1-only oracle);

src/exec/mod.rs

@@ -14803,7 +14803,21 @@ impl Connection {
                         })
                     });
                 let nonseek_case = (negated || !in_col_seekable) && bare_scan;
-                if (nonseek_case || seek_is_in_col) && self.eqp_scalar_bodies_renderable(&[body]) {
+                // With a bare `SCAN` outer, a simple indexed-column subquery is
+                // evaluated by iterating that index (a single `… FOR IN-OPERATOR`
+                // node) rather than materializing a `LIST SUBQUERY` + bloom filter.
+                let in_op_node = if nonseek_case {
+                    self.in_operator_index_node(body)
+                } else {
+                    None
+                };
+                if let Some(node) = in_op_node {
+                    let n_id = *next_id;
+                    *next_id += 1;
+                    out.push((n_id, parent, node));
+                } else if (nonseek_case || seek_is_in_col)
+                    && self.eqp_scalar_bodies_renderable(&[body])
+                {
                     let list_id = *next_id;
                     *next_id += 1;
                     out.push((list_id, parent, String::from("LIST SUBQUERY 1")));
@@ -15116,6 +15130,103 @@ impl Connection {
         Ok(true)
     }
 
+    /// When an `[NOT] IN (SELECT …)` is evaluated by iterating an index on the
+    /// subquery's column instead of materializing its result, SQLite renders a
+    /// single `… FOR IN-OPERATOR` node (a child of the outer `SCAN`) in place of
+    /// the `LIST SUBQUERY` / `CREATE BLOOM FILTER` subtree. This happens for a
+    /// *simple* `SELECT <col> FROM <table> [ORDER BY …]` whose single projected
+    /// column is a plain column that is indexed: a secondary index leading with
+    /// the column renders `USING INDEX <name> FOR IN-OPERATOR`; the rowid /
+    /// INTEGER PRIMARY KEY renders `USING ROWID SEARCH ON TABLE <table> FOR
+    /// IN-OPERATOR`. Any `WHERE`/`GROUP BY`/`HAVING`/`DISTINCT`/`LIMIT`/`OFFSET`,
+    /// a join, a compound/CTE, an expression projection, or an unindexed column
+    /// disqualifies it (→ `None`, so the caller keeps the `LIST SUBQUERY` form).
+    fn in_operator_index_node(&self, body: &Select) -> Option<String> {
+        if body.distinct
+            || body.where_clause.is_some()
+            || !body.group_by.is_empty()
+            || body.having.is_some()
+            || body.limit.is_some()
+            || body.offset.is_some()
+            || !body.compound.is_empty()
+            || !body.ctes.is_empty()
+            || !body.window_defs.is_empty()
+        {
+            return None;
+        }
+        let from = body.from.as_ref()?;
+        if !from.joins.is_empty() {
+            return None;
+        }
+        let tref = &from.first;
+        if tref.subquery.is_some()
+            || tref.tvf_args.is_some()
+            || tref.schema.is_some()
+            || self.is_bare_tvf(tref)
+            || self.is_view(&tref.name)
+        {
+            return None;
+        }
+        // A single plain-column projection (no expression, no `*`).
+        if body.columns.len() != 1 {
+            return None;
+        }
+        let ResultColumn::Expr { expr, .. } = &body.columns[0] else {
+            return None;
+        };
+        let mut proj = expr;
+        while let Expr::Paren(inner) = proj {
+            proj = inner;
+        }
+        let Expr::Column {
+            column,
+            table,
+            schema: None,
+            ..
+        } = proj
+        else {
+            return None;
+        };
+        let meta = self.table_meta(&tref.name, tref.alias.as_deref()).ok()?;
+        // A qualifier on the projected column must name the subquery's table.
+        if let Some(t) = table {
+            let qual = tref.alias.as_deref().unwrap_or(&tref.name);
+            if !t.eq_ignore_ascii_case(qual) {
+                return None;
+            }
+        }
+        let col_idx = meta
+            .columns
+            .iter()
+            .position(|c| c.name.eq_ignore_ascii_case(column));
+        // The rowid / INTEGER PRIMARY KEY of a rowid table → ROWID search form.
+        let is_rowid = !meta.without_rowid
+            && (col_idx == meta.ipk && col_idx.is_some()
+                || (is_rowid_alias(column) && col_idx.is_none()));
+        if is_rowid {
+            return Some(alloc::format!(
+                "USING ROWID SEARCH ON TABLE {} FOR IN-OPERATOR",
+                tref.name
+            ));
+        }
+        // A plain (non-partial, non-expression) secondary index leading with the
+        // column → index-iteration form. Only when the choice is *unambiguous*:
+        // if two or more plain indexes lead with the column, which one SQLite
+        // iterates is a cost-model tiebreak (index width / uniqueness) we can't
+        // reproduce against the stat1-only oracle, so defer to the `LIST SUBQUERY`
+        // form (its pre-existing render) rather than guess the wrong index name.
+        let ci = col_idx?;
+        let ixs = self.indexes_of(&tref.name).ok()?;
+        let mut leading = ixs.iter().filter(|i| {
+            i.partial.is_none() && i.key_exprs.is_none() && i.cols.first() == Some(&ci)
+        });
+        let ix = leading.next()?;
+        if leading.next().is_some() {
+            return None;
+        }
+        Some(alloc::format!("USING INDEX {} FOR IN-OPERATOR", ix.name))
+    }
+
     /// [`eqp_access`](Self::eqp_access), then collapse a *secondary*-index seek to a
     /// plain `SCAN` when the table carries a `NOT INDEXED` hint — the hint forbids
     /// every secondary index (including an implicit `sqlite_autoindex_…` for a

tests/eqp_in_operator.rs

@@ -0,0 +1,105 @@
+//! `[NOT] IN (SELECT …)` over an *indexed* subquery column, evaluated by
+//! iterating that index rather than materializing the result, renders SQLite's
+//! `… FOR IN-OPERATOR` plan node in place of the `LIST SUBQUERY` / `CREATE BLOOM
+//! FILTER` subtree. This fires only when the outer table is scanned (its IN
+//! column is not itself index-seekable) and the subquery is a *simple*
+//! `SELECT <col> FROM <table> [ORDER BY …]` whose single plain column is
+//! indexed: a secondary index leading with the column → `USING INDEX <name> FOR
+//! IN-OPERATOR`; the rowid / INTEGER PRIMARY KEY → `USING ROWID SEARCH ON TABLE
+//! <table> FOR IN-OPERATOR`. Any WHERE/LIMIT/DISTINCT/GROUP/join/expression, an
+//! unindexed column, or an ambiguous multi-index choice keeps the `LIST
+//! SUBQUERY` form. EQP-only: the executed results are unaffected. Verified
+//! byte-for-byte against the sqlite3 3.50.4 CLI.
+
+#![cfg(feature = "std")]
+
+use std::process::Command;
+
+fn sqlite3_available() -> bool {
+    Command::new("sqlite3").arg("--version").output().is_ok()
+}
+
+fn plan(bin: &str, base: &str, sql: &str) -> String {
+    let full = format!("{base} EXPLAIN QUERY PLAN {sql}");
+    let out = Command::new(bin)
+        .arg(":memory:")
+        .arg(&full)
+        .output()
+        .unwrap();
+    String::from_utf8_lossy(&out.stdout)
+        .lines()
+        .filter(|l| !l.trim().is_empty() && !l.starts_with("QUERY PLAN"))
+        .map(|l| l.trim_start_matches(|c: char| " |`*+_-".contains(c)))
+        .collect::<Vec<_>>()
+        .join("#")
+}
+
+// `w` is the scanned outer (no index on `p`); `t` supplies the subquery values
+// with exactly one index per candidate column (unambiguous choice).
+const BASE: &str = "CREATE TABLE t(a INTEGER PRIMARY KEY, b, c);\
+     CREATE INDEX tb ON t(b);\
+     INSERT INTO t VALUES(1,10,100),(2,20,200),(3,30,300);\
+     CREATE TABLE u(x, y); CREATE INDEX ux ON u(x); INSERT INTO u VALUES(10,1),(30,3);\
+     CREATE TABLE w(p, q); INSERT INTO w VALUES(10,1),(20,2),(99,3);";
+
+#[test]
+fn for_in_operator_plan_matches_sqlite() {
+    if !sqlite3_available() {
+        eprintln!("sqlite3 CLI not found; skipping");
+        return;
+    }
+    let g = env!("CARGO_BIN_EXE_graphitesql");
+    for q in [
+        // Secondary-index column → USING INDEX tb FOR IN-OPERATOR.
+        "SELECT * FROM w WHERE p IN (SELECT b FROM t)",
+        "SELECT * FROM w WHERE p NOT IN (SELECT b FROM t)",
+        "SELECT * FROM w WHERE p IN (SELECT b FROM t ORDER BY b)",
+        "SELECT * FROM w WHERE p IN (SELECT (b) FROM t)",
+        "SELECT * FROM w WHERE p IN (SELECT b AS bb FROM t)",
+        "SELECT * FROM w WHERE p IN (SELECT t.b FROM t)",
+        // rowid / INTEGER PRIMARY KEY → USING ROWID SEARCH ON TABLE t.
+        "SELECT * FROM w WHERE p IN (SELECT a FROM t)",
+        "SELECT * FROM w WHERE p IN (SELECT rowid FROM t)",
+        "SELECT * FROM w WHERE p NOT IN (SELECT a FROM t)",
+        // Controls that keep the LIST SUBQUERY / bloom form:
+        "SELECT * FROM w WHERE p IN (SELECT c FROM t)", // unindexed column
+        "SELECT * FROM w WHERE p IN (SELECT b FROM t WHERE c > 0)", // WHERE
+        "SELECT * FROM w WHERE p IN (SELECT b FROM t LIMIT 2)", // LIMIT
+        "SELECT * FROM w WHERE p IN (SELECT DISTINCT b FROM t)", // DISTINCT
+        "SELECT * FROM w WHERE p IN (SELECT b + 1 FROM t)", // expression
+        // Control: the IN column is itself index-seekable → outer SEARCH, RHS
+        // stays a materialized LIST SUBQUERY (no FOR IN-OPERATOR).
+        "SELECT * FROM u WHERE x IN (SELECT b FROM t)",
+        "SELECT * FROM t WHERE b IN (SELECT x FROM u)",
+    ] {
+        assert_eq!(plan("sqlite3", BASE, q), plan(g, BASE, q), "plan for {q}");
+    }
+}
+
+/// The plan node is cosmetic — the executed rows must be identical regardless.
+#[test]
+fn for_in_operator_rows_match() {
+    if !sqlite3_available() {
+        eprintln!("sqlite3 CLI not found; skipping");
+        return;
+    }
+    let g = env!("CARGO_BIN_EXE_graphitesql");
+    for q in [
+        "SELECT p FROM w WHERE p IN (SELECT b FROM t) ORDER BY p",
+        "SELECT p FROM w WHERE p NOT IN (SELECT b FROM t) ORDER BY p",
+        "SELECT p FROM w WHERE p IN (SELECT a FROM t) ORDER BY p",
+    ] {
+        let full = format!("{BASE} {q}");
+        let sq = Command::new("sqlite3")
+            .arg(":memory:")
+            .arg(&full)
+            .output()
+            .unwrap();
+        let gr = Command::new(g).arg(":memory:").arg(&full).output().unwrap();
+        assert_eq!(
+            String::from_utf8_lossy(&sq.stdout),
+            String::from_utf8_lossy(&gr.stdout),
+            "rows for {q}"
+        );
+    }
+}