You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
feat(sql): JOINs — INNER, LEFT OUTER, RIGHT OUTER, FULL OUTER (SQLR-5) (#99)
Adds the four-flavor JOIN quartet with explicit ON conditions and
table aliases. SQLite ships only INNER and LEFT OUTER; we implement
RIGHT and FULL OUTER as well because the per-flavor delta on top of
a shared nested-loop driver is just NULL-padding policy. See
docs/design-decisions.md §14a for the reasoning.
Engine changes:
- New RowScope trait abstracts column lookup; SingleTableScope
preserves the legacy fast path verbatim, JoinedScope handles
multi-table resolution with NULL padding.
- eval_expr / eval_predicate / eval_function / json_fn_* / FTS
helpers / vector-arg extractor refactored to take &dyn RowScope.
- execute_select_rows_joined: left-folded nested-loop driver,
O(N×M) per join level. ON only sees tables in scope at that
join level (forward refs error). ON reuses eval_predicate_scope
so non-zero ints are truthy, matching WHERE.
- Parser: SelectQuery gains table_alias + joins; ProjectionKind
Column carries an optional qualifier for t.col disambiguation.
USING / NATURAL / CROSS / comma joins, plus aggregates / GROUP
BY / DISTINCT over JOIN, surface as friendly NotImplemented.
17 new tests covering all four flavors, NULL padding both ways,
qualified/unqualified resolution, ambiguity, self-join rejection,
three-table chaining, chained LEFT OUTER, NULL ordering on outer
joins, ON-references-later-table errors, and truthy-int ON.
Docs: README + supported-sql.md + design-decisions.md +
architecture.md + sql-engine.md updated.
526 tests passing, 0 failures. Build, fmt, clippy clean.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|`CREATE [UNIQUE] INDEX`| Single-column, named indexes; `IF NOT EXISTS`; persists as a dedicated cell-based B-Tree. INTEGER + TEXT columns only |
159
159
|`INSERT INTO`| Explicit column list required; auto-ROWID for `INTEGER PRIMARY KEY`; multi-row `VALUES (…), (…)`; UNIQUE enforcement; clean type errors (no panics); NULL padding for omitted columns |
160
-
|`SELECT`|`*` or column list with optional `AS alias`; `WHERE`; `DISTINCT`; `GROUP BY col[, col …]`; aggregate projections `COUNT(*)` / `COUNT([DISTINCT] col)` / `SUM` / `AVG` / `MIN` / `MAX`; single-column `ORDER BY [ASC\|DESC]` (also resolves alias and aggregate display names); `LIMIT n`. `WHERE col = literal` probes an index when one exists |
160
+
|`SELECT`|`*` or column list with optional `AS alias`; `WHERE`; `DISTINCT`; `GROUP BY col[, col …]`; aggregate projections `COUNT(*)` / `COUNT([DISTINCT] col)` / `SUM` / `AVG` / `MIN` / `MAX`; `[INNER\|LEFT OUTER\|RIGHT OUTER\|FULL OUTER] JOIN ... ON ...` with table aliases and qualified `t.col` references; single-column `ORDER BY [ASC\|DESC]` (also resolves alias and aggregate display names); `LIMIT n`. `WHERE col = literal` probes an index when one exists |
161
161
|`UPDATE`| Multi-column `SET`; `WHERE`; UNIQUE + type enforcement; arithmetic in assignments (`SET age = age + 1`) |
162
162
|`DELETE`|`WHERE` predicate or full-table delete |
163
163
|`BEGIN` / `COMMIT` / `ROLLBACK`| Real transactions, snapshot-based; WAL-backed commit; single-level (no savepoints); auto-rollback if `COMMIT`'s disk write fails |
@@ -173,7 +173,7 @@ Expressions in `WHERE` and `UPDATE`'s `SET` RHS:
173
173
- String concat — `||`
174
174
- Literals — integer + real numbers, `'single-quoted strings'`, `TRUE` / `FALSE`, `NULL`; parentheses for grouping
175
175
176
-
**Not yet supported** (common ones): joins, subqueries, CTEs, `HAVING`, `LIKE … ESCAPE '<char>'`, `IN (subquery)`, `DISTINCT` on `SUM`/`AVG`/`MIN`/`MAX`, GROUP BY on expressions, expressions in the projection list, `OFFSET`, multi-column `ORDER BY`, savepoints, `ALTER TABLE`, `DROP TABLE`, `DROP INDEX`. The [full list with context](docs/supported-sql.md#not-yet-supported) lives in the reference.
176
+
**Not yet supported** (common ones): subqueries, CTEs, `HAVING`, `LIKE … ESCAPE '<char>'`, `IN (subquery)`, `DISTINCT` on `SUM`/`AVG`/`MIN`/`MAX`, GROUP BY on expressions, expressions in the projection list, `OFFSET`, multi-column `ORDER BY`, savepoints, `JOIN ... USING`, `NATURAL JOIN`, `CROSS JOIN`, comma joins, aggregates / DISTINCT / GROUP BY *over* JOIN results. The [full list with context](docs/supported-sql.md#not-yet-supported) lives in the reference.
177
177
178
178
#### Meta commands
179
179
@@ -229,7 +229,8 @@ The project is staged in phases, each independently shippable. A finished phase
229
229
-[x] Parsing via `sqlparser` (SQLite dialect); typed `SQLRiteError` via `thiserror`
230
230
-[x]`CREATE TABLE` with `PRIMARY KEY`, `UNIQUE`, `NOT NULL`; duplicate-column detection; in-memory `BTreeMap` indexes on PK/UNIQUE columns
231
231
-[x]`INSERT` with auto-ROWID for `INTEGER PRIMARY KEY`, UNIQUE enforcement, NULL padding for missing columns
Copy file name to clipboardExpand all lines: docs/architecture.md
+2-3Lines changed: 2 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -136,9 +136,8 @@ Steps 1–7 are purely in-memory; step 8 is the only disk contact, and after the
136
136
137
137
The roadmap has shipped far enough that the original "deliberately missing" list mostly turned into shipped features. What's still left:
138
138
139
-
-**No query optimizer** beyond the bounded-heap top-k pass for KNN (Phase 7c) and the HNSW probe shortcut (7d.2). Equality-on-PK probes are direct; everything else is a table scan.
140
-
-**No joins.**`INNER` / `LEFT OUTER` / `CROSS` are parsed but rejected by the executor. On the "possible extras" list in [roadmap.md](roadmap.md).
141
-
-**No aggregates.**`COUNT(*)` / `SUM` / `AVG` / `GROUP BY` aren't implemented yet — the parser accepts them but the executor errors. Phase 8 candidate alongside FTS.
139
+
-**No query optimizer** beyond the bounded-heap top-k pass for KNN (Phase 7c) and the HNSW probe shortcut (7d.2). Equality-on-PK probes are direct; everything else is a table scan. Joins use plain nested-loop (O(N×M) per join level); hash / merge joins on equi-join shapes are a future increment.
140
+
-**Aggregates / GROUP BY / DISTINCT over joined results.** The single-table aggregator is wired against one rowid stream; the multi-table join executor produces joined rows but doesn't yet feed them through the aggregator. Surfaces as a clean `NotImplemented` at parse time. The single-table aggregation path (SQLR-3) is fully shipped.
142
141
-**No network layer.** SQLRite is embedded-only. The closest thing is the [`sqlrite-mcp`](mcp.md) server, which is stdio (not network). A real wire protocol isn't on the roadmap.
143
142
-**No streaming row cursor.**`Rows` is currently backed by an eager `Vec` (Phase 5a). The `Rows::next` API is shaped to support a real cursor — the swap is deferred to **5a.2**.
Copy file name to clipboardExpand all lines: docs/design-decisions.md
+19Lines changed: 19 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -156,6 +156,25 @@ Decisions are grouped by the engine layer they concern: parser, storage, concurr
156
156
157
157
---
158
158
159
+
### 14a. Implement RIGHT OUTER and FULL OUTER joins (SQLR-5)
160
+
161
+
**Decision.** SQLRite supports the full quartet — `INNER`, `LEFT OUTER`, `RIGHT OUTER`, `FULL OUTER` — via [`execute_select_rows_joined`](../src/sql/executor.rs). SQLite ships only `INNER` and `LEFT OUTER`; SQLite users typically rewrite a `RIGHT JOIN` as a `LEFT JOIN` with the operands swapped, and a `FULL JOIN` as a `UNION` of `LEFT` and a back-anti-`LEFT`.
162
+
163
+
**Why.** Once the executor has a multi-table scope (the `RowScope` trait), the per-flavor difference is just NULL-padding policy on top of one shared nested-loop driver:
164
+
165
+
-`INNER`: drop unmatched on both sides
166
+
-`LEFT OUTER`: keep unmatched left, NULL the right
167
+
-`RIGHT OUTER`: keep unmatched right, NULL the left
168
+
-`FULL OUTER`: do both
169
+
170
+
Adding the missing two flavors costs ~30 lines in the join driver and a `right_matched: Vec<bool>` to track unmatched right rows across the accumulator. That's much cheaper than the rewrite-by-hand experience SQLite users get, and removes the pedagogical "why doesn't this work" stumble — the whole project's premise is "implement these things to learn how they work", and `RIGHT` / `FULL` are interesting precisely because most engines support them and SQLite's choice not to is itself a design conversation.
171
+
172
+
**Cost.** Slightly more code in the join driver and the doc burden of explaining we diverge from SQLite. The single-table fast path is unchanged, so SQLite users hitting this engine without joins see no behavioral difference. The implementation is plain nested-loop (O(N×M) per join level) with no hash / merge optimization — same complexity as the LEFT path; both flavors will benefit equally when that work lands later.
173
+
174
+
**Cross-reference.** Implementation in [`src/sql/executor.rs`](../src/sql/executor.rs) (`execute_select_rows_joined`); per-flavor table in [`docs/supported-sql.md`](supported-sql.md#join-semantics-sqlr-5).
175
+
176
+
---
177
+
159
178
### 14. Deterministic page-number ordering when saving
160
179
161
180
**Decision.**[`save_database`](../src/sql/pager/mod.rs) sorts table names alphabetically before writing. Same DB contents → same bytes at same page numbers, every time.
`UPDATE` and `DELETE` don't have a dedicated internal struct — the executor pattern-matches the sqlparser types directly because there's less transformation needed.
51
51
52
-
`SelectQuery::projection` is now `Projection::All | Projection::Items(Vec<ProjectionItem>)`, where each item carries a `ProjectionKind::Column(name)` or `ProjectionKind::Aggregate(AggregateCall)` plus an optional `AS alias`. `AggregateCall` covers `COUNT(*)`, `COUNT([DISTINCT] col)`, `SUM` / `AVG` / `MIN` / `MAX` of a bare column. `group_by` is a `Vec<String>` of bare column names (empty = no GROUP BY); the parser validates that every non-aggregate projection item appears in `GROUP BY`.
52
+
`SelectQuery::projection` is now `Projection::All | Projection::Items(Vec<ProjectionItem>)`, where each item carries a `ProjectionKind::Column { qualifier, name }` (qualifier is `Some` for `t.col` shapes, used by JOIN execution to disambiguate) or `ProjectionKind::Aggregate(AggregateCall)` plus an optional `AS alias`. `AggregateCall` covers `COUNT(*)`, `COUNT([DISTINCT] col)`, `SUM` / `AVG` / `MIN` / `MAX` of a bare column. `group_by` is a `Vec<String>` of bare column names (empty = no GROUP BY); the parser validates that every non-aggregate projection item appears in `GROUP BY`.
53
53
54
-
Each parser module still rejects features we don't implement with `SQLRiteError::NotImplemented` — `JOIN`, `HAVING`, `DISTINCT ON (...)`, `GROUP BY` on expressions, `LIKE … ESCAPE '<char>'`, `IN (subquery)`, `OFFSET`, multi-table DELETE, tuple assignment targets, etc. These errors carry the feature name in the message so the user knows what isn't there.
54
+
`SelectQuery::joins` (SQLR-5) is a `Vec<JoinClause>` evaluated left-to-right by `execute_select_rows_joined`. Each clause carries a `JoinType` (`Inner` / `LeftOuter` / `RightOuter` / `FullOuter`), the right-table name + optional alias, and a required `ON` expression. Empty = single-table SELECT, the existing fast path with HNSW / FTS / bounded-heap optimizations.
55
+
56
+
Each parser module still rejects features we don't implement with `SQLRiteError::NotImplemented` — `JOIN ... USING`, `NATURAL JOIN`, `CROSS JOIN`, comma joins, aggregates / GROUP BY / DISTINCT over JOINs, `HAVING`, `DISTINCT ON (...)`, `GROUP BY` on expressions, `LIKE … ESCAPE '<char>'`, `IN (subquery)`, `OFFSET`, multi-table DELETE, tuple assignment targets, etc. These errors carry the feature name in the message so the user knows what isn't there.
-**`ORDER BY`**: single sort key, `ASC` (default) or `DESC`. For non-aggregating queries the key is any expression — including function calls — so KNN queries like `ORDER BY vec_distance_l2(embedding, [...]) LIMIT k` work end-to-end *(Phase 7b)*. For aggregating queries the key resolves against the *output* row by name: a bare identifier matches an alias or a `GROUP BY` column, and a function call like `COUNT(*)` matches an aggregate projection by its canonical display form. Sort key types must match across rows.
177
178
-**`LIMIT`**: non-negative integer literal. `LIMIT 0` is valid (returns zero rows). When `DISTINCT` is in play, `LIMIT` is applied after deduplication so it counts unique rows.
178
179
180
+
### `JOIN` semantics (SQLR-5)
181
+
182
+
Four flavors are supported, all with explicit `ON` conditions:
183
+
184
+
| Flavor | Keeps unmatched rows from… |
185
+
|---|---|
186
+
|`INNER JOIN`| …neither side. Only ON-matched pairs survive. |
187
+
|`LEFT [OUTER] JOIN`| …the left side; right-side columns become `NULL` for unmatched left rows. |
188
+
|`RIGHT [OUTER] JOIN`| …the right side; left-side columns become `NULL` for unmatched right rows. |
189
+
|`FULL [OUTER] JOIN`| …both sides, NULL-padded on the unmatched side. |
190
+
191
+
-**Engine choice:** SQLite ships only `INNER` and `LEFT OUTER`. SQLRite implements all four because the per-flavor differences boil down to NULL-padding policy on top of one shared nested-loop driver — adding `RIGHT` / `FULL` was effectively free once the executor had a multi-table scope. See [`docs/design-decisions.md`](design-decisions.md) for the rationale.
192
+
-**Aliases:**`FROM customers AS c INNER JOIN orders AS o ON c.id = o.customer_id`. When an alias is supplied the original table name leaves scope (SQL standard) — qualifier resolution uses the alias.
193
+
-**Qualified column references:**`<table>.<col>` and `<alias>.<col>` resolve to that specific side. Bare `<col>` references must resolve to exactly one in-scope table; ambiguous references error with a "qualify it as `<table>.col`" hint.
194
+
-**Output of `SELECT *`** over a join is every column of every in-scope table, in source order. Duplicate header names are permitted (SQLite-style). Disambiguate with explicit `SELECT t.col AS t_col, u.col AS u_col`.
195
+
-**Multi-join** chains left-fold: `A JOIN B ON ... JOIN C ON ...` evaluates as `(A ⨝ B) ⨝ C`. Each new clause sees every prior alias / table in its `ON` expression.
196
+
-**Self-joins** require an alias on at least one side: `FROM nodes AS p INNER JOIN nodes AS c ON p.id = c.parent_id`. Without one, you get a `duplicate table reference` error so qualifiers stay unambiguous.
197
+
-**`WHERE` runs after joins.** A `WHERE right.col IS NULL` filter on a `LEFT JOIN` correctly returns left rows with no match (the standard "anti-join via outer-join" idiom).
198
+
-**`ORDER BY` and `LIMIT`** apply to the fully joined row stream.
199
+
-**Algorithm:** plain nested-loop join, O(N×M) per join level. Adequate for an embedded learning database; hash / merge joins on equi-join shapes are a future optimization.
200
+
201
+
#### What's not supported in JOINs
202
+
203
+
-`JOIN ... USING (col)` and `NATURAL JOIN` — explicit `ON` only. (Both are deferred — `USING` is straightforward but adds a column-resolution rule we haven't needed yet.)
204
+
-`CROSS JOIN` (write `INNER JOIN ... ON true` instead) and comma-separated FROM lists.
205
+
- Aggregates / `GROUP BY` / `DISTINCT`*over* a join. The single-table aggregator is wired against one rowid stream; rewiring it for joined rows is a separate increment. Surfaces as a clean `NotImplemented` at parse time.
206
+
-`fts_match` / `bm25_score` inside a JOIN expression. They need to look up an FTS index by column, which is single-table-bound today. Use them on a single-table SELECT first, or fold the FTS lookup into the FROM side.
207
+
179
208
### Index probing
180
209
181
210
The executor includes a tiny optimizer: if the `WHERE` is exactly `<indexed_col> = <literal>` or `<literal> = <indexed_col>`, it probes the index and scans only matching rows. Mixed predicates (`WHERE a = 1 AND b > 2`), range predicates (`WHERE a > 1`), and OR-combined predicates fall back to a full table scan. Aggregating queries (`GROUP BY` / aggregate functions) skip the rowid-shape optimizations (HNSW / FTS / bounded-heap top-k) since every matching row contributes to its group.
@@ -195,7 +224,8 @@ The executor includes a tiny optimizer: if the `WHERE` is exactly `<indexed_col>
195
224
196
225
### What doesn't work
197
226
198
-
-**Joins** of any kind (`INNER`, `LEFT OUTER`, `CROSS`, comma-join)
227
+
-**`CROSS JOIN`**, **comma-separated FROM lists**, **`NATURAL JOIN`**, **`JOIN ... USING (col)`** — explicit `INNER` / `LEFT` / `RIGHT` / `FULL OUTER JOIN ... ON ...` only (see [JOIN semantics](#join-semantics-sqlr-5))
228
+
-**Aggregates** / **`GROUP BY`** / **`DISTINCT`** over a JOIN — pipe through a subquery once subqueries land
199
229
-**Subqueries**, CTEs (`WITH`), views
200
230
-**`HAVING`** — pre-aggregation `WHERE` works; post-aggregation filtering does not yet
201
231
-**`DISTINCT`** on `SUM` / `AVG` / `MIN` / `MAX` (only `COUNT(DISTINCT col)` is supported)
@@ -521,7 +551,8 @@ A REPL launched with `sqlrite --readonly foo.sqlrite` (or `sqlrite::open_databas
521
551
For context when you hit `NotImplemented`. See [Roadmap](roadmap.md) for when these land:
0 commit comments