Skip to content

Commit 998329e

Browse files
joaoh82claude
andauthored
feat(sql): JOINs — INNER, LEFT OUTER, RIGHT OUTER, FULL OUTER (SQLR-5) (#99)
Adds the four-flavor JOIN quartet with explicit ON conditions and table aliases. SQLite ships only INNER and LEFT OUTER; we implement RIGHT and FULL OUTER as well because the per-flavor delta on top of a shared nested-loop driver is just NULL-padding policy. See docs/design-decisions.md §14a for the reasoning. Engine changes: - New RowScope trait abstracts column lookup; SingleTableScope preserves the legacy fast path verbatim, JoinedScope handles multi-table resolution with NULL padding. - eval_expr / eval_predicate / eval_function / json_fn_* / FTS helpers / vector-arg extractor refactored to take &dyn RowScope. - execute_select_rows_joined: left-folded nested-loop driver, O(N×M) per join level. ON only sees tables in scope at that join level (forward refs error). ON reuses eval_predicate_scope so non-zero ints are truthy, matching WHERE. - Parser: SelectQuery gains table_alias + joins; ProjectionKind Column carries an optional qualifier for t.col disambiguation. USING / NATURAL / CROSS / comma joins, plus aggregates / GROUP BY / DISTINCT over JOIN, surface as friendly NotImplemented. 17 new tests covering all four flavors, NULL padding both ways, qualified/unqualified resolution, ambiguity, self-join rejection, three-table chaining, chained LEFT OUTER, NULL ordering on outer joins, ON-references-later-table errors, and truthy-int ON. Docs: README + supported-sql.md + design-decisions.md + architecture.md + sql-engine.md updated. 526 tests passing, 0 failures. Build, fmt, clippy clean. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 9a80bea commit 998329e

7 files changed

Lines changed: 1270 additions & 122 deletions

File tree

README.md

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -157,7 +157,7 @@ sqlrite> DELETE FROM users WHERE age < 30;
157157
| `CREATE TABLE` | `PRIMARY KEY`, `UNIQUE`, `NOT NULL`; duplicate-column detection; types `INTEGER`/`INT`/`BIGINT`/`SMALLINT`, `TEXT`/`VARCHAR`, `REAL`/`FLOAT`/`DOUBLE`/`DECIMAL`, `BOOLEAN`. Auto-creates `sqlrite_autoindex_<table>_<col>` for every PK + UNIQUE column |
158158
| `CREATE [UNIQUE] INDEX` | Single-column, named indexes; `IF NOT EXISTS`; persists as a dedicated cell-based B-Tree. INTEGER + TEXT columns only |
159159
| `INSERT INTO` | Explicit column list required; auto-ROWID for `INTEGER PRIMARY KEY`; multi-row `VALUES (…), (…)`; UNIQUE enforcement; clean type errors (no panics); NULL padding for omitted columns |
160-
| `SELECT` | `*` or column list with optional `AS alias`; `WHERE`; `DISTINCT`; `GROUP BY col[, col …]`; aggregate projections `COUNT(*)` / `COUNT([DISTINCT] col)` / `SUM` / `AVG` / `MIN` / `MAX`; single-column `ORDER BY [ASC\|DESC]` (also resolves alias and aggregate display names); `LIMIT n`. `WHERE col = literal` probes an index when one exists |
160+
| `SELECT` | `*` or column list with optional `AS alias`; `WHERE`; `DISTINCT`; `GROUP BY col[, col …]`; aggregate projections `COUNT(*)` / `COUNT([DISTINCT] col)` / `SUM` / `AVG` / `MIN` / `MAX`; `[INNER\|LEFT OUTER\|RIGHT OUTER\|FULL OUTER] JOIN ... ON ...` with table aliases and qualified `t.col` references; single-column `ORDER BY [ASC\|DESC]` (also resolves alias and aggregate display names); `LIMIT n`. `WHERE col = literal` probes an index when one exists |
161161
| `UPDATE` | Multi-column `SET`; `WHERE`; UNIQUE + type enforcement; arithmetic in assignments (`SET age = age + 1`) |
162162
| `DELETE` | `WHERE` predicate or full-table delete |
163163
| `BEGIN` / `COMMIT` / `ROLLBACK` | Real transactions, snapshot-based; WAL-backed commit; single-level (no savepoints); auto-rollback if `COMMIT`'s disk write fails |
@@ -173,7 +173,7 @@ Expressions in `WHERE` and `UPDATE`'s `SET` RHS:
173173
- String concat — `||`
174174
- Literals — integer + real numbers, `'single-quoted strings'`, `TRUE` / `FALSE`, `NULL`; parentheses for grouping
175175

176-
**Not yet supported** (common ones): joins, subqueries, CTEs, `HAVING`, `LIKE … ESCAPE '<char>'`, `IN (subquery)`, `DISTINCT` on `SUM`/`AVG`/`MIN`/`MAX`, GROUP BY on expressions, expressions in the projection list, `OFFSET`, multi-column `ORDER BY`, savepoints, `ALTER TABLE`, `DROP TABLE`, `DROP INDEX`. The [full list with context](docs/supported-sql.md#not-yet-supported) lives in the reference.
176+
**Not yet supported** (common ones): subqueries, CTEs, `HAVING`, `LIKE … ESCAPE '<char>'`, `IN (subquery)`, `DISTINCT` on `SUM`/`AVG`/`MIN`/`MAX`, GROUP BY on expressions, expressions in the projection list, `OFFSET`, multi-column `ORDER BY`, savepoints, `JOIN ... USING`, `NATURAL JOIN`, `CROSS JOIN`, comma joins, aggregates / DISTINCT / GROUP BY *over* JOIN results. The [full list with context](docs/supported-sql.md#not-yet-supported) lives in the reference.
177177

178178
#### Meta commands
179179

@@ -229,7 +229,8 @@ The project is staged in phases, each independently shippable. A finished phase
229229
- [x] Parsing via `sqlparser` (SQLite dialect); typed `SQLRiteError` via `thiserror`
230230
- [x] `CREATE TABLE` with `PRIMARY KEY`, `UNIQUE`, `NOT NULL`; duplicate-column detection; in-memory `BTreeMap` indexes on PK/UNIQUE columns
231231
- [x] `INSERT` with auto-ROWID for `INTEGER PRIMARY KEY`, UNIQUE enforcement, NULL padding for missing columns
232-
- [x] `SELECT` — projection, `WHERE`, `ORDER BY`, `LIMIT` (single-table, no joins yet)
232+
- [x] `SELECT` — projection, `WHERE`, `ORDER BY`, `LIMIT`
233+
- [x] `JOIN``INNER`, `LEFT OUTER`, `RIGHT OUTER`, `FULL OUTER` with `ON` (SQLR-5)
233234
- [x] `UPDATE ... SET ... WHERE ...` with type + UNIQUE enforcement at write time
234235
- [x] `DELETE ... WHERE ...`
235236
- [x] Expression evaluator: `=`/`<>`/`<`/`<=`/`>`/`>=`, `AND`/`OR`/`NOT`, arithmetic `+`/`-`/`*`/`/`/`%`, string concat `||`, NULL-as-false in `WHERE`

docs/architecture.md

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -136,9 +136,8 @@ Steps 1–7 are purely in-memory; step 8 is the only disk contact, and after the
136136

137137
The roadmap has shipped far enough that the original "deliberately missing" list mostly turned into shipped features. What's still left:
138138

139-
- **No query optimizer** beyond the bounded-heap top-k pass for KNN (Phase 7c) and the HNSW probe shortcut (7d.2). Equality-on-PK probes are direct; everything else is a table scan.
140-
- **No joins.** `INNER` / `LEFT OUTER` / `CROSS` are parsed but rejected by the executor. On the "possible extras" list in [roadmap.md](roadmap.md).
141-
- **No aggregates.** `COUNT(*)` / `SUM` / `AVG` / `GROUP BY` aren't implemented yet — the parser accepts them but the executor errors. Phase 8 candidate alongside FTS.
139+
- **No query optimizer** beyond the bounded-heap top-k pass for KNN (Phase 7c) and the HNSW probe shortcut (7d.2). Equality-on-PK probes are direct; everything else is a table scan. Joins use plain nested-loop (O(N×M) per join level); hash / merge joins on equi-join shapes are a future increment.
140+
- **Aggregates / GROUP BY / DISTINCT over joined results.** The single-table aggregator is wired against one rowid stream; the multi-table join executor produces joined rows but doesn't yet feed them through the aggregator. Surfaces as a clean `NotImplemented` at parse time. The single-table aggregation path (SQLR-3) is fully shipped.
142141
- **No network layer.** SQLRite is embedded-only. The closest thing is the [`sqlrite-mcp`](mcp.md) server, which is stdio (not network). A real wire protocol isn't on the roadmap.
143142
- **No streaming row cursor.** `Rows` is currently backed by an eager `Vec` (Phase 5a). The `Rows::next` API is shaped to support a real cursor — the swap is deferred to **5a.2**.
144143

docs/design-decisions.md

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -156,6 +156,25 @@ Decisions are grouped by the engine layer they concern: parser, storage, concurr
156156

157157
---
158158

159+
### 14a. Implement RIGHT OUTER and FULL OUTER joins (SQLR-5)
160+
161+
**Decision.** SQLRite supports the full quartet — `INNER`, `LEFT OUTER`, `RIGHT OUTER`, `FULL OUTER` — via [`execute_select_rows_joined`](../src/sql/executor.rs). SQLite ships only `INNER` and `LEFT OUTER`; SQLite users typically rewrite a `RIGHT JOIN` as a `LEFT JOIN` with the operands swapped, and a `FULL JOIN` as a `UNION` of `LEFT` and a back-anti-`LEFT`.
162+
163+
**Why.** Once the executor has a multi-table scope (the `RowScope` trait), the per-flavor difference is just NULL-padding policy on top of one shared nested-loop driver:
164+
165+
- `INNER`: drop unmatched on both sides
166+
- `LEFT OUTER`: keep unmatched left, NULL the right
167+
- `RIGHT OUTER`: keep unmatched right, NULL the left
168+
- `FULL OUTER`: do both
169+
170+
Adding the missing two flavors costs ~30 lines in the join driver and a `right_matched: Vec<bool>` to track unmatched right rows across the accumulator. That's much cheaper than the rewrite-by-hand experience SQLite users get, and removes the pedagogical "why doesn't this work" stumble — the whole project's premise is "implement these things to learn how they work", and `RIGHT` / `FULL` are interesting precisely because most engines support them and SQLite's choice not to is itself a design conversation.
171+
172+
**Cost.** Slightly more code in the join driver and the doc burden of explaining we diverge from SQLite. The single-table fast path is unchanged, so SQLite users hitting this engine without joins see no behavioral difference. The implementation is plain nested-loop (O(N×M) per join level) with no hash / merge optimization — same complexity as the LEFT path; both flavors will benefit equally when that work lands later.
173+
174+
**Cross-reference.** Implementation in [`src/sql/executor.rs`](../src/sql/executor.rs) (`execute_select_rows_joined`); per-flavor table in [`docs/supported-sql.md`](supported-sql.md#join-semantics-sqlr-5).
175+
176+
---
177+
159178
### 14. Deterministic page-number ordering when saving
160179

161180
**Decision.** [`save_database`](../src/sql/pager/mod.rs) sorts table names alphabetically before writing. Same DB contents → same bytes at same page numbers, every time.

docs/sql-engine.md

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -45,13 +45,15 @@ The `sqlparser` AST is designed to cover every SQL dialect, so its types are hug
4545
|---|---|---|
4646
| `Statement::CreateTable(CreateTable)` | `CreateQuery { table_name, columns: Vec<ParsedColumn> }` | [`create.rs`](../src/sql/parser/create.rs) |
4747
| `Statement::Insert(Insert)` | `InsertQuery { table_name, columns, rows }` | [`insert.rs`](../src/sql/parser/insert.rs) |
48-
| `Statement::Query(_)` | `SelectQuery { table_name, projection, selection, order_by, limit, distinct, group_by }` | [`select.rs`](../src/sql/parser/select.rs) |
48+
| `Statement::Query(_)` | `SelectQuery { table_name, table_alias, joins, projection, selection, order_by, limit, distinct, group_by }` | [`select.rs`](../src/sql/parser/select.rs) |
4949

5050
`UPDATE` and `DELETE` don't have a dedicated internal struct — the executor pattern-matches the sqlparser types directly because there's less transformation needed.
5151

52-
`SelectQuery::projection` is now `Projection::All | Projection::Items(Vec<ProjectionItem>)`, where each item carries a `ProjectionKind::Column(name)` or `ProjectionKind::Aggregate(AggregateCall)` plus an optional `AS alias`. `AggregateCall` covers `COUNT(*)`, `COUNT([DISTINCT] col)`, `SUM` / `AVG` / `MIN` / `MAX` of a bare column. `group_by` is a `Vec<String>` of bare column names (empty = no GROUP BY); the parser validates that every non-aggregate projection item appears in `GROUP BY`.
52+
`SelectQuery::projection` is now `Projection::All | Projection::Items(Vec<ProjectionItem>)`, where each item carries a `ProjectionKind::Column { qualifier, name }` (qualifier is `Some` for `t.col` shapes, used by JOIN execution to disambiguate) or `ProjectionKind::Aggregate(AggregateCall)` plus an optional `AS alias`. `AggregateCall` covers `COUNT(*)`, `COUNT([DISTINCT] col)`, `SUM` / `AVG` / `MIN` / `MAX` of a bare column. `group_by` is a `Vec<String>` of bare column names (empty = no GROUP BY); the parser validates that every non-aggregate projection item appears in `GROUP BY`.
5353

54-
Each parser module still rejects features we don't implement with `SQLRiteError::NotImplemented``JOIN`, `HAVING`, `DISTINCT ON (...)`, `GROUP BY` on expressions, `LIKE … ESCAPE '<char>'`, `IN (subquery)`, `OFFSET`, multi-table DELETE, tuple assignment targets, etc. These errors carry the feature name in the message so the user knows what isn't there.
54+
`SelectQuery::joins` (SQLR-5) is a `Vec<JoinClause>` evaluated left-to-right by `execute_select_rows_joined`. Each clause carries a `JoinType` (`Inner` / `LeftOuter` / `RightOuter` / `FullOuter`), the right-table name + optional alias, and a required `ON` expression. Empty = single-table SELECT, the existing fast path with HNSW / FTS / bounded-heap optimizations.
55+
56+
Each parser module still rejects features we don't implement with `SQLRiteError::NotImplemented``JOIN ... USING`, `NATURAL JOIN`, `CROSS JOIN`, comma joins, aggregates / GROUP BY / DISTINCT over JOINs, `HAVING`, `DISTINCT ON (...)`, `GROUP BY` on expressions, `LIKE … ESCAPE '<char>'`, `IN (subquery)`, `OFFSET`, multi-table DELETE, tuple assignment targets, etc. These errors carry the feature name in the message so the user knows what isn't there.
5557

5658
## Statement dispatch
5759

docs/supported-sql.md

Lines changed: 34 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -149,7 +149,8 @@ Hex literals, blob literals, and date/time functions are not supported.
149149

150150
```sql
151151
SELECT [DISTINCT] {* | <projection_item>[, <projection_item>, ...]}
152-
FROM <table>
152+
FROM <table> [AS <alias>]
153+
[{INNER | LEFT [OUTER] | RIGHT [OUTER] | FULL [OUTER]} JOIN <table> [AS <alias>] ON <expr>]*
153154
[WHERE <expr>]
154155
[GROUP BY <col>[, <col>, ...]]
155156
[ORDER BY <expr> [ASC|DESC]]
@@ -176,6 +177,34 @@ COUNT([DISTINCT] <column>) -- counts non-NULL values, option
176177
- **`ORDER BY`**: single sort key, `ASC` (default) or `DESC`. For non-aggregating queries the key is any expression — including function calls — so KNN queries like `ORDER BY vec_distance_l2(embedding, [...]) LIMIT k` work end-to-end *(Phase 7b)*. For aggregating queries the key resolves against the *output* row by name: a bare identifier matches an alias or a `GROUP BY` column, and a function call like `COUNT(*)` matches an aggregate projection by its canonical display form. Sort key types must match across rows.
177178
- **`LIMIT`**: non-negative integer literal. `LIMIT 0` is valid (returns zero rows). When `DISTINCT` is in play, `LIMIT` is applied after deduplication so it counts unique rows.
178179

180+
### `JOIN` semantics (SQLR-5)
181+
182+
Four flavors are supported, all with explicit `ON` conditions:
183+
184+
| Flavor | Keeps unmatched rows from… |
185+
|---|---|
186+
| `INNER JOIN` | …neither side. Only ON-matched pairs survive. |
187+
| `LEFT [OUTER] JOIN` | …the left side; right-side columns become `NULL` for unmatched left rows. |
188+
| `RIGHT [OUTER] JOIN` | …the right side; left-side columns become `NULL` for unmatched right rows. |
189+
| `FULL [OUTER] JOIN` | …both sides, NULL-padded on the unmatched side. |
190+
191+
- **Engine choice:** SQLite ships only `INNER` and `LEFT OUTER`. SQLRite implements all four because the per-flavor differences boil down to NULL-padding policy on top of one shared nested-loop driver — adding `RIGHT` / `FULL` was effectively free once the executor had a multi-table scope. See [`docs/design-decisions.md`](design-decisions.md) for the rationale.
192+
- **Aliases:** `FROM customers AS c INNER JOIN orders AS o ON c.id = o.customer_id`. When an alias is supplied the original table name leaves scope (SQL standard) — qualifier resolution uses the alias.
193+
- **Qualified column references:** `<table>.<col>` and `<alias>.<col>` resolve to that specific side. Bare `<col>` references must resolve to exactly one in-scope table; ambiguous references error with a "qualify it as `<table>.col`" hint.
194+
- **Output of `SELECT *`** over a join is every column of every in-scope table, in source order. Duplicate header names are permitted (SQLite-style). Disambiguate with explicit `SELECT t.col AS t_col, u.col AS u_col`.
195+
- **Multi-join** chains left-fold: `A JOIN B ON ... JOIN C ON ...` evaluates as `(A ⨝ B) ⨝ C`. Each new clause sees every prior alias / table in its `ON` expression.
196+
- **Self-joins** require an alias on at least one side: `FROM nodes AS p INNER JOIN nodes AS c ON p.id = c.parent_id`. Without one, you get a `duplicate table reference` error so qualifiers stay unambiguous.
197+
- **`WHERE` runs after joins.** A `WHERE right.col IS NULL` filter on a `LEFT JOIN` correctly returns left rows with no match (the standard "anti-join via outer-join" idiom).
198+
- **`ORDER BY` and `LIMIT`** apply to the fully joined row stream.
199+
- **Algorithm:** plain nested-loop join, O(N×M) per join level. Adequate for an embedded learning database; hash / merge joins on equi-join shapes are a future optimization.
200+
201+
#### What's not supported in JOINs
202+
203+
- `JOIN ... USING (col)` and `NATURAL JOIN` — explicit `ON` only. (Both are deferred — `USING` is straightforward but adds a column-resolution rule we haven't needed yet.)
204+
- `CROSS JOIN` (write `INNER JOIN ... ON true` instead) and comma-separated FROM lists.
205+
- Aggregates / `GROUP BY` / `DISTINCT` *over* a join. The single-table aggregator is wired against one rowid stream; rewiring it for joined rows is a separate increment. Surfaces as a clean `NotImplemented` at parse time.
206+
- `fts_match` / `bm25_score` inside a JOIN expression. They need to look up an FTS index by column, which is single-table-bound today. Use them on a single-table SELECT first, or fold the FTS lookup into the FROM side.
207+
179208
### Index probing
180209

181210
The executor includes a tiny optimizer: if the `WHERE` is exactly `<indexed_col> = <literal>` or `<literal> = <indexed_col>`, it probes the index and scans only matching rows. Mixed predicates (`WHERE a = 1 AND b > 2`), range predicates (`WHERE a > 1`), and OR-combined predicates fall back to a full table scan. Aggregating queries (`GROUP BY` / aggregate functions) skip the rowid-shape optimizations (HNSW / FTS / bounded-heap top-k) since every matching row contributes to its group.
@@ -195,7 +224,8 @@ The executor includes a tiny optimizer: if the `WHERE` is exactly `<indexed_col>
195224

196225
### What doesn't work
197226

198-
- **Joins** of any kind (`INNER`, `LEFT OUTER`, `CROSS`, comma-join)
227+
- **`CROSS JOIN`**, **comma-separated FROM lists**, **`NATURAL JOIN`**, **`JOIN ... USING (col)`** — explicit `INNER` / `LEFT` / `RIGHT` / `FULL OUTER JOIN ... ON ...` only (see [JOIN semantics](#join-semantics-sqlr-5))
228+
- **Aggregates** / **`GROUP BY`** / **`DISTINCT`** over a JOIN — pipe through a subquery once subqueries land
199229
- **Subqueries**, CTEs (`WITH`), views
200230
- **`HAVING`** — pre-aggregation `WHERE` works; post-aggregation filtering does not yet
201231
- **`DISTINCT`** on `SUM` / `AVG` / `MIN` / `MAX` (only `COUNT(DISTINCT col)` is supported)
@@ -521,7 +551,8 @@ A REPL launched with `sqlrite --readonly foo.sqlrite` (or `sqlrite::open_databas
521551
For context when you hit `NotImplemented`. See [Roadmap](roadmap.md) for when these land:
522552

523553
### Joins & composition
524-
- `INNER` / `LEFT OUTER` / `RIGHT OUTER` / `CROSS JOIN`, comma joins
554+
- `INNER` / `LEFT OUTER` / `RIGHT OUTER` / `FULL OUTER JOIN ... ON ...`**supported** (SQLR-5)
555+
- `CROSS JOIN`, comma joins, `NATURAL JOIN`, `JOIN ... USING` — not yet
525556
- Subqueries (scalar, `IN (SELECT ...)`, correlated)
526557
- CTEs (`WITH`), recursive CTEs
527558
- Views (`CREATE VIEW`)

0 commit comments

Comments
 (0)