Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -157,7 +157,7 @@ sqlrite> DELETE FROM users WHERE age < 30;
| `CREATE TABLE` | `PRIMARY KEY`, `UNIQUE`, `NOT NULL`; duplicate-column detection; types `INTEGER`/`INT`/`BIGINT`/`SMALLINT`, `TEXT`/`VARCHAR`, `REAL`/`FLOAT`/`DOUBLE`/`DECIMAL`, `BOOLEAN`. Auto-creates `sqlrite_autoindex_<table>_<col>` for every PK + UNIQUE column |
| `CREATE [UNIQUE] INDEX` | Single-column, named indexes; `IF NOT EXISTS`; persists as a dedicated cell-based B-Tree. INTEGER + TEXT columns only |
| `INSERT INTO` | Explicit column list required; auto-ROWID for `INTEGER PRIMARY KEY`; multi-row `VALUES (…), (…)`; UNIQUE enforcement; clean type errors (no panics); NULL padding for omitted columns |
| `SELECT` | `*` or column list; `WHERE`; single-column `ORDER BY [ASC\|DESC]`; `LIMIT n`. `WHERE col = literal` probes an index when one exists |
| `SELECT` | `*` or column list with optional `AS alias`; `WHERE`; `DISTINCT`; `GROUP BY col[, col …]`; aggregate projections `COUNT(*)` / `COUNT([DISTINCT] col)` / `SUM` / `AVG` / `MIN` / `MAX`; single-column `ORDER BY [ASC\|DESC]` (also resolves alias and aggregate display names); `LIMIT n`. `WHERE col = literal` probes an index when one exists |
| `UPDATE` | Multi-column `SET`; `WHERE`; UNIQUE + type enforcement; arithmetic in assignments (`SET age = age + 1`) |
| `DELETE` | `WHERE` predicate or full-table delete |
| `BEGIN` / `COMMIT` / `ROLLBACK` | Real transactions, snapshot-based; WAL-backed commit; single-level (no savepoints); auto-rollback if `COMMIT`'s disk write fails |
Expand All @@ -166,12 +166,14 @@ Expressions in `WHERE` and `UPDATE`'s `SET` RHS:

- Comparisons — `=`, `<>`, `<`, `<=`, `>`, `>=`
- Null tests — `IS NULL`, `IS NOT NULL`
- Pattern matching — `LIKE`, `NOT LIKE`, `ILIKE` (`%` and `_` wildcards, `\`-escaped literals; case-insensitive ASCII to match SQLite's default)
- Set membership — `IN (list)`, `NOT IN (list)` (literal lists only; subquery form is not supported yet)
- Logical — `AND`, `OR`, `NOT` (SQL three-valued logic; NULL-as-false in `WHERE`)
- Arithmetic — `+`, `-`, `*`, `/`, `%` (integer ops stay integer; any `REAL` promotes to `f64`; divide/modulo by zero is a clean error)
- String concat — `||`
- Literals — integer + real numbers, `'single-quoted strings'`, `TRUE` / `FALSE`, `NULL`; parentheses for grouping

**Not yet supported** (common ones): joins, subqueries, CTEs, `GROUP BY` / aggregates, `DISTINCT`, `LIKE` / `IN`, expressions in the projection list, column aliases, `OFFSET`, multi-column `ORDER BY`, savepoints, `ALTER TABLE`, `DROP TABLE`, `DROP INDEX`. The [full list with context](docs/supported-sql.md#not-yet-supported) lives in the reference.
**Not yet supported** (common ones): joins, subqueries, CTEs, `HAVING`, `LIKE … ESCAPE '<char>'`, `IN (subquery)`, `DISTINCT` on `SUM`/`AVG`/`MIN`/`MAX`, GROUP BY on expressions, expressions in the projection list, `OFFSET`, multi-column `ORDER BY`, savepoints, `ALTER TABLE`, `DROP TABLE`, `DROP INDEX`. The [full list with context](docs/supported-sql.md#not-yet-supported) lives in the reference.

#### Meta commands

Expand Down Expand Up @@ -310,7 +312,7 @@ Lockstep versioning — one dispatch bumps every product to the same `vX.Y.Z`. T

**Possible extras** *(no committed phase)*
- Joins (`INNER`, `LEFT OUTER`, `CROSS` — SQLite does not support `RIGHT`/`FULL OUTER`)
- `GROUP BY`, aggregates (`COUNT`, `SUM`, `AVG`, ...), `DISTINCT`, `LIKE`, `IN`
- `HAVING`, `IN (subquery)`, `BETWEEN`, `GLOB` / `REGEXP`, `GROUP_CONCAT`, window functions
- Composite and expression indexes (with cost analysis)
- Alternate storage engines — LSM/SSTable for write-heavy workloads alongside the B-Tree
- Benchmarks against SQLite
Expand Down
44 changes: 42 additions & 2 deletions docs/sql-engine.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,11 +45,13 @@ The `sqlparser` AST is designed to cover every SQL dialect, so its types are hug
|---|---|---|
| `Statement::CreateTable(CreateTable)` | `CreateQuery { table_name, columns: Vec<ParsedColumn> }` | [`create.rs`](../src/sql/parser/create.rs) |
| `Statement::Insert(Insert)` | `InsertQuery { table_name, columns, rows }` | [`insert.rs`](../src/sql/parser/insert.rs) |
| `Statement::Query(_)` | `SelectQuery { table_name, projection, selection, order_by, limit }` | [`select.rs`](../src/sql/parser/select.rs) |
| `Statement::Query(_)` | `SelectQuery { table_name, projection, selection, order_by, limit, distinct, group_by }` | [`select.rs`](../src/sql/parser/select.rs) |

`UPDATE` and `DELETE` don't have a dedicated internal struct — the executor pattern-matches the sqlparser types directly because there's less transformation needed.

Each parser module also rejects features we don't implement with `SQLRiteError::NotImplemented` — `JOIN`, `GROUP BY`, `HAVING`, `DISTINCT`, `OFFSET`, multi-table DELETE, tuple assignment targets, etc. These errors carry the feature name in the message so the user knows what isn't there.
`SelectQuery::projection` is now `Projection::All | Projection::Items(Vec<ProjectionItem>)`, where each item carries a `ProjectionKind::Column(name)` or `ProjectionKind::Aggregate(AggregateCall)` plus an optional `AS alias`. `AggregateCall` covers `COUNT(*)`, `COUNT([DISTINCT] col)`, `SUM` / `AVG` / `MIN` / `MAX` of a bare column. `group_by` is a `Vec<String>` of bare column names (empty = no GROUP BY); the parser validates that every non-aggregate projection item appears in `GROUP BY`.

Each parser module still rejects features we don't implement with `SQLRiteError::NotImplemented` — `JOIN`, `HAVING`, `DISTINCT ON (...)`, `GROUP BY` on expressions, `LIKE … ESCAPE '<char>'`, `IN (subquery)`, `OFFSET`, multi-table DELETE, tuple assignment targets, etc. These errors carry the feature name in the message so the user knows what isn't there.

## Statement dispatch

Expand Down Expand Up @@ -90,6 +92,9 @@ match query {
|---|---|
| Logical | `AND`, `OR`, `NOT` |
| Comparison | `=`, `<>`, `<`, `<=`, `>`, `>=` |
| Null tests | `IS NULL`, `IS NOT NULL` |
| Pattern | `LIKE`, `NOT LIKE`, `ILIKE` (`%`, `_`, `\`-escape; case-insensitive ASCII) |
| Set | `IN (list)`, `NOT IN (list)` (literal lists only) |
| Arithmetic | `+`, `-`, `*`, `/`, `%` |
| String | `\|\|` |
| Unary | `+`, `-`, `NOT` |
Expand Down Expand Up @@ -139,6 +144,41 @@ Returns top-k from the inverted index in `O(query-term-count × k log k)`. The p

The full canonical FTS reference is in [`docs/fts.md`](fts.md).

### Aggregation phase

When a SELECT contains an aggregate projection or a GROUP BY clause, the
rowid-shaped optimizations don't compose with grouping (every row
contributes to its group), so the executor takes a separate path:

1. Filter by `WHERE` exactly as before — including the index-probe fast
path — to get the matching rowid set.
2. For each matching rowid, derive a **group key** as a
`Vec<DistinctKey>` (one entry per `GROUP BY` column; empty key for
queries with aggregates but no `GROUP BY`).
3. Update one `AggState` per (group, aggregate-projection-slot) —
`AggState` lives in [`src/sql/agg.rs`](../src/sql/agg.rs) and tracks
the SQLite numeric type rules (`SUM` stays `INTEGER` until a `REAL`
input or `i64` overflow promotes it; `AVG` is always `REAL`; `MIN`/`MAX`
reuse the executor's total order; `COUNT(DISTINCT col)` uses a
`HashSet<DistinctKey>`).
4. Emit one output row per group, in projection order — bare-column
slots emit the captured group-key value, aggregate slots emit
`AggState::finalize()`.
5. Apply DISTINCT (post-projection dedup), then ORDER BY (resolved
against the *output* row by alias, bare column name, or aggregate
display form), then LIMIT.

Aggregate function names (`COUNT`/`SUM`/`AVG`/`MIN`/`MAX`) used in WHERE
or any other scalar position get a friendly error redirecting the user
to the projection list (since `HAVING` isn't supported yet). DISTINCT
on `SUM`/`AVG`/`MIN`/`MAX` is rejected at parse time; only
`COUNT(DISTINCT col)` is in v1.

`LIKE` / `ILIKE` use a hand-rolled iterative two-pointer matcher in
`agg.rs::like_match` (no regex dep). `IN (list)` follows SQLite's
three-valued logic for NULL on either side, which collapses to "row
excluded" under WHERE's NULL-as-false rule.

## Two-pass pattern for UPDATE and DELETE

Both `execute_update` and `execute_delete` use the same pattern to satisfy Rust's aliasing rules:
Expand Down
57 changes: 44 additions & 13 deletions docs/supported-sql.md
Original file line number Diff line number Diff line change
Expand Up @@ -148,35 +148,63 @@ Hex literals, blob literals, and date/time functions are not supported.
## `SELECT`

```sql
SELECT {* | col1, col2, ...}
SELECT [DISTINCT] {* | <projection_item>[, <projection_item>, ...]}
FROM <table>
[WHERE <expr>]
[ORDER BY <col> [ASC|DESC]]
[GROUP BY <col>[, <col>, ...]]
[ORDER BY <expr> [ASC|DESC]]
[LIMIT <non-negative-integer>];
```

`<projection_item>` is one of:

```
<column> -- bare column reference
COUNT(*) -- counts every row, including all-NULL ones
COUNT([DISTINCT] <column>) -- counts non-NULL values, optionally deduping
{SUM | AVG | MIN | MAX}(<column>) -- aggregate over a single column
<projection_item> AS <alias> -- optional column alias
```

### What works

- **Projection**: `*` (all columns in declaration order) or a bare column list. Columns not declared on the table are rejected.
- **`WHERE`**: any [expression](#expressions). Evaluated per row; NULL-as-false in WHERE context (three-valued logic collapsed to two-valued for filtering). Includes **`IS NULL`** / **`IS NOT NULL`** for explicit null tests.
- **`ORDER BY`**: single sort key, `ASC` (default) or `DESC`. The sort key can be a bare column reference OR any expression — including function calls — so KNN queries like `ORDER BY vec_distance_l2(embedding, [...]) LIMIT k` work end-to-end *(Phase 7b)*. Sort key types must match; mixing `INTEGER` and `TEXT` across rows under a single `ORDER BY` is a runtime error.
- **`LIMIT`**: non-negative integer literal. `LIMIT 0` is valid (returns zero rows).
- **Projection**: `*` (all columns in declaration order), a bare column list, or an explicit list mixing bare columns and aggregate calls. Each item can carry an optional `AS alias` (the alias becomes the output column header and is recognized by `ORDER BY`).
- **`WHERE`**: any [expression](#expressions). Evaluated per row; NULL-as-false in WHERE context (three-valued logic collapsed to two-valued for filtering). Includes **`IS NULL`** / **`IS NOT NULL`** for explicit null tests, **`LIKE` / `NOT LIKE` / `ILIKE`** for pattern matching, and **`IN (list) / NOT IN (list)`** for set-membership against literal lists.
- **`DISTINCT`**: `SELECT DISTINCT` deduplicates result rows after projection (and after aggregation, when both apply). `NULL` values compare equal to other `NULL`s for dedupe, matching SQL's DISTINCT semantic.
- **`GROUP BY`**: one or more bare column names. Every non-aggregate item in the projection must appear in the `GROUP BY` list (the parser rejects the violation with a clear message). `GROUP BY <col>` without any aggregate behaves like an implicit `DISTINCT <col>`.
- **Aggregates** (SQLR-3): `COUNT(*)`, `COUNT(col)`, `COUNT(DISTINCT col)`, `SUM(col)`, `AVG(col)`, `MIN(col)`, `MAX(col)`. `SUM` over an integer column stays `INTEGER` until a `REAL` input arrives or the running sum overflows `i64` (one-time promotion to `REAL`). `AVG` always returns `REAL` (or `NULL` on empty / all-NULL groups). `MIN` / `MAX` skip NULLs and use the same total order as `ORDER BY`. Aggregates over an empty table or empty group return `0` for `COUNT(*)` / `COUNT(col)` and `NULL` for the rest.
- **`ORDER BY`**: single sort key, `ASC` (default) or `DESC`. For non-aggregating queries the key is any expression — including function calls — so KNN queries like `ORDER BY vec_distance_l2(embedding, [...]) LIMIT k` work end-to-end *(Phase 7b)*. For aggregating queries the key resolves against the *output* row by name: a bare identifier matches an alias or a `GROUP BY` column, and a function call like `COUNT(*)` matches an aggregate projection by its canonical display form. Sort key types must match across rows.
- **`LIMIT`**: non-negative integer literal. `LIMIT 0` is valid (returns zero rows). When `DISTINCT` is in play, `LIMIT` is applied after deduplication so it counts unique rows.

### Index probing

The executor includes a tiny optimizer: if the `WHERE` is exactly `<indexed_col> = <literal>` or `<literal> = <indexed_col>`, it probes the index and scans only matching rows. Mixed predicates (`WHERE a = 1 AND b > 2`), range predicates (`WHERE a > 1`), and OR-combined predicates fall back to a full table scan.
The executor includes a tiny optimizer: if the `WHERE` is exactly `<indexed_col> = <literal>` or `<literal> = <indexed_col>`, it probes the index and scans only matching rows. Mixed predicates (`WHERE a = 1 AND b > 2`), range predicates (`WHERE a > 1`), and OR-combined predicates fall back to a full table scan. Aggregating queries (`GROUP BY` / aggregate functions) skip the rowid-shape optimizations (HNSW / FTS / bounded-heap top-k) since every matching row contributes to its group.

### `LIKE` semantics

- `%` matches any (possibly empty) char sequence; `_` matches exactly one char. `\` escapes the next character so `\%` matches a literal percent. Outside `\%` / `\_` / `\\`, a backslash is itself a literal — matching SQLite's loose default.
- Case folding is **ASCII-only and on by default**, mirroring SQLite's default `PRAGMA case_sensitive_like = OFF`. `LIKE 'a%'` matches both `Apple` and `apple`. Non-ASCII characters compare by code point (no Unicode case folding).
- `LIKE … ESCAPE '<char>'` is not supported. `LIKE ANY (...)` is not supported.
- `NULL LIKE 'pattern'` evaluates to `NULL`; in a `WHERE` that excludes the row.

### `IN` semantics

- Only the literal-list form is supported: `WHERE x IN (1, 2, 3)` and `WHERE x NOT IN (...)`.
- Three-valued logic: if the LHS is `NULL`, the result is `NULL`; if the RHS list contains a `NULL` and no other entry matches, the result is `NULL`. In a `WHERE` both cases collapse to "row excluded", matching SQLite.
- `IN (subquery)`, `IN UNNEST(...)`, and `BETWEEN` are not supported yet.

### What doesn't work

- **Joins** of any kind (`INNER`, `LEFT OUTER`, `CROSS`, comma-join)
- **Subqueries**, CTEs (`WITH`), views
- **`GROUP BY`**, aggregate functions (`COUNT`, `SUM`, `AVG`, `MIN`, `MAX`), `HAVING`
- **`DISTINCT`**
- **`LIKE`**, **`IN`**, `BETWEEN`
- **Expressions in the projection list** (`SELECT age + 1 FROM users`) — projection is bare column references only
- **Multi-column `ORDER BY`**, `NULLS FIRST/LAST` (single sort key only; the sort key itself can be an expression as of Phase 7b)
- **`HAVING`** — pre-aggregation `WHERE` works; post-aggregation filtering does not yet
- **`DISTINCT`** on `SUM` / `AVG` / `MIN` / `MAX` (only `COUNT(DISTINCT col)` is supported)
- **`GROUP BY` on expressions** — bare column names only in v1
- **`LIKE … ESCAPE '<char>'`**, **`IN (subquery)`**, **`BETWEEN`**, **`GLOB`**, **`REGEXP`**
- **Expressions in the projection list** beyond aggregate calls (`SELECT age + 1 FROM users` is still rejected; aggregates are the one allowed expression form)
- **Multi-column `ORDER BY`**, `NULLS FIRST/LAST` (single sort key only)
- **`OFFSET`**
- **Column aliases** (`SELECT name AS n FROM users`)
- **Window functions** (`OVER (...)`, `FILTER (WHERE ...)`, `WITHIN GROUP`)

Any of the above reaches the executor as a parsed AST node that execution doesn't handle, producing either `NotImplemented` or a more specific error (e.g., `joins are not supported`).

Expand Down Expand Up @@ -286,6 +314,9 @@ Expressions work inside `WHERE` (both in `SELECT`, `UPDATE`, `DELETE`) and on th
| Category | Operators |
|---|---|
| Comparison | `=`, `<>`, `<`, `<=`, `>`, `>=` |
| Null tests | `IS NULL`, `IS NOT NULL` |
| Pattern | `LIKE`, `NOT LIKE`, `ILIKE` (`%`, `_`, `\`-escape; case-insensitive ASCII) |
| Set | `IN (list)`, `NOT IN (list)` (literal lists only) |
| Logical | `AND`, `OR`, `NOT` |
| Arithmetic | `+`, `-`, `*`, `/`, `%` |
| String | `\|\|` (concatenation) |
Expand Down
Loading
Loading