Phase 6: schema, system catalog, DDL, constraint enforcement; txn write-job generalization

Author: Thanga-Ganapathy

CHANGELOG.md

@@ -8,6 +8,47 @@ under a category (`Added` / `Changed` / `Fixed` / `Removed` / `Security`).
 
 ## [Unreleased]
 
+### Phase 6 — Schema, catalog & constraints
+
+#### Added
+- `catalog`: strict typed tables over the transaction layer — the schema
+  model (`TableDef`/`ColumnDef`: types, single/composite PK, NOT NULL,
+  UNIQUE, CHECK, DEFAULT values + `now`/`uuid_v7` generators, auto-increment,
+  rowversion, `on_update: now`, `update: free | guarded`) with definition
+  validation, persisted in the in-file system catalog.
+- The system catalog **is** the published-root B+tree (`DECISIONS.md` D14):
+  per table a schema entry, a data-root entry, and an auto-increment sequence
+  entry — one commit covers schema + data atomically and one snapshot pins
+  both consistently.
+- DDL as ordinary write transactions: `create_table`, `drop_table` (frees the
+  whole table tree via deferred reclamation), `add_column` (nullable or
+  constant default; old rows padded lazily on read — D17).
+- Row DML with full constraint enforcement on insert and update; multi-row
+  `insert_many` is atomic (in-batch PK/UNIQUE collisions included); typed
+  per-constraint errors mapped to the `SPEC.md` §9 taxonomy.
+- Provisional `CheckExpr` (comparisons + boolean combinators, SQL 3VL: NULL
+  passes — D15) and provisional scan-based UNIQUE probes until Phase 7's
+  unique indexes (D16).
+- Engine-managed values under the writer: auto-increment (durable sequence,
+  no reuse across reopen), rowversion (1, then +1 per update), `now` /
+  `uuid_v7` defaults and `on_update: now` driven by the injected clock/RNG.
+- Exit-criteria tests: create/reopen/inspect round-trip, one violation test
+  per constraint, guarded persisted/readable, generated values under a
+  manual clock, multi-row atomicity, snapshot consistency across DDL+DML.
+
+#### Changed
+- `txn`: write transactions generalized to a `WriteJob` trait run against a
+  `WriteCtx` (multi-tree edits, typed post-commit outputs); `Db<B>` is now an
+  alias for `JobDb<B, OpsJob>` (API unchanged); the writer classifies job
+  errors by category (`Io`/`Corruption` fatal, others reject the one
+  transaction); new `TxnError::Rejected` carries a higher layer's typed
+  error across the writer thread; `Snapshot` gains `root`/`get_in`/
+  `range_in`/`scan_in` for reading trees under the pinned root.
+- `btree`: `BTree::pages` collects every page of a tree (drop-table
+  reclamation).
+- `types`: `UuidV7Gen` now owns `Arc<dyn Clock>`/`Arc<dyn Rng>` so generator
+  state can span transactions.
+
 ### Phase 5 — Types, values & encoding
 
 #### Added

Cargo.lock

@@ -5,6 +5,74 @@ Per `PLAN.md` §1 rule 6, every resolution of an ambiguity or deviation from
 
 ---
 
+## D17 — Write-path semantics the SPEC leaves open
+
+**Phase:** 6 · **Status:** accepted
+
+`SPEC.md` §4 defines the constraint set but not every edge of the write path.
+Phase 6 fixes these (each is a one-line change later if v2 decides otherwise):
+
+- **PK columns are immutable** — an update touching one is rejected
+  (`PkImmutable`). Change-of-key is delete + insert.
+- **Engine-managed columns reject explicit writes**: `rowversion` and
+  `on_update: now` columns can never be set by the caller.
+- **`rowversion` starts at 1** on insert and bumps by 1 on every update.
+- **`on_update: now` also stamps on insert** (an `updated_at` is never NULL).
+- **Explicit values on auto-increment columns are allowed**; the sequence
+  advances past them (`seq = max(seq, given + 1)`), so generated keys never
+  collide. The sequence is a catalog entry — durable, gaps allowed, no reuse
+  after crash/reopen.
+- **`add column`** requires nullable or a *constant* default; existing rows
+  are padded lazily on read (generators cannot backfill), no rewrite.
+- **UNIQUE ignores NULLs** (multiple NULLs allowed, SQL semantics).
+
+## D16 — UNIQUE enforced by a scan probe until Phase 7
+
+**Phase:** 6 · **Status:** accepted (provisional)
+
+`SPEC.md` §4.1 says UNIQUE is "enforced via a unique index", but `index` is
+Phase 7. Phase 6 enforces the constraint **correctly but provisionally** with
+a full-table scan probe (one scan per write batch). Phase 7's unique indexes
+replace the probe with an index lookup; behavior is unchanged, only cost.
+
+## D15 — Provisional CHECK expressions with SQL three-valued logic
+
+**Phase:** 6 · **Status:** accepted (provisional)
+
+`SPEC.md` §4.1 allows `CHECK(<expr>)` over the row, but the expression
+language (§5.2) arrives with the query layer in Phases 8–9.
+
+**Decision:** Phase 6 ships a minimal `CheckExpr` — column-vs-literal
+comparisons, `and`/`or`/`not`, `is_null`/`is_not_null` — validated at DDL
+(columns exist, literal kinds match) and stored in the catalog. Evaluation
+follows SQL 3VL: a CHECK is violated only when it is definitively **false**;
+NULL/unknown passes. The Phase 8/9 expression engine supersedes the enum
+(same precedent as Phase 3's provisional byte keys).
+
+## D14 — One published root: the catalog tree owns everything
+
+**Phase:** 6 · **Status:** accepted
+
+`ARCHITECTURE.md` §3.5 stores the system catalog "in B+trees referenced from
+the meta page" but leaves the multi-tree commit mechanics open.
+
+**Decision:**
+- The meta page's root **is the catalog B+tree**; every other tree hangs off
+  it. Per table: `("tbl", name)` → schema (changes on DDL), `("root", name)` →
+  data-tree root (changes on every write), `("seq", name)` → auto-increment
+  cursor. A write to table T updates T's tree, then T's root entry, producing
+  one new catalog root — so one root install + one fsync pair commits schema
+  and data atomically, and a snapshot pins both consistently.
+- `txn` generalizes to **`WriteJob`/`WriteCtx`**: a job runs on the writer
+  thread, edits any tree under the root, and returns a typed output delivered
+  after durable commit. `Db<B>` is now an alias for `JobDb<B, OpsJob>` — the
+  Phase 4 API and tests are unchanged. Jobs inherit D8's validate-then-apply
+  contract; the writer classifies job errors **by category** (`Io`/
+  `Corruption` fatal, everything else rejects just that transaction, with the
+  root and freed-list restored defensively). Rejections cross the thread as
+  `TxnError::Rejected(Box<dyn CategorizedError>)` and downcast back to the
+  catalog's typed error at the API surface.
+
 ## D13 — No `u64` value type; `rowversion` columns are `i64`
 
 **Phase:** 5 · **Status:** accepted (SPEC inconsistency, raised)

DECISIONS.md

@@ -168,6 +168,21 @@ impl<'p, B: IoBackend> BTree<'p, B> {
         })
     }
 
+    /// Collect every page id reachable from `root` — the whole tree, in no
+    /// particular order. Used by callers that retire an entire tree (e.g.
+    /// `drop table`) and must hand all of its pages to deferred reclamation.
+    pub fn pages(&self, root: PageId) -> Result<Vec<PageId>> {
+        let mut out = Vec::new();
+        let mut stack = vec![root];
+        while let Some(id) = stack.pop() {
+            out.push(id);
+            if let Node::Internal { children, .. } = self.read_node(id)? {
+                stack.extend(children);
+            }
+        }
+        Ok(out)
+    }
+
     // --- internals -------------------------------------------------------
 
     fn read_node(&self, id: PageId) -> Result<Node> {

crates/btree/src/tree.rs

@@ -11,6 +11,8 @@ description = "Schema model, system catalog, DDL, constraint enforcement (Phase
 
 [dependencies]
 common.workspace = true
+pager.workspace = true
+btree.workspace = true
 types.workspace = true
 txn.workspace = true
 thiserror.workspace = true