feat(storage): free-list + VACUUM to reclaim orphan pages (SQLR-6)

docs/file-format.md

@@ -4,7 +4,7 @@ A SQLRite database is a single file, by convention named `*.sqlrite`. The file i
 
 All multi-byte integers in this format are **little-endian**.
 
-The current on-disk format is **version 4** (Phase 7) by default, with **version 5** written on demand whenever an FTS index is attached to the database (Phase 8c). Decoders accept both v4 and v5; writers preserve the existing version on no-op resaves so a v4 database without FTS stays v4. Files produced by versions 1 – 3 are rejected on open.
+The current on-disk format is **version 4** (Phase 7) by default, with **version 5** written on demand whenever an FTS index is attached to the database (Phase 8c) and **version 6** written on demand whenever a save produces a non-empty freelist (SQLR-6). Decoders accept v4, v5, and v6; writers preserve the existing version on no-op resaves so a v4 database without FTS or freelist stays v4. Files produced by versions 1 – 3 are rejected on open.
 
 ## Page 0 — the database header
 
@@ -15,17 +15,18 @@ The first 4096 bytes of every file are the header page. Only the first 28 bytes
 │ offset │ length │ content                                         │
 ├────────┼────────┼─────────────────────────────────────────────────┤
 │     0  │   16   │ magic:  "SQLRiteFormat\0\0\0"                   │
-│    16  │    2   │ format version (u16 LE) = 4 or 5                │
+│    16  │    2   │ format version (u16 LE) = 4, 5, or 6            │
 │    18  │    2   │ page size      (u16 LE) = 4096                  │
 │    20  │    4   │ total page count (u32 LE), includes page 0      │
 │    24  │    4   │ root page of sqlrite_master (u32 LE)            │
-│    28  │ 4068   │ reserved / zero                                 │
+│    28  │    4   │ freelist head    (u32 LE; 0 = empty) — v6 only  │
+│    32  │ 4064   │ reserved / zero                                 │
 └────────┴────────┴─────────────────────────────────────────────────┘
 ```
 
 The magic string is 14 ASCII bytes (`SQLRiteFormat`) padded with two NUL bytes to fill 16 bytes. It's deliberately different from SQLite's `"SQLite format 3\0"` so the two formats can't be confused on inspection.
 
-`decode_header` in [`src/sql/pager/header.rs`](../src/sql/pager/header.rs) validates all three of (magic, format version, page size) on open. A wrong magic produces `not a SQLRite database`; a wrong version or page size produces `unsupported ...` errors. The decoder accepts both v4 and v5 (anything else is rejected); the parsed `format_version` is propagated through the in-memory `DbHeader` so the writer can preserve it on resave when no version-bumping feature has been added.
+`decode_header` in [`src/sql/pager/header.rs`](../src/sql/pager/header.rs) validates all three of (magic, format version, page size) on open. A wrong magic produces `not a SQLRite database`; a wrong version or page size produces `unsupported ...` errors. The decoder accepts v4, v5, and v6 (anything else is rejected); the parsed `format_version` is propagated through the in-memory `DbHeader` so the writer can preserve it on resave when no version-bumping feature has been added. `freelist_head` is read from bytes [28..32]: v4/v5 files leave that region zero so it always decodes as `0` (an empty freelist), and v6 files store the page number of the first freelist trunk there.
 
 ## Pages 1..page_count — payload pages
 
@@ -53,6 +54,7 @@ Every non-header page starts with a 7-byte header:
 | `2` | `TableLeaf` | Holds a slot directory and a set of cells representing rows of a table. Leaves for one table are linked by sibling `next_page` pointers. |
 | `3` | `Overflow` | Continuation page carrying the spilled body of a single oversized cell. |
 | `4` | `InteriorNode` | Interior B-Tree node. Holds a slot directory of divider cells routing to child pages plus a rightmost-child pointer in the payload header. |
+| `5` | `FreelistTrunk` | One link of the persisted free-page list (SQLR-6). Payload carries `count: u16` followed by `count × u32` free leaf-page numbers; `next_page` chains to the next trunk (0 = end). |
 
 Tag `1` is reserved (it was `SchemaRoot` in format v1; unused in v2). Any other tag on open is a corruption error.
 
@@ -307,6 +309,7 @@ These are not all enforced on open — we validate the header strictly and rely
 - **v3** (Phase 3e) — `sqlrite_master` gains a `type` column; secondary indexes persist as their own cell-based B-Trees whose leaves carry `KIND_INDEX` cells.
 - **v4** (Phase 7a) — value block dispatch gains the `0x04 Vector` tag for the new `VECTOR(N)` column type. Per the [Phase 7 plan's Q8](phase-7-plan.md#q8-file-format-version-bump), later Phase 7 sub-phases (JSON storage, HNSW indexes) added their own value/cell tags inside this same v4 envelope. The `CREATE TABLE` SQL stored in `sqlrite_master` carries vector columns as `VECTOR(N)` in the type position; on open, the engine re-parses that SQL and reconstructs `DataType::Vector(N)` from the `Custom` AST node sqlparser produces.
 - **v5** (Phase 8c, current for FTS-bearing files) — adds the `KIND_FTS_POSTING` cell tag for persisted FTS posting lists. Bumped **on demand** per the [Phase 8 plan's Q10](phase-8-plan.md#q10-file-format-version-bump-strategy): existing v4 databases without FTS keep writing v4 across non-FTS saves; the first save with at least one FTS index attached promotes the file to v5. Decoders accept both v4 and v5; opening a v4 file with a build that supports v5 is a no-op until the user creates an FTS index.
+- **v6** (SQLR-6, current for files with persisted free-page lists) — adds the `freelist_head` field at header bytes [28..32] and the `FreelistTrunk` page tag (`5`). Bumped **on demand**: a save that ends with an empty freelist preserves the existing version; the first save that produces a non-empty freelist promotes the file to v6. Decoders accept v4, v5, and v6; v6 is a strict superset, so opening a v4/v5 file with a v6-aware build is a no-op until the user creates a freelist (e.g., by dropping a table or index). VACUUM clears the freelist but doesn't downgrade.
 
 The page header (7 bytes) and chaining mechanism are stable across future phases. Phase 4's WAL introduces a sibling file (`.sqlrite-wal`) rather than changing the main file format.
 

docs/pager.md

@@ -187,10 +187,23 @@ Without the diff, step 3's "re-serialize every table" would trigger a full file
 
 This only works because `save_database` iterates tables in sorted order — if the order were random, a table that didn't change might land at a different page number, appearing dirty. See [Design decisions §7](design-decisions.md#7-deterministic-page-number-ordering-when-saving).
 
+## Free-page list and VACUUM (SQLR-6)
+
+Save now uses a [`PageAllocator`](../src/sql/pager/allocator.rs) instead of a bare `next_free_page` counter. The allocator pulls pages from three sources, in preference order:
+
+1. **Per-table preferred pool** — every table/index/master is given the page numbers it occupied last save (collected by walking from its old `rootpage`). An unchanged table re-stages byte-identical pages at the same numbers, so the diff pager skips every write for it.
+2. **Global freelist** — pages from dropped tables/indexes that are recorded in the persisted freelist (rooted at `header.freelist_head`).
+3. **Extend** — `next_extend++`, monotonic past the high-water mark.
+
+After staging, pages that were live before this save but didn't get restaged this round (e.g., the leaves of a dropped table) move onto the new freelist. The freelist itself is encoded into a chain of `FreelistTrunk` pages — each trunk holds up to 1021 free leaf-page numbers plus a `next_page` pointer to the following trunk. Trunks consume some of the free pages they describe (a trunk page IS a free page borrowed for metadata), so a freelist of N pages takes `ceil(N / 1022)` trunks and persists `N − T` leaf entries.
+
+`VACUUM;` (a SQL statement) calls [`vacuum_database`](../src/sql/pager/mod.rs), which is `save_database` with empty per-table preferred pools and an empty initial freelist. Allocation falls through to extend on every page → contiguous layout from page 1, no freelist trunks, file truncates to the new high-water mark on the next checkpoint.
+
+Format-version side effect: a save that produces a non-empty freelist promotes the file from v4/v5 to v6 (mirrors Phase 8c's v4→v5 FTS rule). VACUUM clears the freelist but doesn't downgrade — v6 is a strict superset.
+
 ## What it doesn't do (yet)
 
 - **No LRU eviction.** `on_disk` + `wal_cache` together grow with the page count. For a 1 GiB database, that's ~1 GiB of page cache. Bounded cache is future work.
-- **No free-page management.** When a table shrinks, the main file's tail pages are truncated at checkpoint, but there's no free-list to reuse pages inside a grown file.
 - **No per-statement granularity.** The whole database is re-serialized on every commit; the diff keeps the *written* set small but the CPU cost of reserialization is unchanged.
 - **No concurrent reader-and-writer.** Phase 4e graduated to shared/exclusive lock modes (multi-reader *or* single-writer), but POSIX flock can't give us both at once. True concurrent access would need a shared-memory coordination file with read marks — not on the roadmap.
 - **Savepoints / nested transactions.** Phase 4f added top-level `BEGIN` / `COMMIT` / `ROLLBACK` (snapshot-based rollback, auto-save suppressed inside a transaction), but nested `BEGIN` is rejected — real savepoints aren't on the roadmap.

docs/supported-sql.md

@@ -17,6 +17,7 @@ If you're looking for _how_ to use SQLRite (REPL flow, meta-commands, history, e
 | [`ALTER TABLE`](#alter-table) | `RENAME TO`, `RENAME COLUMN`, `ADD COLUMN`, `DROP COLUMN` (one operation per statement) |
 | [`DROP TABLE`](#drop-table) / [`DROP INDEX`](#drop-index) | `IF EXISTS`; single target; auto-indexes refused for `DROP INDEX` |
 | [`BEGIN`](#transactions) / [`COMMIT`](#transactions) / [`ROLLBACK`](#transactions) | Snapshot-based; single-level; WAL-backed commit; auto-rollback on COMMIT disk failure |
+| [`VACUUM`](#vacuum) | Compacts the file: rewrites every live B-Tree contiguously from page 1 and clears the freelist. Bare `VACUUM;` only — no modifiers. |
 
 Statements the parser accepts (because sqlparser understands them in the SQLite dialect) but SQLRite doesn't execute yet return `SQL Statement not supported yet`. The [Not yet supported](#not-yet-supported) section below enumerates the common ones.
 
@@ -259,7 +260,7 @@ DROP TABLE [IF EXISTS] <table>;
 - Reserved-name rejection: `DROP TABLE sqlrite_master` errors with the same message `CREATE TABLE` uses.
 - All indexes attached to the table (auto, explicit, HNSW, FTS) disappear with the table — they live inside the `Table` struct and ride along.
 - Without `IF EXISTS`, dropping a table that doesn't exist errors. With it, that's a benign 0-tables-dropped no-op.
-- **Disk pages are orphaned, not freed.** SQLRite has no free-list yet — the file size doesn't shrink until a future `VACUUM`. The behavior is safe (orphan pages are unreachable from `sqlrite_master` after reopen) but means a write-heavy schema churn won't reclaim space until VACUUM lands.
+- **Disk pages move onto the freelist.** Pages the dropped table occupied are pushed onto a persisted free-page list (SQLR-6) so subsequent `CREATE TABLE` or inserts can reuse them. The file doesn't shrink until [`VACUUM;`](#vacuum) compacts it.
 
 ---
 
@@ -428,6 +429,24 @@ ROLLBACK;  -- nothing was actually deleted
 
 ---
 
+## `VACUUM`
+
+```sql
+VACUUM;
+```
+
+Compacts the database file: rewrites every live table, index, HNSW graph, FTS posting tree, and `sqlrite_master` itself contiguously from page 1, drops the freelist, and lets the next checkpoint truncate the tail.
+
+- **Bare `VACUUM;` only.** Modifiers — `VACUUM FULL`, `VACUUM REINDEX`, table targets, `TO ... PERCENT`, `BOOST` — are parsed (sqlparser supports them) but rejected at execution with `VACUUM modifiers (FULL, REINDEX, table targets, etc.) are not supported`.
+- **Refused inside a transaction.** `BEGIN; VACUUM;` errors with `VACUUM cannot run inside a transaction`. Use `COMMIT;` first, then `VACUUM;`.
+- **No-op on in-memory databases.** Returns a `VACUUM is a no-op for in-memory databases` status string and does nothing — there's no file to compact.
+- **Status string** carries pages and bytes reclaimed: `VACUUM completed. <N> pages reclaimed (<B> bytes).`
+- **Format-version side effect.** A v4/v5 file that has been promoted to v6 by an earlier drop stays at v6 after VACUUM (v6 is a strict superset; we don't downgrade). A file that's already at v4/v5 because no drop ever happened on it doesn't get bumped by VACUUM.
+
+When to run it: any time after a string of `DROP TABLE` / `DROP INDEX` / `ALTER TABLE DROP COLUMN` operations if you care about file size. SQLRite reuses freelist pages on subsequent inserts, so a write-heavy workload may not need VACUUM at all — its main use is reclaiming space when you don't expect to grow back.
+
+---
+
 ## Read-only databases
 
 A REPL launched with `sqlrite --readonly foo.sqlrite` (or `sqlrite::open_database_read_only(path, name)` programmatically) takes a shared POSIX advisory lock instead of an exclusive one. In that mode:

src/sql/executor.rs

@@ -717,6 +717,60 @@ pub fn execute_alter_table(alter: AlterTable, db: &mut Database) -> Result<Strin
     }
 }
 
+/// Executes `VACUUM;` (SQLR-6). Compacts the database file: rewrites
+/// every live table, index, and the catalog contiguously from page 1,
+/// drops the freelist, and truncates the tail at the next checkpoint.
+///
+/// Refuses to run inside a transaction (would publish in-flight writes
+/// out of band); refuses on read-only databases (handled upstream by
+/// the read-only mutation gate); and is a no-op on in-memory databases
+/// (no file to compact). Bare `VACUUM;` only — non-default options
+/// (`FULL`, `REINDEX`, table targets, etc.) are rejected.
+pub fn execute_vacuum(db: &mut Database) -> Result<String> {
+    if db.in_transaction() {
+        return Err(SQLRiteError::General(
+            "VACUUM cannot run inside a transaction".to_string(),
+        ));
+    }
+    let path = match db.source_path.clone() {
+        Some(p) => p,
+        None => {
+            return Ok("VACUUM is a no-op for in-memory databases".to_string());
+        }
+    };
+    // Checkpoint before AND after VACUUM so the main-file size we report
+    // reflects only what VACUUM actually reclaimed — without the leading
+    // checkpoint, `size_before` would be the stale main-file snapshot
+    // (typically 2 pages) while WAL holds the live bytes, making the
+    // bytes-reclaimed delta meaningless.
+    if let Some(pager) = db.pager.as_mut() {
+        let _ = pager.checkpoint();
+    }
+    let size_before = std::fs::metadata(&path).ok().map(|m| m.len()).unwrap_or(0);
+    let pages_before = db
+        .pager
+        .as_ref()
+        .map(|p| p.header().page_count)
+        .unwrap_or(0);
+    crate::sql::pager::vacuum_database(db, &path)?;
+    // Second checkpoint so the main file shrinks now — VACUUM's whole
+    // purpose is to reclaim bytes, so paying the I/O up front is fair.
+    if let Some(pager) = db.pager.as_mut() {
+        let _ = pager.checkpoint();
+    }
+    let size_after = std::fs::metadata(&path).ok().map(|m| m.len()).unwrap_or(0);
+    let pages_after = db
+        .pager
+        .as_ref()
+        .map(|p| p.header().page_count)
+        .unwrap_or(0);
+    let pages_reclaimed = pages_before.saturating_sub(pages_after);
+    let bytes_reclaimed = size_before.saturating_sub(size_after);
+    Ok(format!(
+        "VACUUM completed. {pages_reclaimed} pages reclaimed ({bytes_reclaimed} bytes)."
+    ))
+}
+
 /// Renames a table in `db.tables`. Updates `tb_name`, every secondary
 /// index's `table_name` field, and any auto-index whose name embedded
 /// the old table name. HNSW / FTS index entries don't carry a

src/sql/mod.rs

@@ -160,6 +160,9 @@ pub fn process_command_with_render(query: &str, db: &mut Database) -> Result<Com
 
     // Statements that mutate state — trigger auto-save on success. Read-only
     // SELECTs skip the save entirely to avoid pointless file writes.
+    // VACUUM is a write statement (rewrites the entire file) but it does
+    // its own save internally, so it's also explicitly excluded from the
+    // post-dispatch auto-save block at the bottom.
     let is_write_statement = matches!(
         &query,
         Statement::CreateTable(_)
@@ -169,7 +172,9 @@ pub fn process_command_with_render(query: &str, db: &mut Database) -> Result<Com
             | Statement::Delete(_)
             | Statement::Drop { .. }
             | Statement::AlterTable(_)
+            | Statement::Vacuum(_)
     );
+    let is_vacuum = matches!(&query, Statement::Vacuum(_));
 
     // Early-reject mutations on a read-only database before they touch
     // in-memory state. Phase 4e: without this, a user running INSERT
@@ -338,6 +343,27 @@ pub fn process_command_with_render(query: &str, db: &mut Database) -> Result<Com
         Statement::AlterTable(alter) => {
             message = executor::execute_alter_table(alter, db)?;
         }
+        Statement::Vacuum(vac) => {
+            // SQLR-6 — only bare `VACUUM;` is supported. The crate-level
+            // `VacuumStatement` carries Redshift-style modifiers we don't
+            // implement; reject any non-default flag rather than silently
+            // ignoring it.
+            if vac.full
+                || vac.sort_only
+                || vac.delete_only
+                || vac.reindex
+                || vac.recluster
+                || vac.boost
+                || vac.table_name.is_some()
+                || vac.threshold.is_some()
+            {
+                return Err(SQLRiteError::NotImplemented(
+                    "VACUUM modifiers (FULL, REINDEX, table targets, etc.) are not supported; use bare VACUUM;"
+                        .to_string(),
+                ));
+            }
+            message = executor::execute_vacuum(db)?;
+        }
         _ => {
             return Err(SQLRiteError::NotImplemented(
                 "SQL Statement not supported yet.".to_string(),
@@ -355,7 +381,10 @@ pub fn process_command_with_render(query: &str, db: &mut Database) -> Result<Com
     // mutated, so the caller should know disk is out of sync. The
     // Pager held on `db` diffs against its last-committed snapshot,
     // so only pages whose bytes actually changed are written.
-    if is_write_statement && db.source_path.is_some() && !db.in_transaction() {
+    //
+    // VACUUM is a write-shaped statement but already wrote the file
+    // internally — skip the second save to avoid undoing the compact.
+    if is_write_statement && !is_vacuum && db.source_path.is_some() && !db.in_transaction() {
         let path = db.source_path.clone().unwrap();
         pager::save_database(db, &path)?;
     }