feat(sqlite): pitr & forking

.agent/notes/depot-pitr-compactor-wf-review.md

@@ -0,0 +1,71 @@
+# Review: `04-29-feat_sqlite_pitr_forking` + `04-30-chore_sqlite_comapctor_wf`
+
+## Overview
+
+Two stacked PRs reshaping the SQLite storage backend (renamed from `sqlite-storage` to `depot`):
+
+- **`04-29-feat_sqlite_pitr_forking`** (~35k LoC, 50+ commits) — Per-database branches, fork/restore primitives, bookmarks (ephemeral + pinned, two-phase), namespace forks, S3 cold tier, GC pin recompute, burst-mode quota.
+- **`04-30-chore_sqlite_comapctor_wf`** (~13k LoC) — Reimplements compaction as a Gasoline workflow. New per-branch DB manager workflow + hot/cold/reclaimer companions, `CMP/root` manifest, global pin/proof/dirty indexes (partitions `0x70..=0x75`), `lifecycle_generation` field, `CompactionSignaler` injected into `Db`, 5076-line `workflows/compaction.rs`. Per CLAUDE.md, this workflow is the active compaction authority.
+
+---
+
+## High-Priority Correctness Issues
+
+### Conveyer
+
+1. **Pinned bookmarks bypass the two-phase protocol entirely.** `bookmark.rs:207, 413` write `PinStatus::Ready` synchronously. The two-phase contract requires the request tx to write `Pending`, set `SQLITE_CMP_DIRTY` (or signal the manager workflow via the `CompactionSignaler` already injected into `Db`), and have the cold companion + manager flip `Pending → Ready` after the S3 pin layer is uploaded. As written, no S3 pin layer is ever built and `pin_object_key` is permanently `None`.
+2. **Empty `bk_pin` recompute permanently locks the key at zero.** `bookmark.rs:685` writes `[0;16]` when no pins remain. Subsequent `MutationType::ByteMin` on the same key sees `min(0, new) = 0`, so the next pinned bookmark on this branch never advances `bk_pin`.
+3. **Fork-vs-GC OCC fence missing.** `branch.rs:525 derive_branch_at` reads `bk_pin` only; CLAUDE.md mandates a regular-read of parent `META/manifest.retention_pin_txid` so concurrent GC aborts the fork.
+4. **Truncate cleanup off-by-one for fully-above-EOF SHARDs.** `commit.rs:567` and `takeover.rs:142` use `>` instead of `>=`. A whole-shard truncate (e.g. `new_db_size_pages = 64`) leaves shard 1 alive.
+5. **PIDX/SHARD deletes during truncate use `clear`, not `COMPARE_AND_CLEAR`.** `commit.rs:272-274`. CLAUDE.md is explicit; this clobbers a concurrent compactor write.
+6. **`load_branch_ancestry` off-by-one against `MAX_FORK_DEPTH`.** Loop is `0..=MAX_FORK_DEPTH` (17 iterations); `derive_branch_at:534` caps with `>=`. End-to-end allows reading 17 levels.
+7. **Branch reap leaks parent refcount.** `gc/mod.rs:155-197 sweep_unreferenced_branch_tx` clears the branch keys but never decrements `branches_refcount_key` on the parent set in `derive_branch_at:591-595`.
+8. **Cold tier disabled when commit signaler is wired.** `db.rs:156-173 new_with_compaction_signaler` passes `cold_tier: None`; the only caller is the new envoy path (`pegboard-envoy/src/ws_to_tunnel_task.rs:730-756`). `read.rs:495` short-circuits when cold_tier is None — fork descendants and historical reads break under the new path.
+
+### Compactor workflow
+
+9. **S3 leak when cold publish rejects after upload.** `workflows/compaction.rs:868-895` clears `active_cold_job` without GC'ing the already-uploaded objects. `schedule_stale_cold_output_cleanup` (line 909) only fires for *different* active jobs, not for failed publish of the same job.
+10. **Repair reclaim activities skip the lifecycle generation fence.** `compaction.rs:1652, 1666` set `base_lifecycle_generation: 0`; `cleanup_repair_fdb_outputs_tx` (line 2462) and `DeleteOrphanColdObjects` (line 3022) never call `branch_record_is_live_at_generation`. A repair queued before destroy can run during/after destroy and delete S3 objects.
+11. **Force-compaction reclaim flag silently dropped.** `plan_reclaim_job` takes `_force: bool` and never reads it (`compaction.rs:3868-3873`). Forced reclaim with no actionable lag falls through to `force_noop_reasons`. Hot/cold honor force; reclaim does not.
+12. **Staged-hot install no-op when only DB-pin coverage was staged.** `compaction.rs:1957-1968` requires every PIDX row to be covered by `latest_staged_shards`, but `selected_hot_coverage_txids` only covers DB pins above `hot_watermark_txid`. Pin at `head_txid` only with no intermediate pins → install rejects with "missing staged hot shard".
+13. **`mix_fingerprint` is a hand-rolled non-cryptographic combiner** (`compaction.rs:4058-4066`) used as the OCC fence on `CompactionInputFingerprint`. `sha2::Digest` is already imported (line 11). Collision-resistant hashing matters here because the fingerprint guards "active job identity" through the publish path.
+14. **Reclaim commit-prefix scan uses `continue` instead of `break`** (`compaction.rs:3559-3572`). Txids are ascending big-endian; the rest of the scan is wasted on every reclaim batch.
+
+---
+
+## Convention Violations
+
+- **`Db` cache fields use `parking_lot::Mutex`** (`db.rs:107-123, 193-203`). All called from async contexts — no forced-sync requirement. Should be `tokio::sync::Mutex`. The `Mutex<scc::HashMap>` wrapping is doubly wrong — drop the outer mutex.
+- **Tracing logs in `workflows/compaction.rs` lack the `actor_id` field** the engine convention requires (e.g. `2503-2511, 3064-3074, 3084-3093`); only `database_branch_id` is included.
+- **`anyhow::anyhow!` macro** sprinkled across `commit.rs:677, 724, 738`, `quota.rs:60, 82`, `burst_mode.rs:80`, `takeover.rs:174-232`. Convention is `.context()` / `Error::msg`.
+- **`S3ColdTier::get_object` matches NoSuchKey by `err.to_string().contains("NoSuchKey")`** (`cold_tier/mod.rs:311`). Use the typed `is_no_such_key()` SDK API.
+- **`FaultyColdTier` records metrics under hardcoded `"unknown"` node_id** (`cold_tier/mod.rs:13, 480`).
+- **Test hook `lazy_static!` + `parking_lot::Mutex<Option<…>>`** in `workflows/compaction.rs:51-54` is gated by `cfg(debug_assertions)` rather than `cfg(test)` — shipped in dev builds.
+- **Inline `#[cfg(test)] mod tests` in `conveyer/types.rs:1123`** (not feature-gated). Move to `tests/`.
+
+---
+
+## Test Coverage
+
+**Strengths:** Real UDB via `test_db()`, `FilesystemColdTier` for cold, fault injection via `FaultyColdTier` + custom wrapper tiers. Force-compaction tests correctly wait on durable `force_compaction_results` (`workflow_compaction_skeletons.rs:1502, 1562, 1638, 1712, 1826`). OCC race tests use clean atomic-bool-guarded retry hooks (`fork_database.rs:99`, `fork_namespace.rs:145`).
+
+**Gaps:**
+- **No test for the workflow `Pending → Ready` flip on a pinned bookmark.** Given finding #1, this contract isn't asserted anywhere — neither the success path nor the `Pending → Failed` path on cold upload failure.
+- **No test for crash between cold pin upload and the manager's bookmark-flip publish.**
+- **No test for manager workflow generation bump mid-active-job** — only individual companion-side rejections (`workflow_compaction_skeletons.rs:1960, 2025, 2778, 3007`).
+- **No test for hot OCC abort when a concurrent commit lands between hot-stage read and manager install.**
+
+**Quality concerns:**
+- **`conveyer_commit.rs:760`** does a 1ms wall-clock absence check. Replace with explicit yield/notify.
+- **17 polling helpers in `workflow_compaction_skeletons.rs:125-605`** each implement their own `loop { sleep 25ms }` waiting for durable workflow state. Not retry-til-success masks (each polls a real state condition), but real-clock and duplicative — a single deterministic helper would shave ~250 LoC and CI time.
+- **`fork_common/mod.rs:157-165 assert_storage_error`** uses `err.downcast_ref` (top-level only), while `bookmarks.rs:114` and `gc_pin_recompute_under_bookmark_delete_race.rs:16` use `err.chain().any(...)`. Inconsistent — context-wrapped errors silently pass.
+- **Helper sprawl**: `fault_common`, `fork_common`, `bookmarks.rs:30`, `takeover.rs:13` all redefine `test_db`, `make_db`, `read_value` instead of sharing a single helper module.
+
+---
+
+## Notable Strengths
+
+- `BookmarkStr` newtype validates the 33-char wire format at construction *and* deserialization (`types.rs:69-118`). `MutationType::ByteMin` correctly applied for branch pin atomic-min everywhere it should be.
+- `read.rs` correctly caps ancestor PIDX reads by per-source `versionstamp_cap`, falls through to latest SHARD ≤ cap, and disables the PIDX cache when the read plan has multiple branches (`read.rs:144-147, 174, 285-294`).
+- Cold object retire → grace → `DeleteIssued` → S3 delete → cleanup-Deleted sequence in the workflow (`compaction.rs:4715-4759`) faithfully implements the "completed retired record so the key cannot be republished" invariant.
+- `resolve_namespace_fork_pins` proof walk (`compaction.rs:3214-3394`) materializes `DB_PIN(NamespaceFork)` records before reclaim and treats missing/ambiguous proof as a retention blocker, matching the constraint.

.agent/notes/depot-tier-test-matrix-results.md

@@ -0,0 +1,51 @@
+# Depot Tier Test Matrix Results
+
+Run at: 2026-05-01T15:28:56-07:00
+
+Command:
+
+```bash
+cargo test -p depot
+```
+
+Result: passed.
+
+Summary:
+
+```text
+lib unit tests: 28 passed
+burst_mode: 1 passed
+cold_tier: 4 passed
+conveyer_branch: 19 passed
+conveyer_commit: 15 passed
+conveyer_compaction_payloads: 7 passed
+conveyer_constants: 1 passed
+conveyer_error: 2 passed
+conveyer_keys: 13 passed
+conveyer_page_index: 4 passed
+conveyer_pitr_interval: 2 passed
+conveyer_policy: 4 passed
+conveyer_quota: 4 passed
+conveyer_read: 19 passed
+conveyer_restore_point: 26 passed
+debug: 3 passed
+fork_bucket: 4 passed
+fork_common_helpers: 1 passed
+fork_database: 6 passed
+gc: 4 passed
+gc_pin_recompute_under_restore_point_delete_race: 1 passed
+list_databases: 3 passed
+restore_points: 5 passed
+takeover: 1 passed
+workflow_compaction_payloads: 1 passed
+workflow_compaction_skeletons: 69 passed
+doc tests: 0 passed
+```
+
+Exit status: `0`
+
+Notes:
+
+- The first combined run failed to compile because existing Depot source edits had dropped `anyhow::Error` from `conveyer/commit/helpers.rs` and borrowed temporary `tx.informal()` handles across `try_join!` in `burst_mode.rs`.
+- Those compile blockers were fixed narrowly, then the command above passed.
+- `workflow_compaction_skeletons.rs` now runs its tier-agnostic manager, hot compaction, PITR, and reclaim cases through a local disabled/filesystem workflow matrix. Cold-only and cold-disabled assertions remain single-mode by design.

.agent/notes/driver-test-progress.md

@@ -1,91 +1,32 @@
 # Driver Test Suite Progress
 
-Started: 2026-04-26T14:05:00-07:00
+Started: 2026-05-01
 Config: registry (static), client type (http), encoding (bare)
+Scope: DB driver tests only
 
-## Fast Tests
+## DB Tests
 
-- [x] manager-driver | Manager Driver Tests
-- [x] actor-conn | Actor Connection Tests
-- [x] actor-conn-state | Actor Connection State Tests
-- [x] conn-error-serialization | Connection Error Serialization Tests
-- [x] actor-destroy | Actor Destroy Tests
-- [x] request-access | Request Access in Lifecycle Hooks
-- [x] actor-handle | Actor Handle Tests
-- [x] action-features | Action Features
-- [x] access-control | access control
-- [x] actor-vars | Actor Variables
-- [x] actor-metadata | Actor Metadata Tests
-- [x] actor-onstatechange | Actor onStateChange Tests
-- [ ] actor-db | Actor Database
-- [ ] actor-db-raw | Actor Database Raw Tests
-- [ ] actor-db-init-order | Actor DB Init Order Tests
-- [ ] actor-workflow | Actor Workflow Tests
-- [ ] actor-error-handling | Actor Error Handling Tests
-- [ ] actor-queue | Actor Queue Tests
-- [ ] actor-kv | Actor KV Tests
-- [ ] actor-stateless | Actor Stateless Tests
-- [ ] raw-http | raw http
-- [ ] raw-http-request-properties | raw http request properties
-- [ ] raw-websocket | raw websocket
-- [ ] actor-inspector | Actor Inspector Tests
-- [ ] gateway-query-url | Gateway Query URL Tests
-- [ ] actor-db-pragma-migration | Actor Database Pragma Migration
-- [ ] actor-state-zod-coercion | Actor State Zod Coercion
-- [ ] actor-save-state | Actor Save State Tests
-- [ ] actor-conn-status | Connection Status Changes
-- [ ] gateway-routing | Gateway Routing
-- [ ] lifecycle-hooks | Lifecycle Hooks
-- [ ] serverless-handler | Serverless Handler Tests
-
-## Slow Tests
-
-- [ ] actor-state | Actor State Tests
-- [ ] actor-schedule | Actor Schedule Tests
-- [ ] actor-sleep | Actor Sleep Tests
-- [ ] actor-sleep-db | Actor Sleep Database Tests
-- [ ] actor-lifecycle | Actor Lifecycle Tests
-- [ ] actor-conn-hibernation | Actor Connection Hibernation Tests
-- [ ] actor-run | Actor Run Tests
-- [ ] hibernatable-websocket-protocol | hibernatable websocket protocol
-- [ ] actor-db-stress | Actor Database Stress Tests
-
-## Excluded
-
-- [ ] actor-agent-os | Actor agentOS Tests (skip unless explicitly requested)
+- [x] actor-db | Actor Database
+- [x] actor-db-raw | Actor Database Raw Tests
+- [x] actor-db-pragma-migration | Actor Database Pragma Migration
+- [x] actor-sleep-db | Actor Sleep Database Tests
+- [x] actor-db-stress | Actor Database Stress Tests
+- [x] actor-db-init-order | Actor DB Init Order
 
 ## Log
-
-- 2026-04-26T14:06:57-07:00 manager-driver: PASS
-
-- 2026-04-26T14:07:27-07:00 actor-conn: PASS
-
-- 2026-04-26T14:07:37-07:00 actor-conn-state: PASS
-
-- 2026-04-26T14:07:42-07:00 conn-error-serialization: PASS
-
-- 2026-04-26T14:08:14-07:00 actor-destroy: PASS
-
-- 2026-04-26T14:08:19-07:00 request-access: PASS
-
-- 2026-04-26T14:08:31-07:00 actor-handle: PASS
-
-- 2026-04-26T14:08:31-07:00 action-features: PASS
-
-- 2026-04-26T14:08:46-07:00 access-control: PASS
-
-- 2026-04-26T14:08:51-07:00 actor-vars: PASS
-
-- 2026-04-26T14:08:58-07:00 actor-metadata: PASS
-
-- 2026-04-26T14:08:59-07:00 actor-onstatechange: PASS
-
-- 2026-04-26T14:10:59-07:00 actor-db: FAIL (exit 124)
-
-- 2026-04-26T14:12:00-07:00 runner: stale suite-description filters found for action-features, actor-onstatechange, actor-db, gateway-query-url, and likely other renamed suites; switching to per-file bare filter.
-
-- 2026-04-26T14:12:54-07:00 action-features: PASS (bare file filter)
-
-- 2026-04-26T14:12:59-07:00 actor-onstatechange: PASS (bare file filter)
-
-- 2026-04-26T14:17:33-07:00 actor-db: FAIL (exit 1, bare file filter)
+- 2026-05-01 12:45:05 PDT actor-db: FAIL - 4 failures in static/bare run. First failing test reproduced standalone: `persists across sleep and wake cycles` returned count 0 instead of 1 after sleep/wake.
+- 2026-05-01 13:02:09 PDT actor-db: PASS (13 passed, 26 skipped, 25.4s). Fixed VFS persisted page-1 bootstrap, hot-only sparse page reads, and actor2 serverful reallocate transition ordering.
+- 2026-05-01 13:02:30 PDT actor-db-raw: PASS (5 passed, 10 skipped, 4.5s).
+- 2026-05-01 13:02:52 PDT actor-db-pragma-migration: PASS (4 passed, 8 skipped, 4.0s).
+- 2026-05-01 13:10:52 PDT actor-sleep-db: PASS (14 passed, 58 skipped, 59.1s). Fixed sleep DB fixture hold behavior and made sqlite cleanup terminal for stale actor-context instances.
+- 2026-05-01 13:11:48 PDT actor-db-stress: PASS (3 passed, 27.8s).
+- 2026-05-01 13:12:41 PDT actor-db-init-order: PASS (6 passed, 12 skipped, 7.4s).
+- 2026-05-01 13:13:02 PDT DB TESTS COMPLETE - 6/6 DB file groups passed for static/bare.
+- 2026-05-01 14:22:27 PDT DB TESTS RERUN STARTED - static/bare.
+- 2026-05-01 14:23:06 PDT actor-db rerun: PASS (13 passed, 26 skipped, 23.7s).
+- 2026-05-01 14:23:25 PDT actor-db-raw rerun: PASS (5 passed, 10 skipped, 5.3s).
+- 2026-05-01 14:24:37 PDT actor-db-pragma-migration rerun: PASS (4 passed, 8 skipped, 53.4s).
+- 2026-05-01 14:25:56 PDT actor-sleep-db rerun: PASS (14 passed, 58 skipped, 64.0s).
+- 2026-05-01 14:27:04 PDT actor-db-stress rerun: PASS (3 passed, 28.7s).
+- 2026-05-01 14:28:00 PDT actor-db-init-order rerun: PASS (6 passed, 12 skipped, 7.9s).
+- 2026-05-01 14:28:04 PDT DB TESTS RERUN COMPLETE - 6/6 DB file groups passed for static/bare.

.agent/notes/rivetkit-core-walkthrough.md

@@ -257,7 +257,7 @@ File tags: `0x00` main DB, `0x01` rollback journal, `0x02` WAL, `0x03` SHM. The
 
 **v2 slow path.** Large commits that don't fit the one-shot path encode delta blocks as full LTX v3 frames and stuff them directly under the DELTA chunk keys. There is no `/STAGE` prefix, no fixed one-chunk-per-page mapping. A chunk key may contain a raw 4 KiB page *or* an LTX frame; the v3 decoder handles both.
 
-**The parity invariant.** The native Rust VFS and the WASM TypeScript VFS (`rivetkit-typescript/packages/sqlite-wasm/src/vfs.ts` + `kv.ts`) must be byte-for-byte identical: same chunk size, same key encoding, same PRAGMA settings, same delete-range strategy for truncate, same journal mode. When you change one, change the other in the same commit. A database written by the Rust VFS must be readable by the WASM VFS.
+**Native-only.** The VFS lives only in `rivetkit-rust/packages/rivetkit-sqlite/src/vfs.rs`. There is no runtime WASM/TypeScript VFS — the `@rivetkit/sqlite-wasm` npm package is deprecated and the Rust crate is statically linked into `@rivetkit/rivetkit-napi` via `libsqlite3-sys`. Per-VFS rules (4 KiB chunks, `journal_mode=DELETE`, `locking_mode=EXCLUSIVE`, `auto_vacuum=NONE`) live in this one source.
 
 ---
 
@@ -393,7 +393,7 @@ Consolidated from the above:
 1. KV internal prefixes (`[1]`, `[2]*`, `[5]*`, `[0x08]*`) are reserved. No runtime enforcement. User writes into them corrupt the actor.
 2. `enqueue_and_wait` completion waits ignore the actor abort token. Breaking this breaks hibernation.
 3. Queue metadata rebuilds by full-scan on decode failure. Slow, safe, never lose messages.
-4. Native SQLite VFS and WASM SQLite VFS must match byte-for-byte. Chunk size, key layout, PRAGMAs, truncate strategy, journal mode.
+4. SQLite VFS is native-only (`rivetkit-rust/packages/rivetkit-sqlite/src/vfs.rs`). Chunk size, key layout, PRAGMAs, truncate strategy, journal mode all live here.
 5. Raw `onRequest` HTTP bypasses message-size limits. Action and queue routes do not.
 6. Static native actor HTTP flows through `RegistryDispatcher::handle_fetch`, not `actor/event.rs`. Sleep-timer fixes need both entry points.
 7. WebSocket message/close callbacks run inline under the callback guard, not as dispatch events.