feat: sqlite v2

Author: NathanFlurry

.agent/research/sqlite/sqlite-vfs-ltx-redesign.md

@@ -0,0 +1,44 @@
+{
+  "capturedAt": "2026-04-16T03:05:23.184Z",
+  "vfsVersion": "v1",
+  "source": "examples/sqlite-raw/scripts/benchmark.ts",
+  "pageSizeBytes": 4096,
+  "environment": {
+    "benchmarkHarness": "examples/sqlite-raw wrapper over rivetkit-sqlite-native/examples/v1_baseline_bench.rs",
+    "rttMs": 0,
+    "storage": "in-memory SqliteKv benchmark driver exercising the v1 native VFS",
+    "platform": "linux",
+    "release": "6.1.0-41-amd64",
+    "arch": "x64",
+    "cpuModel": "12th Gen Intel(R) Core(TM) i7-12700KF",
+    "cpuCount": 20,
+    "totalMemoryGiB": 62.56
+  },
+  "workloads": [
+    {
+      "name": "1 MiB insert",
+      "latencyMs": 3.614,
+      "roundTrips": 298
+    },
+    {
+      "name": "10 MiB insert",
+      "latencyMs": 25.892,
+      "roundTrips": 2606
+    },
+    {
+      "name": "hot-row update",
+      "latencyMs": 1.036,
+      "roundTrips": 109
+    },
+    {
+      "name": "cold read",
+      "latencyMs": 1.729,
+      "roundTrips": 228
+    },
+    {
+      "name": "mixed read/write",
+      "latencyMs": 0.824,
+      "roundTrips": 62
+    }
+  ]
+}

.agent/research/sqlite/v1-baseline-bench.json

@@ -0,0 +1 @@
+{"sessionId":"cb4dbb44-01ef-4eef-91a8-ff5ad2f3e6fe","pid":1948414,"acquiredAt":1776308226245}

.claude/scheduled_tasks.lock

@@ -213,12 +213,16 @@ When the user asks to track something in a note, store it in `.agent/notes/` by
 
 ## Architecture
 
+### Deprecated Packages
+- `engine/packages/pegboard-runner/` and associated TypeScript "runner" packages (`engine/sdks/typescript/runner`, `rivetkit-typescript/packages/engine-runner/`) and runner workflows are deprecated. All new actor hosting work targets `engine/packages/pegboard-envoy/` exclusively. Do not add features to or fix bugs in the deprecated runner path.
+
 ### Monorepo Structure
 - This is a Rust workspace-based monorepo for Rivet with the following key packages and components:
 
 - **Core Engine** (`packages/core/engine/`) - Main orchestration service that coordinates all operations
 - **Workflow Engine** (`packages/common/gasoline/`) - Handles complex multi-step operations with reliability and observability
 - **Pegboard** (`packages/core/pegboard/`) - Actor/server lifecycle management system
+- **Pegboard Envoy** (`engine/packages/pegboard-envoy/`) - The active actor-to-engine bridge. All new actor hosting work goes here.
 - **Common Packages** (`/packages/common/`) - Foundation utilities, database connections, caching, metrics, logging, health checks, workflow engine core
 - **Core Packages** (`/packages/core/`) - Main engine executable, Pegboard actor orchestration, workflow workers
 - **Shared Libraries** (`shared/{language}/{package}/`) - Libraries shared between the engine and rivetkit (e.g., `shared/typescript/virtual-websocket/`)
@@ -290,6 +294,12 @@ let error_with_meta = ApiRateLimited { limit: 100, reset_at: 1234567890 }.build(
 **Native SQLite & KV Channel**
 - RivetKit SQLite is served by `@rivetkit/rivetkit-native`. Do not reintroduce SQLite-over-KV or WebAssembly SQLite paths in the TypeScript runtime.
 - The Rust KV-backed SQLite implementation still lives in `rivetkit-typescript/packages/sqlite-native/src/`. When changing its on-disk or KV layout, update the internal data-channel spec in the same change.
+- SQLite v2 slow-path staging writes encoded LTX bytes directly under DELTA chunk keys. Do not expect `/STAGE` keys or a fixed one-chunk-per-page mapping in tests or recovery code.
+- The native VFS uses the same 4 KiB chunk layout and KV key encoding as the WASM VFS. Data is compatible between backends.
+- **The native Rust VFS and the WASM TypeScript VFS must match 1:1.** This includes: KV key layout and encoding, chunk size, PRAGMA settings, VFS callback-to-KV-operation mapping, delete/truncate strategy (both must use `deleteRange`), and journal mode. When changing any VFS behavior in one implementation, update the other. The relevant files are:
+  - Native: `rivetkit-typescript/packages/sqlite-native/src/vfs.rs`, `kv.rs`
+  - WASM: `rivetkit-typescript/packages/sqlite-wasm/src/vfs.ts`, `kv.ts`
+- SQLite VFS v2 storage keys use literal ASCII path segments under the `0x02` subspace prefix with big-endian numeric suffixes so `scan_prefix` and `BTreeMap` ordering stay numerically correct.
 - Full spec: `docs-internal/engine/NATIVE_SQLITE_DATA_CHANNEL.md`
 
 **Inspector HTTP API**

CLAUDE.md

@@ -38,6 +38,7 @@ members = [
   "engine/packages/runner-protocol",
   "engine/packages/runtime",
   "engine/packages/service-manager",
+  "engine/packages/sqlite-storage",
   "engine/packages/telemetry",
   "engine/packages/test-deps",
   "engine/packages/test-deps-docker",
@@ -108,6 +109,7 @@ members = [
   once_cell = "1.20.2"
   opentelemetry-appender-tracing = "0.28.1"
   papaya = "0.2.1"
+  parking_lot = "0.12"
   pest_derive = "2.7"
   portpicker = "0.1"
   prettyplease = "0.2"
@@ -460,6 +462,9 @@ members = [
     [workspace.dependencies.rivet-runtime]
     path = "engine/packages/runtime"
 
+    [workspace.dependencies.sqlite-storage]
+    path = "engine/packages/sqlite-storage"
+
     [workspace.dependencies.rivet-service-manager]
     path = "engine/packages/service-manager"
 

Cargo.lock

@@ -0,0 +1,30 @@
+# SQLite Metrics
+
+- `sqlite_commit_phase_duration_seconds{phase,path}`: Engine-side histogram for commit request phases. `path` is `fast` or `slow`. `phase` is `decode_request`, `meta_read`, `ltx_encode`, `pidx_read`, `udb_write`, or `response_build`.
+- `sqlite_commit_stage_phase_duration_seconds{phase}`: Engine-side histogram for staged commit uploads. `phase` is `decode`, `stage_encode`, or `udb_write`.
+- `sqlite_commit_finalize_phase_duration_seconds{phase}`: Engine-side histogram for staged commit finalize work. `phase` is `stage_promote`, `pidx_write`, or `meta_write`.
+- `sqlite_commit_dirty_page_count{path}`: Histogram of dirty page counts per commit path.
+- `sqlite_commit_dirty_bytes{path}`: Histogram of raw dirty-page bytes per commit path.
+- `sqlite_udb_ops_per_commit{path}`: Histogram of UniversalDB operations per commit path.
+- `sqlite_commit_envoy_dispatch_duration_seconds`: Pegboard-envoy histogram for websocket frame arrival to `sqlite-storage` dispatch.
+- `sqlite_commit_envoy_response_duration_seconds`: Pegboard-envoy histogram for `sqlite-storage` return to websocket response send.
+- `sqlite_commit_phases`: Actor inspector labeled timing metric exposed from `/inspector/metrics`. Values are `request_build`, `serialize`, `transport`, and `state_update`.
+
+## Scrape Points
+
+- Engine and pegboard-envoy Prometheus metrics come from the shared `/metrics` endpoint on port `6430`.
+- Actor-local commit timings come from `GET /inspector/metrics` on the actor gateway route.
+
+## Tracing
+
+- Set `RUST_LOG=sqlite_storage=debug,pegboard_envoy=debug,sqlite_v2_vfs=debug` to emit per-commit phase spans and VFS phase logs.
+- `sqlite-storage` commit handlers use debug spans for the high-level request and sub-phase work.
+- The VFS logs request-build, serialize, transport, and state-update timings after each successful commit.
+
+## Diagnosis
+
+- High `decode_request` or `sqlite_commit_envoy_dispatch_duration_seconds` usually means envoy-side validation or actor lookup is slow before storage work starts.
+- High `meta_read` or `pidx_read` points at UniversalDB read pressure or cache misses.
+- High `ltx_encode` means commit encoding and compression are doing real work. Check dirty page counts and raw dirty bytes together.
+- High `udb_write`, `meta_write`, or `sqlite_commit_envoy_response_duration_seconds` points at write-path latency after encode.
+- A healthy actor should show non-zero `sqlite_commit_phases` totals after commits in `/inspector/metrics`. If SQL runs but those timings stay zero, the native VFS metrics path is broken.