feat(depot-client): add vfs staging cache ttl

Author: NathanFlurry

.agent/specs/sqlite-vfs-staging-cache-ttl.md

@@ -0,0 +1,81 @@
+# SQLite VFS Staging Cache TTL Plan
+
+Date: 2026-05-03
+
+This plan changes the SQLite VFS page cache from a broad second-level pager cache into a short-lived staging cache for speculative pages. Demand pages fetched for `xRead` should be handed to SQLite and then forgotten by the VFS unless a small explicit exception applies.
+
+## Goals
+
+- Avoid retaining pages in VFS memory after SQLite has already received them through `xRead`.
+- Keep startup preload and read-ahead useful by retaining speculative pages briefly.
+- Evict speculative pages on first successful target read so TTL is only the fallback for unused preloads.
+- Keep lazy loading correct when all cache and preload features are disabled.
+- Preserve page 1 handling unless we explicitly decide to make it configurable later.
+
+## Non-Goals
+
+- Do not change the remote `get_pages` protocol.
+- Do not change SQLite pager settings.
+- Do not add read pools back.
+- Do not implement persisted preload hints in this branch.
+
+## Current Behavior
+
+- `resolve_pages` classifies fetched pages as `Target` when SQLite requested them and `Prefetch` when they were predicted.
+- `fetch_initial_pages_for_registration` seeds startup pages as `Startup`.
+- `should_cache_page` allows target, prefetch, and startup caching based on `SqliteVfsPageCacheMode`.
+- Page 1 is always cacheable.
+- Early protected pages live in `protected_page_cache`, which is an `scc::HashMap` with no TTL.
+
+## Proposed Behavior
+
+- Target pages should not be inserted into the VFS page cache by default.
+- Target reads should remove the read pages from the speculative cache after bytes are copied to the caller.
+- Prefetch pages should be inserted into a TTL cache.
+- Startup preload pages should be inserted into the same TTL cache.
+- Page 1 should remain retained for now because it carries page size and database size metadata and is already special-cased throughout the VFS.
+- Protected cache should no longer protect arbitrary early speculative pages forever. It should either be limited to page 1 or removed in favor of the TTL cache.
+
+## Configuration
+
+- Add `RIVETKIT_SQLITE_OPT_VFS_STAGING_CACHE_TTL_MS`.
+- Default to a short TTL such as `30000` ms.
+- A value of `0` disables speculative retention while preserving lazy target fetches.
+- Keep `RIVETKIT_SQLITE_OPT_VFS_PAGE_CACHE_MODE=off` as the stronger kill switch for all non-page-1 VFS caching.
+- Consider narrowing `RIVETKIT_SQLITE_OPT_VFS_PROTECTED_CACHE_PAGES` so it does not pin preload/read-ahead pages indefinitely.
+
+## Implementation Plan
+
+1. Extend `SqliteOptimizationFlags` and `VfsConfig` with a bounded staging TTL field.
+2. Build `page_cache` with `time_to_live(Duration::from_millis(ttl_ms))` when TTL is nonzero.
+3. Split cache insertion semantics so `PageCacheInsertKind::Target` is not retained by default.
+4. Add an explicit `evict_pages_after_target_read` helper that removes read pages from both normal and protected speculative caches.
+5. Call that helper after `io_read` copies returned bytes into SQLite's buffer.
+6. Rework `protected_page_cache` so it cannot pin non-page-1 speculative pages forever.
+7. Keep `seed_main_page` behavior intact for page 1 metadata.
+8. Update metrics naming only if needed. `page_cache_entries` can continue to report retained VFS entries.
+
+## Expected Cache Matrix
+
+| Page source | Retained after fetch | Evicted on target read | TTL fallback |
+| --- | --- | --- | --- |
+| Target `xRead` miss | No | Not needed | No |
+| Read-ahead prefetch | Yes | Yes | Yes |
+| Startup preload | Yes | Yes | Yes |
+| Page 1 | Yes | No | No |
+| Dirty write buffer | Existing behavior | Existing behavior | No |
+
+## Tests
+
+- Add a VFS test proving a target read miss does not increase retained VFS cache entries.
+- Add a VFS test proving prefetch pages are retained before use and removed after target read.
+- Add a VFS test proving startup preload pages are retained briefly and removed after target read.
+- Add a VFS test proving `VFS_STAGING_CACHE_TTL_MS=0` still lazily fetches pages.
+- Add a VFS test proving `VFS_PAGE_CACHE_MODE=off` still lazily fetches pages and does not retain non-page-1 pages.
+- If practical, use Tokio time pause/advance to verify TTL expiry deterministically instead of sleeping.
+
+## Open Questions
+
+- Should page 1 stay permanently retained, or should strict low-memory mode make even page 1 reloadable?
+- Should target retention remain available as an explicit benchmark mode, or should we remove target caching from the shipped matrix?
+- Should `VFS_PROTECTED_CACHE_PAGES` be deprecated in favor of page-1-only retention?

docs-internal/engine/SQLITE_OPTIMIZATIONS.md

@@ -11,7 +11,8 @@ Range page-read protocol details live in `.agent/specs/sqlite-range-page-read-pr
 ## Existing Optimizations
 
 - Actor startup can preload SQLite VFS pages through `OpenConfig.preload_pgnos`, `OpenConfig.preload_ranges`, and persisted `/PRELOAD_HINTS`; first pages, hint mechanisms, and the preload byte budget are configured through central SQLite optimization flags.
-- The VFS keeps an in-memory page cache seeded from `sqlite_startup_data.preloaded_pages`; cache behavior is selected with `RIVETKIT_SQLITE_OPT_VFS_PAGE_CACHE_MODE=off|target|startup|prefetch|all`, with capacity and protected-cache budget configured separately.
+- The VFS keeps a short-lived staging cache for speculative startup preload and read-ahead pages. Direct target pages fetched for `xRead` are not retained after SQLite receives them; speculative entries are invalidated on first target read and also expire through `RIVETKIT_SQLITE_OPT_VFS_STAGING_CACHE_TTL_MS`.
+- VFS staging cache behavior is selected with `RIVETKIT_SQLITE_OPT_VFS_PAGE_CACHE_MODE=off|target|startup|prefetch|all`, with capacity and page-1 protected-cache behavior configured separately.
 - The VFS has speculative read-ahead selected with `RIVETKIT_SQLITE_OPT_READ_AHEAD_MODE=off|bounded|adaptive`; the default bounded budget is 64 pages, which reduced the cold-read benchmark from 1,249 to 368 VFS `get_pages` calls.
 - The VFS tracks bounded recent page hints as hot pages plus coalesced scan ranges; `NativeDatabase::snapshot_preload_hints()` exposes the in-memory plan for future flush wiring.
 - Actor Prometheus metrics expose VFS read counters, fetched bytes, cache hits/misses, and `get_pages` duration at `/gateway/<actor_id>/metrics`.

engine/packages/depot-client/src/optimization_flags.rs

@@ -22,6 +22,7 @@ pub const VFS_PAGE_CACHE_MODE_ENV: &str = "RIVETKIT_SQLITE_OPT_VFS_PAGE_CACHE_MO
 pub const VFS_PAGE_CACHE_CAPACITY_PAGES_ENV: &str =
 	"RIVETKIT_SQLITE_OPT_VFS_PAGE_CACHE_CAPACITY_PAGES";
 pub const VFS_PROTECTED_CACHE_PAGES_ENV: &str = "RIVETKIT_SQLITE_OPT_VFS_PROTECTED_CACHE_PAGES";
+pub const VFS_STAGING_CACHE_TTL_MS_ENV: &str = "RIVETKIT_SQLITE_OPT_VFS_STAGING_CACHE_TTL_MS";
 
 pub const DEFAULT_STARTUP_PRELOAD_MAX_BYTES: usize = 1024 * 1024;
 pub const MAX_STARTUP_PRELOAD_MAX_BYTES: usize = 8 * 1024 * 1024;
@@ -31,6 +32,8 @@ pub const DEFAULT_VFS_PAGE_CACHE_CAPACITY_PAGES: u64 = 50_000;
 pub const MAX_VFS_PAGE_CACHE_CAPACITY_PAGES: u64 = 500_000;
 pub const DEFAULT_VFS_PROTECTED_CACHE_PAGES: usize = 512;
 pub const MAX_VFS_PROTECTED_CACHE_PAGES: usize = 8_192;
+pub const DEFAULT_VFS_STAGING_CACHE_TTL_MS: u64 = 30_000;
+pub const MAX_VFS_STAGING_CACHE_TTL_MS: u64 = 300_000;
 
 #[derive(Debug, Clone, Copy, PartialEq, Eq)]
 pub enum SqliteReadAheadMode {
@@ -102,6 +105,7 @@ pub struct SqliteOptimizationFlags {
 	pub vfs_page_cache_mode: SqliteVfsPageCacheMode,
 	pub vfs_page_cache_capacity_pages: u64,
 	pub vfs_protected_cache_pages: usize,
+	pub vfs_staging_cache_ttl_ms: u64,
 }
 
 impl Default for SqliteOptimizationFlags {
@@ -128,6 +132,7 @@ impl Default for SqliteOptimizationFlags {
 			vfs_page_cache_mode: SqliteVfsPageCacheMode::All,
 			vfs_page_cache_capacity_pages: DEFAULT_VFS_PAGE_CACHE_CAPACITY_PAGES,
 			vfs_protected_cache_pages: DEFAULT_VFS_PROTECTED_CACHE_PAGES,
+			vfs_staging_cache_ttl_ms: DEFAULT_VFS_STAGING_CACHE_TTL_MS,
 		}
 	}
 }
@@ -196,6 +201,11 @@ impl SqliteOptimizationFlags {
 				DEFAULT_VFS_PROTECTED_CACHE_PAGES,
 				MAX_VFS_PROTECTED_CACHE_PAGES,
 			),
+			vfs_staging_cache_ttl_ms: u64_bounded_by_default(
+				read_env(VFS_STAGING_CACHE_TTL_MS_ENV).as_deref(),
+				DEFAULT_VFS_STAGING_CACHE_TTL_MS,
+				MAX_VFS_STAGING_CACHE_TTL_MS,
+			),
 		}
 	}
 }
@@ -307,6 +317,7 @@ mod tests {
 			VFS_PAGE_CACHE_MODE_ENV => Some("off".to_string()),
 			VFS_PAGE_CACHE_CAPACITY_PAGES_ENV => Some("0".to_string()),
 			VFS_PROTECTED_CACHE_PAGES_ENV => Some("0".to_string()),
+			VFS_STAGING_CACHE_TTL_MS_ENV => Some("0".to_string()),
 			_ => None,
 		});
 
@@ -327,6 +338,7 @@ mod tests {
 		assert_eq!(flags.vfs_page_cache_mode, SqliteVfsPageCacheMode::Off);
 		assert_eq!(flags.vfs_page_cache_capacity_pages, 0);
 		assert_eq!(flags.vfs_protected_cache_pages, 0);
+		assert_eq!(flags.vfs_staging_cache_ttl_ms, 0);
 	}
 
 	#[test]
@@ -336,6 +348,7 @@ mod tests {
 			STARTUP_PRELOAD_FIRST_PAGE_COUNT_ENV => Some("nope".to_string()),
 			VFS_PAGE_CACHE_CAPACITY_PAGES_ENV => Some("invalid".to_string()),
 			VFS_PROTECTED_CACHE_PAGES_ENV => Some("invalid".to_string()),
+			VFS_STAGING_CACHE_TTL_MS_ENV => Some("invalid".to_string()),
 			_ => None,
 		});
 		assert_eq!(
@@ -354,6 +367,10 @@ mod tests {
 			invalid.vfs_protected_cache_pages,
 			DEFAULT_VFS_PROTECTED_CACHE_PAGES
 		);
+		assert_eq!(
+			invalid.vfs_staging_cache_ttl_ms,
+			DEFAULT_VFS_STAGING_CACHE_TTL_MS
+		);
 
 		let clamped = SqliteOptimizationFlags::from_env_reader(|key| match key {
 			STARTUP_PRELOAD_MAX_BYTES_ENV => Some((MAX_STARTUP_PRELOAD_MAX_BYTES + 1).to_string()),
@@ -364,6 +381,7 @@ mod tests {
 				Some((MAX_VFS_PAGE_CACHE_CAPACITY_PAGES + 1).to_string())
 			}
 			VFS_PROTECTED_CACHE_PAGES_ENV => Some((MAX_VFS_PROTECTED_CACHE_PAGES + 1).to_string()),
+			VFS_STAGING_CACHE_TTL_MS_ENV => Some((MAX_VFS_STAGING_CACHE_TTL_MS + 1).to_string()),
 			_ => None,
 		});
 		assert_eq!(
@@ -382,5 +400,9 @@ mod tests {
 			clamped.vfs_protected_cache_pages,
 			MAX_VFS_PROTECTED_CACHE_PAGES
 		);
+		assert_eq!(
+			clamped.vfs_staging_cache_ttl_ms,
+			MAX_VFS_STAGING_CACHE_TTL_MS
+		);
 	}
 }

engine/packages/depot-client/src/vfs.rs

@@ -134,6 +134,7 @@ pub struct VfsConfig {
 	pub cache_capacity_pages: u64,
 	pub protected_cache_pages: usize,
 	pub page_cache_mode: SqliteVfsPageCacheMode,
+	pub staging_cache_ttl_ms: u64,
 	pub prefetch_depth: usize,
 	pub adaptive_prefetch_depth: usize,
 	pub max_prefetch_bytes: usize,
@@ -169,11 +170,16 @@ impl VfsConfig {
 				0
 			},
 			protected_cache_pages: if caches_pages {
-				flags.vfs_protected_cache_pages
+				usize::from(flags.vfs_protected_cache_pages > 0)
 			} else {
 				0
 			},
 			page_cache_mode: flags.vfs_page_cache_mode,
+			staging_cache_ttl_ms: if caches_pages {
+				flags.vfs_staging_cache_ttl_ms
+			} else {
+				0
+			},
 			prefetch_depth: if flags.read_ahead {
 				DEFAULT_PREFETCH_DEPTH
 			} else {
@@ -790,9 +796,12 @@ fn push_coalesced_range(ranges: &mut VecDeque<VfsPreloadHintRange>, range: VfsPr
 
 impl VfsState {
 	fn new(config: &VfsConfig) -> Self {
-		let page_cache = Cache::builder()
-			.max_capacity(config.cache_capacity_pages)
-			.build();
+		let mut page_cache_builder = Cache::builder().max_capacity(config.cache_capacity_pages);
+		if config.staging_cache_ttl_ms > 0 {
+			page_cache_builder =
+				page_cache_builder.time_to_live(Duration::from_millis(config.staging_cache_ttl_ms));
+		}
+		let page_cache = page_cache_builder.build();
 		let mut state = Self {
 			db_size_pages: 1,
 			page_size: DEFAULT_PAGE_SIZE,
@@ -841,6 +850,13 @@ impl VfsState {
 			.or_else(|| self.page_cache.get(&pgno))
 	}
 
+	fn evict_target_read_pages(&self, pgnos: &[u32]) {
+		for pgno in pgnos.iter().copied().filter(|pgno| *pgno != 1) {
+			self.page_cache.invalidate(&pgno);
+			self.protected_page_cache.remove_sync(&pgno);
+		}
+	}
+
 	fn seed_page(&mut self, config: &VfsConfig, kind: PageCacheInsertKind, pgno: u32, page: Vec<u8>) {
 		if pgno == 1 {
 			self.seed_main_page(config, kind, page);
@@ -876,7 +892,7 @@ fn cache_page(
 	if !should_cache_page(config, kind, pgno) {
 		return;
 	}
-	if pgno <= config.protected_cache_pages as u32 {
+	if pgno == 1 && config.protected_cache_pages > 0 {
 		let _ = protected_page_cache.upsert_sync(pgno, bytes);
 	} else {
 		page_cache.insert(pgno, bytes);
@@ -887,8 +903,11 @@ fn should_cache_page(config: &VfsConfig, kind: PageCacheInsertKind, pgno: u32) -
 	if pgno == 1 {
 		return true;
 	}
+	if config.staging_cache_ttl_ms == 0 {
+		return false;
+	}
 	match kind {
-		PageCacheInsertKind::Target => config.page_cache_mode.caches_target_pages(),
+		PageCacheInsertKind::Target => false,
 		PageCacheInsertKind::Prefetch => config.page_cache_mode.caches_prefetched_pages(),
 		PageCacheInsertKind::Startup => config.page_cache_mode.caches_startup_preloaded_pages(),
 	}
@@ -2151,7 +2170,7 @@ unsafe extern "C" fn io_read(
 		}
 
 		buf.fill(0);
-		for pgno in requested_pages {
+		for pgno in requested_pages.iter().copied() {
 			let Some(Some(bytes)) = resolved.get(&pgno) else {
 				continue;
 			};
@@ -2167,6 +2186,7 @@ unsafe extern "C" fn io_read(
 			buf[dest_offset..dest_offset + copy_len]
 				.copy_from_slice(&bytes[page_offset..page_offset + copy_len]);
 		}
+		ctx.state.read().evict_target_read_pages(&requested_pages);
 
 		if i_offset as usize + i_amt as usize > file_size {
 			return SQLITE_IOERR_SHORT_READ;

engine/packages/depot-client/tests/inline/vfs.rs

@@ -21,8 +21,8 @@ use tokio::sync::OnceCell;
 
 use crate::optimization_flags::{
 	DEFAULT_STARTUP_PRELOAD_MAX_BYTES, DEFAULT_VFS_PAGE_CACHE_CAPACITY_PAGES,
-	DEFAULT_VFS_PROTECTED_CACHE_PAGES, SqliteOptimizationFlags, SqliteReadAheadMode,
-	SqliteVfsPageCacheMode,
+	DEFAULT_VFS_PROTECTED_CACHE_PAGES, DEFAULT_VFS_STAGING_CACHE_TTL_MS,
+	SqliteOptimizationFlags, SqliteReadAheadMode, SqliteVfsPageCacheMode,
 };
 use crate::query::{BindParam, ColumnValue};
 use crate::vfs::SqliteVfsMetrics;
@@ -55,6 +55,7 @@ fn vfs_config_wires_optimization_flags() {
 		vfs_page_cache_mode: SqliteVfsPageCacheMode::Startup,
 		vfs_page_cache_capacity_pages: DEFAULT_VFS_PAGE_CACHE_CAPACITY_PAGES / 2,
 		vfs_protected_cache_pages: DEFAULT_VFS_PROTECTED_CACHE_PAGES / 2,
+		vfs_staging_cache_ttl_ms: DEFAULT_VFS_STAGING_CACHE_TTL_MS / 2,
 	};
 
 	let config = VfsConfig::from_optimization_flags(flags);
@@ -65,7 +66,11 @@ fn vfs_config_wires_optimization_flags() {
 	);
 	assert_eq!(
 		config.protected_cache_pages,
-		DEFAULT_VFS_PROTECTED_CACHE_PAGES / 2
+		1
+	);
+	assert_eq!(
+		config.staging_cache_ttl_ms,
+		DEFAULT_VFS_STAGING_CACHE_TTL_MS / 2
 	);
 	assert_eq!(config.prefetch_depth, 16);
 	assert!(!config.adaptive_read_ahead);
@@ -141,6 +146,46 @@ fn startup_initial_pages_do_not_require_preload_hints_on_open() {
 	assert_eq!(loaded_pgnos, vec![1, 2, 3, 4]);
 }
 
+#[test]
+fn vfs_staging_cache_retains_only_speculative_pages() {
+	let config = VfsConfig {
+		page_cache_mode: SqliteVfsPageCacheMode::All,
+		staging_cache_ttl_ms: DEFAULT_VFS_STAGING_CACHE_TTL_MS,
+		..VfsConfig::default()
+	};
+	let mut state = VfsState::new(&config);
+
+	state.cache_page(&config, PageCacheInsertKind::Target, 2, vec![2; DEFAULT_PAGE_SIZE]);
+	assert!(state.cached_page(&config, 2).is_none());
+
+	state.cache_page(&config, PageCacheInsertKind::Prefetch, 3, vec![3; DEFAULT_PAGE_SIZE]);
+	state.cache_page(&config, PageCacheInsertKind::Startup, 4, vec![4; DEFAULT_PAGE_SIZE]);
+	assert!(state.cached_page(&config, 3).is_some());
+	assert!(state.cached_page(&config, 4).is_some());
+	assert!(state.protected_page_cache.read_sync(&3, |_, _| ()).is_none());
+
+	state.evict_target_read_pages(&[1, 3, 4]);
+	assert!(state.cached_page(&config, 1).is_some());
+	assert!(state.cached_page(&config, 3).is_none());
+	assert!(state.cached_page(&config, 4).is_none());
+}
+
+#[test]
+fn vfs_staging_cache_ttl_zero_disables_speculative_retention() {
+	let config = VfsConfig {
+		page_cache_mode: SqliteVfsPageCacheMode::All,
+		staging_cache_ttl_ms: 0,
+		..VfsConfig::default()
+	};
+	let mut state = VfsState::new(&config);
+
+	state.cache_page(&config, PageCacheInsertKind::Prefetch, 2, vec![2; DEFAULT_PAGE_SIZE]);
+	state.cache_page(&config, PageCacheInsertKind::Startup, 3, vec![3; DEFAULT_PAGE_SIZE]);
+	assert!(state.cached_page(&config, 1).is_some());
+	assert!(state.cached_page(&config, 2).is_none());
+	assert!(state.cached_page(&config, 3).is_none());
+}
+
 fn next_test_name(prefix: &str) -> String {
 	let id = TEST_ID.fetch_add(1, Ordering::Relaxed);
 	format!("{prefix}-{id}")