fix(runtime): sandbox budget = script CPU-time, not wall clock (ADR-0102 D1, #3295)

Phase 1 of #3275. The QuickJS sandbox now meters each hook/action invocation
against VM-active (CPU) time, not wall clock. Idle host-await time and a nested
hook's own execution (host-side, while the caller VM is parked) are no longer
charged to the caller — so a slow/loaded host or a deep nested-write chain can't
trip the budget while a script is merely waiting (the #3259 flake root cause). A
separate wall-clock ceiling (default 30s, max(ceiling, cpuBudget)) backstops a
body stuck on a never-settling host call.

- runner: slice-stopwatch every VM entry (evalCode / evalCodeAsync / each
  executePendingJobs); the interrupt handler and post-slice checks compare
  against the CPU budget and the wall deadline; distinct errors "exceeded CPU
  budget of Nms" vs "exceeded wall-clock ceiling of Nms while awaiting host
  calls". A mid-slice interrupt — which for an async body surfaces via __error,
  not pending.error — is mapped to the clean budget message at all three sites.
- new knobs: QuickJSScriptRunner wallCeilingMs + env OS_SANDBOX_WALL_CEILING_MS
  (resolveSandboxTimeoutMs gains a 'wallCeiling' kind). Precedence unchanged:
  explicit option > env > built-in default.
- fix: init sliceStart to `start`, not 0 — a 0 init made `now - sliceStart` the
  full epoch millis, firing the interrupt during installCtx and corrupting ctx
  marshalling (__input never set).
- tests: rewrite never-settling-host timeout tests to CPU-burn bodies (assert
  /CPU budget of Nms/) + dedicated small-wallCeilingMs runners for the genuine
  stuck-host cases (assert /wall-clock ceiling/); add the #3259 root-cause guard
  (idle host time resolves) and CPU-vs-ceiling coverage.
- nested-write*.integration: drop the explicit 10s budget — they pass at the
  stock 250ms now, the nested-charging-fix regression guard.
- docs: environment-variables.mdx CPU-time wording + OS_SANDBOX_WALL_CEILING_MS;
  ADR-0102 D1 → Accepted. Changeset carries the FROM→TO semantics note.

574 runtime tests + 26 types tests green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011iJLjqToxNv1aYP3syQtRp

Author: claude

.changeset/sandbox-cpu-budget.md

@@ -0,0 +1,39 @@
+---
+'@objectstack/runtime': minor
+'@objectstack/types': minor
+---
+
+feat(runtime): sandbox budget is script CPU-time, not wall clock (ADR-0102 D1, #3295)
+
+The QuickJS sandbox now meters each hook/action invocation against how much
+**VM-active (CPU) time** the body burns, not wall clock. Idle host-await time and
+a nested hook's own execution (which runs host-side while the caller's VM is
+parked) are no longer charged to the caller — so a slow/loaded host or a deep
+nested-write chain can't trip the budget while a script is merely waiting (the
+root cause of the #3259 CI flake). A separate, generous **wall-clock ceiling**
+(default 30s, `max(ceiling, cpuBudget)`) remains as the backstop for a body stuck
+on a host call that never settles.
+
+What changes for consumers (behaviour, not API signatures):
+
+- **Meaning of the timeout knobs.** `body.timeoutMs`, the `hookTimeoutMs` /
+  `actionTimeoutMs` runner options, and `OS_SANDBOX_HOOK_TIMEOUT_MS` /
+  `OS_SANDBOX_ACTION_TIMEOUT_MS` keep their **names, defaults (250ms / 5000ms),
+  and precedence** — but now bound CPU-time instead of wall-clock. In practice
+  this only *loosens* legitimate slow/nested work; a runaway synchronous script
+  is still cut at the same budget.
+- **Error messages.** `exceeded timeout of Nms` → either `exceeded CPU budget of
+  Nms` (script burned its CPU budget) or `exceeded wall-clock ceiling of Nms
+  while awaiting host calls` (stuck on a never-settling host call). Update any
+  code/tests matching the old string.
+
+New knobs (additive):
+
+- `QuickJSScriptRunner` option `wallCeilingMs` and env `OS_SANDBOX_WALL_CEILING_MS`
+  — tune the wall ceiling (explicit option › env › 30s).
+- `resolveSandboxTimeoutMs` (`@objectstack/types`) gains a `'wallCeiling'` kind.
+
+Also fixes a latent init bug in the new accounting where the interrupt handler
+could fire during `installCtx` and corrupt ctx marshalling. The nested-write
+integration suites now run at the stock 250ms budget (previously forced to 10s),
+which is itself the regression guard for the nested-charging fix.

content/docs/deployment/environment-variables.mdx

@@ -331,8 +331,9 @@ the hosted ObjectOS Cloud control plane.
 | `OS_ENV_CACHE_TTL_MS` | number | `300000` | Env-record cache TTL in ms. |
 | `OS_ARTIFACT_CACHE_TTL_MS` | number | `300000` | Artifact-record cache TTL in ms. |
 | `OS_ARTIFACT_FETCH_TIMEOUT_MS` | number | `60000` | Timeout for remote artifact fetches. Only a positive value is honored — an unset, non-numeric, or `0` value falls back to the 60s default (pass `fetchTimeoutMs: 0` in `artifactSource` config to actually disable the timeout). |
-| `OS_SANDBOX_HOOK_TIMEOUT_MS` | number | `250` | Default wall-clock budget for a sandboxed **hook** body (QuickJS). Each invocation compiles a fresh WASM module and a nested hook compiles another inside the parent's budget, so a loaded/slow host (e.g. an oversubscribed CI runner) may need a larger floor. Only a positive integer is honored; unset / non-numeric / non-positive keeps the 250ms default. A hook body's own declared `timeoutMs` still wins over this. |
-| `OS_SANDBOX_ACTION_TIMEOUT_MS` | number | `5000` | Default wall-clock budget for a sandboxed **action** body (QuickJS). Same resolution rules as the hook variant above (positive integer only; an action body's own `timeoutMs` still wins). |
+| `OS_SANDBOX_HOOK_TIMEOUT_MS` | number | `250` | Default **CPU-time** budget for a sandboxed **hook** body (QuickJS, ADR-0102): how much *VM-active* time a body may burn — idle host-await time and a nested hook's own run are NOT charged. A loaded/slow host rarely needs to raise this now (it is not wall-clock), but the knob remains. Only a positive integer is honored; unset / non-numeric / non-positive keeps the 250ms default. A hook body's own declared `timeoutMs` still wins over this. |
+| `OS_SANDBOX_ACTION_TIMEOUT_MS` | number | `5000` | Default **CPU-time** budget for a sandboxed **action** body (QuickJS). Same resolution rules as the hook variant above (positive integer only; an action body's own `timeoutMs` still wins). |
+| `OS_SANDBOX_WALL_CEILING_MS` | number | `30000` | Wall-clock ceiling (ADR-0102) — the backstop that cuts a hook/action body stuck on a host call that never settles (which burns no CPU, so the CPU budget alone would never fire). The effective ceiling is `max(this, cpuBudget)`, so it can never cut a body still inside its CPU budget. Positive integer only; unset keeps 30s. |
 | `OS_INLINE_SEED_BUDGET_MS` | number | `8000` | Time budget for synchronous seed execution at boot before deferring to a worker. |
 | `OS_TENANT_AUDIT` | flag | `1` | Set to `0` to silence the tenant-isolation audit warnings emitted by the SQL driver. |
 

docs/adr/0102-sandbox-cpu-budget-and-engine-variant.md

@@ -1,6 +1,6 @@
 # ADR-0102: Sandbox Execution Budget & Engine Variant — CPU-time budget with a wall ceiling; per-invocation sync WASM modules, precompiled
 
-**Status**: Proposed (2026-07-19) — spec'd and API-verified in framework#3275; lands as Phase 1 #3295 (D1), Phase 2 #3296 (D2), Phase 3 #3297 (D3). Flips to **Accepted** when Phase 1 merges; D3 is explicitly deferrable without weakening D1/D2.
+**Status**: Accepted (2026-07-19) — **D1 landed** with Phase 1 (#3295): CPU-time budget + wall ceiling in `QuickJSScriptRunner`, nested-write integration tests now green at the stock 250ms budget. D2 (#3296, drop asyncify) and D3 (#3297, precompile) remain **Proposed**; D3 is explicitly deferrable without weakening D1/D2.
 **Deciders**: ObjectStack Protocol Architects
 **Builds on**: the #1867 sandbox redesign (deferred-promise host calls + pump loop) — the mechanism that both retired asyncify's only remaining justification (D2) and created the discrete VM-entry points D1 meters. No earlier ADR covers the sandbox runner; prior art is code history (#3232 honor body `timeoutMs`, #3264 test de-flake, #3270 env-overridable defaults).
 **Tracking**: framework#3275 (implementation-ready spec) · #3295 / #3296 / #3297 (phases) · #3259 (motivating CI flake, closed by the #3270 stopgap)
@@ -41,7 +41,7 @@ The value resolved by `resolveTimeout()` (body `timeoutMs` › enclosing `opts.t
 
 Consequence for the knobs: names, defaults (250ms hooks / 5000ms actions), and precedence are unchanged; only the *dimension* changes. Raising the CPU budget no longer trades away host-latency tolerance, and vice versa. This restores meaning to the 250ms default on any hardware — which is what lets CI drop its 10s floor (a stopgap that today also loosens the runaway bound for every test).
 
-Open point to settle at Phase 1 review: whether the ceiling also gets `OS_SANDBOX_WALL_CEILING_MS` (recommended — symmetric with the CPU knobs, wanted by slow-IO deployments) or stays constructor-only.
+Settled (Phase 1): the ceiling is both a constructor option (`wallCeilingMs`, needed for fast tests) AND env-tunable via `OS_SANDBOX_WALL_CEILING_MS` — symmetric with the CPU knobs and wanted by slow-IO deployments. Precedence: explicit option › env › built-in 30s.
 
 ### D2 — Drop asyncify: per-invocation *sync* modules (isolation unchanged)
 

packages/runtime/src/sandbox/nested-write-real-sqlite.integration.test.ts

@@ -86,9 +86,9 @@ describe('#1867 nested cross-object write — REAL SqlDriver (better-sqlite3, on
     engine.registerDriver(driver, true);
     await engine.init();
     for (const o of [EXPENSE_REPORT, EXPENSE_LINE]) engine.registry.registerObject(o as any);
-    // Generous hook budget — the subject is the nested-write path on a real
-    // driver, not the 250ms default (see nested-write.integration.test.ts).
-    engine.setDefaultBodyRunner(hookBodyRunnerFactory(new QuickJSScriptRunner({ hookTimeoutMs: 10_000 }), { ql: engine, appId: 'expense' }));
+    // Runs at the STOCK 250ms CPU-time budget (ADR-0102 D1): idle host-await time
+    // and the nested rollup's own work are not charged (see nested-write.integration.test.ts).
+    engine.setDefaultBodyRunner(hookBodyRunnerFactory(new QuickJSScriptRunner(), { ql: engine, appId: 'expense' }));
     bindHooksToEngine(engine, [ROLLUP_HOOK as any], { packageId: 'expense' });
     return engine;
   }

packages/runtime/src/sandbox/nested-write.integration.test.ts

@@ -90,13 +90,13 @@ describe('#1867 nested cross-object write from a hook (real engine + sandbox)',
     engine.registerDriver(driver, true);
     await engine.init();
     for (const o of [parent, child]) engine.registry.registerObject(o as any);
-    // Generous hook budget: the subject here is nested-write correctness, not
-    // the 250ms default. Each hook invocation compiles a fresh WASM module and
-    // the nested parent hook compiles another inside the child's budget — on a
-    // loaded CI machine that overhead alone blew 250ms ("hook
-    // 'rollup_parent_total' exceeded timeout of 250ms").
+    // Runs at the STOCK 250ms budget on purpose (ADR-0102 D1): the budget is now
+    // CPU-time, so the child hook parking on its nested `parent.update(...)` — and
+    // the parent hook that fires host-side inside it — is not charged to the
+    // child. If this suite ever needs a generous budget again, that is a
+    // regression in CPU accounting, not an inherent cost of nested writes.
     engine.setDefaultBodyRunner(
-      hookBodyRunnerFactory(new QuickJSScriptRunner({ hookTimeoutMs: 10_000 }), { ql: engine, appId: 'test' }),
+      hookBodyRunnerFactory(new QuickJSScriptRunner(), { ql: engine, appId: 'test' }),
     );
   });