Phase 3a: runtime-neutral session hibernate/wake

[simple-agent-manager]

Summary

• Adds runtime-neutral hibernate/restore for cf-container sessions: private R2 $HOME/Git WIP snapshots, additive D1 metadata, visible degradation, wake-before-proxy, and fresh runtime asset reinjection.
• Hardens the original draft: snapshotting preserves the user's real Git index/branch, restore keeps the original branch, tar extraction rejects traversal and existing symlinks, completion trusts server-derived R2 keys/sizes rather than client claims, and chunked size-limit bypasses are rejected.
• Provisions prefix-scoped R2 lifecycle expiration through Pulumi and injects the same validated TTL into the Worker.

Validation

• pnpm lint (warnings only; no errors)
• pnpm typecheck
• Focused API snapshot tests: 15 passed
• Pulumi infrastructure tests: 43 passed
• go test -race ./internal/server and focused snapshot race tests passed
• pnpm quality:migration-safety, pnpm quality:do-migration-safety, pnpm quality:file-sizes, and pnpm quality:wrangler-bindings
• CI lint/typecheck/build/code-quality/Pulumi/VM-agent checks
• Full staging E2E evidence (required before ready/merge; tracked below)

Staging Verification (REQUIRED for all code changes — merge-blocking)

• Staging deployment green
• Live app verified via Playwright
• Existing workflows confirmed working
• New feature/fix verified on staging — real hibernate → $HOME/WIP snapshot → sleep → fresh wake → restore → native resume/fallback
• Infrastructure verification completed — real container workspace/session lifecycle and heartbeat confirmed
• Mobile and desktop verification notes added for UI changes — N/A: no UI files changed

Staging Verification Evidence

Pending coordinated staging turn 1. This PR remains draft until the complete live flow and cleanup pass.

UI Compliance Checklist (Required for UI changes)

N/A: no UI files changed.

End-to-End Verification (Required for multi-component changes)

• Data flow traced from container handback through vm-agent, callback-auth Worker routes, D1/R2, fresh container wake, restore, runtime injection, and session-host recreation.
• Capability tests cover callback route/service/D1/R2 wiring plus VM-agent Git/tar boundaries.
• Existing behavior assumptions verified against implementation and focused tests.
• Live Cloudflare Container boundary remains the staging gate below.

Data Flow Trace

1. Idle callback ends active work in apps/api/src/routes/projects/agent-activity-callback.ts, causing VmAgentContainer to hibernate before sleep.
2. packages/vm-agent/internal/server/session_snapshot.go creates a non-mutating Git bundle and $HOME tar, then uploads through workspace-scoped callback routes.
3. apps/api/src/routes/workspaces/session-snapshots.ts derives R2 keys, verifies manifest identity and authoritative object sizes, and apps/api/src/services/session-snapshots.ts persists additive D1 metadata.
4. apps/api/src/durable-objects/vm-agent-container.ts launches a fresh container on wake and calls restore before proxying.
5. vm-agent restores $HOME/WIP, calls provisionWorkspaceRuntime to overwrite stale runtime assets/credentials, rebuilds the session host, and reports visible restore status.

Untested Gaps

The real Cloudflare Container sleep/wake boundary is intentionally not claimed from unit tests; it will be exercised on staging before the draft is marked ready.

Post-Mortem (Required for bug fix PRs)

What broke

The initial draft could mutate a user's Git index/branch while snapshotting, leave restore on a temporary branch, accept unbounded chunked uploads, trust client-reported sizes, and follow pre-existing home symlinks during extraction.

Root cause

The first implementation used the live Git index/branch as snapshot scratch state and treated callback-authenticated metadata as authoritative without rechecking storage facts or extraction filesystem state.

Class of bug

Cross-boundary data-integrity and trust-boundary validation gaps.

Why it wasn't caught

The initial tests covered happy-path artifacts and lexical tar traversal, but not exact Git status preservation, existing filesystem symlinks, chunked uploads, or client/storage size disagreement.

Process fix included in this PR

Regression tests now assert exact branch/index preservation, original-branch restore, symlink rejection, required upload lengths, server-derived R2 sizes, and manifest identity. Pulumi tests assert managed TTL cleanup.

Post-mortem file

tasks/archive/2026-07-11-runtime-neutral-session-hibernate-wake.md

Specialist Review Evidence

• All local reviewer checklists completed and findings addressed before merge
• If any reviewer did NOT complete: needs-human-review label added and merge deferred to human — N/A after the complete in-session checklist rerun; the PR remains draft for staging, not review gaps.

Reviewer	Status	Outcome
cloudflare-specialist	ADDRESSED	Added Pulumi-managed prefix expiration aligned with Worker TTL; additive D1/R2 patterns and bindings pass
go-specialist	ADDRESSED	Temporary Git index preserves user state; restore retains branch; changed server race suite passes
security-auditor	ADDRESSED	Workspace-scoped callbacks, server-derived keys/sizes, required lengths, symlink-safe extraction, fresh injection verified
constitution-validator	PASS	TTL, budgets, thresholds, watchdogs, prefix, and limits remain configurable with named defaults
env-validator	PASS	Env interface, generated Worker vars, .env.example, and shared env reference are synchronized
doc-sync-validator	PASS	Task constraints/evidence and configuration references match current implementation
test-engineer	ADDRESSED	Added Git-integrity, symlink, authoritative-size, identity, lifecycle, route, and vertical-slice coverage
task-completion-validator	PASS	Research/checklist/diff/criteria A-F traced; live boundary is explicitly reserved for blocking Phase 6 staging

Exceptions

• Scope: local Miniflare worker vertical-slice execution.
• Rationale: bundled workerd 1.20260329.1 repeatedly SIGSEGVs before test execution in this workspace; CI Worker/VM-agent integration and E2E checks pass, and live staging remains blocking.
• Expiration: staging verification must pass before ready/merge.

Agent Preflight (Required)

• Preflight completed before code changes

Classification

• external-api-change
• cross-component-change
• business-logic-change
• public-surface-change
• docs-sync-change
• security-sensitive-change
• ui-change
• infra-change

External References

Official documentation from Cloudflare was consulted for R2 object lifecycle behavior and the bucket lifecycle API: https://developers.cloudflare.com/r2/buckets/object-lifecycles/ and https://developers.cloudflare.com/api/resources/r2/subresources/buckets/subresources/lifecycle/.

Codebase Impact Analysis

Affected paths: apps/api callback routes, D1 schema/migration, snapshot service, container DO and deployment vars; packages/vm-agent hibernate/restore and tests; infra R2 lifecycle resources; scripts/deploy Pulumi-output to Worker-var synchronization; task and environment reference docs.

Documentation & Specs

Updated apps/api/.env.example, .claude/skills/env-reference/SKILL.md, and tasks/archive/2026-07-11-runtime-neutral-session-hibernate-wake.md with current configuration, superseding authorization, findings, and evidence.

Constitution & Risk Check

Checked additive migration/data-loss rules, Principle XI configurable values, callback authentication scoping, server authority over storage keys and sizes, private snapshot handling, TTL cleanup, visible degradation, and post-restore fresh credential injection. The remaining real-runtime risk is blocked on coordinated staging E2E.

[codspeed-hq]

Merging this PR will not alter performance

✅ 6 untouched benchmarks

---

_{Comparing sam/implement-phase-3a-idea-04cwjb (b16496a) with main (aaa9537)}

[simple-agent-manager]

✅ Wake→reply now works end-to-end on staging (root-caused via Worker logs)

Using wrangler tail sam-api-staging, I traced the restored-session failure to four distinct bugs in the wake path — each was masking the next. All are now fixed and the full flow is verified live.

The four fixes

1. Stale pre-wake state → 410/500 (VmAgentContainer.proxyHttp/fetch): read container state once before wakeFromSnapshot(), then applied the stopped → 410 guard on that stale state, so a freshly-woken container was rejected (surfaced as 500). Now re-reads state after wake. (regression test: vm-agent-container-wake-state.test.ts, fails on pre-fix code.)
2. Node-scoped CALLBACK_TOKEN: wake launched the container with a workspace-scoped token; node callbacks (errors/activity) need node scope → 403. Now signs signNodeCallbackToken(nodeId), matching the normal launch.
3. Workspace-scoped runtime.CallbackToken unset on wake: the message reporter + snapshot callbacks use runtime.CallbackToken (no node fallback). A fresh wake container never ran create-workspace, so it was empty → reporter had "no auth token", replies discarded. The DO now passes a workspace-scoped token on the restore request; the vm-agent persists it via upsertWorkspaceRuntime.
4. Message reporter never primed on restore: the normal path (handleCreateAgentSession) creates/binds the per-workspace reporter with projectID+chatSessionID; the restore path skipped it, so the restored agent's output had nowhere to enqueue. Restore now primes the reporter.

Live end-to-end verification (staging, 023eb65b0)

Fresh cf-container session → set HOME marker + staged/unstaged/untracked Git WIP → container slept → sent a message → container woke, restored, and the agent replied. The agent's reply (persisted to chat, message count 20→77) read back:

• HOME marker HOME-MARKER-november-final ✅
• git status: M README.md, A sam-staged.txt, ?? sam-untracked.txt ✅
• unstaged/staged/untracked contents intact ✅

wake_started → restore 200 → wake_completed → ACP Prompt started/completed → POST /messages 200, zero auth errors. (The agent even correctly refused an earlier attestation-token-style prompt as prompt-injection — nice.)

Housekeeping

• Rebased on main; extracted node diagnostic getters into node-agent-diagnostics.ts to keep node-agent.ts under the 800-line gate.
• All test workspaces cleaned up.

needs-human-review intentionally left on — this is a 4-fix change set spanning the container DO and vm-agent; it should have human sign-off (and the specialist reviewers run) before merge.

[simple-agent-manager]

Specialist Review Evidence

Ran 5 local specialist reviewers on the full diff (git diff origin/main...HEAD).

Reviewer	Verdict	Key outcome
task-completion-validator	PASS	All 4 fixes present & correctly wired; traced the full wake→restore→reply data flow with no gaps; refactor dropped no functions.
security-auditor	PASS	Token-scope checklist all PASS (least-privilege node vs workspace tokens, no token leak, restore endpoint authed via requireNodeManagementAuth, no scope escalation, appropriate TTLs). MEDIUM/LOW = additive defense-in-depth.
go-specialist	PASS w/ notes	No CRITICAL. MEDIUM: prime no-op silent path → fixed (now warns). LOWs deferred.
cloudflare-specialist	PASS w/ notes	HIGH rec: wake concurrency mutex → fixed (ensureAwake, rule 45). MEDIUM/LOW deferred.
test-engineer	ADDRESSED	HIGH: missing tests for token-persist (#3) + reporter-prime (#4) → added Go unit tests + a concurrency regression test.

CRITICAL/HIGH findings — all addressed in commit a291d4fca

1. Wake concurrency (cloudflare HIGH, rule 45): serialized wake behind a promise-chain mutex (ensureAwake) that re-reads lifecycleStatus inside the critical section, so two concurrent requests to a sleeping container launch+restore exactly once. Discriminating concurrency regression test added.
2. Missing tests (test-engineer HIGH): added Go tests locking in the two silent-failure invariants — restore-body workspace token is persisted (callbackTokenForWorkspace/workspaceCallbackToken), and the message reporter is primed on restore. Plus the DO concurrency test above.
3. Silent reporter no-op (go MEDIUM): primeRestoredMessageReporter now logs a warning instead of silently dropping replies when project/chat-session context is missing.

Deferred (LOW / pre-existing, non-regression)

Tracked in tasks/backlog/2026-07-12-cf-container-wake-restore-hardening.md: DO-internal restore wall-clock budget, decouple restore agent-start from request ctx, tar decompression limit, restoreBody filtering before degraded persistence, node-management token TTL decoupling, absoluteControlPlaneURL hostname check, break the pre-existing node-agent.ts re-export circular import, markWakeDegraded status accuracy, ctx threading in skipOversizedUntracked, and an HTTP-level handler test.

Both runtime paths verified end-to-end on staging

• Instant cf-container: hibernate → sleep → wake → restore ($HOME + Git WIP) → agent replies reading back the restored marker + staged/unstaged/untracked files. Re-verified after the mutex change.
• VM (Hetzner): provisioned a real VM; the agent created an agent profile via SAM MCP, then created a file, committed (fd18dd8), and pushed the branch to origin — confirming standard VM functionality is intact.

[sonarqubecloud]

Quality Gate passed

Issues
10 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
1.4% Duplication on New Code

See analysis details on SonarQube Cloud