fix: reconcile dead TaskRunner DO/D1 task lifecycle promptly (Priority 5)

[simple-agent-manager]

Summary

Repairs TaskRunner Durable Object / D1 task lifecycle reconciliation so dead work no longer sits in_progress / awaiting_followup until the 8-hour hard timeout. Adds a separate configurable 24-hour absolute runaway-cost ceiling that fails even demonstrably live tasks, bounding unbounded compute without changing the liveness-gated semantics of TASK_RUN_HARD_TIMEOUT_MS.

Production evidence (2026-07-11, read-only D1/observability): Since 2026-06-01 there were 57 TaskRunner DO completed but task still in 'in_progress' warnings and 37 awaiting_followup 480-minute hard-timeout failures. All 34 D1 rows with the exact 480-minute message were conversation-mode tasks whose workspaces are now deleted; 32 still had agent_sessions.status='running', proving that status is stale liveness. Soft-timeout recovery kept skipping them on shared-node heartbeat, which proves host — not task — liveness. Dominant exit path is normal post-handoff liveness drift (no bounded-window aborted_by_recovery records found), but the recovery race is code-grounded and closed here too.

Changes:

• Composite task-scoped liveness probe (getTaskRuntimeLiveness, stuck-tasks.ts): requires live workspace (running/recovery) and healthy node with recent heartbeat and a task-scoped ACP session with a recent heartbeat. A shared-node heartbeat is never sufficient. Returns { live, conclusive, reason }; inconclusive/errored liveness never fails a task (fail-safe protects genuinely-running long work).
• Prompt reconciliation of DO-completed + D1-active mismatches: transitions to a clear terminal state with sanitized reason only when composite liveness is conclusively dead, gated by a configurable TASK_DO_MISMATCH_GRACE_MS (5 min) instead of waiting for the 8-hour timeout. awaiting_followup is now covered alongside in_progress.
• aborted_by_recovery race closure (state-machine.ts): the zero-row optimistic delegated→in_progress update now re-reads authoritative D1 status — in_progress → DO marks running/done; already-terminal/missing → DO completes without clobbering a legitimate terminal callback; still-transient (superseded) → failTask with a clear message.
• Idempotent terminal callback (callback.ts): a repeated same-terminal callback reuses cleanupTerminalTaskResourcesOrThrow and returns the task instead of throwing a transition conflict.
• Configurable, bounded control loop: new shared DEFAULT_* constants + env fallbacks; candidate query is LIMIT-bounded and the ACP read is bounded per candidate. TASK_RUN_ABSOLUTE_CEILING_MS defaults to 24 hours and terminates even a live-runtime task before any ACP liveness probe, providing a pure runaway-cost backstop.
• Removed now-dead getTaskNodeId / isNodeHeartbeatRecent helpers; corrected diagnostic elapsed-time to use started_at for in_progress/awaiting_followup.
• Docs: configuration.md, .env.example, and the canonical env-reference source document the new configuration, including TASK_RUN_ABSOLUTE_CEILING_MS=86400000.

Validation

• Option 2 focused node-pool regression: 20 passed; mutation check proved the 25-hour live-task test fails when the ceiling branch is removed.
• pnpm typecheck (api clean after shared/providers/cloud-init build)
• pnpm lint on changed files — 0 errors (2 pre-existing non-null-assertion warnings outside this diff)
• Targeted tests: stuck-tasks, task-runner-state-machine, recovery-resilience, task-callback-recoverable-error — 90 passed (node pool + separate callback pool)
• Miniflare workers-pool test tests/workers/scheduled-stuck-tasks.test.ts — passed in the rebased local full pnpm test run and in CI Test.
• Control-loop candidate-volume note (rule 47): This PR changes the sweep candidate-selection behavior for active D1 task rows and adds a per-candidate ACP liveness read for liveness-gated branches. Expected candidate volume: bounded by STUCK_TASK_MAX_CANDIDATES_PER_SWEEP (default 100) via LIMIT. Worst-case per-candidate cost: one indexed workspace+node D1 read; the more expensive ACP-session read (listAcpSessions, bounded by TASK_LIVENESS_MAX_ACP_SESSIONS=5) only runs after cheap workspace/node liveness checks pass, so dead workspaces short-circuit before it. DO getStatus() mismatch probe is gated by TASK_DO_MISMATCH_GRACE_MS. Every selected candidate now has a terminal escape path: without the absolute ceiling, a live-forever task with stale updated_at would sort to the front of the LIMIT window on every tick, burn a liveness probe, and crowd out genuinely stuck work. The ceiling terminates that rule-47 immortal candidate before the DO probe.

Staging Verification (REQUIRED for all code changes — merge-blocking)

• Staging deployment green — pending (see coordination note below)
• Live app verified via Playwright
• Existing workflows confirmed working
• New feature/fix verified on staging
• Infrastructure verification — N/A: no infra changes (no cloud-init / VM agent / DNS / TLS / scripts/deploy changes)
• Mobile/desktop — N/A: no UI changes

Staging Verification Evidence

Staging verification is being completed by follow-up task 01KXAX6D6F5AFKJQDC6WT7DGZK after rebase onto current main and force-push. The needs-human-review label remains until staging evidence is recorded and all gates are green.

End-to-End Verification

Data Flow Trace

1. Cron scheduled() → recoverStuckTasks(env) (apps/api/src/scheduled/stuck-tasks.ts) selects bounded active/awaiting_followup candidates.
2. Per candidate → getTaskRuntimeLiveness(env, task) reads workspaces+nodes (D1) then task-scoped projectDataService.listAcpSessions (ProjectData DO).
3. DO-completed mismatch branch → env.TASK_RUNNER.get(...).getStatus(); if completed && D1 non-terminal && liveness conclusive-dead → mark stuck.
4. Stuck → failTask path writes terminal D1 status + taskStatusEvents + cleanupTerminalTaskResourcesOrThrow.
5. Independent race path: state-machine.ts:transitionToInProgress zero-row update re-reads tasks.status and converges DO/D1.

Untested Gaps

Full cron→DO→D1 reconciliation is covered by the Miniflare workers-pool test scheduled-stuck-tasks.test.ts (recovery interleaving, callback idempotency, dead-mismatch, live-negative) — runs in CI. Live-agent negative case and aborted_by_recovery interleave have node-pool unit coverage that passed locally.

Post-Mortem

What broke

Dead conversation-mode tasks (workspace deleted, agent gone) remained in_progress/awaiting_followup for up to 8 hours because the reconciler only logged DO-completed/D1-active mismatches and its soft-timeout grace trusted shared-node heartbeat as task liveness.

Root cause

stuck-tasks.ts used node heartbeat (host liveness) as a proxy for task liveness and treated the DO/D1 mismatch as informational; the state-machine.ts zero-row optimistic update completed the DO without re-reading D1.

Class of bug

Reporter/liveness-scope mismatch — a coarse-grained signal (shared-node heartbeat, DO "done") used as authority for a finer-grained entity (per-task/workspace/agent liveness), with a check-then-act race across the DO/D1 boundary.

Why it wasn't caught

No test asserted the DO-completed + D1-non-terminal reconciliation against composite liveness; existing coverage stopped at node heartbeat.

Process fix included in this PR

Adds cross-runtime regression coverage (workers-pool) for DO-completed+D1-active, recovery interleaving, live-negative, and callback idempotency, exercising the exact liveness contract that was violated. (No new .claude/rules file — the existing rules 02/11/47 already cover this class; tests are the concrete guard.)

Post-mortem file

Task record tasks/active/2026-07-11-taskrunner-d1-lifecycle-reconciliation.md; idea 01KT90PKF6167SXZ9YZY0R26MM.

Specialist Review Evidence (Required for agent-authored PRs)

• All local reviewers completed; every concrete finding fixed and pushed
• The prior HIGH design decision is resolved: Option 2 implemented; needs-human-review remains for Raphael handoff

Reviewer	Status	Outcome
task-completion-validator	PASS	Rechecked active task record against the rebased diff, checklist, and tests. Implementation covers the task-scoped liveness probe, DO-completed/D1-active reconciliation, state-machine race closure, callback idempotency, env/docs updates, and pending staging/merge items tracked in this wrap-up.
cloudflare-specialist	PASS	D1/DO sweep is LIMIT-bounded, cheap workspace/node reads precede ProjectData DO ACP-session reads, ACP reads are bounded by count + timeout, inconclusive probes are fail-safe, and no D1 migrations/wrangler changes were introduced.
security-auditor	PASS	No new credential exposure, auth bypass, or secret logging. Terminal callback idempotency remains behind the existing callback path, failure reasons are sanitized internal strings, and terminal writes avoid clobbering concurrent terminal states.
test-engineer	PASS	Coverage includes dead DO/D1 mismatch, live negative cases, shared-node heartbeat false-liveness regression, transition-to-in-progress races, idempotent callbacks, env fallback behavior, and the absolute ceiling. Rebased local pnpm test passed.
constitution-validator	PASS	New thresholds and limits use shared DEFAULT_* constants with env overrides and are documented in .env.example, env schema, env-reference, and public configuration docs. No new hardcoded URLs or deployment identifiers.
control-loop-io-budget	PASS	Rule 47 review completed: candidate count is bounded, per-candidate I/O is staged/timeout-bounded, absolute ceiling gives immortal live candidates a terminal escape path, and dead/inconclusive/live branches have explicit outcomes.

RESOLVED: Option 2 implemented

A separate TASK_RUN_ABSOLUTE_CEILING_MS backstop now defaults to 24 hours and fails a task even when its ACP session keeps a fresh heartbeat. This intentionally deviates from the task literal "prevent early failure regardless of elapsed duration" wording.

Justification:

1. It caps unbounded compute for a looping agent that keeps its heartbeat fresh. No other SAM mechanism does: reconciliation is idle-gated and never fires on a chattering loop, while node max-lifetime cleanup skips nodes with active workspaces.
2. Rule 47 requires every sweep candidate to have an exit path. Without the ceiling, a live-forever task becomes an immortal stuck-task-sweep candidate: stale updated_at sorts it to the front of the LIMIT window every tick, burning a probe and crowding out genuinely stuck tasks. The ceiling check runs before the liveness probe.
3. SAM already has precedent for failing provably live tasks when the compaction-loop detector identifies runaway token spend.

TASK_RUN_HARD_TIMEOUT_MS semantics are unchanged; the new, higher ceiling is independent. awaiting_followup is not exempt.

Exceptions (If any)

• Scope: none currently open for specialist review evidence.
• Rationale: workers-pool coverage now passed in local full pnpm test and CI.
• Expiration: N/A

Agent Preflight (Required)

• Preflight completed before code changes

Classification

• cross-component-change
• business-logic-change
• docs-sync-change

External References

N/A: internal reconciliation logic. Grounded in read-only production D1/observability evidence (documented in the task file + idea 01KT90PKF6167SXZ9YZY0R26MM) and existing .claude/rules/47-control-loop-io-budget.md, .claude/rules/11-fail-fast-patterns.md.

Codebase Impact Analysis

• apps/api/src/scheduled/stuck-tasks.ts (reconciler + liveness probe)
• apps/api/src/durable-objects/task-runner/state-machine.ts (race closure)
• apps/api/src/routes/tasks/callback.ts (idempotent terminal callback)
• apps/api/src/env.ts, packages/shared/src/constants/task-execution.ts + index.ts (config)
• apps/www/.../reference/configuration.md, apps/api/.env.example (docs)

Documentation & Specs

Updated apps/www/src/content/docs/docs/reference/configuration.md and apps/api/.env.example for the three new env vars.

Constitution & Risk Check

• Principle XI (no hardcoded values): all new thresholds/limits are DEFAULT_* constants with env overrides.
• Rule 47 (control-loop I/O budget): candidate LIMIT bound, tiered/gated per-candidate reads, terminal/skip escape path for every candidate.
• Rule 11 (fail-fast identity/liveness at boundaries): liveness is fail-safe — inconclusive never fails a task.
• Key risk: false-positive early failure of a live long task — mitigated by requiring conclusive-dead composite liveness and the conclusive fail-safe.

[codspeed-hq]

Merging this PR will not alter performance

✅ 6 untouched benchmarks

---

_{Comparing sam/priority-5-repair-taskrunner-kaz1z8 (2f5bf6a) with main (688db71)}

[simple-agent-manager]

Staging wrap-up attempt for PR #1567 stopped before merge.

Completed:

• Rebasing sam/priority-5-repair-taskrunner-kaz1z8 on current origin/main completed cleanly and was force-pushed.
• Local gates passed in /tmp/sam-pr1567: dependency-order builds, pnpm typecheck, pnpm lint, pnpm test.
• Specialist Review Evidence table was populated in the PR body; local parser passed when needs-human-review is excluded.
• Staging deployment succeeded: https://github.com/raphaeltm/simple-agent-manager/actions/runs/29189352941
• Browser regression passed via token-login: POST https://api.sammy.party/api/auth/token-login returned 200, /api/projects returned 200, /api/admin/tasks/stuck returned 200, dashboard loaded.

Blocker:

• The required staging verification specifically asks to confirm a dead/completed TaskRunner DO with a non-terminal D1 task row is transitioned promptly.
• A controlled stopped-runtime task (01KXAZ3QAGE04F3GNK71B3HW3F) showed dead runtime shape (in_progress task, workspace stopped) but this exercises the 4-hour in-progress timeout path, not the prompt DO-completed mismatch branch.
• A second controlled task (01KXAZWSS6QRABP3FGXTTKXV8R) completed normally at 2026-07-12T11:03:22.550Z, which was suitable for setting up the DO-completed/D1-nonterminal mismatch. However, the available CF_TOKEN is read-only for D1 writes: Cloudflare returned 7500: You do not have permission to perform this operation for the narrow controlled update on that task row. No public/admin app route exposes an equivalent safe task-status override.
• Because the required E2E branch could not be verified, I left needs-human-review in place and did not merge.

Cleanup:

• The first controlled probe task was terminalized through the supported app route as failed with cleanup reason. The second controlled task remains normally completed; its workspace was stopped through the supported workspace stop route.

[sonarqubecloud]

Quality Gate passed

Issues
2 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud