fix: bound long-running TUI agents with a wall-clock ceiling so a busy-but-stuck agent can't run unbounded

The idle-timeout reaper resets on every PTY chunk, but Claude Code repaints
its (Ns · esc to interrupt) working counter ~1x/sec while any tool/API call is
in flight — including one stuck retrying a stalled Bedrock/network operation.
A busy-but-stuck agent therefore keeps lastOutputAt advancing forever, so
idle-reap never fires and the run had no ceiling at all: agent-b1c56083's
/do:next --swarm claim-issue orchestration churned its counter for 98 minutes
before Claude Code's own internal 'Operation timed out' finally stopped it.
Had the CLI not self-terminated, the agent would have run unbounded, holding
its lane, blocking the app cooldown, and leaking a shell session.

The long-running agent TUI path now arms an absolute wall-clock backstop from
prompt submission (default 3h, provider-configurable via tuiMaxRuntimeMs),
mirroring the one-shot runner's existing hardTimeoutTimer. On fire it salvages
as success if .agent-done exists, else finalizes as a needs-manual-finish
max-runtime-timeout failure with a captured worktree diff.

Author: atomantic

.changelog/NEXT.md

@@ -8,6 +8,7 @@
 - **A live agent TUI no longer gets reaped mid-paste.** The idle-timeout reaper only tracked `lastOutputAt` (PTY output), not client input, so pasting a large multi-line command into a running agent's session could sit in a silent reflow/commit window with no echo yet — indistinguishable from "idle" to the 5s poll. If the session happened to already be near its idle threshold, the reaper killed the PTY mid-paste, showing "[Session removed externally]" in the Shell page. The idle timer in `agentTuiSpawning.js` now also checks a new `shell.js` export, `getLastInputAt()`, and skips the reap while input arrived within the last 15s. This is deliberately keyed on input *recency*, not "is a Shell viewer attached": a regular (non-external) session's socket stays bound after its viewer navigates away — only external one-shot `tuiPromptRunner.js` runs get released via `shell:release-views` — so an attachment-based guard would have permanently suppressed idle-complete for any agent glanced at once in the Shell UI. (`server/services/shell.js`, `server/services/agentTuiSpawning.js`)
 - **Swarm claim-issue runs now write the completion sentinel so the task actually finishes.** In `/do:next --swarm` mode the orchestrator ships every issue via a PR and never touches the working tree, so it was skipping the `.agent-done` completion sentinel — leaving the CoS task hanging with no summary as if the run never ended. Phase C of the swarm directive now explicitly instructs the orchestrator to write the sentinel (with a run summary of the issues claimed, their PRs, and merge outcomes) after the serialized merges, while still skipping the per-agent `/simplify`/push steps. (`server/services/cosTaskGenerator.js`)
 - **[issue-2190] OpenCode Ollama providers now accept any --model flag.** The static `OPENCODE_CONFIG_CONTENT` lacked a models map, so OpenCode rejected every model with "not valid" and the agent sat idle. The config is now built dynamically at spawn time, declaring the provider's configured models (plus the model being run) under `provider.ollama.models` with bare ids — the exact shape OpenCode's `@ai-sdk/openai-compatible` provider validates `--model ollama/<id>` against. Applied across every OpenCode spawn path (CoS CLI/TUI agents, the toolkit CLI/TUI runners, and the light fire-and-collect CLI run), and a migration backfills existing installs' `data/providers.json` so a configured model's map is persisted too. Fixes the TUI eval path for local Ollama models.
+- **A busy-but-stuck TUI agent can no longer run unbounded — a wall-clock ceiling now backstops the idle reaper.** The idle-timeout reaper resets on every PTY chunk, but Claude Code repaints its `(Ns · esc to interrupt)` working counter roughly once a second while *any* tool/API call is in flight — including one stuck retrying a stalled Bedrock/network operation. A "busy-but-stuck" agent therefore keeps `lastOutputAt` advancing forever, so idle-reap never fires and the run had no ceiling at all: a real `/do:next --swarm` claim-issue orchestration churned its counter for 98 minutes (2026-07-06, agent-b1c56083) before Claude Code's *own* internal "Operation timed out" finally stopped it — had the CLI not self-terminated, the agent would have run unbounded, holding its lane, blocking the app's cooldown, and leaking a shell session. The long-running agent TUI path now arms an absolute wall-clock backstop from prompt submission (default 3h, provider-configurable via `tuiMaxRuntimeMs`), mirroring the one-shot runner's existing `hardTimeoutTimer`. If it fires with no `.agent-done` sentinel present, the run finalizes as a needs-manual-finish FAILURE (`max-runtime-timeout`) with the same "check for open or merged-but-uncleaned PRs" guidance as the merge-queue/review-loop idle-timeout paths, capturing any uncommitted worktree diff for post-mortem first; if the sentinel *is* present the run is salvaged as success. (`server/services/agentTuiSpawning.js`, `server/lib/tuiHandshake.js`, `server/lib/aiToolkit/providers.js`, `server/lib/aiToolkit/validation.js`, `server/lib/validation.js`)
 - **[issue-2191] TUI agents with zero file changes are now marked failed instead of silently succeeding.** A TUI agent that idled out after showing working indicators but made no uncommitted file changes was previously finalized as `success: idle-complete` — deleting the worktree and any evidence of what happened. Now the idle-complete path checks `git status` in the worktree: if the tree is clean, it finalizes as `failure: idle-no-changes` with the TUI buffer tail as error context. Any uncommitted diff is also captured to `worktree-diff.txt` in the agent archive dir before cleanup, so post-mortems can see what (if anything) was left behind.
 - **[issue-2193] Concurrent agent spawns no longer race on git worktree creation.** When two tasks for the same app spawned in the same evaluation tick, the first `git worktree add` grabbed git's per-repo worktree lock and the second failed with a lock error — blocking that task with `blockedCategory: worktree-failed` and leaving an orphan branch (`cos/task-…/agent-…`) with no worktree. Worktree creation now serializes per source repo through a single promise tail (the `issueWriteTail` pattern), so same-tick adds queue instead of colliding; `git worktree add` also retries a few times with backoff on lock errors to absorb residual external contention; and on final failure the partially-created branch is deleted (`git branch -D` + `git worktree prune`) so orphan branches stop accumulating. (`server/services/worktreeManager.js`)
 - **Scheduled improvement tasks with no configured model now follow the active provider's default instead of a stale hardcoded one.** When a self-improvement or managed-app improvement task had no explicit model in its schedule config, the generator pinned a hardcoded `claude-opus-4-5-20251101` into the task — a model that had since dropped out of the `claude-code-tui` provider config, so the scheduler spawned `claude --model claude-opus-4-5-20251101` even after the provider default moved to Opus 4.8. Since a per-task model pin is honored above the provider default (it's passed straight through to the CLI), that stale literal overrode the configured default on every scheduled run. The generator no longer pins a literal: with no configured model it leaves the task's model unset, so model selection resolves the active provider's tier/default model at spawn time and always uses a model the provider actually lists. An explicit per-task or per-schedule model pin still wins as before. (`server/services/cosTaskGenerator.js`)

server/lib/aiToolkit/providers.js

@@ -449,7 +449,11 @@ export function createProviderService(config = {}) {
         secretEnvVars: providerData.secretEnvVars || [],
         headlessArgs: providerData.headlessArgs || [],
         tuiPromptDelayMs: providerData.tuiPromptDelayMs || 2500,
-        tuiIdleTimeoutMs: providerData.tuiIdleTimeoutMs || 180000
+        tuiIdleTimeoutMs: providerData.tuiIdleTimeoutMs || 180000,
+        // Absolute wall-clock ceiling for long-running TUI agents (3h). The idle
+        // reaper can't bound a busy-but-stuck agent whose working counter keeps
+        // repainting; the consumer (agentTuiSpawning) enforces this backstop.
+        tuiMaxRuntimeMs: providerData.tuiMaxRuntimeMs || 10800000
       };
 
       data.providers[id] = provider;

server/lib/aiToolkit/validation.js

@@ -86,7 +86,11 @@ export const providerSchema = z.object({
   secretEnvVars: z.array(z.string()).optional(),
   headlessArgs: z.array(z.string()).optional(),
   tuiPromptDelayMs: z.number().int().min(250).max(60000).optional(),
-  tuiIdleTimeoutMs: z.number().int().min(10000).max(1800000).optional()
+  tuiIdleTimeoutMs: z.number().int().min(10000).max(1800000).optional(),
+  // Absolute wall-clock ceiling for a long-running TUI agent (idle-reap can't
+  // bound a busy-but-stuck agent — see DEFAULT_TUI_MAX_RUNTIME_MS). Min 1min,
+  // max 12h to cover the longest legitimate multi-hour orchestration.
+  tuiMaxRuntimeMs: z.number().int().min(60000).max(43200000).optional()
 });
 
 export const runSchema = z.object({

server/lib/tuiHandshake.js

@@ -457,6 +457,24 @@ export function scheduleSubmitEnters(write, isFinalized) {
 export const DEFAULT_TUI_PROMPT_DELAY_MS = 2500;
 export const DEFAULT_TUI_IDLE_TIMEOUT_MS = 180000;
 
+// Absolute wall-clock ceiling for a long-running TUI agent, applied from prompt
+// submission. This is the honest backstop the idle reaper CAN'T be: the reaper
+// resets on every PTY chunk, but Claude Code repaints its `(Ns · …)` working
+// counter ~1×/sec while ANY tool/API call is in flight — INCLUDING one stuck
+// retrying a stalled Bedrock/network operation. A "busy-but-stuck" agent keeps
+// `lastOutputAt` advancing forever, so idle-reap never fires and the run has NO
+// ceiling at all (real incident 2026-07-06: agent-b1c56083 churned the counter
+// for 98min on a `/do:next --swarm` claim-issue task before Claude Code's OWN
+// internal "Operation timed out" finally stopped it — had the CLI not self-
+// terminated, the agent would have run unbounded, holding a lane, blocking the
+// app's cooldown, and leaking a shell session). The one-shot runner already has
+// this backstop (tuiPromptRunner.js `hardTimeoutTimer`); the agent path omitted
+// it. 3h sits comfortably above the longest legitimate single run observed
+// (swarm claim-issue orchestrations routinely take 40–98min, and the merge-queue
+// / review-loop idle windows are 15min each) while bounding a genuinely-stuck
+// agent. Provider-configurable via `tuiMaxRuntimeMs`.
+export const DEFAULT_TUI_MAX_RUNTIME_MS = 3 * 60 * 60 * 1000;
+
 // Extended idle threshold applied ONLY while a `/do:next --swarm` orchestrator
 // is in its Phase C serialized merge queue (issue #2074). Merging PRs one at a
 // time makes each subsequent PR rebase onto the new `main` and re-run required

server/lib/validation.js

@@ -202,7 +202,11 @@ export const providerSchema = z.object({
   envVars: z.record(z.string()).optional(),
   headlessArgs: z.array(z.string()).optional(),
   tuiPromptDelayMs: z.number().int().min(250).max(60000).optional(),
-  tuiIdleTimeoutMs: z.number().int().min(10000).max(1800000).optional()
+  tuiIdleTimeoutMs: z.number().int().min(10000).max(1800000).optional(),
+  // Absolute wall-clock ceiling for long-running TUI agents (mirrors the
+  // aiToolkit providerSchema; the idle reaper can't bound a busy-but-stuck agent
+  // — see DEFAULT_TUI_MAX_RUNTIME_MS in tuiHandshake.js). Min 1min, max 12h.
+  tuiMaxRuntimeMs: z.number().int().min(60000).max(43200000).optional()
 });
 
 // Run command schema

server/services/agentTuiSpawning.js

@@ -26,6 +26,7 @@ import { isAntigravityCommand } from '../lib/antigravity.js';
 import {
   DEFAULT_TUI_PROMPT_DELAY_MS,
   DEFAULT_TUI_IDLE_TIMEOUT_MS,
+  DEFAULT_TUI_MAX_RUNTIME_MS,
   MERGE_QUEUE_IDLE_TIMEOUT_MS,
   REVIEW_LOOP_IDLE_TIMEOUT_MS,
   READY_POLL_INTERVAL_MS,
@@ -272,7 +273,8 @@ export function buildTuiSpawnConfig(provider, model, { systemPromptFile = null }
     args,
     commandLine: [command, ...args].map(shellQuote).join(' '),
     promptDelayMs: provider?.tuiPromptDelayMs || DEFAULT_TUI_PROMPT_DELAY_MS,
-    idleTimeoutMs: provider?.tuiIdleTimeoutMs || DEFAULT_TUI_IDLE_TIMEOUT_MS
+    idleTimeoutMs: provider?.tuiIdleTimeoutMs || DEFAULT_TUI_IDLE_TIMEOUT_MS,
+    maxRuntimeMs: provider?.tuiMaxRuntimeMs || DEFAULT_TUI_MAX_RUNTIME_MS
   };
 }
 
@@ -398,6 +400,12 @@ export async function spawnTuiAgent({
   let pasteEnterTimer = null;
   let pasteVerifyTimer = null;
   let submitEnterTimer = null;
+  // Absolute wall-clock backstop — armed once the prompt is SUBMITTED, cleared
+  // in finish(). Unlike the idle reaper (which resets on every PTY repaint and
+  // so never fires for a busy-but-stuck agent whose working counter keeps
+  // ticking), this bounds the total run so a hung provider/CLI can't run
+  // unbounded. See DEFAULT_TUI_MAX_RUNTIME_MS for the incident.
+  let maxRuntimeTimer = null;
 
   const streamingStrip = createStreamingAnsiStripper();
 
@@ -549,6 +557,7 @@ export async function spawnTuiAgent({
     if (pasteEnterTimer) { clearInterval(pasteEnterTimer); pasteEnterTimer = null; }
     if (pasteVerifyTimer) { clearInterval(pasteVerifyTimer); pasteVerifyTimer = null; }
     if (submitEnterTimer) { clearInterval(submitEnterTimer); submitEnterTimer = null; }
+    if (maxRuntimeTimer) { clearTimeout(maxRuntimeTimer); maxRuntimeTimer = null; }
     // Release the post-paste accumulator even when finalize fires mid-paste-
     // window. The pasteEnterTimer's own cleanup path nulls this too, but if
     // finalize comes from elsewhere (shell-exit, command-not-found, user
@@ -901,6 +910,41 @@ export async function spawnTuiAgent({
         () => shellService.writeToSession(sessionId, '\r'),
         () => finalized
       );
+      // Arm the absolute wall-clock backstop from submission (once — a paste
+      // retry re-enters submitEnter but must not stack timers). The idle reaper
+      // can't bound a busy-but-stuck agent because Claude Code's working counter
+      // keeps repainting through a stalled provider retry, resetting lastOutputAt
+      // forever; this timer is the honest ceiling regardless of PTY chatter.
+      if (!maxRuntimeTimer) {
+        maxRuntimeTimer = setTimeout(() => {
+          if (finalized) return;
+          // Salvage net: if the agent already wrote its .agent-done sentinel the
+          // run truly finished (the TUI just never idled/exited), so complete as
+          // success — mirrors the one-shot runner's response-file salvage. The
+          // 2s doneSentinelTimer normally catches this first; this covers the
+          // boundary where it lands right at the deadline.
+          const salvaged = doneSentinelPath && existsSync(doneSentinelPath);
+          if (salvaged) {
+            finish({ success: true, exitCode: 0, reason: 'max-runtime-sentinel' }).catch(err => {
+              emitLog('error', `Failed to finalize TUI agent ${agentId} at max-runtime salvage: ${err.message}`, { agentId });
+            });
+            return;
+          }
+          // Capture any uncommitted work for post-mortem before cleanup, then
+          // fail with a needs-manual-finish message — a stuck orchestrator may
+          // have left PRs/worktrees behind (same recovery guidance as the
+          // merge-queue/review-loop idle-timeout paths).
+          captureWorktreeDiff(cwd, agentDir).catch(() => {});
+          finish({
+            success: false,
+            exitCode: 124,
+            error: `TUI agent exceeded its max runtime of ${Math.round(tuiConfig.maxRuntimeMs / 60000)}min — the provider/CLI likely hung (a stalled request keeps the working counter repainting so the idle reaper never fires); check for open or merged-but-uncleaned PRs and finish them manually.`,
+            reason: 'max-runtime-timeout',
+          }).catch(err => {
+            emitLog('error', `Failed to finalize TUI agent ${agentId} at max-runtime: ${err.message}`, { agentId });
+          });
+        }, tuiConfig.maxRuntimeMs);
+      }
     };
 
     // Confirms the TUI actually received the paste before we submit. The
@@ -1233,7 +1277,8 @@ export async function spawnTuiAgent({
       tuiSessionId: sessionId,
       tuiCommand: tuiConfig.commandLine,
       tuiKind,
-      tuiIdleTimeoutMs: tuiConfig.idleTimeoutMs
+      tuiIdleTimeoutMs: tuiConfig.idleTimeoutMs,
+      tuiMaxRuntimeMs: tuiConfig.maxRuntimeMs
     }
   });
 

server/services/agentTuiSpawning.test.js

@@ -317,7 +317,10 @@ describe('spawnTuiAgent runtime', () => {
     args: [],
     commandLine: 'codex',
     promptDelayMs: 100,
-    idleTimeoutMs: 50
+    idleTimeoutMs: 50,
+    // Large so the wall-clock backstop never fires during the modest fake-timer
+    // advances the idle/paste tests perform (the max-runtime test overrides it).
+    maxRuntimeMs: 3600000
   };
 
   function runSpawn(overrides = {}) {
@@ -621,6 +624,89 @@ describe('spawnTuiAgent runtime', () => {
     );
   });
 
+  // ── 1c. Absolute wall-clock backstop reaps a busy-but-stuck agent ───────────
+  // The idle reaper resets on every PTY chunk, so an agent whose working counter
+  // keeps repainting through a stalled provider retry never idles out and would
+  // run unbounded (real incident 2026-07-06: agent-b1c56083 churned for 98min).
+  // The max-runtime timer is the honest ceiling: it fires from submission
+  // regardless of PTY chatter and, with no .agent-done sentinel present,
+  // finalizes as a needs-manual-finish FAILURE.
+  it('max-runtime: reaps a still-chattering agent as failure once the wall-clock ceiling elapses', async () => {
+    let resolveComplete;
+    const completeDone = new Promise((r) => { resolveComplete = r; });
+    vi.mocked(agentLifecycle.finalizeAgent).mockImplementation(async () => { resolveComplete(); });
+
+    // Idle window LARGER than the max-runtime ceiling so the idle reaper can't
+    // win — this isolates the wall-clock backstop (the real-world stuck agent
+    // keeps its working counter ticking, so idle never fires anyway).
+    runSpawn({ tuiConfig: { ...defaultTuiConfig, idleTimeoutMs: 600000, maxRuntimeMs: 30000 } });
+    await flushMicrotasks();
+
+    await capturedOnData(Buffer.from('Codex booting...\n'));
+    await flushMicrotasks();
+    await vi.advanceTimersByTimeAsync(2000);
+    await flushMicrotasks();
+
+    // Prompt echo → paste verification passes → submit-Enter fires → the
+    // max-runtime timer is armed.
+    await capturedOnData(Buffer.from('do the thing\n'));
+    await flushMicrotasks();
+    await vi.advanceTimersByTimeAsync(3600);
+    await flushMicrotasks();
+
+    // A busy agent that keeps chattering — but the idle window (600s) is huge so
+    // only the 30s wall-clock ceiling can reap it. Advance past the ceiling.
+    await capturedOnData(Buffer.from('(1s · thinking with high effort)\n'));
+    await vi.advanceTimersByTimeAsync(31000);
+    await flushMicrotasks();
+
+    vi.useRealTimers();
+    await completeDone;
+
+    expect(agentLifecycle.finalizeAgent).toHaveBeenCalledWith(
+      expect.objectContaining({
+        agentId: 'agent-1',
+        success: false,
+        completionReason: 'max-runtime-timeout',
+      })
+    );
+  });
+
+  // ── 1d. A written .agent-done sentinel is never overridden by a FAILURE reap ─
+  // If the agent wrote .agent-done, the run truly finished — the max-runtime
+  // ceiling firing would be a false failure. The 2s sentinel poll normally
+  // finalizes it as success first; the max-runtime timer's own salvage branch
+  // (existsSync check) is the boundary backstop mirroring the one-shot runner's
+  // response-file salvage. Either way, with the sentinel present the run must
+  // finalize as SUCCESS — never as a max-runtime FAILURE.
+  it('max-runtime does not fail a run whose .agent-done sentinel exists', async () => {
+    vi.mocked(existsSync).mockReturnValue(true);
+
+    let resolveComplete;
+    const completeDone = new Promise((r) => { resolveComplete = r; });
+    vi.mocked(agentLifecycle.finalizeAgent).mockImplementation(async () => { resolveComplete(); });
+
+    runSpawn({ tuiConfig: { ...defaultTuiConfig, maxRuntimeMs: 30000 } });
+    await flushMicrotasks();
+
+    await capturedOnData(Buffer.from('Codex booting...\n'));
+    await flushMicrotasks();
+    await vi.advanceTimersByTimeAsync(2000);
+    await flushMicrotasks();
+
+    await capturedOnData(Buffer.from('do the thing\n'));
+    await flushMicrotasks();
+    await vi.advanceTimersByTimeAsync(3600);
+    await flushMicrotasks();
+
+    vi.useRealTimers();
+    await completeDone;
+
+    const call = vi.mocked(agentLifecycle.finalizeAgent).mock.calls.at(-1)?.[0];
+    expect(call?.success).toBe(true);
+    expect(call?.completionReason).not.toBe('max-runtime-timeout');
+  });
+
   // ── 1b. Command exited before the prompt → don't paste into the bare shell ───
   // The TUI command (claude/codex/…) runs as a CHILD of the persistent PTY
   // shell, so if it exits at startup the PTY stays open and onExit never fires.