Skip to content

Commit 722338d

Browse files
alari76claude
andauthored
feat: pausable child timeouts, broader allowlist, worktree status (Joe 3/5) (#503)
* feat: realtime blocked-child notifications for the orchestrator When a child session blocks on a tool approval or question, the parent orchestrator now gets an immediate push notification with the requestId and the exact API call to respond, instead of the child silently dying at timeout. Adds a 'blocked' child status, stops auto-denying pending prompts when the last client leaves an agent session, and replaces the headless-agent fast-deny with the normal prompt flow (5-min timeout remains the backstop). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: persistent notification outbox with replay when the orchestrator returns Notifications to the orchestrator (Agent Joe) were silently dropped when its Claude process was not running — terminal child-session events and monitor alerts vanished, so Joe woke up with no idea what happened while it was down. - New OrchestratorOutbox: JSON-file-backed queue (survives restarts, capped at 200 items) with a 60s background flusher that replays queued items as a single digest message once the orchestrator session is alive again. Flush is gated on the rate-limit circuit breaker and clears the queue before sending to avoid double-delivery. - sendOrchestratorNotification now enqueues to the outbox when the parent is unreachable and returns true (the outbox owns delivery from that point); false only when queueing itself fails. - OrchestratorMonitor.deliverToOrchestrator enqueues instead of dropping when the orchestrator process is down. - Flusher wired up at server boot in ws-server.ts. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: pausable child timeouts, broader allowlist, worktree status in spawn response Blocked children used to burn their whole timeout waiting on an approval the orchestrator never saw — with PR1's realtime blocked notifications in place, the timeout now pauses while a child waits for an answer. - Working-time clock pauses when a child blocks on an approval/question and resumes (with the remaining budget) once the prompt is answered; a separate 30-min blocked-time cap still terminates abandoned children. - DEFAULT_TIMEOUT_MS raised from 10 to 30 minutes; spawn API now accepts and validates timeoutMs (1 min – 4 h), documented in Joe's CLAUDE.md. - AGENT_CHILD_ALLOWED_TOOLS broadened: python3, pytest, sed, rg, jq, mkdir, cp, mv, touch (rm/sudo/docker still require approval). - ChildSession now reports worktree status ('active'/'failed'/'none') and worktreePath so the orchestrator knows when isolation failed. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
1 parent 718b449 commit 722338d

4 files changed

Lines changed: 254 additions & 13 deletions

File tree

server/orchestrator-children.test.ts

Lines changed: 136 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,5 @@
11
/** Tests for OrchestratorChildManager — verifies spawn, status tracking,
22
* listing, timeout, and prompt generation. */
3-
/* eslint-disable @typescript-eslint/no-explicit-any */
43
import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'
54

65
vi.mock('./config.js', () => ({
@@ -15,7 +14,7 @@ vi.mock('./orchestrator-outbox.js', () => ({
1514
getOrchestratorOutbox: () => ({ enqueue: () => {} }),
1615
}))
1716

18-
import { OrchestratorChildManager, type ChildSessionRequest } from './orchestrator-children.js'
17+
import { OrchestratorChildManager, AGENT_CHILD_ALLOWED_TOOLS, type ChildSessionRequest } from './orchestrator-children.js'
1918

2019
// ---------------------------------------------------------------------------
2120
// Helpers
@@ -138,6 +137,30 @@ describe('OrchestratorChildManager', () => {
138137
expect(prompt).toContain('Worktree Not Available')
139138
})
140139

140+
it('reports worktree status "active" with the worktree path on success', async () => {
141+
const child = await manager.spawn(makeRequest({ useWorktree: true }))
142+
143+
expect(child.worktree).toBe('active')
144+
expect(child.worktreePath).toBe('/repos/myproject-wt-child123')
145+
})
146+
147+
it('reports worktree status "failed" when worktree creation fails', async () => {
148+
sessions = makeMockSessions(false)
149+
manager = new OrchestratorChildManager(sessions)
150+
151+
const child = await manager.spawn(makeRequest({ useWorktree: true }))
152+
153+
expect(child.worktree).toBe('failed')
154+
expect(child.worktreePath).toBeNull()
155+
})
156+
157+
it('reports worktree status "none" when no worktree was requested', async () => {
158+
const child = await manager.spawn(makeRequest({ useWorktree: false }))
159+
160+
expect(child.worktree).toBe('none')
161+
expect(child.worktreePath).toBeNull()
162+
})
163+
141164
it('records failed status when session creation throws', async () => {
142165
sessions.create = vi.fn(() => { throw new Error('create failed') })
143166

@@ -272,6 +295,28 @@ describe('OrchestratorChildManager', () => {
272295
})
273296
})
274297

298+
// -------------------------------------------------------------------------
299+
// Default allowlist
300+
// -------------------------------------------------------------------------
301+
302+
describe('AGENT_CHILD_ALLOWED_TOOLS', () => {
303+
it('includes the broadened dev toolset', () => {
304+
for (const tool of [
305+
'Bash(python3:*)', 'Bash(pytest:*)',
306+
'Bash(sed:*)', 'Bash(rg:*)', 'Bash(jq:*)',
307+
'Bash(mkdir:*)', 'Bash(cp:*)', 'Bash(mv:*)', 'Bash(touch:*)',
308+
]) {
309+
expect(AGENT_CHILD_ALLOWED_TOOLS).toContain(tool)
310+
}
311+
})
312+
313+
it('still excludes destructive commands', () => {
314+
for (const tool of ['Bash(rm:*)', 'Bash(sudo:*)', 'Bash(docker:*)', 'Bash:*', 'Bash']) {
315+
expect(AGENT_CHILD_ALLOWED_TOOLS).not.toContain(tool)
316+
}
317+
})
318+
})
319+
275320
// -------------------------------------------------------------------------
276321
// Timeout
277322
// -------------------------------------------------------------------------
@@ -305,6 +350,95 @@ describe('OrchestratorChildManager', () => {
305350
expect(child.error).toContain('Timed out')
306351
expect(child.completedAt).toBeTruthy()
307352
})
353+
354+
it('pauses the working clock while the child is blocked on a prompt', async () => {
355+
const pendingApprovals = new Map([['req-1', { toolInput: { command: 'git push' } }]])
356+
sessions.get = vi.fn(() => ({
357+
claudeProcess: { isAlive: vi.fn(() => true), stop: vi.fn() },
358+
outputHistory: [],
359+
pendingToolApprovals: pendingApprovals,
360+
pendingControlRequests: new Map(),
361+
}))
362+
363+
const child = await manager.spawn(makeRequest({ timeoutMs: 60_000, parentSessionId: 'parent-1' }))
364+
365+
// Burn half the working budget, then block on a prompt
366+
vi.advanceTimersByTime(30_000)
367+
for (const cb of sessions._promptListeners) cb(child.id, 'permission', 'Bash', 'req-1')
368+
expect(child.status).toBe('blocked')
369+
370+
// Way past the original working deadline while blocked — must NOT time out
371+
vi.advanceTimersByTime(10 * 60_000)
372+
expect(child.status).toBe('blocked')
373+
374+
// Prompt answered: a result arrives with no pendings → clock resumes
375+
pendingApprovals.clear()
376+
sessions.get = vi.fn(() => ({
377+
claudeProcess: { isAlive: vi.fn(() => true), stop: vi.fn() },
378+
outputHistory: [{ type: 'output', data: 'pushed and created a pull request '.repeat(10) }],
379+
pendingToolApprovals: new Map(),
380+
pendingControlRequests: new Map(),
381+
}))
382+
for (const cb of sessions._resultListeners) cb(child.id, false)
383+
384+
await vi.waitFor(() => {
385+
expect(child.status).toBe('completed')
386+
})
387+
})
388+
389+
it('times out a child that stays blocked past the blocked-time budget', async () => {
390+
const pendingApprovals = new Map([['req-1', { toolInput: { command: 'git push' } }]])
391+
sessions.get = vi.fn(() => ({
392+
claudeProcess: { isAlive: vi.fn(() => true), stop: vi.fn() },
393+
outputHistory: [],
394+
pendingToolApprovals: pendingApprovals,
395+
pendingControlRequests: new Map(),
396+
}))
397+
398+
const child = await manager.spawn(makeRequest({ timeoutMs: 60_000, parentSessionId: 'parent-1' }))
399+
400+
for (const cb of sessions._promptListeners) cb(child.id, 'permission', 'Bash', 'req-1')
401+
expect(child.status).toBe('blocked')
402+
403+
// Exceed the 30-minute blocked budget
404+
vi.advanceTimersByTime(31 * 60_000)
405+
406+
await vi.waitFor(() => {
407+
expect(child.status).toBe('timed_out')
408+
})
409+
expect(child.error).toContain('pending approval')
410+
})
411+
412+
it('resumes the working clock after unblock with the remaining budget', async () => {
413+
const pendingApprovals = new Map([['req-1', { toolInput: { command: 'git push' } }]])
414+
const makeSession = (pending: Map<string, unknown>) => ({
415+
claudeProcess: { isAlive: vi.fn(() => true), stop: vi.fn() },
416+
outputHistory: [],
417+
pendingToolApprovals: pending,
418+
pendingControlRequests: new Map(),
419+
})
420+
sessions.get = vi.fn(() => makeSession(pendingApprovals))
421+
422+
const child = await manager.spawn(makeRequest({ timeoutMs: 60_000, parentSessionId: 'parent-1' }))
423+
424+
// Use 30s of the budget, block, then unblock via a result event
425+
vi.advanceTimersByTime(30_000)
426+
for (const cb of sessions._promptListeners) cb(child.id, 'permission', 'Bash', 'req-1')
427+
vi.advanceTimersByTime(5 * 60_000)
428+
429+
pendingApprovals.clear()
430+
sessions.get = vi.fn(() => makeSession(new Map()))
431+
for (const cb of sessions._resultListeners) cb(child.id, false)
432+
// Result with empty output → nudge consumed? No — process alive, PR policy,
433+
// no "pull request" text → nudge fires and monitoring continues with the
434+
// resumed clock. Remaining working budget is ~30s.
435+
vi.advanceTimersByTime(31_000)
436+
437+
await vi.waitFor(() => {
438+
expect(child.status).toBe('timed_out')
439+
})
440+
expect(child.error).toContain('working time')
441+
})
308442
})
309443

310444
// -------------------------------------------------------------------------

server/orchestrator-children.ts

Lines changed: 98 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,11 @@ export interface ChildSessionRequest {
3232
deployAfter: boolean
3333
/** Use a git worktree for isolation. */
3434
useWorktree: boolean
35-
/** Timeout in ms (default 10 minutes). */
35+
/**
36+
* Working-time timeout in ms (default 30 minutes). Time spent blocked on
37+
* a pending approval/question does not count against this budget — blocked
38+
* time has its own separate cap (MAX_BLOCKED_MS).
39+
*/
3640
timeoutMs?: number
3741
/** Optional model override. */
3842
model?: string
@@ -69,6 +73,15 @@ export interface ChildSession {
6973
* parent orchestrator. Used to enforce single-fire idempotency.
7074
*/
7175
terminalNotifiedAt: string | null
76+
/**
77+
* Worktree isolation outcome:
78+
* - 'active': worktree created, session runs isolated
79+
* - 'failed': worktree was requested but creation failed (running in repo)
80+
* - 'none': worktree was not requested
81+
*/
82+
worktree: 'active' | 'failed' | 'none'
83+
/** Absolute path of the worktree when active. */
84+
worktreePath: string | null
7285
}
7386

7487
/**
@@ -83,7 +96,13 @@ export type ChildNotifyFn = (args: OrchestratorNotifyArgs) => boolean
8396
// ---------------------------------------------------------------------------
8497

8598
const MAX_CONCURRENT = 5
86-
const DEFAULT_TIMEOUT_MS = 600_000 // 10 minutes
99+
const DEFAULT_TIMEOUT_MS = 1_800_000 // 30 minutes of working time
100+
/**
101+
* Separate budget for time spent blocked on a pending approval or question.
102+
* The working-time clock is paused while blocked; this cap ensures a child
103+
* waiting on an answer that never comes still terminates eventually.
104+
*/
105+
const MAX_BLOCKED_MS = 1_800_000 // 30 minutes
87106
const CHILD_RETENTION_MS = 3_600_000 // keep completed/failed children for 1 hour
88107
const MAX_RETAINED_CHILDREN = 100 // hard cap on total entries
89108
const MAX_NOTIFIED_PROMPT_IDS = 500 // cap on the blocked-prompt dedup set
@@ -108,6 +127,12 @@ export const AGENT_CHILD_ALLOWED_TOOLS = [
108127
// Build / lint / test tools
109128
'Bash(node:*)', 'Bash(tsc:*)', 'Bash(eslint:*)', 'Bash(prettier:*)',
110129
'Bash(cargo:*)', 'Bash(go:*)', 'Bash(make:*)', 'Bash(pip:*)',
130+
// Python toolchain (linting/tests in Python repos)
131+
'Bash(python3:*)', 'Bash(pytest:*)',
132+
// Text/data processing (read-only or scoped to working dir)
133+
'Bash(sed:*)', 'Bash(rg:*)', 'Bash(jq:*)',
134+
// Non-destructive file management (no rm — deletion still needs approval)
135+
'Bash(mkdir:*)', 'Bash(cp:*)', 'Bash(mv:*)', 'Bash(touch:*)',
111136
// Safe filesystem inspection (read-only)
112137
'Bash(ls:*)', 'Bash(cat:*)', 'Bash(wc:*)',
113138
'Bash(head:*)', 'Bash(tail:*)', 'Bash(sort:*)', 'Bash(diff:*)',
@@ -126,6 +151,12 @@ export class OrchestratorChildManager {
126151
private notify: ChildNotifyFn
127152
/** Prompt requestIds already reported to the parent (single-fire per prompt). */
128153
private notifiedPromptIds = new Set<string>()
154+
/**
155+
* Per-child timeout controllers — lets the prompt handler pause the
156+
* working-time clock the moment a child blocks on an approval/question,
157+
* instead of waiting for the next monitor event.
158+
*/
159+
private timeoutControllers = new Map<string, { pause(): void; resume(): void }>()
129160

130161
constructor(sessions: SessionManager, opts?: { notify?: ChildNotifyFn }) {
131162
this.sessions = sessions
@@ -153,6 +184,8 @@ export class OrchestratorChildManager {
153184
if (!child || TERMINAL_STATUSES.has(child.status)) return
154185

155186
child.status = 'blocked'
187+
// Pause the working-time clock while the child waits for an answer.
188+
this.timeoutControllers.get(sessionId)?.pause()
156189

157190
// Single-fire per requestId (re-broadcasts on client join would otherwise
158191
// spam the parent). Prompts without a requestId can't be deduped or
@@ -294,6 +327,8 @@ export class OrchestratorChildManager {
294327
result: null,
295328
error: null,
296329
terminalNotifiedAt: null,
330+
worktree: request.useWorktree ? 'failed' : 'none', // upgraded to 'active' on success
331+
worktreePath: null,
297332
}
298333
this.children.set(sessionId, child)
299334

@@ -315,7 +350,10 @@ export class OrchestratorChildManager {
315350
let worktreeFailed = false
316351
if (request.useWorktree) {
317352
const wtPath = await this.sessions.createWorktree(sessionId, request.repo, request.branchName)
318-
if (!wtPath) {
353+
if (wtPath) {
354+
child.worktree = 'active'
355+
child.worktreePath = wtPath
356+
} else {
319357
worktreeFailed = true
320358
console.warn(`[orchestrator-child] Failed to create worktree for ${sessionId}, falling back to main directory`)
321359
}
@@ -530,19 +568,62 @@ export class OrchestratorChildManager {
530568
let settled = false
531569
const settle = () => { if (!settled) { settled = true; resolve() } }
532570

533-
// Timeout handler
534-
const timer = setTimeout(() => {
571+
// ---- Pausable working-time clock -----------------------------------
572+
// The working budget (timeoutMs) only burns while the child is doing
573+
// work. When the child blocks on an approval/question, the clock is
574+
// paused and a separate blocked-time cap (MAX_BLOCKED_MS) takes over
575+
// so an unanswered prompt still terminates the child eventually.
576+
let remainingMs = timeoutMs
577+
let workStartedAt = Date.now()
578+
let workTimer: ReturnType<typeof setTimeout> | null = null
579+
let blockedTimer: ReturnType<typeof setTimeout> | null = null
580+
581+
const clearTimers = () => {
582+
if (workTimer) { clearTimeout(workTimer); workTimer = null }
583+
if (blockedTimer) { clearTimeout(blockedTimer); blockedTimer = null }
584+
}
585+
586+
const fireTimeout = (error: string) => {
535587
if (settled) return
536588
child.status = 'timed_out'
537-
child.error = `Timed out after ${timeoutMs}ms`
589+
child.error = error
538590
child.completedAt = new Date().toISOString()
591+
clearTimers()
539592

540593
const session = this.sessions.get(child.id)
541594
if (session?.claudeProcess?.isAlive()) {
542595
session.claudeProcess.stop()
543596
}
544597
settle()
545-
}, timeoutMs)
598+
}
599+
600+
const pause = () => {
601+
if (settled || !workTimer) return
602+
clearTimeout(workTimer)
603+
workTimer = null
604+
remainingMs = Math.max(0, remainingMs - (Date.now() - workStartedAt))
605+
blockedTimer ??= setTimeout(() => {
606+
fireTimeout(`Timed out after waiting ${MAX_BLOCKED_MS}ms for a pending approval/answer`)
607+
}, MAX_BLOCKED_MS)
608+
}
609+
610+
const resume = () => {
611+
if (settled || workTimer) return
612+
if (blockedTimer) { clearTimeout(blockedTimer); blockedTimer = null }
613+
workStartedAt = Date.now()
614+
workTimer = setTimeout(() => {
615+
fireTimeout(`Timed out after ${timeoutMs}ms of working time`)
616+
}, remainingMs)
617+
}
618+
619+
// Expose pause/resume to the prompt handler (handleChildPrompt).
620+
this.timeoutControllers.set(child.id, { pause, resume })
621+
622+
// Start the working clock.
623+
workStartedAt = Date.now()
624+
workTimer = setTimeout(() => {
625+
fireTimeout(`Timed out after ${timeoutMs}ms of working time`)
626+
}, remainingMs)
546627

547628
// Result hook: Claude completed a turn
548629
const onResult = (sessionId: string, isError: boolean) => {
@@ -552,7 +633,7 @@ export class OrchestratorChildManager {
552633
child.status = 'failed'
553634
child.error = 'Session was deleted'
554635
child.completedAt = new Date().toISOString()
555-
clearTimeout(timer)
636+
clearTimers()
556637
settle()
557638
return
558639
}
@@ -564,9 +645,15 @@ export class OrchestratorChildManager {
564645
// act. Keep monitoring; the next result/exit event re-evaluates.
565646
if (session.pendingToolApprovals.size > 0 || session.pendingControlRequests.size > 0) {
566647
child.status = 'blocked'
648+
pause()
567649
return
568650
}
569651

652+
// The prompt (if any) was answered — restart the working clock so
653+
// post-approval work draws from the remaining working budget.
654+
resume()
655+
if (child.status === 'blocked') child.status = 'running'
656+
570657
const text = this.extractText(session.outputHistory)
571658
// Check if the final step was done; if not, nudge (keep listening)
572659
if (this.ensureFinalStep(child, session, text, nudgedIds, supersededMsgs)) return
@@ -575,7 +662,7 @@ export class OrchestratorChildManager {
575662
child.result = text || null
576663
child.error = isError ? 'Claude returned an error' : null
577664
child.completedAt = new Date().toISOString()
578-
clearTimeout(timer)
665+
clearTimers()
579666
settle()
580667
}
581668

@@ -590,7 +677,7 @@ export class OrchestratorChildManager {
590677
child.result = text || null
591678
child.error = text.length <= 100 ? 'Claude exited without sufficient output' : null
592679
child.completedAt = new Date().toISOString()
593-
clearTimeout(timer)
680+
clearTimers()
594681
settle()
595682
}
596683

@@ -601,6 +688,7 @@ export class OrchestratorChildManager {
601688
// Unsubscribe listeners to prevent accumulation across spawn() calls
602689
unsubResult?.()
603690
unsubExit?.()
691+
this.timeoutControllers.delete(child.id)
604692
// Safety net: ensure isProcessing is cleared when monitoring ends.
605693
// handleClaudeResult should have already done this, but edge cases
606694
// (nudge race, missed result event) can leave the flag stuck.

0 commit comments

Comments
 (0)