feat: template fixes, child transcript endpoint, org-aware repo discovery (Joe 5/5) (#505)

* feat: realtime blocked-child notifications for the orchestrator

When a child session blocks on a tool approval or question, the parent
orchestrator now gets an immediate push notification with the requestId
and the exact API call to respond, instead of the child silently dying
at timeout. Adds a 'blocked' child status, stops auto-denying pending
prompts when the last client leaves an agent session, and replaces the
headless-agent fast-deny with the normal prompt flow (5-min timeout
remains the backstop).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: persistent notification outbox with replay when the orchestrator returns

Notifications to the orchestrator (Agent Joe) were silently dropped when
its Claude process was not running — terminal child-session events and
monitor alerts vanished, so Joe woke up with no idea what happened while
it was down.

- New OrchestratorOutbox: JSON-file-backed queue (survives restarts,
  capped at 200 items) with a 60s background flusher that replays queued
  items as a single digest message once the orchestrator session is
  alive again. Flush is gated on the rate-limit circuit breaker and
  clears the queue before sending to avoid double-delivery.
- sendOrchestratorNotification now enqueues to the outbox when the
  parent is unreachable and returns true (the outbox owns delivery from
  that point); false only when queueing itself fails.
- OrchestratorMonitor.deliverToOrchestrator enqueues instead of
  dropping when the orchestrator process is down.
- Flusher wired up at server boot in ws-server.ts.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: pausable child timeouts, broader allowlist, worktree status in spawn response

Blocked children used to burn their whole timeout waiting on an approval
the orchestrator never saw — with PR1's realtime blocked notifications in
place, the timeout now pauses while a child waits for an answer.

- Working-time clock pauses when a child blocks on an approval/question
  and resumes (with the remaining budget) once the prompt is answered;
  a separate 30-min blocked-time cap still terminates abandoned children.
- DEFAULT_TIMEOUT_MS raised from 10 to 30 minutes; spawn API now accepts
  and validates timeoutMs (1 min – 4 h), documented in Joe's CLAUDE.md.
- AGENT_CHILD_ALLOWED_TOOLS broadened: python3, pytest, sed, rg, jq,
  mkdir, cp, mv, touch (rm/sudo/docker still require approval).
- ChildSession now reports worktree status ('active'/'failed'/'none')
  and worktreePath so the orchestrator knows when isolation failed.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: verify child completion against ground truth instead of transcript sniffing

Completion checks used to grep the child transcript for keywords like
"pull request" — false-positive on mentions, false-negative on terse
output — and treated >100 chars of output as success on process exit.

- isFinalStepMissing asks the real systems: gh pr list --head <branch>
  for 'pr' policy, git ls-remote --heads origin <branch> for 'merge'
  (run in the child's worktree, injectable exec for tests). Transcript
  sniffing remains only as a fallback when the command itself fails.
- Children are nudged once when the final step is missing; if it still
  doesn't land, the terminal notification carries a "Completion not
  verified" note so the orchestrator knows to follow up.
- Exit-path status now derives from ground truth, not output length;
  removed the dead supersededMsgs tracking.
- Joe's CLAUDE.md template: drop the "poll children every 30 min" cron
  example — the server pushes blocked/terminal notifications and owns
  completion verification now.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: template fixes + version refresh, child transcript endpoint, org-aware repo discovery (Joe 5/5)

- Fix broken backtick escaping in CLAUDE_MD_TEMPLATE so seeded files render
  real code fences, and stamp the template with a version comment; CLAUDE.md
  is now system-managed and refreshed automatically when the template version
  bumps (PROFILE.md/REPOS.md remain seed-once user memory).
- Add GET /api/orchestrator/children/:id/transcript so the orchestrator can
  read the tail of a child's output (e.g. after "Completion not verified")
  without attaching to the session.
- Make repo discovery in the monitor recurse one level so org-style layouts
  (REPOS_ROOT/org/repo) are picked up alongside flat ones.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

Author: alari76

server/orchestrator-manager.test.ts

@@ -33,6 +33,8 @@ import {
   isOrchestratorSession,
   ensureOrchestratorRunning,
   getOrchestratorSessionId,
+  readTemplateVersion,
+  CLAUDE_MD_TEMPLATE_VERSION,
 } from './orchestrator-manager.js'
 
 function fakeSessionManager(existingSession?: any) {
@@ -97,15 +99,56 @@ describe('ensureOrchestratorDir', () => {
     expect(mockWriteFileSync).toHaveBeenCalledTimes(3)
   })
 
-  it('does not overwrite existing files', () => {
-    // All paths exist
+  it('does not overwrite existing files when CLAUDE.md is current', () => {
+    // All paths exist and CLAUDE.md carries the current template version
     mockExistsSync.mockReturnValue(true)
+    mockReadFileSync.mockReturnValue(
+      `<!-- codekin-template-version: ${CLAUDE_MD_TEMPLATE_VERSION} -->\n# custom`,
+    )
 
     ensureOrchestratorDir()
 
     expect(mockMkdirSync).not.toHaveBeenCalled()
     expect(mockWriteFileSync).not.toHaveBeenCalled()
   })
+
+  it('refreshes CLAUDE.md when its template version is stale, leaving seed files alone', () => {
+    mockExistsSync.mockReturnValue(true)
+    // Unstamped (pre-versioning) CLAUDE.md → version 0 → stale
+    mockReadFileSync.mockReturnValue('# Agent — old template without a stamp')
+
+    ensureOrchestratorDir()
+
+    const writtenPaths = mockWriteFileSync.mock.calls.map((c: any[]) => c[0])
+    expect(writtenPaths).toEqual(['/tmp/test-data/orchestrator/CLAUDE.md'])
+    const written = mockWriteFileSync.mock.calls[0][1] as string
+    expect(written).toContain(`<!-- codekin-template-version: ${CLAUDE_MD_TEMPLATE_VERSION} -->`)
+  })
+})
+
+describe('readTemplateVersion', () => {
+  it('returns 0 when the file does not exist', () => {
+    mockExistsSync.mockReturnValue(false)
+    expect(readTemplateVersion('/tmp/none/CLAUDE.md')).toBe(0)
+  })
+
+  it('returns 0 when the file has no version stamp', () => {
+    mockExistsSync.mockReturnValue(true)
+    mockReadFileSync.mockReturnValue('# no stamp here')
+    expect(readTemplateVersion('/tmp/x/CLAUDE.md')).toBe(0)
+  })
+
+  it('parses the stamped version', () => {
+    mockExistsSync.mockReturnValue(true)
+    mockReadFileSync.mockReturnValue('<!-- codekin-template-version: 7 -->\n# hi')
+    expect(readTemplateVersion('/tmp/x/CLAUDE.md')).toBe(7)
+  })
+
+  it('returns 0 when reading throws', () => {
+    mockExistsSync.mockReturnValue(true)
+    mockReadFileSync.mockImplementation(() => { throw new Error('EACCES') })
+    expect(readTemplateVersion('/tmp/x/CLAUDE.md')).toBe(0)
+  })
 })
 
 describe('getOrCreateOrchestratorId', () => {

server/orchestrator-manager.ts

@@ -35,7 +35,17 @@ Agent ${AGENT_DISPLAY_NAME} tracks repositories you work with in Codekin.
 (none yet — ${AGENT_DISPLAY_NAME} will populate this as you work)
 `
 
-const CLAUDE_MD_TEMPLATE = `# Agent ${AGENT_DISPLAY_NAME} — Codekin Orchestrator
+/**
+ * Bump this whenever CLAUDE_MD_TEMPLATE changes. Already-seeded CLAUDE.md
+ * files carrying an older (or no) version stamp are refreshed on boot —
+ * without this, installs keep running on stale orchestrator instructions
+ * forever. CLAUDE.md is system-managed; user memory lives in PROFILE.md,
+ * REPOS.md and journal/, which are never overwritten.
+ */
+export const CLAUDE_MD_TEMPLATE_VERSION = 2
+
+const CLAUDE_MD_TEMPLATE = `<!-- codekin-template-version: ${CLAUDE_MD_TEMPLATE_VERSION} -->
+# Agent ${AGENT_DISPLAY_NAME} — Codekin Orchestrator
 
 You are ${AGENT_DISPLAY_NAME}, a calm and friendly ops manager inside Codekin.
 You help users keep their repositories healthy, their workflows running
@@ -174,6 +184,12 @@ curl -s "http://localhost:$CODEKIN_PORT/api/orchestrator/children" \\
 # Get specific child session
 curl -s "http://localhost:$CODEKIN_PORT/api/orchestrator/children/SESSION_ID" \\
   -H "Authorization: Bearer $CODEKIN_AUTH_TOKEN"
+
+# Read the tail of a child's transcript (what Claude actually output).
+# Useful when a child stops with "Completion not verified" or gets stuck.
+# ?limit caps the returned characters (default 5000, max 50000).
+curl -s "http://localhost:$CODEKIN_PORT/api/orchestrator/children/SESSION_ID/transcript?limit=10000" \\
+  -H "Authorization: Bearer $CODEKIN_AUTH_TOKEN"
 \`\`\`
 
 ## Scheduling Reminders & Recurring Tasks
@@ -214,31 +230,31 @@ inspect the worktree and finish it or respawn.
 Sessions can get stuck waiting for tool approvals or user answers. You can
 discover and unblock them:
 
-\\\`\\\`\\\`bash
+\`\`\`bash
 # List all sessions with pending prompts
 curl -s "http://localhost:$CODEKIN_PORT/api/orchestrator/sessions/pending-prompts" \\
   -H "Authorization: Bearer $CODEKIN_AUTH_TOKEN"
-\\\`\\\`\\\`
+\`\`\`
 
-Returns sessions with their pending prompts, including the \\\`requestId\\\`,
-\\\`toolName\\\`, and \\\`promptType\\\` ("permission" or "question").
+Returns sessions with their pending prompts, including the \`requestId\`,
+\`toolName\`, and \`promptType\` ("permission" or "question").
 
 ### Giving Approvals to Stuck Sessions
 If a child session is blocked on a tool approval and you're confident it's
 safe, you can approve it directly:
 
-\\\`\\\`\\\`bash
+\`\`\`bash
 curl -s -X POST "http://localhost:$CODEKIN_PORT/api/orchestrator/sessions/SESSION_ID/respond" \\
   -H "Authorization: Bearer $CODEKIN_AUTH_TOKEN" \\
   -H "Content-Type: application/json" \\
   -d '{"requestId": "REQUEST_ID", "value": "allow"}'
-\\\`\\\`\\\`
+\`\`\`
 
-Values: \\\`"allow"\\\`, \\\`"deny"\\\`, \\\`"always_allow"\\\`, or free text for question prompts.
+Values: \`"allow"\`, \`"deny"\`, \`"always_allow"\`, or free text for question prompts.
 
 **Guidelines for giving approvals:**
 - Only approve tools you understand — if unsure, ask the user
-- Prefer \\\`"allow"\\\` over \\\`"always_allow"\\\` for child sessions
+- Prefer \`"allow"\` over \`"always_allow"\` for child sessions
 - Never approve destructive commands (rm -rf, git push --force, DROP TABLE)
   without user confirmation
 - For question prompts, provide a reasonable answer or ask the user
@@ -299,9 +315,9 @@ Users can manage trust directly in chat:
 4. Read skill-profile.json for guidance style adaptation
 5. Check for new audit reports that may have landed
 6. Check for decisions pending outcome assessment
-7. **Re-establish cron jobs** — cron jobs do not survive session restarts, so always re-create your standard recurring checks on startup:
+7. **Re-establish cron jobs** — cron jobs do not survive session restarts, so re-create your scheduled work on startup:
    - Report check: \`cron: "3 9 * * *"\`, \`prompt: "Check for new audit reports across all managed repos and triage any new findings"\`
-   - Child session monitor: \`cron: "*/30 * * * *"\`, \`prompt: "Check child session status and unblock any stuck sessions"\`
+   - Do NOT create a child-session polling cron — the server pushes blocked/terminal notifications to you in realtime.
 8. Greet the user with a brief, friendly status update
 
 ### Greeting Guidelines
@@ -321,15 +337,32 @@ export function ensureOrchestratorDir(): void {
   const journalDir = join(ORCHESTRATOR_DIR, 'journal')
   if (!existsSync(journalDir)) mkdirSync(journalDir, { recursive: true })
 
-  // Seed files only if they don't exist (preserve user edits)
+  // Seed memory files only if they don't exist (preserve user edits)
   const seeds: [string, string][] = [
     [join(ORCHESTRATOR_DIR, 'PROFILE.md'), PROFILE_TEMPLATE],
     [join(ORCHESTRATOR_DIR, 'REPOS.md'), REPOS_TEMPLATE],
-    [join(ORCHESTRATOR_DIR, 'CLAUDE.md'), CLAUDE_MD_TEMPLATE],
   ]
   for (const [path, content] of seeds) {
     if (!existsSync(path)) writeFileSync(path, content, 'utf-8')
   }
+
+  // CLAUDE.md is system-managed: refresh it whenever the embedded template
+  // version is older than the current one (or missing entirely).
+  const claudeMdPath = join(ORCHESTRATOR_DIR, 'CLAUDE.md')
+  if (readTemplateVersion(claudeMdPath) < CLAUDE_MD_TEMPLATE_VERSION) {
+    writeFileSync(claudeMdPath, CLAUDE_MD_TEMPLATE, 'utf-8')
+  }
+}
+
+/** Parse the template version stamp from a seeded CLAUDE.md; 0 when absent. */
+export function readTemplateVersion(path: string): number {
+  try {
+    if (!existsSync(path)) return 0
+    const match = /<!-- codekin-template-version: (\d+) -->/.exec(readFileSync(path, 'utf-8'))
+    return match ? parseInt(match[1], 10) : 0
+  } catch {
+    return 0
+  }
 }
 
 /** Get or create a stable session UUID that persists across restarts. */

server/orchestrator-monitor.test.ts

@@ -1,8 +1,11 @@
 /** Tests for the passive-repo notifier predicate — see #issue: alerts were
  * firing for repos with zero enabled workflows, recommending de-scheduling
  * workflows that didn't exist. */
-import { describe, it, expect } from 'vitest'
-import { hasEnabledWorkflowForRepo } from './orchestrator-monitor.js'
+import { describe, it, expect, beforeEach, afterEach } from 'vitest'
+import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from 'fs'
+import { tmpdir } from 'os'
+import { join } from 'path'
+import { hasEnabledWorkflowForRepo, discoverRepoPathsUnder } from './orchestrator-monitor.js'
 import type { ReviewRepoConfig } from './workflow-config.js'
 
 const make = (overrides: Partial<ReviewRepoConfig>): ReviewRepoConfig => ({
@@ -46,3 +49,55 @@ describe('hasEnabledWorkflowForRepo', () => {
     expect(hasEnabledWorkflowForRepo(repoPath, entries)).toBe(false)
   })
 })
+
+describe('discoverRepoPathsUnder', () => {
+  let root: string
+
+  beforeEach(() => {
+    root = mkdtempSync(join(tmpdir(), 'codekin-discover-'))
+  })
+
+  afterEach(() => {
+    rmSync(root, { recursive: true, force: true })
+  })
+
+  function makeRepo(...segments: string[]): string {
+    const repo = join(root, ...segments)
+    mkdirSync(join(repo, '.git'), { recursive: true })
+    return repo
+  }
+
+  it('returns [] when the root does not exist', () => {
+    expect(discoverRepoPathsUnder(join(root, 'missing'))).toEqual([])
+  })
+
+  it('finds flat repos directly under the root', () => {
+    const a = makeRepo('repo-a')
+    const b = makeRepo('repo-b')
+    expect(discoverRepoPathsUnder(root).sort()).toEqual([a, b].sort())
+  })
+
+  it('finds org-style repos one level down (root/org/repo)', () => {
+    const flat = makeRepo('flat-repo')
+    const nested = makeRepo('my-org', 'nested-repo')
+    expect(discoverRepoPathsUnder(root).sort()).toEqual([flat, nested].sort())
+  })
+
+  it('does not recurse past depth 2', () => {
+    makeRepo('a', 'b', 'too-deep')
+    expect(discoverRepoPathsUnder(root)).toEqual([])
+  })
+
+  it('does not descend into repos looking for nested repos', () => {
+    const outer = makeRepo('outer')
+    mkdirSync(join(outer, 'vendor', 'inner', '.git'), { recursive: true })
+    expect(discoverRepoPathsUnder(root)).toEqual([outer])
+  })
+
+  it('ignores plain files at both levels', () => {
+    writeFileSync(join(root, 'README.md'), 'hi')
+    mkdirSync(join(root, 'org'))
+    writeFileSync(join(root, 'org', 'notes.txt'), 'hi')
+    expect(discoverRepoPathsUnder(root)).toEqual([])
+  })
+})

server/orchestrator-monitor.ts

@@ -293,29 +293,46 @@ export class OrchestratorMonitor {
     notification.delivered = true
   }
 
-  /** Discover repo paths from REPOS_ROOT. */
+  /** Discover repo paths from REPOS_ROOT (see discoverRepoPathsUnder). */
   private discoverRepoPaths(): string[] {
-    if (!existsSync(REPOS_ROOT)) return []
-    try {
-      return readdirSync(REPOS_ROOT)
-        .map(name => join(REPOS_ROOT, name))
-        .filter(p => {
-          try {
-            return statSync(p).isDirectory() && existsSync(join(p, '.git'))
-          } catch {
-            return false
-          }
-        })
-    } catch {
-      return []
-    }
+    return discoverRepoPathsUnder(REPOS_ROOT)
   }
 }
 
 // ---------------------------------------------------------------------------
 // Helpers
 // ---------------------------------------------------------------------------
 
+/**
+ * Discover git repositories under a root directory, recursing one level into
+ * non-repo directories so org-style layouts (root/org/repo) are picked up
+ * alongside flat ones (root/repo). Unreadable entries are skipped.
+ */
+export function discoverRepoPathsUnder(root: string): string[] {
+  if (!existsSync(root)) return []
+  const repos: string[] = []
+  const scan = (dir: string, depth: number): void => {
+    let entries: string[]
+    try {
+      entries = readdirSync(dir)
+    } catch {
+      return
+    }
+    for (const name of entries) {
+      const p = join(dir, name)
+      try {
+        if (!statSync(p).isDirectory()) continue
+        if (existsSync(join(p, '.git'))) repos.push(p)
+        else if (depth < 2) scan(p, depth + 1)
+      } catch {
+        // unreadable entry — skip
+      }
+    }
+  }
+  scan(root, 1)
+  return repos
+}
+
 /**
  * Suppress passive-repo alerts unless at least one workflow is enabled for the
  * repo — there's nothing to "de-schedule" otherwise, so the nag is noise.