Skip to content
This repository was archived by the owner on Apr 11, 2026. It is now read-only.

Commit 85b501e

Browse files
committed
feat(ops): add dual-tier worker timeout, deadlock detection, and workflow fixes
- Add timeout: 2700000 (45min) to Agent() call in phases.md Step 7 - Add timeout handling in Step 8 with flowctl fail for timed-out workers - Add heartbeat instruction to worker Phase 5 (every 60s) - Add deadlock detection in Step 3 with cause classification - Add no-progress watchdog (stops after 2 unchanged waves) - Add spec hash comparison at wave start with warning on mismatch - Change approval timeout from silent skip to flowctl block Task: fn-110-workflow-reliability-and-performance.5
1 parent 178fe62 commit 85b501e

2 files changed

Lines changed: 66 additions & 3 deletions

File tree

agents/worker.md

Lines changed: 19 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -89,7 +89,7 @@ SendMessage(to: "coordinator", summary: "Blocked: <TASK_ID>",
8989
```
9090
- On `status: approved` → proceed with the edit.
9191
- On `status: rejected` → emit a `Blocked:` summary and skip the file.
92-
- On timeout → note in completion evidence and continue with alternative approach.
92+
- On timeout → mark task as blocked: `$FLOWCTL block <TASK_ID> "Approval timeout: requested access to <path>"` and continue with alternative approach.
9393

9494
**Via SendMessage (non-Teams mode):**
9595
```
@@ -192,6 +192,16 @@ For each entry returned, fetch its content and include verbatim in your context:
192192
```
193193
These are lightweight narrative handoffs from earlier tasks in this epic — read them to understand what upstream work surprised the previous worker, what decisions they made, and what gotchas to watch for. Skip gracefully if the list is empty (new epic) or if `outputs.enabled` is false.
194194

195+
**Spec hash verification (mid-wave protection):**
196+
If the coordinator passed a `SPEC_HASH` value in your prompt, compare it against the current spec:
197+
```bash
198+
CURRENT_HASH=$(echo "$(<FLOWCTL> cat <TASK_ID>)" | shasum -a 256 | cut -d' ' -f1)
199+
if [ "$CURRENT_HASH" != "$SPEC_HASH" ]; then
200+
echo "Warning: spec for <TASK_ID> changed since wave start (hash mismatch)"
201+
fi
202+
```
203+
Continue execution but note the mismatch in evidence.
204+
195205
Parse the spec carefully. Identify:
196206
- Acceptance criteria
197207
- Dependencies on other tasks
@@ -320,6 +330,12 @@ echo "BASE_COMMIT=$BASE_COMMIT"
320330
```
321331
Save this - you'll pass it to impl-review so it only reviews THIS task's changes.
322332

333+
**Heartbeat signaling:** Every 60 seconds during implementation, call:
334+
```bash
335+
$FLOWCTL heartbeat --task $TASK_ID
336+
```
337+
This signals liveness to the coordinator. The coordinator checks heartbeats every 3 minutes. If no heartbeat is received within that window, the worker is considered hung and may be terminated.
338+
323339
### Wave-Checkpoint-Wave Execution
324340

325341
When a task touches **3+ files**, use the Wave pattern to parallelize file I/O. This achieves 3-4x speedup over sequential reads/edits.
@@ -376,7 +392,7 @@ If more files remain (tests, docs, config), repeat: parallel read → checkpoint
376392
message: "Access request for <TASK_ID>.\nFile: <path>\nReason: <why needed>\nCurrent owner: <task-id if known>")
377393
```
378394
2. Wait for `status: approved`/`rejected` (API) or "Access granted:"/"Access denied:" (fallback)
379-
3. If timeout, skip the file and note it in your completion evidence
395+
3. If timeout, mark task as blocked: `$FLOWCTL block <TASK_ID> "Approval timeout: requested access to <path>"` and continue with alternative approach
380396
4. On rejected/denied, find an alternative approach that stays within your owned files
381397

382398
**This is not optional.** Do not bypass this check even if you believe the lock system will catch violations. Self-enforcement is the primary guard; hooks are the backup.
@@ -719,7 +735,7 @@ If you catch yourself thinking any of these, stop and follow the correct action:
719735
| Thought | Reality |
720736
|---------|---------|
721737
| "I need to edit a file not in OWNED_FILES" | Create a `flowctl approval create --kind file_access` (or fallback "Need file access:" message) and WAIT. Do not edit. |
722-
| "The coordinator isn't responding" | `approval show --wait --timeout 600` blocks ≤10min; on fallback path wait 60s. If timeout, skip the file and note it in evidence. |
738+
| "The coordinator isn't responding" | `approval show --wait --timeout 600` blocks ≤10min; on fallback path wait 60s. If timeout, call `$FLOWCTL block <TASK_ID> "Approval timeout: requested access to <file>"` and continue with alternative approach. |
723739
| "I'll just edit it, the lock check will catch it" | Don't rely on hooks. Self-enforce OWNED_FILES. |
724740
| "TEAM_MODE doesn't matter for this task" | If TEAM_MODE=true is set, follow the protocol. Always. |
725741
| "It's a small edit, nobody will notice" | Ownership violations break parallel safety for everyone. |

skills/flow-code-work/phases.md

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -134,6 +134,34 @@ After restarts, find ready tasks normally:
134134
$FLOWCTL ready --epic <epic-id> --json
135135
```
136136

137+
**Deadlock detection:** If ready tasks = 0 AND no tasks are in_progress, check for stalled state:
138+
139+
```bash
140+
TASKS_JSON=$($FLOWCTL tasks --epic <epic-id> --json)
141+
FAILED=$(echo "$TASKS_JSON" | jq '[.[] | select(.status=="failed")] | length')
142+
BLOCKED=$(echo "$TASKS_JSON" | jq '[.[] | select(.status=="blocked")] | length')
143+
TODO_WITH_DEPS=$(echo "$TASKS_JSON" | jq '[.[] | select(.status=="todo" and (.dependencies | length > 0))] | length')
144+
```
145+
146+
Classify the stall:
147+
- If `FAILED > 0`: "Epic stalled: $FAILED tasks failed, blocking downstream. Consider restart or skip."
148+
- If `BLOCKED > 0`: "Epic stalled: $BLOCKED tasks blocked. Check block reasons."
149+
- If all remaining are `todo` with unmet deps: "Possible circular dependency. Run `$FLOWCTL validate --epic <epic-id>`."
150+
151+
Stop the wave loop in any of these cases.
152+
153+
**No-progress watchdog:** Track `completed_count` at the start of each wave:
154+
155+
```bash
156+
COMPLETED_COUNT=$(echo "$TASKS_JSON" | jq '[.[] | select(.status=="done")] | length')
157+
```
158+
159+
If `completed_count` is unchanged after 2 consecutive waves:
160+
```
161+
"No progress detected across 2 waves. Stopping."
162+
```
163+
Stop the wave loop.
164+
137165
Collect ALL ready tasks (no unresolved dependencies). If no ready tasks, check for completion review gate (see Step 10 below).
138166

139167
### Step 4. Readiness Check
@@ -144,6 +172,18 @@ Before starting, validate each task spec is implementation-ready:
144172
$FLOWCTL cat <task-id>
145173
```
146174

175+
**Spec hash snapshot (mid-wave protection):**
176+
177+
Record a content hash for each task spec at wave start:
178+
```bash
179+
SPEC_HASH_<task-id>=$(echo "$($FLOWCTL cat <task-id>)" | shasum -a 256 | cut -d' ' -f1)
180+
```
181+
Workers compare this hash during re-anchor (Phase 2). If the spec changed since wave start, log:
182+
```
183+
"Warning: spec for <task-id> changed since wave start (hash mismatch)"
184+
```
185+
Continue execution but note the mismatch in evidence.
186+
147187
**Check these fields exist and are non-empty:**
148188
- `## Description` — what to build (not just a title)
149189
- `## Acceptance` — at least one testable `- [ ]` criterion
@@ -238,6 +278,7 @@ Agent({
238278
team_name: "flow-<epic-id>",
239279
isolation: "worktree",
240280
run_in_background: true,
281+
timeout: 2700000,
241282
prompt: "$WORKER_PROMPT
242283
243284
TASK_ID: <task-id>
@@ -263,6 +304,12 @@ Spawn ALL ready task workers in a SINGLE message with multiple Agent tool calls.
263304

264305
Wait for all workers to complete.
265306

307+
**Worker timeout handling:** If a worker times out (exceeds the 45-minute ceiling):
308+
```bash
309+
$FLOWCTL fail <task-id> "Worker timeout after 45min"
310+
```
311+
This triggers the retry logic in the task lifecycle (`up_for_retry` if retries remaining). Log the timeout and continue processing other workers in the wave.
312+
266313
**Merge-back** (after all workers return):
267314

268315
```bash

0 commit comments

Comments
 (0)