feat(agent): stuck/runaway guard + surface the failure behind a max_turns cap (ABCA-662) (#600)

* feat(agent): stuck/runaway guard — break a repeating-failing-command loop

Live-caught ABCA-483: a one-line README task burned ALL 100 turns (~22 min,
$1.53) because the agent re-ran the SAME failing command (mise //cdk:test →
JS-heap OOM, exit 134) over and over, yak-shaving the build env instead of
finishing. Nothing noticed until the hard max_turns cap killed it.

New `stuck_guard.py` (pure, dep-free): PostToolUse records each tool result;
a coarse (tool_name, command) signature tracks consecutive failures. A new
between-turns hook (registered after cancel, before nudge — same
short-circuit rationale) then:
 - STEER once at 3 repeats: inject 'stop retrying X — work around it or
 finish with what you have';
 - BAIL at 6 repeats: end the turn loop early (continue_=False) so the task
 fails fast instead of grinding to the cap.
Conservative by design: keys on a REPEATING FAILURE (not a raw turn count),
so a large task making varied progress never trips it; unknown tool output
counts as success. Fail-open everywhere — a guard bug can't wedge the agent.

Platform-agnostic, MAIN-BOUND: lives entirely in agent/src (stuck_guard.py +
hooks.py); surfaces via the channel-neutral error_message. The honest reason
is then rendered per-channel by the classifier / failure-reply (several fixes).

1158 agent tests pass (ruff + ty clean).

* fix: stuck-guard is advisory-only — drop the bail, keep a one-time steer

Reviewing the guard for false-positive risk (user: 'make sure the turn limit
isn't too aggressive') showed the bail path was unsafe: distinguishing a true
spin from a legitimately-iterating agent (re-running the same test command as
it fixes failures one by one) from raw output is genuinely fragile — a digit-
normalizing fingerprint that ignores volatile GC timings ALSO collapses
'test file_0' vs 'file_1' (the progress signal), so it would kill a working
agent. A false-positive KILL is far worse than a false-positive nudge.

So: removed BAIL entirely. The guard now only ever injects a ONE-TIME advisory
steer when the same command fails with IDENTICAL output N times; the max_turns
cap (with a fix's honest 'Exceeded max turns' reason) is the real runaway backstop.
A false positive costs exactly one extra advisory comment. Platform-agnostic,
main-bound.

* fix(agent): explain WHY a task hit max_turns + catch loop-of-variations early (ABCA-662)

ABCA-662 capped at max_turns, but the CAUSE was masked: it had finished the code
and then thrashed ~15 turns retrying a failing 'git push' (remote: invalid
credentials) every which way — a DIFFERENT command each turn, so the existing
per-signature stuck-guard (same command × identical output ≥3) never tripped,
and the terminal reason read only 'Exceeded max turns'. max_turns is a CORRECT
classification, but the user can't tell 'genuinely needed more turns' from 'spun
on an error' — and raising the cap just burns more tokens.

Two changes, both advisory (no hard kill — keeps the K10 no-false-kill stance):
- stuck_guard: add a signature-agnostic TRAILING WINDOW (last 6 tool outcomes).
  When ≥5 share the SAME byte-identical failure fingerprint across DIFFERENT
  commands, steer once ('you're spinning on one failing op — stop, it won't
  resolve by retrying'). EXACT fingerprint (no digit-blur) so a healthy
  iterate-and-fix loop (same cmd, a different test failing each run) never trips
  it — the K10 case, still covered by its test. Also exposes recent_failure_
  summary().
- pipeline/hooks: the between-turns hook latches that summary; the terminal path
  appends it to a max_turns reason → 'error_max_turns … — spinning on failing
  tool calls (last: git push → invalid credentials)'. Only enriches max_turns;
  a productive run adds nothing.
- error-classifier: a new bucket for 'error_max_turns … spinning on failing tool
  calls' → 'Ran out of turns while stuck on a failing step' with the honest
  remedy (raising --max-turns won't help; fix the failing op / check the env),
  ordered before the generic max_turns bucket. Tests: window-spin + K10 no-trip
  + summary + pipeline enrichment + classifier bucket.

* fix(errors): don't over-claim on a max_turns spin — offer BOTH remedies (ABCA-662 review)

The prior 'spinning on failing tool calls' classifier bucket asserted 'raising
--max-turns won't help'. That over-claims: the trailing window can't distinguish
(a) a HARD blocker no turn count fixes from (b) a LONG task that hit a
transient/recoverable snag near the end and just ran out — which is 662 itself
(siblings 661/663 pushed fine with the same token → its 'invalid credentials'
was a transient race that more turns / a retry WOULD have cleared). Reframe the
bucket to SURFACE what it was stuck on and present BOTH paths (environment
blocker → fix + retry; transient/recoverable or a shorter sibling got past it →
just retry / raise --max-turns), letting the failure KIND in the detail tell the
reader which applies. Title 'Ran out of turns retrying a failing step'. The
early window-steer (agent told to STOP and report while it still has turns) is
the real mechanism that lets a long task escape the thrash; this copy is just
the honest post-hoc explanation. Test updated to assert both remedies, not the
over-claim.

* fix(errors): max_turns stays "Exceeded max turns" — observe the repeated failure, don't diagnose it

Reviewer concern on the ABCA-662 work: re-titling a max_turns failure as "Ran out
of turns retrying a failing step" over-claims. The stuck-guard's trailing window is
only the last ~6 tool calls, which genuinely CANNOT distinguish a hard blocker (bad
creds, no permission — more turns won't help) from a long task that made real
progress and hit a recoverable snag only at the tail (more turns / a retry WOULD
help — 662 itself: siblings pushed fine with the same token). Framing the whole run
around its last 6 calls misrepresents the latter.

- error-classifier: drop the separate "retrying a failing step" bucket. A max_turns
  failure stays the plain "Exceeded max turns"; the description/remedy point the
  reader at the observed detail and still surface the environment-blocker path,
  but assert NO cause.
- stuck_guard.recent_failure_summary: emit a NEUTRAL observation ("last tool calls
  repeated: `<cmd>` → <err>") instead of "spinning on failing tool calls". The
  mid-run steer TO the agent (an advisory nudge, not a user surface) still says
  "spinning" — that's fine, it's coaching the agent, not classifying the outcome.
- tests updated to pin the neutral wording + assert the title is NOT re-framed.

Author: isadeks

agent/src/hooks.py

@@ -34,6 +34,7 @@
 from policy import APPROVAL_RATE_LIMIT, FLOOR_TIMEOUT_S, Outcome
 from progress_writer import _generate_ulid
 from shell import log, log_error_cw
+from stuck_guard import StuckGuard
 
 if TYPE_CHECKING:
     from policy import PolicyEngine
@@ -172,6 +173,27 @@ def reset_blocker_reason() -> None:
 _reset_blocker_reason_for_tests = reset_blocker_reason
 
 
+# ABCA-662: latch of the stuck-guard's "why it's spinning" summary, refreshed by
+# the between-turns hook. Read by the pipeline's terminal path so a max_turns
+# failure can say WHY it capped ("Exceeded max turns — spinning on failing tool
+# calls: git push → invalid credentials") vs. a task that genuinely used its
+# turns. Process-lifetime (one task), last-writer-wins (the most recent window
+# is the most relevant to the cap). Distinct from the blocker latch: this is a
+# SOFT diagnostic (advisory), not a canonical BLOCKED[…] terminal reason.
+_LAST_STUCK_SUMMARY: str | None = None
+
+
+def last_stuck_summary() -> str | None:
+    """Return the latched stuck-guard summary for the max_turns terminal reason."""
+    return _LAST_STUCK_SUMMARY
+
+
+def reset_stuck_summary() -> None:
+    """Clear the stuck-summary latch (per-task, alongside the blocker latch)."""
+    global _LAST_STUCK_SUMMARY
+    _LAST_STUCK_SUMMARY = None
+
+
 def detect_egress_denial(text: str) -> tuple[bool, str | None]:
     """Scan tool output for an egress-denial signature (#251).
 
@@ -1039,6 +1061,7 @@ async def post_tool_use_hook(
     *,
     trajectory: _TrajectoryWriter | None = None,
     progress: Any = None,
+    stuck_guard: StuckGuard | None = None,
 ) -> dict:
     """PostToolUse hook: screen tool output for secrets/PII.
 
@@ -1051,6 +1074,11 @@ async def post_tool_use_hook(
     present, an egress-denial signature in the tool output emits an
     ``egress_denied`` blocker event (#251) — best-effort observability,
     never alters the screening decision.
+
+    K7: when a ``stuck_guard`` is supplied, every tool result is recorded so a
+    between-turns hook can detect a repeating failing command (the ABCA-483
+    spin loop) and steer / bail. Recording is best-effort and never alters the
+    screening outcome.
     """
     _PASS_THROUGH: dict = {"hookSpecificOutput": {"hookEventName": "PostToolUse"}}
     _FAIL_CLOSED: dict = {
@@ -1113,6 +1141,14 @@ async def post_tool_use_hook(
                 resource=host,
             )
 
+    # K7: feed the stuck-guard (best-effort — a tracking error must never block
+    # the screening path that follows).
+    if stuck_guard is not None:
+        try:
+            stuck_guard.record_tool_result(tool_name, hook_input.get("tool_input"), tool_response)
+        except Exception as exc:
+            log("WARN", f"stuck-guard record raised (ignored): {type(exc).__name__}: {exc}")
+
     try:
         result = scan_tool_output(tool_response)
     except Exception as exc:
@@ -1395,6 +1431,49 @@ def _cancel_between_turns_hook(ctx: dict) -> list[str]:
     return []
 
 
+def _stuck_guard_between_turns_hook(ctx: dict) -> list[str]:
+    """K7: nudge the agent when it repeats the SAME failing command (ABCA-483).
+
+    Reads the per-task :class:`StuckGuard` stamped on ``ctx`` (by
+    :func:`stop_hook`). When the same command has failed with identical output
+    enough times in a row, the guard returns a ``steer`` action — a ONE-TIME
+    advisory message telling the agent to stop retrying and either work around
+    the failure or finish with what it has. Returned as injected text, so the
+    SDK continues the turn with the steer as the next user message.
+
+    ADVISORY ONLY: the guard never kills the task (the bail path was removed —
+    distinguishing a true spin from a legitimately-iterating agent is too
+    fragile to justify an auto-kill; the ``max_turns`` cap is the real
+    backstop). A false positive here costs exactly one extra advisory comment.
+
+    Runs AFTER cancel (cancel wins — never steer a dying agent) but its own
+    no-op-when-cancelled guard makes ordering robust. Fail-open: any error is
+    swallowed so a guard bug can never wedge a healthy agent.
+    """
+    if ctx.get("_cancel_requested"):
+        return []
+    guard = ctx.get("stuck_guard")
+    if guard is None:
+        return []
+    try:
+        action = guard.evaluate()
+        # ABCA-662: refresh the "why it's spinning" latch every turn. When the
+        # trailing window is failure-dominated this returns a one-liner; otherwise
+        # None (which clears the latch — a task that recovered isn't "stuck"). Read
+        # by the terminal path so a later max_turns cap explains itself.
+        global _LAST_STUCK_SUMMARY
+        _LAST_STUCK_SUMMARY = guard.recent_failure_summary()
+    except Exception as exc:
+        log("WARN", f"stuck-guard evaluate raised (ignored): {type(exc).__name__}: {exc}")
+        return []
+
+    if action.kind == "steer":
+        _emit_nudge_milestone(ctx, "stuck_steer", action.message[:_NUDGE_PREVIEW_LEN])
+        log("NUDGE", f"stuck-guard STEER injected: {action.signature}")
+        return [action.message]
+    return []
+
+
 # Global list of between-turns hooks.  Cancel MUST run first so it can
 # short-circuit nudges on cancelled tasks (no point injecting nudges into a
 # dying agent — worse, the nudge reader mutates DDB state that the agent will
@@ -1406,6 +1485,10 @@ def _cancel_between_turns_hook(ctx: dict) -> list[str]:
 # nudge reader to preserve cancel-wins semantics.
 between_turns_hooks: list[BetweenTurnsHook] = [
     _cancel_between_turns_hook,
+    # K7 stuck-guard (advisory): injects a one-time "stop retrying X" nudge when
+    # the same command keeps failing identically. Runs after cancel (never steer
+    # a dying agent); order vs nudge/denial is cosmetic since it never bails.
+    _stuck_guard_between_turns_hook,
     _nudge_between_turns_hook,
     # Chunk 3 (finding #2): denial injection runs LAST so both cancel and
     # nudge short-circuits pre-empt it. The hook explicitly re-checks
@@ -1423,6 +1506,7 @@ async def stop_hook(
     task_id: str,
     progress: Any = None,
     engine: Any = None,
+    stuck_guard: Any = None,
 ) -> dict:
     """Stop hook: run registered between-turns hooks; block if they produce text.
 
@@ -1445,6 +1529,7 @@ async def stop_hook(
         "task_id": task_id,
         "progress": progress,
         "engine": engine,
+        "stuck_guard": stuck_guard,
     }
 
     # Cancel-before-nudge short-circuit.
@@ -1526,6 +1611,11 @@ def build_hook_matchers(
         SyncHookJSONOutput,
     )
 
+    # K7: one stuck-guard per task (== per build_hook_matchers call). The
+    # PostToolUse closure feeds it every tool result; the Stop closure reads it
+    # between turns to steer / bail on a repeating failing command.
+    _stuck_guard = StuckGuard()
+
     # Closure-based wrapper matches the HookCallback signature exactly:
     # (HookInput, str | None, HookContext) -> Awaitable[HookJSONOutput]
     async def _pre(
@@ -1568,7 +1658,12 @@ async def _post(
     ) -> HookJSONOutput:
         try:
             result = await post_tool_use_hook(
-                hook_input, tool_use_id, ctx, trajectory=trajectory, progress=progress
+                hook_input,
+                tool_use_id,
+                ctx,
+                trajectory=trajectory,
+                progress=progress,
+                stuck_guard=_stuck_guard,
             )
             return SyncHookJSONOutput(**result)
         except Exception as exc:
@@ -1593,6 +1688,7 @@ async def _stop(
                 task_id=stop_task_id,
                 progress=progress,
                 engine=engine,
+                stuck_guard=_stuck_guard,
             )
         except Exception as exc:
             log(

agent/src/pipeline.py

@@ -466,6 +466,21 @@ def _resolve_overall_task_status(
     agent_status = agent_result.status
     err = agent_result.error
 
+    # ABCA-662: a max_turns cap is a CORRECT classification, but on its own it
+    # doesn't say WHETHER the task genuinely needed the turns or SPUN on a failing
+    # operation until it ran out (662 thrashed on a failing `git push` → invalid
+    # credentials, retried every which way, and capped). When the stuck-guard's
+    # trailing window was failure-dominated, append its one-line summary so the
+    # reason distinguishes "ran long" from "looped on an error" — the classifier
+    # still buckets it as max_turns, but a human sees the real cause. Only enriches
+    # the max_turns reason; a task that used its turns productively adds nothing.
+    if err and "error_max_turns" in err:
+        from hooks import last_stuck_summary
+
+        stuck = last_stuck_summary()
+        if stuck and stuck not in err:
+            err = f"{err} — {stuck}"
+
     if agent_status in ("success", "end_turn") and build_ok:
         return "success", err
 
@@ -696,9 +711,12 @@ def run_task(
         # in principle dispatch a second run_task in the same process — reset
         # here so a stale BLOCKED[...] reason can never leak into this task's
         # terminal error_message (the latch is a scalar, not task_id-keyed).
-        from hooks import reset_blocker_reason
+        from hooks import reset_blocker_reason, reset_stuck_summary
 
         reset_blocker_reason()
+        # ABCA-662: same per-task reset for the stuck-guard recent-failure latch,
+        # so a prior task's observation can't leak into this task's max_turns copy.
+        reset_stuck_summary()
         # --trace accumulator (design §10.1): when the task opted into
         # trace, ``_TrajectoryWriter`` keeps an in-memory copy of each
         # event so the pipeline can gzip+upload the full trajectory to