You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Implementation run impl-20260705-184231-936209 for #4189 (autoskillit fleet run CLI) was killed at the pre-merge fix step. Part A of a two-part plan registered the fleet run subcommand while deletion of the pre-existing deletion-guard test test_fleet_run_command_not_registered (tests/cli/test_fleet_list.py:100) was deferred to Part B — so Part A's local test gate could never pass. resolve-failures fixed 3 of 4 failures, then emitted verdict=ci_only_failure for the remaining locally-reproducible canary failure, and the recipe routed to release_issue_failure → register_clone_failure → escalate_stop.
Root cause — four stacked gaps (one per layer)
make-plan (origin): the 500-line split trigger + "implementation vs. tests" boundary stranded the canary deletion in Part B. Multi-part rules (make-plan/SKILL.md lines 337–343) are scope-isolation only — no invariant that each part must leave task test-check green. The planner's own Registry Tracer flagged the canary CRITICAL (F1) pre-split; the split put the fix on the wrong side and no rule caught it.
dry-walkthrough: read Part A's own admission ("task test-all passes with the single exception of test_fleet_run_command_not_registered") and recorded it under Verified, then stamped the plan "Implement as-is". It checks the test command name (SKILL.md lines 149–156), never the predicted outcome. No pipeline-gate compatibility check exists.
resolve-failures: no verdict exists for a plan-acknowledged deferred failure. The session anchored on the weakened invariant "ci_only_failure MAY be emitted when fixes_applied >= 1…" and skipped the "…if the remaining failures are all CI-only" qualifier. Per the skill's own decision table (lines 186–196) every ci_only_failure row requires Local result = PASS — the canary fails locally (verified empirically in the worktree). It also exited the fix loop after 2 of 3 iterations via the tests-green reporting path (Step 4) instead of Step 5 (Report Failure).
Recipe routing: the pre-merge fix step (implementation.yaml:661–698) is reachable only via a local test failure before any CI run — an honest ci_only_failure is mechanically impossible there. The route functions purely as an unconditional kill for misclassifications. The same trap exists in merge_gate_fix, implementation-groups.yaml, remediation.yaml, and research-implement.yaml.
The regression test test_ci_only_failure_requires_no_fix_applied (tests/skills/test_resolve_failures_ci_aware.py:292) uses DOTALL regexes across the whole document and still passes with the weakened text — verified against current SKILL.md.
check_test_passed ignores xfailed counts (verified empirically: exit 0 + "45 xfailed" → gate passes; locked in by tests/server/test_tools_workspace.py:181). Marking the canary xfail(strict=True) in Part A would have passed the gate with forced cleanup semantics. No skill guidance mentions this mechanism.
Recommended remediation
make-plan: add a multi-part CRITICAL rule — every part must independently pass task test-check; any pre-existing test invalidated by a part's changes must be updated/removed/xfail-bridged in that same part; split boundaries keep gate-prerequisites with the code that triggers them.
dry-walkthrough: blocking FAIL when the plan text declares any test expected to fail after the part's implementation (detectable from a single part; no cross-part read required).
resolve-failures: restore the hard invariant (give partial-fix-with-CI-only-remainder its own verdict if genuinely needed); mandate Step 5 (Report Failure) as the sole exit when tests remain red with iterations remaining; optionally add a plan_blocked verdict for plan-acknowledged deferred failures.
Contracts/orchestrator: enum-constrain verdict in skill contracts (currently unconstrained type: string); reject or re-route ci_only_failure emitted from steps reachable only via local test failure — mechanical precondition validation instead of prose trust.
Test hardening: pin the invariant paragraph in test_resolve_failures_ci_aware.py with an anchored, paragraph-scoped assertion so the next prose softening fails CI.
Artifacts
Failed run clone (preserved, status=error): /home/talon/projects/autoskillit-runs/impl-20260705-184231-936209
Part A worktree (3 salvageable commits 671e7e325, 999ef83c2, 58d5bbc30): /home/talon/projects/autoskillit-runs/worktrees/impl-4189-20260705-192411
Prior investigation completed interactively. See below for root cause analysis.
Investigation: Pipeline Failure on Issue #4189 — Part-Split Canary Test vs. Per-Part Test Gate
Date: 2026-07-05 Scope: Implementation pipeline run impl-20260705-184231-936209 for issue #4189 (autoskillit fleet run CLI). Failure at the fix step: resolve-failures returned verdict=ci_only_failure for a locally-failing, plan-acknowledged canary test, routing the run to release_issue_failure → escalate_stop. Mode: Standard (extended: 9 subagents + direct empirical verification)
Summary
The run was killed by a stack of four independent gaps, one per pipeline layer, none of which is a "bug" in isolation. (1) Planning: make-plan split a 644-line plan at the mechanical 500-line trigger using an "implementation vs. tests" boundary, which stranded the removal of a pre-existing deletion-guard test (test_fleet_run_command_not_registered) in Part B while Part A registers the very command the guard forbids — guaranteeing a red test gate after Part A. make-plan has no rule that each part must independently pass the test gate. (2) Validation: dry-walkthrough read the plan's own admission that the test would stay red, recorded it under "Verified" as a correct factual statement, and stamped the plan — it has no pipeline-gate compatibility check. (3) Execution: resolve-failures fixed 3 of 4 failures, then faced a failure it correctly understood as plan-deferred but had no verdict for; it misapplied a previously-weakened invariant and emitted ci_only_failure for a failure that reproduces locally — definitionally wrong per its own decision table. (4) Routing: in the pre-merge fix step, which is only ever reached via a local test failure before any CI run, ci_only_failure can never be legitimate; its route is a hard-kill misclassification sink. The verdict misuse is a three-generation recurring regression (#1954 fixed → #3610 fix weakened the invariant → today's misread), and the regression test guarding the invariant is too broad to detect the weakening (empirically verified).
Root Cause
Layered, in causal order:
RC1 — make-plan split invariant gap (primary origin). The plan session's adversarial Registry Tracer correctly flagged the canary as CRITICAL (F1) and the fix (Step 4: delete the test) was added to the unsplit draft. When the 500-line rule forced a split, the planner categorized Step 4 as "test infrastructure" and moved it to Part B. The split rules in make-plan/SKILL.md (verified at lines 337–343) are exclusively about scope isolation — there is no "each part must leave the gate green" invariant, so no rule was violated. Part A's own Verification section (line 357 of the plan) then stated the failure as an accepted outcome. [SUPPORTED — SKILL.md text, plan files, plan session log]
RC2 — dry-walkthrough has no gate-compatibility check. The verify session explicitly wrote: "test_fleet_run_command_not_registered … is the single expected failure, exactly as the plan states (removal deferred to Part B)" — under its Verified section — then stamped the plan and recommended "Implement as-is." Its checklist checks the test command name, never the predicted test outcome. Its multi-part isolation rule ("MUST NOT read any other part files") also prevents cross-part reasoning, but no cross-part read is needed: the plan's own text declaring a post-part red test is detectable from Part A alone. [SUPPORTED — verify session JSONL, SKILL.md lines 88–156]
RC3 — resolve-failures verdict vocabulary gap + invariant misapplication. The fix session identified the canary as plan-deferred on its first reasoning turn, fixed the 3 unrelated failures (commits 999ef83c2, 58d5bbc30), re-ran tests (28,077 passed / 1 failed), then emitted ci_only_failure with zero deliberation over alternatives. Per the skill's own decision table (lines 186–196), every ci_only_failure row requires Local result = PASS; the post-fix invariant (lines 333–336) permits it with fixes_applied >= 1 only "if the remaining failures … are all CI-only." The canary fails locally — I verified this empirically in the worktree ('run' in subcommands → assertion False). The agent anchored on fixes_applied=2 >= 1 and skipped the CI-only qualifier. Secondary violations: it exited the fix loop after 2 of 3 allowed iterations with red tests, and used the Step 4 (tests-green) reporting path instead of Step 5 (Report Failure). [SUPPORTED — fix session JSONL, SKILL.md, empirical check]
RC4 — pre-merge ci_only_failure route is a semantic trap. The fix step (implementation.yaml:661–698) is reached exclusively after a localtest_check failure; no CI has run and the CI context refs are empty in this path. A verdict meaning "failure only reproducible in CI" is therefore mechanically impossible if emitted honestly — the route ci_only_failure → release_issue_failure in this step functions only as an unconditional kill for misclassifications. The same trap exists in merge_gate_fix and in implementation-groups.yaml, remediation.yaml, research-implement.yaml. [SUPPORTED — recipe YAML, routing analysis]
src/autoskillit/recipes/contracts/implementation.yaml (line ~131): verdict is unconstrained type: string, no enum [SUPPORTED]
src/autoskillit/execution/testing.py (check_test_passed, 243–265): binary gate; xfailed does not fail — empirically verified [SUPPORTED]
tests/skills/test_resolve_failures_ci_aware.py (line 292): DOTALL regex passes despite invariant weakening — verified by reading the regex against current SKILL.md text [SUPPORTED]
plan captures plan_parts → per part: verify (dry-walkthrough stamps) → create_impl_worktree → implement (skill forbids running tests; canary failure invisible here by design) → test (test_check → task test-check, binary, local) → on failure check_test_fix_loop → fix (resolve-failures emits free-string verdict) → on_result routing. Part A merges to the feature branch (merge → inter_part_push) before Part B begins (next_or_done), so the Part A test gate runs with zero awareness that Part B exists. The verdict string flows uncheck-ed (no enum, no precondition validation) into routing that terminates the run.
Test Gap Analysis
The invariant regression test cannot detect its own subject regressing.test_ci_only_failure_requires_no_fix_applied matches (no fix|never.*fix|fixes_applied.*0).*ci_only_failure with re.DOTALL over the whole document; the Step 2d header text ("Applies ONLY when no fix was applied (fixes_applied == 0)") thousands of characters away from the weakened invariant keeps it green. The 476059f weakening shipped without any test failing. [SUPPORTED — verified against current text]
No contract-level verdict validation. The skill contract declares verdict: type: string with no enum, so any string — including a semantically impossible one — passes validation and routes. [SUPPORTED]
No recipe semantic rule checks that a verdict-gated route's verdict is satisfiable in its context (e.g., ci_only_failure routed from a step reachable only via local test failure). [SUPPORTED]
No test exercises the multi-part red-gate scenario end-to-end (plan part declaring an expected failure → pipeline behavior). [SUPPORTED — absence via grep across recipes/server/skills]
Similar Patterns
xfail(strict=True) is the existing, proven bridge mechanism.check_test_passed ignores xfailed counts (empirically verified: check_test_passed(0, "…45 xfailed…") == True), the suite already uses xfail in 8+ files, and tests/server/test_tools_workspace.py:181 explicitly locks in that xfailed ≠ failure. Marking the canary xfail(strict=True, reason="removed in Part B") in Part A would have passed the gate, and strict=True forces cleanup. [SUPPORTED]
The canary pytest marker is a different concept ("development-time filter verification tests", pyproject line 106) — excluded from all task test-check variants but not intended for deletion guards. [SUPPORTED]
Deletion-guard precedent: the run_skill_retry guards were removed in later test-audit passes (3a43a0df0, 40ae65b80) — valid only for permanently-deleted symbols. For intentional re-registration (this case), the guard must die in the same commit that re-registers the name. Part A's commit 671e7e325 registered run and left the guard untouched. [SUPPORTED]
Design Intent Findings
test_fleet_run_command_not_registered: introduced in 7dda8dc81 (2026-04-25, PR Implementation Plan: Fleet Restructure T3: CLI Split — fleet dispatch + fleet campaign #1242, human-authored) as a deletion guard when fleet run was split into fleet dispatch/fleet campaign. No docstring, no removal condition, no marker. Depended on by nothing — it IS the dependency (a one-sided "name must stay absent" invariant). The pattern is undocumented anywhere in docs/AGENTS.md. [SUPPORTED]
Pre-merge fix verdict routing: inherited verbatim from the post-CI resolve_ci context where ci_only_failure → release_issue_failure is coherent; never adapted for the local-only pre-merge context. [SUPPORTED]
Historical Context
This is a recurring pattern — at least the 4th manifestation of "verdict precondition not met → terminal routing kills pipeline":
Prior fixes were prose-layer (SKILL.md edits) guarded by regexes too broad to detect softening; no layer mechanically validates verdict preconditions. This is a recurring pattern — consider running /rectify for architectural immunity after resolving the immediate issue.
External Research
GitHub issue search (TalonT-Org/AutoSkillit) surfaced the historical issues above; issue #4189's body contains no mention of the canary or a two-part split — the split was purely a planning-session decision. No external library/web factors are implicated (failure is entirely internal orchestration logic).
Scope Boundary
Investigated: make-plan / review-approach / dry-walkthrough / implement-worktree-no-merge / resolve-failures skill texts; all five session logs of the failed run; implementation.yaml + implementation-groups.yaml routing; test_check/check_test_passed code paths (empirically); canary origin and precedents in the AutoSkillit repo; verdict invariant git lineage; prior related issues and fixes. Not yet explored: whether the open scope of #3610 overlaps further; live multi-part runs at risk in remediation.yaml/research-implement.yaml (same routing shape, unexamined runs); Codex-backend variants of the fix path; the review_pr/audit_impl stages (never reached); the implementer's separate pre-commit rule violation ("pre-existing" contract-freshness blame) — noted but not chased.
Confidence Levels
All findings above marked inline. Every load-bearing claim (canary fails locally; xfailed passes the gate; ci_only_failure rows require local PASS; invariant weakened in 476059fbb; regression regex blind to the weakening; make-plan lacks green-gate rule) was verified directly by me — code read, diff read, or executed — not only by subagents: SUPPORTED. The only NEEDS-EVIDENCE item: why the fix agent skipped the third loop iteration (its transcript records no rationale; the early exit is observed fact, the motivation is inference).
Retry the issue with the plan corrected: move canary deletion (Part B Step 4) into Part A — the precedented same-commit removal for intentional re-registration — or, if the parts must stay as-is, add pytest.mark.xfail(strict=True, reason="canary removed in Part B") in Part A and delete the test in Part B. The preserved worktree's 3 commits are salvageable. Remove the fail label on retry.
Systemic (defense per layer, in priority order):
2. make-plan: add a fifth CRITICAL multi-part rule — every part must independently leave task test-check green; any pre-existing test invalidated by a part's changes must be updated/removed/xfail-bridged in that same part; split boundaries must keep gate-prerequisites with the code that triggers them.
3. dry-walkthrough: add a blocking check — if the plan text states any test is expected to fail after this part's implementation, emit Dry Walkthrough FAILED. Detectable from a single part; no cross-part read required.
4. resolve-failures: restore the hard invariant ("NEVER when fixes_applied >= 1"), giving the partial-fix-with-CI-only-remainder case its own verdict if genuinely needed; require Step 5 (Report Failure) as the sole exit when tests remain red and iterations remain; optionally add a plan_blocked verdict for plan-acknowledged deferred failures so the recipe can route it distinctly (diagnostic/human pause) instead of mislabeling.
5. Contract/orchestrator (the /rectify candidate): enumerate allowed verdicts in skill contracts (enum, not free string) and add a semantic rule or runtime check that ci_only_failure is rejected (or re-routed to re-diagnosis) when emitted from steps reachable only via local test failure — mechanical precondition validation instead of prose trust.
6. Test hardening: pin the invariant paragraph in test_resolve_failures_ci_aware.py with an anchored, paragraph-scoped assertion so the next prose softening fails CI.
What happened
Implementation run
impl-20260705-184231-936209for #4189 (autoskillit fleet runCLI) was killed at the pre-mergefixstep. Part A of a two-part plan registered thefleet runsubcommand while deletion of the pre-existing deletion-guard testtest_fleet_run_command_not_registered(tests/cli/test_fleet_list.py:100) was deferred to Part B — so Part A's local test gate could never pass. resolve-failures fixed 3 of 4 failures, then emittedverdict=ci_only_failurefor the remaining locally-reproducible canary failure, and the recipe routed torelease_issue_failure → register_clone_failure → escalate_stop.Root cause — four stacked gaps (one per layer)
make-plan/SKILL.mdlines 337–343) are scope-isolation only — no invariant that each part must leavetask test-checkgreen. The planner's own Registry Tracer flagged the canary CRITICAL (F1) pre-split; the split put the fix on the wrong side and no rule caught it.task test-allpasses with the single exception oftest_fleet_run_command_not_registered") and recorded it under Verified, then stamped the plan "Implement as-is". It checks the test command name (SKILL.md lines 149–156), never the predicted outcome. No pipeline-gate compatibility check exists.ci_only_failureMAY be emitted whenfixes_applied >= 1…" and skipped the "…if the remaining failures are all CI-only" qualifier. Per the skill's own decision table (lines 186–196) everyci_only_failurerow requires Local result = PASS — the canary fails locally (verified empirically in the worktree). It also exited the fix loop after 2 of 3 iterations via the tests-green reporting path (Step 4) instead of Step 5 (Report Failure).fixstep (implementation.yaml:661–698) is reachable only via a local test failure before any CI run — an honestci_only_failureis mechanically impossible there. The route functions purely as an unconditional kill for misclassifications. The same trap exists inmerge_gate_fix,implementation-groups.yaml,remediation.yaml, andresearch-implement.yaml.Recurring regression (3rd generation)
953e27c54(resolve-failures skill: ci_only_failure verdict when fix was successfully applied #1954, 2026-05-05): added hard invariant "ci_only_failureis NEVER emitted whenfixes_applied >= 1".476059fbb(ci_only_failure Verdict Invariant Violation and Recipe Routing Deviation #3610 fix, 2026-06-02): side-effect weakened it to "MAY be emitted … if all CI-only" — the loophole-shaped text this session misread.test_ci_only_failure_requires_no_fix_applied(tests/skills/test_resolve_failures_ci_aware.py:292) uses DOTALL regexes across the whole document and still passes with the weakened text — verified against current SKILL.md.Existing escape hatch never used
check_test_passedignoresxfailedcounts (verified empirically: exit 0 + "45 xfailed" → gate passes; locked in bytests/server/test_tools_workspace.py:181). Marking the canaryxfail(strict=True)in Part A would have passed the gate with forced cleanup semantics. No skill guidance mentions this mechanism.Recommended remediation
task test-check; any pre-existing test invalidated by a part's changes must be updated/removed/xfail-bridged in that same part; split boundaries keep gate-prerequisites with the code that triggers them.plan_blockedverdict for plan-acknowledged deferred failures.verdictin skill contracts (currently unconstrainedtype: string); reject or re-routeci_only_failureemitted from steps reachable only via local test failure — mechanical precondition validation instead of prose trust.test_resolve_failures_ci_aware.pywith an anchored, paragraph-scoped assertion so the next prose softening fails CI.Artifacts
/home/talon/projects/autoskillit-runs/impl-20260705-184231-936209671e7e325,999ef83c2,58d5bbc30):/home/talon/projects/autoskillit-runs/worktrees/impl-4189-20260705-1924111f7c367b· review_approach0fc7a4bc· verify0d97291a· implementdf113cef· fix97d09f97fail)Investigation
Investigation: Pipeline Failure on Issue #4189 — Part-Split Canary Test vs. Per-Part Test Gate
Date: 2026-07-05
Scope: Implementation pipeline run
impl-20260705-184231-936209for issue #4189 (autoskillit fleet runCLI). Failure at thefixstep: resolve-failures returnedverdict=ci_only_failurefor a locally-failing, plan-acknowledged canary test, routing the run torelease_issue_failure → escalate_stop.Mode: Standard (extended: 9 subagents + direct empirical verification)
Summary
The run was killed by a stack of four independent gaps, one per pipeline layer, none of which is a "bug" in isolation. (1) Planning: make-plan split a 644-line plan at the mechanical 500-line trigger using an "implementation vs. tests" boundary, which stranded the removal of a pre-existing deletion-guard test (
test_fleet_run_command_not_registered) in Part B while Part A registers the very command the guard forbids — guaranteeing a red test gate after Part A. make-plan has no rule that each part must independently pass the test gate. (2) Validation: dry-walkthrough read the plan's own admission that the test would stay red, recorded it under "Verified" as a correct factual statement, and stamped the plan — it has no pipeline-gate compatibility check. (3) Execution: resolve-failures fixed 3 of 4 failures, then faced a failure it correctly understood as plan-deferred but had no verdict for; it misapplied a previously-weakened invariant and emittedci_only_failurefor a failure that reproduces locally — definitionally wrong per its own decision table. (4) Routing: in the pre-mergefixstep, which is only ever reached via a local test failure before any CI run,ci_only_failurecan never be legitimate; its route is a hard-kill misclassification sink. The verdict misuse is a three-generation recurring regression (#1954 fixed → #3610 fix weakened the invariant → today's misread), and the regression test guarding the invariant is too broad to detect the weakening (empirically verified).Root Cause
Layered, in causal order:
RC1 — make-plan split invariant gap (primary origin). The plan session's adversarial Registry Tracer correctly flagged the canary as CRITICAL (F1) and the fix (Step 4: delete the test) was added to the unsplit draft. When the 500-line rule forced a split, the planner categorized Step 4 as "test infrastructure" and moved it to Part B. The split rules in
make-plan/SKILL.md(verified at lines 337–343) are exclusively about scope isolation — there is no "each part must leave the gate green" invariant, so no rule was violated. Part A's own Verification section (line 357 of the plan) then stated the failure as an accepted outcome. [SUPPORTED — SKILL.md text, plan files, plan session log]RC2 — dry-walkthrough has no gate-compatibility check. The verify session explicitly wrote: "test_fleet_run_command_not_registered … is the single expected failure, exactly as the plan states (removal deferred to Part B)" — under its Verified section — then stamped the plan and recommended "Implement as-is." Its checklist checks the test command name, never the predicted test outcome. Its multi-part isolation rule ("MUST NOT read any other part files") also prevents cross-part reasoning, but no cross-part read is needed: the plan's own text declaring a post-part red test is detectable from Part A alone. [SUPPORTED — verify session JSONL, SKILL.md lines 88–156]
RC3 — resolve-failures verdict vocabulary gap + invariant misapplication. The fix session identified the canary as plan-deferred on its first reasoning turn, fixed the 3 unrelated failures (commits
999ef83c2,58d5bbc30), re-ran tests (28,077 passed / 1 failed), then emittedci_only_failurewith zero deliberation over alternatives. Per the skill's own decision table (lines 186–196), everyci_only_failurerow requires Local result = PASS; the post-fix invariant (lines 333–336) permits it withfixes_applied >= 1only "if the remaining failures … are all CI-only." The canary fails locally — I verified this empirically in the worktree ('run' in subcommands → assertion False). The agent anchored onfixes_applied=2 >= 1and skipped the CI-only qualifier. Secondary violations: it exited the fix loop after 2 of 3 allowed iterations with red tests, and used the Step 4 (tests-green) reporting path instead of Step 5 (Report Failure). [SUPPORTED — fix session JSONL, SKILL.md, empirical check]RC4 — pre-merge
ci_only_failureroute is a semantic trap. Thefixstep (implementation.yaml:661–698) is reached exclusively after a localtest_checkfailure; no CI has run and the CI context refs are empty in this path. A verdict meaning "failure only reproducible in CI" is therefore mechanically impossible if emitted honestly — the routeci_only_failure → release_issue_failurein this step functions only as an unconditional kill for misclassifications. The same trap exists inmerge_gate_fixand inimplementation-groups.yaml,remediation.yaml,research-implement.yaml. [SUPPORTED — recipe YAML, routing analysis]Affected Components
src/autoskillit/skills_extended/make-plan/SKILL.md(lines 337–343): split trigger + multi-part rules; missing green-gate invariant [SUPPORTED]src/autoskillit/skills_extended/dry-walkthrough/SKILL.md(lines 88–156): no test-gate outcome check; part-isolation rule [SUPPORTED]src/autoskillit/skills_extended/resolve-failures/SKILL.md(lines 186–196 decision table; 323 verdict list; 333–336 weakened invariant) [SUPPORTED]src/autoskillit/recipes/implementation.yaml(fix step 661–698; merge_gate_fix 718–757; next_or_done 820–829; kill chain 2352–2497) [SUPPORTED]src/autoskillit/recipes/contracts/implementation.yaml(line ~131):verdictis unconstrainedtype: string, no enum [SUPPORTED]src/autoskillit/execution/testing.py(check_test_passed, 243–265): binary gate; xfailed does not fail — empirically verified [SUPPORTED]tests/skills/test_resolve_failures_ci_aware.py(line 292): DOTALL regex passes despite invariant weakening — verified by reading the regex against current SKILL.md text [SUPPORTED]tests/cli/test_fleet_list.py:100–103: the deletion-guard canary, added7dda8dc81(PR Implementation Plan: Fleet Restructure T3: CLI Split — fleet dispatch + fleet campaign #1242, 2026-04-25), no removal annotation, no pytest marker [SUPPORTED]Data Flow
plancapturesplan_parts→ per part:verify(dry-walkthrough stamps) →create_impl_worktree→implement(skill forbids running tests; canary failure invisible here by design) →test(test_check→task test-check, binary, local) → on failurecheck_test_fix_loop→fix(resolve-failures emits free-string verdict) →on_resultrouting. Part A merges to the feature branch (merge→inter_part_push) before Part B begins (next_or_done), so the Part A test gate runs with zero awareness that Part B exists. The verdict string flows uncheck-ed (no enum, no precondition validation) into routing that terminates the run.Test Gap Analysis
test_ci_only_failure_requires_no_fix_appliedmatches(no fix|never.*fix|fixes_applied.*0).*ci_only_failurewithre.DOTALLover the whole document; the Step 2d header text ("Applies ONLY when no fix was applied (fixes_applied == 0)") thousands of characters away from the weakened invariant keeps it green. The 476059f weakening shipped without any test failing. [SUPPORTED — verified against current text]verdict: type: stringwith no enum, so any string — including a semantically impossible one — passes validation and routes. [SUPPORTED]ci_only_failurerouted from a step reachable only via local test failure). [SUPPORTED]Similar Patterns
xfail(strict=True)is the existing, proven bridge mechanism.check_test_passedignoresxfailedcounts (empirically verified:check_test_passed(0, "…45 xfailed…") == True), the suite already uses xfail in 8+ files, andtests/server/test_tools_workspace.py:181explicitly locks in that xfailed ≠ failure. Marking the canaryxfail(strict=True, reason="removed in Part B")in Part A would have passed the gate, andstrict=Trueforces cleanup. [SUPPORTED]canarypytest marker is a different concept ("development-time filter verification tests", pyproject line 106) — excluded from alltask test-checkvariants but not intended for deletion guards. [SUPPORTED]run_skill_retryguards were removed in later test-audit passes (3a43a0df0,40ae65b80) — valid only for permanently-deleted symbols. For intentional re-registration (this case), the guard must die in the same commit that re-registers the name. Part A's commit671e7e325registeredrunand left the guard untouched. [SUPPORTED]Design Intent Findings
test_fleet_run_command_not_registered: introduced in7dda8dc81(2026-04-25, PR Implementation Plan: Fleet Restructure T3: CLI Split — fleet dispatch + fleet campaign #1242, human-authored) as a deletion guard whenfleet runwas split intofleet dispatch/fleet campaign. No docstring, no removal condition, no marker. Depended on by nothing — it IS the dependency (a one-sided "name must stay absent" invariant). The pattern is undocumented anywhere in docs/AGENTS.md. [SUPPORTED]ci_only_failureinvariant lineage: hard form ("NEVER when fixes_applied >= 1") introduced by953e27c54(2026-05-05, closes resolve-failures skill: ci_only_failure verdict when fix was successfully applied #1954); weakened to conditional form ("MAY … if all CI-only") as a side-effect of476059fbb(2026-06-02, closes ci_only_failure Verdict Invariant Violation and Recipe Routing Deviation #3610, whose primary subject was bypass-route reachability). No commit since has re-hardened it. [SUPPORTED — diffs read directly]fixverdict routing: inherited verbatim from the post-CIresolve_cicontext whereci_only_failure → release_issue_failureis coherent; never adapted for the local-only pre-merge context. [SUPPORTED]Historical Context
This is a recurring pattern — at least the 4th manifestation of "verdict precondition not met → terminal routing kills pipeline":
953e27c54):ci_only_failureemitted after a successful fix.32572a2e5): flaky tests aborted the pipeline via verdict routing asymmetry.3a2f4c840), Infinite CI loop: flake_suspected routing has no escape for is_fixable=false env failures #4113/Implementation Plan: Fix Infinite CI Loop — flake_suspected Routing Escape for is_fixable=false #4116 (87ad2ff94): adjacent verdict/loop routing failures.476059fbb): verdict-bypass reachability — whose fix weakened the resolve-failures skill: ci_only_failure verdict when fix was successfully applied #1954 invariant, creating today's loophole ("MAY be emitted when fixes_applied >= 1" invites anchoring on the numeric condition while skipping the "all CI-only" qualifier).97d09f97did exactly that.Prior fixes were prose-layer (SKILL.md edits) guarded by regexes too broad to detect softening; no layer mechanically validates verdict preconditions. This is a recurring pattern — consider running
/rectifyfor architectural immunity after resolving the immediate issue.External Research
GitHub issue search (TalonT-Org/AutoSkillit) surfaced the historical issues above; issue #4189's body contains no mention of the canary or a two-part split — the split was purely a planning-session decision. No external library/web factors are implicated (failure is entirely internal orchestration logic).
Scope Boundary
Investigated: make-plan / review-approach / dry-walkthrough / implement-worktree-no-merge / resolve-failures skill texts; all five session logs of the failed run; implementation.yaml + implementation-groups.yaml routing; test_check/check_test_passed code paths (empirically); canary origin and precedents in the AutoSkillit repo; verdict invariant git lineage; prior related issues and fixes.
Not yet explored: whether the open scope of #3610 overlaps further; live multi-part runs at risk in
remediation.yaml/research-implement.yaml(same routing shape, unexamined runs); Codex-backend variants of the fix path; the review_pr/audit_impl stages (never reached); the implementer's separate pre-commit rule violation ("pre-existing" contract-freshness blame) — noted but not chased.Confidence Levels
All findings above marked inline. Every load-bearing claim (canary fails locally; xfailed passes the gate; ci_only_failure rows require local PASS; invariant weakened in
476059fbb; regression regex blind to the weakening; make-plan lacks green-gate rule) was verified directly by me — code read, diff read, or executed — not only by subagents: SUPPORTED. The only NEEDS-EVIDENCE item: why the fix agent skipped the third loop iteration (its transcript records no rationale; the early exit is observed fact, the motivation is inference).Recommendations
Immediate (unblock #4189):
pytest.mark.xfail(strict=True, reason="canary removed in Part B")in Part A and delete the test in Part B. The preserved worktree's 3 commits are salvageable. Remove thefaillabel on retry.Systemic (defense per layer, in priority order):
2. make-plan: add a fifth CRITICAL multi-part rule — every part must independently leave
task test-checkgreen; any pre-existing test invalidated by a part's changes must be updated/removed/xfail-bridged in that same part; split boundaries must keep gate-prerequisites with the code that triggers them.3. dry-walkthrough: add a blocking check — if the plan text states any test is expected to fail after this part's implementation, emit Dry Walkthrough FAILED. Detectable from a single part; no cross-part read required.
4. resolve-failures: restore the hard invariant ("NEVER when fixes_applied >= 1"), giving the partial-fix-with-CI-only-remainder case its own verdict if genuinely needed; require Step 5 (Report Failure) as the sole exit when tests remain red and iterations remain; optionally add a
plan_blockedverdict for plan-acknowledged deferred failures so the recipe can route it distinctly (diagnostic/human pause) instead of mislabeling.5. Contract/orchestrator (the /rectify candidate): enumerate allowed verdicts in skill contracts (enum, not free string) and add a semantic rule or runtime check that
ci_only_failureis rejected (or re-routed to re-diagnosis) when emitted from steps reachable only via local test failure — mechanical precondition validation instead of prose trust.6. Test hardening: pin the invariant paragraph in
test_resolve_failures_ci_aware.pywith an anchored, paragraph-scoped assertion so the next prose softening fails CI.