Skip to content

Admission truth-telling: any-pass capability override + pruning-filter erasure of ERROR rules let statically-doomed codex dispatches through #4171

Closed
Closed
Admission truth-telling: any-pass capability override + pruning-filter erasure of ERROR rules let statically-doomed codex dispatches through#4171
Labels
bugExisting behavior is brokenrecipe:remediationRoute: investigate/decompose before implementationstagedImplementation staged and waiting for promotion to main

Description

@Trecek
Trecek
opened on Jul 3, 2026

Summary

On 2026-07-02 PM, a codex implementation run for #3987 reached 35 minutes of successful work (plan → review-approach → dry-walkthrough → implement with 4 real commits via minimax provider routing → test_check) and then crashed pre-launch at the fix step: RuntimeError: Skill 'resolve-failures' requires backend ['claude-code'] but session backend is 'codex'. The runtime gate is correct and provider-aware — the defect is that admission admitted a pipeline whose crash was statically knowable. Deep investigation (11 agents, challenge round, empirical probes against the real code, 3 validators; full report appended) established that two admission defenses are silently disabled, and this issue is the remediation package for making admission tell the truth again. All findings below were verified first-hand (probes re-run by validators).

Key established facts:

  1. Any-pass admission shipped by the previous remediation. PR Capability Admission Provider Override Remediation #4168 (4590df0ca, 2026-07-01 23:52, first shipped in 0.10.835 — the remediation pipeline output for issue Codex dispatch admission: complete missing-step diagnostics, seam tests, ADR #4167) rewrote _provider_aware_capability_overrides (src/autoskillit/server/tools/_auto_overrides.py:91–128) from all-must-pass (bail-on-first-miss) to any-suffices: one covered guarded step (implement: minimax) flips backend_supports_git_write="true" (resolution_path="any_pass" at :116) and un-prunes all 9 guarded run_skill steps (the ingredient also gates run_python gate steps — the capability loop counts run_skill steps only, _auto_overrides.py:83). Issue Codex dispatch admission: complete missing-step diagnostics, seam tests, ADR #4167's own "Explicitly out of scope (evaluated and killed)" section rejected any-pass for exactly the failure that then occurred; the plan appended to the issue re-endorsed it; the pinning test was renamed and inverted in the same commit (test_provider_aware_capability_override_partial_overrides_stays_false → ..._flips_true, partial_bail assertions deleted — diff verified).
  2. The ERROR-severity admission rule that would still have refused the recipe is structurally inert. backend-incompatible-skill (src/autoskillit/recipe/rules/rules_backend_compat.py:19–68, Severity.ERROR) is correctly wired into validity (compute_recipe_validity, recipe/registry.py:245–254: any ERROR semantic finding → valid=False). But the pre-prune baseline context is built without a skill_resolver (recipe/_api.py:376: make_validation_context(active_recipe, backend_name=backend_name)), so the rule yields [] there (early-out at rules_backend_compat.py:30); post-prune it fires with the resolver present (_api.py:439–447); then filter_pruning_false_positives (recipe/_rule_helpers.py:274–287 — "keep only post-prune findings that also appeared pre-prune") erases every one of its findings whenever any step was pruned (_api.py:452) — which is essentially every normal run: post_run_diagnostics (6 steps), review_approach, and absent issue_url all default-prune. Validity is computed on the filtered set (_api.py:490). The inverse case is a live user-visible regression today: a config that disarms all pruning (diagnostics + review_approach enabled, issue_url supplied) lets the 13 findings — including correctly-routed steps — reach validity, refusing a fully-covered working config.
  3. Empirical proof (probes preserved in the appended report): real load_and_validate on codex with the partial user config → valid=True, zero backend-incompatible-skill findings; the identical call with every skip-ingredient forced truthy (filter disarmed) → 13 ERROR findings and valid=False. Admission validity currently depends on whether an unrelated ingredient happened to prune a step.
  4. The rule is also provider-blind: the 13 findings include implement — a step correctly routed to a claude-code worker via ANTHROPIC_BASE_URL. So the two defects cancel in the common case (the rule would over-refuse if it were heard, and it is never heard), which is why neither was noticed.
  5. Four steps escape all admission surfaces under ANY semantics: resolve_queue_merge_conflicts (implementation.yaml:1695), resolve_direct_merge_conflicts (:1900), resolve_immediate_merge_conflicts (:2005), ci_conflict_fix (:2239) invoke resolve-merge-conflicts (git_metadata_write → claude-code-only) but are guarded only by skip_when_false: inputs.open_pr — never pruned on codex, never checked by the capability admission (which only inspects backend_supports_git_write-guarded steps), and only the provider-aware rule fix below can make them visible. The same 13-step topology exists in remediation.yaml (with assess/merge_gate_assess) and implementation-groups.yaml (identical names) — validator-verified.
  6. User-side unblock is config-only — pre-R0 (documented in the appended report; applying it is the user's action, not this issue's scope; post-R0 the block becomes model selection only, no longer a reachability requirement — see R0 and the out-of-scope note): per-step provider overrides for all 13 git-write-skill steps per recipe make every dispatch gate pass — empirically proven by a simulation that runs the real resolution code for every run_skill step of both recipes.

The missing mechanism (the true origin of the class — established after user correction, git-verified)

The intended design — a codex orchestrator automatically launching Claude Code CLI worker sessions for any skill that requires claude-code, with zero provider configuration — EXISTED and is in the git history:

Commit Date What happened
ea488fb1e / 968258826 (#3569, implements #3495) Jun 1 Added _skill_requires_claude: backend_override = claude-code if (_provider_override or _skill_requires_claude) — a claude-requiring skill on a non-claude orchestrator auto-launches a claude-code worker; logged as backend_override_activated, reason="skill_requirement"
89b042c24 / 98e685574 (#3680, fixes #3675) Jun 3 Over-classification bug: EVERY capability was hand-marked claude-only, forcing ALL dispatches to claude-code. The fix correctly derived required_backends from codex_status — and deleted the _skill_requires_claude routing mechanism entirely as collateral, leaving backend_override provider-only
c73887014 (#3892) Jun 7 Correctly classified git_metadata_write across 8 skills (resolve-failures, resolve-merge-conflicts, resolve-review, retry-worktree, generate-report, resolve-claims-review, resolve-research-review, make-plan — the last corrected again by b9017136f/#3902) — but with the routing gone, the requirement became a hard crash instead of a routing signal

Every "requires backend ['claude-code']" failure since Jun 7 — and the entire gate_backend_write/provider-override admission pendulum (#4094→#4168) — is downstream scaffolding compensating for that missing mechanism. This also explains why claude-code-orchestrator runs "just work": their workers inherit claude-code trivially. Issue #3495 remains OPEN; the mechanism was never restored after the data defect that motivated its removal was fixed.

Remediation Scope

R0 — Restore the capability-driven backend override (the primary fix; blast radius: MEDIUM)

Reinstate the _skill_requires_claude disjunct in run_skill's backend_override derivation (tools_execution.py:762–770), exactly the #3569 shape (including the reason="skill_requirement" structured log). This is safe now precisely because #3680 fixed the condition it was removed for: required_backends is derived, and only git_metadata_write and open_kitchen are not-applicable, so only genuinely claude-requiring skills route. One structural restriction (verified): four skills derive their requirement from the open_kitchen capability (open-kitchen, close-kitchen, sous-chef, enrich-issues) — auto-routing a kitchen-lifecycle skill into a worker session would nest kitchens. None is dispatched via run_skill by any bundled recipe today (grep-verified), but the guard must be structural, not incidental: trigger the auto-route only when git_metadata_write is among the skill's uses_capabilities (not open_kitchen), or equivalently exclude open_kitchen-derived requirements from the disjunct. Effect: a codex orchestrator launches Claude Code CLI workers for git-write skills with zero provider config — provider overrides become model selection (e.g., MiniMax), not reachability requirements.

Two halves, both required:

  1. Runtime routing — the disjunct itself; the :801 gate then passes naturally (effective backend = claude-code).
  2. Admission alignment — _provider_aware_capability_overrides must account for the auto-route when resolving backend_supports_git_write: a guarded run_skill step's effective worker backend is claude-code whenever its skill requires it, so the ingredient's input becomes claude-binary availability, not provider coverage. Without this half, a zero-config codex run still prunes/refuses steps that R0's routing could serve. Specification (resolves an ambiguity adversarial review caught): the binary probe is fail-closed here — binary absence under capability auto-routing is capability infeasibility, refused at admission with an explicit "claude binary not found" remedy, NOT a fail-open warning (the campaign's fail-open availability contract applies to provider-endpoint probes, not to the binary the route depends on; without this distinction, a missing binary would regress from today's clean 5.6ms gate crash to a muddy spawned-process crash — ensure_pre_launch() returns [] and no headless-path binary check exists today). Implementation shape: probe internally (shutil.which("claude"), one call per invocation — no process-local cache infra exists yet; building one is optional) so the 5 existing call sites (tools_kitchen.py ×2, tools_recipe.py ×2, tools_fleet_dispatch.py) need no signature change; add new resolution_path values ("capability_route", "capability_route_no_binary") and extend the exhaustive value list in CapabilityResolutionDetail's docstring (core/types/_type_results.py:222–234) — do NOT reuse "any_pass", whose escape-hatch messaging is provider-specific. Credential note: an auto-routed worker with no provider override runs the claude CLI against the default Anthropic endpoint and requires Anthropic credentials; credential absence surfaces at launch (out of R0's probe scope — document as a prerequisite in the refusal/doc surfaces).

Co-updates (all verified against the current tree; the #3680 test changes shipped inside its own squash — the "Part B" its PR body promised never landed separately):

  • tests/server/test_tools_execution_backend_mixing.py::test_no_skill_requires_claude_logic (:367–376) is an anti-restoration guard asserting the literal _skill_requires_claude symbol is absent from tools_execution.py — it will block R0 at CI and must be replaced (not deleted) with a test pinning the restored behavior under a decision-record docstring citing this issue.
  • ::test_skill_backend_requirement_triggers_incompatibility_gate (:109) and ::test_backend_incompatibility_does_not_emit_override_log (:153) pin the pre-R0 contract (requirement → crash, no override log) and will fail under R0 — both flip to asserting the routing fires (backend_override_activated, reason="skill_requirement"), with decision-record docstrings.
  • The backend_override_activated log's reason field distinguishes skill_requirement from provider_profile (restore the Implementation Plan: Skill-Capability-Aware Backend Override + Structured Logging #3569 shape).

R1 — Scope filter_pruning_false_positives to the graph-aware rules it was written for (blast radius: MEDIUM)

The filter's own docstring names its purpose: suppressing pruning-induced false positives from graph-aware rules ("dead-output, capture-inversion"). Scope it to exactly those rule names (or, equivalently and additionally, give the pre-prune baseline context the same skill_resolver the post-prune context gets — fix the _api.py:376 asymmetry) so that ERROR-severity rules can never be silently erased by the intersection.

R2 — Make backend-incompatible-skill effective-backend-aware (blast radius: MEDIUM)

Post-R0, "provider-aware" generalizes to effective-backend-aware: the rule must evaluate each step against the same effective-backend resolution the runtime uses — capability auto-route (R0) first, provider override second. With R0 in place the rule stops firing for claude-requiring steps whenever the worker path exists, and fires truthfully when it does not (e.g., claude binary unavailable).

Before flagging a surviving step, resolve its provider profile exactly as the runtime does — the same _resolve_provider_profile call already used per-step at server/tools/_preflight.py:68–78 (call at :68–73, predicate at :74–78) and _auto_overrides.py:96–101. A step whose resolved profile carries ANTHROPIC_BASE_URL on a backend with anthropic_provider_capable=False routes to a claude-code worker at dispatch (tools_execution.py:762–776) and must not be flagged. This requires threading per-step routing data into the rule's inputs — and the IL layering constrains how: ProvidersConfig is IL-1 config and ValidationContext is IL-2 recipe, and import-linter contract "IL-2 recipe imports only IL-0 and workspace" forbids recipe/ importing config/. The data must therefore arrive as a pre-resolved IL-0 plain type (e.g., dict[str, bool] step-name → routes-to-anthropic-capable-worker, or a frozenset[str] of routed step names), computed at IL-3 (tools_kitchen.py, beside the existing _provider_aware_capability_overrides call) and threaded through load_and_validate's signature into make_validation_context/ValidationContext. This is the same parameter-passing pattern the split-plane campaign's S1 formalizes; this fix must not wait for S1, but must not violate the layering either.

Sizing note (adversarial review, verified): R0 is independently deployable

R0 touches IL-3 server files (tools_execution.py, _auto_overrides.py) and their tests; R1+R2 touch IL-2 recipe files (_api.py, _rule_helpers.py, rules_backend_compat.py). The mutual-dependency constraint below binds R1↔R2 only — R0 can land before them without creating either regression, and the planner may decompose accordingly rather than serializing the routing fix behind R2's ValidationContext-threading design.

⚠ R1 and R2 MUST land in the same change — mutual dependency (verified both directions)

  • R1 without R2: the 13 findings (which include correctly-routed steps) survive the filter → valid=False → a fully-covered, working provider config is refused at admission. Worse than the status quo.
  • R2 without R1: the rule stops flagging routed steps, but its remaining (true) findings are still erased by the filter → still inert.
  • R1+R2 together: the rule flags exactly the surviving steps that will crash at dispatch → admission refuses exactly the configs that fail at runtime — the admission↔runtime consistency invariant in miniature. This also makes the 4 open_pr-guarded steps (fact 5) visible to admission for the first time.

R3 — Admission semantics decision recorded, not assumed (blast radius: LOW)

With R1+R2, any-pass becomes defensible (partial configs are truthfully refused by the rule naming the exact uncovered steps) — but that is a decision to record, not to assume: the split-plane campaign's S0/D1 (ADR) owns the final semantics. Interim requirement: whatever semantics stand after this fix must be pinned by a test carrying a decision-record docstring — the #4168 flip proved a bare assertion does not hold the line (the test was simply renamed and inverted in the same commit that changed behavior).

R4 — Regression seams (blast radius: LOW)

  1. Admission↔dispatch agreement seam test: for real ProvidersConfig shapes (partial / full / none) × real bundled recipes (implementation, remediation, implementation-groups) on the codex backend, assert that every step surviving pruning has a dispatch-time effective backend satisfying its skill's backend_requirements — i.e., admit ⟹ dispatchable. Post-R0, the simulated routing predicate must cover BOTH disjuncts of the backend_override expression (_skill_requires_claude first, _provider_override second — the restored tools_execution.py:762–770 shape), not provider routing alone. The investigation's simulation script (.autoskillit/temp/investigate/simulate_codex_pipeline.py, referenced in the appended report) is the working prototype: it exercises _provider_aware_capability_overrides, load_and_validate, _resolve_provider_profile, and the tools_execution.py:762–776 predicate with no mocks.
  2. Filter/severity invariant test: ERROR-severity semantic findings must survive filter_pruning_false_positives (or the filter must be provably scoped to its named graph-aware rules). Today nothing pins the pre-prune context construction or the filter's breadth.
  3. Coverage for the 4 open_pr-guarded git-write steps (parametrized over all three recipes) — currently zero admission coverage of any kind.

R5 — Process finding for the record (no code)

The #4168 deviation entered at make-plan: the plan appended to issue #4167 contradicted the issue's own kill-list, and audit/review validated diff-vs-plan (which matched). Issue #4167's §C.2 requested claimed-scope-vs-diff validation; this incident shows it must also cover plan-vs-issue-scope consistency (does the plan contradict the issue's explicit out-of-scope/killed-alternatives sections?). Second occurrence of the pathology (#4164 "test-only" mislabel was the first).

Explicitly out of scope (evaluated and killed)

  • Wholesale revert of Capability Admission Provider Override Remediation #4168 — loses the genuine full-enumeration diagnostics and fleet sentinel it also shipped; and all-must-pass never covered the 4 open_pr-guarded steps, so the crash class survives a revert.
  • Recipe YAML changes (adding backend_supports_git_write guards to the 4 conflict steps) — changes semantics for all backends plus contract regeneration; R2 makes them admission-visible without YAML churn; YAML/gate decisions belong to the campaign's D1/D2.
  • Hoisting the backend-compat gate before the arg gate — real but orthogonal (round-trip waste, not correctness); already specced in open issue run_skill gate hardening: backend-compat check ordered after arg gate, non-self-healing gate errors, no dispatch-time enforcement of skip_when_false pruning #3975 R1; do not fold in.
  • The user-side config block itself — a user action on ~/.autoskillit/config.yaml, documented in the appended report; not a code deliverable. Note: post-R0 the block is unnecessary for reachability — provider overrides go back to their intended role, selecting models/providers for worker sessions (e.g., implement: minimax), with routing handled by capability requirements automatically.
  • The full per-step resolution architecture — owned by the split-plane dispatch admission campaign (task document); this issue is the narrow interim truth-telling fix that the campaign's S1–S3 later subsume.

Related

Investigation

Prior investigation completed interactively (deep mode). Full report with evidence citations follows.

Investigation: Codex Full-Run Blockers — resolve-failures Gate Crash + Complete Forward Walkthrough

Date: 2026-07-02
Scope: Root cause of the fix-step crash ("Skill 'resolve-failures' requires backend ['claude-code'] but session backend is 'codex'") in the afternoon #3987 run; complete forward enumeration of EVERY remaining blocker in the implementation and remediation recipes under a codex orchestrator; empirical simulation of the exact config that clears them all; history of every codex-run blocker ever hit.
Mode: Deep Analysis (2 batches — 11 agents; challenge round; empirical probes run against real code; 3 validators)

Summary

The pipeline crashed at the fix step not because of a broken gate but because of a three-layer admission failure that let a doomed step be dispatched. The runtime gate (_check_backend_compat, tools_execution.py:93–154) is correct and provider-aware: it evaluates the effective backend after per-step provider routing. implement passed it because config routes it to minimax (session log proves: backend=claude-code, provider_used=minimax, model=MiniMax-M3); fix crashed because the user config covers only implement — and admission had let the run proceed anyway. Admission passed because (1) the #4167 remediation (PR #4168, 4590df0ca, 0.10.835, merged Jul 1 23:52) changed _provider_aware_capability_overrides from all-must-pass to any-suffices — semantics the source issue's own out-of-scope section had adversarially killed, but which the pipeline's plan re-endorsed and shipped with the pinning test rewritten (partial_overrides_stays_false → partial_overrides_flips_true); and (2) the ERROR-severity backend-incompatible-skill semantic rule that would still have refused the recipe is silently erased by filter_pruning_false_positives — the pre-prune baseline context is built without a skill_resolver (_api.py:376), so the rule yields nothing pre-prune, and the filter ("keep only findings that also appeared pre-prune") deletes its post-prune ERROR findings whenever any step was pruned — which is nearly always, since post_run_diagnostics defaults false. Both mechanisms were confirmed empirically by running the real load_and_validate: with the filter armed, valid=True and zero backend-incompatible-skill findings; with pruning disarmed, 13 ERROR findings surface and valid=False. The forward walkthrough enumerated every remaining blocker: the entire blocker set is config-fixable today — a 26-entry providers.recipe_overrides block (13 implementation + 13 remediation step names) empirically clears every admission and dispatch gate (simulation ran the real resolution code end-to-end). Everything downstream of the git-write-skill seam (merge, audit, PR, review, CI, merge queue, cleanup) has been proven working on codex by seven historical end-to-end successes.

Root Cause

Layer 1 — Any-pass admission shipped by the remediation pipeline (the enabling regression). PR #4168 (4590df0ca) rewrote the guarded-step loop in _auto_overrides.py: the bail (if not has_base_url: return base — all-must-pass) became any_has_base_url accumulation, flipping backend_supports_git_write="true" if any guarded step has ANTHROPIC_BASE_URL (_auto_overrides.py:91–128, resolution_path="any_pass" at :116 — personally verified, diff personally verified). The source issue #4167 listed any-pass under "Explicitly out of scope (evaluated and killed)" (it re-creates exactly this failure: uncovered guarded steps survive pruning and dispatch on codex), yet the plan section appended to the same issue endorsed "any-suffices is correct (the runtime gates provide per-step defense-in-depth)", the implement step shipped it, the pinning test was renamed and its assertions flipped in the same commit, and audit/review validated diff-vs-plan. The plan contradicted its own issue's kill-list and no gate checks plan-vs-issue-scope consistency — the same pathology as #4164's "test-only" mislabel. [SUPPORTED — commit diff and test flip personally verified]

Layer 2 — The admission rule that would have caught it is structurally inert. backend-incompatible-skill (recipe/rules/rules_backend_compat.py:19–68, Severity.ERROR) checks each surviving run_skill step's skill_info.backend_requirements against ctx.backend_name and DOES wire into validity (compute_recipe_validity, registry.py:245–254: any ERROR semantic finding → valid=False — personally verified). But: the pre-prune baseline pass is make_validation_context(active_recipe, backend_name=backend_name) with no skill_resolver (_api.py:376) → the rule returns [] there (rules_backend_compat.py:30); the post-prune pass has the resolver and fires; then, whenever any step was pruned (any(v is False) — true in every normal run because the six post_run_diagnostics steps prune), filter_pruning_false_positives (_rule_helpers.py:274–287) intersects post-prune findings with the pre-prune baseline and deletes every backend-incompatible finding. Validity is computed on the filtered set (_api.py:490). Empirical proof: real load_and_validate on codex with the current config → valid=True, 149 suggestions, zero backend findings; same call with every skip-ingredient forced truthy (filter disarmed) → 13 backend-incompatible-skill ERROR findings, valid=False. The filter's docstring says it exists for graph-aware rules (dead-output, capture-inversion); it over-applies to all rules. [SUPPORTED — both probes run first-hand: .autoskillit/temp/investigate/probe_rule_firing.py, probe_filter_erasure.py]

Layer 3 — Per-step runtime resolution then did exactly what it should. At dispatch, run_skill resolves the provider per step (tools_execution.py:762–776): implement → minimax profile → ANTHROPIC_BASE_URL present → backend_override=claude-code → gate passes; fix → no override anywhere in config (recipe_overrides.implementation contains only implement: minimax; retry_worktree commented out in step_overrides) → effective backend stays codex → resolve-failures requires claude-code (its uses_capabilities includes git_metadata_write, whose codex_status="not-applicable" derives required_backends={"claude-code"} — _type_constants_registries.py:307–310, 350–356) → correct fail-closed crash in 5.6ms, no session launched, no state harmed. The recipe's on_failure routing then worked as designed: release_issue (fail label), register_clone_status, clean halt. [SUPPORTED]

Important corrective vs. the transcript's appearance: the "note" that both implement and resolve-failures carry the identical backend requirement kills the theory that the skill declarations are inconsistent; and the gate-ordering theory (#3975 R1) is real but is a round-trip-waste defect, not this crash's cause — the gate correctly receives the post-override backend (:801–810 call site, :772–776 effective-backend computation, both personally verified).

Affected Components

  • src/autoskillit/server/tools/_auto_overrides.py:91–128 — any-pass loop (any_has_base_url, resolution_path="any_pass"), shipped by 4590df0ca [SUPPORTED — verified]
  • src/autoskillit/recipe/rules/rules_backend_compat.py:19–68 — ERROR-severity rule; provider-blind (flags even correctly-routed steps — the 13 findings include implement); early-out on skill_resolver is None [SUPPORTED]
  • src/autoskillit/recipe/_api.py:376 — pre-prune baseline context built without skill_resolver (asymmetric with the post-prune context at :439–447) [SUPPORTED — verified]
  • src/autoskillit/recipe/_rule_helpers.py:274–287 — filter_pruning_false_positives intersects ALL rules with the baseline (docstring intent: graph-aware rules only) [SUPPORTED — verified]
  • src/autoskillit/recipe/registry.py:245–254 — compute_recipe_validity blocks on ERROR semantic findings (the wiring exists; it just never receives the findings) [SUPPORTED — verified]
  • src/autoskillit/server/tools/tools_execution.py:93–154, 762–776, 801–810 — the (correct) runtime gate and provider routing [SUPPORTED — verified]
  • src/autoskillit/core/types/_type_constants_registries.py:307–310, 321–357 — capability registry; git_metadata_write/open_kitchen are the only not-applicable (blocking) capabilities [SUPPORTED]
  • ~/.autoskillit/config.yaml — recipe_overrides: implementation.implement: minimax, remediation.implement: minimax only; step_overrides: compose_pr/diagnose_ci/prepare_pr (non-guarded); retry_worktree commented out [SUPPORTED — read with tokens redacted]
  • src/autoskillit/recipes/implementation.yaml — 9 backend_supports_git_write-guarded run_skill steps + 4 git-write-skill steps guarded only by open_pr (resolve_queue_merge_conflicts :1695, resolve_direct_merge_conflicts :1900, resolve_immediate_merge_conflicts :2005, ci_conflict_fix :2239) that no admission surface has ever checked [SUPPORTED]

Data Flow (the failure chain, verified end-to-end)

  1. open_kitchen → _provider_aware_capability_overrides: implement resolves minimax (has_base_url=True) → any_pass → backend_supports_git_write="true".
  2. load_and_validate → pruning keeps all 9 guarded steps (ingredient true); prunes the 6 diagnostics steps (post_run_diagnostics=false) → _skip_resolutions contains False → filter armed.
  3. Semantic rules: pre-prune baseline (no resolver) → no backend findings; post-prune → 13 ERROR findings → filter erases all 13 → valid=True, dispatch_feasible=True → recipe served.
  4. Pipeline runs: plan (codex native, 18m40s) → review-approach → dry-walkthrough → worktree → implement (claude-code via minimax, 7m28s, 4 real commits) → gate_backend_write true → test_check FAILS (2 real test failures) → loop iteration 1 → run_skill(fix).
  5. run_skill: _resolve_provider_profile("fix", "implementation", …) → anthropic default, no ANTHROPIC_BASE_URL → backend_override=None → effective backend codex → _check_backend_compat → crash (5.6ms) → on_failure → release (fail label) → register clone error → halt at 42m36s.

The Complete Forward Walkthrough (the "no more step-by-step discovery" deliverable)

Empirical simulation (.autoskillit/temp/investigate/simulate_codex_pipeline.py — runs the real _provider_aware_capability_overrides, load_and_validate, _resolve_provider_profile, capability-registry resolution, and the exact tools_execution.py:762–776 predicate for every run_skill step of both recipes):

Current config — steps that WILL crash, in pipeline order (all via the same mechanism; each is one run_skill dispatch away once reached):

# implementation step skill reached when
1 fix resolve-failures test failure (~26% of runs historically) — today's crash
2 retry_worktree retry-worktree implement hits context limit
3 merge_gate_fix resolve-failures merge gate test failure
4 rebase_conflict_fix resolve-merge-conflicts pre-merge rebase conflict
5 resolve_review resolve-review review returns changes_requested (~75% historically)
6 resolve_pre_review_conflicts resolve-merge-conflicts pre-review conflict
7 resolve_ci resolve-failures CI failure
8 resolve_pre_resolve_conflicts resolve-merge-conflicts pre-CI-resolve conflict
9–12 resolve_queue_merge_conflicts, resolve_direct_merge_conflicts, resolve_immediate_merge_conflicts, ci_conflict_fix resolve-merge-conflicts merge-queue/direct/immediate-merge conflicts — guarded only by open_pr, so they crash on codex under ANY admission semantics and were never visible to any admission check

Remediation recipe mirrors: assess, merge_gate_assess, retry_worktree, rebase_conflict_fix, resolve_review, resolve_pre_review_conflicts, resolve_ci, resolve_pre_resolve_conflicts + the same 4 conflict fixers.

Steps that pass on codex natively (no blocking capability): plan/make-plan, review-approach, dry-walkthrough/verify, audit-impl, prepare-pr, all 13 arch-lens-*, compose-pr, review-pr, diagnose-ci, analyze-pipeline-health ×6. All non-run_skill surfaces are safe: every MCP server tool the recipe uses (merge_worktree, push_to_remote, test_check, wait_for_ci, enqueue_pr, wait_for_merge_queue, check_repo_merge_state, check_pr_mergeable, release_issue, register_clone_status, run_cmd scripts, all 20+ run_python callables) executes in the unsandboxed server process with zero backend conditionals (exhaustively enumerated; create_impl_worktree.sh and the gh-CLI merge commands proven live). One latent fail-open: review_loop_gate/review_gate_post_hook key their state file on Path.cwd(), which may differ from work_dir — the review-ordering rail silently doesn't enforce (permissive, not blocking).

THE FIX (config-only — empirically proven to clear every gate in both recipes):

providers:
  recipe_overrides:
    implementation:
      implement: minimax            # present already
      retry_worktree: minimax
      fix: minimax
      merge_gate_fix: minimax
      rebase_conflict_fix: minimax
      resolve_review: minimax
      resolve_pre_review_conflicts: minimax
      resolve_ci: minimax
      resolve_pre_resolve_conflicts: minimax
      resolve_queue_merge_conflicts: minimax
      resolve_direct_merge_conflicts: minimax
      resolve_immediate_merge_conflicts: minimax
      ci_conflict_fix: minimax
    remediation:
      implement: minimax            # present already
      retry_worktree: minimax
      assess: minimax
      merge_gate_assess: minimax
      rebase_conflict_fix: minimax
      resolve_review: minimax
      resolve_pre_review_conflicts: minimax
      resolve_ci: minimax
      resolve_pre_resolve_conflicts: minimax
      resolve_queue_merge_conflicts: minimax
      resolve_direct_merge_conflicts: minimax
      resolve_immediate_merge_conflicts: minimax
      ci_conflict_fix: minimax
    implementation-groups:          # identical 13-step set (validator-verified)
      implement: minimax
      retry_worktree: minimax
      fix: minimax
      merge_gate_fix: minimax
      rebase_conflict_fix: minimax
      resolve_review: minimax
      resolve_pre_review_conflicts: minimax
      resolve_ci: minimax
      resolve_pre_resolve_conflicts: minimax
      resolve_queue_merge_conflicts: minimax
      resolve_direct_merge_conflicts: minimax
      resolve_immediate_merge_conflicts: minimax
      ci_conflict_fix: minimax

Note: implementation-groups.yaml (the third gate-equipped recipe) carries the identical 9 guarded + 4 open_pr-guarded step names — all step names in all three sections verified against the actual YAML by an independent validator.

Validated non-blockers for the next run (each checked to ground): re-claiming #3987 works — the fail label is auto-removed when in-progress is applied (removes_on_entry in the label metadata; the claim check looks only for in-progress); every recipe model: field is '' so no model-alias failure on routed steps (and resolve_model_identity suppresses non-anthropic model identifiers anyway); skill injection for routed workers follows the effective backend (init_session receives _effective_backend_obj, so resolve-failures IS injected into the claude-code worker's .claude/skills/); test_check and all worktree scripts are backend-independent (server-side).

With this block the simulation shows: admission PASS, every step's effective backend satisfies its skill's requirements (routed steps → claude-code; native steps → codex, no requirement). Tier-0 _resolve_provider_profile lookup is a pure step-name match — it works identically for the open_pr-guarded steps (verified in server/_guards.py). No step requires a code change to dispatch. Known caveat (cosmetic): minimax responses may omit Anthropic cache-token fields → cache metrics recorded as 0 for routed steps.

Next-risk list after the config fix (ranked, from historical evidence — not blockers today):

  1. Dirty-tree merge rejection — Jun 7 precedent: resolve-failures left uncommitted files; merge_worktree rejected 3×. The commit instructions now exist in the skill (Step 0.5 + context-limit section) but there is NO structural clean-tree enforcement at skill exit; the recipe's dirty_tree route goes to the fix loop (max 3), which re-runs resolve-failures (whose Step 0.5 commits) — self-healing in principle, instruction-dependent in practice.
  2. detect_zero_changes false-zero — exit-128 crash fixed (5a971233d, count→0 fallback + write-evidence override), but a run with real commits, a clean tree, and no has_implementation_progress context could still close as no-changes.
  3. Merge-queue timeout (900s → register_clone_unconfirmed, lost tracking) and check_commits_ahead exit-128 (check=True → clean escalation).
  4. The current stuck run's 2 test failures are trivial one-line fixes (add compose-pr to _KNOWN_OPTIONAL_CAPTURE_EXEMPTIONS in tests/recipe/test_contract_strength.py:94–98; add "invalid head" to compose-pr SKILL.md Step 5 permanent list) — well within one fix-loop iteration.

Salvageable state: worktree /home/talon/projects/autoskillit-runs/worktrees/impl-3987-20260702-161005 is clean with the 4 implementation commits intact; remote branch compose-pr-…/3987 exists at base SHA; issue #3987 is OPEN with recipe:implementation + fail labels (the pipeline released it properly this time).

Test Gap Analysis

  1. The admission↔runtime round-trip test — deferred twice — remains the missing seam. Nothing asserts "every step that survives pruning is dispatchable" (per-step provider resolution × skill requirements). The simulation script in this investigation is effectively that test, written as a probe.
  2. The Capability Admission Provider Override Remediation #4168 tests were rewritten to bless the deviation: test_provider_aware_capability_override_partial_overrides_stays_false → ..._flips_true, partial_bail assertions deleted (personally verified in the diff). The verdict-pinning-with-decision-record discipline (task-doc S9 mechanics) did not exist to stop it.
  3. No test exercises the filter×rule interaction: nothing asserts that ERROR-severity rules survive filter_pruning_false_positives, and nothing pins the pre-prune context's construction (resolver asymmetry is invisible to every existing test).
  4. tests/recipe/test_backend_reachability.py::TestPrunedRecipeSatisfiability uses _backend_capability_overrides (never the provider-aware variant) — the any-pass survival scenario has zero coverage.
  5. The 4 open_pr-guarded git-write steps have no admission coverage of any kind, and no test notes their existence.

Similar Patterns

Design Intent Findings

Historical Context

The complete codex blocker sequence (from all orchestrator rollouts, May 29 → today): (1) MCP server/tool visibility (May 29–31, fixed by kitchen gating) → (2) codex output-flush races (empty_output/completed_no_flush, Jun 3–4) → (3) skill backend requirements (dry-walkthrough/make-plan/resolve-failures, Jun 4–9 — recurring) → (4) dry-walkthrough stamp gate (Jun 5) → (5) worktree ref mismatch 128 (Jun 5, fixed 5a971233d) → (6) dirty-tree merge rejection (Jun 7) → (7) HOOK_REGISTRY fix-required dispatch block (Jun 11) → (8) recipe structural validation (Jun 11) → (9–11) gate_backend_write runtime → admission (Jun 12–Jul 1) → (12) provider-override all-must-pass refusal (Jul 1 PM) → (13) lock_ingredients server-authoritative rejection (Jul 2 AM) → (14) this incident. Seven full end-to-end codex successes exist (May 31 #3428; Jun 2 #3468; Jun 3 #3659 + token-reserve #144; Jun 5 #146/PR 187; Jun 8 #3875/PR 3914; Jun 10 #147/PR 189 and #148/PR 190) — proving merge_worktree, audit-impl, compose-pr, review-pr, resolve-review, CI wait, merge queue, release, diagnostics all work on codex. Those ran before the git-write guards; under the post-Jun-12 recipe, no codex run has passed test_check yet — but the only untraversed gates between here and the proven stages are exactly the 12 config-fixable dispatch crashes enumerated above. This IS a recurring pattern (admission/runtime divergence class) — the split-plane campaign task document is the architectural remediation; consider /rectify only after the interim fixes land.

External Research

The parity survey grounds the recommendation: Kubernetes makes kubelet admission call the scheduler's own AdmissionCheck function (same code, two layers — kubelet/lifecycle/predicate.go v1.32.0); Bazel fails mandatory toolchain resolution at analysis, never at execution; kubectl --dry-run=server is trustworthy precisely because it runs the REAL admission chain (k8s blog 2019-01-14), while Ansible check-mode diverges because modules re-implement it; GitHub Actions' unvalidated runner labels (jobs queue forever — community #31587/#50926) is the documented anti-pattern our system currently matches; parse-don't-validate (King 2019) names the type-level end-state. Principle for AutoSkillit: admission must evaluate the same per-step predicate the runtime evaluates — one function, two callers (which is precisely the task document's S1/S5).

Scope Boundary

Investigated: the crash mechanics end-to-end (gate, requirements derivation, provider resolution, config state); admission semantics archaeology (#4166→#4168 diffs personally verified); the rule/filter erasure (probed empirically, both directions); every run_skill step × skill × capability of both recipes; every non-run_skill surface (tools, scripts, callables, hooks); complete codex-run history (30+ rollouts, all blockers tabulated); external parity patterns; residual post-fix risks.

Not yet explored: live execution of the merge/PR/CI stages under the CURRENT recipe on codex (simulated + historically proven only — first real traversal will be the next run); minimax output-quality on conflict-resolution workloads (routed steps have run implement-class work only); the review_loop_gate cwd mismatch (flagged, fail-open, not blocking); whether the fail label on #3987 affects re-claiming (release was clean; claim logic treats fail as informational).

Recommendations

Single recommendation: apply the proven 26-entry config block now (unblocks a full run today), and file one code-side remediation for the admission truth-telling defects (the interim, narrow version of the campaign's per-step principle).

Immediate (config, no code): replace providers.recipe_overrides with the block above. Empirically verified to clear every gate in both recipes. Uncomment/remove the stale # retry_worktree line in step_overrides to avoid confusion. Then re-run issue #3987 fresh (worktree with 4 commits is reusable if preferred; the 2 test failures are one-line fixes).

Code-side remediation (one issue; MUST land as one unit — the two rule fixes are mutually dependent):

  1. Scope filter_pruning_false_positives to the graph-aware rules it was written for (dead-output, capture-inversion — per its own docstring) so ERROR-severity rules can never be silently erased; equivalently/additionally give the pre-prune baseline context the same skill_resolver (fix the _api.py:376 asymmetry).
  2. Make backend-incompatible-skill per-step provider-aware: before flagging a surviving step, resolve its provider profile (the same _resolve_provider_profile call used at _preflight.py:74–78 and _auto_overrides.py:96–101); a step whose profile carries ANTHROPIC_BASE_URL on a non-anthropic_provider_capable backend routes to claude-code and must not be flagged. Warning: shipping fix 1 without fix 2 refuses the fully-covered valid config (the 13 findings include correctly-routed steps — verified); shipping 2 without 1 leaves the rule inert. Together they make admission refuse exactly the configs that would crash at runtime — the admission↔runtime consistency in miniature.
  3. With 1+2 in place, any-pass becomes defensible (the rule now truthfully guards partial configs); alternatively revert to all-must-pass — the ADR decision (task doc S0/D1) owns the final semantics; the interim need is only that SOME layer tells the truth.
  4. Regression seams: convert the simulation into a real test (per-step admission↔dispatch agreement over real config shapes); a test pinning that ERROR-severity rules survive the pruning filter; parametrized coverage of the 4 open_pr-guarded steps; restore a decision-record-cited pinning test for the admission verdict (the Capability Admission Provider Override Remediation #4168 flip proved plain assertions don't hold the line).
  5. Process note for the record: the Capability Admission Provider Override Remediation #4168 deviation entered at make-plan (plan contradicted the issue's own kill-list) — the claimed-scope-vs-issue-scope check requested in Codex dispatch admission: complete missing-step diagnostics, seam tests, ADR #4167 §C.2 would have caught it; it should cover plan-vs-issue consistency, not only diff-vs-plan.

Killed alternatives:

Breakage Analysis

  • Recommendation: config block (26 entries)
    • Breakage surface: none in code; behavioral — 12 more steps route to minimax/claude-code workers (model-quality on conflict resolution unproven; cache metrics under-count cosmetically). Tier-0 resolution verified for all step names.
    • Prior reverts: none (config).
    • Risk: LOW.
  • Recommendation: filter scoping + provider-aware rule (must ship together)

Delivery ledger (2026-07-03)

Do not re-plan R1/R2/R4 from this issue. Remaining work lives in #4174 (code) and #4173 (process).

Delivery ledger update (2026-07-03, final)

R0 — DELIVERED by PR #4176 (merge commit 8e9c1a037; see #4174 for the full ledger). With R1/R2/R4 delivered by PR #4172, R3 decision records now pinned in the flipped tests (vacuous-gate, backend-mixing, composition matrix), and R5 tracked and remediated via #4173 (PR #4175), every remediation enumerated by this issue is delivered. Closing.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugExisting behavior is brokenrecipe:remediationRoute: investigate/decompose before implementationstagedImplementation staged and waiting for promotion to main

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions