You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
open_kitchen(name="implementation") fails on the first orchestrator message of every Codex-backend session in v0.10.752 with:
Recipe 'implementation' failed structural validation: unknown structural error (stage=recipe_validation, errors=[], suggestions contain only the usual warnings)
The fail-closed guard added by PR #3995 (61802dce4) gates on result["valid"], but load_and_validate computes valid from semantic-rule findings evaluated on the unpruned recipe. With the Codex capability override (backend_supports_git_write="false"), the backend-incompatible-skill rule emits 13 ERROR-severity findings — 9 on steps that are pruned immediately afterward (skip_when_false: inputs.backend_supports_git_write) and 4 on inputs.open_pr-route-guarded steps that the conformance test explicitly exempts via _ROUTE_GUARDED_COMPAT_EXCEPTIONS but compute_recipe_validity does not. So valid=False even though the served (pruned) content is structurally sound (non-empty: 85,728 bytes with defer_unresolved=True / 77,686 with False; zero dangling routes, errors=[]).
This is a latent semantic mismatch dating to PR #3515 (merged Jun 1), which threaded backend_name into validation and silently expanded the meaning of valid from "structurally usable" to "no backend-incompatible step anywhere in the YAML, even guarded ones." No consumer was updated for the expansion:
open_kitchen (both guard sites: first-open tools_kitchen.py:711, deferred-recall :624) now fails closed on it → total Codex breakage.
Sequencing defect (the root): in recipe/_api.py (load_and_validate): run_semantic_rules runs at line 397 on the unpruned recipe → _prune_skipped_steps at line 408 → compute_recipe_validity at line 457 consumes the pre-prune findings.
backend-incompatible-skill (rules/rules_backend_compat.py:27-68) has no awareness of skip_when_false guards or route-guard exemptions → 13 ERROR findings for codex.
compute_recipe_validity (registry.py:245-254) folds any ERROR-severity semantic/contract finding into valid → valid=False, content non-empty, errors=[].
Scope of fix (from validated investigation; killed alternatives documented in the appended report)
Compute validity from the post-prune recipe. Move semantic-rule evaluation (or at minimum the validity-relevant findings derivation) after _prune_skipped_steps. The ValidationContext must be rebuilt with the pruned recipe — not reused from the pre-prune build.
Single source of truth for route-guard exemptions. Promote _ROUTE_GUARDED_COMPAT_EXCEPTIONS (currently test-only: tests/recipe/test_backend_reachability.py:31-38) into a production constant consumed by both the validity computation path and the conformance test.
Fleet parity.fleet/_api.py dispatch validation must pass the backend capability overrides (mirroring _promote_capability_keys in the open_kitchen path) so fleet's pruning matches what would actually be served. Note fleet already contains an IL-safe local mirror, _build_capability_overrides (fleet/_api.py:112, currently used only for effective_ingredients at :422, not for validation) — this is a one-line call-site change.
Regression tests through the real producer. Bundled-recipe × backend validity assertions through the real load_and_validate with real capability overrides, asserting valid is True, non-empty content, errors == []. Note: the test added by Rectify: Surface LoadRecipeResult Errors and Fail-Closed on Invalid Content #3995 (test_load_and_validate_produces_valid_content_after_step_pruning) was one assertion short — it never asserts valid.
Implementer cautions (validated):
Confirm _api_cache.copy_resulterrors-list aliasing before shipping (cached findings must reflect post-prune state).
Deferred-recall recall path uses defer_unresolved=False vs first-open not _user_overrides_present — both passes must agree on the finding basis or first-open vs recall can disagree on valid.
Re-run the 8×2 validity matrix after the change to confirm currently-valid cells stay valid (finding sets shrink post-prune; warning-count test assertions in tests/recipe/ may need updates).
Semantic-rule authors should be told rules now see the pruned graph (document in recipe/AGENTS.md).
Extend _RECIPE_NAMES in tests/recipe/test_backend_reachability.py:24 (currently implementation / implementation-groups / remediation) to include merge-prs: its 8 codex errors are all on backend_supports_git_write-guarded steps, so the sequencing fix alone makes it valid — but nothing currently guards against regression there.
Decide explicitly WHERE the promoted route-guard exemption is consumed: (a) filter findings before compute_recipe_validity (exempted findings stay visible in suggestions), (b) inside the backend-incompatible-skill rule (findings disappear entirely), or (c) inside compute_recipe_validity. The choice changes what callers see in suggestions — pick one and document it.
Adjacent consumer with the same defect class (decide: in-scope or follow-up ticket): validate_from_path (recipe/_api_listing.py:71, backing the validate_recipe MCP tool) runs run_semantic_rules (:131) and NEVER calls _prune_skipped_steps, so it reports backend-incompatible false positives on correctly guarded steps whenever a backend name is supplied.
Date: 2026-06-11 Scope: Regression in open_kitchen(name="implementation") with the Codex backend failing on the first orchestrator message with errors=[] and "unknown structural error"; root cause, test-gap analysis, and process-gap analysis. Mode: Deep Analysis (2 exploration batches + adversarial challenge round + post-report validation; 9 subagents)
Summary
The failure is caused by the interaction of a latent semantic mismatch (present since PR #3515, merged Jun 1) with a new fail-closed guard (PR #3995 / commit 61802dce4, deployed in v0.10.752 — two commits ahead of the local checkout at 8c2ced093). load_and_validate runs semantic validation rules on the unpruned recipe and folds the resulting findings into the valid flag, so any backend-restricted skill referenced by a step — even one correctly guarded by skip_when_false and pruned moments later — produces an ERROR-severity backend-incompatible-skill finding that makes valid=False for Codex sessions. Before #3995 nothing in open_kitchen consumed valid, so the flag's poisoned value was invisible. #3995 added if not result.get("valid", False) or not result.get("content", "") as a fail-closed gate (intended to stop the silent content-wipe bug from run impl-20260610-095047-486925), which now trips on every Codex open_kitchen for the implementation recipe. The misleading message "unknown structural error" arises because the failure envelope renders only the structural errors list (result["errors"], empty) while the findings that actually drove valid=False are ERROR-severity items inside result["suggestions"]. Empirically confirmed against the installed v0.10.752 package: codex → valid=False (13 ERROR findings, content 78–86KB (defer-flag dependent), errors=[]); claude-code → valid=True for implementation. The same guard also breaks open_kitchen("research") for all backends (1 pre-existing ERROR finding), and the same valid=False mismatch has been silently rejecting Codex fleet dispatch at fleet/_api.py:374 since #3515.
Root Cause
Causal chain (all steps empirically confirmed on the installed v0.10.752 package):
Backend capability override.CodexBackend declares git_metadata_writable=False (execution/backends/codex.py:521); _backend_capability_overrides produces {"backend_supports_git_write": "false"} and open_kitchen merges it last (unoverridable) into ingredient overrides (tools_kitchen.py:561), then calls load_and_validate(..., backend_name="codex") (tools_kitchen.py:648-656).
Sequencing defect (the root). In recipe/_api.py (load_and_validate): semantic rules run at line 397 on the unpruned recipe; _prune_skipped_steps runs at line 408; compute_recipe_validity at line 457 consumes the pre-prune findings. The backend-incompatible-skill rule (rules/rules_backend_compat.py:27-68) has no awareness of skip_when_false guards.
13 ERROR findings for codex on implementation. 9 findings are on steps guarded by skip_when_false: inputs.backend_supports_git_write (pruned at load time — pure false positives); 4 are on inputs.open_pr-route-guarded steps that the conformance test exempts via _ROUTE_GUARDED_COMPAT_EXCEPTIONS (tests/recipe/test_backend_reachability.py:31-38) but compute_recipe_validity (registry.py:245-254) does not.
Result shape.valid=False, content non-empty (~78–86KB; pruning and route repair work correctly — no dangling routes), errors=[] (no structural errors), 13 error-severity + ~223 warning-severity items in suggestions.
Misleading message._recipe_validation_error_response (tools_kitchen.py:100-115) renders detail only from result.get("errors", []) (structural list); empty → falls back to the literal "unknown structural error". The actual cause (error-severity semantic findings) sits unrendered in suggestions.
Notes:
_apply_triage_gate only filters rule=="stale-contract" suggestions (repository.py:139-218); it neither affects valid nor strips the error findings.
The visible transcript showed only warning-severity suggestions because the response JSON was truncated in the terminal rendering; the 13 error items are present in the full payload.
src/autoskillit/recipe/registry.py:245-254 (compute_recipe_validity): folds ANY ERROR-severity semantic/contract finding into valid with no prune/route-guard awareness [SUPPORTED]
src/autoskillit/recipe/rules/rules_backend_compat.py:27-68 (backend-incompatible-skill): fires on guarded/prunable steps [SUPPORTED]
src/autoskillit/server/tools/tools_kitchen.py (v0.10.752: guard at :624 and :711; _recipe_validation_error_response at :100-115): gates on valid; renders only structural errors [SUPPORTED]
src/autoskillit/fleet/_api.py:349-394: same valid=False gate (line 374) rejects Codex campaign dispatch (FLEET_RECIPE_INVALID) since Implementation Plan: Thread backend_name Through Recipe Validation API #3515; passes backend_name but no capability overrides, so the internal prune pass runs but removes nothing (the skip_when_false: inputs.backend_supports_git_write guards evaluate against the YAML default 'true') [SUPPORTED]
src/autoskillit/recipes/research.yaml: 1 pre-existing ERROR finding (audit-impl-remediation-route) → valid=False for ALL backends → open_kitchen("research") blocked everywhere on v0.10.752 [SUPPORTED]
src/autoskillit/recipe/_cmd_rpc_issues.py:293,303: \$ in f-strings → import-time SyntaxWarning (cosmetic; separate defect; see External Research) [SUPPORTED]
src/autoskillit/recipe/_api_cache.py (copy_result): cached-result copy does not independently copy the new errors list (aliasing risk only) [NEEDS-EVIDENCE]
Six compounding mechanisms let this reach runtime:
The commit's own test was one assertion short.test_load_and_validate_produces_valid_content_after_step_pruning (added by Rectify: Surface LoadRecipeResult Errors and Fail-Closed on Invalid Content #3995, tests/recipe/test_api.py) calls the REAL load_and_validate with codex + backend_supports_git_write=false — the exact production scenario — but asserts only "no dangling-route errors" and "content non-empty". Adding assert result["valid"] is True would have failed and caught the regression pre-merge.
The conformance test validates the right invariant in the wrong order.test_pruned_recipe_has_no_backend_incompatible_findings (Implementation Plan: Backend-Parametrized Recipe Reachability Tests + Codex Smoke Pipeline Variant #3985, tests/recipe/test_backend_reachability.py:50-75) calls _prune_skipped_steps FIRST, then run_semantic_rules on the pruned recipe — the correct order, which production does not use. It bypasses load_and_validate entirely, so it verifies the desired property while the production code path computes the opposite.
Every open_kitchen-level test hand-mocks load_and_validate. All gating tests (test_tools_kitchen_envelope.py, test_tools_kitchen_visibility.py, test_open_kitchen_deferred_recall.py, test_tools_kitchen_gate.py) construct the result dict by hand. The new Rectify: Surface LoadRecipeResult Errors and Fail-Closed on Invalid Content #3995 envelope tests mock valid=False WITH non-empty errors — the one shape production never produces in this failure (valid=False + errors=[]) is untested.
No end-to-end test crosses the recipe→server boundary. No test exercises bundled YAML → real load_and_validate (with backend capability overrides) → open_kitchen gating decision, for any backend.
CI excludes the only tests that could see it at runtime..github/workflows/tests.yml runs task test-check, whose Taskfile definition (Taskfile.yml:70) bakes in -m "not smoke and not canary"; the Codex smoke target (task test-smoke-codex) is manual-only, credential-gated, and only validates raw codex exec NDJSON — it never opens a kitchen. PR CI additionally runs path-filtered (AUTOSKILLIT_TEST_FILTER=conservative); the server cascade does not include recipe/ integration tests, so a server-only diff would skip them entirely (in Rectify: Surface LoadRecipeResult Errors and Fail-Closed on Invalid Content #3995's case recipe/_api.py was touched, so recipe/test_api.py DID run — and passed, see gap 1).
Review gates are diff-scoped and text-scoped. dry-walkthrough, audit-impl, and review-pr all ran on Rectify: Surface LoadRecipeResult Errors and Fail-Closed on Invalid Content #3995 (per the commit's pipeline signature). None has an instruction to (a) verify that a newly-consumed key's semantics match the guard's intent when producer logic lives OUTSIDE the diff (compute_recipe_validity and rules_backend_compat.py were not in the diff), or (b) empirically execute the gated code path. The dry-walkthrough Historical Regression Check only catches symbol-level re-additions of previously deleted code, not same-failure-mode recurrence through new symbols.
Similar Patterns
Producer/consumer key-contract tests exist but were not applied to this boundary.tests/contracts/test_hook_bridge_coverage.py asserts _quota_guard_hook_payload() produces exactly the keys its consumer reads — the precise pattern missing for LoadRecipeResult → tools_kitchen.py. Existing test_load_recipe_result_is_typed checks only TypedDict annotations, not runtime values or semantics.
validate_from_path (the validate_recipe MCP tool) already returns {"valid", "errors", "findings", ...} — Rectify: Surface LoadRecipeResult Errors and Fail-Closed on Invalid Content #3995's added errors field mirrored that shape, but load_and_validate puts semantic findings in suggestions, not errors, which is exactly the asymmetry the failure envelope tripped over.
fleet/_api.py:374 is the older sibling of the same defect: gates dispatch on valid while error detail lives in suggestions (it at least renders error-severity suggestions in its message — the open_kitchen envelope does not).
Design Intent Findings
LoadRecipeResult contract: created in aa95ed0dd (Split Monolithic Files — recipe/ Package Root (3 Files) #2948, monolith split) with no errors field; structural error strings were computed and then discarded at the TypedDict boundary. errors field added by 61802dce4 (Rectify: Surface LoadRecipeResult Errors and Fail-Closed on Invalid Content #3995, 2026-06-10). No AGENTS.md/doc defines the result schema or a *Result convention; the TypedDict (_recipe_ingredients.py:110, total=False) is the only contract. Consumers: tools_kitchen.py, fleet/_api.py, hooks/formatters/_fmt_recipe.py, _api.py cache layer. [SUPPORTED]
Step pruning + route repair (_prune_skipped_steps): works correctly in v0.10.752 — 9 codex-incompatible steps pruned with no dangling routes (8 when=None catch-alls + 1 on_success target). The prune machinery is NOT the defect; the defect is validating before pruning. [SUPPORTED]
Historical Context
Part A (project log mining): no prior /investigate logs found in ~/.claude/projects/-home-talon-projects-generic_automation_mcp matching this root cause.
Part B/C (git history): this is the sixth fix in ~14 days in the same "Codex backend × recipe pruning/content delivery" family — and #3995 itself says so in its commit message ("the sixth manifestation of the step-pruning/content-divergence family"):
Every fix addressed the layer where the symptom appeared; none added the end-to-end invariant "every bundled recipe × supported backend loads with valid=True and non-empty content through the production path." Additionally, parallel branches diverged during #3995's development (b62f26b81 duplicated a test-isolation fixture; ff18b21e4 — NOT an ancestor of the merged commit — carried overlapping YAML fixes), indicating concurrent uncoordinated work on the same surface.
This is a recurring pattern — consider running /rectify for architectural immunity after resolving the immediate issue.
External Research
No external library involvement; the defect is internal logic. Two empirically verified environment facts (local verification, not web research):
The installed tool (uv tool env, Python 3.13.2) at v0.10.752 is byte-identical to origin/develop (dae5f5ded) for all files examined; no local recipe shadowing exists in ~/projects/token-reserve/.autoskillit/recipes/ (directory absent).
The SyntaxWarning: invalid escape sequence '\$' at _cmd_rpc_issues.py:293,303 comes from \$ inside f-strings (GraphQL variable syntax cargo-culted from shell escaping — Python needs no escape for $). Cosmetic on Python 3.13 (compile-time warning, emitted once before .pyc caching); Python has long signaled invalid escapes will eventually become hard errors, so it should be fixed (drop the backslashes), but it is unrelated to the regression.
Scope Boundary
Investigated: open_kitchen gating flow (both first-open and deferred-recall paths) in v0.10.752; load_and_validate sequencing; compute_recipe_validity; backend-incompatible-skill rule; backend capability override derivation; all 8 bundled recipes × {claude-code, codex} validity matrix; #3995's full diff, intent, tests, and pipeline gates; fleet dispatch path; CI workflow/smoke coverage; test path filtering; review-skill checklists; 6-commit recurrence history; the \$ SyntaxWarning.
Not yet explored: whether Codex fleet campaigns were ever attempted/observed failing in production telemetry since #3515 (code-level breakage is confirmed; runtime occurrence not mined); the errors-list aliasing in _api_cache.copy_result (flagged, unconfirmed impact); whether other semantic rules besides backend-incompatible-skill and audit-impl-remediation-route can produce ERROR findings that poison valid for specific configurations; the exact fix for the research recipe's audit-impl-remediation-route error.
Recommendations
Single converged recommendation (three coordinated changes — one root fix, one contract fix, one reporting fix):
Root fix — compute validity from the post-prune recipe with shared route-guard exemptions. In load_and_validate, run semantic rules (or at minimum re-derive the validity-relevant findings) on the pruned recipe, and move _ROUTE_GUARDED_COMPAT_EXCEPTIONS out of the test into a production constant consumed by both the backend-incompatible-skill rule path and the conformance test (single source of truth). This makes valid mean what every consumer assumes — "this recipe, as it will actually be served, is usable" — and simultaneously repairs Codex fleet dispatch (fleet/_api.py:374) which has been broken since Implementation Plan: Thread backend_name Through Recipe Validation API #3515. Implementer notes (validated): (a) make_validation_context is currently built from the pre-prune active_recipe — it must be rebuilt with the pruned recipe, not reused; (b) fleet's call must also pass the backend capability overrides (see _promote_capability_keys at tools_kitchen.py:560, which makes them unoverridable in the open_kitchen path) so fleet's pruning matches what would be served; (c) confirm the _api_cache.copy_resulterrors-list aliasing before shipping, since cached findings must reflect post-prune state; (d) re-run the 8×2 validity matrix after the change to confirm the currently-valid cells stay valid.
Contract fix — encode the envelope invariant. Enforce (by construction or contract test) that whenever valid=False, the failure envelope can name a cause: _recipe_validation_error_response must merge error-severity suggestions into the rendered detail when errors is empty, so "unknown structural error" becomes structurally unreachable. Extend the test_hook_bridge_coverage.py producer/consumer pattern to the LoadRecipeResult → tools_kitchen.py boundary.
Test-gap fix — the missing end-to-end invariant. Add a parametrized test (the challenge round's matrix probe is effectively its prototype): for every bundled recipe × supported backend (with that backend's real capability overrides), call the REAL load_and_validate and assert valid is True, content non-empty, errors == []; plus one server-layer test driving real open_kitchen on a bundled recipe. This single test fails today for 5 codex cells and 1 claude-code cell (research), i.e., it catches both the regression and the latent research-recipe defect. Wire at least the claude-code/codex matrix variant into CI (it requires no API credentials — it is pure validation, not a live session).
Also: fix the research recipe's audit-impl-remediation-route ERROR (currently blocks open_kitchen("research") on all backends) — note this needs its own analysis, because change #1 will NOT fix it if the rule fires on a step that survives pruning; and remove the \$ escapes in _cmd_rpc_issues.py:293,303.
Killed alternatives:
Gate open_kitchen on content/errors only (drop valid from the guard). Cheapest fix and faithful to Rectify: Surface LoadRecipeResult Errors and Fail-Closed on Invalid Content #3995's original intent, but killed as the primary because it leaves valid semantically poisoned — Codex fleet dispatch stays broken, and the next consumer of valid reintroduces this class of failure.
Make backend-incompatible-skill simulate pruning internally (skip guarded steps). Killed: duplicates _prune_skipped_steps evaluation logic inside a rule (redundancy → drift risk), and other present/future rules would still see the unpruned graph.
Breakage surface: every consumer of suggestions/findings counts — hooks/formatters/_fmt_recipe.py (finding-count rendering), fleet/_api.py:374-388 (rejection messages), warning-count assertions in tests/recipe/ (post-prune evaluation reduces the ~236-finding set), and the deferred-recall path's defer_unresolved=False recall (tools_kitchen.py:583) which may prune differently from the first pass — both passes must use the same finding basis or first-open vs recall can disagree on valid.
Prior reverts: none found on _api.py validation ordering (git log --grep="revert" on the mechanism files: no hits); however 6 prior fixes in 14 days on this surface mean any change here needs the new E2E matrix test in the SAME commit.
Downstream contract violations: LoadRecipeResult shape is unchanged (no key changes), so formatter/cache contracts hold; semantic-rule authors must know rules now see the pruned graph (document in recipe/AGENTS.md).
Risk level: MEDIUM (multi-consumer behavioral shift, mitigated by the matrix test landing first/with it).
Breakage surface: callers parsing user_visible_message strings (none found doing exact-match); _FMT_OPEN_KITCHEN_SUPPRESSED in hooks/formatters/_fmt_recipe.py suppresses errors from agent-visible output and may need the merged detail whitelisted.
Summary
open_kitchen(name="implementation")fails on the first orchestrator message of every Codex-backend session in v0.10.752 with:The fail-closed guard added by PR #3995 (
61802dce4) gates onresult["valid"], butload_and_validatecomputesvalidfrom semantic-rule findings evaluated on the unpruned recipe. With the Codex capability override (backend_supports_git_write="false"), thebackend-incompatible-skillrule emits 13 ERROR-severity findings — 9 on steps that are pruned immediately afterward (skip_when_false: inputs.backend_supports_git_write) and 4 oninputs.open_pr-route-guarded steps that the conformance test explicitly exempts via_ROUTE_GUARDED_COMPAT_EXCEPTIONSbutcompute_recipe_validitydoes not. Sovalid=Falseeven though the served (pruned) content is structurally sound (non-empty: 85,728 bytes withdefer_unresolved=True/ 77,686 withFalse; zero dangling routes,errors=[]).This is a latent semantic mismatch dating to PR #3515 (merged Jun 1), which threaded
backend_nameinto validation and silently expanded the meaning ofvalidfrom "structurally usable" to "no backend-incompatible step anywhere in the YAML, even guarded ones." No consumer was updated for the expansion:tools_kitchen.py:711, deferred-recall:624) now fails closed on it → total Codex breakage.fleet/_api.py:374; the gate itself predates Implementation Plan: Thread backend_name Through Recipe Validation API #3515 — added by Rectify: Dispatch Validation Bypass — Fleet Dispatch Skips All Semantic Rules #2347) has been rejecting Codex campaigns withFLEET_RECIPE_INVALIDsince Implementation Plan: Thread backend_name Through Recipe Validation API #3515 threadedbackend_nameinto fleet's validation call. Fleet passes no capability overrides toload_and_validate, so its internal prune pass removes nothing.Backend impact matrix (empirical, installed v0.10.752)
(The
researchrow is a distinct pre-existing recipe defect — tracked separately; see Related work.)Root cause chain (all empirically confirmed on installed v0.10.752)
CodexBackenddeclaresgit_metadata_writable=False(execution/backends/codex.py:521);_backend_capability_overridesyields{"backend_supports_git_write": "false"}, merged last/unoverridable (tools_kitchen.py:560-561via_promote_capability_keys).recipe/_api.py(load_and_validate):run_semantic_rulesruns at line 397 on the unpruned recipe →_prune_skipped_stepsat line 408 →compute_recipe_validityat line 457 consumes the pre-prune findings.backend-incompatible-skill(rules/rules_backend_compat.py:27-68) has no awareness ofskip_when_falseguards or route-guard exemptions → 13 ERROR findings for codex.compute_recipe_validity(registry.py:245-254) folds any ERROR-severity semantic/contract finding intovalid→valid=False,contentnon-empty,errors=[].not result.get("valid", False); the envelope renders only the structuralerrorslist → misleading "unknown structural error" (envelope rendering is a separate parallel ticket; see Related work).Scope of fix (from validated investigation; killed alternatives documented in the appended report)
_prune_skipped_steps. TheValidationContextmust be rebuilt with the pruned recipe — not reused from the pre-prune build._ROUTE_GUARDED_COMPAT_EXCEPTIONS(currently test-only:tests/recipe/test_backend_reachability.py:31-38) into a production constant consumed by both the validity computation path and the conformance test.fleet/_api.pydispatch validation must pass the backend capability overrides (mirroring_promote_capability_keysin the open_kitchen path) so fleet's pruning matches what would actually be served. Note fleet already contains an IL-safe local mirror,_build_capability_overrides(fleet/_api.py:112, currently used only foreffective_ingredientsat :422, not for validation) — this is a one-line call-site change.load_and_validatewith real capability overrides, assertingvalid is True, non-emptycontent,errors == []. Note: the test added by Rectify: Surface LoadRecipeResult Errors and Fail-Closed on Invalid Content #3995 (test_load_and_validate_produces_valid_content_after_step_pruning) was one assertion short — it never assertsvalid.Implementer cautions (validated):
_api_cache.copy_resulterrors-list aliasing before shipping (cached findings must reflect post-prune state).defer_unresolved=Falsevs first-opennot _user_overrides_present— both passes must agree on the finding basis or first-open vs recall can disagree onvalid.tests/recipe/may need updates).recipe/AGENTS.md)._RECIPE_NAMESintests/recipe/test_backend_reachability.py:24(currently implementation / implementation-groups / remediation) to includemerge-prs: its 8 codex errors are all onbackend_supports_git_write-guarded steps, so the sequencing fix alone makes it valid — but nothing currently guards against regression there.compute_recipe_validity(exempted findings stay visible insuggestions), (b) inside thebackend-incompatible-skillrule (findings disappear entirely), or (c) insidecompute_recipe_validity. The choice changes what callers see insuggestions— pick one and document it.validate_from_path(recipe/_api_listing.py:71, backing thevalidate_recipeMCP tool) runsrun_semantic_rules(:131) and NEVER calls_prune_skipped_steps, so it reports backend-incompatible false positives on correctly guarded steps whenever a backend name is supplied.Related work
valid=Truefor codex × implementation/planner viaload_and_validatewith no xfail machinery — it cannot pass CI until this lands. T5-P3-A1-WP1 Provide a merge-blocking CI test that proves every bundled recipe composes validly under every registered backend, with explicit data-table governance for known unsupported combos and structural me... #4008 can land first: its designxfail(strict=True)s should-work-but-broken cells (passing CI while the bug exists); after this fix those markers must be removed (strict xfail turns XPASS into failure). This fix is a prerequisite of T5-P4-A8-WP2 Prove that planner.yaml composes valid:true under the codex backend capability override set. #4026 and a marker-removal trigger for T5-P3-A1-WP1 Provide a merge-blocking CI test that proves every bundled recipe composes validly under every registered backend, with explicit data-table governance for known unsupported combos and structural me... #4008 — an extension of neither.tools_kitchen.py; no file overlap with this ticket's scope (recipe/_api.py,registry.py,rules_backend_compat.py,fleet/_api.py).Parallel split
Filed alongside (no file overlap, parallel-safe):
_cmd_rpc_issues.py\$SyntaxWarningInvestigation
Investigation: open_kitchen "unknown structural error" — Codex Backend Total Breakage (v0.10.752)
Date: 2026-06-11
Scope: Regression in
open_kitchen(name="implementation")with the Codex backend failing on the first orchestrator message witherrors=[]and "unknown structural error"; root cause, test-gap analysis, and process-gap analysis.Mode: Deep Analysis (2 exploration batches + adversarial challenge round + post-report validation; 9 subagents)
Summary
The failure is caused by the interaction of a latent semantic mismatch (present since PR #3515, merged Jun 1) with a new fail-closed guard (PR #3995 / commit
61802dce4, deployed in v0.10.752 — two commits ahead of the local checkout at8c2ced093).load_and_validateruns semantic validation rules on the unpruned recipe and folds the resulting findings into thevalidflag, so any backend-restricted skill referenced by a step — even one correctly guarded byskip_when_falseand pruned moments later — produces an ERROR-severitybackend-incompatible-skillfinding that makesvalid=Falsefor Codex sessions. Before #3995 nothing inopen_kitchenconsumedvalid, so the flag's poisoned value was invisible. #3995 addedif not result.get("valid", False) or not result.get("content", "")as a fail-closed gate (intended to stop the silent content-wipe bug from runimpl-20260610-095047-486925), which now trips on every Codexopen_kitchenfor the implementation recipe. The misleading message "unknown structural error" arises because the failure envelope renders only the structural errors list (result["errors"], empty) while the findings that actually drovevalid=Falseare ERROR-severity items insideresult["suggestions"]. Empirically confirmed against the installed v0.10.752 package: codex →valid=False(13 ERROR findings, content 78–86KB (defer-flag dependent),errors=[]); claude-code →valid=Truefor implementation. The same guard also breaksopen_kitchen("research")for all backends (1 pre-existing ERROR finding), and the samevalid=Falsemismatch has been silently rejecting Codex fleet dispatch atfleet/_api.py:374since #3515.Root Cause
Causal chain (all steps empirically confirmed on the installed v0.10.752 package):
CodexBackenddeclaresgit_metadata_writable=False(execution/backends/codex.py:521);_backend_capability_overridesproduces{"backend_supports_git_write": "false"}andopen_kitchenmerges it last (unoverridable) into ingredient overrides (tools_kitchen.py:561), then callsload_and_validate(..., backend_name="codex")(tools_kitchen.py:648-656).recipe/_api.py(load_and_validate): semantic rules run at line 397 on the unpruned recipe;_prune_skipped_stepsruns at line 408;compute_recipe_validityat line 457 consumes the pre-prune findings. Thebackend-incompatible-skillrule (rules/rules_backend_compat.py:27-68) has no awareness ofskip_when_falseguards.skip_when_false: inputs.backend_supports_git_write(pruned at load time — pure false positives); 4 are oninputs.open_pr-route-guarded steps that the conformance test exempts via_ROUTE_GUARDED_COMPAT_EXCEPTIONS(tests/recipe/test_backend_reachability.py:31-38) butcompute_recipe_validity(registry.py:245-254) does not.valid=False,contentnon-empty (~78–86KB; pruning and route repair work correctly — no dangling routes),errors=[](no structural errors), 13 error-severity + ~223 warning-severity items insuggestions.tools_kitchen.py:711first open;:624deferred-recall — the user's second call hit this one) fires onnot result.get("valid", False)._recipe_validation_error_response(tools_kitchen.py:100-115) renders detail only fromresult.get("errors", [])(structural list); empty → falls back to the literal"unknown structural error". The actual cause (error-severity semantic findings) sits unrendered insuggestions.Notes:
_apply_triage_gateonly filtersrule=="stale-contract"suggestions (repository.py:139-218); it neither affectsvalidnor strips the error findings.Affected Components
src/autoskillit/recipe/_api.py(load_and_validate, lines 352-457): rules-before-prune sequencing; validity computed from pre-prune findings [SUPPORTED]src/autoskillit/recipe/registry.py:245-254(compute_recipe_validity): folds ANY ERROR-severity semantic/contract finding intovalidwith no prune/route-guard awareness [SUPPORTED]src/autoskillit/recipe/rules/rules_backend_compat.py:27-68(backend-incompatible-skill): fires on guarded/prunable steps [SUPPORTED]src/autoskillit/server/tools/tools_kitchen.py(v0.10.752: guard at :624 and :711;_recipe_validation_error_responseat :100-115): gates onvalid; renders only structural errors [SUPPORTED]src/autoskillit/fleet/_api.py:349-394: samevalid=Falsegate (line 374) rejects Codex campaign dispatch (FLEET_RECIPE_INVALID) since Implementation Plan: Thread backend_name Through Recipe Validation API #3515; passesbackend_namebut no capability overrides, so the internal prune pass runs but removes nothing (theskip_when_false: inputs.backend_supports_git_writeguards evaluate against the YAML default'true') [SUPPORTED]src/autoskillit/recipes/research.yaml: 1 pre-existing ERROR finding (audit-impl-remediation-route) →valid=Falsefor ALL backends →open_kitchen("research")blocked everywhere on v0.10.752 [SUPPORTED]src/autoskillit/recipe/_cmd_rpc_issues.py:293,303:\$in f-strings → import-timeSyntaxWarning(cosmetic; separate defect; see External Research) [SUPPORTED]src/autoskillit/recipe/_api_cache.py(copy_result): cached-result copy does not independently copy the newerrorslist (aliasing risk only) [NEEDS-EVIDENCE]Backend impact matrix (empirical, installed v0.10.752):
Data Flow
Test Gap Analysis
Six compounding mechanisms let this reach runtime:
test_load_and_validate_produces_valid_content_after_step_pruning(added by Rectify: Surface LoadRecipeResult Errors and Fail-Closed on Invalid Content #3995,tests/recipe/test_api.py) calls the REALload_and_validatewith codex +backend_supports_git_write=false— the exact production scenario — but asserts only "no dangling-route errors" and "content non-empty". Addingassert result["valid"] is Truewould have failed and caught the regression pre-merge.test_pruned_recipe_has_no_backend_incompatible_findings(Implementation Plan: Backend-Parametrized Recipe Reachability Tests + Codex Smoke Pipeline Variant #3985,tests/recipe/test_backend_reachability.py:50-75) calls_prune_skipped_stepsFIRST, thenrun_semantic_ruleson the pruned recipe — the correct order, which production does not use. It bypassesload_and_validateentirely, so it verifies the desired property while the production code path computes the opposite.load_and_validate. All gating tests (test_tools_kitchen_envelope.py,test_tools_kitchen_visibility.py,test_open_kitchen_deferred_recall.py,test_tools_kitchen_gate.py) construct the result dict by hand. The new Rectify: Surface LoadRecipeResult Errors and Fail-Closed on Invalid Content #3995 envelope tests mockvalid=FalseWITH non-emptyerrors— the one shape production never produces in this failure (valid=False+errors=[]) is untested.load_and_validate(with backend capability overrides) →open_kitchengating decision, for any backend..github/workflows/tests.ymlrunstask test-check, whose Taskfile definition (Taskfile.yml:70) bakes in-m "not smoke and not canary"; the Codex smoke target (task test-smoke-codex) is manual-only, credential-gated, and only validates rawcodex execNDJSON — it never opens a kitchen. PR CI additionally runs path-filtered (AUTOSKILLIT_TEST_FILTER=conservative); theservercascade does not includerecipe/integration tests, so a server-only diff would skip them entirely (in Rectify: Surface LoadRecipeResult Errors and Fail-Closed on Invalid Content #3995's caserecipe/_api.pywas touched, sorecipe/test_api.pyDID run — and passed, see gap 1).compute_recipe_validityandrules_backend_compat.pywere not in the diff), or (b) empirically execute the gated code path. The dry-walkthrough Historical Regression Check only catches symbol-level re-additions of previously deleted code, not same-failure-mode recurrence through new symbols.Similar Patterns
tests/contracts/test_hook_bridge_coverage.pyasserts_quota_guard_hook_payload()produces exactly the keys its consumer reads — the precise pattern missing forLoadRecipeResult→tools_kitchen.py. Existingtest_load_recipe_result_is_typedchecks only TypedDict annotations, not runtime values or semantics.validate_from_path(thevalidate_recipeMCP tool) already returns{"valid", "errors", "findings", ...}— Rectify: Surface LoadRecipeResult Errors and Fail-Closed on Invalid Content #3995's addederrorsfield mirrored that shape, butload_and_validateputs semantic findings insuggestions, noterrors, which is exactly the asymmetry the failure envelope tripped over.fleet/_api.py:374is the older sibling of the same defect: gates dispatch onvalidwhile error detail lives insuggestions(it at least renders error-severity suggestions in its message — the open_kitchen envelope does not).Design Intent Findings
LoadRecipeResultcontract: created inaa95ed0dd(Split Monolithic Files — recipe/ Package Root (3 Files) #2948, monolith split) with noerrorsfield; structural error strings were computed and then discarded at the TypedDict boundary.errorsfield added by61802dce4(Rectify: Surface LoadRecipeResult Errors and Fail-Closed on Invalid Content #3995, 2026-06-10). No AGENTS.md/doc defines the result schema or a*Resultconvention; the TypedDict (_recipe_ingredients.py:110,total=False) is the only contract. Consumers:tools_kitchen.py,fleet/_api.py,hooks/formatters/_fmt_recipe.py,_api.pycache layer. [SUPPORTED]validflag semantics drift:compute_recipe_validitydocstring says "no schema, semantic, or contract errors." Befored1bdab3f8(Implementation Plan: Thread backend_name Through Recipe Validation API #3515, merged Jun 1) threadedbackend_namethrough validation,backend-incompatible-skillalways returned[], sovalidde facto meant "structurally usable." After Implementation Plan: Thread backend_name Through Recipe Validation API #3515 it silently expanded to "no backend-incompatible steps anywhere in the YAML, even guarded ones." No consumer was updated for the expansion;fleet/_api.py:374(the only pre-Rectify: Surface LoadRecipeResult Errors and Fail-Closed on Invalid Content #3995 gate onvalid) began rejecting Codex dispatch from that day. [SUPPORTED]61802dce4to rectify runimpl-20260610-095047-486925/ issue Silent recipe content wipe on codex: unrepairable dangling route after pruning #3992, whereopen_kitchenstampedsuccess: trueovercontent=""+valid=False(dangling-route wipe) and the Codex orchestrator improvised the pipeline from the Mermaid diagram. The guard's intent (never deliver empty/broken content silently) is sound; its predicate (valid) conflates "advisory findings exist" with "content is unusable." [SUPPORTED]_prune_skipped_steps): works correctly in v0.10.752 — 9 codex-incompatible steps pruned with no dangling routes (8when=Nonecatch-alls + 1on_successtarget). The prune machinery is NOT the defect; the defect is validating before pruning. [SUPPORTED]Historical Context
Part A (project log mining): no prior
/investigatelogs found in~/.claude/projects/-home-talon-projects-generic_automation_mcpmatching this root cause.Part B/C (git history): this is the sixth fix in ~14 days in the same "Codex backend × recipe pruning/content delivery" family — and #3995 itself says so in its commit message ("the sixth manifestation of the step-pruning/content-divergence family"):
3c50c8fc0293283d22358c8a168207deda91884e1f2ae61802dce4validassertion)Every fix addressed the layer where the symptom appeared; none added the end-to-end invariant "every bundled recipe × supported backend loads with
valid=Trueand non-empty content through the production path." Additionally, parallel branches diverged during #3995's development (b62f26b81duplicated a test-isolation fixture;ff18b21e4— NOT an ancestor of the merged commit — carried overlapping YAML fixes), indicating concurrent uncoordinated work on the same surface.This is a recurring pattern — consider running
/rectifyfor architectural immunity after resolving the immediate issue.External Research
No external library involvement; the defect is internal logic. Two empirically verified environment facts (local verification, not web research):
origin/develop(dae5f5ded) for all files examined; no local recipe shadowing exists in~/projects/token-reserve/.autoskillit/recipes/(directory absent).SyntaxWarning: invalid escape sequence '\$'at_cmd_rpc_issues.py:293,303comes from\$inside f-strings (GraphQL variable syntax cargo-culted from shell escaping — Python needs no escape for$). Cosmetic on Python 3.13 (compile-time warning, emitted once before.pyccaching); Python has long signaled invalid escapes will eventually become hard errors, so it should be fixed (drop the backslashes), but it is unrelated to the regression.Scope Boundary
Investigated: open_kitchen gating flow (both first-open and deferred-recall paths) in v0.10.752;
load_and_validatesequencing;compute_recipe_validity;backend-incompatible-skillrule; backend capability override derivation; all 8 bundled recipes × {claude-code, codex} validity matrix; #3995's full diff, intent, tests, and pipeline gates; fleet dispatch path; CI workflow/smoke coverage; test path filtering; review-skill checklists; 6-commit recurrence history; the\$SyntaxWarning.Not yet explored: whether Codex fleet campaigns were ever attempted/observed failing in production telemetry since #3515 (code-level breakage is confirmed; runtime occurrence not mined); the
errors-list aliasing in_api_cache.copy_result(flagged, unconfirmed impact); whether other semantic rules besidesbackend-incompatible-skillandaudit-impl-remediation-routecan produce ERROR findings that poisonvalidfor specific configurations; the exact fix for the research recipe'saudit-impl-remediation-routeerror.Recommendations
Single converged recommendation (three coordinated changes — one root fix, one contract fix, one reporting fix):
load_and_validate, run semantic rules (or at minimum re-derive the validity-relevant findings) on the pruned recipe, and move_ROUTE_GUARDED_COMPAT_EXCEPTIONSout of the test into a production constant consumed by both thebackend-incompatible-skillrule path and the conformance test (single source of truth). This makesvalidmean what every consumer assumes — "this recipe, as it will actually be served, is usable" — and simultaneously repairs Codex fleet dispatch (fleet/_api.py:374) which has been broken since Implementation Plan: Thread backend_name Through Recipe Validation API #3515. Implementer notes (validated): (a)make_validation_contextis currently built from the pre-pruneactive_recipe— it must be rebuilt with the pruned recipe, not reused; (b) fleet's call must also pass the backend capability overrides (see_promote_capability_keysattools_kitchen.py:560, which makes them unoverridable in the open_kitchen path) so fleet's pruning matches what would be served; (c) confirm the_api_cache.copy_resulterrors-list aliasing before shipping, since cached findings must reflect post-prune state; (d) re-run the 8×2 validity matrix after the change to confirm the currently-valid cells stay valid.valid=False, the failure envelope can name a cause:_recipe_validation_error_responsemust merge error-severitysuggestionsinto the rendered detail whenerrorsis empty, so "unknown structural error" becomes structurally unreachable. Extend thetest_hook_bridge_coverage.pyproducer/consumer pattern to theLoadRecipeResult→tools_kitchen.pyboundary.load_and_validateand assertvalid is True,contentnon-empty,errors == []; plus one server-layer test driving realopen_kitchenon a bundled recipe. This single test fails today for 5 codex cells and 1 claude-code cell (research), i.e., it catches both the regression and the latent research-recipe defect. Wire at least the claude-code/codex matrix variant into CI (it requires no API credentials — it is pure validation, not a live session).Also: fix the research recipe's
audit-impl-remediation-routeERROR (currently blocksopen_kitchen("research")on all backends) — note this needs its own analysis, because change #1 will NOT fix it if the rule fires on a step that survives pruning; and remove the\$escapes in_cmd_rpc_issues.py:293,303.Killed alternatives:
content/errorsonly (dropvalidfrom the guard). Cheapest fix and faithful to Rectify: Surface LoadRecipeResult Errors and Fail-Closed on Invalid Content #3995's original intent, but killed as the primary because it leavesvalidsemantically poisoned — Codex fleet dispatch stays broken, and the next consumer ofvalidreintroduces this class of failure.backend-incompatible-skillsimulate pruning internally (skip guarded steps). Killed: duplicates_prune_skipped_stepsevaluation logic inside a rule (redundancy → drift risk), and other present/future rules would still see the unpruned graph.Breakage Analysis
suggestions/findings counts —hooks/formatters/_fmt_recipe.py(finding-count rendering),fleet/_api.py:374-388(rejection messages), warning-count assertions intests/recipe/(post-prune evaluation reduces the ~236-finding set), and the deferred-recall path'sdefer_unresolved=Falserecall (tools_kitchen.py:583) which may prune differently from the first pass — both passes must use the same finding basis or first-open vs recall can disagree onvalid._api.pyvalidation ordering (git log --grep="revert"on the mechanism files: no hits); however 6 prior fixes in 14 days on this surface mean any change here needs the new E2E matrix test in the SAME commit.LoadRecipeResultshape is unchanged (no key changes), so formatter/cache contracts hold; semantic-rule authors must know rules now see the pruned graph (document inrecipe/AGENTS.md).user_visible_messagestrings (none found doing exact-match);_FMT_OPEN_KITCHEN_SUPPRESSEDinhooks/formatters/_fmt_recipe.pysuppresseserrorsfrom agent-visible output and may need the merged detail whitelisted.