Rectify: Surface LoadRecipeResult Errors and Fail-Closed on Invalid Content

[Trecek]

Summary

open_kitchen silently returns content: "" with success: true when post-prune validation detects dangling routes or route-consistency errors. The architectural weakness is threefold:

1. load_and_validate() computes structural errors but discards them — LoadRecipeResult has no errors field, so the reason for valid=False is lost at the TypedDict boundary.
2. open_kitchen stamps success: true unconditionally — it never checks result["valid"] or result["content"] before returning.
3. _prune_skipped_steps produces unrepairable dangling routes silently — when a pruned step has no computable redirect (no on_success, no when=None catch-all in on_result), the route-repair loop is a no-op, and the only signal is a list of strings returned from _validate_no_dangling_routes that gets appended to a local variable and then discarded.

Part A addresses the core data-loss path: surfacing structural errors through LoadRecipeResult, making open_kitchen fail-closed on invalid content, and adding tests that catch this class of bug. Part B will address the recipe YAML fixes (adding when=None catch-all to resolve_review) and broader vacuous-assertion cleanup.

Problem

When the orchestrator backend is Codex, open_kitchen(name="implementation") returns success: true, valid: false, and content: "" — the entire recipe body is silently withheld. The orchestrator receives no explanation for the empty content, has no prompt instruction for this state, and is structurally blocked from every recovery path (recipe_read_guard.py PreToolUse hook + sous-chef raw-read prohibition + load_recipe returning the same empty result).

In run impl-20260610-095047-486925 (TalonT-Org/the-token-reserve#147, PR TalonT-Org/the-token-reserve#189, AutoSkillit 0.10.745, 2026-06-10 09:49–11:31), the Codex orchestrator improvised the entire 60-minute pipeline from the Mermaid diagram field, the ingredients table, and the sous-chef prompt — executing steps the composition had pruned as backend-incompatible (e.g. implement → implement-worktree-no-merge, declared requires backend ['claude-code']), with no step routing, no capture: contracts, no retry counts, and zero record_pipeline_step calls. The PR happened to merge green, but the run cannot be trusted to have followed the composed plan.

Reproduced at repo HEAD 18f410893 (0.10.748). This is the sixth manifestation of the step-pruning/content-divergence family.

Root cause chain

1. Guard introduction (2026-06-09): 207deda91 (Implementation Plan: Backend-Conditional Routing — gate_backend_write Step #3960) introduced the backend_supports_git_write ingredient and added skip_when_false: inputs.backend_supports_git_write to 7 steps in implementation.yaml (and 7 in remediation.yaml) including resolve_review — without giving resolve_review a when=None catch-all route. The bug became reachable <24h before the failing run.
2. Codex capability override: _backend_capability_overrides() (src/autoskillit/server/tools/_auto_overrides.py:19-28) maps Codex's git_metadata_writable=False to backend_supports_git_write="false", injected into session overrides at src/autoskillit/server/tools/tools_kitchen.py:540, promoted over user-supplied overrides via _promote_capability_keys (line 542), and passed to load_and_validate (lines 627-635).
3. Unrepairable route: _prune_skipped_steps() (src/autoskillit/recipe/_recipe_composition.py:294-415 (redirect derivation 352-364)) prunes resolve_review but derives redirect = None — the step has no on_success and all four of its on_result conditions carry when clauses (the repair logic deliberately never derives a redirect from on_failure/on_exhausted) (src/autoskillit/recipes/implementation.yaml:1089-1097). The surviving enrich_diff_context (on_success: resolve_review, on_failure: resolve_review, implementation.yaml:1064-1065) is left dangling. The code comment acknowledges this path: "redirect stays None and _validate_no_dangling_routes catches dangling refs."
4. Silent content wipe: _validate_no_dangling_routes returns ["Step 'enrich_diff_context' routes to unknown step 'resolve_review'"] and load_and_validate() sets raw = "" (src/autoskillit/recipe/_api.py:415-418). The errors list feeds only compute_recipe_validity(); LoadRecipeResult has no errors field, so the reason never reaches the caller.
5. No fail signal, no recovery: open_kitchen returns success: true, kitchen: "open" without checking valid/content (tools_kitchen.py:690-692; the deferred-recall branch at 605-607 is equally unchecked). Neither sous-chef SKILL.md nor any orchestrator prompt covers content: "". The [FIX] Rectify: Backend-Specific Orchestrator Prompt Supplements — Codex Raw Recipe File Read Bypass — PART A #3736/Implementation Plan: Codex Sous-Chef Recipe Context Loss via Auto-Compaction #3947 defenses (raw-read prohibition, recipe_read_guard.py, compaction disable) eliminate self-recovery, converting the state into a trap.

Reproduction (repo HEAD 18f410893, 0.10.748)

load_and_validate("implementation", project_dir=Path("/home/talon/projects/token-reserve"),
                  backend_name="codex", lister=DefaultSkillResolver(),
                  ingredient_overrides={"backend_supports_git_write": "false",
                                        "post_run_diagnostics": "false"})
# → valid: False, content_len: 0, 13 error-severity backend-incompatible-skill suggestions,
#   exactly one dangling-route error (instrumented): enrich_diff_context → resolve_review
# same call with backend_name="claude-code" → valid: True, content_len: 86,942

Claude Code is unaffected because git_metadata_writable=True produces no pruning of these steps — full content is delivered. This is not Codex-side truncation: the run's rollout log records the full open_kitchen results (90,635 and 86,672 bytes) with zero truncation markers; tool_output_token_limit=50000 and compaction-disable were correctly written to config.toml.

Remediation targets

1. Surface the wipe reason: add errors: list[str] to LoadRecipeResult and populate it whenever raw is cleared (_api.py:415-426).
2. Fail loudly in open_kitchen: when valid is False and content == "", return a failure envelope (success: false + the dangling-route errors) — never success: true with no recipe.
3. Orchestrator halt instruction: sous-chef/orchestrator-prompt rule — if open_kitchen returns empty content, halt and escalate; never execute from the diagram.
4. Fix the recipe: give resolve_review a computable redirect (when=None catch-all) or guard enrich_diff_context's routes consistently so the pruned subgraph is removed atomically; audit the other backend_supports_git_write-guarded steps — 11 in implementation.yaml at HEAD, added across 207deda91, 884e1f2ae, d88bda83a — and the other bundled recipes (remediation.yaml gained 7 guards in the same commit) for the same exposure.
5. Close the test escape hatch: replace the continue at tests/recipe/test_hidden_ingredients.py:966-975 with an assertion (and ideally a semantic validation rule) that every prunable step has a computable redirect; add a codex-composition test asserting valid is True AND non-empty content AND all surviving steps present for each bundled recipe.
6. Secondary: bundled skills are invisible to Codex sessions (no --plugin-dir equivalent; src/autoskillit/workspace/session_skills.py:319 excludes BUNDLED), so the orchestrator could not map a post-hoc "run analyze health" request to /autoskillit:analyze-pipeline-health; consider listing invocable bundled skill commands in the Codex orchestrator prompt or open_kitchen result.

Closes #3992

Implementation Plan

Plan file: /home/talon/projects/autoskillit-runs/remediation-20260610-175022-808728/.autoskillit/temp/rectify/rectify_silent_recipe_content_wipe_part_a_2026-06-10_180512.md

🤖 Generated with Claude Code via AutoSkillit

Token Usage Summary

Step	Model	count	uncached	output	cache_read	peak_ctx	turns	cache_write	time
rectify*	opus[1m]	1	4.9k	23.2k	1.3M	97.0k	43	82.9k	21m 11s
review_approach*	opus[1m]	1	4.5k	6.1k	260.0k	49.0k	14	35.4k	5m 23s
dry_walkthrough*	opus	1	41	9.6k	867.8k	89.3k	35	67.7k	7m 33s
implement*	MiniMax-M3	1	3.7M	9.0k	0	0	79	0	7m 44s
audit_impl*	opus[1m]	1	1.2k	11.3k	257.3k	48.2k	18	39.9k	4m 33s
prepare_pr*	MiniMax-M3	1	284.5k	2.9k	0	0	12	0	1m 7s
compose_pr*	MiniMax-M3	1	351.0k	1.3k	0	0	12	0	51s
review_pr*	opus[1m]	1	43	26.5k	980.9k	89.0k	36	68.0k	6m 17s
Total			4.4M	89.8k	3.6M	97.0k		294.0k	54m 42s

* Step used a non-Anthropic provider; caching behavior may differ.

Token Efficiency

Step	LoC Changed	cache_read/LoC	cache_write/LoC	output/LoC
rectify	0	—	—	—
review_approach	0	—	—	—
dry_walkthrough	0	—	—	—
implement	243	0.0	0.0	37.0
audit_impl	0	—	—	—
prepare_pr	0	—	—	—
compose_pr	0	—	—	—
review_pr	0	—	—	—
Total	243	14903.1	1210.0	369.7

Model Usage Breakdown

Model	steps	uncached	output	cache_read	cache_write	time
opus[1m]	4	10.7k	67.0k	2.8M	226.3k	37m 26s
opus	1	41	9.6k	867.8k	67.7k	7m 33s
MiniMax-M3	3	4.4M	13.2k	0	0	9m 42s