Ingredient authority feedback contract: silent-clobber vs hard-reject asymmetry + non-actionable lock_ingredients rejection stalls Codex pipelines

[Trecek]

Summary

On 2026-07-02 a Codex orchestrator running the implementation recipe for issue #3987 was asked mid-run to enable the analyze-pipeline-health step. Following the served sous-chef guidance ("user asks to enable a step → lock_ingredients"), it re-opened the kitchen with overrides={post_run_diagnostics:"true"} — silently clobbered by the server with a clean success and zero warning — then called lock_ingredients({post_run_diagnostics:"true"}), which was hard-rejected: "Cannot lock server-authoritative ingredients: ['post_run_diagnostics']. These are set by the server and cannot be overridden." The bare {success:false, error} envelope carried no severity, remediation, or "your pipeline is unaffected" signal. Codex echoed the raw JSON as its final answer and ended its turn — zero further tool calls — with the pipeline otherwise healthy (context at 64.9%, the next step implement entirely independent of the failed lock), leaving #3987 claimed in-progress and an orphaned remote branch. A deep-mode investigation (18 subagents across 2 batches, adversarial challenge round, 3 blast-radius analyses, 3 post-report validators; full report appended below) established a three-layer orchestrator-facing contract defect. Every individual server component behaved exactly as designed; this issue is the remediation package for the contract seams between them.

Key established facts (all personally re-verified at cited lines):

• post_run_diagnostics is hidden: true + authority: config (src/autoskillit/recipes/implementation.yaml:100-104), resolved from config diagnostics.post_run_analysis (default false; config/ingredient_defaults.py:155), and a member of SERVER_AUTHORITATIVE_INGREDIENTS (ingredient_defaults.py:123-126, derived from CONFIG_AUTHORITY_KEYS, core/types/_type_constants.py:290-301).
• Silent-clobber side: open_kitchen merges {**_session_overrides, **(overrides or {}), **_config_layer} (tools_kitchen.py:680) — the config layer wins unconditionally — and _check_override_keys (tools_kitchen.py:508-522) explicitly subtracts SERVER_AUTHORITATIVE_INGREDIENTS before computing warnings, so the caller gets neither application, nor warning, nor rejection. The subtraction was introduced by 85e4e255f (Rectify: Ingredient Naming Asymmetry Causes Orchestrator Confusion #3500) with no documented rationale.
• Second silent entry point: load_recipe (tools_recipe.py:238) performs the byte-identical merge with NO _check_override_keys call at all — and it is the ADR-0004 sanctioned post-compaction re-delivery channel, exactly the moment a re-sent override is most likely. get_recipe is NOT a surface (accepts no caller overrides, tools_kitchen.py:444-468).
• Hard-reject side: lock_ingredients rejects the same keys (tools_kitchen.py:1192-1204) with a bare envelope that violates the repo's own conventions — no user_visible_message, stage, or retriable (cf. ToolFailureEnvelope in server/tools/_types.py:203-221; _kitchen_failure_envelope at tools_kitchen.py:80 and _dispatch_infeasible_response at :133-160 both carry user_visible_message; even the ingredient-lock guard's deny message includes remediation, hooks/guards/ingredient_lock_guard.py:63-65).
• Guidance routing: sous-chef SKILL.md (INGREDIENT LOCKING, skills/sous-chef/SKILL.md:843-877) instructs "user says 'turn on X' → lock_ingredients" with no server-authoritative carve-out, no mention that lock_ingredients can fail, and no configuration-tool failure handling anywhere in the file. The open_kitchen docstring (tools_kitchen.py:546) actively teaches "Use to activate hidden features" for overrides. The served recipe YAML exposes hidden ingredient declarations verbatim including authority: config — and no served text anywhere defines what that field means.
• The override path has never worked: 9 open_kitchen calls with post_run_diagnostics:"true" exist across Jun 5–Jul 2 sessions; all were silent no-ops. The recipe's diagnostics gate (six terminal steps, implementation.yaml:1431/2325/2381/2401/2430/2459) has never fired automatically in any traceable run. The June 8 "precedent" health report was a manual run_skill /autoskillit:analyze-pipeline-health dispatch after the user ordered it directly (440.9s, real report produced), and that session's agent stated on the record that re-opening does not mutate the running pipeline.
• First occurrence, novel seam: the 2026-07-02 rejection is the only time this error has appeared as a real tool response in any log corpus (all other matches are source-code reads). Prior fixes (d70a4b38b/Rectify: Defense-in-Depth — Server-Authoritative Ingredient Override Registry #3185, e9dd32b4e/[FIX] Add Missing Config-Authority Test Coverage #3190, 795f93d41/[FEATURE] Session-scoped ingredient locking with structural enforcement #3381, 85e4e255f/Rectify: Ingredient Naming Asymmetry Causes Orchestrator Confusion #3500, 7bf89aff0/Rectify: Ingredient Authority Enforcement Gap — Config-Authority Keys Without Enforcement Backing #3973) all harden the enforcement direction (server value must win); none asked what the orchestrator is told when enforcement fires, or what it should do next. No existing issue tracks this asymmetry.
• Halt context: context exhaustion ruled out (167,778 of 258,400 tokens = 64.9%; 6% quota). The stop matches documented Codex/gpt-5.x premature-turn-termination dispositions (openai/codex #10828, #14242, #28704), aggravated by success:false living in a normal MCP result body rather than the spec's isError channel — the model receives a structurally successful call and must infer severity from prose (MCP spec 2025-06-18; Google AIP-129 explicitly warns silent-drop of client-set values causes declarative clients to loop or abort).

Remediation Scope

R1 — Actionable rejection envelope in lock_ingredients (blast radius: LOW)

Replace the bare dict at tools_kitchen.py:1192-1204 with the repo-standard failure envelope carrying retriable: false, stage, and a user_visible_message that states, in order: (a) this failure does NOT affect the running pipeline — continue executing recipe steps; (b) how the ingredient is actually controlled — four keys are config-backed (base_branch → branching.default_base_branch, local_review_rounds → review.local_review_rounds, adversarial_review_level → plan.adversarial_review_level, post_run_diagnostics → diagnostics.post_run_analysis), while is_fleet_dispatch/dispatch_id are env-derived (DISPATCH_ID_ENV_VAR, ingredient_defaults.py:163-165) and fixed at session launch — for those two the message must say "set by the dispatch runtime; not user-configurable", not "edit config"; (c) after the user edits config, re-calling open_kitchen picks up the new value — verified: load_config has no caching (config/_config_loader.py:189-193); config is re-read on every open_kitchen call, so config-edit + re-open genuinely works mid-session; (d) for post_run_diagnostics specifically, the sanctioned manual alternative: run_skill /autoskillit:analyze-pipeline-health (the historically proven path).

Constraints and co-updates (verified):

• Keep the phrase "server-authoritative" in error — tests/server/test_lock_ingredients.py:101 asserts the substring.
• input_failure_envelope (_types.py:240-250) does NOT populate user_visible_message by default — pass it explicitly.
• tests/arch/test_subpackage_isolation.py:969 hardcodes tools_kitchen.py at 1321 lines — update in whichever commit last touches the file (R1 and R4 both add lines).
• No ingredient→config-path mapping exists anywhere in the codebase today (verified: the paths live only as inline attribute accesses in resolve_ingredient_defaults) — create a SERVER_AUTHORITATIVE_CONFIG_PATHS: dict[str, str] constant beside SERVER_AUTHORITATIVE_INGREDIENTS in config/ingredient_defaults.py (config-backed keys only) and drive both R1's message and R2's warning text from it.
• Add shape assertions (stage non-empty, retriable is False, user_visible_message present and actionable), parametrized over post_run_diagnostics AND base_branch (the existing test covers only base_branch).

R2 — Visible clobber warnings on BOTH silent entry points (blast radius: MEDIUM)

1. _check_override_keys (tools_kitchen.py:508-522): stop subtracting SERVER_AUTHORITATIVE_INGREDIENTS; emit an informative warning per masked key ("Override for server-authoritative ingredient 'X' ignored — server value 'Y' (from config ) wins; set the config key and re-call open_kitchen to change it"). The merge/clobber itself is unchanged (server still wins). Both call sites: deferred-recall :781-787 and normal :893-900. To render the winning value 'Y', _check_override_keys must gain access to the resolved server values — its current signature carries only (overrides, declared, session_keys) — e.g., add a config_layer: dict[str, str] parameter and update both existing call sites plus the new load_recipe site.
2. Formatter triangle (critical — without it the fix is invisible to its audience): warnings is absent from the OpenKitchenResult TypedDict (recipe/_recipe_ingredients.py:134-164) and from both _FMT_OPEN_KITCHEN_RENDERED/_FMT_OPEN_KITCHEN_SUPPRESSED (hooks/formatters/_fmt_recipe.py:163-193), and _fmt_recipe_body never reads it — the pretty-output hook would drop the new warnings for all formatted sessions (both backends). The fix must add warnings to the TypedDict, to _FMT_OPEN_KITCHEN_RENDERED, and implement rendering; tests/infra/test_pretty_output_hook_infra.py::test_coverage_registry_entries_are_valid (:126) enforces the triangle once the TypedDict changes.
3. load_recipe (tools_recipe.py:238): apply the same warning + rendering treatment (mirror set: LoadRecipeResult at _recipe_ingredients.py:110-131, _FMT_LOAD_RECIPE_RENDERED/SUPPRESSED at _fmt_recipe.py:24-50), relocate _check_override_keys (with its extended signature) into a shared server/tools/ helper module rather than importing across sibling tool modules — the established pattern (_preflight.py, _claim_helpers.py, _auto_overrides.py); update the tools AGENTS.md file table in the same commit.
4. backend_supports_git_write handling is unchanged (it is excluded via session keys, not the server-authoritative subtraction). Fleet sessions cannot call open_kitchen (orchestrator-only guard) — no spurious fleet warnings; no bundled skill passes server-authoritative overrides.
5. Tests: new warning-content assertions on both open_kitchen paths and on load_recipe; extend test_open_kitchen_config_authority_overrides_caller (tests/server/test_tools_kitchen_envelope.py:406) to assert the warning (it currently only checks the clobbered value downstream).

R3 — Sous-chef carve-out + scoped failure rule (blast radius: MEDIUM, LOW with the anchor test)

1. Add a server-authoritative carve-out to the INGREDIENT LOCKING section — placed before the ### When to call lock_ingredients heading (:850), not appended after the examples, or the model may still fire lock_ingredients before reaching the exception. List the six never-lockable/never-overridable keys with the response pattern "requires config change: " for the four config-backed keys (base_branch, local_review_rounds, adversarial_review_level, post_run_diagnostics) and "set by the dispatch runtime at session launch; not changeable" for is_fleet_dispatch/dispatch_id; for post_run_diagnostics add the manual alternative (run_skill /autoskillit:analyze-pipeline-health).
2. Add a failure-handling rule scoped to the server-authoritative rejection: such a rejection is advisory — report it to the user in one sentence and continue the pipeline. Do NOT write a blanket "never halt on configuration-tool failure": an infrastructure lock-write failure means a user instruction was silently unenforced, and the existing halt doctrine (SKILL.md :166, :775, :836-839) correctly applies there.
3. Anchoring test (required — prevents prose/frozenset drift): an exogenous coupling test asserting every SERVER_AUTHORITATIVE_INGREDIENTS member appears within the extracted INGREDIENT LOCKING section specifically — not anywhere in the 12k-token file, where e.g. dispatch_id already appears in unrelated sections and would false-pass. Parametrize over sorted(SERVER_AUTHORITATIVE_INGREDIENTS) (the existing pattern test_quota_post_warning_trigger_coupled_to_sous_chef_skill in tests/contracts/test_exogenous_string_coupling.py is single-constant; section-extraction helpers exist in tests/_helpers.py). Without it the list goes stale the first time the frozenset changes, and no existing test would notice.
4. Size impact negligible (~100 tokens on a ~12,300-token file, served on every open_kitchen); no existing sous-chef contract test asserts absence of content, so nothing breaks.

R4 — Docstring corrections (blast radius: LOW)

• open_kitchen overrides param (tools_kitchen.py:546): qualify "Use to activate hidden features" — ingredients declaring authority: config cannot be set via overrides; they resolve from server config and caller values are ignored.
• lock_ingredients docstring: state that server-authoritative ingredients are rejected (the error names them).
• Optional: one legend line where served content is assembled explaining authority: config.
• No test asserts docstring text (verified); only the shared line-count guard applies.

R5 — Cross-tool regression seams

• A consistency test pinning the post-fix contract: every SERVER_AUTHORITATIVE_INGREDIENTS key produces feedback on BOTH surfaces (warning in open_kitchen/load_recipe, actionable rejection in lock_ingredients) — today the two surfaces are tested only in isolation and nothing relates them. Natural home: a new test class in tests/server/test_lock_ingredients.py (it already exercises the kitchen tool surface).
• A lock-survival test across a deferred-recall re-open (gap: tests/server/test_kitchen_lifecycle.py:134-170 covers only close/open cycles; locks live in the overlay file, untouched by re-open, deleted only via _close_kitchen_handler :397-401). Natural home: tests/server/test_open_kitchen_deferred_recall.py (exists, dedicated to the deferred-recall path).

Explicitly out of scope (evaluated and killed)

• Demoting post_run_diagnostics from server authority / adding a runtime-enable tool — contradicts the explicit defense-in-depth intent (ingredient_defaults.py:118-122, introduced by d70a4b38b: "preventing LLM-supplied values from winning"); two sanctioned paths already exist (config-edit + re-open, which works mid-session; manual run_skill). Weakening the registry re-opens the drift class it was built to close.
• Stripping hidden ingredients from served content — removes the orchestrator's ability to explain constraints; users name ingredients aloud anyway (both incidents started from user phrasing), so discovery-prevention fails its own premise.
• Blanket "never halt on configuration-tool failure" rule — conflicts with the halt doctrine for infrastructure failures where a user instruction silently goes unenforced.
• Migrating server errors to MCP isError: true — spec-correct and would give clients a structural severity channel, but it is a server-wide contract change across ~40 tools and all backends; record as a candidate ADR question, not part of this remediation.
• A re-open guard ("kitchen already open" error) — multi-open is by design (fbfb3589e/[FIX] Double open_kitchen Context Cost — Pre-Reveal / Deferred-Recall Architectural Immunity #4092 deferred-recall path) and config-edit + re-open is the sanctioned refresh mechanism; a guard would break the legitimate recovery flow.

Context: incident recovery (already performed)

• Issue compose-pr: Add shell-level retry around gh pr create for transient API failures #3987: in-progress label removed and the orphaned branch compose-pr-add-shell-level-retry-around-gh-pr-create-for-tra/3987 deleted on 2026-07-02 (it sat at base SHA with zero commits). Note: no reaper would ever have released the label — the liveness/stale-claim infrastructure operates only on fleet DispatchRecords, and interactive claim_and_resolve_issue claims create none (server/tools/_claim_helpers.py:83-91).
• To actually enable the pipeline-health step today: set diagnostics.post_run_analysis: true in ~/.autoskillit/config.yaml (picked up at the next open_kitchen, including mid-session re-opens), or run /autoskillit:analyze-pipeline-health directly via run_skill.

Related

• Prior fixes hardening the enforcement direction (none touched the feedback seam): d70a4b38b (Rectify: Defense-in-Depth — Server-Authoritative Ingredient Override Registry #3185), e9dd32b4e ([FIX] Add Missing Config-Authority Test Coverage #3190), 795f93d41 ([FEATURE] Session-scoped ingredient locking with structural enforcement #3381), 85e4e255f (Rectify: Ingredient Naming Asymmetry Causes Orchestrator Confusion #3500), 7bf89aff0 (Rectify: Ingredient Authority Enforcement Gap — Config-Authority Keys Without Enforcement Backing #3973).
• Adjacent open issues (different gaps, not duplicates): Session-scoped ingredient locking: configure → overlay → hook enforcement for recipe step control #3357, Ingredient naming asymmetry causes orchestrator confusion (audit vs audit_impl) #3462, Ingredient lock bypass: step_name='' defeats both enforcement layers #3452, Hidden ingredient overrides invisible to LLM — skip_when_false evaluates against YAML default instead of resolved config value #3065, Move canonical CONFIG_AUTHORITY_KEYS to core/types/_type_constants.py and add cross-layer consistency test #3970, Fleet dispatch does not enforce backend_supports_git_write authority contract #3971, Double open_kitchen context cost: recall re-sends full recipe; Codex amplifiers #4089.
• Class adjacency: Codex dispatch admission: complete missing-step diagnostics, seam tests, ADR #4167 (dispatch-admission refusal diagnostics) — this is the second consecutive orchestrator-facing refusal surface found lacking actionable guidance in a week; the split-plane admission campaign's disclosure/observability principles (complete enumeration, structured refusal records, actionable envelopes) apply to this seam as well.
• Coordination notes (in-flight work): (1) Codex dispatch admission: complete missing-step diagnostics, seam tests, ADR #4167's remediation is executing now and touches tools_recipe.py's load_recipe function (~15 lines from R2's insertion point at :238) — independent changes, but expect a rebase if both are in flight. (2) The planned split-plane admission campaign's S2(c) extends the same OpenKitchenResult TypedDict and _FMT_OPEN_KITCHEN_* frozensets (adding user_visible_message/missing_provider_steps/escape_hatch) that R2 extends with warnings; the formatter field-coverage test is an exclusive invariant, so whichever lands second rebases the frozensets. (3) That campaign's S5 cites "parity with the existing lock_ingredients denial" as its quality bar — after R1 lands, the reference target is the repaired envelope, not the pre-R1 bare dict.
• Full investigation report with evidence citations appended below.

Investigation

Prior investigation completed interactively (deep mode). Full report with evidence citations follows.

Investigation: Codex Pipeline Halt After lock_ingredients Server-Authoritative Rejection (post_run_diagnostics)

Date: 2026-07-02
Scope: Codex-backend implementation pipeline for issue #3987 halting after a mid-run user request to enable the analyze-pipeline-health step; kitchen re-open semantics; ingredient authority model; orchestrator-facing contracts; test gaps; full history of the pattern.
Mode: Deep Analysis (2 exploration batches — 11 agents, challenge round, 3 blast-radius/breakage agents, historical recurrence check, external research)

Summary

At 16:24:51 UTC the user told the running Codex orchestrator "there is an Analyze Pipeline health step that defaults to Turn Off, but you should run it at the end." Codex — which had already seen the hidden ingredient post_run_diagnostics in the recipe YAML served by open_kitchen — re-called open_kitchen with overrides={post_run_diagnostics:"true"} (silently clobbered by the server, no warning) and then, following the sous-chef guidance "user asks to enable a step → lock_ingredients", called lock_ingredients({post_run_diagnostics:"true"}). That call was hard-rejected: post_run_diagnostics is a server-authoritative ingredient settable only via config diagnostics.post_run_analysis. The rejection envelope was a bare {"success": false, "error": ...} with no severity, remediation, or "your pipeline is unaffected" signal. Codex reasoned ~15 seconds, echoed the raw error JSON as its final answer, and ended its turn — making zero further tool calls despite the next recipe step (implement) being entirely unaffected. Nothing was corrupted or lost: the worktree, plan, branch, and existing locks were all intact and the run was resumable; but the pipeline made no further progress, leaving issue #3987 claimed in-progress and an orphaned branch at base SHA. The failure is a three-layer contract defect (silent-clobber vs hard-reject asymmetry; served guidance that routes exactly this user request into the rejection; a non-actionable error envelope handed to a model with a documented disposition to end turns after failures), not a malfunction of any single component — every server component behaved as designed.

Root Cause

Layer 1 — Contract asymmetry between open_kitchen and lock_ingredients for the same keys. open_kitchen merges overrides as {**_session_overrides, **(overrides or {}), **_config_layer} (src/autoskillit/server/tools/tools_kitchen.py:680); _config_layer is build_config_authoritative_layer(...) — the resolved config values for SERVER_AUTHORITATIVE_INGREDIENTS — and, being rightmost, unconditionally clobbers caller values. The caller receives no signal: _check_override_keys (tools_kitchen.py:508-522) explicitly subtracts SERVER_AUTHORITATIVE_INGREDIENTS before computing unknown-key warnings, so a server-authoritative override is neither applied, nor warned about, nor rejected. The sibling tool lock_ingredients hard-rejects the identical keys (tools_kitchen.py:1192-1204) with a terse envelope that violates the repo's own failure-envelope conventions (no user_visible_message, stage, or retriable — cf. ToolFailureEnvelope in server/tools/_types.py, _kitchen_failure_envelope, _dispatch_infeasible_response). [SUPPORTED — all lines personally verified]

Layer 2 — Served guidance routes exactly this user request into the rejection. The sous-chef SKILL.md (appended verbatim to every open_kitchen response, tools_kitchen.py:939-945) instructs: *"User says 'turn on review_approach' → lock_ingredients(locked={"review_approach": "true"}...)"* and *"Use lock_ingredients for user-requested skip/enable instructions"* (src/autoskillit/skills/sous-chef/SKILL.md, INGREDIENT LOCKING section, ~lines 845-877 — quoted text personally verified). It contains zero mention of server-authoritative ingredients, zero mention that lock_ingredientscan returnsuccess:false, and zero guidance for handling configuration-tool failures. The open_kitchendocstring compounds it: theoverridesparameter is documented as *"Use to activate hidden features"* — actively teaching the silently-clobbered path. The full recipe YAML served incontentexposes hidden ingredients verbatim, includingpost_run_diagnosticswithhidden: true/authority: config (src/autoskillit/recipes/implementation.yaml:100-104), but no served text anywhere explains what authority: config` means. Codex's behavior was a faithful reading of everything it was shown. [SUPPORTED]

Layer 3 — Orchestrator fragility on ambiguous failure. The error reached Codex untransformed (verified in the rollout: mcp_tool_call_end content identical to the server's JSON) as a protocol-level success whose body says success:false — MCP's isError channel was not used. With no severity or remediation signal, gpt-5.5 treated it as terminal: ~15s of reasoning, then the raw error echoed as final_answer, then task_complete. Context exhaustion is ruled out (167,778 of 258,400 context-window tokens = 64.9% used; 6% quota). This matches documented Codex/gpt-5.x premature-turn-termination patterns (openai/codex #10828, #14242, #28704). [SUPPORTED for the event sequence; the causal attribution to model disposition vs. deliberate "pause and report" is discussed under Severity framing]

Severity framing (challenge-round correction): task_complete ends a turn, not the session. The interactive session remained open; a user reply of "continue" would have resumed the pipeline. Nothing was destroyed. The defect is therefore best stated as: Codex declined to continue an unaffected pipeline and reported a raw JSON error instead of either continuing or explaining the constraint — a stall with orphaned external state (claimed issue, published branch), not a destructive collapse. Whether pausing to report was defensible is arguable for an interactive session; surfacing unexplained raw JSON and taking no recovery action is not.

Direct answers to the triggering questions:

1. Why did it "collapse"? Because the one tool call that failed gave the model no way to know the failure was ignorable, and nothing in its instructions covered configuration-tool failures. The pipeline itself was healthy; the halt was an orchestrator decision made on an information vacuum the server created.
2. Why did it re-open the kitchen? Three served signals point there: the docstring says overrides activate hidden features; overrides can only be passed to open_kitchen; and the sous-chef text says to lock "immediately after open_kitchen". Re-opening is also architecturally sanctioned — the deferred-recall path (tools_kitchen.py:577,632-637) exists precisely to serve repeat calls cheaply, and re-open-after-config-edit is the only mechanism to pick up a config change mid-session (config is re-read on every open_kitchen call: resolve_ingredient_defaults at tools_kitchen.py:657, ingredient_defaults.py:155). The re-open was useless here (the override was clobbered) but harmless: handler skipped, locks preserved, step set recomputed to an identical result.
3. "It only needs to succeed once" — multi-open is by design (no re-open guard has ever existed; fbfb3589e PR [FIX] Double open_kitchen Context Cost — Pre-Reveal / Deferred-Recall Architectural Immunity #4092 made repeat calls cheap rather than forbidden). The defect is not the re-open; it is that the override path silently no-ops and the lock path hard-rejects, with no surface telling the orchestrator the truth.

Affected Components

• src/autoskillit/server/tools/tools_kitchen.py:680 — override merge; config layer clobbers caller values silently [SUPPORTED]
• src/autoskillit/server/tools/tools_kitchen.py:508-522 — _check_override_keys masks server-authoritative keys from warnings (both call sites: deferred-recall ~:781-787, normal ~:893-900) [SUPPORTED]
• src/autoskillit/server/tools/tools_kitchen.py:1192-1204 — lock_ingredients hard rejection, bare envelope [SUPPORTED]
• src/autoskillit/config/ingredient_defaults.py:118-126, 155, 161, 208-210 — SERVER_AUTHORITATIVE_INGREDIENTS (= CONFIG_AUTHORITY_KEYS − {source_dir, backend_supports_git_write}), resolution from cfg.diagnostics.post_run_analysis, exception fallback "false", build_config_authoritative_layer [SUPPORTED]
• src/autoskillit/core/types/_type_constants.py:290-301 — CONFIG_AUTHORITY_KEYS (8 keys incl. post_run_diagnostics) [SUPPORTED]
• src/autoskillit/recipes/implementation.yaml:100-104 — ingredient declaration (hidden: true, authority: config, default 'false'); six terminal diagnostic steps gated by skip_when_false: inputs.post_run_diagnostics (:1431, :2325, :2381, :2401, :2430, :2459 — one per pipeline exit path, all optional: true, all invoking /autoskillit:analyze-pipeline-health) [SUPPORTED]
• src/autoskillit/skills/sous-chef/SKILL.md ~:845-877 — INGREDIENT LOCKING guidance with no server-authoritative carve-out and no failure handling [SUPPORTED — verified verbatim]
• open_kitchen MCP docstring (overrides param) — "Use to activate hidden features" [SUPPORTED — verified against the live MCP contract]
• src/autoskillit/hooks/formatters/_fmt_recipe.py:163-193 — warnings key absent from both rendered and suppressed field lists for open_kitchen (any future warning would be dropped by the formatter) [SUPPORTED]
• src/autoskillit/server/tools/tools_recipe.py:238 — load_recipe performs the byte-identical silent-clobber merge with NO _check_override_keys call at all (zero authority-feedback references in the file); it is the ADR-0004 post-compaction re-delivery channel and fleet-dispatch-tagged, so the same defect exists on a second entry point [SUPPORTED — personally verified]
• src/autoskillit/config/_config_dataclasses.py:301-302, src/autoskillit/config/defaults.yaml:119-120 — DiagnosticsConfig.post_run_analysis: bool = False [SUPPORTED]
• ~/.autoskillit/config.yaml — diagnostics.post_run_analysis: false at incident time (project config has no diagnostics block) [SUPPORTED]

Data Flow

Override lifecycle for open_kitchen(name="implementation", overrides={post_run_diagnostics:"true"}):

1. resolve_ingredient_defaults(project_dir) (tools_kitchen.py:657) re-reads config (per call, not cached) → {"post_run_diagnostics": "false", ...} since cfg.diagnostics.post_run_analysis is false.
2. _config_layer = build_config_authoritative_layer(_defaults) → contains post_run_diagnostics: "false".
3. _merged_overrides = {**_session_overrides, **(overrides or {}), **_config_layer} (:680) → caller's "true" overwritten.
4. load_and_validate(..., ingredient_overrides=_merged_overrides) prunes all six diagnostic steps (skip_when_false false) and interpolates hidden refs with "false".
5. _check_override_keys (:781-787 deferred path) subtracts server-authoritative keys → no warning. Response is a clean success; served content byte-identical to the previous open (verified in the rollout: call-3 output identical to call-2).
6. lock_ingredients({post_run_diagnostics:"true"}, pipeline_id="3987") → set(locked) & SERVER_AUTHORITATIVE_INGREDIENTS non-empty (:1192) → {"success": False, "error": "Cannot lock server-authoritative ingredients: ['post_run_diagnostics']. These are set by the server and cannot be overridden."}. No state mutated; prior locks (review_approach, audit_impl) intact in .autoskillit/temp/.hook_config_overlay.json.
7. Error passes unmodified through the MCP layer (lock_ingredients is in pretty_output_hook._UNFORMATTED_TOOLS; no suppression) → Codex → final answer → task_complete at 16:25:40.940Z. No further tool calls.

Kitchen re-open mechanics: call 3 sets _is_deferred_recall=True (name matches recipe_name, gate enabled — :632-637); _skip_handler = gate_infrastructure_ready (:577) skips all infrastructure; active_recipe_steps is recomputed (:731) but to an identical set; kitchen_id, gate, quota task, pipeline tracker, and the lock overlay are untouched. The overlay file is deleted only via _close_kitchen_handler() (:397-401), invoked by close_kitchen (:994).

Timeline (from ~/.codex/sessions/2026/07/02/rollout-2026-07-02T08-58-55-019f238d-fb2b-7543-8bd6-d2ad4d04f8fe.jsonl, UTC)

Time	Event
15:59:09	open_kitchen #1 (bare, discovery)
15:59:34	User: implement issue #3987 with review approach + audit
15:59:51	open_kitchen #2 (task/issue_url/review_approach/audit_impl)
15:59:56	lock_ingredients({review_approach, audit_impl}, pipeline_id="3987") → success
16:00:05–16:00:48	bootstrap_clone → claim_and_resolve (#3987 → in-progress) → create_and_publish_branch → run_skill make-plan
16:24:51	make-plan returns success (25m37s); user injection about the health step arrives same moment
16:25:13	open_kitchen #3 (+post_run_diagnostics:"true") → clean success, override silently clobbered
16:25:22	lock_ingredients({post_run_diagnostics:"true"}) → rejection
16:25:37–40	~15s reasoning → raw error JSON as final answer → task_complete (turn ends; 64.9% context used)

Orphaned state at halt: issue #3987 OPEN + in-progress; remote branch compose-pr-add-shell-level-retry-around-gh-pr-create-for-tra/3987 at base SHA 49dafcf67 with zero commits; clean worktree at /home/talon/projects/autoskillit-runs/impl-20260702-090006-277778 with the completed plan at .autoskillit/temp/make-plan/compose_pr_gh_pr_create_retry_plan_2026-07-02_091102.md.

Test Gap Analysis

Why no test caught this:

1. The silent-clobber side is untested as a caller-visible contract. test_open_kitchen_config_authority_overrides_caller (tests/server/test_tools_kitchen_envelope.py:406) asserts the downstream value wins but never asserts what the caller is told (no warnings assertion). No test passes a server-authoritative key and checks the response for any signal.
2. The rejection side is tested only for firing, not for shape. test_lock_ingredients_rejects_server_authoritative (tests/server/test_lock_ingredients.py:84-101) asserts success is False and the substring "server-authoritative" — using base_branch, not post_run_diagnostics; nothing asserts retriable/stage/user_visible_message or conformance to ToolFailureEnvelope.
3. No cross-tool consistency invariant. Nothing asserts any relationship between the keys open_kitchen silently masks (:515) and the keys lock_ingredients hard-rejects (:1192) — both derive from the same frozenset but are tested in isolation.
4. No lock-state-across-reopen test. test_back_to_back_open_close_open_resets_infrastructure (tests/server/test_kitchen_lifecycle.py:134-170) covers close/open cycles, not deferred-recall re-opens with locks in place.
5. The guidance layer is untestable as written. Sous-chef contract tests (tests/contracts/test_sous_chef_*.py) assert presence of specific passages; no test could flag a missing carve-out that was never written.

Similar Patterns

• Same trap shape, five more ingredients. Every SERVER_AUTHORITATIVE_INGREDIENTS member is a plausible user request an orchestrator would try to satisfy the same way: base_branch ("merge to feature/x"), local_review_rounds ("do 2 local reviews"), adversarial_review_level ("use strict review"), plus is_fleet_dispatch/dispatch_id (lower plausibility). All would follow the identical silent-clobber → hard-reject → bare-error path.
• Contrast within the same file: _dispatch_infeasible_response (tools_kitchen.py:133-160) and _kitchen_failure_envelope (:80) both carry user_visible_message; the ingredient-lock guard's deny message even includes remediation ("Call lock_ingredients(unlock=[...]) to release." — hooks/guards/ingredient_lock_guard.py:63-65). The server-authoritative rejection is the outlier.
• backend_supports_git_write is deliberately excluded from the server-authoritative set (capability flag, lockable) — evidence the set's membership is a considered design surface, not an accident.

Design Intent Findings

• lock_ingredients + server-authoritative rejection: introduced together in 795f93d41 (2026-05-30, PR [FEATURE] Session-scoped ingredient locking with structural enforcement #3381) to stop orchestrator drift on user skip/enable instructions (prior incidents Orchestrator step-skipping recurrence: structural enforcement + context-anxiety immunity #1083, Fleet tag visibility leak: fleet/fleet-dispatch tags never disabled at startup #3337, Hidden ingredient overrides invisible to LLM — skip_when_false evaluates against YAML default instead of resolved config value #3065, investigate: 'auto' ingredient is dead code — investigation_complete marker never triggers skip #3238). The rejection was pre-emptive registry hygiene, not a response to an observed lock attempt. Callers/consumers: run_skill enforcement (tools_execution.py:157-201, 604, 631), ingredient_lock_guard.py PreToolUse hook (applies to Codex "works-as-is"), overlay file lifecycle (close_kitchen deletes).
• SERVER_AUTHORITATIVE_INGREDIENTS registry: d70a4b38b (2026-05-27, PR Rectify: Defense-in-Depth — Server-Authoritative Ingredient Override Registry #3185) — "third instance of the same pattern: a server-knowable value left to LLM judgment"; explicit intent comment survives at ingredient_defaults.py:118-122 ("preventing LLM-supplied values from winning"). Canonical set moved to IL-0 in 11e41d792 (PR Implementation Plan: Move Canonical CONFIG_AUTHORITY_KEYS to core/types/_type_constants.py #3972).
• The silent-mask in _check_override_keys: introduced in 85e4e255f (2026-05-31, PR Rectify: Ingredient Naming Asymmetry Causes Orchestrator Confusion #3500, ingredient naming asymmetry). The subtraction is deliberate (avoids false "unknown key" warnings for accepted-then-clobbered keys) but carries no comment or commit-message justification — the asymmetry with the one-day-older hard-reject was never annotated anywhere.
• post_run_diagnostics / diagnostics.post_run_analysis: 0440da032 (2026-05-25, PR Implement post-run pipeline health analysis with developer config toggle #3006) — "post-run pipeline health analysis with developer config toggle". Never user-settable in its entire history; protected by the registry two days after introduction. The gated step runs /autoskillit:analyze-pipeline-health (Haiku scanner subagents + adversarial validation; skills_extended/analyze-pipeline-health/SKILL.md, agents/pipeline-health-scanner.md).
• Multi-open of open_kitchen: no re-open guard has ever existed (git log -S finds none). fbfb3589e (2026-06-12, PR [FIX] Double open_kitchen Context Cost — Pre-Reveal / Deferred-Recall Architectural Immunity #4092) added gate_infrastructure_ready to make repeat calls cheap (deferred recall), confirming multi-open as the intended contract. docs/decisions/0004-recipe-redelivery.md sanctions load_recipe as the re-delivery channel post-compaction; kitchen lifecycle re-open is otherwise undocumented.

Historical Context

This exact failure is a first occurrence — the 2026-07-02 session is the only time "Cannot lock server-authoritative ingredients" has ever appeared as a real tool response (all other grep hits in session logs are source code being read). No prior investigation covers this root cause (log-dir scan found none; the 2026-07-01 investigation of dispatch_infeasible/gate_backend_write is an adjacent-but-distinct admission-control seam).

But the silent no-op half has a long, unnoticed history. Nine open_kitchen calls with post_run_diagnostics:"true" in overrides exist across Jun 5–Jul 2 (sessions 019e9817, 019e9b43, 019e9ea6, 019ea7cc, 019eaccc, 019eb4c2, 019eb90b, 019ebc31, 019f238d). None ever enabled the step. The June 8 session (019ea7cc) is the smoking gun: its agent eventually stated on the record, "Re-opening the kitchen doesn't mutate the already-running pipeline state. It just returns a freshly resolved recipe snapshot. I treated that as a way to reload the recipe with post_run_diagnostics=true, but…" (verified verbatim in the rollout), and the health analysis only ran because the user said "Okay, then run the diagnostic step" and the agent dispatched run_skill /autoskillit:analyze-pipeline-health manually (440.9s, real report produced). The recipe's post_run_diagnostics gate has never fired automatically in any traceable run; the 8 existing health reports (May 27–Jun 3) came from manual dispatches or predecessor skill names (May sessions unpreserved; a brief config-true window in May cannot be confirmed or excluded).

Prior fixes were adjacent, not preventive — verdict: novel seam, not a recurrence. d70a4b38b/#3185 (registry completeness), e9dd32b4e/#3190 (enforcement test coverage), 795f93d41/#3381 (drift-prevention locks + the hard-reject), 85e4e255f/#3500 (naming validation + the silent-mask), 7bf89aff0/#3973 (fleet-layer enforcement). All harden the enforcement direction (server value must win); none asked "when enforcement fires, what is the orchestrator told, and what does it do next?" No open or closed GitHub issue describes the asymmetry or the missing remediation guidance (searched: lock_ingredients, server-authoritative, post_run_diagnostics, ingredient authority, silently ignored override; nearest: #3357, #3462, #3452, #3970, #3971 — all different gaps).

Recurring-pattern note: while this seam is novel, it is the second consecutive week an orchestrator-facing refusal surface (dispatch admission on 2026-07-01, ingredient authority on 2026-07-02) produced an uninformative refusal that stalled or blocked a Codex run — the class "refusal surfaces lack actionable guidance" is recurring even though the mechanisms differ. Consider /rectify for the class after the immediate remediation.

External Research

• MCP error channels: the spec (2025-06-18) defines protocol errors and isError: true tool-execution errors; a success:false JSON body in a normal result is neither — the model receives a structurally successful call and must infer severity from prose. https://modelcontextprotocol.io/specification/2025-06-18/server/tools
• Codex/gpt-5.x turn-termination disposition: openai/codex #10828 (ends turn despite promising to continue), #14242 (abandons MCP server after one empty probe), #28704 (marks failed MCP tool permanently unavailable), #30523 (turns end without function_call); context exhaustion (#19842) ruled out here by the rollout's own token counts.
• Ignore-vs-reject for server-owned fields — no industry consensus, but a directly-on-point warning: Google AIP-203 OUTPUT_ONLY mandates clear + never error; Kubernetes uses layered semantics (silent-drop for status, 409 ownership conflict for SSA, 422 for immutable fields); Terraform hard-errors at plan time for Computed-only attributes. Google AIP-129 explicitly warns that silently discarding a client-set value causes declarative clients to loop or abort — the exact failure mode observed. https://google.aip.dev/203, https://google.aip.dev/129
• Actionable tool errors: Anthropic tool-use guidance: prompt-engineer error responses to communicate "specific and actionable improvements"; detailed parameter descriptions are the highest-leverage lever against invalid calls. https://www.anthropic.com/engineering/writing-tools-for-agents
• JSON Schema/OpenAPI readOnly: annotation-only ("ignored or rejected by the owning authority" — both allowed), so schema alone cannot carry the contract; description text and server feedback must.

Scope Boundary

Investigated: the full 2026-07-02 rollout and all related session logs (Jun 5–Jul 2); kitchen lifecycle/re-open internals; the complete ingredient authority model (four mechanisms) and its enforcement points; lock storage/enforcement; served guidance surfaces (sous-chef SKILL.md, docstrings, formatters, served YAML); design-intent git archaeology of all five involved mechanisms; test coverage across tests/server/, tests/arch/, tests/contracts/, tests/config/, tests/infra/; GitHub issue landscape; external specs and Codex behavior reports; blast radius of every recommended change.

Not yet explored: whether Claude Code-backend orchestrators exhibit the same halt behavior on this rejection (grep of both log corpora confirms zero actual rejection envelopes — every match is source-code reads — so there is genuinely no occurrence to compare); the May 2026 config history (sessions unpreserved — whether post_run_analysis was ever true is unknowable from surviving evidence); a server-wide audit of all tools' failure envelopes for convention conformance (only kitchen-adjacent tools were checked); whether configure_order/configure_fleet should gain a diagnostics toggle (deliberately left as a design question for the ADR-style decision in the remediation).

Recommendations

Single recommendation: make the server-authoritative ingredient contract symmetric, visible, and actionable at every orchestrator-facing surface. Four coordinated components (one issue-sized remediation package), with co-update obligations from the breakage analysis:

R1 — Actionable rejection envelope in lock_ingredients (tools_kitchen.py:1192-1204). Replace the bare dict with the repo-standard failure envelope carrying retriable: false, stage, and a user_visible_message that states, in order: (a) this failure does NOT affect the running pipeline — continue executing recipe steps; (b) the ingredient is controlled by config key diagnostics.post_run_analysis (map each config-backed key to its config path; is_fleet_dispatch/dispatch_id are env-derived at session launch and not user-configurable); (c) after the user edits config, re-calling open_kitchen picks the new value up (config is re-read per call); (d) the immediate manual alternative for this specific case: run_skill /autoskillit:analyze-pipeline-health (the historically proven path). Constraints: keep the phrase "server-authoritative" in error (asserted by tests/server/test_lock_ingredients.py:101); note input_failure_envelope does not populate user_visible_message by default — pass it explicitly. Co-updates: tests/arch/test_subpackage_isolation.py:969 hardcodes tools_kitchen.py at 1321 lines; add shape assertions (stage, retriable is False, user_visible_message) parametrized over post_run_diagnostics and base_branch. Risk: LOW.

R2 — Visible clobber warning in open_kitchen (_check_override_keys, tools_kitchen.py:508-522, both call sites). Stop subtracting SERVER_AUTHORITATIVE_INGREDIENTS; instead emit an informative warning per masked key: "Override for server-authoritative ingredient 'X' ignored — server value 'Y' (from config ) wins; set the config key and re-call open_kitchen to change it." The merge/clobber itself is unchanged. Critical co-update: the warnings response key is currently invisible — it is absent from OpenKitchenResult (recipe/_recipe_ingredients.py:134-164) and from both formatter field lists (_fmt_recipe.py:163-193), and _fmt_recipe_body never reads it, so formatted sessions (including Codex) would never see the warning. The fix must add warnings to the TypedDict, to _FMT_OPEN_KITCHEN_RENDERED, and implement its rendering — tests/infra/test_pretty_output_hook_infra.py::test_coverage_registry_entries_are_valid (:126) enforces this triangle once the TypedDict changes. backend_supports_git_write handling is unchanged (excluded via session keys). Fifth surface (validator-found, personally verified): load_recipe (tools_recipe.py:238) performs the identical merge with no _check_override_keys call and no warnings field in LoadRecipeResult (_recipe_ingredients.py:110-131) or its formatter lists (_fmt_recipe.py:24-50) — R2 must extend the same warning + rendering treatment there (it is the documented post-compaction recovery path, exactly when a re-sent override is most likely). get_recipe is NOT a surface (accepts no caller overrides, tools_kitchen.py:444-468). Risk: MEDIUM (now five-file-plus coordination), no existing test asserts the current silence.

R3 — Sous-chef guidance carve-out + scoped failure rule (skills/sous-chef/SKILL.md INGREDIENT LOCKING section). Add: (a) the list of never-lockable/never-overridable ingredients with the response pattern "requires config change: " and the post_run_diagnostics → manual run_skill alternative; (b) a failure-handling rule scoped to the server-authoritative rejection: such a rejection is advisory — report it to the user in one sentence and continue the pipeline. Do NOT write a blanket "never halt on config-tool failure": an infrastructure lock-write failure means a user instruction was not honored, and the existing halt doctrine (SKILL.md :166, :775, :836-839) correctly applies there. Placement matters: the carve-out must appear before the ### When to call lock_ingredients heading (:850) — appended after the examples, the model may still fire lock_ingredients before reaching the exception. Co-update: add an exogenous coupling test (pattern: tests/contracts/test_exogenous_string_coupling.py::test_quota_post_warning_trigger_coupled_to_sous_chef_skill) asserting every SERVER_AUTHORITATIVE_INGREDIENTS member appears in the section — otherwise the prose drifts the first time the frozenset changes. Size impact negligible (~100 tokens on a ~12,300-token file). Risk: MEDIUM (drift without the anchor; rule-scoping matters).

R4 — Docstring corrections (zero test exposure). open_kitchen overrides param: qualify "activate hidden features" — ingredients with authority: config cannot be set via overrides; they resolve from server config. lock_ingredients docstring: state that server-authoritative ingredients are rejected and will be named in the error. Optionally add one legend line where served content is assembled explaining authority: config. Risk: LOW (only the tools_kitchen.py line-count guard, shared with R1).

Plus regression seams (from Test Gap Analysis): a cross-tool consistency test pinning the post-fix contract ("every server-authoritative key produces feedback on BOTH surfaces: warning in open_kitchen, actionable rejection in lock_ingredients"); a lock-survival-across-deferred-recall-reopen test.

Immediate user-side recovery (no code): issue #3987 needs its in-progress label released manually — no reaper will ever release it: the liveness/stale-claim infrastructure operates only on fleet DispatchRecords, and interactive claim_and_resolve_issue claims create none, so future claim attempts stay blocked until the label is removed (or the run resumed — the clean worktree at impl-20260702-090006-277778 and the finished plan are reusable); the remote branch at base SHA can be reused or deleted. To actually enable the health step: set diagnostics.post_run_analysis: true in ~/.autoskillit/config.yaml (picked up at the next open_kitchen), or have the orchestrator run /autoskillit:analyze-pipeline-health directly at the end of a run — mid-run, "edit config, then re-open the kitchen" genuinely works.

Killed alternatives:

• Demote post_run_diagnostics from server authority / add a runtime-enable tool — contradicts the explicit defense-in-depth intent (ingredient_defaults.py:118-122, d70a4b38b: prevent LLM-supplied values winning); two sanctioned paths (config+re-open; manual run_skill) already exist; weakening the registry re-opens the drift class it was built to close.
• Strip hidden ingredients from served content — removes the orchestrator's ability to explain constraints, breaks nothing but hides everything; users will still name ingredients aloud (both incidents started from user phrasing), so discovery-prevention fails its own premise.
• Blanket "never halt on configuration-tool failure" rule — conflicts with the halt doctrine for infrastructure failures where a user instruction silently goes unenforced (breakage analysis, C3 check 4).
• Switch server errors to MCP isError: true — would give clients a structural severity channel (spec-correct), but it is a server-wide contract change across ~40 tools and all backends, far exceeding this defect; record as a candidate ADR question, not part of this remediation.
• Re-open guard ("kitchen already open" error) — would break the sanctioned config-refresh mechanism and the deferred-recall design (PR [FIX] Double open_kitchen Context Cost — Pre-Reveal / Deferred-Recall Architectural Immunity #4092); multi-open is intended.

Breakage Analysis

• Recommendation: R1 (envelope change)
  • Breakage surface: tests/server/test_lock_ingredients.py:100-101 (survives if "server-authoritative" retained in error); tests/arch/test_subpackage_isolation.py:969 line-count guard (certain, trivial update); pretty_output_hook generic formatter renders extra keys harmlessly; no jq/TypedDict consumers of the response shape; _types.py import already present in tools_kitchen.py.
  • Prior reverts: none (git log --grep=revert -- tools_kitchen.py empty).
  • Downstream contract violations: none found.
  • Risk level: LOW
• Recommendation: R2 (silent-mask removal + warning)
  • Breakage surface: formatter triangle — OpenKitchenResult TypedDict (recipe/_recipe_ingredients.py:134), _FMT_OPEN_KITCHEN_RENDERED/SUPPRESSED (_fmt_recipe.py:163-193), _fmt_recipe_body rendering, enforced by tests/infra/test_pretty_output_hook_infra.py:126; test_open_kitchen_config_authority_overrides_caller needs extension (not currently failing). Extending to load_recipe adds the mirror set: tools_recipe.py:238, LoadRecipeResult, _FMT_LOAD_RECIPE_RENDERED/SUPPRESSED (_fmt_recipe.py:24-50), plus an import-vs-duplicate decision for _check_override_keys across the two tool modules. Fleet sessions cannot call open_kitchen (orchestrator-only guard), so no spurious fleet warnings; no skill instructs passing server-authoritative overrides.
  • Prior reverts: none touching _check_override_keys (sole commit: 85e4e255f).
  • Downstream contract violations: the warning is dropped silently unless the formatter work is included — the fix is incomplete without it.
  • Risk level: MEDIUM
• Recommendation: R3 (sous-chef text)
  • Breakage surface: no existing sous-chef contract test breaks (they assert presence, not absence); drift risk between prose list and the frozenset is the main hazard — no doc/code coupling test exists today for ingredient names.
  • Prior reverts: none; no size-trim history on the file.
  • Downstream contract violations: potential doctrine conflict if the failure rule is not scoped to server-authoritative rejections.
  • Risk level: MEDIUM (drift), mitigated to LOW with the anchoring test.
• Recommendation: R4 (docstrings)
  • Breakage surface: no test asserts docstring text; line-count guard shared with R1.
  • Prior reverts: none.
  • Downstream contract violations: none.
  • Risk level: LOW