Skip to content

load_recipe re-serve corrupts routing in live kitchen: clone.on_success rewired to register_clone_failure on 2nd+ serving (0.10.853) #4208

Description

@Trecek

Problem

Within a live kitchen session on 0.10.853, load_recipe re-serves of the remediation recipe carry corrupted routing: clone.on_success is rewired from claim_and_resolve to register_clone_failure. The first serving (via open_kitchen) is correct; the second and third servings (via load_recipe) are corrupted. An orchestrator that re-loads the recipe then dutifully routes a successful clone to the failure path and the dispatch dies.

Evidence (dispatch e6a45f7f, session d8d50e44, 2026-07-07 ~20:52)

Impact

Any pipeline whose orchestrator calls load_recipe after open_kitchen (a legitimate, common pattern) can fail immediately after clone — blocks the remediation queue and any recipe run on 0.10.853. Non-deterministic at fleet level because not all orchestrators re-load.

Expected behavior

Every serving of a recipe within a session is identical and correct: recall must serve an immutable snapshot; any post-serve mutation must operate on a copy. A serve-idempotence test (serve N times in one session, diff routing) should gate this permanently.

Prior Remediation Attempt (round 1) + descoping decision

Round 1 (dispatch d01edea9, 3h25m): Part A is complete on pushed branch impl-4208-20260707-230557session_serve_overrides snapshot closing the serve-context asymmetry (1cc88f305), arch-guard fixes (e31bdd5a3), and 5 serve-idempotence tests (529016f1f, on the branch history). Part B (skip-guard-requires-on-success rule) repeatedly context-exhausted with ~93 recipe-layer failures and is descoped to #4209.

Guidance for round 2 — SHIP PART A ONLY: fetch branch impl-4208-20260707-230557; take the Part A commits (529016f1f, 1cc88f305, e31bdd5a3) and NONE of the Part B commits (40aeaade9, f178603c0 — those belong to #4209); verify the serve-idempotence tests pass and the full suite is green at that scope; then PR/review/merge closing this issue. Do not implement any new rules.

Round 2 tail status + remaining defect (for round 3)

PR #4210 exists and carries: Part A (serve-overrides snapshot + serve-idempotence tests), 7 review-resolution fixes (review approved, id 4654241982), and an attempted test-isolation fix (6eb5cd0ff: reset_session_serve_overrides helper + fixture wiring) that proved INSUFFICIENT.

Remaining defect blocking merge: tests/server/test_tools_agents.py::test_unlock_agent_pack_session_scoped fails order-dependently in CI ("assert not True") — 3 occurrences (merge-group run 28944804388, branch runs 28938034846/28952937562); passes in isolation and in some full runs. Diagnosis so far: when _state._ctx is left non-None by an earlier test on the same xdist worker (with ctx.backend wired / kitchen gate state), the victim's bare Client(mcp) lifespan boot re-enables tags via the pre-reveal path — clearing only session_serve_overrides/session_serve_defer_unresolved (the 6eb5cd0 fix) does not prevent this. The real leak is broader lifecycle state (_state._ctx and/or gate/kitchen-open state) surviving between tests.

Round 3 scope: adopt PR #4210's branch (do NOT open a new PR — push to it); root-cause the ordering leak properly (find the polluter class; decide whether the fixture must fully reset _state._ctx/gate state, or the victim test needs hermetic setup, or the boot pre-reveal path needs a guard); make the test deterministically safe under xdist; prove with repeated targeted orderings via PYTEST_TEST_PATHS=... task test-check plus full task test-check; then the PR merges via the queue (review already approved).

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugExisting behavior is brokenrecipe:remediationRoute: investigate/decompose before implementationstagedImplementation staged and waiting for promotion to main

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions