Skip to content

Commit bf3632d

Browse files
Trecekclaude
andauthored
fix: thread effective_backend_map and admission tests for Codex (#4198)
## Summary Threads `effective_backend_map` through the dispatch engine so that per-step backend routing is evaluated correctly during Codex fleet runs, eliminating the spurious `FLEET_RECIPE_INVALID` exit caused by steps being validated against the flat codex backend instead of their actual routed backend. Also adds two real-path admission tests (REQ-021 and REQ-028) that exercise the full `_execute_fleet_run → execute_dispatch → _run_dispatch → load_and_validate` path and verify that the CLI and kitchen dispatch paths pass an identical effective backend map. <details> <summary>Individual Group Plans</summary> ### Group 1: Rectify: Fleet Run Codex Effective Backend Map `autoskillit fleet run --backend codex` exits 3 with `FLEET_RECIPE_INVALID` because the fleet dispatch engine's internal validation call — inside `_run_dispatch` in `fleet/_api.py` — has never accepted `effective_backend_map` as a parameter. Without the per-step routing map, the `backend-incompatible-skill` rule evaluates every `run_skill` step against the flat codex backend, so steps that would be auto-routed to Claude Code workers via the R0 capability route are rejected as incompatible before dispatch starts. The architectural root cause is broader than the CLI: the dispatch engine (`execute_dispatch` / `_run_dispatch`) forms a wall — any per-step backend map computed by an IL-3 caller has nowhere to go, so `_run_dispatch`'s inner `load_and_validate` always runs without it. `dispatch_food_truck` computes the map correctly in its preflight but silently drops it at the `execute_dispatch` boundary because the parameter does not exist. The CLI path lands in the same engine with no preflight at all. A parallel structural gap exists in `validate_from_path`: this function and its `RecipeRepository` protocol have never accepted `effective_backend_map`, making the `validate_recipe` MCP tool constitutionally incapable of per-step routing validation. Part A closes these threading gaps and delivers behavioral + structural tests. Part B (separate task) introduces a `BackendAdmissionParams` bundle in `core/types/` and replaces the opt-in hardcoded arch-test enumeration with an exhaustive scan. ### Group 2: Implementation Plan: Fleet Run Codex Admission Tests — REQ-021 and REQ-028 Two missing tests remain from the issue #4197 implementation: 1. **REQ-021 (Test 1b):** `test_fleet_run_cli_codex_fails_without_claude_binary` — the existing test `test_fleet_run_cli_codex_fails_closed_on_invalid_recipe` mocks `_execute_fleet_run` itself via `AsyncMock`, bypassing all real admission logic. The replacement exercises the real `_execute_fleet_run` → `execute_dispatch` → `_run_dispatch` → `load_and_validate` path with `shutil.which=None` to verify fail-closed behavior. 2. **REQ-028 (Test 1h):** `test_admission_agreement_cli_matches_kitchen_path` — the existing test calls `_compute_effective_backend_map` twice with identical arguments and verifies it is deterministic. The replacement is a spy-based end-to-end threading test verifying that the CLI path and the kitchen path pass an identical `effective_backend_map` to `execute_dispatch`, parametrized over `["codex", "claude-code"]`. Two registry gaps found by Registry Tracer must also be fixed: - `tests/cli/AGENTS.md` line 47: file description does not mention the new test 1b. - `tests/arch/test_feature_markers.py` `_FLEET_CROSS_DIR_FILES`: `test_fleet_run_admission.py` is absent from the frozenset despite carrying `pytest.mark.feature("fleet")`. All other REQ-001–REQ-020, REQ-022–REQ-027, REQ-029 are satisfied by prior rounds. </details> Closes #4197 ## Implementation Plan Plan files: - `.autoskillit/temp/rectify/rectify_fleet-run-codex-effective-backend-map_2026-07-06_222543_part_a.md` - `.autoskillit/temp/make-plan/fleet_run_codex_admission_tests_plan_2026-07-06_235701.md` 🤖 Generated with [Claude Code](https://claude.com/claude-code) via AutoSkillit <!-- autoskillit:pipeline-signature steps=prepare_pr,run_arch_lenses,compose_pr,annotate_pr_diff,review_pr --> ## Token Usage Summary | Step | Model | count | uncached | output | cache_read | peak_ctx | turns | cache_write | time | |------|-------|-------|----------|--------|------------|----------|-------|-------------|------| | investigate* | fable | 1 | 23.2k | 38.8k | 1.6M | 148.5k | 43 | 287.4k | 16m 56s | | rectify* | sonnet | 1 | 289 | 43.9k | 3.3M | 132.8k | 86 | 133.6k | 28m 34s | | dry_walkthrough* | sonnet | 2 | 3.7k | 27.7k | 1.2M | 70.9k | 60 | 98.0k | 12m 44s | | implement* | sonnet | 2 | 1.4k | 70.3k | 9.2M | 175.6k | 231 | 227.0k | 25m 37s | | retry_worktree* | sonnet | 1 | 244 | 19.3k | 2.7M | 128.7k | 77 | 111.3k | 12m 34s | | audit_impl* | sonnet | 3 | 342 | 62.0k | 1.7M | 86.0k | 98 | 187.1k | 25m 56s | | make_plan* | sonnet | 1 | 153 | 54.5k | 1.0M | 108.9k | 39 | 109.3k | 22m 29s | | assess* | sonnet | 1 | 166 | 8.8k | 1.4M | 100.7k | 54 | 82.0k | 8m 8s | | dispatch:cf252062-b70b-468e-8c78-49c3d7239610* | sonnet | 1 | 1.5k | 483 | 16.3M | 133.5k | 181 | 309.4k | 3h 0m | | merge_gate_assess* | sonnet | 1 | 94 | 4.3k | 504.1k | 61.4k | 28 | 43.1k | 4m 6s | | prepare_pr* | sonnet | 1 | 52 | 6.0k | 230.2k | 57.2k | 16 | 40.8k | 1m 58s | | compose_pr* | sonnet | 1 | 67 | 3.2k | 263.0k | 37.5k | 17 | 18.8k | 1m 14s | | **Total** | | | 31.1k | 339.3k | 39.4M | 175.6k | | 1.6M | 5h 40m | \* *Step used a non-Anthropic provider; caching behavior may differ.* ## Token Efficiency | Step | LoC Changed | cache_read/LoC | cache_write/LoC | output/LoC | |------|-------------|----------------|-----------------|------------| | investigate | 0 | — | — | — | | rectify | 0 | — | — | — | | dry_walkthrough | 0 | — | — | — | | implement | 342 | 26883.4 | 663.9 | 205.5 | | retry_worktree | 430 | 6228.0 | 258.9 | 45.0 | | audit_impl | 0 | — | — | — | | make_plan | 0 | — | — | — | | assess | 2 | 722743.0 | 41017.5 | 4402.5 | | dispatch:cf252062-b70b-468e-8c78-49c3d7239610 | 0 | — | — | — | | merge_gate_assess | 0 | — | — | — | | prepare_pr | 0 | — | — | — | | compose_pr | 0 | — | — | — | | **Total** | **774** | 50956.4 | 2129.1 | 438.4 | ## Model Usage Breakdown | Model | steps | uncached | output | cache_read | cache_write | time | |-------|-------|----------|--------|------------|-------------|------| | fable | 1 | 23.2k | 38.8k | 1.6M | 287.4k | 16m 56s | | sonnet | 11 | 8.0k | 300.6k | 37.8M | 1.4M | 5h 23m | --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent c0828fd commit bf3632d

14 files changed

Lines changed: 565 additions & 0 deletions

src/autoskillit/cli/fleet/_fleet_run.py

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,32 @@ async def _execute_fleet_run(
4444

4545
ctx = make_context(cfg, project_dir=Path.cwd())
4646

47+
from autoskillit.server import ( # noqa: PLC0415
48+
_compute_effective_backend_map,
49+
_provider_aware_capability_overrides,
50+
)
51+
52+
_recipe_info = ctx.recipes.find(recipe, ctx.project_dir) if ctx.recipes else None
53+
_raw_steps = (
54+
ctx.recipes.load(_recipe_info.path).steps
55+
if _recipe_info is not None and ctx.recipes is not None
56+
else None
57+
)
58+
_prov_overrides, _ = _provider_aware_capability_overrides(
59+
dispatch_backend,
60+
recipe,
61+
ctx.config.providers,
62+
_raw_steps,
63+
skill_resolver=ctx.skill_resolver,
64+
)
65+
_effective_backend_map = _compute_effective_backend_map(
66+
_raw_steps,
67+
dispatch_backend.name if dispatch_backend else None,
68+
ctx.config.providers,
69+
recipe,
70+
skill_resolver=ctx.skill_resolver,
71+
)
72+
4773
effective_backend = dispatch_backend or ctx.backend
4874
has_ufa = (
4975
effective_backend.capabilities.has_unguarded_filesystem_access
@@ -91,6 +117,8 @@ async def quota_refresher(_cfg: object) -> None:
91117
resume_session_id=resume_session_id,
92118
prior_dispatch_id=prior_dispatch_id,
93119
dispatch_backend=dispatch_backend,
120+
provider_capability_overrides=_prov_overrides,
121+
effective_backend_map=_effective_backend_map,
94122
)
95123

96124

src/autoskillit/core/types/_type_protocols_recipe.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,7 @@ def validate_from_path(
5353
*,
5454
backend_name: str | None = None,
5555
ingredient_overrides: dict[str, str] | None = None,
56+
effective_backend_map: dict[str, str] | None = None,
5657
) -> dict[str, Any]: ...
5758

5859
def list_all(

src/autoskillit/fleet/_api.py

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -192,6 +192,7 @@ async def execute_dispatch(
192192
caller_instructions: str | None = None,
193193
provider_capability_overrides: dict[str, str] | None = None,
194194
dispatch_backend: CodingAgentBackend | None = None,
195+
effective_backend_map: dict[str, str] | None = None,
195196
) -> DispatchResult:
196197
"""Execute a single food truck dispatch.
197198
@@ -260,6 +261,7 @@ def _reject(error_code: FleetErrorCode, message: str, **kwargs: Any) -> Dispatch
260261
caller_instructions=caller_instructions,
261262
provider_capability_overrides=provider_capability_overrides,
262263
dispatch_backend=dispatch_backend,
264+
effective_backend_map=effective_backend_map,
263265
)
264266
except asyncio.CancelledError:
265267
raise
@@ -322,6 +324,7 @@ async def _run_dispatch(
322324
caller_instructions: str | None = None,
323325
provider_capability_overrides: dict[str, str] | None = None,
324326
dispatch_backend: CodingAgentBackend | None = None,
327+
effective_backend_map: dict[str, str] | None = None,
325328
) -> DispatchResult:
326329
"""Inner dispatch body — called after lock acquisition."""
327330
from autoskillit.fleet.state import (
@@ -367,6 +370,7 @@ async def _run_dispatch(
367370
ingredient_overrides=_merged_ingredients,
368371
temp_dir=tool_ctx.temp_dir,
369372
backend_name=_effective_backend.name if _effective_backend else None,
373+
effective_backend_map=effective_backend_map,
370374
)
371375
except ProcessStaleError as exc:
372376
return DispatchResult(

src/autoskillit/recipe/_api_listing.py

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,7 @@ def validate_from_path(
7878
lister: SkillLister | None = None,
7979
backend_name: str | None = None,
8080
ingredient_overrides: dict[str, str] | None = None,
81+
effective_backend_map: dict[str, str] | None = None,
8182
) -> dict[str, Any]:
8283
"""Validate a recipe YAML file at the given path.
8384
@@ -132,6 +133,7 @@ def validate_from_path(
132133
available_skills=frozenset(s.name for s in lister.list_all()),
133134
skill_resolver=_skill_resolver,
134135
backend_name=backend_name,
136+
effective_backend_map=effective_backend_map,
135137
)
136138
_pre_prune_findings = run_semantic_rules(pre_prune_ctx)
137139
recipe, _skip_resolutions = _prune_skipped_steps(
@@ -143,6 +145,7 @@ def validate_from_path(
143145
available_skills=known_skills,
144146
skill_resolver=_skill_resolver,
145147
backend_name=backend_name,
148+
effective_backend_map=effective_backend_map,
146149
)
147150
report = ctx.dataflow
148151
semantic_findings = run_semantic_rules(ctx)

src/autoskillit/recipe/repository.py

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -126,12 +126,14 @@ def validate_from_path(
126126
*,
127127
backend_name: str | None = None,
128128
ingredient_overrides: dict[str, str] | None = None,
129+
effective_backend_map: dict[str, str] | None = None,
129130
) -> dict[str, Any]:
130131
return _api.validate_from_path(
131132
script_path,
132133
temp_dir_relpath=temp_dir_relpath,
133134
backend_name=backend_name,
134135
ingredient_overrides=ingredient_overrides,
136+
effective_backend_map=effective_backend_map,
135137
)
136138

137139
def list_all(

src/autoskillit/server/__init__.py

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,9 @@
5757
"version_info",
5858
"make_context",
5959
"resolve_backend_override",
60+
# Admission helpers consumed by CLI (re-exported to avoid cross-submodule imports)
61+
"_provider_aware_capability_overrides",
62+
"_compute_effective_backend_map",
6063
# Wire-format compatibility middleware
6164
"ClaudeCodeCompatMiddleware",
6265
# Session-type visibility dispatcher (callable by tests)
@@ -140,6 +143,10 @@
140143
from autoskillit.server.tools import ( # noqa: E402, F401
141144
tools_workspace as _tools_workspace,
142145
)
146+
from autoskillit.server.tools._auto_overrides import ( # noqa: E402, F401
147+
_compute_effective_backend_map,
148+
_provider_aware_capability_overrides,
149+
)
143150
from autoskillit.server.tools.tools_kitchen import _build_tool_category_listing # noqa: E402, F401
144151

145152
for tag in sorted(ALL_VISIBILITY_TAGS):

src/autoskillit/server/tools/tools_fleet_dispatch.py

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -341,6 +341,7 @@ async def dispatch_food_truck(
341341
_fleet_load_result: dict[str, Any] = {}
342342
_capability_overrides: dict[str, str] = {}
343343
_cap_detail: CapabilityResolutionDetail | None = None
344+
_effective_backend_map: dict[str, str] | None = None
344345
if tool_ctx.recipes is not None:
345346
try:
346347
_preflight_recipe_info = tool_ctx.recipes.find(recipe, tool_ctx.project_dir)
@@ -471,6 +472,7 @@ async def dispatch_food_truck(
471472
caller_instructions=caller_instructions,
472473
provider_capability_overrides=_capability_overrides,
473474
dispatch_backend=dispatch_backend,
475+
effective_backend_map=_effective_backend_map,
474476
)
475477
except TimeoutError:
476478
logger.error(

src/autoskillit/server/tools/tools_recipe.py

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -369,11 +369,19 @@ async def validate_recipe(script_path: str) -> str:
369369
_validate_recipe_steps,
370370
skill_resolver=tool_ctx.skill_resolver,
371371
)
372+
_validate_effective_backend_map = _compute_effective_backend_map(
373+
_validate_recipe_steps,
374+
tool_ctx.backend.name if tool_ctx.backend else None,
375+
tool_ctx.config.providers,
376+
_validate_recipe_name,
377+
skill_resolver=tool_ctx.skill_resolver,
378+
)
372379
result = tool_ctx.recipes.validate_from_path(
373380
Path(script_path),
374381
temp_dir_relpath=temp_dir_display_str(tool_ctx.config.workspace.temp_dir),
375382
backend_name=tool_ctx.backend.name if tool_ctx.backend else None,
376383
ingredient_overrides=_cap_overrides,
384+
effective_backend_map=_validate_effective_backend_map,
377385
)
378386
return json.dumps(result)
379387
except Exception as exc:

tests/arch/test_capability_admission_control.py

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -337,6 +337,112 @@ def test_run_skill_references_git_metadata_write_capability() -> None:
337337
)
338338

339339

340+
def test_execute_dispatch_accepts_effective_backend_map() -> None:
341+
"""execute_dispatch signature must include effective_backend_map parameter."""
342+
import inspect
343+
344+
from autoskillit.fleet._api import execute_dispatch
345+
346+
sig = inspect.signature(execute_dispatch)
347+
assert "effective_backend_map" in sig.parameters, (
348+
"execute_dispatch must accept effective_backend_map — "
349+
"without it the CLI and dispatch_food_truck cannot thread the per-step map "
350+
"into the engine's internal load_and_validate call."
351+
)
352+
353+
354+
def _find_load_and_validate_call_in_run_dispatch(src: str) -> ast.Call | None:
355+
"""Return the load_and_validate Call node inside _run_dispatch, or None."""
356+
tree = ast.parse(src)
357+
for node in ast.walk(tree):
358+
if isinstance(node, ast.AsyncFunctionDef) and node.name == "_run_dispatch":
359+
for child in ast.walk(node):
360+
if (
361+
isinstance(child, ast.Call)
362+
and isinstance(child.func, ast.Attribute)
363+
and child.func.attr == "load_and_validate"
364+
):
365+
return child
366+
return None
367+
368+
369+
def test_run_dispatch_passes_effective_backend_map_to_load_and_validate() -> None:
370+
"""_run_dispatch must pass effective_backend_map kwarg to load_and_validate."""
371+
api_path = SRC_ROOT / "fleet" / "_api.py"
372+
src = api_path.read_text(encoding="utf-8")
373+
call_node = _find_load_and_validate_call_in_run_dispatch(src)
374+
assert call_node is not None, (
375+
"load_and_validate call not found inside _run_dispatch in fleet/_api.py"
376+
)
377+
kw_names = {kw.arg for kw in call_node.keywords if kw.arg is not None}
378+
assert "effective_backend_map" in kw_names, (
379+
"_run_dispatch must pass effective_backend_map to load_and_validate — "
380+
"without it the engine's validation always runs without per-step routing context."
381+
)
382+
383+
384+
def test_execute_fleet_run_calls_provider_aware_capability_overrides() -> None:
385+
"""_execute_fleet_run must call _provider_aware_capability_overrides (CLI admission parity)."""
386+
fleet_run_path = SRC_ROOT / "cli" / "fleet" / "_fleet_run.py"
387+
tree = ast.parse(fleet_run_path.read_text(encoding="utf-8"))
388+
assert _has_call_in_function(
389+
tree, "_execute_fleet_run", "_provider_aware_capability_overrides"
390+
), (
391+
"_execute_fleet_run must call _provider_aware_capability_overrides — "
392+
"without this, the CLI path does not compute provider-aware capability signals "
393+
"before dispatching, causing backend-incompatible-skill errors for Codex runs."
394+
)
395+
396+
397+
def test_execute_fleet_run_calls_compute_effective_backend_map() -> None:
398+
"""_execute_fleet_run must call _compute_effective_backend_map (CLI admission parity)."""
399+
fleet_run_path = SRC_ROOT / "cli" / "fleet" / "_fleet_run.py"
400+
tree = ast.parse(fleet_run_path.read_text(encoding="utf-8"))
401+
assert _has_call_in_function(tree, "_execute_fleet_run", "_compute_effective_backend_map"), (
402+
"_execute_fleet_run must call _compute_effective_backend_map — "
403+
"without this, the CLI path never computes per-step routing and always "
404+
"evaluates backend-compat with a flat backend map."
405+
)
406+
407+
408+
def test_dispatch_food_truck_forwards_effective_backend_map_to_execute_dispatch() -> None:
409+
"""dispatch_food_truck must pass effective_backend_map to execute_dispatch call."""
410+
fleet_path = SRC_ROOT / "server" / "tools" / "tools_fleet_dispatch.py"
411+
src = fleet_path.read_text(encoding="utf-8")
412+
tree = ast.parse(src)
413+
for node in ast.walk(tree):
414+
if isinstance(node, ast.AsyncFunctionDef) and node.name == "dispatch_food_truck":
415+
for child in ast.walk(node):
416+
if (
417+
isinstance(child, ast.Call)
418+
and isinstance(child.func, ast.Name)
419+
and child.func.id == "execute_dispatch"
420+
):
421+
kw_names = {kw.arg for kw in child.keywords if kw.arg is not None}
422+
assert "effective_backend_map" in kw_names, (
423+
"dispatch_food_truck must pass effective_backend_map to "
424+
"execute_dispatch — without it, the per-step routing map "
425+
"computed by the preflight is silently dropped at the engine boundary."
426+
)
427+
return
428+
pytest.fail("execute_dispatch call not found inside dispatch_food_truck")
429+
pytest.fail("dispatch_food_truck function not found in tools_fleet_dispatch.py")
430+
431+
432+
def test_validate_from_path_accepts_effective_backend_map() -> None:
433+
"""validate_from_path signature must accept effective_backend_map."""
434+
import inspect
435+
436+
from autoskillit.recipe._api_listing import validate_from_path
437+
438+
sig = inspect.signature(validate_from_path)
439+
assert "effective_backend_map" in sig.parameters, (
440+
"validate_from_path must accept effective_backend_map — "
441+
"without it the validate_recipe MCP tool cannot thread per-step routing "
442+
"context into backend-compat rule evaluation."
443+
)
444+
445+
340446
def test_compute_effective_backend_map_accepts_skill_resolver() -> None:
341447
"""_compute_effective_backend_map must accept a skill_resolver parameter."""
342448
auto_overrides_path = SRC_ROOT / "server" / "tools" / "_auto_overrides.py"

tests/arch/test_feature_markers.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@
3232
_TESTS_ROOT / "cli" / "test_fleet_status.py",
3333
_TESTS_ROOT / "cli" / "test_fleet_list.py",
3434
_TESTS_ROOT / "cli" / "test_fleet_run.py",
35+
_TESTS_ROOT / "cli" / "test_fleet_run_admission.py",
3536
_TESTS_ROOT / "server" / "test_tools_dispatch.py",
3637
_TESTS_ROOT / "server" / "test_tools_dispatch_validation.py",
3738
_TESTS_ROOT / "server" / "test_tools_dispatch_params.py",

0 commit comments

Comments
 (0)