Implementation Plan: T5-P3-A1-WP1 Recipe × Backend Composition Matrix CI Gate#4080
Merged
Trecek merged 4 commits intoJun 12, 2026
Conversation
Adds tests/recipe/test_recipe_backend_composition_matrix.py that validates every bundled recipe composes validly under every registered backend via full cross-product parametrization (currently 20 recipes x 2 backends = 40 cells). Governance tables: - DECLARED_UNSUPPORTED frozenset: by-design unsupported combos (currently empty) - KNOWN_BROKEN dict: should-work-but-broken combos (4 cells, tracking #4069) decorated with xfail(strict=True) so unexpected passes force removal - UNSUPPORTED_REASONS dict: skip message + issue_ref for DECLARED_UNSUPPORTED entries Two meta-tests prevent governance rot: - test_declared_unsupported_orphan_check: every DECLARED_UNSUPPORTED entry must reference a live recipe + backend - test_matrix_collection_count: matrix = R x B - U (structural guard) Per-cell assertions: result[valid] is True, content non-empty, no '[post-prune] dangling route:' suggestions, no backend-incompatible-skill ERROR suggestions. Also registers the new file in tests/_test_filter.py LAYER_CASCADE_CONSERVATIVE workspace block (direct autoskillit.workspace importer) and adds a row to tests/recipe/AGENTS.md for sub-CLAUDE.md completeness.
…trix The composition matrix test imports autoskillit.server, .execution, and .workspace — same as test_backend_reachability.py — to exercise the full pruning + validation pipeline. Add the per-file allowlist entry so test_test_files_respect_layer_boundaries passes. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…unsupported entries _MATRIX_IDS was built before DECLARED_UNSUPPORTED was defined, making it impossible to exclude unsupported combos. Reorder so DECLARED_UNSUPPORTED is defined first, then filter it in the _MATRIX_IDS comprehension. This makes test_matrix_collection_count correct once any entries are added. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
_MATRIX_IDS already excludes DECLARED_UNSUPPORTED tuples at module level (L36), so the runtime check at the top of test_recipe_backend_matrix_cell could never be true. Removing the dead guard also eliminates the latent KeyError in UNSUPPORTED_REASONS access. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Trecek
deleted the
t5-p3-a1-wp1-provide-a-merge-blocking-ci-test-that-proves-ev/4008
branch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add
tests/recipe/test_recipe_backend_composition_matrix.py— a parametrized test module that validates every bundled recipe composes validly under every registered backend. The module uses:BACKEND_REGISTRY × all_validated_recipe_names()(currently 20 recipes × 2 backends = 40 cells)DECLARED_UNSUPPORTEDfrozenset for by-design unsupported combos (currently empty — all combos are supported)KNOWN_BROKENdict for should-work-but-broken combos decorated withxfail(strict=True)(currently 4 cells: agent-eval and skill-eval on both backends, tracking Resolve dispatch-readiness exemptions for agent-eval and skill-eval recipes #4069)UNSUPPORTED_REASONSdict mapping eachDECLARED_UNSUPPORTEDentry to{reason, issue_ref}DECLARED_UNSUPPORTEDentries)Per live cell, the test calls
load_and_validate(recipe_name, project_dir=_PROJECT_ROOT, backend_name=backend_name, ingredient_overrides=_backend_capability_overrides(get_backend(backend_name)), lister=DefaultSkillResolver())and asserts:result['valid'] is True,result['content']is non-empty, no[post-prune] dangling route:suggestion, nobackend-incompatible-skillERROR suggestion.Closes #4008
Implementation Plan
Plan file:
/home/talon/projects/autoskillit-runs/impl-20260611-182135-648906/.autoskillit/temp/make-plan/t5-p3-a1-wp1-recipe-backend-composition-matrix_plan_2026-06-11_182600.md🤖 Generated with Claude Code via AutoSkillit
Token Usage Summary
* Step used a non-Anthropic provider; caching behavior may differ.
Token Efficiency
Model Usage Breakdown