[FEATURE] Rectify: Agent-Eval Detection Criteria Schema Drift Immunity#3885
Merged
Trecek merged 4 commits intoJun 7, 2026
Merged
Conversation
…nd semantic drift guard
Extract VALID_CRITERION_TYPES and REQUIRED_CRITERION_KEYS as canonical
frozenset constants in smoke_utils/_eval.py and use them in the runtime
validator. Add conformance tests that read a structured-format fixture
through parse_agent_eval_manifests, a negative test verifying the
constants drive rejection, and an entry for each eval callable in
skill_contracts.yaml. Add a criterion-schema-drift semantic rule that
fires at recipe-validation time when bundled manifests still use
plain-string detection_criteria entries, and update the existing
make-plan-canaries.json fixtures to use the structured {text, type}
format that the runtime validator requires.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…mmit - Fix cross-package submodule import: use smoke_utils (package init) not smoke_utils._eval - Update _eval.py json.dumps line numbers in schema version convention allowlist (+3 offset) - Bump recipe/rules file count exemption from 46 to 47 for rules_criterion_schema_drift.py - Update pyright suppression allowlist line 248→251 for recipe/__init__.py - Add tests/arch/fixtures/* to test-filter-manifest.yaml Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace smoke_utils import with local _REQUIRED_CRITERION_KEYS constant to avoid transitive import chain (smoke_utils → _telemetry → pipeline → config) that breaks IL-2 recipe layer contracts. Add conformance test to guard against divergence between local and canonical constants. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Subprocess tests for fleet import isolation inherit env from the parent process (MCP server or xdist worker), which can include vars that trigger circular imports in freshly-created venvs (fastmcp 3.3.1 _install_hints). Pass a minimal clean env instead, and capture stderr for better diagnostics on both tests. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Trecek
force-pushed
the
agent-eval-prep-skill-md-detection-criteria-schema-is-outdat/3611
branch
from
536f3cd to
3cf211a
Compare
Trecek
deleted the
agent-eval-prep-skill-md-detection-criteria-schema-is-outdat/3611
branch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The
agent-eval-prepSKILL.md documentsdetection_criteriaas plain strings but the validator (_eval.py) requires structured{text, type}objects. This is a class of bug where documentation-as-code-instruction drifts from the enforcement layer because there is no shared schema definition and no automated conformance test bridging the two.The architectural immunity creates a single canonical schema constant in the validation code, exposes it for testing, adds a conformance test that reads a fixture derived from the SKILL.md and feeds it through the validator, and registers a semantic rule that fires at recipe-validation time if manifests use the wrong format — making drift impossible without a test failure.
Closes #3611
Implementation Plan
Plan file:
/home/talon/projects/autoskillit-runs/remediation-20260607-131119-345378/.autoskillit/temp/rectify/rectify_agent_eval_schema_drift_immunity_2026-06-07_131119.md🤖 Generated with Claude Code via AutoSkillit
Token Usage Summary
* Step used a non-Anthropic provider; caching behavior may differ.
Model Usage Breakdown