feat: add validate-codemod-recipe agent skill#94
Merged
Conversation
Adds an agent skill that captures the iterative workflow for validating codemod migration recipes against a real codebase: run codemod → lint → inspect diffs → identify issues → fix AST transform → add test fixtures → run codemod tests → reset & re-run Includes a taxonomy of common codemod bugs (false positives, redundant transforms, broken references, type/value mismatch, formatting damage, dangling artifacts) and concrete fix patterns for each.
- Add decision framework for when to fix vs skip: generic fixes that improve the codemod for any consumer vs repo-specific patches - Add diagnostic signals section: using metrics output to detect false positives, verifying modified files actually reference the target type, tracking diff count between runs
There was a problem hiding this comment.
Pull request overview
Adds a new agent skill, validate-codemod-recipe, under .agents/skills/ to document a repeatable QA workflow for running codemod migration recipes against a real target repo, inspecting diffs, and iterating on fixes + fixtures until output is clean.
Changes:
- Introduces the
validate-codemod-recipeskill with an end-to-end validation loop (run → lint → diff review → fix transform → add fixtures → re-run). - Documents a concrete issue taxonomy (false positives, broken references, type/value mismatches, formatting damage, dangling artifacts) plus common remediation patterns.
- Defines fixture/test expectations (positive/negative fixtures, optional
metrics.jsonrules) and operational pitfalls (registry vs local sources,--targetsemantics, lint-staged/prettier interference).
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+22
to
+26
| | Input | Example | Required | | ||
| | ---------------- | ----------------------------------------------- | -------- | | ||
| | Codemod path | `../codemods/codemods/v1.51.0/migration-recipe` | Yes | | ||
| | Target directory | `packages`, `plugins`, or `.` | Yes | | ||
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds the
validate-codemod-recipeagent skill to.agents/skills/. This skill captures the iterative workflow for validating codemod migration recipes by running them against a real codebase and fixing issues found in the output.Why
Running codemods against a real monorepo like
backstage/backstagesurfaces bugs that unit tests miss — false positives, broken references, type/value mismatches, and redundant transforms. This workflow was developed while QA-ing the v1.51.0 migration recipe (see #93) and is now codified as a reusable skill.The workflow
What the skill covers
metrics.jsonformatyarn dlxusage, registry vs local sources,--targetsemantics, metrics trailing newlines, lint-staged interference