|
| 1 | +# Codemod Batch Test |
| 2 | + |
| 3 | +Tests the v1-to-v2 codemod against real-world repos to find bugs, missing transforms, and gaps. |
| 4 | + |
| 5 | +## How it works |
| 6 | + |
| 7 | +For each repo in `repos.json`, the batch test: |
| 8 | + |
| 9 | +1. Clones the repo (or resets an existing clone) |
| 10 | +2. Installs dependencies |
| 11 | +3. Runs baseline checks (typecheck, build, test, lint) to confirm the repo is healthy |
| 12 | +4. Runs the codemod using the programmatic API |
| 13 | +5. Packs local SDK packages as tarballs and rewrites `package.json` deps to use them (so the test runs against the current SDK branch, not published npm versions) |
| 14 | +6. Re-installs dependencies |
| 15 | +7. Re-runs the same checks |
| 16 | +8. Writes structured JSON reports |
| 17 | + |
| 18 | +Errors that appear in step 7 but not step 3 are codemod-introduced regressions. |
| 19 | + |
| 20 | +## Usage |
| 21 | + |
| 22 | +```bash |
| 23 | +# Build all SDK packages first (tarballs need built dist/) |
| 24 | +pnpm build:all |
| 25 | + |
| 26 | +# Run the batch test |
| 27 | +pnpm --filter @modelcontextprotocol/codemod batch-test |
| 28 | + |
| 29 | +# Clean cloned repos, results, and tarballs |
| 30 | +pnpm --filter @modelcontextprotocol/codemod batch-test:clean |
| 31 | +``` |
| 32 | + |
| 33 | +## Output |
| 34 | + |
| 35 | +Results are written to `batch-test/results/`: |
| 36 | + |
| 37 | +- `summary.json` — overview across all repos: which passed, which failed, error counts |
| 38 | +- `<repo-slug>/report.json` — per-repo detail: baseline vs post-codemod check results, codemod diagnostics, change counts |
| 39 | + |
| 40 | +## Repo manifest (`repos.json`) |
| 41 | + |
| 42 | +An array of repo entries. Each entry specifies a GitHub repo and one or more packages within it. |
| 43 | + |
| 44 | +```json |
| 45 | +{ |
| 46 | + "repo": "owner/repo-name", |
| 47 | + "ref": "main", |
| 48 | + "packages": [ |
| 49 | + { |
| 50 | + "dir": "packages/mcp-server", |
| 51 | + "sourceDir": "src", |
| 52 | + "checks": { |
| 53 | + "typecheck": "npx tsc --noEmit", |
| 54 | + "build": "npm run build", |
| 55 | + "test": "npm run test", |
| 56 | + "lint": null |
| 57 | + } |
| 58 | + } |
| 59 | + ] |
| 60 | +} |
| 61 | +``` |
| 62 | + |
| 63 | +| Field | Required | Default | Description | |
| 64 | +| ---------------------- | -------- | -------------------------------------- | ----------------------------------------------------------------- | |
| 65 | +| `repo` | yes | — | GitHub `owner/name` | |
| 66 | +| `ref` | no | `main` | Branch or tag to clone | |
| 67 | +| `packages` | no | `[{ "dir": ".", "sourceDir": "src" }]` | Package targets within the repo | |
| 68 | +| `packages[].dir` | yes | — | Path to package root (where `package.json` lives) | |
| 69 | +| `packages[].sourceDir` | no | `src` | Source directory relative to `dir` (passed to codemod) | |
| 70 | +| `packages[].checks` | no | auto-detect | Override check commands; set a value to `null` to skip that check | |
| 71 | + |
| 72 | +When `checks` is omitted, the runner auto-detects commands from the package's `package.json` scripts (probing names like `typecheck`, `build`, `test`, `lint`). The package manager is auto-detected from the lockfile at the repo root. |
| 73 | + |
| 74 | +## Analyzing results |
| 75 | + |
| 76 | +`analyze-prompt.md` contains instructions for Claude Code to run the batch test and produce a categorized analysis. Each error is classified as: |
| 77 | + |
| 78 | +| Category | Meaning | |
| 79 | +| ------------------- | -------------------------------------------------- | |
| 80 | +| `codemod-bug` | A transform produced incorrect output | |
| 81 | +| `missing-transform` | The codemod should handle this pattern but doesn't | |
| 82 | +| `manual-migration` | Expected — requires human judgment | |
| 83 | +| `repo-specific` | Unusual pattern not worth handling in the codemod | |
| 84 | + |
| 85 | +## Adding a repo |
| 86 | + |
| 87 | +1. Edit `repos.json` and add an entry |
| 88 | +2. Run `pnpm --filter @modelcontextprotocol/codemod batch-test` |
| 89 | +3. Check `results/<repo-slug>/report.json` for new findings |
| 90 | + |
| 91 | +For monorepos, list each package that uses `@modelcontextprotocol/sdk` as a separate entry in `packages`. |
| 92 | + |
| 93 | +## Iteration workflow |
| 94 | + |
| 95 | +``` |
| 96 | +1. Run the batch test |
| 97 | +2. Review results — identify codemod bugs / missing transforms |
| 98 | +3. Fix the codemod transforms |
| 99 | +4. Run batch-test:clean, then re-run the batch test |
| 100 | +5. Confirm the fixes resolved the issues |
| 101 | +6. Repeat |
| 102 | +``` |
0 commit comments