Problem
We're starting to rely on agentic detection of project structure (frameworks, monorepo layouts) in the wizard, but we don't yet have an automated way to catch regressions in that detection. The agentic source-map detection landed in #689 and the broader monorepo support is tracked in #113 — both would benefit from a guardrail.
Proposal
Add a CI/eval step that exercises framework and monorepo detection against a set of representative fixture projects, so we can verify detection behavior on every change and catch regressions before they ship.
Rough shape:
- A corpus of fixture repos / manifests covering common cases (single-app, pnpm/yarn/npm workspaces, mixed framework monorepos, Next.js in a subpackage, etc.)
- Run the detector against each fixture and assert on the structured JSON output (detected projects, frameworks, chosen manifest globs)
- Wire it into CI so detection changes are evaluated automatically
References
Filed from a Slack thread — reminder from the error tracking team to add a framework + monorepo detection CI/eval step.
Problem
We're starting to rely on agentic detection of project structure (frameworks, monorepo layouts) in the wizard, but we don't yet have an automated way to catch regressions in that detection. The agentic source-map detection landed in #689 and the broader monorepo support is tracked in #113 — both would benefit from a guardrail.
Proposal
Add a CI/eval step that exercises framework and monorepo detection against a set of representative fixture projects, so we can verify detection behavior on every change and catch regressions before they ship.
Rough shape:
References
Filed from a Slack thread — reminder from the error tracking team to add a framework + monorepo detection CI/eval step.