test(cli): add matrix and generated build coverage (#1046)

Author: AmanVarshney01

.github/workflows/test.yaml

@@ -44,17 +44,9 @@ jobs:
       - name: Build Template Generator
         run: cd packages/template-generator && bun run build
 
-      - name: Build and Test CLI
+      - name: Test CLI
         working-directory: apps/cli
-        run: bun run build && bun run test:ci
+        run: bun run test:ci
         env:
           AGENT: 1
           BTS_TELEMETRY: 0
-
-      - name: Setup pnpm
-        uses: pnpm/action-setup@v6
-        with:
-          version: latest
-
-      - name: Publish smoke test (npm, pnpm, bun)
-        run: bun run smoke:publish

AGENTS.md

@@ -63,3 +63,17 @@ This repo is a Bun + Turborepo monorepo.
   - linked issue (if applicable),
   - verification steps run (`bun run check`, relevant tests),
   - screenshots/GIFs for web UI changes.
+
+## Agent skills
+
+### Issue tracker
+
+Issues are tracked in GitHub Issues for `AmanVarshney01/create-better-t-stack` using the `gh` CLI. See `docs/agents/issue-tracker.md`.
+
+### Triage labels
+
+The canonical triage roles map directly to same-named GitHub labels. See `docs/agents/triage-labels.md`.
+
+### Domain docs
+
+This repo uses a single-context domain doc layout. See `docs/agents/domain.md`.

CONTEXT.md

@@ -0,0 +1,108 @@
+# Better T Stack
+
+Better T Stack is a project scaffolding CLI that turns a stack selection into a reproducible TypeScript starter project.
+
+## Language
+
+**Project Configuration**:
+A complete selection of stack choices used to scaffold one project.
+_Avoid_: Config blob, options bag
+
+**Core Stack**:
+The primary framework and infrastructure choices that define the generated project's shape.
+_Avoid_: Main options, basics
+
+**Addon**:
+An optional capability layered onto a generated project without changing the core stack identity.
+_Avoid_: Plugin, extra
+
+**Example**:
+A generated sample feature included to demonstrate the selected stack.
+_Avoid_: Demo, sample app
+
+**Exhaustive Matrix**:
+The set of all meaningful core-stack combinations after normalizing unordered selections.
+_Avoid_: All permutations
+
+**Addon Compatibility Coverage**:
+Focused coverage proving addon rules, interactions, and representative generated output without crossing every addon subset through the exhaustive matrix.
+_Avoid_: Addon exhaustive product
+
+**Input Shape**:
+The structural form of user-provided CLI input before it is normalized into a project configuration.
+_Avoid_: Edge case, weird input
+
+**Virtual Generation**:
+Generation of the project tree in memory without writing the scaffolded project to disk.
+_Avoid_: Dry run
+
+**Filesystem Scaffolding**:
+Creation of the generated project on disk.
+_Avoid_: Real generation
+
+**Curated Build Set**:
+A small representative set of generated projects that are installed, typechecked, or built.
+_Avoid_: Build every matrix case
+
+**Compatibility Oracle**:
+A test-owned model of which project configurations should be accepted or rejected.
+_Avoid_: Validator reuse
+
+**Matrix Job**:
+A separately runnable test job that exercises large project-configuration matrices outside the default fast test suite.
+_Avoid_: Normal test run
+
+**Default Suite**:
+The fast test suite that runs focused unit, integration, and regression tests during ordinary development.
+_Avoid_: Main matrix
+
+**Matrix Smoke**:
+A representative matrix slice that runs in regular CI to catch broad compatibility regressions without running the full matrix.
+_Avoid_: Mini exhaustive test
+
+**Full Matrix Job**:
+The sharded long-running matrix job that exercises the complete normalized core-stack matrix.
+_Avoid_: Default tests
+
+**Create Path**:
+The CLI workflow that turns a project configuration into a new scaffolded project.
+_Avoid_: Init path, generation path
+
+**Add Path**:
+The CLI workflow that adds capabilities to an existing Better T Stack project.
+_Avoid_: Update path, addon install path
+
+**MCP Surface**:
+The Model Context Protocol server and tools that expose Better T Stack planning, creation, addon, guidance, and schema operations to agents.
+_Avoid_: MCP addon
+
+## Relationships
+
+- A **Project Configuration** contains one **Core Stack**
+- A **Project Configuration** may include zero or more **Addons**
+- A **Project Configuration** may include zero or more **Examples**
+- The **Exhaustive Matrix** covers **Core Stack** combinations
+- **Addon Compatibility Coverage** covers addon-specific rules outside the exhaustive matrix
+- **Input Shape** checks validate raw CLI input before project-configuration behavior is evaluated
+- **Virtual Generation** and **Filesystem Scaffolding** are two ways to exercise project generation
+- A **Curated Build Set** proves generated projects compile without building every matrix case
+- A **Compatibility Oracle** predicts expected pass/fail before production validation runs
+- A **Matrix Job** provides broad confidence without making the default test suite slow
+- The **Default Suite**, **Matrix Smoke**, and **Full Matrix Job** are the three testing tiers
+- The **Create Path** owns exhaustive project-configuration coverage
+- The **Add Path** has a focused suite over existing-project state and addon behavior
+- The **MCP Surface** is a critical CLI surface alongside the **Create Path** and **Add Path**
+- The **MCP Surface** uses focused contract coverage instead of participating in the **Full Matrix Job**
+
+## Example dialogue
+
+> **Dev:** "Should the exhaustive tests include every order of addons?"
+> **Domain expert:** "No. The **Exhaustive Matrix** covers normalized **Project Configurations**; addon ordering is an **Input Shape** concern."
+
+## Flagged ambiguities
+
+- "all possible combinations" was ambiguous between literal permutations and meaningful stack selections; resolved: use the **Exhaustive Matrix** for normalized core-stack combinations, with separate **Input Shape** tests for ordering, duplicates, and invalid `none` mixtures.
+- "all addon combinations" was ambiguous between rule coverage and a full addon-subset product; resolved: use **Addon Compatibility Coverage** instead of crossing every addon subset through the **Exhaustive Matrix**.
+- "all tests" was ambiguous between the default test suite and long-running exhaustive coverage; resolved: use a separate **Matrix Job** for broad matrix coverage.
+- "test the MCP" was ambiguous between matrix participation and protocol contract coverage; resolved: the **MCP Surface** gets focused contract coverage.
+- "complete CLI tests" was ambiguous between new-project creation and adding to existing projects; resolved: the **Create Path** gets exhaustive matrix coverage first, while the **Add Path** gets a focused complete suite.

apps/cli/bunfig.toml

@@ -2,8 +2,8 @@
 # Preload setup file for global setup/teardown
 preload = ["./test/setup.ts"]
 
-# Per-test timeout (3 minutes for smoke tests)
-timeout = 180000
+# Default per-test timeout. Long-running smoke/build tests set explicit timeouts.
+timeout = 30000
 
 # Skip test files from coverage reports
 coverageSkipTestFiles = true

apps/cli/package.json

@@ -63,10 +63,9 @@
     "build": "tsdown --publint",
     "dev": "tsdown --watch",
     "check-types": "tsc --noEmit",
-    "test": "bun run build && bun test",
-    "test:watch": "bun run build && bun test --watch",
-    "test:coverage": "bun run build && bun test --coverage",
-    "test:ci": "bun run build && CI=1 bun test --bail=5",
+    "test": "bun test",
+    "test:ci": "bun run build && BTS_SKIP_EXTERNAL_COMMANDS=1 BTS_TEST_MODE=1 bun test --bail=5",
+    "test:complete": "cd ../.. && bun run build && cd apps/cli && BTS_MATRIX_MODE=smoke BTS_BUILD_SAMPLES=1 bun test --bail=5",
     "prepublishOnly": "npm run build"
   },
   "dependencies": {

apps/cli/src/utils/compatibility-rules.ts

@@ -265,6 +265,8 @@ export function isExampleTodoAllowed(
 }
 
 export function isExampleAIAllowed(backend?: ProjectConfig["backend"], frontends: Frontend[] = []) {
+  if (backend === "none") return false;
+
   const includesSolid = frontends.includes("solid");
   const includesAstro = frontends.includes("astro");
   if (includesSolid || includesAstro) return false;
@@ -447,6 +449,14 @@ export function validateExamplesCompatibility(
     return validationErr("The 'ai' example is not compatible with the Solid frontend.");
   }
 
+  if (examplesArr.includes("ai") && (frontend ?? []).includes("astro")) {
+    return validationErr("The 'ai' example is not compatible with the Astro frontend.");
+  }
+
+  if (examplesArr.includes("ai") && backend === "none") {
+    return validationErr("The 'ai' example requires a backend.");
+  }
+
   // Convex AI example only supports React-based frontends
   if (examplesArr.includes("ai") && backend === "convex") {
     const frontendArr = frontend ?? [];

apps/cli/test/addons.test.ts

@@ -944,7 +944,7 @@ describe("Addon Configurations", () => {
           orm: "drizzle",
           auth: "better-auth",
           api: webCase.api,
-          examples: ["todo", "ai"],
+          examples: webCase.frontend === "astro" ? ["todo"] : ["todo", "ai"],
           dbSetup: "turso",
           webDeploy: "none",
           serverDeploy: "none",

apps/cli/test/basic-configurations.test.ts

@@ -106,10 +106,7 @@ describe("Basic Configurations", () => {
   });
 
   describe("Installation Options", () => {
-    // Skip install test in CI to avoid timeouts
-    const runInstallTest = process.env.CI ? it.skip : it;
-
-    runInstallTest(
+    it.skipIf(Boolean(process.env.CI))(
       "should work with install enabled",
       async () => {
         const result = await runTRPCTest({

apps/cli/test/clerk-matrix.test.ts

@@ -63,7 +63,7 @@ function needsServerClerkPublishableKey(backend: string, api: string) {
 }
 
 describe("Clerk matrix", () => {
-  it("should generate every supported Clerk combination", { timeout: 30_000 }, async () => {
+  it("should generate every supported Clerk combination", async () => {
     const standardFrontendCombos = buildFrontendCombos(standardWeb);
     const selfFrontendCombos = buildFrontendCombos(selfWeb, { requireWeb: true });
 
@@ -388,5 +388,5 @@ describe("Clerk matrix", () => {
 
     expect(combos).toHaveLength(499);
     expect(failures).toEqual([]);
-  });
+  }, 30_000);
 });

apps/cli/test/examples.test.ts

@@ -195,6 +195,48 @@ describe("Example Configurations", () => {
       expectError(result, "The 'ai' example is not compatible with the Solid frontend");
     });
 
+    it("should fail with AI example + Astro frontend", async () => {
+      const result = await runTRPCTest({
+        projectName: "ai-astro-fail",
+        examples: ["ai"],
+        backend: "hono",
+        runtime: "bun",
+        database: "sqlite",
+        orm: "drizzle",
+        auth: "none",
+        api: "orpc",
+        frontend: ["astro"],
+        addons: ["none"],
+        dbSetup: "none",
+        webDeploy: "none",
+        serverDeploy: "none",
+        expectError: true,
+      });
+
+      expectError(result, "The 'ai' example is not compatible with the Astro frontend");
+    });
+
+    it("should fail with AI example + no backend", async () => {
+      const result = await runTRPCTest({
+        projectName: "ai-no-backend-fail",
+        examples: ["ai"],
+        backend: "none",
+        runtime: "none",
+        database: "none",
+        orm: "none",
+        auth: "none",
+        api: "none",
+        frontend: ["tanstack-router"],
+        addons: ["none"],
+        dbSetup: "none",
+        webDeploy: "none",
+        serverDeploy: "none",
+        expectError: true,
+      });
+
+      expectError(result, "The 'ai' example requires a backend");
+    });
+
     it("should work with AI example + Convex + React frontend", async () => {
       const result = await runTRPCTest({
         projectName: "ai-convex-react",