Skip to content

Commit 6ac3fa7

Browse files
authored
chore(test): pin fixture deps and manifest-drive the e2e harness (#280)
* test(e2e): pin fixture dependencies, migrate to npm, and overhaul fixture lifecycle Pin every E2E fixture project's dependencies and check in package-lock.json files so fixture installs are reproducible and skip scaffolding the upstream templates from scratch on each run. Switch fixtures from bun-based installs to npm to match what users would do on a freshly scaffolded app. Replace the useFixture helper with createGetFixture, returning { fixture, users } so each test owns its own user lifecycle and tracks created users for explicit cleanup. Add scripts/lib/fixture-deps for the shared pinning logic plus tests. Migrate test infrastructure off scripts/run-tests.ts to native bun test --parallel for both unit and E2E pipelines, and fold test/e2e into the lint, format, and typecheck commands. * test(e2e): drive refresh script from a fixtures manifest Each fixture test file calls `describe()` at the top level, so the refresh script could no longer load them outside `bun test`. Move every fixture's config into `test/e2e/fixtures.manifest.ts` keyed by fixture name, and have `createFixtureHarness(name)` look up the config plus fixtureDir from the manifest, embedding both on the returned `Fixture`. The refresh script imports the manifest directly, dropping all test-file imports and the `CLERK_REFRESH_FIXTURES` env-var shim. Each fixture test file shrinks to a typed `createFixtureHarness("<name>")` call, and the helpers (`runFixtureTests`, `runFileExistsTest`, `runBrowserTests`) read config from the harness handle instead of threading it through every call. * build: tolerate ignored files in nano-staged hooks `oxfmt`/`oxlint` exit with an error when every staged file is ignored by the formatter/linter config, which breaks commits that only touch `test/e2e/fixtures/**`. Pass `--no-error-on-unmatched-pattern` to both so fixture-only commits no longer fail the pre-commit hook. * docs(changeset): add empty changeset * test(e2e): tighten fixture setup env handling and key validation Inline CLERK_PLATFORM_API_KEY plumbing per CLI invocation instead of a shared helper, validate publishable/secret keys with regex expectations, and force color in the op-wrapped subprocess for readable output. * test(e2e): replace key regex with presence checks in fixture setup * chore(test): refresh e2e fixtures Regenerates every fixture via `bun run e2e:refresh-fixtures` to pick up upstream template/dependency drift since the harness overhaul landed. * refactor(scripts): batch fixture dependency resolution and simplify range lookup - scripts/lib/fixture-deps.ts: parallelize per-field resolveVersion calls via Promise.all - scripts/lib/fixture-deps.ts: replace nested ternary in validatePinnedDependencyRanges with Array.find * test(e2e): tolerate transient rm errors during fixture cleanup EFAULT from node:fs/promises rm in afterAll was failing the Nuxt fixture even though the test itself passed (oven-sh/bun#28958 and #9298 surface EFAULT in place of the real errno). Wrap fixture-setup cleanup in a best-effort safeRm that logs and continues; the OS reclaims /tmp anyway.
1 parent da59716 commit 6ac3fa7

85 files changed

Lines changed: 34331 additions & 1070 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.changeset/beige-lilies-nail.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
---
2+
---

.claude/rules/e2e.md

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -79,7 +79,7 @@ A single test runner used by both `bun run test` and `bun run test:e2e`. Each te
7979
Each fixture directory contains:
8080

8181
- Framework source files (scaffolded by `config.scaffoldCmd`)
82-
- A `.test.ts` file that exports a `config: FixtureConfig` and calls `runFixtureTest()` and `runBrowserTest()`
82+
- A `.test.ts` file that exports a `config: FixtureConfig` and calls `runFixtureTests()` and `runBrowserTests()`
8383

8484
### FixtureConfig
8585

@@ -102,11 +102,11 @@ Defined in `test/e2e/lib/types.ts`:
102102
5. Parse `.env` / `.env.local` for publishable and secret keys (uses `detectPublishableKeyName` / `detectSecretKeyName` from CLI source)
103103
6. `bun install`
104104

105-
### Build + typecheck test (`runFixtureTest`)
105+
### Build + typecheck test (`runFixtureTests`)
106106

107107
Runs the framework build command, then `tsc --noEmit`. If the fixture has a `typecheck` script in its `package.json`, that's used instead of bare `tsc` (handles React Router's `react-router typegen`).
108108

109-
### Browser auth test (`runBrowserTest`)
109+
### Browser auth test (`runBrowserTests`)
110110

111111
1. Creates a disposable test user via `clerk api /users -X POST` (uses `+clerk_test` email suffix for OTP bypass)
112112
2. Starts the framework's dev server on a dynamic port
@@ -131,19 +131,19 @@ In CI, use `bunx playwright install chromium --with-deps` to include system-leve
131131

132132
Fixture files run in parallel (concurrency controlled by the runner, defaults to CPU count). Each fixture uses an isolated temp directory and `CLERK_CONFIG_DIR`, so there is no shared mutable state. Do not use `test.concurrent` within individual fixture files.
133133

134-
Within each test file, `useFixture()` runs `setupFixture()` once in `beforeAll` and shares the result with both the build test and browser test. This avoids duplicating the expensive setup.
134+
Within each test file, `createFixtureHarness()` runs `setupFixture()` once in `beforeAll` and shares the result with both the build test and browser test. This avoids duplicating the expensive setup.
135135

136136
## Adding a new fixture
137137

138138
1. Create `test/e2e/fixtures/<name>/`
139139
2. Scaffold the framework manually or via `bun run e2e:refresh-fixtures`
140-
3. Add a `<name>.test.ts` exporting `config: FixtureConfig` and calling `runFixtureTest()` and `runBrowserTest()`
140+
3. Add a `<name>.test.ts` exporting `config: FixtureConfig` and calling `runFixtureTests()` and `runBrowserTests()`
141141
4. Add a `README.md` in the fixture directory describing the project
142142

143143
Helper functions are in `test/e2e/lib/`:
144144

145145
- `fixture-setup.ts` - `setupFixture`
146-
- `fixture-test.ts` - `useFixture`, `runFixtureTest`, `runBrowserTest`
146+
- `fixture-test.ts` - `createFixtureHarness`, `runFixtureTests`, `runFileExistsTest`, `runBrowserTests`
147147
- `dev-server.ts` - `startDevServer` (allocates a port internally and retries on collision), `killDevServer`, `buildDevCommand`
148148
- `test-user.ts` - `createTestUser`, `deleteTestUser`
149149
- `logger.ts` - `log`, `debug` (shared logging; set `CLERK_E2E_DEBUG=1` for verbose output)

.oxfmtrc.json

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
{
2+
"$schema": "./node_modules/oxfmt/configuration_schema.json",
3+
"ignorePatterns": ["test/e2e/fixtures/**"]
4+
}

.oxlintrc.json

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,10 @@
11
{
2+
"$schema": "./node_modules/oxlint/configuration_schema.json",
23
"rules": {
34
"unicorn/no-process-exit": "error",
45
"no-console": "error"
56
},
7+
"ignorePatterns": ["test/e2e/fixtures/**"],
68
"overrides": [
79
{
810
"files": [
@@ -18,7 +20,8 @@
1820
"files": [
1921
"scripts/**",
2022
"packages/cli-core/src/**/*.test.ts",
21-
"packages/cli-core/src/test/**"
23+
"packages/cli-core/src/test/**",
24+
"test/e2e/**"
2225
],
2326
"rules": {
2427
"no-console": "off"

CONTRIBUTING.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -88,7 +88,7 @@ bunx playwright install chromium
8888

8989
bun run test:e2e:op # Run all E2E tests (secrets from 1Password)
9090
bun run test:e2e:op -- --filter react # Run only tests matching "react"
91-
bun run test:e2e:op -- --debug # Verbose helper logging (sets CLERK_E2E_DEBUG=1)
91+
bun run test:e2e:op -- --debug # Force serial execution for parsing logs (sets CLERK_E2E_DEBUG=1)
9292
bun run test:e2e:op -- --har # Capture HAR files to test/e2e/.har for network debugging
9393
bun run test:e2e:op -- --har-dir ./out # Capture HAR files to a custom directory
9494
bun run e2e:refresh-fixtures # Re-scaffold fixture projects from upstream CLIs

package.json

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -7,14 +7,14 @@
77
"scripts": {
88
"build": "bun run --filter @clerk/cli-core build",
99
"dev": "bun run --cwd packages/cli-core dev",
10-
"test": "bun run scripts/run-tests.ts --pattern 'packages/cli-core/src/**/*.test.ts' --pattern 'scripts/**/*.test.ts'",
11-
"test:e2e": "bun run scripts/run-tests.ts --pattern 'test/e2e/*.test.ts' --retries 1",
10+
"test": "bun test 'packages/cli-core/src/' 'scripts/' --parallel --only-failures",
11+
"test:e2e": "bun test 'test/e2e/' --retry 1 --parallel --only-failures",
1212
"test:e2e:op": "bun run scripts/run-e2e-op.ts",
1313
"e2e:refresh-fixtures": "bun run scripts/refresh-e2e-fixtures.ts",
14-
"typecheck": "bun run --filter './packages/*' typecheck && tsc --noEmit -p scripts/tsconfig.json",
15-
"lint": "bun run --filter './packages/*' lint && oxlint -c .oxlintrc.json scripts/",
16-
"format": "bun run --filter './packages/*' format && oxfmt --write scripts/",
17-
"format:check": "bun run --filter './packages/*' format:check && oxfmt --check scripts/",
14+
"typecheck": "bun run --filter './packages/*' typecheck && tsc --noEmit -p scripts/tsconfig.json && tsc --noEmit -p test/e2e/tsconfig.json",
15+
"lint": "bun run --filter './packages/*' lint && oxlint -c .oxlintrc.json scripts/ test/e2e/",
16+
"format": "bun run --filter './packages/*' format && oxfmt --write scripts/ test/e2e/",
17+
"format:check": "bun run --filter './packages/*' format:check && oxfmt --check scripts/ test/e2e/",
1818
"check:patches": "bun run scripts/check-patches.ts",
1919
"build:compile": "bun run --filter @clerk/cli-core build:compile",
2020
"version-packages": "bun changeset version",
@@ -42,10 +42,10 @@
4242
},
4343
"nano-staged": {
4444
"*.{ts,tsx,js,jsx}": [
45-
"oxfmt --write",
46-
"oxlint -c .oxlintrc.json"
45+
"oxfmt --write --no-error-on-unmatched-pattern",
46+
"oxlint -c .oxlintrc.json --no-error-on-unmatched-pattern"
4747
],
48-
"*.{md,json,css}": "oxfmt --write"
48+
"*.{md,json,css}": "oxfmt --write --no-error-on-unmatched-pattern"
4949
},
5050
"engines": {
5151
"bun": ">=1.3.10"

packages/cli-core/package.json

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,8 @@
1313
"typecheck": "tsc --noEmit -p tsconfig.json",
1414
"lint": "oxlint src/",
1515
"format": "oxfmt --write src/",
16-
"format:check": "oxfmt --check src/"
16+
"format:check": "oxfmt --check src/",
17+
"test": "bun test src/ --parallel"
1718
},
1819
"dependencies": {
1920
"@clerk/cli-extras": "workspace:*",

packages/cli-core/src/test/integration/lib/harness.ts

Lines changed: 20 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -256,7 +256,24 @@ mock.module(
256256

257257
// ── Real config module ───────────────────────────────────────────────────────
258258

259-
export const { _setConfigDir, readConfig, setProfile } = await import("../../../lib/config.ts");
259+
type ConfigModule = typeof import("../../../lib/config.ts");
260+
261+
let configModulePromise: Promise<ConfigModule> | null = null;
262+
263+
function getConfigModule(): Promise<ConfigModule> {
264+
configModulePromise ??= import("../../../lib/config.ts");
265+
return configModulePromise;
266+
}
267+
268+
export async function readConfig(): ReturnType<ConfigModule["readConfig"]> {
269+
return (await getConfigModule()).readConfig();
270+
}
271+
272+
export async function setProfile(
273+
...args: Parameters<ConfigModule["setProfile"]>
274+
): ReturnType<ConfigModule["setProfile"]> {
275+
return (await getConfigModule()).setProfile(...args);
276+
}
260277

261278
// ── Mock data ────────────────────────────────────────────────────────────────
262279

@@ -526,6 +543,7 @@ function setEnv(key: string, value: string) {
526543
*/
527544
export async function setupTest(): Promise<TestHarness> {
528545
const tempDir = await mkdtemp(join(tmpdir(), "clerk-integration-"));
546+
const { _setConfigDir } = await getConfigModule();
529547
_setConfigDir(tempDir);
530548
process.cwd = () => tempDir;
531549
setEnv("CLERK_PLATFORM_API_KEY", "test_platform_key");
@@ -558,6 +576,7 @@ export async function setupTest(): Promise<TestHarness> {
558576
* temporary directory.
559577
*/
560578
export async function teardownTest(harness: TestHarness): Promise<void> {
579+
const { _setConfigDir } = await getConfigModule();
561580
currentHarness = null;
562581
assertPromptQueuesEmpty();
563582
http.assertRoutesConsumed();

scripts/lib/fixture-deps.test.ts

Lines changed: 134 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,134 @@
1+
import { describe, expect, test } from "bun:test";
2+
import {
3+
applyPackageJsonOverrides,
4+
assertPinnedDependencyRanges,
5+
resolveDependencySpecsToExactVersions,
6+
validatePinnedDependencyRanges,
7+
} from "./fixture-deps.ts";
8+
9+
describe("applyPackageJsonOverrides", () => {
10+
test("merges dependency overrides into package.json", () => {
11+
const pkg: {
12+
dependencies?: Record<string, string>;
13+
devDependencies?: Record<string, string>;
14+
} = {
15+
dependencies: {
16+
existing: "^1",
17+
},
18+
};
19+
20+
applyPackageJsonOverrides(pkg, {
21+
dependencies: {
22+
added: "^2",
23+
},
24+
devDependencies: {
25+
dev: "^3",
26+
},
27+
});
28+
29+
expect(pkg.dependencies).toEqual({
30+
existing: "^1",
31+
added: "^2",
32+
});
33+
expect(pkg.devDependencies).toEqual({
34+
dev: "^3",
35+
});
36+
});
37+
});
38+
39+
describe("validatePinnedDependencyRanges", () => {
40+
test("allows satisfying generated dependencies without changing package.json", () => {
41+
const pkg = {
42+
dependencies: {
43+
"fixture-framework": "1.2.3",
44+
react: "^18",
45+
},
46+
};
47+
48+
const warnings = validatePinnedDependencyRanges(pkg, { "fixture-framework": "^1" });
49+
50+
expect(warnings).toEqual([]);
51+
expect(pkg.dependencies["fixture-framework"]).toBe("1.2.3");
52+
});
53+
54+
test("warns and keeps generated dependency when it falls outside the configured range", () => {
55+
const pkg = {
56+
dependencies: {
57+
"fixture-framework": "2.0.0",
58+
},
59+
};
60+
61+
const warnings = validatePinnedDependencyRanges(pkg, { "fixture-framework": "^1" });
62+
63+
expect(pkg.dependencies["fixture-framework"]).toBe("2.0.0");
64+
expect(warnings).toEqual([
65+
'fixture-framework generated version "2.0.0" does not satisfy pinned range "^1"',
66+
]);
67+
});
68+
});
69+
70+
describe("assertPinnedDependencyRanges", () => {
71+
test("throws when pinned dependency validation fails", () => {
72+
const pkg = {
73+
dependencies: {
74+
"fixture-framework": "2.0.0",
75+
},
76+
};
77+
78+
expect(() =>
79+
assertPinnedDependencyRanges(pkg, { "fixture-framework": "^1" }, "fixture-name"),
80+
).toThrow(
81+
'Pinned dependency validation failed for fixture-name:\n - fixture-framework generated version "2.0.0" does not satisfy pinned range "^1"',
82+
);
83+
});
84+
});
85+
86+
describe("resolveDependencySpecsToExactVersions", () => {
87+
test("rewrites generated dependency ranges to exact versions", async () => {
88+
const pkg = {
89+
dependencies: {
90+
"@clerk/react": "latest",
91+
react: "^19.0.0",
92+
"already-exact": "1.2.3",
93+
},
94+
devDependencies: {
95+
typescript: "~5.9.0",
96+
},
97+
};
98+
const resolved: string[] = [];
99+
const versions: Record<string, string> = {
100+
"react@^19.0.0": "19.2.6",
101+
"typescript@~5.9.0": "5.9.3",
102+
};
103+
104+
await resolveDependencySpecsToExactVersions(pkg, async (name, spec) => {
105+
resolved.push(`${name}@${spec}`);
106+
return versions[`${name}@${spec}`]!;
107+
});
108+
109+
expect(pkg).toEqual({
110+
dependencies: {
111+
"@clerk/react": "latest",
112+
react: "19.2.6",
113+
"already-exact": "1.2.3",
114+
},
115+
devDependencies: {
116+
typescript: "5.9.3",
117+
},
118+
});
119+
expect(resolved).toEqual(["react@^19.0.0", "typescript@~5.9.0"]);
120+
});
121+
122+
test("resolves pinned dependency ranges to exact satisfying versions", async () => {
123+
const pkg = {
124+
dependencies: {
125+
"fixture-framework": "^1",
126+
},
127+
};
128+
129+
await resolveDependencySpecsToExactVersions(pkg, async () => "1.2.3");
130+
131+
expect(pkg.dependencies["fixture-framework"]).toBe("1.2.3");
132+
expect(validatePinnedDependencyRanges(pkg, { "fixture-framework": "^1" })).toEqual([]);
133+
});
134+
});

0 commit comments

Comments
 (0)