Skip to content

Commit 599ffc1

Browse files
caio-pizzolcaio-pizzol
committed
chore(dx): add pnpm test:visual orchestrator and testing guide
- Add scripts/test-visual.mjs: interactive wrapper around layout:compare with auth preflight, corpus check, and npm version selection - Add TESTING.md: comprehensive developer testing guide with decision matrix for which tests to run - Fix scripts/test-layout.mjs: use pnpm --filter instead of npm --workspace - Move tests/behavior/AGENTS.md → CLAUDE.md (symlink for compat) - Update CLAUDE.md, layout-snapshots README, and visual CLAUDE.md with cross-references to the new unified commands
1 parent bf83d00 commit 599ffc1

10 files changed

Lines changed: 792 additions & 276 deletions

File tree

‎CLAUDE.md‎

Lines changed: 18 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -130,32 +130,38 @@ Note: `packages/sdk/tools/__init__.py` is a manual file (Python package marker)
130130

131131
## Testing
132132

133-
Three types of tests, each with a different purpose:
133+
| What to verify | Command | Speed |
134+
|---|---|---|
135+
| Logic works? | `pnpm test` | seconds |
136+
| Editing works? | `pnpm test:behavior` | minutes |
137+
| Rendering regressed? | `pnpm test:visual` | ~10 min |
134138

135139
### Unit Tests (Vitest)
136140

137-
Co-located with source code as `feature.test.ts` next to `feature.ts`. Run with `pnpm test` from a package directory. Test pure logic, data transformations, and utilities in isolation.
141+
Co-located with source code as `feature.test.ts` next to `feature.ts`. Test pure logic, data transformations, and utilities in isolation.
138142

139143
- Framework: **Vitest** (config at `vitest.config.mjs`)
140144
- Most coverage in `packages/super-editor/` (526 files) and `packages/layout-engine/` (150 files)
141145
- Run a single package: `pnpm --filter <package> test`
142146

143147
### Behavior Tests (Playwright)
144148

145-
End-to-end tests that exercise features through the browser. Located in `e2e-tests/` and `tests/behavior/`.
149+
End-to-end tests that exercise editing features through the browser. Located in `tests/behavior/`.
146150

147-
- Framework: **Playwright**
148-
- Test editing commands, collaboration, import/export through a running SuperDoc instance
151+
- Framework: **Playwright** (Chromium, Firefox, WebKit)
152+
- Tests editing commands, formatting, tables, comments, tracked changes, lists, toolbar
153+
- Asserts on document state, not pixels — see `tests/behavior/README.md`
149154

150-
### Visual Regression Tests (Playwright + R2)
155+
### Visual Regression Tests (`pnpm test:visual`)
151156

152-
Screenshot-based tests that catch rendering changes. Located in `tests/visual/`. See `tests/visual/CLAUDE.md` for full details.
157+
Compares layout engine output (JSON structure) across ~382 test documents against a published npm version. This is the primary tool for catching rendering regressions.
153158

154-
- Framework: **Playwright** with pixel-diff comparison
155-
- Test documents auto-discovered from `test-data/rendering/*.docx`
156-
- Baselines stored in **Cloudflare R2** (not git), generated from `stable` branch in CI
157-
- CI runs as a **soft gate** — pixel diffs post a PR comment but don't block merge
158-
- **Community PRs cannot upload to R2** (requires Cloudflare credentials). A team member must generate baselines after merge.
159+
- Run: `pnpm test:visual` (interactive — prompts for reference version)
160+
- Flags: `--reference <version>`, `--match <pattern>`, `--limit <n>`
161+
- Handles auth, corpus download, build, and comparison automatically
162+
- Reports written to `tests/layout-snapshots/reports/`
163+
- Lower-level access: `pnpm layout:compare` (same engine, no interactive UX)
164+
- One-time setup: `npx wrangler login` (for corpus download from R2)
159165

160166
## Brand & Design System
161167

‎TESTING.md‎

Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,115 @@
1+
# Testing Guide
2+
3+
How to verify your changes before pushing.
4+
5+
## Quick Reference
6+
7+
| What to verify | Command | Speed | CI Gate |
8+
|---|---|---|---|
9+
| Logic works? | `pnpm test` | ~30s | Hard |
10+
| Editing works? | `pnpm test:behavior` | ~3 min | Hard |
11+
| Rendering regressed? | `pnpm test:visual` | ~10 min | Manual |
12+
13+
## Unit Tests
14+
15+
Test pure logic — data transformations, algorithms, style resolution, layout math.
16+
17+
```bash
18+
pnpm test # all packages
19+
pnpm test:layout # layout engine packages only
20+
pnpm test:editor # super-editor only
21+
pnpm --filter <pkg> test # specific package
22+
```
23+
24+
Tests are co-located with source code as `feature.test.ts` next to `feature.ts`. Framework: Vitest.
25+
26+
## Behavior Tests
27+
28+
Test editing interactions through a real browser — typing, formatting, tables, comments, tracked changes, clipboard, toolbar.
29+
30+
```bash
31+
pnpm test:behavior # all browsers, headless
32+
pnpm test:behavior -- --project=chromium # single browser
33+
pnpm test:behavior:headed # watch the browser
34+
pnpm test:behavior:ui # Playwright UI mode
35+
```
36+
37+
These assert on **document state**, not pixels. Located in `tests/behavior/`. See `tests/behavior/README.md` for writing tests.
38+
39+
**First-time setup:**
40+
41+
```bash
42+
pnpm --filter @superdoc-testing/behavior setup # install browser binaries
43+
```
44+
45+
## Visual Regression (Layout Comparison)
46+
47+
Compare layout engine output (JSON structure) across ~382 real-world documents against a published npm version. This is the primary tool for catching rendering regressions.
48+
49+
```bash
50+
pnpm test:visual # interactive
51+
pnpm test:visual -- --reference 1.16.0 # specific version
52+
pnpm test:visual -- --match tables --limit 5 # filtered, faster
53+
```
54+
55+
The command handles everything: corpus download, build, snapshot generation, comparison.
56+
57+
**First-time setup:**
58+
59+
```bash
60+
npx wrangler login # Cloudflare auth for downloading test documents
61+
pnpm test:visual # downloads corpus automatically on first run
62+
```
63+
64+
After the first run, the corpus is cached locally — no auth needed for subsequent runs.
65+
66+
**Reports** are written to `tests/layout-snapshots/reports/`. Each report includes a `summary.md` with changed documents and a `docs/` folder with per-document diffs.
67+
68+
**Advanced:** For lower-level access, use `pnpm layout:compare` directly. See `tests/layout-snapshots/README.md`.
69+
70+
## When to Run What
71+
72+
| I changed... | Run |
73+
|---|---|
74+
| A utility function or algorithm | `pnpm test` |
75+
| An editing command or extension | `pnpm test` + `pnpm test:behavior` |
76+
| Layout engine or style resolution | `pnpm test` + `pnpm test:visual` |
77+
| DomPainter rendering | `pnpm test` + `pnpm test:visual` |
78+
| PM adapter (data conversion) | `pnpm test` + `pnpm test:visual` |
79+
| Table rendering or spacing | All three |
80+
| Super-converter (import/export) | `pnpm test` + `pnpm test:visual` |
81+
82+
## CI Behavior
83+
84+
| Suite | Runs on PRs | Blocks merge |
85+
|---|---|---|
86+
| Unit tests | Yes | Yes |
87+
| Behavior tests | Yes (sharded across 3 runners) | Yes |
88+
| Visual regression | No (run manually) | No |
89+
90+
## Troubleshooting
91+
92+
**`pnpm test:visual` says auth expired:**
93+
94+
```bash
95+
npx wrangler login
96+
```
97+
98+
**Behavior tests fail with port conflict:**
99+
100+
```bash
101+
node scripts/free-port.mjs 9990
102+
pnpm test:behavior
103+
```
104+
105+
**Want to debug a behavior test visually:**
106+
107+
```bash
108+
pnpm test:behavior:headed # see the browser
109+
pnpm test:behavior:ui # Playwright inspector
110+
pnpm test:behavior:trace # record traces
111+
```
112+
113+
**Layout comparison shows many diffs but none are from your PR:**
114+
115+
You're probably comparing against an old npm version. The diffs include all changes on `main` since that release. Use `npm@next` (the default) for the closest baseline to current `main`.

‎package.json‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,7 @@
6161
"corpus:pull": "node scripts/corpus/pull.mjs",
6262
"corpus:push": "node scripts/corpus/push.mjs",
6363
"corpus:update-registry": "node scripts/corpus/update-registry.mjs",
64+
"test:visual": "bun scripts/test-visual.mjs",
6465
"watch": "pnpm --prefix packages/superdoc run watch:es",
6566
"check:all": "pnpm run format && pnpm run lint:fix && pnpm --prefix packages/super-editor run types:build && pnpm run test",
6667
"local:publish": "pnpm --prefix packages/superdoc version prerelease --preid=local && pnpm --prefix packages/superdoc publish --registry http://localhost:4873",
@@ -83,6 +84,7 @@
8384
"sdk:release:dry": "node packages/sdk/scripts/sdk-release.mjs --dry-run"
8485
},
8586
"devDependencies": {
87+
"@clack/prompts": "^1.0.1",
8688
"@commitlint/cli": "catalog:",
8789
"@commitlint/config-conventional": "catalog:",
8890
"@eslint/js": "catalog:",

‎pnpm-lock.yaml‎

Lines changed: 20 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎scripts/test-layout.mjs‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10,8 +10,8 @@ const workspaces = [
1010
];
1111

1212
for (const workspace of workspaces) {
13-
const result = spawnSync('npm', ['run', 'test', `--workspace=${workspace}`], {
14-
stdio: 'inherit'
13+
const result = spawnSync('pnpm', ['--filter', workspace, 'test'], {
14+
stdio: 'inherit',
1515
});
1616

1717
if (result.status !== 0) {

0 commit comments

Comments
 (0)