Skip to content

Commit dceb479

Browse files
josephfuscojosephfusco
committed
Build/Test Tools: Expand visual regression test coverage for admin pages.
1 parent dbf8614 commit dceb479

7 files changed

Lines changed: 470 additions & 162 deletions

File tree

‎.github/workflows/visual-regression-tests.yml‎

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
name: Visual Regression Tests
2+
3+
on:
4+
workflow_dispatch:
5+
6+
concurrency:
7+
group: ${{ github.workflow }}-${{ github.ref }}
8+
cancel-in-progress: true
9+
10+
# Disable permissions for all available scopes by default.
11+
# Any needed permissions should be configured at the job level.
12+
permissions: {}
13+
14+
env:
15+
LOCAL_DIR: build
16+
PUPPETEER_SKIP_DOWNLOAD: ${{ true }}
17+
18+
jobs:
19+
visual-regression:
20+
name: Visual Regression
21+
runs-on: ubuntu-24.04
22+
timeout-minutes: 30
23+
permissions:
24+
contents: read
25+
26+
steps:
27+
- name: Configure environment variables
28+
run: |
29+
echo "PHP_FPM_UID=$(id -u)" >> "$GITHUB_ENV"
30+
echo "PHP_FPM_GID=$(id -g)" >> "$GITHUB_ENV"
31+
32+
- name: Checkout repository
33+
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
34+
with:
35+
show-progress: ${{ runner.debug == '1' && 'true' || 'false' }}
36+
fetch-depth: 0
37+
persist-credentials: false
38+
39+
- name: Determine baseline commit
40+
id: baseline
41+
run: |
42+
echo "current_sha=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT"
43+
echo "base_sha=$(git merge-base HEAD origin/trunk)" >> "$GITHUB_OUTPUT"
44+
45+
- name: Set up Node.js
46+
uses: actions/setup-node@6044e13b5dc448c55e2357c09f80417699197238 # v6.2.0
47+
with:
48+
node-version-file: '.nvmrc'
49+
cache: npm
50+
51+
- name: Install npm dependencies
52+
run: npm ci
53+
54+
- name: Install Playwright browsers
55+
run: npx playwright install --with-deps chromium
56+
57+
- name: Checkout merge base
58+
run: git checkout ${{ steps.baseline.outputs.base_sha }}
59+
60+
- name: Build WordPress (baseline)
61+
run: npm run build
62+
63+
- name: Start Docker environment
64+
run: npm run env:start
65+
66+
- name: Install WordPress
67+
run: npm run env:install
68+
69+
- name: Generate baseline snapshots
70+
run: npx wp-scripts test-playwright --config tests/visual-regression/playwright.config.js --update-snapshots
71+
72+
- name: Checkout current commit
73+
run: git checkout ${{ steps.baseline.outputs.current_sha }}
74+
75+
- name: Rebuild WordPress
76+
run: npm run build
77+
78+
- name: Restart Docker environment
79+
run: npm run env:stop && npm run env:start
80+
81+
- name: Reinstall WordPress
82+
run: npm run env:install
83+
84+
- name: Run visual comparison
85+
run: npx wp-scripts test-playwright --config tests/visual-regression/playwright.config.js
86+
87+
- name: Upload report artifacts
88+
if: always()
89+
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0
90+
with:
91+
name: visual-regression-report-${{ github.run_id }}
92+
path: |
93+
artifacts/visual-report
94+
artifacts/test-results
95+
if-no-files-found: ignore
96+
include-hidden-files: true
97+

‎package.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -129,7 +129,7 @@
129129
"test:php": "node ./tools/local-env/scripts/docker.js run --rm php ./vendor/bin/phpunit",
130130
"test:coverage": "npm run test:php -- --coverage-html ./coverage/html/ --coverage-php ./coverage/php/report.php --coverage-text=./coverage/text/report.txt",
131131
"test:e2e": "wp-scripts test-playwright --config tests/e2e/playwright.config.js",
132-
"test:visual": "wp-scripts test-playwright --config tests/visual-regression/playwright.config.js",
132+
"test:visual": "bash tests/visual-regression/compare-branches.sh",
133133
"typecheck:php": "node ./tools/local-env/scripts/docker.js run --rm php composer phpstan",
134134
"gutenberg:checkout": "node tools/gutenberg/checkout-gutenberg.js",
135135
"gutenberg:build": "node tools/gutenberg/build-gutenberg.js",

‎tests/visual-regression/README.md‎

Lines changed: 44 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -2,10 +2,50 @@
22

33
These tests make use of Playwright, with a setup very similar to that of the e2e tests.
44

5+
## Prerequisites
6+
7+
- Node.js >= 20.10.0
8+
- A running WordPress environment (`npm run env:start`)
9+
- Playwright browsers installed (`npx playwright install chromium`)
10+
511
## How to Run the Tests Locally
612

7-
1. Check out trunk.
8-
2. Run `npm run test:visual` to generate some base snapshots.
9-
3. Check out the feature branch to be tested.
10-
4. Run `npm run test:visual` again. If any tests fail, the diff images can be found in `artifacts/`
13+
### Single-command comparison (recommended)
14+
15+
From a feature branch with a clean working tree, run:
16+
17+
```bash
18+
npm run test:visual
19+
```
20+
21+
This will automatically:
22+
1. Build WordPress and start the environment.
23+
2. Check out trunk to generate baseline snapshots.
24+
3. Return to your feature branch.
25+
4. Rebuild and compare the current branch against those baselines.
26+
5. Generate and open the HTML report with side-by-side visual diffs.
27+
28+
### Step-by-step comparison
29+
30+
Useful when iterating on a feature branch — generate baselines once, then re-run the comparison as you make changes without regenerating baselines every time.
31+
32+
```bash
33+
# 1. Check out trunk and generate baselines.
34+
git checkout trunk
35+
npm run test:visual
36+
37+
# 2. Check out your feature branch and compare.
38+
git checkout my-feature-branch
39+
npm run test:visual
40+
```
41+
42+
If any tests fail, diff images can be found in `artifacts/`.
43+
44+
### Direct Playwright execution
45+
46+
To run Playwright directly against existing snapshots (useful for quick iterations or updating baselines manually):
1147

48+
```bash
49+
npx wp-scripts test-playwright --config tests/visual-regression/playwright.config.js
50+
npx wp-scripts test-playwright --config tests/visual-regression/playwright.config.js --update-snapshots
51+
```

‎tests/visual-regression/compare-branches.sh‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
#!/bin/bash
2+
set -e
3+
4+
CURRENT_BRANCH=$(git rev-parse --abbrev-ref HEAD)
5+
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
6+
CONFIG="$SCRIPT_DIR/playwright.config.js"
7+
8+
if ! git diff-index --quiet HEAD --; then
9+
echo "Error: Uncommitted changes detected. Please commit or stash before running."
10+
exit 1
11+
fi
12+
13+
echo "Ensuring Playwright browsers are installed..."
14+
npx playwright install chromium
15+
16+
npm run build
17+
npm run env:start
18+
npm run env:install
19+
20+
# On trunk: generate baseline snapshots only.
21+
# On a feature branch: generate baselines from trunk, then compare.
22+
#
23+
# Manual workflow (still supported):
24+
# 1. Check out trunk → npm run test:visual (generates baselines)
25+
# 2. Check out feature branch → npm run test:visual (compares against baselines)
26+
#
27+
# Single-command workflow:
28+
# From a feature branch → npm run test:visual (does both automatically)
29+
if [ "$CURRENT_BRANCH" = "trunk" ]; then
30+
echo "On trunk — generating baseline snapshots..."
31+
npx wp-scripts test-playwright --config "$CONFIG" --update-snapshots
32+
exit 0
33+
fi
34+
35+
# Always restore the original branch on exit, even on failure.
36+
trap 'git checkout "$CURRENT_BRANCH" 2>/dev/null' EXIT
37+
38+
# Note: the CI workflow uses `git merge-base HEAD origin/trunk` instead of
39+
# trunk's tip. Baselines can differ if trunk has moved since your branch point.
40+
echo "Generating baselines from trunk..."
41+
git checkout trunk
42+
npm run build
43+
npm run env:start
44+
npm run env:install
45+
npx wp-scripts test-playwright --config "$CONFIG" --update-snapshots
46+
47+
echo "Comparing against $CURRENT_BRANCH..."
48+
git checkout "$CURRENT_BRANCH"
49+
npm run build
50+
# Full restart required — WordPress stores the build version in the database
51+
# during env:install. Without a restart, the second install may detect the
52+
# same version and skip setup, leaving stale state from the baseline branch.
53+
npm run env:stop && npm run env:start
54+
npm run env:install
55+
# Allow the comparison to "fail" (i.e. detect diffs) without exiting
56+
# the script, so the HTML report still generates and opens locally.
57+
npx wp-scripts test-playwright --config "$CONFIG" || true

‎tests/visual-regression/config/screenshot.css‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
/*
2+
* Global stylesheet applied during screenshot capture.
3+
*
4+
* Hides volatile elements that change between environments or runs,
5+
* preventing false positives in visual regression comparisons.
6+
* Applied via Playwright's stylePath config option.
7+
*
8+
* See: https://playwright.dev/docs/test-snapshots#stylepath
9+
*/
10+
11+
/*
12+
* Uses `visibility: hidden` instead of `display: none` to preserve
13+
* each element's layout space. Collapsing elements with `display: none`
14+
* would shift surrounding content and cause false positives elsewhere.
15+
*/
16+
17+
/* WordPress version/update nag in the admin footer. */
18+
#footer-upgrade {
19+
visibility: hidden !important;
20+
}
21+
22+
/* Admin bar user-specific content (Howdy, gravatar). */
23+
#wp-admin-bar-root-default {
24+
visibility: hidden !important;
25+
}
26+
27+
/* Gutenberg plugin menu item — not present in all environments. */
28+
#toplevel_page_gutenberg {
29+
visibility: hidden !important;
30+
}
31+
32+
/* Gravatar images — external service, different per environment. */
33+
.avatar {
34+
visibility: hidden !important;
35+
}
36+
37+
/* Date columns in list tables — relative timestamps shift between runs. */
38+
.column-date {
39+
visibility: hidden !important;
40+
}
41+
42+
/* Dashboard widgets with dynamic counts and activity. */
43+
#dashboard_right_now .inside,
44+
#dashboard_activity .inside {
45+
visibility: hidden !important;
46+
}
47+
48+
/* Update-related timestamps. */
49+
.update-last-checked {
50+
visibility: hidden !important;
51+
}
52+
53+
/* Admin notices — various nags (PHP deprecation, updates, etc.). */
54+
.notice,
55+
.update-nag,
56+
.updated,
57+
.error:not(#error),
58+
#message {
59+
visibility: hidden !important;
60+
}
61+
62+
/* General Settings — live date/time preview changes on every run. */
63+
#local-time,
64+
.example {
65+
visibility: hidden !important;
66+
}
67+
68+
/* Users list table — post counts vary based on test data. */
69+
.column-posts {
70+
visibility: hidden !important;
71+
}

‎tests/visual-regression/playwright.config.js‎

Lines changed: 40 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ import { defineConfig } from '@playwright/test';
77
/**
88
* WordPress dependencies
99
*/
10+
// CJS require — the @wordpress/scripts config does not export ESM.
1011
const baseConfig = require( '@wordpress/scripts/config/playwright.config' );
1112

1213
process.env.WP_ARTIFACTS_PATH ??= path.join( process.cwd(), 'artifacts' );
@@ -15,9 +16,47 @@ process.env.STORAGE_STATE_PATH ??= path.join(
1516
'storage-states/admin.json'
1617
);
1718

19+
const reporter = [
20+
[ 'list' ],
21+
...( process.env.CI ? [ [ 'github' ] ] : [] ),
22+
[
23+
'html',
24+
{
25+
open: process.env.CI ? 'never' : 'always',
26+
outputFolder: path.join(
27+
process.env.WP_ARTIFACTS_PATH,
28+
'visual-report'
29+
),
30+
},
31+
],
32+
];
33+
1834
const config = defineConfig( {
1935
...baseConfig,
20-
globalSetup: undefined,
36+
fullyParallel: true,
37+
// No retries — visual diffs are expected when regressions exist;
38+
// retrying would just re-confirm the same diff.
39+
retries: 0,
40+
workers: process.env.CI ? 1 : undefined,
41+
reporter,
42+
use: {
43+
...baseConfig.use,
44+
viewport: { width: 1280, height: 720 },
45+
},
46+
expect: {
47+
toHaveScreenshot: {
48+
// Only disables CSS animations/transitions. JavaScript-driven
49+
// animations (e.g. jQuery .animate()) can still cause flakes.
50+
animations: 'disabled',
51+
// Captures the entire scrollable page, not just the viewport.
52+
// The viewport width (1280) still matters — it controls layout.
53+
fullPage: true,
54+
// 1% tolerance — catches real regressions while ignoring
55+
// sub-pixel anti-aliasing differences across environments.
56+
maxDiffPixelRatio: 0.01,
57+
stylePath: path.join( __dirname, 'config', 'screenshot.css' ),
58+
},
59+
},
2160
webServer: {
2261
...baseConfig.webServer,
2362
command: 'npm run env:start',

0 commit comments

Comments
 (0)