Skip to content

Commit 99746bf

Browse files
balzssclaude
balzss
and
claude
committed
feat(regression-test): capture screenshots across canvas, light, and dark themes
Render each component page under multiple themes and screenshot each one, so the visual regression suite compares canvas, light, and dark side by side. - layout.tsx reads the `?theme=` query param and applies the matching theme via InstUISettingsProvider. It renders `canvas` for the first paint and switches in an effect to avoid a hydration mismatch, exposing `data-theme` on <html> so the spec can wait for the theme to be applied before screenshotting. - spec.cy.ts is now data-driven: a PAGES list (slug/title/wait/a11y-skip) times a THEMES list. Each page is screenshotted as `<slug>-<theme>` before the axe check so an a11y failure can never drop a visual baseline. Axe runs per theme with skipFailures and the totals are asserted once at the end, keeping a11y a gate across all themes without aborting the other themes' screenshots. - Screenshot capture moved out of the support afterEach into the spec. Screenshot filenames change from `<title>` to `<slug>-<theme>`, so the first run re-baselines every image. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 1d89888 commit 99746bf

4 files changed

Lines changed: 182 additions & 247 deletions

File tree

‎regression-test/README.md‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,8 @@ A small Next.js app that imports `@instructure/ui` locally and exposes one page
66
- **Detect a11y issues** — axe-core runs against every page.
77
- **Detect unexpected console errors** — the spec's `afterEach` hook asserts `console.error` was not called.
88

9+
Each page is captured once **per theme** (`canvas`, `light`, `dark`), so screenshots are named `<slug>-<theme>.png`. The theme is selected with the `?theme=<key>` query param, which `src/app/layout.tsx` reads and applies via `InstUISettingsProvider`. Add or remove themes with the `THEMES` array in `cypress/e2e/spec.cy.ts` (this multiplies the screenshot/baseline count).
10+
911
See the [visual regression testing guide](../docs/testing/visual-regression.md) for the full CI pipeline, the diff report UI, and tuning notes.
1012

1113
## Why npm instead of pnpm?
@@ -41,6 +43,7 @@ Run the Cypress suite against the running server:
4143
## Adding a new component
4244

4345
1. Create `src/app/<component-name>/page.tsx`. Start the file with `'use client'` and wrap the rendered markup in an element with the `axe-test` class — that's what the axe-core check selects against.
44-
2. Add a corresponding `it(...)` block in `cypress/e2e/spec.cy.ts` that visits `http://localhost:3000/<component-name>`, calls `cy.injectAxe()`, and `cy.checkA11y('.axe-test', axeOptions, terminalLog)`.
45-
3. If the component animates or loads content asynchronously, add a `cy.wait(<ms>)` before `injectAxe()`.
46-
4. Commit and push. The first PR run will show the new screenshot as "New"; merging the PR promotes it to a baseline automatically.
46+
2. Add an entry to the `PAGES` array in `cypress/e2e/spec.cy.ts` with the page's `slug` and a human-readable `title`. The suite generates the visit, per-theme screenshots, and the axe check for you.
47+
- If the component animates or loads content asynchronously, set `wait: <ms>`.
48+
- If the page has a known a11y issue, set `a11y: false` with an `a11ySkipReason` (ticket or short note) until it's fixed.
49+
3. Commit and push. The first PR run will show the new screenshots (one per theme) as "New"; merging the PR promotes them to baselines automatically.

‎regression-test/cypress/e2e/spec.cy.ts‎

Lines changed: 137 additions & 212 deletions
Original file line numberDiff line numberDiff line change
@@ -72,219 +72,144 @@ const axeOptions: { runOnly: RunOnly } = {
7272
}
7373
}
7474

75-
describe('visual regression test', () => {
76-
it('Metric, Pill, Tag, TimeSelect, Text', () => {
77-
cy.visit('http://localhost:3000/small-components')
78-
cy.injectAxe()
79-
cy.checkA11y('.axe-test', axeOptions, terminalLog)
80-
})
81-
82-
it('Alert', () => {
83-
cy.visit('http://localhost:3000/alert')
84-
cy.injectAxe()
85-
cy.checkA11y('.axe-test', axeOptions, terminalLog)
86-
})
87-
88-
it('Avatar', () => {
89-
cy.visit('http://localhost:3000/avatar')
90-
cy.wait(300) // images render a frame later, Chromatic needs a bit more delay
91-
cy.injectAxe()
92-
cy.checkA11y('.axe-test', axeOptions, terminalLog)
93-
})
94-
95-
it('Badge', () => {
96-
cy.visit('http://localhost:3000/badge')
97-
cy.injectAxe()
98-
cy.checkA11y('.axe-test', axeOptions, terminalLog)
99-
})
100-
101-
it('Billboard', () => {
102-
cy.visit('http://localhost:3000/billboard')
103-
cy.injectAxe()
104-
cy.checkA11y('.axe-test', axeOptions, terminalLog)
105-
})
106-
107-
it('Breadcrumb', () => {
108-
cy.visit('http://localhost:3000/breadcrumb')
109-
cy.wait(300) // wait for text to be truncated
110-
//TODO There are a11y issues, fix INSTUI-4676 before uncommenting
111-
//cy.injectAxe()
112-
//cy.checkA11y('.axe-test', axeOptions, terminalLog)
113-
})
114-
115-
it('Button and derivatives', () => {
116-
cy.visit('http://localhost:3000/button')
117-
cy.wait(100)
118-
cy.injectAxe()
119-
cy.checkA11y('.axe-test', axeOptions, terminalLog)
120-
})
121-
122-
it('Byline', () => {
123-
cy.visit('http://localhost:3000/byline')
124-
cy.injectAxe()
125-
cy.checkA11y('.axe-test', axeOptions, terminalLog)
126-
})
127-
128-
it('Calendar', () => {
129-
cy.visit('http://localhost:3000/calendar')
130-
cy.injectAxe()
131-
cy.checkA11y('.axe-test', axeOptions, terminalLog)
132-
})
133-
134-
it('Checkbox', () => {
135-
cy.visit('http://localhost:3000/checkbox')
136-
cy.wait(100) // needed so checkbox dont trigger axe check fails
137-
cy.injectAxe()
138-
cy.checkA11y('.axe-test', axeOptions, terminalLog)
139-
})
140-
141-
it('Checkboxgroup', () => {
142-
cy.visit('http://localhost:3000/checkboxgroup')
143-
cy.wait(300)
144-
cy.injectAxe()
145-
cy.checkA11y('.axe-test', axeOptions, terminalLog)
146-
})
147-
148-
it('ColorPicker', () => {
149-
cy.visit('http://localhost:3000/colorpicker')
150-
cy.wait(300)
151-
cy.injectAxe()
152-
// TODO: Fix ARIA violations before enabling a11y check
153-
// - aria-allowed-attr (critical): 1 node
154-
// - aria-prohibited-attr (serious): 3 nodes
155-
// ColorMixer has aria-disabled on plain div without proper role
156-
// cy.checkA11y('.axe-test', axeOptions, terminalLog)
157-
})
158-
159-
it('Contextview', () => {
160-
cy.visit('http://localhost:3000/contextview')
161-
cy.injectAxe()
162-
cy.checkA11y('.axe-test', axeOptions, terminalLog)
163-
})
164-
165-
it('Custom and Lucide icons', () => {
166-
cy.visit('http://localhost:3000/custom-icons')
167-
cy.injectAxe()
168-
cy.checkA11y('.axe-test', axeOptions, terminalLog)
169-
})
170-
171-
it('Dateinput, DateInput2', () => {
172-
cy.visit('http://localhost:3000/dateinput')
173-
cy.wait(400)
174-
cy.injectAxe()
175-
cy.checkA11y('.axe-test', axeOptions, terminalLog)
176-
})
177-
178-
it('DateTimeInput', () => {
179-
cy.visit('http://localhost:3000/datetimeinput')
180-
cy.wait(400)
181-
cy.injectAxe()
182-
cy.checkA11y('.axe-test', axeOptions, terminalLog)
183-
})
184-
185-
it('Drilldown', () => {
186-
cy.visit('http://localhost:3000/drilldown')
187-
cy.wait(300) // Drilldown dropdown renders a frame later, Chromatic needs a bit more delay
188-
cy.injectAxe()
189-
cy.checkA11y('.axe-test', axeOptions, terminalLog)
190-
})
191-
192-
it('Filedrop', () => {
193-
cy.visit('http://localhost:3000/filedrop')
194-
cy.injectAxe()
195-
cy.checkA11y('.axe-test', axeOptions, terminalLog)
196-
})
197-
198-
it('Form errors', () => {
199-
cy.visit('http://localhost:3000/form-errors')
200-
cy.wait(300)
201-
cy.injectAxe()
202-
cy.checkA11y('.axe-test', axeOptions, terminalLog)
203-
})
204-
205-
it('Heading', () => {
206-
cy.visit('http://localhost:3000/heading')
207-
cy.injectAxe()
208-
cy.checkA11y('.axe-test', axeOptions, terminalLog)
209-
})
210-
211-
it('Img', () => {
212-
cy.visit('http://localhost:3000/img')
213-
cy.wait(100) // images may render a frame later
214-
cy.injectAxe()
215-
cy.checkA11y('.axe-test', axeOptions, terminalLog)
216-
})
217-
218-
it('Link', () => {
219-
cy.visit('http://localhost:3000/link')
220-
cy.injectAxe()
221-
cy.checkA11y('.axe-test', axeOptions, terminalLog)
222-
})
223-
224-
it('Menu', () => {
225-
cy.visit('http://localhost:3000/menu')
226-
cy.wait(300)
227-
// TODO Fix INSTUI-4677 before enabling this
228-
//cy.injectAxe()
229-
//cy.checkA11y('.axe-test', axeOptions, terminalLog)
230-
})
231-
232-
it('Options', () => {
233-
cy.visit('http://localhost:3000/options')
234-
cy.injectAxe()
235-
cy.checkA11y('.axe-test', axeOptions, terminalLog)
236-
})
237-
238-
it('Pagination', () => {
239-
cy.visit('http://localhost:3000/pagination')
240-
cy.wait(400) // needed so tooltips dont trigger axe check fails
241-
cy.injectAxe()
242-
cy.checkA11y('.axe-test', axeOptions, terminalLog)
243-
})
244-
245-
it('Progressbar', () => {
246-
cy.visit('http://localhost:3000/progressbar')
247-
cy.injectAxe()
248-
cy.checkA11y('.axe-test', axeOptions, terminalLog)
249-
})
250-
251-
it('Select, SimpleSelect', () => {
252-
cy.visit('http://localhost:3000/select')
253-
cy.wait(300)
254-
cy.injectAxe()
255-
cy.checkA11y('.axe-test', axeOptions, terminalLog)
256-
})
257-
258-
it('Table', () => {
259-
cy.visit('http://localhost:3000/table')
260-
cy.injectAxe()
261-
cy.checkA11y('.axe-test', axeOptions, terminalLog)
262-
})
263-
264-
it('Tabs', () => {
265-
cy.visit('http://localhost:3000/tabs')
266-
cy.injectAxe()
267-
cy.checkA11y('.axe-test', axeOptions, terminalLog)
268-
})
269-
270-
it('Tooltip', () => {
271-
cy.visit('http://localhost:3000/tooltip')
272-
cy.wait(300) // tooltips render a frame later, Chromatic needs a bit more delay
273-
cy.injectAxe()
274-
cy.checkA11y('.axe-test', axeOptions, terminalLog)
275-
})
75+
const BASE_URL = 'http://localhost:3000'
76+
77+
// Themes to capture. Each page is screenshotted once per theme (see the
78+
// `?theme=` handling in src/app/layout.tsx). Adding a theme here multiplies the
79+
// screenshot/baseline count by one.
80+
const THEMES = ['canvas', 'light', 'dark'] as const
81+
82+
type PageSpec = {
83+
// URL segment and page directory under src/app/<slug>/page.tsx
84+
slug: string
85+
// Human-readable Cypress test title
86+
title: string
87+
// Extra settle time (ms) for pages with async/animated content
88+
wait?: number
89+
// Run the axe a11y check for this page (default true). Some pages have known
90+
// issues tracked separately and opt out until fixed.
91+
a11y?: boolean
92+
// Reason/ticket for skipping a11y, for the record
93+
a11ySkipReason?: string
94+
}
27695

277-
it('TreeBrowser', () => {
278-
cy.visit('http://localhost:3000/treebrowser')
279-
cy.wait(1000) // large timeout is needed for CI (to finish animation?)
280-
// TODO axe fails with color contrast issues, try to remove animations from TreeBrowser
281-
//cy.injectAxe()
282-
//cy.checkA11y('.axe-test', axeOptions, terminalLog)
283-
})
96+
const PAGES: PageSpec[] = [
97+
{ slug: 'small-components', title: 'Metric, Pill, Tag, TimeSelect, Text' },
98+
{ slug: 'alert', title: 'Alert' },
99+
{ slug: 'avatar', title: 'Avatar', wait: 300 },
100+
{ slug: 'badge', title: 'Badge' },
101+
{ slug: 'billboard', title: 'Billboard' },
102+
{
103+
slug: 'breadcrumb',
104+
title: 'Breadcrumb',
105+
wait: 300,
106+
a11y: false,
107+
a11ySkipReason: 'INSTUI-4676'
108+
},
109+
{ slug: 'button', title: 'Button and derivatives', wait: 100 },
110+
{ slug: 'byline', title: 'Byline' },
111+
{ slug: 'calendar', title: 'Calendar' },
112+
{ slug: 'checkbox', title: 'Checkbox', wait: 100 },
113+
{ slug: 'checkboxgroup', title: 'Checkboxgroup', wait: 300 },
114+
{
115+
slug: 'colorpicker',
116+
title: 'ColorPicker',
117+
wait: 300,
118+
a11y: false,
119+
a11ySkipReason:
120+
'ColorMixer ARIA violations (aria-allowed-attr, aria-prohibited-attr)'
121+
},
122+
{ slug: 'contextview', title: 'Contextview' },
123+
{ slug: 'custom-icons', title: 'Custom and Lucide icons' },
124+
{ slug: 'dateinput', title: 'Dateinput, DateInput2', wait: 400 },
125+
{ slug: 'datetimeinput', title: 'DateTimeInput', wait: 400 },
126+
{ slug: 'drilldown', title: 'Drilldown', wait: 300 },
127+
{ slug: 'filedrop', title: 'Filedrop' },
128+
{ slug: 'form-errors', title: 'Form errors', wait: 300 },
129+
{ slug: 'heading', title: 'Heading' },
130+
{ slug: 'img', title: 'Img', wait: 100 },
131+
{ slug: 'link', title: 'Link' },
132+
{
133+
slug: 'menu',
134+
title: 'Menu',
135+
wait: 300,
136+
a11y: false,
137+
a11ySkipReason: 'INSTUI-4677'
138+
},
139+
{ slug: 'options', title: 'Options' },
140+
{ slug: 'pagination', title: 'Pagination', wait: 400 },
141+
{ slug: 'progressbar', title: 'Progressbar' },
142+
{ slug: 'select', title: 'Select, SimpleSelect', wait: 300 },
143+
{ slug: 'table', title: 'Table' },
144+
{ slug: 'tabs', title: 'Tabs' },
145+
{ slug: 'tooltip', title: 'Tooltip', wait: 300 },
146+
{
147+
slug: 'treebrowser',
148+
title: 'TreeBrowser',
149+
wait: 1000,
150+
a11y: false,
151+
a11ySkipReason: 'axe color-contrast failures; animations'
152+
},
153+
{ slug: 'view', title: 'View' }
154+
]
155+
156+
const SCREENSHOT_OPTIONS = {
157+
capture: 'fullPage',
158+
overwrite: true,
159+
disableTimersAndAnimations: true
160+
} as const
284161

285-
it('View', () => {
286-
cy.visit('http://localhost:3000/view')
287-
cy.injectAxe()
288-
cy.checkA11y('.axe-test', axeOptions, terminalLog)
162+
describe('visual regression test', () => {
163+
PAGES.forEach(({ slug, title, wait, a11y = true }) => {
164+
it(title, () => {
165+
// Track a11y violations across all themes so a violation in one theme does
166+
// not abort the others (skipFailures below). We assert the total at the end
167+
// to keep a11y as a gate while still capturing every screenshot.
168+
let violationCount = 0
169+
170+
THEMES.forEach((theme) => {
171+
cy.visit(`${BASE_URL}/${slug}?theme=${theme}`)
172+
// Wait until the requested theme has actually been applied before doing
173+
// anything else (layout.tsx sets data-theme in an effect after mount).
174+
cy.get(`html[data-theme="${theme}"]`)
175+
if (wait) {
176+
cy.wait(wait)
177+
}
178+
179+
const name = `${slug}-${theme}`
180+
cy.task('recordMeta', { name, pagePath: `/${slug}` }, { log: false })
181+
// Wait until web fonts have finished loading before capturing. Otherwise
182+
// the screenshot can be taken mid-load, when text is still rendered in a
183+
// fallback font with different metrics — producing inconsistent, flaky
184+
// baselines.
185+
cy.document({ log: false }).then((doc) => doc.fonts.ready)
186+
// Screenshot BEFORE the a11y check so an a11y failure can never leave a
187+
// page without a visual baseline.
188+
cy.screenshot(name, SCREENSHOT_OPTIONS)
189+
190+
if (a11y) {
191+
cy.injectAxe()
192+
cy.checkA11y(
193+
'.axe-test',
194+
axeOptions,
195+
(violations) => {
196+
terminalLog(violations)
197+
violationCount += violations.length
198+
},
199+
// skipFailures: don't throw here — collect and assert once at the end
200+
true
201+
)
202+
}
203+
})
204+
205+
if (a11y) {
206+
cy.then(() => {
207+
expect(
208+
violationCount,
209+
'total a11y violations across themes'
210+
).to.equal(0)
211+
})
212+
}
213+
})
289214
})
290215
})

0 commit comments

Comments
 (0)