Skip to content

Commit f635ee3

Browse files
balzssclaude
andcommitted
ci: keep visual-regression baselines refreshing and surface Cypress failures
The baseline job ran Cypress without continue-on-error, so failing a11y/console assertions aborted the job before the "Push baselines" step — silently freezing the visual-baselines branch on every merge since the dark/light theme suite landed. Decouple baseline capture from the assertion gate: Cypress runs with continue-on-error, the screenshots always flatten and push (if: always()), and a trailing step fails the job for visibility if Cypress failed. Baselines now refresh on every merge regardless of the a11y/console outcome. Also add a "Cypress assertions" pass/fail line to the sticky PR comment so failures are visible even when the visual diff is clean, and remove the leftover TEMP diff-demo fixture step and regression-test/cypress/diff-demo/. Refs: INSTUI-5137 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 30c7251 commit f635ee3

5 files changed

Lines changed: 23 additions & 50 deletions

File tree

.github/workflows/visual-baselines.yml

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,15 @@ jobs:
4242
run: npx cypress install
4343
working-directory: regression-test
4444

45+
# Capturing baselines is decoupled from the a11y/console assertions: the
46+
# screenshots are valid renders regardless of whether those assertions
47+
# pass, so a failing test must never block the baseline refresh (that's
48+
# what silently froze the baselines branch before). We let Cypress fail
49+
# here, always publish the screenshots below, then re-surface the failure
50+
# in a trailing step so master still shows red and notifies.
4551
- name: Run Cypress
52+
id: cypress
53+
continue-on-error: true
4654
uses: cypress-io/github-action@v7
4755
env:
4856
ELECTRON_EXTRA_LAUNCH_ARGS: "--remote-debugging-port=9222"
@@ -53,11 +61,13 @@ jobs:
5361
working-directory: regression-test
5462

5563
- name: Flatten screenshots
64+
if: always()
5665
run: |
5766
mkdir -p baseline-publish/screenshots
5867
find regression-test/cypress/screenshots -name '*.png' -exec cp {} baseline-publish/screenshots/ \;
5968
6069
- name: Push baselines to visual-baselines branch
70+
if: always()
6171
run: |
6272
cd baseline-publish
6373
git init -b visual-baselines
@@ -67,3 +77,12 @@ jobs:
6777
git commit -m "update baselines from ${GITHUB_SHA}"
6878
git remote add origin "https://x-access-token:${{ secrets.GITHUB_TOKEN }}@github.com/${GITHUB_REPOSITORY}.git"
6979
git push --force origin visual-baselines
80+
81+
# Baselines have been refreshed by this point; now fail loudly if the
82+
# suite had regressions (a11y/console violations or a broken page) so the
83+
# failure is visible on master rather than silently swallowed.
84+
- name: Fail if Cypress reported failures
85+
if: steps.cypress.outcome == 'failure'
86+
run: |
87+
echo "::error::Baselines were published, but the regression suite had failing tests (a11y/console/spec). See the Run Cypress step logs."
88+
exit 1

.github/workflows/visual-regression.yml

Lines changed: 4 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -77,19 +77,6 @@ jobs:
7777
mkdir -p regression-test/.actual
7878
find regression-test/cypress/screenshots -name '*.png' -exec cp {} regression-test/.actual/ \;
7979
80-
# TEMP (remove before merge): this PR renames every screenshot with a
81-
# `-<theme>` suffix, so nothing matches the old baselines and the report
82-
# has zero "changed" rows — leaving the new diff visualization impossible
83-
# to preview. Inject one synthetic baseline/actual pair that differs in a
84-
# few localized spots so the report always has a "changed" example.
85-
# Fixtures live in regression-test/cypress/diff-demo/. Delete this step
86-
# and that directory to revert.
87-
- name: TEMP inject diff-demo fixture
88-
run: |
89-
mkdir -p regression-test/.baselines regression-test/.actual
90-
cp regression-test/cypress/diff-demo/baseline.png regression-test/.baselines/diff-demo.png
91-
cp regression-test/cypress/diff-demo/actual.png regression-test/.actual/diff-demo.png
92-
9380
- name: Diff and generate report
9481
id: diff
9582
working-directory: regression-test
@@ -173,7 +160,9 @@ jobs:
173160
message: |
174161
## Visual regression report
175162
176-
${{ steps.summary.outputs.status }}
163+
**Cypress assertions (a11y + console errors):** ${{ steps.cypress.outcome == 'failure' && format('❌ **Failing** — see the [run logs]({0}/{1}/actions/runs/{2}).', github.server_url, github.repository, github.run_id) || '✅ Passing' }}
164+
165+
**Visual diff:** ${{ steps.summary.outputs.status }}
177166
178167
| Status | Count |
179168
|---|---|
@@ -186,7 +175,7 @@ jobs:
186175
187176
${{ steps.summary.outputs.diff_images }}
188177
189-
<sub>Baselines come from the `visual-baselines` branch. They refresh on every merge to `master`.</sub>
178+
<sub>Baselines come from the `visual-baselines` branch. They refresh on every merge to `master`. The <b>Cypress assertions</b> line above covers a11y (axe) and console-error checks — a ❌ there means the suite found real issues even if the visual diff is clean.</sub>
190179
191180
- name: Fail if regressions or Cypress failed
192181
if: steps.diff.outcome == 'failure' || steps.cypress.outcome == 'failure'

regression-test/cypress/diff-demo/README.md

Lines changed: 0 additions & 35 deletions
This file was deleted.
-2.02 KB
Binary file not shown.
-2 KB
Binary file not shown.

0 commit comments

Comments
 (0)