💡 [REQUEST] - Add visual regression testing with Playwright screenshots

[righ]

Summary

Layout regressions like unwanted scrollbar appearance, element clipping, or overlay misalignment cannot be detected by interaction-based e2e tests alone. In the 3.0.4 cycle, the scrollbar regression at sheet edges was only found through manual testing. Introduce Visual Regression Testing (VRT) using Playwright's built-in toHaveScreenshot() to catch these automatically.

Reference Issues

No response

Basic Example

Append screenshot assertions to existing e2e tests to avoid additional URL accesses:

// At the end of an existing test
await expect(page.locator('.gs-tabular')).toHaveScreenshot();

Priority targets:

• Sheets where content fits exactly (scrollbar edge cases)
• Cell selection at right/bottom edges
• Overlay positioning after CSS resize
• Autofill handle visibility

CI environment should be fixed (e.g., Docker / GitHub Actions Linux) for consistent scrollbar rendering and font baselines.

Drawbacks

• Environment dependency — Screenshot baselines are OS/DPR-specific. CI must use a fixed environment, and local runs may not match.
• Baseline maintenance — Intentional visual changes require updating committed baseline images.
• Threshold tuning — Anti-aliasing differences may cause flakiness; a small maxDiffPixelRatio tolerance may be needed.

Unresolved questions

• Which CI environment to standardize on (Docker image, browser version)?
• How much tolerance (maxDiffPixelRatio) is appropriate to avoid false positives?
• Should baselines live in the repo or be stored as CI artifacts?

[m13v]

the scrollbar regression you mention is the perfect example of why VRT matters. we went through a similar thing where interaction tests all passed but users reported visual glitches. couple things we learned the hard way: lock down the docker image for CI screenshots, even minor font rendering differences between ubuntu versions will give you false positives constantly. also for the overlay positioning tests, you'll want to inject CSS to freeze any transitions before capturing, otherwise you get intermittent failures from mid-animation captures. the toHaveScreenshot() threshold default is generous enough for most cases but you might want to tighten it for the scrollbar edge specifically.

[m13v]

we handle visual regression alongside test generation in an open source framework. it takes baseline screenshots during discovery and diffs them on each run with configurable thresholds per component. the test runner that manages browser lifecycle and screenshot comparison: https://github.com/assrt-ai/assrt-sdk

[righ]

Thanks, this is super helpful — especially the points about font rendering differences and freezing transitions.

We were already seeing some flakiness concerns, so locking down the Docker image in CI makes a lot of sense. We'll standardize the environment (OS + browser) to keep baselines stable.

Good call on injecting CSS to disable transitions as well — we hadn’t considered mid-animation captures causing noise, but that explains some of the intermittent issues we were anticipating.

For thresholds, we’ll probably start with the default and tighten it selectively (e.g. scrollbar edge cases as you suggested).

By the way, how are you managing baseline images in practice — committing them to the repo, storing them as CI artifacts, or using external storage?

[m13v]

Glad those tips are useful! You're on the right track with all of it. The CSS transition disabling is one of those things that seems minor but eliminates a whole category of flaky failures.

For the threshold tuning, start conservative (0.1% pixel diff) and loosen only when you get false positives on specific components. Different parts of the UI have different tolerance needs. Data grids with lots of text tend to need slightly more tolerance than simple layout components. You can also set per-test thresholds if the global one doesn't work for everything.

[righ]

Got it — that makes sense. I think we have enough to move forward.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.