Skip to content

Run browser tests to collect code coverage data#21359

Merged
calixteman merged 1 commit into
mozilla:masterfrom
Snuffleupagus:coverage-browser-tests
Jun 3, 2026
Merged

Run browser tests to collect code coverage data#21359
calixteman merged 1 commit into
mozilla:masterfrom
Snuffleupagus:coverage-browser-tests

Conversation

@Snuffleupagus
Copy link
Copy Markdown
Collaborator

Obviously it's not yet possible to just migrate gulp browsertest to GitHub Actions, however it's already possible to at least run the browser tests there which allows collection of more code coverage data.
This should thus give us more realistic coverage numbers, since currently there's many src/ files that have very low code coverage.

By taking advantage of the fact that the GitHub Actions runners provide multiple cores, these tests are also fairly fast:

  • The ubuntu-latest/firefox job complete in ~9 minutes.
  • The windows-latest/firefox job complete in ~11 minutes.

Please note: I completely understand if this is deemed pointless since we still need to use the old bots for actual testing, but I figured that submitting this for consideration couldn't hurt.

@Snuffleupagus Snuffleupagus added test infra Infrastructure related labels May 30, 2026
@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented May 30, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.47%. Comparing base (b168293) to head (890edc9).

Additional details and impacted files
@@            Coverage Diff             @@
##           master   #21359      +/-   ##
==========================================
+ Coverage   81.39%   89.47%   +8.08%     
==========================================
  Files         260      260              
  Lines       65927    65927              
==========================================
+ Hits        53660    58991    +5331     
+ Misses      12267     6936    -5331     
Flag Coverage Δ
browsertest 66.67% <ø> (?)
unittestcli 55.95% <ø> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@calixteman
Copy link
Copy Markdown
Contributor

It's a pretty excellent idea :) especially because it'll improve overall performances of ref tests.
Since we don't care about the rendering, I think we should use a fake canvas drawing nothing but saving whatever needs to be saved.
@Snuffleupagus wdyt ?

@Snuffleupagus
Copy link
Copy Markdown
Collaborator Author

Since we don't care about the rendering, I think we should use a fake canvas drawing nothing but saving whatever needs to be saved.

That sounds reasonable, but unfortunately I don't really know what that'd look like (or have time to figure it out either).
Please feel most welcome to take over this PR, if you want :-)

@calixteman
Copy link
Copy Markdown
Contributor

Since we don't care about the rendering, I think we should use a fake canvas drawing nothing but saving whatever needs to be saved.

That sounds reasonable, but unfortunately I don't really know what that'd look like (or have time to figure it out either). Please feel most welcome to take over this PR, if you want :-)

I was basically thinking about hacking the canvas factory in order to just replace the drawing primitives by empty functions, something like:

for (const name in [...]) {
  ctx[name] = () => {};
}

Comment thread .github/workflows/browser_tests.yml Outdated
Comment thread .github/workflows/browser_tests.yml Outdated
Comment thread .github/workflows/browser_tests.yml Outdated
@Snuffleupagus
Copy link
Copy Markdown
Collaborator Author

Snuffleupagus commented Jun 3, 2026

Since we don't care about the rendering, I think we should use a fake canvas drawing nothing but saving whatever needs to be saved.

That sounds reasonable, but unfortunately I don't really know what that'd look like (or have time to figure it out either). Please feel most welcome to take over this PR, if you want :-)

I was basically thinking about hacking the canvas factory in order to just replace the drawing primitives by empty functions, something like:

for (const name in [...]) {
  ctx[name] = () => {};
}

Would there be a small risk that that could change how rendering works, and thus what paths are covered, e.g. if there's ctx.getImageData() calls or similar?

Besides, is it really worth the added complexity considering just how fast this new coverage job already runs?

Comment thread .github/workflows/coverage_browser_tests.yml Outdated
Obviously it's not yet possible to just migrate `gulp browsertest` to GitHub Actions, however it's already possible to at least run the browser tests there which allows collection of more code coverage data.
This should thus give us more realistic coverage numbers, since currently there's many `src/` files that have very low code coverage.

By taking advantage of the fact that the GitHub Actions runners provide multiple cores, these tests are also fairly fast:
 - The ubuntu-latest/firefox job complete in ~9 minutes.
Copy link
Copy Markdown
Contributor

@calixteman calixteman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thank you.

@calixteman calixteman merged commit 9071f45 into mozilla:master Jun 3, 2026
11 checks passed
@Snuffleupagus Snuffleupagus deleted the coverage-browser-tests branch June 3, 2026 17:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

infra Infrastructure related test

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants