Add multi-browser automated testing in CI pipeline

[saismrutiranjan18]

The Web Monetization extension supports multiple browsers (Chrome, Firefox, Safari, Edge), but there's no automated cross-browser compatibility testing in the CI/CD pipeline.

• Manual testing across browsers is time-consuming and error-prone
• Browser-specific bugs may slip through to production
• No guarantee that features work consistently across all supported browsers
• Issue Look into automated testing for Safari extension #1080 mentions Safari testing, but we need comprehensive multi-browser testing

[sidvishnoi]

There's an end to end testing with playwright, which tests extension on Chromium (and Chrome and Edge). Playwright doesn't support Firefox extensions last I tried. Safari extensions are no go for sure.

Do you have alternate strategies? We were exploring a pipeline based on the Selenium Web driver, but the effort was too high and our priorities were shifted towards developing. The existing Chrome e2e tests gave us a good enough confidence, but definitely looking into a more general strategy that works cross platform and cross browser.

[saismrutiranjan18]

There's an end to end testing with playwright, which tests extension on Chromium (and Chrome and Edge). Playwright doesn't support Firefox extensions last I tried. Safari extensions are no go for sure.

Do you have alternate strategies? We were exploring a pipeline based on the Selenium Web driver, but the effort was too high and our priorities were shifted towards developing. The existing Chrome e2e tests gave us a good enough confidence, but definitely looking into a more general strategy that works cross platform and cross browser.

Thanks for the detailed response @sidvishnoi!

I understand the Playwright limitations for Firefox and Safari. Your phased approach makes sense.

I'm interested in helping with this! Here's what I can contribute:

Short-term (I can start with):

1. I'll research Selenium WebDriver for Firefox extension testing and create a POC
2. Document my findings - feasibility, test execution time, maintenance overhead
3. If it works well, we can decide whether to maintain parallel frameworks or find a unified solution

Questions before I start:

• Should I create a new branch for the Selenium POC or work in a fork?
• Are there specific test scenarios you'd like me to prioritize for Firefox?
• Do we have any existing Firefox-specific bugs that I should use as test cases?

I agree the Chromium matrix (Chrome + Edge + Brave) is a quick win. Should we tackle that first, or can I explore both in parallel?

Also, for the Chromium matrix, I can help set that up if you share the current Playwright test structure. I have experience with GitHub Actions matrix strategies.

Let me know how you'd like to proceed, and I'll get started!

[sidvishnoi]

I'll research Selenium WebDriver for Firefox extension testing and create a POC

I'd suggest looking into it in a cross-browser way, as we don't want to have different test setups for each browser separately if possible.

Should I create a new branch for the Selenium POC or work in a fork?

Fork would be preferred at this stage. You can send a PR he

Are there specific test scenarios you'd like me to prioritize for Firefox?

We've tests in tests/e2e directory. In particular, making connect.spec.ts (or equivalent file in new approach) would validate most of the things.

Do we have any existing Firefox-specific bugs that I should use as test cases?

Not at this stage. Let's first look into a working POC that tests can work.

[...] for the Chromium matrix, [...]

We already have a Chromium matrix in the nighty-build GitHub Actions workflow.

---

Ideally, we should have a single way to write tests. Like with Playwright, we get same tests for Chromium browsers - don't have to write them in a tool specific manner (i.e. we don't want to duplicate tests for Selenium & Playwright, just written differently).

Going through docs/TESTING.md would be helpful to understand the current testing setup.