Add automated browser-based tests to check examples on web editor for console errors

[nbogie]

Addresses #84

What is this?

Adds playwright-based config and tests to check the p5.sound examples in place on the web editor, checking they load and don't have any errors in the console.

How to run the tests (with test-runner UI)

This is recommended as it allows easier exploration of test failures:

1. once, to install playwright

npm install

2. launch the test runner's UI

npm run test:integration:ui

3. in the gui that opens, click "projects" and ensure "firefox" and "chromium" are checked. (see image)

4. Also in the gui, press the play button on the top test or above it. (see image)

Alternative: How to run the tests (unattended):

npm run test:integration 

This will present a html report when it finishes.

What are the additions and changes?

• Adds playwright as a dev dependency in package.json
  • (playwright also downloads browsers (chromium and firefox) but will not do so until the user runs the tests for the first time)
• Adds playwright.config.js, specifying, for example, where the tests are (./tests/) and which browsers to use.
• Adds tests/integration/test-examples-on-web-editor.spec.js This is the main test file that instructs playwright what web pages to load , what to click, and what to check for.
• Adds tests/integration/about-these-tests.md documenting the testing (and some problems faced in automated testing of the web editor).
• Adds the playwright-generated .github/workflows/playwright.yml.DISABLED - this is untested.

Limitations:

• It runs the tests against firefox and chromium but not safari (webkit) nor on any mobile browsers. Research is needed in how to have webkit granted microphone and webcam permissions and to allow audio context start without user interaction.
• Because I've only written ONE standardised test to exercise ALL p5.sound examples, certain conditions have to be met for an example to be compatible:
  • Example must render a canvas.
  • Example must not require user permissions beyond microphone and camera.
• No CI setup has been tested. It is envisaged this tool will be plenty useful when used on an ad-hoc basis. Certainly once that proves true it would be worth hooking up the CI. The playwright-generated workflow for github actions has been placed in the repo but disabled: .github/workflows/playwright.yml.DISABLED

AI usage disclosure

Claude Code did most of the implementation on this, including research into chrome and firefox support for pre-emptive grants of microphone + video, troubleshooting the main issue with testing the web editor, and the bulk of the write-up about-these-tests.md. That said, I've reworked the code to how I want it, I have read and understood each line, and the PR write-up is all mine.

[nbogie]

update: silenced the headless firefox test run to match chrome.

[ksen0]

Minor extra notes from trying this with @ogbabydiesal :

1. Fresh install will require npx playwright install after npm install as well to get the browser drivers
2. Tommy got 25 passed and 19 failed (yours was 29 and 15) which is a bit surprising, however

As next steps Tommy will work on creating an examples directory in the repository, which should be used instead. Would that require much of a change to this PR?

Thanks so much for getting this started!

[nbogie]

I am happy to alter it to work from examples/ this weekend. It should generally be a simplification, I think.

(copied from my discord response:)

If the examples are going to continue to be published on the web editor (I hope so - I love the availability and the subsequent discoverability in the collection), then I'd probably argue mildly that we might try to have the tests be capable of running against sketches in both locations, assuming it doesn't add a lot of maintenance.

In the end the actual test process doesn't change wherever they are, only some implementation details for each step.

1. access the sketch
2. start sketch if necessary
3. interact minimally with the canvas
4. wait a second
5. check no errors in console

Pros of running the automated tests against the editor:

• Tests what the large majority of users will actually be interacting with
• Ensures, after a library or example change, that all examples have been successfully uploaded / updated / created, including assets
  • unless the publication step from examples/ is automated these transfers/updates will be a significant source of errors

Cons of running the automated tests against the editor:

• Changes to the web editor can break the tests
• Testing framework is slightly more complex, slower, and less obvious ("why does the test wait here before pressing play?")
  • maintenance, troubleshooting, and other contribution is therefore harder
• Test machine needs to be online to run
• (Hypothetical) eventual CI run of the tests would be flakier due to web editor dependency (outages, quirks, weather)
  • if we engineered the tests to be able to run in both locations, I'd suggest CI runs them only locally

[ogbabydiesal]

hey @nbogie I created a repo with collection in the examples folder, there's only one for now but this would be the folder hierarchy if you'd like to test with that. I'll add the rest soon

https://github.com/processing/p5.sound.js.git

I hear you about running the tests in the editor as that is how many people will interact with them first, if your into the idea of running them both places, and it isn't too much work you can use that repo!

[ksen0]

@ogbabydiesal not sure if that's the intended link but can you make the examples dir be inthis repo please, like in p5.sound.js? Totally ok if there's just 1 example! Thanks so much both

[nbogie]

Hi @ksen0 @ogbabydiesal
In case it's helpful i've made a PR with all the examples downloaded verbatim into examples/ and a tracking issue.

(Sorry if this work is already underway elsewhere).

[nbogie]

@ksen0 @ogbabydiesal I've updated this PR to also test the local examples/ that are now in main.

AI use disclosure: rewrite, and about-these-tests mostly done by claude code.

(about-these-tests is a bit technical still, i know, but if you find no issues I'd like to merge this, as a starting point)

To run the browser-based tests locally we need a little web server to serve the examples - I added one as a dependency. (Or we could write a small from node.js primitives.)

[nbogie]

Fresh install will require npx playwright install after npm install as well to get the browser drivers

Thanks, I've now addressed this in about-these-tests.md

I think playwright tells you what to do when you don't have the necessary browser(s) installed, so if someone doesn't read that document, it's not a total loss.

(I'm wary of putting npx playwright install as a postinstall action which would automatically trigger after each npm install.)

[ogbabydiesal]

just did a test and see that many of my firefox tests fail but when testing these examples in the browser there are no errors. for example, this test fails for firefox:

https://editor.p5js.org/thomasjohnmartinez/sketches/z-KkeTrcu

but you will see that there are no errors when running the sketch. (i will note that I can cause an error by going out of bounds with one of the parameters but wouldn't this cause the test to fail in chrome as well? )

let me know what you think!

UPDATE: I ran the test again one by one, and that test passed so it might be that the tests are inconsistent? It did help me find a bug in example 4 though (the most recent library adds a feature that is not in the release yet, the example takes advantage of this feature but is broken until a new version is pushed).

[nbogie]

Thanks @ogbabydiesal . I didn't look into the firefox failures. I get the same result as you when running that example sketch manually (no error).

Some possibilities:

• By default playwright runs four browsers in parallel. I'm surprised if firefox doesn't like this, but we can reduce them to run only one at a time. (As suggested by your discovered fix). Update: tried this - no difference.
• Playwright might be using a different firefox than we are using (I don't think this is the cause).

I suggest we merge the tests into main (even with these firefox issues) so they're more easily accessible for others to work on, and so that the (fairly healthy) chromium tests (for example) can be used to provide confidence to rapid changes to the examples and library itself.

[nbogie]

Hi @ogbabydiesal
I tried having playwright run the firefox (local) tests one by one rather than in parallel. It made no difference - 7 fails: examples 4, 5, 6, 6, 10, 11, 18, pt-to-tone.

I ran the test again one by one and that test passed

Are you saying that ALL examples passed their tests when the tests are run individually in the playwright UI? They still fail for me when I run them one by one.

Also, can you be specific as to whether you're talking about the local or web-editor examples?

I'll investigate a bit further

[nbogie]

Here's a video of an intermittent failure on firefox of example 005 in the web editor
https://www.loom.com/share/a06ff5a09fb34f44b3743ff08f53dd67