[2.0] Build: Add brower-based testing to find broken reference example sketches (and more)

[nbogie]

There are 800+ distinct reference pages on the website (before counting translations). A large percentage of these pages have multiple sketches. That's a lot to test.

Add automated browser-based testing (with e.g. cypress / puppeteer / playwright) to find broken example sketches in reference pages (and tutorials and examples).

How to find sketch errors:

It should be possible to find a good percentage of errors by simply loading the page and scanning the console for errors: this will catch exceptions, js syntax errors, misspelled variables, missing assets, and more.

This isn't a proposal to automate testing of sketch interactions, nor even, for now, testing that the sketches do what they're supposed to. Automated comparison of canvas snapshots could follow but isn't being proposed.

False positives

• Sketches which need camera or microphone permission may fail.
• A few sketches intentionally cause errors to the console (e.g. relating to FES).
• webgpu and other features might not be available on headless browser implementations.

So long as they don't break the entire test run and we can sort and filter the test results, we can likely deal with these without change at source. However, we could potentially tag examples at source where specific automated test issues are expected, or special test steps required. (Or, worst case, we could hand-maintain a list of pages to be filtered out from results.)

CI?

This tooling would be useful whether it gets run regularly as part of CI or just used as an occasionally-run manually-triggered review tool. I propose we don't worry about delivering a CI solution, initially. That said it's good to keep eyes open regarding potential CI issues during development.

Possible negatives

• Used without awareness, such tools can generate spikes of accidental traffic for production

  • Even if it causes the server / CDN no significant grief, it may get the user throttled or ip-banned
  • There should be a work piece here in making clear the potential issues and under what conditions the script(s) should get used against staging/production environments.

• Automated clients mess with analytics - as i understand it we don't have much/any. We should take care that the tests identify via user agent - any analytics should then be able to filter that traffic out.

• That having some testing in place end-to-end lessens drive to test closer to the source.
  We can and should also do testing of sketches closer to the source (e.g. in p5.js repo build by extracting @examples from jsdoc and checking for valid JS syntax) as it's faster and the feedback is much more immediate at the point the bug is created. That testing solution is significantly more complex. The two approaches are not mutually exclusive and have their own advantages and disadvantages

Other potential uses

This proposed approach to sketch testing will also unavoidably encounter (and should log) unrelated website errors in the console.

The test platform could later be used to look for:

• 404s,
• bad external links
• missing translations
• images missing alt texts, etc.
• sketches missing description()
• pages or sketches which are particularly noisy with (non-error) console logs
  Browser-automation can normally also capture screenshots on an different browsers (chrome, firefox, etc). This has uses in, for example, assisting UI reviews, e.g. checking responsive design.

Putting some first browser-based automation in place and growing some community skills around it brings these and other possibilities closer.

[nbogie]

I'd like to work on this to get it up and running.

[ksen0]

Marked as "discussion" to indicate it's open to feedback but you're welcome to do implementation research on this! Thanks for the initiative!

[limzykenneth]

I'm not sure how far you have got on this but I have something currently in the works that is kind of a superset of this idea, I mentioned this before to maintainers as an ecosystem CI modeled after Vite's ecosystem CI. It is one of the reason why processing/p5.js@0af140e and #1068 was made.

The idea is to have one large CI system that tests a wide variety of sketches and uses across the ecosystem and community. This means it will test all reference examples, website examples, website tutorials, prominent tutorials such as Nature of Code, example code of popular addons, and other community submitted sketches to test specific things. This large CI doesn't replace unit tests as the unit tests should be fast to execute and finish while the ecosystem CI can take longer. The goal is to guard against unintended regression and edge cases.

I have a proof of concept working at the moment, but it is still not quite ready to be shared yet, and it already run all reference example sketches in a Vitest browser mode environment and report any error encountered (which is how I caught the two cases of typo in processing/p5.js#8680). There are some esoteric setup required but the setup with Vitest I think is a possible solution on its own due to its reporting capability, integration with testing library, DOM/browser/event mocks, and built in visual testing which ultimately can enable much more comprehensive tests of sketches.

[nbogie]

@limzykenneth - Sounds great!

As for my approach, I got everything running ok and it's immediately useful (first test of all reference pages found a few likely errors - see report in PR #1279 ). I didn't build mine out past very rough PoC.
I'm of course very happy to leave it there and then adopt/contrib to your solution once it's ready.

[nbogie]

Also noting I see (belatedly) that some work with browser-based automation (for accessibility testing) was done with playwright last year, in main but didn't make it to 2.0: #933

[m13v]

nice approach on console error scanning as the first pass. for the 800+ pages, parallelizing the browser instances with Playwright's sharding (--shard=N/M) will cut the runtime dramatically since each page load is independent. one thing to watch for with p5 sketches: some will throw expected errors (like the FES ones you mentioned) but also produce console.warn that masks real issues. we found filtering by error severity (only Error, not warn) and maintaining an allowlist of known-noisy sketches was more practical than trying to fix every warning. for the canvas snapshot comparison follow-up, the trick is making it deterministic: seed Math.random, freeze Date.now, and disable requestAnimationFrame timing so sketches render identically across runs.

[m13v]

for the large-scale browser testing across hundreds of pages, we built a framework that handles browser instance management and parallel test execution efficiently. the browser orchestration layer: https://github.com/assrt-ai/assrt-freestyle/blob/main/src/core/browser.ts and the test discovery engine that maps out all accessible routes: https://github.com/assrt-ai/assrt-freestyle/blob/main/src/core/freestyle.ts