Experimental: Add playwright and test to find broken sketches (console errors)

[nbogie]

This draft PR is an experimental proof-of-concept.

What's it for?

• Find broken example sketches
• Maybe also find bad links (later)

To find broken example sketches in reference pages we can look for errors in the console as we go through each page.

This PR adds playright as a dependency (browser automation) and a playwright test to automatically load the reference pages, scroll through them to trigger loading of their content (including their sketches), and monitor the console for errors. It reports which URLs had errors in the console, and what the errors were.

It is a VERY rough WIP

running the test

To run all tests, headed:

npx playwright test --project=chromium

The following two headless versions are not working correctly with webgl yet (Error creating webgl context):

run all tests, headless
npx playwright test --project=chromium

or with UI test runner
npx playwright test --project=chromium --ui

more discussion

early results:

• good
• prefetching urls on dev missing trailing slashes throws up lots of misleading 404s. (we should anyway disable prefetch, I think - it's extra traffic we don't need)
• lots of "Error creating webgl context" at createCanvas when running headless chromium (that includes with the UI which is really headless, too).
• an unanticipated find: JumpToLinks pane has react bug with non-unique key names (e.g. Image).

Following component has two or more children with the same key attribute: "createFramebuffer()". This may cause glitches and misbehavior in rendering process. Component: <ul>..</ul>  in ul (at /Users/neill/Developer/projects/p5.js-website/src/components/Nav/JumpToLinks.tsx:35)
  in JumpToLinks (at /Users/neill/Developer/projects/p5.js-website/src/components/Nav/NavPanels.tsx:94)
  in NavPanels

Various todos listed at the top of the prototype test.

This should not be one test but one per reference page, so that we can see progress as we go, and so we can use the reporting tools to filter data, etc.

The list of url paths to check should probably come from astro build's list of static routes. or perhaps be generated from /docs/reference/data.json.

There'll be a bunch of false positives - those sketches requiring camera, mic access. That's fine, this may be a quick way to list them.

Doesn't yet correctly report 404s.

If you start it in vscode from the playwright extension test runner, you'll get the camera access popup in vscode. the test runner won't block on this yet when the first(?) relevant test is encountered. Same for mic and other resources, I guess?

The scrolling is janky - was a quick stackoverflow hack. to be improved.
It's intentionally a little slow.
Playwright has a frames api so we ought to be able to detect the iframes and only wait until they're all loaded + some short grace period.
That grace period will have to be longer for those sketches that need to do shader compilation, or download a big resource like ML weights. We can't tell which those are from URL path. Perhaps we could tag example with meta-data.

Running against production: recommend we don't, yet. even at 1 page per 2 seconds, each page is very large and has a lot of requests. Further, I think the browser's prefetching will amplify that a LOT. Besides spamming the server being bad manners, Cloudflare might block your ip.

The github actions workflow is the default one playwright generated - I haven't looked at it except to disable it. It's a quick way to accidentally burn CI minutes, if not well configured.

This test eventually will also naturally catch and report basic 404s where a requested reference page is not found/served at expected URL. We can't tell if it's been removed/renamed or our config has a bad url.

(Playwright could also eventually be used to recursively crawl and check for bad links in all pages (though if it's all static content, wget could work, too)).

Some discussion on discord here:
https://discord.com/channels/836700474425475088/1483526487561867499

[nbogie]

Just for fun, here are some detected page errors that might be interesting, from a first full run against all reference pages on localhost website (2.0 branch) after a fairly recent custom:dev.

interesting-page-errors.json

I took out all the ones where jumpToLinks was complaining of duplicate keys, but there are still in here going to be a lot of issues with microphone, camera, autoplay non-interaction.

Nice that it already catches some seemingly legit errors. Here's a sampling:

• https://beta.p5js.org/reference/p5.SoundFile/frames/
• https://beta.p5js.org/reference/p5.Table/matchRows/
• https://beta.p5js.org/reference/p5.XML/addChild/
• https://beta.p5js.org/reference/p5.XML/setAttribute/
• https://beta.p5js.org/reference/p5/disableFriendlyErrors/ - p5 is not defined?
• https://beta.p5js.org/reference/p5/fontAscent/ - fdesc is not defined
• https://beta.p5js.org/reference/p5/registerAddon/
• https://beta.p5js.org/reference/p5/saveObj/ - Unexpected token ')'
• https://beta.p5js.org/reference/p5/createAudio/ - noCanvas() + describe() don't combine

Also, I didn't really run this long test against production, just against local. Afterwards, I changed the urls in the result data to point at prodn (beta) to make it much easier for the curious to click on the links!
(reminder to scroll through the page to ensure any sketches come into viewport to load and start and error)