[p5.js 2.0] Bring back a way to show the results of visual tests visually

[davepagurek]

Increasing access

In 1.x, when I first added visual tests, there was also a page you could go to where you could see what we were expecting and what the results are:

I think this view is really helpful in showing contributors how our testing works, and what is already covered by tests. We don't currently have this in 2.0.

Most appropriate sub-area of p5.js?

• Accessibility
• Color
• Core/Environment/Rendering
• Data
• DOM
• Events
• Image
• IO
• Math
• Typography
• Utilities
• WebGL
• Build process
• Unit testing
• Internationalization
• Friendly errors
• Other (specify if possible)

Feature request details

Our tests are now run in vitest, and within that testing framework, I have not (yet) found a way to do a custom HTML display of test results in order to show pictures. For now, we just output a base64-encoded image URL that you can paste into your browser. This works for failed test cases, but makes it hard to get an overview of the current state of tests.

Is there a way in vitest to make a custom report that could embed the expected and actual visual test results? Or maybe in github actions?

[ksen0]

If displaying visual test results in GH Actions is doable, I think that would be really exciting!

[Vaivaswat2244]

For displaying the results in GH actions, I was thinking of the following steps

• Making visual tests save expected vs. actual images to a consistent location with a structured naming pattern.
• Can create a script that builds an HTML page showing side-by-side comparisons of expected vs. actual images for each test case.
  And then while running the tests,
• Generate the HTML report using the script.
• Upload the report directory as an artifact.

For this I can update the current github workflow in ci-test.yml or I can create a new workflow for visual testing.

@davepagurek @ksen0 let me know your thoughts on this...

Now for viewing the results, we may have some options,
Firstly, a more brute force approach
After a GitHub Actions workflow completes, going to the workflow run page, from the summary tab we can download a zip file and then open the index.html file in the browser
This method works for every workflow run, including pull requests, but requires manual downloading and extraction. (might get tedious for new contributors)

Secondly, we can implement github pages deployment,
The report will be automatically published to https://[username].github.io/[repository]/
This URL is publicly accessible without downloading anything also can be automatically updated after each successful run on the main branch

[davepagurek]

That sounds good @Vaivaswat2244! I think if we save the actual results similar to how we save the expected results (but added to the .gitignore, since they don't need to be checked into source control), then we can collect them at the end of a test run and generate an HTML report.

Making a downloadable zip would be a good start since there might be some questions for a GH pages deploy that don't have easy answers, e.g. do we have to keep the pages live for every test run forever? So for now just having a downloadable file might work.

Another thought: we might be able to have a single .html file rather than a zip if, when generating the page, rather than using <img src="path/to/image" />, we load the image data and write it into a data url like <img src="data:image/png;base64,iVBORw0KG..." />. Then, all the image data is also contained in the HTML file.

[limzykenneth]

With Vitest, there is a preview window which for tests that draws to an onscreen canvas does show up with content so eventually we can look into utilizing it. Additionally, we may want to consider transitioning to use Vitest Image Snapshot which does more or less the same thing as we are while having a more unified framework for writing the test that keep track of visual regression.

[davepagurek]

I actually initially explored the jest-image-snapshot library mentioned in the docs, but found that it wasn't flexible enough to handle the sorts of diff checking that we need to deal with per-platform variations in things like text rendering. We could potentially still swap out the backend to use vitest snapshot saving under the hood though, combined with our current custom diffing.

Also, there will be a pr05 project this year trying to make a unified backend for visual testing in both Processing and p5.js, which likely would involve looking into integration with the p5 test infrastructure and whatever infrastructure Processing uses. Vitest integration could be a part of that, potentially. (EDIT: pr05 now, not GSoC)