Make image reference tests work on CI

[dawbarton]

Add functionality to download a tarball from a GitHub release and compare against the current branch.

Currently downloads a test tarball from issue #145. The generate.jl script will generate the relevant tarball to upload as a release.

• Change url of tarball to be a release URL.
• Hook this up to the CI

When merged this fixes #145

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 74.03%. Comparing base (819bbec) to head (5944f1a).
⚠️ Report is 40 commits behind head on master.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #146      +/-   ##
==========================================
- Coverage   78.39%   74.03%   -4.36%     
==========================================
  Files          10       10              
  Lines         597      647      +50     
==========================================
+ Hits          468      479      +11     
- Misses        129      168      +39     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[dawbarton]

@Kolaru - after a bit of messing around this should all be working now. The workflow is

1. It downloads the latest set of reference images from GitHub releases (URL needs changing - see below).
2. It compares against the current branch - the comparison is pixel perfect at the moment. (I think that needs softening a bit to account for small changes in anti-aliasing.)
3. Any differences that are found are uploaded as an artifact for further inspection. See this example.

All you need to do is make the reference images using the generate.jl script, upload as a release, and adjust the URL in compare.jl. (Cf. Makie's download url)

[dawbarton]

Not sure why coverage has gone down slightly. I have got it uploading the coverage files from the reference tests as well. I suspect that I don't understand how the coverage test works; the coverage test fails before the reference tests have finished, suggesting that the coverage from that CI run doesn't get used.

[Kolaru]

Thanks a lot, this is stellar!

If I understand correctly, you made a separate GH action so that we can examine the images if the reference tests fail, right?

Now I just have to figure out the release: I'm thinking of tagging the main branch after this is merged, and patch the URL immediatly afterwards.

I'll review the PR in more depth later this week/this weekend.

[dawbarton]

At the moment it's separate just because it was the easiest way to check that it works. It could be combined with the existing unit tests, which would then have the advantage that it only runs if the unit tests pass. (It could easily consume quite a bit of CI time since Makie isn't the fastest to precompile.) If you want, I can make that change later in the week.

That said, the unit tests run on more OSes than the reference tests do, and so putting them in the same action will consume more CI time regardless.

I don't think it matters whether you patch the URL before or after the release. Though after is going to be easier since you don't have to guess at what the URL format should be.

[Kolaru]

Sorry for the long time coming back to this.

I think I figured out everything (I just already released the refimages), and everything should work once merged.

Thanks so much for your help, this is stellar!