test: refactor CI

[2bndy5]

per request from #1432 (comment)

closes #1366

Changes

• remove all test jobs from test.yml workflow except the one that runs on s390x platform.

• rename the test.yml Display name accordingly: "Tests (s390x)"

• skip building ppc64le wheels when not triggered on master branch.

• reconfigure wheels.yml workflow triggers. The wheels.yml CI now runs for

  1. any push to master branch
  2. any change in a PR that targets master branch (excluding when PR changes only affect docs/ path)
  3. any tag (starting with "v") is pushed

• auto-cancel wheels.yml CI run if a new run is triggered (does not cancel runs on master branch).

• added skip-existing param in case deployment to PyPI suffers a network error and the CI just needs to be re-run.

• cherry-pick changes from ci: change triggers to prevent dual runs on prs #1366 (about CI triggers for lint and spell-check CI workflows)

• use pytest.ini in cibuildwheel isolated env

• added step to job that deploys wheels to PyPI. This new step creates a GitHub Release for the tag that was pushed. Uses release notes extracted from CHANGELOG.md (during sdist job)

[2bndy5]

I noticed that the tests take about 20 seconds for each wheel. Here are some tips to conditionally reduce the number of tests executed:

Using pytest markers

If you want to exclude certain tests where the platform architecture should not affect the outcome, then you can use pytest.mark:

@pytest.mark.platform_agnostic
def test_something() -> None: ...

# pytest.ini
markers = 
  platform_agnostic: test that is not affected by platform architecture

# pyproject.toml
[[tool.cibuildwheel.override]]
select = "cp3{11,12,13,14,14t}-{macosx,win,linux}-*"
# skip platform-agnostic tests for all wheels built on
#   macos, windows, and linux using
#   any python version 3.11-3.14(t)
# NOTE: assumes python 3.10 is minimum supported version.
test-command = "pytest -m 'no platform_agnostic'"

Using pytest.skip_if

For tests that are agnostic of python version, you can use pytest.skip_if:

@pytest.skip_if(
    sys.version > (3, 10),
    reason="only tested on min supported python ver",
)
def test_something() -> None: ...

[jdavid]

LGTM. There is just the changelog generation to be removed, unless you think it can be generated from the CHANGELOG.md file, with --notes maybe?

[2bndy5]

unless you think it can be generated from the CHANGELOG.md file, with --notes maybe?

The thought did occur, but it wouldn't be a simple solution. I'd have to parse the changelog file to get only the recently added entry/section. I think that would require a few lines of python code.

Alternatively, (without --generate-notes), the release description would be blank. Instead of blank, I could simply link to the CHANGELOG.md file (a static line of text).

[2bndy5]

I added .github/workflows/parse_release_notes.py script to get the release's title and notes from the CHANGELOG.md file. When a tag is pushed, the version is extracted from the tag and used to identify the CHANGELOG entry.

This means that each CHANGELOG entry must start with the following pattern:

# <major>.<minor>.<patch>

This conforms to the current pattern already used in the CHANGELOG.md file, but the new py script might behave unexpectedly if this pattern is not observed in the future.

[jdavid]

Awesome, thanks!