Should `pytest_xdist_auto_num_workers` respect `-X cpu_count`/`PYTHON_CPU_COUNT` on Python 3.13+?

[the-13th-letter]

Background

Python 3.13 introduced new functionality to address the case "how many (logical) CPU cores can this Python program use" via the os.process_cpu_count function. In particular, it also introduced the related -X cpu_count option and PYTHON_CPU_COUNT environment variable, which can be used to override the auto-detected value, thus serving a similar purpose to the PYTEST_XDIST_AUTO_NUM_WORKERS environment variable. These two override mechanisms were retrofitted to the os.cpu_count function as well (the variant that does not take CPU affinity into account).

The issue

pytest-xdist does not explicitly respect these standard Python override mechanisms in its pytest_xdist_auto_num_workers hook. Assuming that PYTEST_XDIST_AUTO_NUM_WORKERS is unset:

• If psutil is available, then pytest-xdist will dismiss -X cpu_count/PYTHON_CPU_COUNT, because psutil uses C code to calculate the true logical process count available to this Python process.¹
• Otherwise, os.sched_getaffinity will be queried, which also dismisses -X cpu_count/PYTHON_CPU_COUNT.

The question

Should these override mechanisms be respected?

• I would argue yes, because they clearly state intent (beyond just the pytest-xdist interaction) for Python to lie claim to the script that only so-and-so many CPU cores are available.
• However, the code was written with pre-3.13 semantics in mind, and still runs on pre-3.13 systems, so retrofitting these overrides there is a compatibility break, and should perhaps not be taken lightly.
• Or perhaps the correct choice here is to go version-specific, i.e., to only respect the overrides on 3.13 and higher.

Going forward

If the override mechanisms are to be respected (whether in general, or on 3.13 and higher only), I would be willing to write a PR.

Footnotes

1. It makes sense that psutil – which aims to query and report true system properties – would not respect these Python-specific overrides. ↩

[RonnyPfannschmidt]

i was not aware of this addition - -n auto should take it into account

[the-13th-letter]

@RonnyPfannschmidt, I reiterate my offer to write a PR. Which variant do the pytest devs prefer? 3.13+ only, or all Pythons?

[RonnyPfannschmidt]

If its not avaliable pre 3.13 it makes no sense pre 3.13

[the-13th-letter]

os.process_cpu_count is not available pre 3.13, yes. But I'm asking specifically if you would like me to backport the -X cpu_count/PYTHON_CPU_COUNT support itself back to pre 3.13 as well? I feel that we can be consistent with stock Python behavior (i.e., support the overrides only on 3.13 and later, because that was when they were introduced) or consistent as a feature (i.e., pytest-xdist supports -X cpu_count/PYTHON_CPU_COUNT, no matter the Python version, similar to how PYTEST_XDIST_AUTO_NUM_WORKERS doesn't depend on the Python version).

I would think the latter is preferable, from the perspective of someone testing some Python software against multiple Python versions, and where your tox¹ configuration has identical command-lines for all pytest calls in the testing environments. Hence my asking if this is worth implementing manually for pre 3.13 too.

Footnotes

1. …or, in my case, hatch, but the argument is the same. ↩

[RonnyPfannschmidt]

i did some research - and based on the cpython spec

its sensible to support this on at least cpython in general, perhaps pypy as well if compatible

[the-13th-letter]

Okay. I'll draft a PR this weekend. I've taken a quick glance at the tests, and they're fiddly enough that I would want to spend a larger block of time at once on them to make sure I'm adapting them properly, instead of rushing them.