FEAT: support running coverage measurements through `coverage.py` directly, instead of via `pytest-cov`

[neutrinoceros]

I discovered this while helping on #207, but the issue I want to report is actually quite orthogonal to that PR, so I'm opening this ticket separately.

In short, I would like to make pytest-cov a choice rather than a hard requirement for coverage reporting; this plugin is merely more than a middle man and provides no actual functionality on its own, so I would like to eliminate it from dependency graphs to reduce the "attack surface" (not exclusively, but also in the security sense) for dependees.

Running coverage directly isn't really challenging in itself, it should be as simple as the following substitution:

pytest ... --cov --cov-report=xml

coverage run  --parallel-mode -m pytest
coverage combine
coverage xml

Note that, by virtue of coverage being a hard dependency to pytest-cov, it would be perfectly backwards compatible to use it directly.

The difficulty lies in how to approach this; here's a summary of the design constraints we're facing.

• tox.yml (via tox itself) can only append arguments to a command via matrix.pytest_flag, which tox redirects as posargs
• in order to automatically form a coverage command dynamically, I need to be able to prepend coverage run -m before the pytest invoke
• I would also need to somehow be able to add dynamically generated commands to invoke coverage combine or coverage xml ...

This just seems impossible to do in the most general case. My preferred solution would be to renounce doing this stuff dynamically and let users define their own commands (using pytest-cov or coverage directly, at their discretion); but of course that'd be a breaking change. I will try to come up with a backward compatible transition plan.

Originally posted by @neutrinoceros in #207 (comment)

[arybhatt4533]

after reading your issue . I understand your problem can i solve it am able to this work with efficiently

[neutrinoceros]

How would you go about it ?

[arybhatt4533]

Idea: Make pytest-cov optional

Right now the project requires pytest-cov for coverage reports.
However, the actual work can be done directly by the coverage package, which is already a dependency. Because of that, we don’t necessarily need to force users to install pytest-cov.

🔧 Practical workflow without pytest-cov
coverage run --parallel-mode -m pytest
coverage combine
coverage xml

What this does:

The first command runs pytest and collects coverage

The second merges coverage from parallel runs or multiple tox environments

The last one creates the same XML report that tools expect today

🔄 How to avoid breaking existing setups

The idea is to:

Use the above workflow as the default

Let users override it in their tox settings if they prefer using pytest-cov

Default (when users don’t configure anything):

commands =
coverage run --parallel-mode -m pytest {posargs}
coverage combine
coverage xml

If someone wants to keep using pytest-cov:

commands =
pytest --cov --cov-report=xml {posargs}

👍 Why this is better

pytest-cov becomes optional instead of mandatory

Smaller and safer dependency surface

Users still have the choice to keep using the plugin

Existing workflows won’t break

🙋‍♂️ Next step

If this approach looks fine, I can go ahead and prepare a PR based on it.
#neutrinoceros

[neutrinoceros]

Is there a person behind the LLM I seem to be talking to ?

[arybhatt4533]

Don’t worry, I did not write that myself. It is the AI system replying automatically. I just use it to explain things more clearly.
#neutrinoceros

[neutrinoceros]

I disagree that it clarifies anything. Please don't post LLM generated messages or code on this bug tracker, it is not helpful and to some degree, even rude. Thanks.

[Flamki]

I’d like to take this on.

Thanks for the clear problem statement and constraints. I agree with the goal: make pytest-cov optional while keeping current behavior stable for existing users.

Plan

I’ll implement this as a backward-compatible enhancement to tox.yml:

1. Add a new input, coverage-tool, with two supported values:

   • pytest-cov (default, keeps current behavior)
   • coverage.py (new mode)

2. Keep existing behavior unchanged in default mode:

   • Continue appending pytest-cov flags for XML coverage output.

3. In coverage.py mode:

   • Skip pytest-cov-specific flag injection.
   • After tox runs, generate XML via coverage.py directly:
     • coverage combine (best effort)
     • coverage xml

4. Keep Codecov upload flow the same:

   • still controlled by the existing coverage input (codecov).

Why this approach

• No breaking change for current users.
• Lets projects choose direct coverage.py usage and drop pytest-cov from their dependency graph.
• Small, contained workflow change with clear migration path.

What I’ll update

• tools/tox_matrix.py (matrix field + validation + flag behavior)
• .github/workflows/tox.yml (new input + conditional coverage XML step)
• docs/source/tox.rst (document both modes and usage guidance)
• Re-embed workflow scripts with the existing repo helper script

Validation

I’ll verify:

• default pytest-cov mode behaves exactly as today
• coverage.py mode generates coverage.xml
• Codecov upload still works in both modes when enabled

If that sounds good, I’ll open a PR linked to #330 with focused commits and updated docs/tests.

[saksham242007]

I’d like to work on this. My plan is to review the current workflow that uses pytest-cov and try switching it to coverage.py directly running tests with coverage run and generating the report from there while keeping the CI output and artifacts the same.
Does this sound good?

[Cadair]

The backwards compatible-ness of this related to #383.

Although, I'm in the camp of it's not worth making things more complex for the sake of removing pytest-cov.