Codecov not behaving as expected

[jescalada]

Codecov seems to be behaving oddly: the coverage report in #1332 shows a substantial loss in coverage (which was around 84% a few months ago). On top of that, the lines covered has nearly doubled, so there may be lines being counted towards the "coverage" but not actually needed for testing.

Also, the CI still seems to pass despite the lower coverage on both the patch and overall project.

We should double-check that:

• Coverage is calculating correctly
• Patch/project coverage below 80% actually fails the CI
• Testing-relevant lines in feat(ssh): Add SSH agent forwarding #1332 are being included, and non-relevant ones excluded

Some more context from #1327:

SSH Implementation PR:

@fabiovincenzi reported PR

#1332 for SSH agent forwarding is open, with additional tests added.
Coverage numbers discussed; some discrepancies between CI and codecov reporting noted.
Review requested from all, especially those deploying SSH features.
@jescalada to raise an issue to investigate coverage reporting accuracy.

Additional Context

#1327, #1332

[fabiovincenzi]

The coverage discrepancy between local reports (80.00%) and Codecov (77.17%) seems to exists before SSH PR. This ~3% gap has always existed and appears to be due to how Codecov calculates "Project Coverage" using a proprietary formula, different from the standard line coverage calculation.

Key Findings

The ~3% Discrepancy Has Always Existed

I compared coverage reports on the same commits to verify this gap pre-exists:

Upstream main (commit 930a5da, no SSH code):

• Local coverage: 83.5%
• Codecov: 80.40%

This SSH branch (commit 0b0a020):

• Local coverage: 80.0%
• Codecov: 77.17%

Codecov's "Project Coverage" appears to be calculated differently from simple line coverage:

• Local/CI: Direct calculation from lcov.info: covered_lines / total_lines
• Codecov "Project Coverage": Proprietary formula that could factor in:
  • Patch coverage (changed lines)
  • Base vs head comparison
  • File-level impact weighting
  • Historical trends

Discrepancies like this are common and have been reported by many projects in the Codecov community.

Coverage Progression Over Time

• ~Few months ago: ~84% Codecov coverage
• After vitest migration (PR #1202): ~81% Codecov (
• After this SSH PR: 77.17% Codecov

[jescalada]

@fabiovincenzi I think it's safe to close this if there isn't any obvious way to fix it. On the bright side, the publicly reported coverage is on the conservative end, the actual coverage may well be closer to the one shown in the CI (~93%).