Send a mail when I-Build has regressions.

[mickaelistria]

Problem

Regressions in I-Builds seem to be ignored and unnoticed by most committers.

Suggestion

When an I-Build has regression compared to some baselines (eg previous I-Build, previous release...), send a mail to the mailing-list one that matter.
This could most likely be achieved as part of each platform-specific build. The tricky part seems to be mostly getting the JUnit reports in XML format. once we have such file, the list of regressions can be sorted out with some bash, eg

$ curl --compressed https://ci.eclipse.org/ls/job/jdt-core-incubator/job/dom-with-javac/lastSuccessfulBuild/testReport/api/xml | xmllint --xpath '/testResult/suite/case[status != "PASSED"]/*[name() = "className" or name() = "name"]/text()' - | sed 'N; s/\n/ /' - | sort > baselineFailures
$ curl --compressed https://ci.eclipse.org/ls/job/jdt-core-incubator/job/PR-714/lastSuccessfulBuild/testReport/api/xml | xmllint --xpath '/testResult/suite/case[status != "PASSED"]/*[name() = "className" or name() = "name"]/text()' - | sed 'N; s/\n/ /' - | sort > prFailures
$ comm -2 -3 prFailures baselineFailures > regressions

when regressions is not an empty file, a mail could then be sent with the list.

Community

• I understand suggesting an enhancement doesn't mandate anyone to implement it. Other contributors may consider this suggestion, or not, at their own convenience. The most efficient way to get it fixed is that I implement it myself and contribute it back as a good quality patch to the project.

[iloveeclipse]

It would be great if we could just send mail test results as we had before, with a number of executed/failed tests per platform build.

Build XYZ - OS - Java version
Executed N tests / Failed M tests.

"Regressions" shouldn't be needed as default number for failing tests should be zero.

[HannesWell]

I also think implementing this should help to make test failures more visible and hopefully leads to more people taking care.

In order to increase the chance that at least committers receive this mail we should consider to mandate that committers subscribe to the mailing-lists:

• eclipse-dev
• platform-releng-dev (receives already I-build info)

I think it would be suitable to send test-results to platform-releng-dev too, but still having committers subscribed to eclipse-dev would IMO be useful as kind of a 'cross-SDK-project' mailing list. This would also avoid that one has to send mails to four or five mailing lists, if something has to be shared across Eclipse-Platform, Equinox, JDT and PDE.
If we agree on this we could ask the EF for support on this as part of the on boarding process of new committers.

This could most likely be achieved as part of each platform-specific build. The tricky part seems to be mostly getting the JUnit reports in XML format. once we have such file, the list of regressions can be sorted out with some bash, eg

Aren't the files feeding the [Test Results for I20241014-0530] page sufficient?
These are available for all platforms/configs in

• https://download.eclipse.org/eclipse/downloads/drops4/I20241014-0530/testresults/xml/

or per platform/config e.g. in

• https://download.eclipse.org/eclipse/downloads/drops4/I20241014-0530/testresults/ep434I-unit-cen64-gtk3-java17_linux.gtk.x86_64_17/

But I'm not sure if considering just xml in the latter location yields to the exact same files. E.g. when searching for org.eclipse.ua.tests. I get a difference.

"Regressions" shouldn't be needed as default number for failing tests should be zero.

Maybe we could just send the mail when there are test-failures or the number of tests has declined (which might be a configuration error)?

I can look into implementing this after after EclipseCon.

[mickaelistria]

I think it would be suitable to send test-results to platform-releng-dev too, but still having committers subscribed to eclipse-dev would IMO be useful as kind of a 'cross-SDK-project' mailing list.

eclipse-dev is supposed to be for mandatory for committers to subscribe. However, thre is nothing enforcing it so it can easily be forgotten.
IMO, the mail can go to both eclipse-dev and platform-releng-dev. Any committer must be made aware of failures or regressions.

Aren't the files feeding the [Test Results for I20241014-0530] page sufficient?

Yes. So now we know where to get the data, the question is more where can we hook the script that consumes it. Is there a CI step that joins all platform test results? As I was not sure about it, that's the reason why I first thought about sending 1 mail per platform.

Maybe we could just send the mail when there are test-failures or the number of tests has declined (which might be a configuration error)?

That would already be a major achievement.

[HannesWell]

Aren't the files feeding the [Test Results for I20241014-0530] page sufficient?

Yes. So now we know where to get the data, the question is more where can we hook the script that consumes it. Is there a CI step that joins all platform test results? As I was not sure about it, that's the reason why I first thought about sending 1 mail per platform.

Yes there is for example https://ci.eclipse.org/releng/job/Releng/job/ep-collectResults/ defined in

eclipse.platform.releng.aggregator/JenkinsJobs/Releng/collectResults.groovy

Line 1 in 6c8faac

job('Releng/ep-collectResults'){

The only non triviality is to check if the current invocation is the last one or not. Because we don't want to send a mail after each completion of a stage.

It should become relatively simple once #1950 is implemented and a test-result regression mail can be send in a subsequent stage, after the matrix/parallel stage for testing has completed. This way Jenkins would handle waiting all test results implicitly.