[do not merge]: Testing force re-build of all packages #210

smuzaffar · 2024-06-13T07:20:45Z

Do not merge

cmsbuild · 2024-06-13T07:21:05Z

A new Pull Request was created by @smuzaffar for branch V00-34-XX.

@cmsbuild, @smuzaffar, @aandvalenzuela, @iarspider can you please review it and eventually sign? Thanks.
@antoniovilela, @sextonkennedy, @rappoccio you are the release manager for this.
cms-bot commands are listed here

cmsbuild · 2024-06-13T07:21:05Z

cms-bot internal usage

smuzaffar · 2024-06-13T07:21:26Z

please test

cmsbuild · 2024-06-15T03:38:09Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e0b45c/39889/summary.html
COMMIT: aaddec3
CMSSW: CMSSW_14_1_X_2024-06-13-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/pkgtools/210/39889/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially removed 16 lines from the logs
Reco comparison results: 4 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3345018
DQMHistoTests: Total failures: 3
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3344995
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 202 log files, 165 edm output root files, 48 DQM output files
TriggerResults: no differences found

smuzaffar · 2024-07-16T07:43:31Z

please test

cmsbuild · 2024-07-17T03:23:11Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e0b45c/40418/summary.html
COMMIT: aaddec3
CMSSW: CMSSW_14_1_X_2024-07-15-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/pkgtools/210/40418/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found 1 errors in the following unit tests:

---> test test-particleLevel_fromMiniAod had ERRORS

Comparison Summary

Summary:

You potentially removed 18 lines from the logs
Reco comparison results: 8772 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3345094
DQMHistoTests: Total failures: 14313
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3330761
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 202 log files, 165 edm output root files, 48 DQM output files
TriggerResults: found differences in 2 / 46 workflows

smuzaffar · 2025-01-29T10:16:06Z

test parameters:

set_env = NO_PACKAGE_STORE

smuzaffar · 2025-01-29T10:19:12Z

please test

cmsbuild · 2025-01-29T22:20:57Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e0b45c/44019/summary.html
COMMIT: aaddec3
CMSSW: CMSSW_15_0_X_2025-01-28-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/pkgtools/210/44019/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e0b45c/44019/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e0b45c/44019/git-merge-result

Comparison Summary

Summary:

You potentially added 1 lines to the logs
Reco comparison results: 3 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 4016770
DQMHistoTests: Total failures: 44
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 4016706
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
Checked 218 log files, 189 edm output root files, 50 DQM output files
TriggerResults: no differences found

cmsbuild · 2025-11-21T14:45:55Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e0b45c/49584/summary.html
COMMIT: 31906e6
CMSSW: CMSSW_16_0_X_2025-11-20-1100/el8_amd64_gcc14
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/pkgtools/210/49584/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed Unit Tests

I found 1 errors in the following unit tests:

---> test pytorch-test_simple_dnn had ERRORS

Comparison Summary

Summary:

You potentially added 308 lines to the logs
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 83238 differences found in the comparisons
Reco comparison had 2 failed jobs
DQMHistoTests: Total files compared: 51
DQMHistoTests: Total histograms compared: 3905633
DQMHistoTests: Total failures: 343835
DQMHistoTests: Total nulls: 324
DQMHistoTests: Total successes: 3561454
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -14732.880999999998 KiB( 50 files compared)
DQMHistoSizes: changed ( 10224.0 ): -0.192 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 13034.0 ): -0.685 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 17034.0 ): 4.320 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 250202.181 ): -0.176 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 25202.0 ): -0.127 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 29634.0,... ): -1843.433 KiB EcalDigisV/SelectiveReadout
DQMHistoSizes: changed ( 29634.0,... ): -951.648 KiB EcalDigisV/EcalDigiTask
DQMHistoSizes: changed ( 29634.0,... ): -151.247 KiB EcalHitsV/EcalSimHitsValidation
DQMHistoSizes: changed ( 7.3 ): -4.381 KiB SiStrip/MechanicalView
Checked 218 log files, 188 edm output root files, 51 DQM output files
TriggerResults: found differences in 16 / 49 workflows

cmsbuild · 2025-11-24T12:51:28Z

Pull request #210 was updated.

smuzaffar · 2025-11-24T12:51:32Z

please test with cms-sw/cms-bot#2623 for CMSSW_16_0_DEVEL_X

smuzaffar · 2025-11-24T12:57:01Z

please test

smuzaffar · 2025-11-24T12:57:37Z

please test with cms-sw/cms-bot#2623 for CMSSW_16_0_ROOT6_X/el8_amd64_gcc13

smuzaffar · 2025-11-24T12:58:11Z

please test with cms-sw/cms-bot#2623 for CMSSW_16_0_ROOT6_X/el8_aarch64_gcc13

cmsbuild · 2025-11-24T13:04:03Z

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e0b45c/49643/summary.html
COMMIT: fafbad0
CMSSW: CMSSW_16_0_ROOT6_X_2025-11-23-2300/el8_aarch64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/pkgtools/210/49643/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed External Build

I found compilation warning when building: See details on the summary page.

smuzaffar · 2025-11-24T13:19:49Z

please test with #229,cms-sw/cms-bot#2623 for CMSSW_16_0_ROOT6_X/el8_aarch64_gcc13

cmsbuild · 2025-11-24T20:44:10Z

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e0b45c/49642/summary.html
COMMIT: fafbad0
CMSSW: CMSSW_16_0_DEVEL_X_2025-11-23-2300/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/pkgtools/210/49642/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed External Build

I found compilation error when building:

/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc13/external/cmake/3.31.7-7e60cfa32399d7fcefded5d28f26f8b1/share/cmake-3.31/Modules/FetchContent.cmake:2384 (__FetchContent_Populate)
external/CMakeLists.txt:134 (FetchContent_MakeAvailable)


-- Configuring incomplete, errors occurred!
error: Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.DyAVaZ (%build)

RPM build errors:
Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.DyAVaZ (%build)

* The action "build-install-external+celeritas+v0.6.0-8f9c5ec608ac02cbd1a4424896b3e4ff" was not completed successfully because The following dependencies could not complete:

cmsbuild · 2025-11-24T23:08:07Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e0b45c/49645/summary.html
COMMIT: fafbad0
CMSSW: CMSSW_16_0_ROOT6_X_2025-11-23-2300/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/pkgtools/210/49645/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

@akritkbehera [ROOT6] Updated root to tip of branch master cmsdist#10219

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e0b45c/49645/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e0b45c/49645/git-merge-result

Failed Unit Tests

I found 3 errors in the following unit tests:

---> test testTorchSimpleDnn had ERRORS
---> test test_DavixReadV_http had ERRORS
---> test test_DavixRead_http had ERRORS

Comparison Summary

Summary:

You potentially added 209 lines to the logs
Reco comparison results: 5 differences found in the comparisons
Reco comparison had 2 failed jobs
DQMHistoTests: Total files compared: 51
DQMHistoTests: Total histograms compared: 3905633
DQMHistoTests: Total failures: 12761
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3892852
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 50 files compared)
Checked 218 log files, 188 edm output root files, 51 DQM output files
TriggerResults: no differences found

cmsbuild · 2025-11-25T04:43:04Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e0b45c/49644/summary.html
COMMIT: fafbad0
CMSSW: CMSSW_16_0_X_2025-11-24-1100/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/pkgtools/210/49644/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed Unit Tests

I found 2 errors in the following unit tests:

---> test test_DavixReadV_http had ERRORS
---> test test_DavixRead_http had ERRORS

Comparison Summary

Summary:

You potentially removed 1 lines from the logs
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 10 differences found in the comparisons
Reco comparison had 2 failed jobs
DQMHistoTests: Total files compared: 51
DQMHistoTests: Total histograms compared: 3905633
DQMHistoTests: Total failures: 21
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3905592
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 50 files compared)
Checked 218 log files, 188 edm output root files, 51 DQM output files
TriggerResults: no differences found

smuzaffar · 2025-11-25T09:07:49Z

please test with cms-sw/cms-bot#2623 for CMSSW_16_0_ROOT6_X/el8_aarch64_gcc13

cmsbuild · 2025-11-25T10:14:26Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e0b45c/49659/summary.html
COMMIT: fafbad0
CMSSW: CMSSW_16_0_DEVEL_X_2025-11-23-2300/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/pkgtools/210/49659/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed Unit Tests

I found 2 errors in the following unit tests:

---> test test_DavixReadV_http had ERRORS
---> test test_DavixRead_http had ERRORS

Comparison Summary

Summary:

You potentially added 183 lines to the logs
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 4 differences found in the comparisons
Reco comparison had 2 failed jobs
DQMHistoTests: Total files compared: 51
DQMHistoTests: Total histograms compared: 3905633
DQMHistoTests: Total failures: 7
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3905606
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 50 files compared)
Checked 218 log files, 188 edm output root files, 51 DQM output files
TriggerResults: no differences found

cmsbuild · 2025-11-26T05:31:04Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e0b45c/49661/summary.html
COMMIT: fafbad0
CMSSW: CMSSW_16_0_ROOT6_X_2025-11-24-2300/el8_aarch64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/pkgtools/210/49661/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed Unit Tests

I found 4 errors in the following unit tests:

---> test TestDQMOnlineClient-ecalgpu_dqm_sourceclient had ERRORS
---> test TestDQMOnlineClient-hcalgpu_dqm_sourceclient had ERRORS
---> test TestDQMOnlineClient-pfgpu_dqm_sourceclient had ERRORS
and more ...

smuzaffar · 2025-12-12T10:16:50Z

please test with cms-sw/cms-bot#2623 for CMSSW_16_0_ROOT6_X/el8_aarch64_gcc13

cmsbuild · 2025-12-13T00:22:33Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e0b45c/49923/summary.html
COMMIT: fafbad0
CMSSW: CMSSW_16_0_ROOT6_X_2025-12-11-2300/el8_aarch64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/pkgtools/210/49923/install.sh to create a dev area with all the needed externals and cmssw changes.

[do not merge]: Testing force re-build of all packages

aaddec3

cmsbuild added orp-pending pending-signatures externals-pending tests-pending labels Jun 13, 2024

cmsbuild added tests-started requires-external and removed tests-pending labels Jun 13, 2024

cmsbuild added tests-approved and removed tests-started labels Jun 15, 2024

cmsbuild added tests-started and removed tests-approved labels Jul 16, 2024

cmsbuild added tests-rejected and removed tests-started labels Jul 17, 2024

smuzaffar closed this Sep 22, 2024

smuzaffar deleted the test-rebuild branch September 22, 2024 16:17

smuzaffar restored the test-rebuild branch January 29, 2025 09:43

smuzaffar reopened this Jan 29, 2025

cmsbuild removed the requires-external label Jan 29, 2025

cmsbuild added tests-started and removed tests-rejected labels Jan 29, 2025

cmsbuild added tests-rejected and removed tests-started labels Nov 21, 2025

Update comment from 'test' to 'test2'

fafbad0

cmsbuild added tests-pending and removed tests-rejected labels Nov 24, 2025

cmsbuild added tests-started requires-external and removed tests-pending labels Nov 24, 2025

cmsbuild removed the requires-external label Nov 24, 2025

cmsbuild added the requires-external label Nov 24, 2025

smuzaffar mentioned this pull request Nov 24, 2025

protection against missing build type in stats #229

Merged

cmsbuild added tests-rejected and removed tests-started labels Nov 25, 2025

smuzaffar mentioned this pull request Dec 12, 2025

improve cpu utilization monitoring #231

Merged

[do not merge]: Testing force re-build of all packages #210

Are you sure you want to change the base?

[do not merge]: Testing force re-build of all packages #210

Uh oh!

Conversation

smuzaffar commented Jun 13, 2024

Uh oh!

cmsbuild commented Jun 13, 2024

Uh oh!

cmsbuild commented Jun 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

smuzaffar commented Jun 13, 2024

Uh oh!

cmsbuild commented Jun 15, 2024

Comparison Summary

Uh oh!

smuzaffar commented Jul 16, 2024

Uh oh!

cmsbuild commented Jul 17, 2024

Unit Tests

Comparison Summary

Uh oh!

smuzaffar commented Jan 29, 2025

Uh oh!

smuzaffar commented Jan 29, 2025

Uh oh!

cmsbuild commented Jan 29, 2025

Comparison Summary

Uh oh!

cmsbuild commented Nov 21, 2025

Failed Unit Tests

Comparison Summary

Uh oh!

cmsbuild commented Nov 24, 2025

Uh oh!

smuzaffar commented Nov 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

smuzaffar commented Nov 24, 2025

Uh oh!

smuzaffar commented Nov 24, 2025

Uh oh!

smuzaffar commented Nov 24, 2025

Uh oh!

cmsbuild commented Nov 24, 2025

Failed External Build

Uh oh!

smuzaffar commented Nov 24, 2025

Uh oh!

cmsbuild commented Nov 24, 2025

Failed External Build

Uh oh!

cmsbuild commented Nov 24, 2025

Failed Unit Tests

Comparison Summary

Uh oh!

cmsbuild commented Nov 25, 2025

Failed Unit Tests

Comparison Summary

Uh oh!

smuzaffar commented Nov 25, 2025

Uh oh!

cmsbuild commented Nov 25, 2025

Failed Unit Tests

Comparison Summary

Uh oh!

cmsbuild commented Nov 26, 2025

Failed Unit Tests

Uh oh!

smuzaffar commented Dec 12, 2025

Uh oh!

cmsbuild commented Dec 13, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

cmsbuild commented Jun 13, 2024 •

edited

Loading

smuzaffar commented Nov 24, 2025 •

edited

Loading