From 97a9c9624de97f49952aefd8ebe17307026c3ccd Mon Sep 17 00:00:00 2001 From: RafaelGSS Date: Wed, 16 Apr 2025 03:03:56 -0300 Subject: [PATCH 1/9] Blog: add update to Security CI incident Co-Authored-By: Antoine du Hamel Co-Authored-By: Richard Lau Co-Authored-By: Matteo Collina --- .../vulnerability/march-2025-ci-incident.md | 66 ++++++++++++++++++- .../example_attack_test_infra.svg | 4 ++ .../blog/vulnerability/example_test_infra.svg | 4 ++ 3 files changed, 72 insertions(+), 2 deletions(-) create mode 100644 apps/site/public/static/images/blog/vulnerability/example_attack_test_infra.svg create mode 100644 apps/site/public/static/images/blog/vulnerability/example_test_infra.svg diff --git a/apps/site/pages/en/blog/vulnerability/march-2025-ci-incident.md b/apps/site/pages/en/blog/vulnerability/march-2025-ci-incident.md index 635daabd05051..ef8aba8298892 100644 --- a/apps/site/pages/en/blog/vulnerability/march-2025-ci-incident.md +++ b/apps/site/pages/en/blog/vulnerability/march-2025-ci-incident.md @@ -1,13 +1,72 @@ --- -date: '2025-03-31T16:30:00.617Z' +date: '2025-04-17T16:30:00.617Z' category: vulnerability title: Node.js Test CI Security Incident layout: blog-post author: Node.js Technical Steering Committee --- -On March 21st, the Node.js project received a security report regarding our development infrastructure via [our bug bounty program](https://hackerone.com/nodejs). We immediately restricted access while implementing corrective actions. +# _(Update 16-April-2025)_ Node.js Test CI Security Incident – Full Disclosure + +## **Summary** + +On March 21, 2025, we received a [security report via HackerOne](https://hackerone.com/reports/3050534) (link restricted at time of writing), detailing a successful compromise of several Node.js test CI hosts. + +According to the HackerOne report, the exploit proceeded as follows: + +1. Submit a valid pull request to nodejs/node. +2. Wait for a maintainer to add the `request-ci` label (this label is added to every pull request with non-documentation changes). +3. After approval, update the pull request using an outdated Git commit timestamp. +4. When Jenkins pipelines trigger, they fetch and execute code from the forked pull request. +5. Attain code execution on Node.js Jenkins agents. + +Upon review, we identified that the `request-ci` label step simplifies but is **not required** to carry out the attack. A similar attack could be used against the `commit-queue` label, thus potentially allowing an attacker to land an unauthorized code change. + +The core issue stems from a Time-of-Check-Time-of-Use (TOCTOU) vulnerability between initiating a CI build and the moment the Jenkins job checks out the code. Previously, CI jobs used Git references (`refs/pull//head`), which attackers can alter after triggering the CI. Importantly, the collaborator initiating the CI build did nothing wrong—the pull request appeared safe when CI was triggered. + +![Example of workflow for starting Jenkins CI on a Github Pull Request][example_test_infra] + +![Example of attack in the Node.js test infra][example_attack_test_Infra] + +## **Remediation** + +In response to this security incident, the Node.js security team took measures to mitigate risks and secure the infrastructure. + +- Immediately upon confirmation of the vulnerability, access to initiate new Jenkins CI runs was restricted to prevent further compromise while the team validated the report. +- All compromised hosts (24 machines) were swiftly identified, removed from Jenkins, and rebuilt to eliminate any potential residual risk left over from the initial ingress. +- Security improvements were implemented in Jenkins jobs to validate commit SHAs, ensuring jobs only executed trusted and verified code. +- `request-ci` and `commit-queue` labels now act relying on validated commit SHAs instead of comparing dates. +- Comprehensive audits were carried out across 140 Jenkins jobs, prioritizing frequently used ones, to detect and remediate vulnerabilities. +- Identified vulnerable GitHub workflows were temporarily disabled, promptly patched, and re-enabled with enhanced security measures. +These targeted actions significantly strengthened the security posture of our CI infrastructure, preventing the recurrence of similar potential +intrusions and ensuring safe operations moving forward. + +## **Timeline** + +- **Friday, 21 March 2025**: Report received on Hackerone. Initial triage confirmed the report as a genuine issue. The ability to start new Jenkins CI runs was restricted to prevent any further machine compromises. +- **Monday, 24 March 2025:** All compromised machines (totalling 24\) were identified and removed from Jenkins (pending a complete rebuild). Initial attempts to evaluate all 140 jobs defined in our Jenkins instance for vulnerability. Work started on updating the most often used vulnerable jobs to take an expected commit SHA and only proceed if the SHA of the code checked out on the machine matches. +- **Tuesday, 25 March 2025:** Some affected hosts rebuilt. The updated jobs failed on macOS and were investigated and updated again. +- **Wednesday, 26 March 2025**: More jobs updated and affected hosts rebuilt. Some GitHub workflows also identified as being vulnerable to similar attacks and disabled. +- **Thursday, 27 March 2025**: Validation logic in the updated jobs tweaked again to allow daily testing on non-pull request branches. Decision taken to disable all remaining jobs that had not been evaluated for the vulnerability or identified as needing the fix applied. More machines rebuilt. +- **Friday 28 March 2025:** Ability to start jobs on Jenkins was reenabled for Node.js collaborators. Some lesser used jobs are still disabled. GitHub workflows patched and re-enabled. +- **Wednesday, 2 April 2025**: More machines rebuilt. +- **Thursday, 3 April 2025**: Benchmarking and libuv CI jobs updated. + +## **Security vs. Developer Experience** + +Over 100 volunteers maintain the Node.js project. Our processes aim to streamline CI initiation and verification of contributions across approximately 100 Jenkins runners spanning multiple operating systems and CPU architectures. +The existing CI system design anticipates potential compromises, recognizing the need to balance security with developer convenience. + +## **Volunteer Organization** + +As a volunteer-driven organization, such security incidents significantly disrupt our operational capabilities. We **strongly** **recommend** that security researchers **avoid** unauthorized attempts to breach our systems. Instead, please coordinate responsibly through our official HackerOne program. + +--- + +# Node.js Test CI Security Incident – Notice + +On March 21st, the Node.js project received a security report regarding our development infrastructure via [our bug bounty program](https://hackerone.com/nodejs). We immediately restricted access while implementing corrective actions. The reported issue did not impact the Node.js runtime and there is no risk to users of Node.js. No action by Node.js users is required. The development infrastructure is expected to be available to the community by April 15 or sooner. @@ -19,3 +78,6 @@ A full report of this incident will be available forthcoming. We appreciate the The current Node.js security policy can be found at [https://nodejs.org/security/](/security/). Please follow the process outlined in if you wish to report a vulnerability in Node.js. Subscribe to the low-volume announcement-only nodejs-sec mailing list at to stay up to date on security vulnerabilities and security-related releases of Node.js and the projects maintained in the nodejs GitHub organization. + +[example_test_infra]: /static/images/blog/vulnerability/example_test_infra.svg +[example_attack_test_Infra]: /static/images/blog/vulnerability/example_attack_test_infra.svg diff --git a/apps/site/public/static/images/blog/vulnerability/example_attack_test_infra.svg b/apps/site/public/static/images/blog/vulnerability/example_attack_test_infra.svg new file mode 100644 index 0000000000000..18edfa7801ac1 --- /dev/null +++ b/apps/site/public/static/images/blog/vulnerability/example_attack_test_infra.svg @@ -0,0 +1,4 @@ + + +PR #XYZMaintaineradd'request'ci'labelCertifySafervalidationtriggersrequested bya maintainer?PR has notchanged since'request-ci' labelif trueFetch /PR_ID/HEADnode-test-1node-test-2node-test-...node-test-NJenkins Test CIClusterAttacker send a commit changingthe HEAD of the PR just validatedTOCTOU \ No newline at end of file diff --git a/apps/site/public/static/images/blog/vulnerability/example_test_infra.svg b/apps/site/public/static/images/blog/vulnerability/example_test_infra.svg new file mode 100644 index 0000000000000..3d2135477ef60 --- /dev/null +++ b/apps/site/public/static/images/blog/vulnerability/example_test_infra.svg @@ -0,0 +1,4 @@ + + +PR #XYZMaintaineradd'request'ci'labelCertifySafervalidationtriggersrequested bya maintainer?PR has notchanged since'request-ci' labelif trueFetch /PR_ID/HEADnode-test-1node-test-2node-test-...node-test-NJenkins Test CICluster \ No newline at end of file From 9ad38a2eb64ac2ac5a07bedce13cc593a8f5b560 Mon Sep 17 00:00:00 2001 From: RafaelGSS Date: Fri, 18 Apr 2025 10:47:41 -0300 Subject: [PATCH 2/9] fixup! Blog: add update to Security CI incident --- .../vulnerability/march-2025-ci-incident.md | 34 +++++++++---------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/apps/site/pages/en/blog/vulnerability/march-2025-ci-incident.md b/apps/site/pages/en/blog/vulnerability/march-2025-ci-incident.md index ef8aba8298892..33e10c54961b2 100644 --- a/apps/site/pages/en/blog/vulnerability/march-2025-ci-incident.md +++ b/apps/site/pages/en/blog/vulnerability/march-2025-ci-incident.md @@ -1,14 +1,14 @@ --- -date: '2025-04-17T16:30:00.617Z' +date: '2025-04-21T16:30:00.617Z' category: vulnerability title: Node.js Test CI Security Incident layout: blog-post author: Node.js Technical Steering Committee --- -# _(Update 16-April-2025)_ Node.js Test CI Security Incident – Full Disclosure +# _(Update 21-April-2025)_ Node.js Test CI Security Incident – Full Disclosure -## **Summary** +## Summary On March 21, 2025, we received a [security report via HackerOne](https://hackerone.com/reports/3050534) (link restricted at time of writing), detailing a successful compromise of several Node.js test CI hosts. @@ -28,7 +28,7 @@ The core issue stems from a Time-of-Check-Time-of-Use (TOCTOU) vulnerability bet ![Example of attack in the Node.js test infra][example_attack_test_Infra] -## **Remediation** +## Remediation In response to this security incident, the Node.js security team took measures to mitigate risks and secure the infrastructure. @@ -42,25 +42,25 @@ In response to this security incident, the Node.js security team took measures t These targeted actions significantly strengthened the security posture of our CI infrastructure, preventing the recurrence of similar potential intrusions and ensuring safe operations moving forward. -## **Timeline** +## Timeline -- **Friday, 21 March 2025**: Report received on Hackerone. Initial triage confirmed the report as a genuine issue. The ability to start new Jenkins CI runs was restricted to prevent any further machine compromises. -- **Monday, 24 March 2025:** All compromised machines (totalling 24\) were identified and removed from Jenkins (pending a complete rebuild). Initial attempts to evaluate all 140 jobs defined in our Jenkins instance for vulnerability. Work started on updating the most often used vulnerable jobs to take an expected commit SHA and only proceed if the SHA of the code checked out on the machine matches. -- **Tuesday, 25 March 2025:** Some affected hosts rebuilt. The updated jobs failed on macOS and were investigated and updated again. -- **Wednesday, 26 March 2025**: More jobs updated and affected hosts rebuilt. Some GitHub workflows also identified as being vulnerable to similar attacks and disabled. -- **Thursday, 27 March 2025**: Validation logic in the updated jobs tweaked again to allow daily testing on non-pull request branches. Decision taken to disable all remaining jobs that had not been evaluated for the vulnerability or identified as needing the fix applied. More machines rebuilt. -- **Friday 28 March 2025:** Ability to start jobs on Jenkins was reenabled for Node.js collaborators. Some lesser used jobs are still disabled. GitHub workflows patched and re-enabled. -- **Wednesday, 2 April 2025**: More machines rebuilt. -- **Thursday, 3 April 2025**: Benchmarking and libuv CI jobs updated. +1. **Friday, 21 March 2025**: Report received on Hackerone. Initial triage confirmed the report as a genuine issue. The ability to start new Jenkins CI runs was restricted to prevent any further machine compromises. +2. **Monday, 24 March 2025:** All compromised machines (totalling 24\) were identified and removed from Jenkins (pending a complete rebuild). Initial attempts to evaluate all 140 jobs defined in our Jenkins instance for vulnerability. Work started on updating the most often used vulnerable jobs to take an expected commit SHA and only proceed if the SHA of the code checked out on the machine matches. +3. **Tuesday, 25 March 2025:** Some affected hosts rebuilt. The updated jobs failed on macOS and were investigated and updated again. +4. **Wednesday, 26 March 2025**: More jobs updated and affected hosts rebuilt. Some GitHub workflows also identified as being vulnerable to similar attacks and disabled. +5. **Thursday, 27 March 2025**: Validation logic in the updated jobs tweaked again to allow daily testing on non-pull request branches. Decision taken to disable all remaining jobs that had not been evaluated for the vulnerability or identified as needing the fix applied. More machines rebuilt. +6. **Friday, 28 March 2025:** Ability to start jobs on Jenkins was reenabled for Node.js collaborators. Some lesser used jobs are still disabled. GitHub workflows patched and re-enabled. +7. **Wednesday, 2 April 2025**: More machines rebuilt. +8. **Thursday, 3 April 2025**: Benchmarking and libuv CI jobs updated. -## **Security vs. Developer Experience** +## Security vs. Developer Experience -Over 100 volunteers maintain the Node.js project. Our processes aim to streamline CI initiation and verification of contributions across approximately 100 Jenkins runners spanning multiple operating systems and CPU architectures. +Over 300 volunteers maintain the Node.js project. Our processes aim to streamline CI initiation and verification of contributions across approximately 100 Jenkins runners spanning multiple operating systems and CPU architectures. The existing CI system design anticipates potential compromises, recognizing the need to balance security with developer convenience. -## **Volunteer Organization** +## Volunteer Organization -As a volunteer-driven organization, such security incidents significantly disrupt our operational capabilities. We **strongly** **recommend** that security researchers **avoid** unauthorized attempts to breach our systems. Instead, please coordinate responsibly through our official HackerOne program. +As a volunteer-driven organization, such security incidents significantly disrupt our operational capabilities. We **strongly recommend** that security researchers **avoid** unauthorized attempts to breach our systems. Instead, please coordinate responsibly through our official HackerOne program. --- From 2d7443794c43577a79e7ba1c62e0531e31ced680 Mon Sep 17 00:00:00 2001 From: RafaelGSS Date: Fri, 18 Apr 2025 10:50:07 -0300 Subject: [PATCH 3/9] fixup! fixup! Blog: add update to Security CI incident --- apps/site/pages/en/blog/vulnerability/march-2025-ci-incident.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/apps/site/pages/en/blog/vulnerability/march-2025-ci-incident.md b/apps/site/pages/en/blog/vulnerability/march-2025-ci-incident.md index 33e10c54961b2..56f93c2963eb5 100644 --- a/apps/site/pages/en/blog/vulnerability/march-2025-ci-incident.md +++ b/apps/site/pages/en/blog/vulnerability/march-2025-ci-incident.md @@ -42,6 +42,8 @@ In response to this security incident, the Node.js security team took measures t These targeted actions significantly strengthened the security posture of our CI infrastructure, preventing the recurrence of similar potential intrusions and ensuring safe operations moving forward. +The change we implemented now requires every pull request to be approved before running the Jenkins CI - or for collaborators to specify the individual SHA. + ## Timeline 1. **Friday, 21 March 2025**: Report received on Hackerone. Initial triage confirmed the report as a genuine issue. The ability to start new Jenkins CI runs was restricted to prevent any further machine compromises. From 10547a97523d291e64684daf69c2234778a8be09 Mon Sep 17 00:00:00 2001 From: Matteo Collina Date: Tue, 22 Apr 2025 03:20:02 -0700 Subject: [PATCH 4/9] Update apps/site/pages/en/blog/vulnerability/march-2025-ci-incident.md Co-authored-by: Antoine du Hamel Signed-off-by: Matteo Collina --- .../site/pages/en/blog/vulnerability/march-2025-ci-incident.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/apps/site/pages/en/blog/vulnerability/march-2025-ci-incident.md b/apps/site/pages/en/blog/vulnerability/march-2025-ci-incident.md index 56f93c2963eb5..6851a26f28a22 100644 --- a/apps/site/pages/en/blog/vulnerability/march-2025-ci-incident.md +++ b/apps/site/pages/en/blog/vulnerability/march-2025-ci-incident.md @@ -51,7 +51,8 @@ The change we implemented now requires every pull request to be approved before 3. **Tuesday, 25 March 2025:** Some affected hosts rebuilt. The updated jobs failed on macOS and were investigated and updated again. 4. **Wednesday, 26 March 2025**: More jobs updated and affected hosts rebuilt. Some GitHub workflows also identified as being vulnerable to similar attacks and disabled. 5. **Thursday, 27 March 2025**: Validation logic in the updated jobs tweaked again to allow daily testing on non-pull request branches. Decision taken to disable all remaining jobs that had not been evaluated for the vulnerability or identified as needing the fix applied. More machines rebuilt. -6. **Friday, 28 March 2025:** Ability to start jobs on Jenkins was reenabled for Node.js collaborators. Some lesser used jobs are still disabled. GitHub workflows patched and re-enabled. +6. **Friday, 28 March 2025:** GitHub workflows were patched and `commit-queue` was re-enabled. +7. **Tuesday, 1 April 2025:** Ability to start jobs on Jenkins was reenabled for Node.js collaborators. Some lesser used jobs are still disabled. 7. **Wednesday, 2 April 2025**: More machines rebuilt. 8. **Thursday, 3 April 2025**: Benchmarking and libuv CI jobs updated. From 6375469905ef106732401f1ac5c6f46463dfbcb5 Mon Sep 17 00:00:00 2001 From: Matteo Collina Date: Tue, 22 Apr 2025 07:18:11 -0700 Subject: [PATCH 5/9] Update apps/site/pages/en/blog/vulnerability/march-2025-ci-incident.md Co-authored-by: Antoine du Hamel Signed-off-by: Matteo Collina --- apps/site/pages/en/blog/vulnerability/march-2025-ci-incident.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/apps/site/pages/en/blog/vulnerability/march-2025-ci-incident.md b/apps/site/pages/en/blog/vulnerability/march-2025-ci-incident.md index 6851a26f28a22..c41b8257345c3 100644 --- a/apps/site/pages/en/blog/vulnerability/march-2025-ci-incident.md +++ b/apps/site/pages/en/blog/vulnerability/march-2025-ci-incident.md @@ -52,7 +52,7 @@ The change we implemented now requires every pull request to be approved before 4. **Wednesday, 26 March 2025**: More jobs updated and affected hosts rebuilt. Some GitHub workflows also identified as being vulnerable to similar attacks and disabled. 5. **Thursday, 27 March 2025**: Validation logic in the updated jobs tweaked again to allow daily testing on non-pull request branches. Decision taken to disable all remaining jobs that had not been evaluated for the vulnerability or identified as needing the fix applied. More machines rebuilt. 6. **Friday, 28 March 2025:** GitHub workflows were patched and `commit-queue` was re-enabled. -7. **Tuesday, 1 April 2025:** Ability to start jobs on Jenkins was reenabled for Node.js collaborators. Some lesser used jobs are still disabled. +7. **Tuesday, 1 April 2025:** Ability to start jobs on Jenkins and via `request-ci` was reenabled. Some lesser used jobs were still disabled. 7. **Wednesday, 2 April 2025**: More machines rebuilt. 8. **Thursday, 3 April 2025**: Benchmarking and libuv CI jobs updated. From 1112eb4f8308d0a5c960b6b3b5a5c0e80f67a3db Mon Sep 17 00:00:00 2001 From: Matteo Collina Date: Tue, 22 Apr 2025 21:42:46 +0200 Subject: [PATCH 6/9] Update apps/site/pages/en/blog/vulnerability/march-2025-ci-incident.md Signed-off-by: Matteo Collina --- apps/site/pages/en/blog/vulnerability/march-2025-ci-incident.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/apps/site/pages/en/blog/vulnerability/march-2025-ci-incident.md b/apps/site/pages/en/blog/vulnerability/march-2025-ci-incident.md index c41b8257345c3..308800607ec91 100644 --- a/apps/site/pages/en/blog/vulnerability/march-2025-ci-incident.md +++ b/apps/site/pages/en/blog/vulnerability/march-2025-ci-incident.md @@ -63,7 +63,7 @@ The existing CI system design anticipates potential compromises, recognizing the ## Volunteer Organization -As a volunteer-driven organization, such security incidents significantly disrupt our operational capabilities. We **strongly recommend** that security researchers **avoid** unauthorized attempts to breach our systems. Instead, please coordinate responsibly through our official HackerOne program. +As a volunteer-driven organization, we rely on people dedicating their time to work on unglamorous tasks, such as hardening CI, handling security reports, and assembling releases. Even good-faith research against our live systems could significantly disrupt our operations. As always, we welcome all sorts of contributions, including penetration testing. We ask researchers to give us a heads up on what they are attempting to do on live systems and to keep an auditable record of their actions through our HackerOne program or by contacting the Node.js Technical Steering Committee directly (tsc@iojs.org). More on that in our [SECURITY.md](https://github.com/nodejs/node/blob/main/SECURITY.md) file. --- From 2348bca3c5040ebabf9a3525052c3361045d2546 Mon Sep 17 00:00:00 2001 From: Matteo Collina Date: Wed, 23 Apr 2025 06:50:39 -0700 Subject: [PATCH 7/9] Update apps/site/pages/en/blog/vulnerability/march-2025-ci-incident.md Signed-off-by: Matteo Collina --- apps/site/pages/en/blog/vulnerability/march-2025-ci-incident.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/apps/site/pages/en/blog/vulnerability/march-2025-ci-incident.md b/apps/site/pages/en/blog/vulnerability/march-2025-ci-incident.md index 308800607ec91..ee83d6d01b548 100644 --- a/apps/site/pages/en/blog/vulnerability/march-2025-ci-incident.md +++ b/apps/site/pages/en/blog/vulnerability/march-2025-ci-incident.md @@ -1,5 +1,5 @@ --- -date: '2025-04-21T16:30:00.617Z' +date: '2025-04-23T16:30:00.617Z' category: vulnerability title: Node.js Test CI Security Incident layout: blog-post From 336e39d9bf69ba4c595af19c46b0fe365b62a9c8 Mon Sep 17 00:00:00 2001 From: Matteo Collina Date: Wed, 23 Apr 2025 06:50:56 -0700 Subject: [PATCH 8/9] Update apps/site/pages/en/blog/vulnerability/march-2025-ci-incident.md Signed-off-by: Matteo Collina --- apps/site/pages/en/blog/vulnerability/march-2025-ci-incident.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/apps/site/pages/en/blog/vulnerability/march-2025-ci-incident.md b/apps/site/pages/en/blog/vulnerability/march-2025-ci-incident.md index ee83d6d01b548..3e4d7e7f0d190 100644 --- a/apps/site/pages/en/blog/vulnerability/march-2025-ci-incident.md +++ b/apps/site/pages/en/blog/vulnerability/march-2025-ci-incident.md @@ -6,7 +6,7 @@ layout: blog-post author: Node.js Technical Steering Committee --- -# _(Update 21-April-2025)_ Node.js Test CI Security Incident – Full Disclosure +# _(Update 23-April-2025)_ Node.js Test CI Security Incident – Full Disclosure ## Summary From 4ba20dee5f82fdf552b65a29a21009cbe06176da Mon Sep 17 00:00:00 2001 From: Matteo Collina Date: Wed, 23 Apr 2025 16:24:34 +0200 Subject: [PATCH 9/9] fixup Signed-off-by: Matteo Collina --- .../pages/en/blog/vulnerability/march-2025-ci-incident.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/apps/site/pages/en/blog/vulnerability/march-2025-ci-incident.md b/apps/site/pages/en/blog/vulnerability/march-2025-ci-incident.md index 3e4d7e7f0d190..12c13318ddce2 100644 --- a/apps/site/pages/en/blog/vulnerability/march-2025-ci-incident.md +++ b/apps/site/pages/en/blog/vulnerability/march-2025-ci-incident.md @@ -53,8 +53,8 @@ The change we implemented now requires every pull request to be approved before 5. **Thursday, 27 March 2025**: Validation logic in the updated jobs tweaked again to allow daily testing on non-pull request branches. Decision taken to disable all remaining jobs that had not been evaluated for the vulnerability or identified as needing the fix applied. More machines rebuilt. 6. **Friday, 28 March 2025:** GitHub workflows were patched and `commit-queue` was re-enabled. 7. **Tuesday, 1 April 2025:** Ability to start jobs on Jenkins and via `request-ci` was reenabled. Some lesser used jobs were still disabled. -7. **Wednesday, 2 April 2025**: More machines rebuilt. -8. **Thursday, 3 April 2025**: Benchmarking and libuv CI jobs updated. +8. **Wednesday, 2 April 2025**: More machines rebuilt. +9. **Thursday, 3 April 2025**: Benchmarking and libuv CI jobs updated. ## Security vs. Developer Experience