Skip to content

[Experimental PR] Benchmark any Frontier Release on the CI #1090

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Malmahrouqi3 wants to merge 18 commits into from
Closed

[Experimental PR] Benchmark any Frontier Release on the CI #1090

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
2a7df55
recycled my old code to benchmark any frontier release on the CI
Dec 15, 2025
896ae5a
new changes
Dec 15, 2025
8ecb5c8
Update workflow reference for benchmark dispatch
Malmahrouqi3 Dec 15, 2025
66c348b
Update owner and repo in release workflow
Malmahrouqi3 Dec 15, 2025
549e4de
Fix owner and repo strings in release workflow
Malmahrouqi3 Dec 15, 2025
437451d
default value for bench-frontier
Dec 15, 2025
c4c4793
Delete .github/workflows/release.yml
Malmahrouqi3 Dec 16, 2025
e8ea2ba
Add concurrency support to benchmark workflow
Malmahrouqi3 Dec 16, 2025
fe92610
Fix syntax for BENCH_TAG in workflow file
Malmahrouqi3 Dec 16, 2025
e63b788
Refactor BENCH_TAG assignment in workflow
Malmahrouqi3 Dec 16, 2025
1726cdf
Fix BENCH_TAG assignment in workflow file
Malmahrouqi3 Dec 16, 2025
f29513f
Removed Junk for Prepare context step (forgotten sudo commands)
Malmahrouqi3 Dec 16, 2025
2f89aa7
Remove context preparation steps from workflow
Malmahrouqi3 Dec 16, 2025
eebe250
clean up workflow
Malmahrouqi3 Dec 16, 2025
d15c914
Update bench-frontier.yml for cluster and checkout
Malmahrouqi3 Dec 16, 2025
9626e6c
Configure Frontier cluster with OpenMP interface
Malmahrouqi3 Dec 16, 2025
fa29525
Update bench-frontier.yml
Malmahrouqi3 Dec 16, 2025
9dd5133
Merge branch 'master' into benchmark-all-releases-on-frontier-runner
Malmahrouqi3 Dec 17, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
86 changes: 86 additions & 0 deletions .github/workflows/bench-frontier.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
name: 'Benchmark Frontier Releases'

concurrency:
group: bench-frontier
cancel-in-progress: false


on:
push:
pull_request:
workflow_dispatch:
inputs:
tag:
description: 'tag to Benchmark'
required: true
default: 'v5.1.3'

jobs:
self:
name: "${{ matrix.name }} (${{ matrix.device }}${{ matrix.interface != 'none' && format('-{0}', matrix.interface) || '' }})"
strategy:
fail-fast: false
matrix:
include:
- cluster: frontier
name: Oak Ridge | Frontier (CCE)
group: phoenix
labels: frontier
flag: f
device: gpu
interface: acc
build_script: "bash .github/workflows/frontier/build.sh gpu acc bench"
- cluster: frontier
name: Oak Ridge | Frontier (CCE)
group: phoenix
labels: frontier
flag: f
device: gpu
interface: omp
build_script: "bash .github/workflows/frontier/build.sh gpu omp bench"
runs-on:
group: ${{ matrix.group }}
labels: ${{ matrix.labels }}
timeout-minutes: 480
env:
ACTIONS_RUNNER_FORCE_ACTIONS_NODE_VERSION: node16
ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
Comment on lines +45 to +47
Copy link
Contributor

@coderabbitai coderabbitai bot Dec 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Upgrade runner to support Node20 or remove deprecated Node16 override.

Node.js 16 reached end of life on November 12, 2024 and is no longer supported by GitHub Actions. Forcing Node16 with ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true bypasses security protections and exposes the workflow to known vulnerabilities.

Update the self-hosted Frontier runner to support Node20 or later, then remove these environment variables.

</comment_end>

🤖 Prompt for AI Agents 
.github/workflows/bench-frontier.yml lines 44-46: the workflow forces Node16 by
setting ACTIONS_RUNNER_FORCE_ACTIONS_NODE_VERSION and
ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION which is deprecated and insecure; either
upgrade the self-hosted Frontier runner to a Node20+ environment and then remove
these two environment variables from the workflow, or if upgrade cannot be
completed immediately, replace the override with a documented temporary comment
and an action that enforces a supported Node version (Node20) on the runner;
ensure the final change removes both ACTIONS_RUNNER_FORCE_ACTIONS_NODE_VERSION
and ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION entries and that the runner is
verified to support Node20 before committing.

BENCH_TAG: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.tag != '' && github.event.inputs.tag || 'v5.1.3' }}
Comment on lines +45 to +48
Copy link
Contributor

@coderabbitai coderabbitai bot Dec 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

cat -n .github/workflows/bench-frontier.yml

Repository: MFlowCode/MFC

Length of output: 5526

🌐 Web query:

Node.js 16 end of life EOL date GitHub Actions

💡 Result:

  • Node.js 16 upstream EOL: September 11, 2023. [1]
  • GitHub Actions runner EOL (Node16 removed): November 12, 2024. [2]

🏁 Script executed:

# Search for any comments or documentation about why Node16 is forced
rg -i "node16|node version|unsecure" .github/workflows/ -B 2 -A 2

Repository: MFlowCode/MFC

Length of output: 568

🏁 Script executed:

# Check git history for context on why this was added
git log --oneline --all -- .github/workflows/bench-frontier.yml | head -20

Repository: MFlowCode/MFC

Length of output: 129

🏁 Script executed:

# Check for any issues, comments, or discussions about Node20 upgrade
rg -i "node20|node version|upgrade.*runner|frontier.*node" --type md --type txt

# Search for any configuration or documentation about runner setup
find . -name "*.md" -o -name "*.txt" | xargs grep -l -i "frontier\|node" 2>/dev/null | head -10

Repository: MFlowCode/MFC

Length of output: 522

🏁 Script executed:

# Look for any recent issues or PRs mentioning this
git log --all --oneline --grep="node\|frontier" -i | head -20

Repository: MFlowCode/MFC

Length of output: 1099

🏁 Script executed:

# Search specifically for when Node16 forcing was added to this workflow
git log -p --all -- .github/workflows/bench-frontier.yml | grep -A 5 -B 5 "ACTIONS_RUNNER_FORCE_ACTIONS_NODE_VERSION" | head -30

Repository: MFlowCode/MFC

Length of output: 39

🏁 Script executed:

# Check if there are any open issues or discussions about Node version
rg "ACTIONS_ALLOW_USE_UNSECURE" -B 5 -A 5

Repository: MFlowCode/MFC

Length of output: 39

🏁 Script executed:

# Look for Frontier runner documentation or setup notes
find . -path "*frontier*" -name "*.sh" -o -path "*frontier*" -name "*.md" | head -20

Repository: MFlowCode/MFC

Length of output: 253

Upgrade the Frontier runner to support Node20 or newer.

Node.js 16 reached end of life on November 12, 2024. GitHub Actions no longer includes Node16 in the runner, making the use of ACTIONS_RUNNER_FORCE_ACTIONS_NODE_VERSION: node16 and ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true a security concern. These environment variables force a deprecated, unsupported Node.js version and bypass security protections.

Remove these environment variables and update the self-hosted Frontier runner to support Node20 or later.

🤖 Prompt for AI Agents 
In .github/workflows/bench-frontier.yml around lines 44 to 47, remove the
environment variables forcing Node16 (ACTIONS_RUNNER_FORCE_ACTIONS_NODE_VERSION
and ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION) and stop overriding runner Node
versions in the workflow; instead update the self-hosted Frontier runner(s) to
install and support Node20+ (or use a GitHub-hosted runner that already provides
Node20), verify workflows run under Node20, and then delete those env entries
and any references that bypass runner security.

steps:
- name: Clone - PR
uses: actions/checkout@v4
with:
path: pr

- name: Clone - Master
uses: actions/checkout@v4
with:
repository: MFlowCode/MFC
ref: master
path: master
Comment on lines +55 to +60
Copy link
Contributor

@coderabbitai coderabbitai bot Dec 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

Wrong git ref: should checkout the tag, not master branch.

This workflow benchmarks a specific release tag, but line 58 checks out the master branch instead of ${{ env.BENCH_TAG }}. This defeats the entire purpose of the workflow—you'll be benchmarking the master branch every time instead of the specified release.

Apply this diff to fix:

       - name: Clone - Master
         uses: actions/checkout@v4
         with:
           repository: MFlowCode/MFC
-          ref: master
+          ref: ${{ env.BENCH_TAG }}
           path: master
🤖 Prompt for AI Agents 
.github/workflows/bench-frontier.yml lines 54-59: the checkout step currently
uses ref: master which causes the workflow to benchmark the master branch
instead of the intended release tag; update the checkout ref to use the
BENCH_TAG environment variable (ref: ${{ env.BENCH_TAG }}) so the workflow
checks out the specified tag, leaving repository and path as-is.


- name: Checkout PR repo
uses: actions/checkout@v3
with:
path: pr

- name: Setup & Build
run: |
(cd master && ${{ matrix.build_script }}) &
wait %1
Comment on lines +67 to +70
Copy link
Contributor

@qodo-code-review qodo-code-review bot Dec 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: Simplify the Setup & Build step by removing the unnecessary backgrounding (&) and wait %1. The build script can be run directly in the foreground. [general, importance: 3]

Suggested change
- name: Setup & Build
run: |
(cd master && ${{ matrix.build_script }}) &
wait %1
- name: Setup & Build
run: |
cd master && ${{ matrix.build_script }}

Comment on lines +69 to +70
Copy link
Contributor

@coderabbitai coderabbitai bot Dec 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

Fix incorrect wait syntax for bash.

Line 128 uses wait %1, which is Windows cmd syntax. In bash, use wait (no arguments) to wait for all background jobs, or wait $! for the last background process.

Apply this diff:

          (cd master && ${{ matrix.build_script }}) & 
-         wait %1
+         wait
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
(cd master && ${{ matrix.build_script }}) &
wait %1
(cd master && ${{ matrix.build_script }}) &
wait
🤖 Prompt for AI Agents 
In .github/workflows/bench-frontier.yml around lines 127 to 128, the workflow
uses Windows cmd style wait syntax `wait %1`; replace it with the correct bash
form—either `wait` to wait for all background jobs or `wait $!` to wait for the
last background process depending on intent—so change the line to use the
appropriate bash wait and ensure the background job invocation remains `(cd
master && ${{ matrix.build_script }}) &` before the wait.


- name: Bench (Master)
run: bash master/.github/scripts/run_parallel_benchmarks.sh ${{ matrix.device }} ${{ matrix.interface }} ${{ matrix.cluster }}

- name: Print Logs
if: always()
run: |
cat master/bench-${{ matrix.device }}.* 2>/dev/null || true

- name: Archive Logs (Frontier)
uses: actions/upload-artifact@v4
with:
name: ${{ matrix.cluster }}-${{ matrix.device }}
path: |
master/bench-${{ matrix.device }}.*
master/build/benchmarks/*
124 changes: 0 additions & 124 deletions .github/workflows/bench.yml
View file
Open in desktop

This file was deleted.

127 changes: 0 additions & 127 deletions .github/workflows/cleanliness.yml
View file
Open in desktop

This file was deleted.

50 changes: 0 additions & 50 deletions .github/workflows/coverage.yml
View file
Open in desktop

This file was deleted.

Loading
Loading