Skip to content

Turing WPI support #1335

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sbryngelson merged 5 commits into from
Apr 3, 2026
+57 −1
Merged

Turing WPI support #1335

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
faa0693
add turing WPI modules and template
Mar 24, 2026
21e7bd3
Merge branch 'MFlowCode:master' into master
dgvacarevelo Apr 2, 2026
a10ab03
Switch to CPU-only mode since GPU support on Turing is still experime…
Apr 2, 2026
acdcb2f
No spaces in mail-type directive
Apr 2, 2026
91ae402
Merge branch 'MFlowCode:master' into master
dgvacarevelo Apr 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion toolchain/bootstrap/modules.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,8 @@ if [ -v $u_c ]; then
log "$BR""Brown$W: Oscar (o)"
log "$B""DoD$W: Carpenter Cray (cc) | Carpenter GNU (c) | Nautilus (n)"
log "$OR""Florida$W: HiPerGator (h)"
log_n "($G""a$W/$G""f$W/$G""s$W/$G""w$W/$B""tuo$W/$C""b$W/$C""e$CR/$C""d/$C""dai$CR/$Y""p$CR/$R""r$CR/$B""cc$CR/$B""c$CR/$B""n$CR/$BR""o"$CR"/$OR""h"$CR"): "
log "$C""WPI $W: Turing (t)"
log_n "($G""a$W/$G""f$W/$G""s$W/$G""w$W/$B""tuo$W/$C""b$W/$C""e$CR/$C""d/$C""dai$CR/$Y""p$CR/$R""r$CR/$B""cc$CR/$B""c$CR/$B""n$CR/$BR""o"$CR"/$OR""h"$CR/$C""t""$CR"): "
read u_c
log
fi
Expand Down
4 changes: 4 additions & 0 deletions toolchain/modules
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,3 +102,7 @@ h-gpu CC=/apps/compilers/nvhpc/25.9/Linux_x86_64/25.9/comm_libs/mpi/bin/mpicc
h-gpu CXX=/apps/compilers/nvhpc/25.9/Linux_x86_64/25.9/comm_libs/mpi/bin/mpicxx
h-gpu FC=/apps/compilers/nvhpc/25.9/Linux_x86_64/25.9/comm_libs/mpi/bin/mpifort
h-gpu NVCOMPILER_COMM_LIBS_HOME=/apps/compilers/nvhpc/25.9/Linux_x86_64/25.9/comm_libs/12.9

t WPI Turing
t-all slurm
t-cpu gcc/12.1.0/i6yk33f openmpi/4.1.3/ebae7zc python/3.13.5/6anz4qy
51 changes: 51 additions & 0 deletions toolchain/templates/turing.mako
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
#!/usr/bin/env bash

<%namespace name="helpers" file="helpers.mako"/>

% if engine == 'batch':
#SBATCH --nodes=${nodes}
#SBATCH --ntasks-per-node=${tasks_per_node}
#SBATCH --cpus-per-task=1
#SBATCH --job-name="${name}"
#SBATCH --time=${walltime}
% if partition:
#SBATCH --partition=${partition}
% endif
% if account:
#SBATCH --account="${account}"
% endif
#SBATCH --output="${name}.out"
#SBATCH --error="${name}.err"
#SBATCH --export=ALL
% if email:
#SBATCH --mail-user=${email}
#SBATCH --mail-type=BEGIN,END,FAIL
% endif
% endif

${helpers.template_prologue()}

ok ":) Loading modules:\n"
cd "${MFC_ROOT_DIR}"
. ./mfc.sh load -c t -m ${'g' if gpu_enabled else 'c'}
Copy link
Copy Markdown
Contributor

@qodo-code-review qodo-code-review bot Apr 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Action required

1. Gpu mode missing module set 🐞 Bug ≡ Correctness

toolchain/templates/turing.mako uses gpu_enabled to select -m g, but toolchain/modules defines no
t-gpu entry, so mfc.sh load will omit compiler/MPI modules and MPI runs can fail due to missing
runtime libraries. This is triggered whenever a user runs with --gpu on Turing (gpu_enabled=true).
Agent Prompt 
### Issue description
`turing.mako` selects `-m g` when `gpu_enabled` is true, but there is no `t-gpu` module set in `toolchain/modules`. As a result, GPU mode on Turing will only load `t-all` (currently just `slurm`) and will miss required compiler/MPI modules.

### Issue Context
`gpu_enabled` is driven by the `--gpu` CLI flag, so users can unintentionally hit this even on a CPU-only cluster.

### Fix Focus Areas
- toolchain/templates/turing.mako[28-33]
- toolchain/modules[106-108]
- toolchain/bootstrap/modules.sh[105-114]

### Suggested fix options
1) If Turing is CPU-only: change the template to always load CPU modules (`-m c`) and optionally error if `gpu_enabled` is requested.
2) If GPU runs are intended: add `t-gpu ...` in `toolchain/modules` (either alias to `t-cpu` or add the real GPU toolchain modules) so `mfc.sh load -c t -m g` loads a complete environment.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Copy link
Copy Markdown
Contributor Author

@dgvacarevelo dgvacarevelo Apr 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The template and modules targets CPU-only simulations, as GPU support on Turing remains experimental and restricted to single-processor runs.

cd - > /dev/null
echo


% for target in targets:
${helpers.run_prologue(target)}

% if not mpi:
(set -x; ${profiler} "${target.get_install_binpath(case)}")
% else:
(set -x; ${profiler} \
srun --mpi=pmi2 --ntasks=${nodes*tasks_per_node} \
"${target.get_install_binpath(case)}")
% endif

${helpers.run_epilogue(target)}

echo
% endfor

${helpers.template_epilogue()}
Loading