Reduce MPI oversubscription: default BLAS/OpenMP threads to 1 per rank

[gthyagi]

Problem

On many systems (especially with OpenBLAS defaults), each MPI rank can spawn multiple BLAS/OpenMP threads. This causes oversubscription (e.g. np=8 with 10 BLAS threads => up to 80 runnable threads), resulting in poor scaling and large walltime regressions.

What this PR changes

1. ./uw launcher now sets safe defaults for common thread-pool env vars only if they are not already set:
   • OMP_NUM_THREADS=1
   • OPENBLAS_NUM_THREADS=1
   • MKL_NUM_THREADS=1
   • VECLIB_MAXIMUM_THREADS=1
   • NUMEXPR_NUM_THREADS=1
2. underworld3/__init__.py applies the same MPI-safe default policy at import time for MPI runs (size > 1) when vars are unset.
3. Adds a rank-0 runtime warning when MPI is used and any thread pool is explicitly configured >1.
4. Documents the policy and override controls in docs/advanced/parallel-computing.md.

Why this approach

• Fixes the performance pitfall for users by default.
• Preserves user control: explicit env settings are respected.
• Keeps behavior predictable across launcher and direct Python execution paths.

Override controls

• Disable automatic caps:
  • UW_DISABLE_THREAD_CAPS=1
• Suppress warning (while keeping explicit >1 settings):
  • UW_SUPPRESS_THREAD_WARNING=1

Scope and compatibility

• No solver API changes.
• No change to single-rank behavior.
• MPI behavior is safer by default; advanced users can opt out.

Underworld development team with AI support from Claude Code

[lmoresi]

Review: overlap with PR #75

Thanks @gthyagi — this is good defensive work. A few notes on how it fits with the parallel scaling investigation:

Root cause is MPICH, not just thread oversubscription

The thread capping helps (it was burning 200%+ CPU on single-proc runs due to OpenBLAS defaults), but the catastrophic parallel scaling regression you reported in #68 is actually caused by MPICH 4.x collective operations on macOS. We confirmed this with pure C MPI benchmarks — no Python, no PETSc, no OpenBLAS involved. See the detailed diagnosis in #68.

PR #75 (feature/openmpi-platform-default) switches the default MPI to OpenMPI on macOS, which fixes the parallel scaling. It also includes the thread capping via pixi.toml activation environment:

[activation.env]
OMP_NUM_THREADS = "1"
OPENBLAS_NUM_THREADS = "1"

What to keep from this PR

The __init__.py safety net is genuinely useful — it protects users who run UW3 outside of pixi (e.g. pip install, or on clusters with system MPI). The pixi activation only covers pixi-managed environments.

The uw launcher changes will conflict with #75 since both modify the same file. The pixi activation approach in #75 makes the launcher-level capping redundant (pixi sets the env vars before any process launches), so the uw script portion here can be dropped.

The docs update to parallel-computing.md is worth keeping.

Suggested path forward

1. Rebase this onto development (not feature/boundary-integrals)
2. Drop the uw script changes (covered by pixi activation in Switch default MPI to OpenMPI on macOS (fixes #68) #75)
3. Keep the __init__.py thread policy and the docs update
4. Merge Switch default MPI to OpenMPI on macOS (fixes #68) #75 first (it fixes the actual scaling regression), then this PR adds the Python-level safety net on top

Also — please try the OpenMPI build on your machine to confirm the scaling fix. Instructions are in the #68 comment and in PR #75.

Underworld development team with AI support from Claude Code

[lmoresi]

@gthyagi - I didn't have much luck fixing the scaling issue when I tried knocking back the thread over-subscription, but I was not a thorough with the number of settings. Does this sort out the scaling for you ?

We can combine the PRs to grab all the effective fixes. The main thing for you is to rebase onto development.

[gthyagi]

@lmoresi I’ll implement the proposed path-forward steps. I agree that the changes in the uw launch script are redundant.

Also, the thread over-subscription fix alone did not resolve the negative MPI scaling—it only partially addressed the CPU spike issue. The CPU spikes appear to result from a combination of MPICH communication behaviour and multiple thread subscriptions, which is why I referenced this PR in the context of the negative MPI scaling.

[gthyagi]

@lmoresi I have rebased onto the development branch and tested the OpenMPI branch on Mac. Both PRs appear ready to merge.

[gthyagi]

This PR also includes the boundary-integral branch, which I am still testing. I created a new PR (#76) that cherry-picks only the BLAS/OpenMP thread changes. Therefore, this PR is no longer needed and will be closed.