Severe negative MPI scaling with the latest dev branch

[gthyagi]

Summary

From recent timing plots of stokes_timing.txt, the BH92 no-fault benchmark shows severe performance degradation with increasing MPI ranks.

---

Additional Note

I believe this issue is not specific to the BH92 setup and can likely be replicated with any Stokes solver configuration.

In this case, I was running linear solves only (no nonlinear iterations), so the scaling degradation appears to occur at the linear solver / MPI / preconditioner level rather than due to nonlinear solve complexity.

---

The measured PETSc Time (sec) (Max column) is:

ncpus=1: 35.91 s
ncpus=2: 32.02 s
ncpus=4: 55.96 s
ncpus=6: 509.5 s
ncpus=8: 1397 s

Expected Behaviour

For this problem size, we expect:

• Monotonic decrease in runtime with increasing MPI ranks, or
• Mild saturation due to communication overhead

Observed Behaviour

• Runtime increases significantly beyond 2 MPI ranks
• Severe scaling breakdown between 4 → 6 → 8 ranks
• ~40× slowdown at 8 ranks compared to 1 rank

This suggests a parallel scaling or MPI-related performance issue.

---

Environment

• Underworld3 branch: development (local working branch with write_timestep / XDMF updates)
• Python: 3.12 (amr-dev pixi environment)
• MPI: MPICH 4.3.2 (ch4:ofi)
• PETSc (from logs): 3.24.5
• Date observed: March 3, 2026

---

Timing Plot

[gthyagi]

An older version of Underworld3 solved the same Stokes system using 8 CPUs with a total runtime of:
Total Time (Runtime) : 32.488 sec

[bknight1]

Have you changed PETSc_options? How does 1 CPU compare in runtime between the old and new branches?

[gthyagi]

At least in the scripts, the PETSc options are the same. I’m not sure whether there are any differences in the internal options within the code.

[gthyagi]

MPI scaling from an older UW3 version:

[gthyagi]

Check partial fix #74

[gthyagi]

Problem

On macOS (UW latest, amr-dev pixi env), Ex_Stokes_Kramer_latest.py scales negatively with MPI:

• ncpus=1: ~20–35 s
• ncpus=2: ~69–90 s (slower than 1 rank)

PETSc timing indicates the 2-rank run is communication-dominated (MatMult, VecScatter*, high MPI message count), not compute-dominated.

Root Cause

This is a runtime transport issue in the MPI stack on this machine (MPICH + OFI/libfabric), not a benchmark logic issue.

By default, OFI auto-selects a provider. On this macOS setup, the selected/default provider is inefficient for local multi-rank communication, causing large communication overhead and poor scaling.

What FI_PROVIDER=tcp Means

• FI_PROVIDER is a libfabric/OFI environment variable.
• tcp forces OFI to use the TCP sockets provider.
• This overrides auto-provider selection.

Effect:

1. Forces a stable transport backend.
2. Avoids the slow default provider path seen on this system.
3. Reduces MPI communication overhead for this benchmark.

Verified Solution

Run MPI jobs with:

FI_PROVIDER=tcp pixi run -m /path/to/underworld3 -e amr-dev mpiexec -n 2 python benchmarks/annulus/Ex_Stokes_Kramer_latest.py

[lmoresi]

Root Cause

This is not an Underworld3 bug. It's a performance regression in MPICH 4.x collective operations on macOS (Apple Silicon). Point-to-point MPI operations are unaffected — only collectives like MPI_Allreduce suffer catastrophic slowdown at ≥4 processes.

We isolated this through systematic elimination:

1. Different UW3 solvers (GAMG, Jacobi, fieldsplit) → same problem → not solver-specific
2. Different PETSc builds (conda-forge, custom) → same problem → not PETSc-specific
3. Pure mpi4py benchmark (no PETSc/UW3) → same problem → not UW3 or PETSc
4. Pure C MPI benchmark (no Python at all) → same problem → not Python
5. Different MPICH builds (ch4:ofi, ch3:nemesis) → same problem → not transport-specific
6. OpenMPI 5.0.10 → no problem → MPICH-specific on macOS

Raw MPI benchmark (pure C, 100k iterations)

nprocs	MPICH 4.3.2 Allreduce 8B	OpenMPI 5.0.10 Allreduce 8B
2	1.8 µs	0.4 µs
4	54 µs	1.5 µs
6	(timed out)	14.8 µs

MPICH point-to-point (ping-pong) is fine at all process counts — the problem is specific to collective operations.

UW3 Stokes solver benchmark

2D Stokes solve, ~2700 triangular elements, P2/P1, fieldsplit Schur complement.
Machine: M1 MacBook Pro (8 cores), other processes running, so times are indicative not absolute.

nprocs	MPICH (old)	OpenMPI (conda PETSc)	OpenMPI (custom PETSc/AMR)
1	5.4s	7.0s	4.7s
2	6.4s	5.8s	5.8s
4	31.9s	7.4s	6.9s
6	169.3s	13.4s	18.0s

The MPICH build degrades catastrophically (31× slowdown at 6 procs). OpenMPI scales normally.

Fix

We've switched the default MPI to OpenMPI on macOS. This is on branch feature/openmpi-platform-default.

The changes are:

• pixi.toml: Primary environments (default, dev, amr-dev, etc.) now use platform-conditional dependencies — OpenMPI on macOS, MPICH on Linux. Explicit override environments (mpich-dev, amr-openmpi-dev, etc.) are available when you need the non-default MPI. We have not yet tested whether MPICH has the same issue on Linux.
• petsc-custom/build-petsc.sh: Auto-detects MPI from the pixi environment and sets PETSC_ARCH accordingly (petsc-4-uw-openmpi or petsc-4-uw-mpich). Both builds co-exist under the same PETSc source tree.
• uw script: Setup wizard now includes MPI choice with platform-appropriate defaults.

We also added OMP_NUM_THREADS=1 and OPENBLAS_NUM_THREADS=1 to the environment activation to prevent OpenMP thread oversubscription (conda-forge OpenBLAS defaults to 8 threads per process, which compounds the problem under MPI).

Rebuild Instructions (AMR / custom PETSc users)

If you're using amr-dev (or any AMR environment), you need to rebuild PETSc with OpenMPI. Your existing MPICH build is preserved — both co-exist.

# 1. Pull the branch
git fetch origin
git checkout feature/openmpi-platform-default

# 2. Re-run setup (will pick OpenMPI on macOS automatically)
./uw setup
#   → Choose: AMR = Yes, Tier = Developer, MPI = Platform default
#   → This writes "amr-dev" to .pixi-env and installs OpenMPI dependencies

# 3. Build PETSc with OpenMPI (~1 hour)
./uw petsc-local-build
#   → Auto-detects OpenMPI, builds under petsc-4-uw-openmpi
#   → Your old petsc-4-uw build is untouched

# 4. Build underworld3
./uw build

# 5. Verify
./uw doctor

Preserving the old MPICH build

The old petsc-4-uw build still works. To make it findable under the new naming convention:

cd petsc-custom/petsc
ln -s petsc-4-uw petsc-4-uw-mpich

Then you can switch between MPI implementations:

./uw set-env amr-dev        # OpenMPI (macOS default)
./uw set-env amr-mpich-dev  # MPICH (uses the symlinked old build)
./uw build                  # Rebuild UW3 for the selected environment

Standalone reproducer (no UW3 needed)

Pure C MPI benchmark

#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int main(int argc, char **argv) {
    MPI_Init(&argc, &argv);
    int rank, size;
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &size);

    int iters = 100000;
    double t0, t1;

    /* Allreduce - 8 bytes */
    double val = 1.0, result;
    MPI_Barrier(MPI_COMM_WORLD);
    t0 = MPI_Wtime();
    for (int i = 0; i < iters; i++)
        MPI_Allreduce(&val, &result, 1, MPI_DOUBLE, MPI_SUM, MPI_COMM_WORLD);
    t1 = MPI_Wtime();
    if (rank == 0)
        printf("nprocs=%d, Allreduce 8B: %.1f us/call\n", size, (t1-t0)/iters*1e6);

    /* Ping-pong if size >= 2 */
    if (size >= 2) {
        double msg[1000];
        memset(msg, 0, sizeof(msg));
        MPI_Barrier(MPI_COMM_WORLD);
        if (rank == 0) {
            t0 = MPI_Wtime();
            for (int i = 0; i < iters; i++) {
                MPI_Send(msg, 1000, MPI_DOUBLE, 1, 0, MPI_COMM_WORLD);
                MPI_Recv(msg, 1000, MPI_DOUBLE, 1, 1, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
            }
            t1 = MPI_Wtime();
            printf("nprocs=%d, Ping-pong 8KB: %.1f us/rtt\n", size, (t1-t0)/iters*1e6);
        } else if (rank == 1) {
            for (int i = 0; i < iters; i++) {
                MPI_Recv(msg, 1000, MPI_DOUBLE, 0, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
                MPI_Send(msg, 1000, MPI_DOUBLE, 0, 1, MPI_COMM_WORLD);
            }
        }
    }

    MPI_Finalize();
    return 0;
}

Compile and run: mpicc -O2 -o mpi_bench mpi_bench.c && mpirun -np 4 ./mpi_bench

[lmoresi]

Fixed by PR #75 (merged to development). Default MPI on macOS is now OpenMPI, which does not have the collective-operation scaling bug. Thread capping also added as a secondary measure.

See the detailed diagnosis and rebuild instructions earlier in this thread.