[Draft] OpenACC port of halo exchanges #1355

Open

abishekg7 wants to merge 146 commits intoMPAS-Dev:developfrom

abishekg7:framework/acc_halo_exch

Commits on Jan 21, 2026

add new deformation coefficients to registries

Jimy Dudhia
authored and
mgduda
committed
initialize new deformation coefficients

Jimy Dudhia
authored and
mgduda
committed
add pointers for new coefficients

Jimy Dudhia
authored and
mgduda
committed
add initialization of new arrays to zero

Jimy Dudhia
authored and
mgduda
committed
pass new arrays to dyn_tend_work - compiles

Jimy Dudhia
authored and
mgduda
committed
add deformation formulation to 2d Smagorinsky kdiff
mgduda
committed
bug fixes for computing the deformation coefficients for the 2D Smagorinsky scheme on Cartesian planes.

skamaroc
authored and
mgduda
committed
Fixed error in w_x, w_y deformation coefficients.

skamaroc
authored and
mgduda
committed
New module file for the dissipation routines.

skamaroc
authored and
mgduda
committed
(1) Changes to the Makefile to compile new dissipation module.
mgduda
committed
Modifications to use mpas_atm_dimensions in the dissipation module

skamaroc
authored and
mgduda
committed
Adding configuration string for the les models in the Registry.
mgduda
committed
Moved the dissipation term computations for u, theta and w from
mgduda
committed
Added initialization for convective boundary layer LES case.

skamaroc
authored and
mgduda
committed
add calcalate_n2 routine (no functional code yet)

Jimy Dudhia
authored and
mgduda
committed
add to arg list of calculate_n2

Jimy Dudhia
authored and
mgduda
committed
add dry N^2

Jimy Dudhia
authored and
mgduda
committed
fix theta to be in diag pool

Jimy Dudhia
authored and
mgduda
committed
fix to use theta_m and qv instead of non-updated theta

Jimy Dudhia
authored and
mgduda
committed
use local 1d theta array for efficiency

Jimy Dudhia
authored and
mgduda
committed
add qtot drag term to thetav in dry case

Jimy Dudhia
authored and
mgduda
committed
add cloudy conditions N2 calculation following WRF and Durran and Klemp (1982)

Jimy Dudhia
authored and
mgduda
committed
remove thetav use

Jimy Dudhia
authored and
mgduda
committed
add bn2 to output and commented out test call
mgduda
committed
added a 3D vertical eddy viscosity to the Registry for use in LES appliations.

skamaroc
authored and
mgduda
committed
Added a horizontal eddy_viscosity array to go along with the vertical

skamaroc
authored and
mgduda
committed
Bug fixes in the new LES mixing routines.

skamaroc
authored and
mgduda
committed
added test calls to the new 3D LES schemes. This is currently hardwired
mgduda
committed
Bug fixes for the supercell test case using the 3D Smagorinsky LES model.

skamaroc
authored and
mgduda
committed
switched signs of the vertical turbulent fluxes and vertical flux divergence

skamaroc
authored and
mgduda
committed
Added new lower boundary conditions for LES simulations:
mgduda
committed
Added a scalar variable named tke. This is the turbulent kinetic energy

skamaroc
authored and
mgduda
committed
Added passive test of prognostic tke. Does not yet pass debug step.

skamaroc
authored and
mgduda
committed
1.5 order prognostic tke mods. The prognostic tke does not feed back on the

skamaroc
authored and
mgduda
committed
Added mixing routines for scalars within the LES models,

skamaroc
authored and
mgduda
committed
Clean up and changes to run the prognostic tke les model. initial tests

skamaroc
authored and
mgduda
committed
bug fixes and the addition of the Prandtl number formulation to the

skamaroc
authored and
mgduda
committed
changed Prandtl number to 1/3 for LES simulations in the MPAS_constants module.

skamaroc
authored and
mgduda
committed
implemented a perturbation coriolis term option for the LES cases.

skamaroc
authored and
mgduda
committed
introduced the sheared PBL test case for the LES implementation, including setting

skamaroc
authored and
mgduda
committed
Implemented SAS case sounding in LES initialization.

skamaroc
authored and
mgduda
committed
Added the SAS LES time-varying lower boundary moisture and heat fluxes.
mgduda
committed
changed the initialization for the random potential temperature preturbation,

skamaroc
authored and
mgduda
committed
Added a 3D diagnostic variable: the inverse Prandtl number used in the

skamaroc
authored and
mgduda
committed
Threaded the 3D inverse Prandtl number through to the routines that need it

skamaroc
authored and
mgduda
committed
ustm passed to u_dissipation_3d, diag_physics pool accessed

Jimy Dudhia
authored and
mgduda
committed
ustm add pointer remove blanks in pool statement

Jimy Dudhia
authored and
mgduda
committed
add les_model check to bypass pbl calls

Jimy Dudhia
authored and
mgduda
committed
add time_of_day_seconds calculation (compiles)

Jimy Dudhia
authored and
mgduda
committed
add flux_les_sas call in driver_sfclayer routine

Jimy Dudhia
authored and
mgduda
committed
add fluxes to sfclay arguments and pass in

Jimy Dudhia
authored and
mgduda
committed
pass heat flux args to sfclay1d

Jimy Dudhia
authored and
mgduda
committed
copy specified fluxes to hfx and qfx before sfclay1d

Jimy Dudhia
authored and
mgduda
committed
add MOL and BR calculations to sfclay1d

Jimy Dudhia
authored and
mgduda
committed
add ustm drag in u_dissipation_3d les_sas_test

Jimy Dudhia
authored and
mgduda
committed
skip hfx and qfx calcs in sfclay for specified_flux

Jimy Dudhia
authored and
mgduda
committed
pass hfx and qfx to dissipation and use for les_sas

Jimy Dudhia
authored and
mgduda
committed
comment out duplicate flux_les_sas in dissipiation module

Jimy Dudhia
authored and
mgduda
committed
Set dummy TSK and UST=USTM (no VCONV)

Jimy Dudhia
authored and
mgduda
committed
sign correction for MOL

Jimy Dudhia
authored and
mgduda
committed
pass clock atm_do_timestep to physics_driver to driver_sfclayer

Jimy Dudhia
authored and
mgduda
committed
put k loop around prandtl_inverse calc in dissipation

Jimy Dudhia
authored and
mgduda
committed
bug fixes - fixed an uninitialized value in the w turbulent flux at the model top,

skamaroc
authored and
mgduda
committed
hfx = heat_flux*rho*cp (factor added)

Jimy Dudhia
authored and
mgduda
committed
add print for ust_edge

Jimy Dudhia
authored and
mgduda
committed
initialize landmask, lu_index(ivgtyp), xland = 1

Jimy Dudhia
authored and
mgduda
committed
change hardwired value of f - need to generize to use fEdge

Jimy Dudhia
authored and
mgduda
committed
fix rdzw/rdzu for w dissipation, use fEdge instead of coriolis_value

Jimy Dudhia
authored and
mgduda
committed
revert to hardwired f until fEdge can be used

Jimy Dudhia
authored and
mgduda
committed
convert moisture_flux w'q' to surface flux qfx

Jimy Dudhia
authored and
mgduda
committed
moisture_flux in dynamics should be w'q' = qfx/rho

Jimy Dudhia
authored and
mgduda
committed
add commented fEdge line while still using hardwired f

dudhia
authored and
mgduda
committed
initial prep to use les_test option

dudhia
authored and
mgduda
committed
fix for compile

dudhia
authored and
mgduda
committed
fixes to compile init_atmosphere

dudhia
authored and
mgduda
committed
to compile atmosphere

dudhia
authored and
mgduda
committed
fix small error in tke shear production and add surface heat flux

dudhia
authored and
mgduda
committed
remove specialized physics code to leave dynonly les

dudhia
authored and
mgduda
committed
mods to scale les-supercell case and remove surface fluxes from les_test

dudhia
authored and
mgduda
committed
add seed tke = 0.1 at all points

dudhia
authored and
mgduda
committed
fix scaling for x_period and y_period for cases

dudhia
authored and
mgduda
committed
revert supercell a_scale to 1.0

dudhia
authored and
mgduda
committed
add config_les_surface to registry and routines (not used yet)

dudhia
authored and
mgduda
committed
clean up and reduce duplication

dudhia
authored and
mgduda
committed
remove redundant dissipation routines

dudhia
authored and
mgduda
committed
Merge with Bill Skamarock updated LES modules

dudhia
authored and
mgduda
committed
add surface flux namelist options

dudhia
authored and
mgduda
committed
Add explicit declarations of iCell, k, and p in the calculate_n2 routine
mgduda
committed

Commits on Jan 30, 2026

Commits on Feb 3, 2026

Commits on Feb 4, 2026

Commits on Feb 13, 2026

Commits on Feb 20, 2026

Commits on Feb 24, 2026

Some more fixes to get GPU runs working
abishekg7
committed

Commits on Feb 26, 2026

Add data movement for some fields under the mpas_halo_groups

G. Dylan Dickerson
authored and
abishekg7
committed
Add a data region and acc kernels to the 2D packing code

G. Dylan Dickerson
authored and
abishekg7
committed
Add the update directives that should have been part of the last commit

G. Dylan Dickerson
authored and
abishekg7
committed
Comment out data present region, see if this causes an error

G. Dylan Dickerson
authored and
abishekg7
committed
Expand the data managed on the GPU for the halo exchange

G. Dylan Dickerson
authored and
abishekg7
committed
Remove the OpenACC management of recvBuf

G. Dylan Dickerson
authored and
abishekg7
committed
Add update host(sendBuf) back, address answer diff

G. Dylan Dickerson
authored and
abishekg7
committed
Expand to other packing kernels, only update sendBuf after packing finishes

G. Dylan Dickerson
authored and
abishekg7
committed
Change to simple integers to access the buffers and the field arrays

G. Dylan Dickerson
authored and
abishekg7
committed
Add kernels to unpacking loops and use a data present region to try to force GPUDirect MPI

G. Dylan Dickerson
authored and
abishekg7
committed
Change from data copyin regions to enter/exit directives for the r?arr variables

G. Dylan Dickerson
authored and
abishekg7
committed
Re-enable update host for sendBuf, add update device recvBuf

G. Dylan Dickerson
authored and
abishekg7
committed
Remove update directives, use acc host_data use_device(...) near MPI calls instead

G. Dylan Dickerson
authored and
abishekg7
committed
checkpoints: acc pack + cuda aware mpi working
abishekg7
committed
seems to be working
abishekg7
committed
Optimized packing and unpacking loops. Adding timers and other cleanup
abishekg7
committed
Working savepoint
abishekg7
committed
u_2 and w_2 need to be copied out after dynamics + cleanup
abishekg7
committed
using attach in a directive instead of the acc_attach library call
abishekg7
committed
Using attach clause in parallel region will also auto detach at end of region
abishekg7
committed
Reverting the indexing in loops and comment cleanup
abishekg7
committed
New namelist option to switch on or off GPU-Aware MPI
abishekg7
committed
Adding a dependency to mpas_timer.o in mpas_halo.o
abishekg7
committed