You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
</div><!-- fragment --><p>This is particularly useful when computations are done in Python to generate the case.</p>
153
153
<h1><aclass="anchor" id="autotoc_md3"></a>
154
154
(Optional) Accepting command line arguments</h1>
155
-
<p>Input files can accept <b>positional</b>command line arguments, forwarded by <code>mfc.sh run</code>. Consider this example from the 3D_weak_scaling case:</p>
155
+
<p>Input files can accept command line arguments, forwarded by <code>mfc.sh run</code>. Consider this example from the <code>scaling</code> case:</p>
<divclass="line">parser.add_argument(<spanclass="stringliteral">"gbpp"</span>, type=int, metavar=<spanclass="stringliteral">"MEM"</span>, default=16, help=<spanclass="stringliteral">"Adjusts the problem size per rank to fit into [MEM] GB of GPU memory"</span>)</div>
<divclass="line">parser.add_argument(<spanclass="stringliteral">"-s"</span>, <spanclass="stringliteral">"--scaling"</span>, type=str, metavar=<spanclass="stringliteral">"SCALING"</span>, choices=[<spanclass="stringliteral">"weak"</span>, <spanclass="stringliteral">"strong"</span>], help=<spanclass="stringliteral">"Whether weak- or strong-scaling is being exercised."</span>)</div>
165
165
<divclass="line"></div>
166
166
<divclass="line"><spanclass="comment"># Your parsed arguments are here</span></div>
</div><!-- fragment --><p>The first argument is always a JSON string representing <code>mfc.sh run</code>'s internal state. It contains all the runtime information you might want from the build/run system. We hide it from the help menu with <code>help=argparse.SUPPRESS</code> since it is not meant to be passed in by users. You can add as many additional positional arguments as you may need.</p>
167
+
<divclass="line">args = parser.parse_args()</div>
168
+
</div><!-- fragment --><p>The first argument is always a JSON string representing <code>mfc.sh run</code>'s internal state. It contains all the runtime information you might want from the build/run system. You can add as many additional arguments as you may need.</p>
169
169
<p>To run such a case, use the following format:</p>
170
-
<divclass="fragment"><divclass="line">./mfc.sh run <path/to/case.py> <positional arguments> <regular mfc.sh run arguments></div>
171
-
</div><!-- fragment --><p>For example, to run the 3D_weak_scaling case with <code>gbpp=2</code>:</p>
172
-
<divclass="fragment"><divclass="line">./mfc.sh run examples/3D_weak_scaling/case.py 2 -t pre_process -j 8</div>
170
+
<divclass="fragment"><divclass="line">./mfc.sh run <path/to/case.py> <mfc.sh run arguments> -- <case arguments></div>
171
+
</div><!-- fragment --><p>For example, to run the <code>scaling</code> case in "weak-scaling" mode:</p>
172
+
<divclass="fragment"><divclass="line">./mfc.sh run examples/scaling/case.py -t pre_process -j 8 -- --scaling weak</div>
173
173
</div><!-- fragment --><h1><aclass="anchor" id="autotoc_md4"></a>
174
174
Parameters</h1>
175
175
<p>There are multiple sets of parameters that must be specified in the python input file:</p><oltype="1">
<tdclass="markdownTableBodyRight"><code>run_time_info</code></td><tdclass="markdownTableBodyCenter">Logical </td><tdclass="markdownTableBodyLeft">Output run-time information </td></tr>
196
+
<tdclass="markdownTableBodyRight"><code>run_time_info</code></td><tdclass="markdownTableBodyCenter">Logical </td><tdclass="markdownTableBodyLeft">Output run-time information </td></tr>
197
+
<trclass="markdownTableRowEven">
198
+
<tdclass="markdownTableBodyRight"><code>rdma_mpi</code></td><tdclass="markdownTableBodyCenter">Logical </td><tdclass="markdownTableBodyLeft">(GPUs) Enable RDMA for MPI communication. </td></tr>
197
199
</table>
198
200
<ul>
199
201
<li><code>run_time_info</code> generates a text file that includes run-time information including the CFL number(s) at each time-step.</li>
202
+
<li><code>rdma_mpi</code> optimizes data transfers between GPUs using Remote Direct Memory Access (RDMA). The underlying MPI implementation and communication infrastructure must support this feature, detecting GPU pointers and performing RDMA accordingly.</li>
<p>The <ahref="case.py"><b>Scaling</b></a> case can exercise both weak- and strong-scaling. It adjusts itself depending on the number of requested ranks.</p>
171
+
<p>This directory also contains a collection of scripts used to test strong-scaling on OLCF Frontier. They required modifying MFC to collect some metrics but are meant to serve as a reference to users wishing to run similar experiments.</p>
172
+
<h2><aclass="anchor" id="autotoc_md37"></a>
173
+
Weak Scaling</h2>
174
+
<p>Pass <code>--scaling weak</code>. The <code>--memory</code> option controls (approximately) how much memory each rank should use, in Gigabytes. The number of cells in each dimension is then adjusted according to the number of requested ranks and an approximation for the relation between cell count and memory usage. The problem size increases linearly with the number of ranks.</p>
175
+
<h2><aclass="anchor" id="autotoc_md38"></a>
176
+
Strong Scaling</h2>
177
+
<p>Pass <code>--scaling strong</code>. The <code>--memory</code> option controls (approximately) how much memory should be used in total during simulation, across all ranks, in Gigabytes. The problem size remains constant as the number of ranks increases.</p>
178
+
<h2><aclass="anchor" id="autotoc_md39"></a>
179
+
Example</h2>
180
+
<p>For example, to run a weak-scaling test that uses ~4GB of GPU memory per rank on 8 2-rank nodes with case optimization, one could:</p>
181
+
<divclass="fragment"><divclass="line">./mfc.sh run examples/scaling/case.py -t pre_process simulation \</div>
<p>Reference: P. D. Lax, Weak solutions of nonlinear hyperbolic equations and their numerical computation, Communications on pure and applied mathematics 7 (1) (1954) 159–193.</p>
<p>The <ahref="case.py"><b>3D_weak_scaling</b></a> case depends on two parameters:</p>
213
-
<ul>
214
-
<li><b>The number of MPI ranks</b> (<em>procs</em>): As <em>procs</em> increases, the problem size per rank remains constant. <em>procs</em> is determined using information provided to the case file by <code>mfc.sh run</code>.</li>
215
-
<li><b>GPU memory usage per rank</b> (<em>gbpp</em>): As <em>gbpp</em> increases, the problem size per rank increases and the number of timesteps decreases so that wall times consistent. <em>gbpp</em> is a user-defined optional argument to the <ahref="case.py">case.py</a> file. It can be specified right after the case filepath when invoking <code>mfc.sh run</code>.</li>
216
-
</ul>
217
-
<p>Weak scaling benchmarks can be produced by keeping <em>gbpp</em> constant and varying <em>procs</em>.</p>
218
-
<p>For example, to run a weak scaling test that uses ~4GB of GPU memory per rank on 8 2-rank nodes with case optimization, one could:</p>
219
-
<divclass="fragment"><divclass="line">./mfc.sh run examples/3D_weak_scaling/case.py 4 -t pre_process simulation \</div>
<p>Reference: V. A. Titarev, E. F. Toro, Finite-volume WENO schemes for three-dimensional conservation laws, Journal of Computational Physics 201 (1) (2004) 238–260.</p>
<p><imgsrc="linear_theory.jpg" alt="Linear Theory Comparision" class="inline"/></p>
270
-
<h1><aclass="anchor" id="autotoc_md58"></a>
274
+
<h1><aclass="anchor" id="autotoc_md61"></a>
271
275
Lid-Driven Cavity Problem (2D)</h1>
272
276
<p>Reference: Bezgin, D. A., & Buhendwa A. B., & Adams N. A. (2022). JAX-FLUIDS: A fully-differentiable high-order computational fluid dynamics solver for compressible two-phase flows. arXiv:2203.13760</p>
273
277
<p>Reference: Ghia, U., & Ghia, K. N., & Shin, C. T. (1982). High-re solutions for incompressible flow using the Navier-Stokes equations and a multigrid method. Journal of Computational Physics, 48, 387-411</p>
0 commit comments