Merge branch 'master' into phase3-riemann

Author: sbryngelson

.github/scripts/submit-slurm-job.sh

@@ -52,7 +52,10 @@ case "$cluster" in
         account="CFD154"
         job_prefix="MFC"
         qos="hackathon"
-        extra_sbatch=""
+        # Let each job's slurmstepd broker its own steps instead of routing
+        # every srun through slurmctld. The in-job test suite launches ~1700+
+        # srun steps per allocation, which congests the Frontier controller.
+        extra_sbatch="#SBATCH --stepmgr"
         test_time="01:59:00"
         bench_time="01:59:00"
         gpu_partition_dynamic=false
@@ -62,7 +65,7 @@ case "$cluster" in
         account="CFD154"
         job_prefix="MFC"
         qos="hackathon"
-        extra_sbatch=""
+        extra_sbatch="#SBATCH --stepmgr"
         test_time="01:59:00"
         bench_time="01:59:00"
         gpu_partition_dynamic=false

README.md

@@ -53,12 +53,13 @@
 
 ## Why MFC?
 
-- **Exascale GPU performance** - Ideal weak scaling to 43K+ GPUs. Near compute-roofline behavior. [Compile-time case optimization](https://mflowcode.github.io/documentation/running.html) for up to 10x speedup.
-- **Compact codebase** - ~40K lines of Fortran with [Fypp](https://fypp.readthedocs.io/en/stable/fypp.html) metaprogramming. Small enough to read and modify; powerful enough for [Gordon Bell](https://awards.acm.org/bell).
+- **Performant at exascale** - Ideal weak scaling to 43K+ GPUs. Near compute-roofline behavior. [Compile-time case optimization](https://mflowcode.github.io/documentation/running.html) for ~10x speedup.
+- **Compact code** - ~40K lines of Fortran with [Fypp](https://fypp.readthedocs.io/en/stable/fypp.html) metaprogramming. Small enough to read and modify; powerful enough for [Gordon Bell](https://awards.acm.org/bell).
 - **Native multi-phase** - 4, 5, and 6-equation models, phase change, surface tension, bubble dynamics, and Euler-Lagrange particle tracking, all built in.
 - **Portable** - NVIDIA and AMD GPUs, CPUs, laptops to exascale. Docker, Codespaces, Homebrew, and [16+ HPC system templates](https://mflowcode.github.io/documentation/running.html).
 - **Tested** - 500+ regression tests per PR with line-level [coverage](https://app.codecov.io/gh/MFlowCode/MFC) across GNU, Intel, Cray, and NVIDIA compilers.
-- **Truly open** - MIT license, active [Slack](https://join.slack.com/t/mflowcode/shared_invite/zt-y75wibvk-g~zztjknjYkK1hFgCuJxVw), and responsive development team.
+- **Easy to shape** - If MFC is missing something you need, do not get stuck maintaining an old private copy. Open a PR and make your feature, fix, machine support, or workflow part of the maintained codebase.
+- **Open** - MIT license, active [Slack](https://join.slack.com/t/mflowcode/shared_invite/zt-y75wibvk-g~zztjknjYkK1hFgCuJxVw), and a responsive development team.
 
 > If MFC is useful to your work, please ⭐ star the repo and [cite it](#citation)!
 
@@ -115,7 +116,7 @@ Run `./mfc.sh <command> --help` for detailed options, or see the [full documenta
 ## Is this _really_ exascale?
 
 MFC weak scales to the full machines on [El Capitan](https://hpc.llnl.gov/hardware/compute-platforms/el-capitan) (MI300A), [Frontier](https://www.olcf.ornl.gov/frontier/) (MI250X), and [Alps](https://www.cscs.ch/computers/alps) (GH200) with near-ideal efficiency.
-MFC is a SPEChpc benchmark candidate, part of the JSC JUPITER Early Access Program, and used OLCF Frontier and LLNL El Capitan early access systems.
+MFC is an SPEChpc benchmark candidate, part of the JSC JUPITER Early Access Program, and used OLCF Frontier and LLNL El Capitan early access systems.
 
 <p align="center">
     <picture>
@@ -263,12 +264,12 @@ MFC is under the MIT license (see [LICENSE](LICENSE) for full text).
 
 Federal sponsors have supported MFC development, including the US Department of Defense (DOD), the National Institutes of Health (NIH), the Department of Energy (DOE) and National Nuclear Security Administration (NNSA), and the National Science Foundation (NSF).
 
-MFC computations have used many supercomputing systems. A partial list is below
+MFC computations have been run on many supercomputing systems. A partial list is below
   * OLCF Frontier and Summit, and testbeds Wombat, Crusher, and Spock (allocation CFD154, PI Bryngelson).
   * LLNL El Capitan, Tuolumne, and Lassen; El Capitan early access system Tioga.
   * NCSA Delta and DeltaAI, PSC Bridges(1/2), SDSC Comet and Expanse, Purdue Anvil, TACC Stampede(1-3), and TAMU ACES via ACCESS-CI allocations from Bryngelson, Colonius, Rodriguez, and more.
   * DOD systems Blueback, Onyx, Carpenter, Nautilus, and Narwhal via the DOD HPCMP program.
-  * Sandia National Labs systems Doom and Attaway, and testbed systems Weaver and Vortex.
+  * Sandia National Labs systems, Doom and Attaway, and testbed systems, Weaver and Vortex.
 
 ---
 

benchmarks/ibm/case.py

@@ -68,6 +68,7 @@
             "riemann_solver": 2,
             "wave_speeds": 1,
             "avg_state": 2,
+            "fd_order": 2,
             "bc_x%beg": -3,
             "bc_x%end": -3,
             "bc_y%beg": -3,

docs/documentation/case.md

@@ -369,7 +369,7 @@ Additional details on this specification can be found in [NACA airfoil](https://
 
 - For STL/OBJ geometry (geometry 5 or 12), set `model_id` to index into the `stl_models` array and specify `model_filepath`, `model_scale`, `model_translate`, and `model_threshold` on that entry.
 
-- `moving_ibm` sets the method by which movement will be applied to the immersed boundary. Using 0 will result in no movement. Using 1 will result 1-way coupling where the boundary moves at a constant rate and applied forces to the fluid based upon it's own motion. In 1-way coupling, the fluid does not apply forces back onto the IB. Using 2 will result in 2-way coupling, where the boundary pushes on the fluid and the fluid pushes back on the boundary via pressure and viscous forces. If external forces are applied, the boundary will also experience those forces.
+- `moving_ibm` sets the method by which movement will be applied to the immersed boundary. Using 0 will result in no movement. Using 1 will result 1-way coupling where the boundary moves at a constant rate and applied forces to the fluid based upon its own motion. In 1-way coupling, the fluid does not apply forces back onto the IB. Using 2 will result in 2-way coupling, where the boundary pushes on the fluid and the fluid pushes back on the boundary via pressure and viscous forces. If external forces are applied, the boundary will also experience those forces.
 
 - `vel(i)` is the initial linear velocity of the IB in the x, y, z direction for i=1, 2, 3. When `moving_ibm` equals 2, this velocity is just the starting speed of the object, which will then accelerate due to external forces. If `moving_ibm` equals 1, then this is constant if it is a number, or can be described analytically with an expression.
 
@@ -379,15 +379,15 @@ Additional details on this specification can be found in [NACA airfoil](https://
   Available variables: `x` (`x_cc(i)`), `y` (`y_cc(j)`), `z` (`z_cc(k)`), `t` (current simulation time), and `r` (the IB patch radius).
   The same intrinsic functions and `pi` constant apply; bare `e` is not available.
 
-- `coefficient_of_restitution` is a number from 0 (exclusive) to 1 (inclusive) describing how elastic IB collisions are. 0 is for perfectly inellastic collisions while 1 is for perfectly ellastic collisions.
+- `coefficient_of_restitution` is a number from 0 (exclusive) to 1 (inclusive) describing how elastic IB collisions are. 0 is for perfectly inelastic collisions while 1 is for perfectly elastic collisions.
 
-- `collision_model` is an integer to select the collision model being used for IB collisions. Using 0 disables collisions and collisiono checking. 1 enables the soft-sphere collision model, where all IBs must be circles or sphere and those IBs can collide with each other as well as walls.
+- `collision_model` is an integer to select the collision model being used for IB collisions. Using 0 disables collisions and collision checking. 1 enables the soft-sphere collision model, where all IBs must be circles or sphere and those IBs can collide with each other as well as walls.
 
-- `collision_time` is approximately the amount of simulation time used to resolve collisions. This is handled by modifying the spring gonstant used to apply collision forces.
+- `collision_time` is approximately the amount of simulation time used to resolve collisions. This is handled by modifying the spring constant used to apply collision forces.
 
 - `ib_coefficient_of_friction` is the coefficient of friction used in IB collisions.
 
-- `ib_neighborhood_radius` controls the size of the neighborhood size. This value defaults to 1, which indicates that any given rank is aware of IB's up to 1 ranks away. This parameter is required to strong-scale a case when IB's eventually grow to be larger than one full processor domain wide.
+- `ib_neighborhood_radius` controls the size of the neighborhood size. This value defaults to 1, which indicates that any given rank is aware of IBs up to 1 ranks away. This parameter is required to strong-scale a case when IBs eventually grow to be larger than one full processor domain wide.
 
 #### Particle Clouds
 

docs/index.html

@@ -63,6 +63,7 @@
             { name: "Burstwave lithotripsy",               image: "res/simulations/k.png", computer: "Delta",      computerUrl: "https://www.ncsa.illinois.edu/research/project-highlights/delta/",              accelerators: "128 A100s", walltime: "30m",  source: "https://www.youtube.com/watch?v=XWsUTaJXGF8" },
             { name: "Cavitation fragments kidney stone",   image: "res/simulations/d.png", computer: "Summit",     computerUrl: "https://www.olcf.ornl.gov/summit/",                                            accelerators: "576 V100s", walltime: "30m",  source: "https://doi.org/10.48550/arXiv.2305.09163" },
             { name: "Focused ultrasound on kidney stone",  image: "res/simulations/v.png", computer: "Bridges2",   computerUrl: "https://www.psc.edu/resources/bridges-2/",                                     accelerators: "10 V100s",  walltime: "2h",   source: "https://www.youtube.com/watch?v=z8j3NH-Y6i0" },
+            { name: "Microbubble-enhanced HIFU in a liver", image: "res/simulations/y.png", computer: "Delta",      computerUrl: "https://www.ncsa.illinois.edu/research/project-highlights/delta/",              accelerators: "128 CPU cores", walltime: "26m",  source: "https://www.youtube.com/watch?v=TbR0MEdG8OU" },
             { name: "Kidney stone stress waves",           image: "res/simulations/l.png", computer: "Bridges2",   computerUrl: "https://www.psc.edu/resources/bridges-2/",                                     accelerators: "8 V100s",   walltime: "20m",  source: "https://www.youtube.com/watch?v=Q2L0J68qnRw" },
             { name: "Whale bubble net feeding",            image: "res/simulations/p.png", computer: "Delta",      computerUrl: "https://www.ncsa.illinois.edu/research/project-highlights/delta/",              accelerators: "128 A100s", walltime: "30m",  source: "https://www.youtube.com/watch?v=6EpP6tdCZSA" },
             { name: "Earplug acoustics (kinetic energy)",  image: "res/simulations/q.png", computer: "Delta",      computerUrl: "https://www.ncsa.illinois.edu/research/project-highlights/delta/",              accelerators: "8 A100s",   walltime: "5h",   source: "https://www.youtube.com/watch?v=xSW5wZkdbrc" },

docs/res/simulations/y.png

@@ -54,6 +54,7 @@
             "bc_y%end": -3,
             "ib": "T",
             "num_ibs": 1,
+            "fd_order": 2,
             "viscous": "T",
             # Formatted Database Files Structure Parameters
             "format": "silo",

examples/2D_backward_facing_step/case.py

@@ -52,6 +52,7 @@
             "bc_y%end": -2,
             "ib": "T",
             "num_ibs": 1,
+            "fd_order": 2,
             # Formatted Database Files Structure Parameters
             "format": "silo",
             "precision": "double",

examples/2D_forward_facing_step/case.py

@@ -60,6 +60,7 @@
             # Set IB to True and add 1 patch
             "ib": "T",
             "num_ibs": 1,
+            "fd_order": 2,
             "viscous": "T",
             # Formatted Database Files Structure Parameters
             "format": "silo",

examples/2D_ibm/case.py

@@ -64,6 +64,7 @@
             # Set IB to True and add 1 patch
             "ib": "T",
             "num_ibs": 1,
+            "fd_order": 2,
             # Formatted Database Files Structure Parameters
             "format": "silo",
             "precision": "double",