diff --git a/README.md b/README.md index a966b918fc..e39cc664d5 100644 --- a/README.md +++ b/README.md @@ -4,207 +4,96 @@

+

An exascale many-physics flow solver

+

- + Tests - + Contributors - + Slack - + License - + Codecov - + Contributing

- - - - + Docs - + Codespaces - - - + DockerHub + - - -

- -

- - - - - Star History Chart - + Homebrew

-> **If MFC helps your work, please ⭐ the repo and cite it!** +> **2025 Gordon Bell Prize Finalist** - MFC simulates compressible multi-phase flows at exascale, using [Fypp](https://fypp.readthedocs.io/en/stable/fypp.html) metaprogramming in ~40K lines of Fortran. It conducted the [largest known public CFD simulation](https://arxiv.org/abs/2505.07392) at **200 trillion grid points** and 1 quadrillion degrees of freedom, scaling ideally to >43K AMD APUs on [El Capitan](https://hpc.llnl.gov/hardware/compute-platforms/el-capitan) and >33K AMD GPUs on [Frontier](https://www.olcf.ornl.gov/frontier/). -### Who uses MFC +
+ +
-MFC runs at exascale on the world's fastest supercomputers: -- **OLCF Frontier** (>33K AMD MI250X GPUs) -- **LLNL El Capitan** (>43K AMD MI300A APUs) -- **LLNL Tuolumne**, **CSCS Alps**, and many others +## Why MFC? -### Try MFC +- **Exascale GPU performance** - Ideal weak scaling to 43K+ GPUs. Near compute-roofline behavior. [Compile-time case optimization](https://mflowcode.github.io/documentation/running.html) for up to 10x speedup. +- **Compact codebase** - ~40K lines of Fortran with [Fypp](https://fypp.readthedocs.io/en/stable/fypp.html) metaprogramming. Small enough to read and modify; powerful enough for [Gordon Bell](https://awards.acm.org/bell). +- **Native multi-phase** - 4, 5, and 6-equation models, phase change, surface tension, bubble dynamics, and Euler-Lagrange particle tracking, all built in. +- **Portable** - NVIDIA and AMD GPUs, CPUs, laptops to exascale. Docker, Codespaces, Homebrew, and [16+ HPC system templates](https://mflowcode.github.io/documentation/running.html). +- **Tested** - 500+ regression tests per PR with line-level [coverage](https://app.codecov.io/gh/MFlowCode/MFC) across GNU, Intel, Cray, and NVIDIA compilers. +- **Truly open** - MIT license, active [Slack](https://join.slack.com/t/mflowcode/shared_invite/zt-y75wibvk-g~zztjknjYkK1hFgCuJxVw), and responsive development team. + +> If MFC is useful to your work, please ⭐ star the repo and [cite it](#citation)! + +## Try MFC | Path | Command | | --- | --- | -| **Codespaces** (fastest) πŸ’¨ | Click the "Codespaces" badge above to launch in 1 click | +| **Codespaces** (fastest) πŸ’¨ | [Open a Codespace](https://github.com/codespaces/new?hide_repo_select=true&ref=master&repo=MFlowCode%2FMFC) - pre-built, zero install | | **Docker** 🐳 | `docker run -it --rm --entrypoint bash sbryngelson/mfc:latest-cpu` | -| **Homebrew** (macOS) 🍺 | `brew install mflowcode/mfc/mfc && mfc $(brew --prefix mfc)/examples/1D_sodshocktube/case.py -n 2` | -| **Local build** πŸ’» | `./mfc.sh build -j $(nproc) && ./mfc.sh test -j $(nproc)` | - -**Welcome!** -MFC simulates compressible multi-phase flows, [among other things](#what-else-can-this-thing-do). -It uses metaprogramming and is short (~40K lines of Fortran) and portable. -MFC conducted the largest known CFD simulation at 200 trillion grid points, and 1 quadrillion degrees of freedom (as of September 2025). -MFC is a 2025 Gordon Bell Prize Finalist. - -

- - - - - - - - - +| **Homebrew** (macOS) 🍺 | `brew install mflowcode/mfc/mfc` | +| **From source** πŸ’» | `git clone https://github.com/MFlowCode/MFC && cd MFC && ./mfc.sh build -j $(nproc)` | -
-Is MFC useful for you? Consider citing it or giving a star! -

- -```bibtex -@article{wilfong26, - Author = {Benjamin Wilfong and Henry {Le Berre} and Anand Radhakrishnan and Ansh Gupta and Daniel J. Vickers and Diego Vaca-Revelo and Dimitrios Adam and Haocheng Yu and Hyeoksu Lee and Jose Rodolfo Chreim and Mirelys {Carcana Barbosa} and Yanjun Zhang and Esteban Cisneros-Garibay and Aswin Gnanaskandan and Mauro {Rodriguez Jr.} and Reuben D. Budiardja and Stephen Abbott and Tim Colonius and Spencer H. Bryngelson}, - Title = {{MFC 5.0: A}n exascale many-physics flow solver}, - journal = {Computer Physics Communications}, - year = {2026}, - volume = {322}, - pages = {110055}, - doi = {10.1016/j.cpc.2026.110055}, -} +Your first simulation: +```bash +./mfc.sh run examples/3D_shockdroplet/case.py -n $(nproc) ``` +Visualize the output in `examples/3D_shockdroplet/silo_hdf5/` with ParaView, VisIt, or your favorite tool. +For detailed build instructions (Linux, macOS, Windows/WSL, HPC clusters), see the [Getting Started guide](https://mflowcode.github.io/documentation/getting-started.html). -MFC is used on the latest leadership-class supercomputers. -It scales ideally to exascale; [tens of thousands of GPUs on NVIDIA- and AMD-GPU machines](#is-this-really-exascale) on Oak Ridge Frontier, LLNL El Capitan, CSCS Alps, among others. -MFC is a SPEChpc benchmark candidate, part of the JSC JUPITER Early Access Program, and used OLCF Frontier and LLNL El Capitan early access systems. - Get in touch with Spencer if you have questions! We have an [active Slack channel](https://join.slack.com/t/mflowcode/shared_invite/zt-y75wibvk-g~zztjknjYkK1hFgCuJxVw) and development team. MFC has high- and low-level documentation, visualizations, and more on [its website](https://mflowcode.github.io/). -## An example - -We keep many examples. -Here are some of them! -MFC can execute high-fidelity simulations of shock-droplet interaction (see `examples/3d_shockdroplet`) - -

- Shock Droplet Example -

+## Examples -This one simulates high-Mach flow over an airfoil: +MFC ships with [137+ example cases](https://mflowcode.github.io/documentation/examples.html). +Here is a high-Mach flow over an airfoil (see [`examples/2D_ibm_airfoil/`](examples/2D_ibm_airfoil/)):

- Airfoil Example
+ High-Mach airfoil flow

-And here is a high-amplitude acoustic wave reflecting and emerging through a circular orifice: +And a high-amplitude acoustic wave reflecting and emerging through a circular orifice:

- Orifice Example
+ Acoustic wave through orifice

- -## Getting started - -You can navigate [to this webpage](https://mflowcode.github.io/documentation/md_getting-started.html) to get you get started using MFC on your local machine, cluster, or supercomputer! -It's rather straightforward. -Some examples are below. - -### Codespaces and Containers - - - - - -For a _very_ quick start, open a GitHub Codespace to load a pre-configured Docker container and familiarize yourself with MFC commands. -Click <> Code (green button at top right) β†’ Codespaces (right tab) β†’ + (create a codespace). - -Alternatively, run MFC directly with Docker on your local machine: -```bash -docker run -it --rm --entrypoint bash sbryngelson/mfc:latest-cpu -``` -Once inside the container, navigate to `/opt/MFC` to access MFC and run examples. - -> ****Note:**** Codespaces is a free service with a monthly quota of compute time and storage usage. -> It is recommended for testing commands, troubleshooting, and running simple case files without installing dependencies or building MFC on your device. -> Don't conduct any critical work here! -> To learn more, please see [how Docker & Containers work](https://mflowcode.github.io/documentation/md_docker.html). - -### macOS quick start (Homebrew) - - - - - -Install the prebuilt package and run an example: - -```bash -brew install mflowcode/mfc/mfc -mkdir -p ~/mfc_quickstart && cd ~/mfc_quickstart -cp $(brew --prefix mfc)/examples/1D_sodshocktube/case.py . -mfc case.py -n 2 -``` - -Use `-n X` to select the number of MPI processes. For developer commands (`build`, `test`, etc.), clone the repo and use `./mfc.sh`. - -### macOS from source - -We'll give a brief introduction for building from source on MacOS below. -Using [brew](https://brew.sh), install MFC's dependencies: -```shell -brew install coreutils python cmake fftw hdf5 gcc boost open-mpi lapack -``` -You're now ready to build and test MFC! -Put it to a local directory via -```shell -git clone https://github.com/MFlowCode/MFC -cd MFC -``` -and be sure MFC knows where to find Boost by appending to your dotfiles and sourcing them again -```shell -echo -e "export BOOST_INCLUDE='$(brew --prefix --installed boost)/include'" | tee -a ~/.bash_profile ~/.zshrc -. ~/.bash_profile 2>/dev/null || . ~/.zshrc 2>/dev/null -! [ -z "${BOOST_INCLUDE+x}" ] && echo 'Environment is ready!' || echo 'Error: $BOOST_INCLUDE is unset. Please adjust the previous commands to fit with your environment.' -``` -then you can build MFC and run the test suite! -```shell -./mfc.sh build -j $(nproc) -./mfc.sh test -j $(nproc) -``` -And... you're done! - ## Toolchain | Command | Description | @@ -213,27 +102,23 @@ And... you're done! | `./mfc.sh run case.py` | Run a simulation case (interactive or batch: SLURM/PBS/LSF) | | `./mfc.sh test` | Run the test suite | | `./mfc.sh validate case.py` | Check a case file for errors before running | -| `./mfc.sh init my_case` | Create a new case from a template | +| `./mfc.sh new my_case` | Create a new case from a template | | `./mfc.sh clean` | Remove build artifacts | | `./mfc.sh interactive` | Launch interactive menu-driven interface | -Run `./mfc.sh --help` for detailed options, or see the [full documentation](https://mflowcode.github.io/documentation/index.html). Tab completion for bash and zsh is auto-installed after you have run `./mfc.sh generate` (or any non-`init` command) at least once. Play with the examples in `examples/` ([showcased here](https://mflowcode.github.io/documentation/md_examples.html)). - -The shock-droplet interaction case above was run via -```shell -./mfc.sh run -n $(nproc) ./examples/3d_shockdroplet/case.py -``` -where `$(nproc)` is the number of cores the example will run on (and the number of physical cores on your CPU device). -You can visualize the output data in `examples/3d_shockdroplet/silo_hdf5` via Paraview, Visit, or your favorite software. +Run `./mfc.sh --help` for detailed options, or see the [full documentation](https://mflowcode.github.io/documentation/index.html). Tab completion for bash and zsh is auto-installed after you have run `./mfc.sh generate` (or any non-`new` command) at least once. Play with the examples in `examples/` ([showcased here](https://mflowcode.github.io/documentation/examples.html)). ## Is this _really_ exascale? -[OLCF Frontier](https://www.olcf.ornl.gov/frontier/) is the first exascale supercomputer. -The weak scaling of MFC on this machine shows near-ideal utilization. -We also scale ideally to >98% of LLNL El Capitan. +MFC weak scales to the full machines on [El Capitan](https://hpc.llnl.gov/hardware/compute-platforms/el-capitan) (MI300A), [Frontier](https://www.olcf.ornl.gov/frontier/) (MI250X), and [Alps](https://www.cscs.ch/computers/alps) (GH200) with near-ideal efficiency. +MFC is a SPEChpc benchmark candidate, part of the JSC JUPITER Early Access Program, and used OLCF Frontier and LLNL El Capitan early access systems.

- Scaling + + + + Weak scaling on El Capitan, Frontier, and Alps +

## What else can this thing do? @@ -246,10 +131,10 @@ We also scale ideally to >98% of LLNL El Capitan. * Multi- and single-component * 4, 5, and 6 equation models for multi-component/phase features * Kapila and Allaire models 5-equation models -* Multi- and single-phase +* Multi- and single-phase * Phase change via p, pT, and pTg schemes * Grids - * 1-3D Cartesian, cylindrical, axisymmetric. + * 1-3D Cartesian, cylindrical, axisymmetric. * Arbitrary grid stretching for multiple domain regions. * Complex/arbitrary geometries via immersed boundary method * STL geometry files supported @@ -293,10 +178,10 @@ We also scale ideally to >98% of LLNL El Capitan. * GPU compatible on NVIDIA ([P/V/A/H]100, GH200, etc.) and AMD (MI[1/2/3]00+) GPU and APU hardware * Ideal weak scaling to 100% of the largest GPU and superchip supercomputers - * \>43K AMD APUs (MI300A) on [LLNL El Capitan](https://hpc.llnl.gov/hardware/compute-platforms/el-capitan) - * \>3K AMD APUs (MI300A) on [LLNL Tuolumne](https://hpc.llnl.gov/hardware/compute-platforms/tuolumne) - * \>33K AMD GPUs (MI250X) on [OLCF Frontier](https://www.olcf.ornl.gov/frontier/) - * \>10K NVIDIA GPUs (V100) on [OLCF Summit](https://www.olcf.ornl.gov/summit/) + * \>43K AMD APUs (MI300A) on [LLNL El Capitan](https://hpc.llnl.gov/hardware/compute-platforms/el-capitan) + * \>3K AMD APUs (MI300A) on [LLNL Tuolumne](https://hpc.llnl.gov/hardware/compute-platforms/tuolumne) + * \>33K AMD GPUs (MI250X) on [OLCF Frontier](https://www.olcf.ornl.gov/frontier/) + * \>10K NVIDIA GPUs (V100) on [OLCF Summit](https://www.olcf.ornl.gov/summit/) * Near compute roofline behavior * Compile-time case optimization (hard-codes parameters for significant speedup) * RDMA (remote data memory access; GPU-GPU direct communication) via GPU-aware MPI on NVIDIA (CUDA-aware MPI) and AMD GPU systems @@ -320,6 +205,18 @@ Ref. 1 includes all modern MFC features, including GPU acceleration and many new If referencing MFC's (GPU) performance, consider citing ref. 1 and 2, which describe the solver and its design. The original open-source release of MFC is ref. 3, which should be cited for provenance as appropriate. +

+ + DOI 10.1016/j.cpc.2026.110055 + + + Zenodo DOI + + + GitHub stars + +

+ ```bibtex @article{wilfong26, Author = {Benjamin Wilfong and Henry {Le Berre} and Anand Radhakrishnan and Ansh Gupta and Daniel J. Vickers and Diego Vaca-Revelo and Dimitrios Adam and Haocheng Yu and Hyeoksu Lee and Jose Rodolfo Chreim and Mirelys {Carcana Barbosa} and Yanjun Zhang and Esteban Cisneros-Garibay and Aswin Gnanaskandan and Mauro {Rodriguez Jr.} and Reuben D. Budiardja and Stephen Abbott and Tim Colonius and Spencer H. Bryngelson}, @@ -353,7 +250,7 @@ The original open-source release of MFC is ref. 3, which should be cited for pro ``` ## License - + Copyright 2021 Spencer Bryngelson and Tim Colonius. MFC is under the MIT license (see [LICENSE](LICENSE) for full text). @@ -367,3 +264,15 @@ MFC computations have used many supercomputing systems. A partial list is below * NCSA Delta and DeltaAI, PSC Bridges(1/2), SDSC Comet and Expanse, Purdue Anvil, TACC Stampede(1-3), and TAMU ACES via ACCESS-CI allocations from Bryngelson, Colonius, Rodriguez, and more. * DOD systems Blueback, Onyx, Carpenter, Nautilus, and Narwhal via the DOD HPCMP program. * Sandia National Labs systems Doom and Attaway, and testbed systems Weaver and Vortex. + +--- + +

+ + + + + Star History Chart + + +

diff --git a/docs/documentation/expectedPerformance.md b/docs/documentation/expectedPerformance.md index e5daec9496..132bdb01c3 100644 --- a/docs/documentation/expectedPerformance.md +++ b/docs/documentation/expectedPerformance.md @@ -165,22 +165,24 @@ __All grind times are in nanoseconds (ns) per grid point (gp) per equation (eq) Weak scaling results are obtained by increasing the problem size with the number of processes so that work per process remains constant. -### AMD MI250X GPU +### GPU weak scaling -MFC weask scales to (at least) 65,536 AMD MI250X GPUs on OLCF Frontier with 96% efficiency. -This corresponds to 87% of the entire machine. +MFC weak scales on multiple exascale GPU platforms with high efficiency: +- **LLNL El Capitan**: AMD MI300A APUs +- **OLCF Frontier**: AMD MI250X GPUs (65,536 GCDs, 87% of the machine, 96% efficiency) +- **CSCS Alps**: NVIDIA GH200 GPUs - + -### NVIDIA V100 GPU +### NVIDIA V100 GPU (historical) -MFC weak scales to (at least) 13,824 V100 NVIDIA V100 GPUs on OLCF Summit with 97% efficiency. +MFC weak scales to (at least) 13,824 NVIDIA V100 GPUs on OLCF Summit with 97% efficiency. This corresponds to 50% of the entire machine. -### IBM Power9 CPU -MFC Weak scales to 13,824 Power9 CPU cores on OLCF Summit to within 1% of ideal scaling. +### IBM Power9 CPU (historical) +MFC weak scales to 13,824 Power9 CPU cores on OLCF Summit to within 1% of ideal scaling. diff --git a/docs/index.html b/docs/index.html index 173ffdd8f9..90e34352c5 100644 --- a/docs/index.html +++ b/docs/index.html @@ -14,9 +14,15 @@ - MFC | Home - - + MFC | Exascale Multiphase Flow Solver + + + + + + + + @@ -34,8 +40,7 @@ ]; const scalings = [ - { image: "res/weakScaling/frontier.svg", label: "Oak Ridge Frontier (AMD MI250X GPUs)" }, - { image: "res/weakScaling/summit.svg", label: "Oak Ridge Summit (NVIDIA V100 GPUs)" }, + { image: "res/weakScaling/weakscaling.png", label: "Weak scaling on El Capitan (MI300A), Frontier (MI250X), and Alps (GH200)" }, ]; document.addEventListener("DOMContentLoaded", () => { @@ -71,7 +76,7 @@ `).join(""); document.getElementById("ft-scaling").innerHTML = scalings.map(s => ` -
+
${s.label}
@@ -118,8 +123,11 @@
-
+
An exascale compressible multiphase and multiphysics flow code. + + 2025 Gordon Bell Prize Finalist - 200+ trillion grid points on 43,000+ GPUs +
@@ -127,6 +135,7 @@ GitHub + GitHub stars @@ -198,7 +207,7 @@

Multiple federal sponsors have supported MFC development, including the US Department of Defense (DOD), the National Institutes of Health (NIH), the Department of Energy (DOE), and the National Science Foundation (NSF). - MFC computations use OLCF Frontier, Summit, and Wombat; LLNC LC Tuolumne and Lassen; and many ACCESS-CI systems + MFC computations use OLCF Frontier, Summit, and Wombat; LLNL LC El Capitan, Tuolumne, and Lassen; and many ACCESS-CI systems
diff --git a/docs/res/weakScaling/weakscaling-dark.png b/docs/res/weakScaling/weakscaling-dark.png new file mode 100644 index 0000000000..8bfdbd8529 Binary files /dev/null and b/docs/res/weakScaling/weakscaling-dark.png differ diff --git a/docs/res/weakScaling/weakscaling.png b/docs/res/weakScaling/weakscaling.png new file mode 100644 index 0000000000..e5b5c44a0c Binary files /dev/null and b/docs/res/weakScaling/weakscaling.png differ