|
1 | 1 | # GROMACS |
2 | 2 |
|
3 | | -[GROMACS](https://www.gromacs.org/) `mdrun` PME molecular dynamics — a **whole-application FFT backend A/B**. Swaps only the single-precision FFTW library (`libfftw3f.so.3`) under one unchanged binary via `LD_PRELOAD`; GROMACS comes from the EESSI module (`GROMACS/2026.2-foss-2025b`). |
| 3 | +[GROMACS](https://www.gromacs.org/) `mdrun` PME molecular dynamics on the SpaceMiT X60 — two independent backend axes measured end-to-end: |
4 | 4 |
|
5 | | -Benchmark source: [opensolvers/benchmarks/gromacs](https://github.com/opensolvers/benchmarks/tree/main/gromacs) — `gen-water-box.sh` and `run-gmx-fft-ab.sh`. |
| 5 | +1. **FFT axis** — swap single-precision `libfftw3f.so.3` via `LD_PRELOAD` (stock EESSI build, `SIMD: None`) |
| 6 | +2. **Force axis** — hand-written **`impl_riscv_rvv/`** SIMD backend (`SIMD: RISCV_RVV`) or upstream 2026.3 autovec 1×1 kernel |
6 | 7 |
|
7 | | -This is the **FFT-axis** companion to [Quantum ESPRESSO](qe.html)'s BLAS-axis probe. FFT microbench numbers: [FFTW](../scientific-libs/fftw.html). |
| 8 | +Benchmark source: [opensolvers/benchmarks/gromacs](https://github.com/opensolvers/benchmarks/tree/main/gromacs) — `gen-water-box.sh`, `run-gmx-fft-ab.sh`, and [`rvv-backend/`](https://github.com/opensolvers/benchmarks/tree/main/gromacs/rvv-backend) (patch + `apply-and-build.sh`). |
8 | 9 |
|
9 | | -## Why GROMACS as an FFT probe |
| 10 | +FFT microbench numbers: [FFTW](../scientific-libs/fftw.html). BLAS-axis counterpart: [Quantum ESPRESSO](qe.html). |
10 | 11 |
|
11 | | -PME splits long-range electrostatics into a real-space **`Force`** term and a reciprocal-space **mesh** part whose cost is a pair of 3D FFTs per step. On this build GROMACS reports **`SIMD instructions: None`** — force kernels are scalar — so swapping `libfftw3f` moves only the `PME 3D-FFT` / `PME mesh` rows in GROMACS's cycle table; `Force`, neighbor search, and constraints are built-in controls. |
| 12 | +--- |
12 | 13 |
|
13 | | -## Correctness — RVV FFT reproduces scalar MD |
| 14 | +## FFT axis — PME 3D-FFT (`run-gmx-fft-ab.sh`) |
14 | 15 |
|
15 | | -Same `mdrun`, same `md.tpr`, `libfftw3f` swapped (17.2k-atom SPC water, 2000 steps): |
| 16 | +Swaps only `libfftw3f` under one unchanged EESSI `GROMACS/2026.2-foss-2025b` binary. BLAS pinned to scalar OpenBLAS; serial (`-ntmpi 1 -ntomp 1`) by design to isolate FFT. |
| 17 | + |
| 18 | +On the stock build GROMACS reports **`SIMD instructions: None`** — force kernels are scalar, so `PME 3D-FFT` / `PME mesh` are the swapped rows; `Force` (90% of runtime) is the built-in control. |
| 19 | + |
| 20 | +### Correctness — RVV FFT reproduces scalar MD |
| 21 | + |
| 22 | +Same `mdrun`, same `md.tpr`, 17.2k-atom SPC water, 2000 steps: |
16 | 23 |
|
17 | 24 | | FFT backend | Avg potential energy | Δ vs scalar | |
18 | 25 | | ----------- | -------------------: | ----------: | |
19 | 26 | | Scalar (`libfftw3f`, ~944 KB) | **−236755 kJ/mol** | — | |
20 | 27 | | r5v / RVV (`libfftw3f`, ~14 MB, 282k RVV instr) | **−236592 kJ/mol** | **0.069%** | |
21 | 28 |
|
22 | | -Physically the same trajectory — expected single-precision rounding between codelet sets. |
| 29 | +### Performance — isolated FFT step (Orange Pi RV2) |
23 | 30 |
|
24 | | -## Performance — SPC water PME (Orange Pi RV2) |
25 | | - |
26 | | -Serial, 48×48×48 PME grid. BLAS pinned to scalar OpenBLAS via FlexiBLAS; only `LD_PRELOAD` swaps FFT: |
| 31 | +Serial, 48×48×48 PME grid. GROMACS cycle accounting (WALL): |
27 | 32 |
|
28 | 33 | | Activity | Scalar | r5v (RVV) | Speedup | Kind | |
29 | 34 | | -------- | -----: | --------: | ------: | ---- | |
30 | 35 | | **`PME 3D-FFT`** | 21.095 s | 17.152 s | **1.23×** | **FFT (swapped axis)** | |
31 | 36 | | `PME mesh` (total) | 73.281 s | 69.792 s | 1.05× | FFT + spread/gather/solve | |
32 | | -| `PME solve Elec` | 9.116 s | 9.139 s | 1.00× | scalar (control) | |
33 | | -| **`Force`** | 821.35 s | 819.97 s | 1.00× | **scalar kernels (90% of run)** | |
34 | | -| `Neighbor search` | 6.731 s | 6.714 s | 1.00× | scalar (control) | |
| 37 | +| **`Force`** | 821.35 s | 819.97 s | 1.00× | scalar control (90%) | |
| 38 | +| `Neighbor search` | 6.731 s | 6.714 s | 1.00× | scalar control | |
| 39 | + |
| 40 | +RVV wins **~1.23×** on the isolated 3D-FFT step; **`Force` confirms the A/B** (821.35 vs 819.97 s, 0.17% noise). Whole-app effect is small because `Force` dominates. |
| 41 | + |
| 42 | +--- |
| 43 | + |
| 44 | +## Force axis — RVV SIMD backend (`rvv-backend/`) |
| 45 | + |
| 46 | +The scalar `Force` term that dilutes the FFT swap is itself optimizable. [`rvv-backend/`](https://github.com/opensolvers/benchmarks/tree/main/gromacs/rvv-backend) ships a hand-written **`impl_riscv_rvv/`** backend for GROMACS **2026.2** (~1545 lines, float/mixed precision) — ARM SVE as the VLA template. |
| 47 | + |
| 48 | +Built on the X60 under EESSI GCC 14.3 with `GMX_SIMD=RISCV_RVV`. `gmx --version` reports **`SIMD instructions: RISCV_RVV`**; float SIMD unit tests **158/158** pass. |
| 49 | + |
| 50 | +Same serial protocol (SPC water PME, `md.tpr`, 2000 steps, `-ntmpi 1 -ntomp 1 -nb cpu -pme cpu`): |
| 51 | + |
| 52 | +| Metric | Scalar (`SIMD: None`) | RVV (`SIMD: RISCV_RVV`) | Speedup | |
| 53 | +| ------ | --------------------: | ------------------------: | ------: | |
| 54 | +| **`Force` wall** | 821.35 s | **187.37 s** | **4.38×** | |
| 55 | +| Total wall | 910.33 s | **275.40 s** | **3.31×** | |
| 56 | +| Performance | 0.380 ns/day | **1.256 ns/day** | **3.31×** | |
| 57 | +| Avg potential energy | −236755 kJ/mol | −236653 kJ/mol | Δ 0.043% (correct) | |
35 | 58 |
|
36 | | -The **`Force` row confirms the A/B** — 821.35 vs 819.97 s (0.17%, noise). RVV wins **~1.23×** on the isolated 3D-FFT step, but `Force` dominates the whole run. |
| 59 | +**The profile inverts:** with `Force` cut ~4.4×, PME mesh becomes the leading cost (28% vs 8% of runtime) — so the FFT axis matters more after vectorizing `Force`. |
37 | 60 |
|
38 | | -## Where GROMACS sits on the dilution spectrum |
| 61 | +Status: captured as a patch for reproduction; **no upstream PR yet**. Double precision not ported. |
| 62 | + |
| 63 | +--- |
| 64 | + |
| 65 | +## Force axis — three levers compared |
| 66 | + |
| 67 | +All measured on Orange Pi RV2, same SPC water system, 2000 steps, 1 core (unless noted): |
| 68 | + |
| 69 | +| Force lever | GROMACS | Mechanism | `Force` wall | ns/day | vs scalar | |
| 70 | +| ----------- | ------- | --------- | -----------: | -----: | --------: | |
| 71 | +| Scalar 4×4 (baseline) | 2026.2 | none (`SIMD: None`) | 821.35 s | 0.380 | 1.00× | |
| 72 | +| GCC `-march=…v` 4×4 autovec | 2026.2 | compiler on 4×4 kernel | ~836 s* | 0.372 | **0.98× (slower)** | |
| 73 | +| **Upstream 1×1 autovec** | **2026.3** | clang `#pragma`, `GMX_NBNXN_PLAINC_1X1=1` | **480.14 s** | **0.543** | **1.43×** | |
| 74 | +| **Hand-written `impl_riscv_rvv/`** | **2026.2** | `GMX_SIMD=RISCV_RVV` | **187.37 s** | **1.256** | **3.31×** | |
| 75 | + |
| 76 | +<sub>*4×4 autovec Force scaled from 1000-step run; ns/day directly comparable.</sub> |
| 77 | + |
| 78 | +The **2026.3 autovec 1×1** path (clang ≥ 19, `GMX_ENABLE_NBNXM_CPU_VECTORIZATION=on`) is a real win — **1.74× whole-app** vs the same binary's default 4×4 — but the hand-written backend is still **2.31× faster** whole-app (1.256 vs 0.543 ns/day). Autovec is VLEN-agnostic and upstream-native; the dedicated backend targets fixed VLEN=256 (`-mrvv-vector-bits=zvl`). |
| 79 | + |
| 80 | +**Free lever:** threading — serial A/B wastes 7 of 8 cores. Same system at `-ntomp 8` reaches **~2.60 ns/day** (~6.9× vs serial baseline). |
| 81 | + |
| 82 | +--- |
| 83 | + |
| 84 | +## Backend-dilution spectrum |
39 | 85 |
|
40 | 86 | | Probe | Axis | Isolated speedup | Whole-app effect | |
41 | 87 | | ----- | ---- | ---------------: | ---------------- | |
42 | 88 | | [OpenBLAS verification](../scientific-libs/blas.html#verification) | BLAS | ~2.3× | (pure kernel) | |
43 | | -| [QE](qe.html) (DFT SCF) | BLAS | ~1.5–2.0× on BLAS routines | ~1.2–1.3× (FFT half untouched) | |
| 89 | +| [BLIS](../scientific-libs/blis.html) vs OpenBLAS | BLAS | ~1.29× (1T, N=4096) | (microbench) | |
| 90 | +| [QE](qe.html) (DFT SCF) | BLAS | ~1.5–2.0× on BLAS routines | ~1.2–1.3× (FFT untouched) | |
44 | 91 | | [QE + FFTW](../scientific-libs/fftw.html) | FFT | ~1.02× on `fftw` | **~0%** (`FFTW_ESTIMATE`) | |
45 | | -| **GROMACS** (this page) | **FFT** | **~1.23× on `PME 3D-FFT`** | small (`Force` = 90%) | |
| 92 | +| **GROMACS FFT** (this page) | FFT | **~1.23× on `PME 3D-FFT`** | small (`Force` = 90%) | |
| 93 | +| **GROMACS Force** (`rvv-backend/`) | **SIMD** | **~4.38× on `Force`** | **~3.31×** | |
46 | 94 |
|
47 | | -GROMACS is the mirror of QE: QE speeds up BLAS and is diluted by FFT; GROMACS speeds up FFT and is diluted by scalar force kernels. |
| 95 | +GROMACS FFT is the mirror of QE: QE speeds up BLAS and is diluted by FFT; the FFT swap is diluted by scalar `Force`. Vectorizing `Force` moves the whole app — then PME/FFT becomes the next target. |
| 96 | + |
| 97 | +--- |
48 | 98 |
|
49 | 99 | ## Gotchas |
50 | 100 |
|
51 | | -- GROMACS mixed precision links **`libfftw3f.so.3`** (single precision), not the double-precision `libfftw3.so.3` from the QE/FFTW work. |
| 101 | +- GROMACS mixed precision links **`libfftw3f.so.3`** (single precision), not double-precision `libfftw3.so.3` from QE/FFTW work. |
52 | 102 | - Fixed `-nsteps` keeps FFT call count identical between backends (4002 here). |
53 | 103 | - RVV `libfftw3f` build is slow to compile (~2 h on one X60); scalar ~15 min. |
| 104 | +- GCC `-march=rv64gcv_zvl256b` on stock 4×4 `Force` emits 164k RVV instrs but runs **~2% slower** — gather-heavy kernel defeats autovec. |
| 105 | +- 2026.3 autovec 1×1 needs **clang ≥ 19** and runtime **`GMX_NBNXN_PLAINC_1X1=1`**; confirm via `Using plain-C-1x1` in `md.log`. |
| 106 | +- RVV-backend builds need GCC 14.3's `libstdc++` (`CXXABI_1.3.15`) on `LD_LIBRARY_PATH` at runtime. |
54 | 107 |
|
55 | 108 | ## Reproduce |
56 | 109 |
|
57 | 110 | ```bash |
| 111 | +# FFT axis |
58 | 112 | module load GROMACS/2026.2-foss-2025b |
59 | 113 | ./gen-water-box.sh |
60 | 114 | bash run-gmx-fft-ab.sh v1 |
| 115 | + |
| 116 | +# Force axis (hand-written RVV backend) |
| 117 | +cd gromacs/rvv-backend |
| 118 | +./apply-and-build.sh /path/to/gromacs-2026.2 |
61 | 119 | ``` |
62 | 120 |
|
63 | 121 | **Toolchain:** GROMACS 2026.2 / foss-2025b (EESSI), FFTW 3.3.10 `libfftw3f`, Orange Pi RV2 (SpaceMiT X60). |
0 commit comments