Merge pull request #29 from opensolvers/site-updates

Sync benchmarks: GROMACS, ScaLAPACK, FFTW/QE

Author: hugomeiland

README.md

@@ -17,7 +17,7 @@ Each RISC-V board exposes several compute paths. We benchmark and tune them inde
 | **Specific** | Silicon-specific custom units beyond RVV | X60 **IME** (`smt.vmadot`) int8 on [RV2](boards/RV2.html) / [F3](boards/F3.html); [ONNX Runtime](apps/onnx.html) int4 via MLAS |
 | **GPU** | Discrete or integrated accelerators | On the roadmap — not yet in our RISC-V board benchmarks |
 
-Recent highlights on the Orange Pi RV2 (SpaceMiT X60, RVV): fixing an OpenBLAS `gemv_n` bug restores correctness across [BLAS](scientific-libs/blas.html), [LAPACK](scientific-libs/lapack.html), [ELPA](scientific-libs/elpa.html), [HPL](apps/hpl.html), and [Quantum ESPRESSO](apps/qe.html) — with patched RVV reaching **10.53 GFLOP/s** on Linpack, **1.58×** on a dense eigensolve, and **1.31×** on a 64-atom Si DFT SCF. [FFTW `r5v`](scientific-libs/fftw.html) beats scalar **1.06–1.60×** under `FFTW_MEASURE`; planner choice alone is worth **3–5×**. On the IME path, setting `accuracy_level=4` on ONNX `MatMulNBits` nodes unlocks **9–10×** int4 LLM decode — see [ONNX Runtime](apps/onnx.html) and [MLAS](scientific-libs/mlas.html).
+Recent highlights on the Orange Pi RV2 (SpaceMiT X60, RVV): fixing an OpenBLAS `gemv_n` bug restores correctness across [BLAS](scientific-libs/blas.html), [LAPACK](scientific-libs/lapack.html), [ELPA](scientific-libs/elpa.html), [ScaLAPACK](scientific-libs/scalapack.html), [HPL](apps/hpl.html), and [Quantum ESPRESSO](apps/qe.html). [FFTW `r5v`](scientific-libs/fftw.html) wins **1.06–1.60×** in isolation but **~0%** inside a real QE SCF (`FFTW_ESTIMATE`); [GROMACS](apps/gromacs.html) sees **1.23×** on isolated `PME 3D-FFT`. ONNX `accuracy_level=4` unlocks **9–10×** int4 decode — [ONNX Runtime](apps/onnx.html) / [MLAS](scientific-libs/mlas.html).
 
 ## Scientific libs
 
@@ -29,7 +29,8 @@ Library-level probes — performance *and* numerical correctness:
 - **[LAPACK](scientific-libs/lapack.html)** — LAPACK path via NumPy `eigvalsh`
 - **[ELPA](scientific-libs/elpa.html)** — dense eigensolver (CP2K / VASP class workloads)
 - **[MLAS](scientific-libs/mlas.html)** — ONNX Runtime QNBit int4 GEMM; isolated IME kernel rates on X60
-- **[FFTW](scientific-libs/fftw.html)** — RVV `r5v` backend A/B vs scalar; planner-aware `tests/bench` on X60
+- **[FFTW](scientific-libs/fftw.html)** — RVV `r5v` backend A/B; QE FFT-axis shows ~0% end-to-end despite micro wins
+- **[ScaLAPACK](scientific-libs/scalapack.html)** — distributed `PDSYEV`; stock RVV hangs, patched **1.09×**
 
 ## Apps
 
@@ -38,6 +39,7 @@ End-to-end application benchmarks on the same boards and EESSI toolchain:
 - **[HPL](apps/hpl.html)** — High Performance Linpack; cross-board summary and A/B configs from [opensolvers/benchmarks](https://github.com/opensolvers/benchmarks)
 - **[Quantum ESPRESSO](apps/qe.html)** — plane-wave DFT SCF (`pw.x`); whole-application BLAS backend A/B with per-routine timers
 - **[ONNX Runtime](apps/onnx.html)** — int4 `MatMulNBits` LLM decode; `accuracy_level=4` unlocks X60 IME (**9–10×**)
+- **[GROMACS](apps/gromacs.html)** — PME MD; FFT-axis A/B via `libfftw3f` `LD_PRELOAD` (**1.23×** on `PME 3D-FFT`)
 
 ## Boards
 

_config.yml

@@ -21,6 +21,8 @@ navigation_apps:
     url: /apps/qe.html
   - title: ONNX Runtime
     url: /apps/onnx.html
+  - title: GROMACS
+    url: /apps/gromacs.html
 
 navigation_scientific_libs:
   - title: BLAS
@@ -37,3 +39,5 @@ navigation_scientific_libs:
     url: /scientific-libs/mlas.html
   - title: FFTW
     url: /scientific-libs/fftw.html
+  - title: ScaLAPACK
+    url: /scientific-libs/scalapack.html

apps/gromacs.md

@@ -0,0 +1,63 @@
+# GROMACS
+
+[GROMACS](https://www.gromacs.org/) `mdrun` PME molecular dynamics — a **whole-application FFT backend A/B**. Swaps only the single-precision FFTW library (`libfftw3f.so.3`) under one unchanged binary via `LD_PRELOAD`; GROMACS comes from the EESSI module (`GROMACS/2026.2-foss-2025b`).
+
+Benchmark source: [opensolvers/benchmarks/gromacs](https://github.com/opensolvers/benchmarks/tree/main/gromacs) — `gen-water-box.sh` and `run-gmx-fft-ab.sh`.
+
+This is the **FFT-axis** companion to [Quantum ESPRESSO](qe.html)'s BLAS-axis probe. FFT microbench numbers: [FFTW](../scientific-libs/fftw.html).
+
+## Why GROMACS as an FFT probe
+
+PME splits long-range electrostatics into a real-space **`Force`** term and a reciprocal-space **mesh** part whose cost is a pair of 3D FFTs per step. On this build GROMACS reports **`SIMD instructions: None`** — force kernels are scalar — so swapping `libfftw3f` moves only the `PME 3D-FFT` / `PME mesh` rows in GROMACS's cycle table; `Force`, neighbor search, and constraints are built-in controls.
+
+## Correctness — RVV FFT reproduces scalar MD
+
+Same `mdrun`, same `md.tpr`, `libfftw3f` swapped (17.2k-atom SPC water, 2000 steps):
+
+| FFT backend | Avg potential energy | Δ vs scalar |
+| ----------- | -------------------: | ----------: |
+| Scalar (`libfftw3f`, ~944 KB) | **−236755 kJ/mol** | — |
+| r5v / RVV (`libfftw3f`, ~14 MB, 282k RVV instr) | **−236592 kJ/mol** | **0.069%** |
+
+Physically the same trajectory — expected single-precision rounding between codelet sets.
+
+## Performance — SPC water PME (Orange Pi RV2)
+
+Serial, 48×48×48 PME grid. BLAS pinned to scalar OpenBLAS via FlexiBLAS; only `LD_PRELOAD` swaps FFT:
+
+| Activity | Scalar | r5v (RVV) | Speedup | Kind |
+| -------- | -----: | --------: | ------: | ---- |
+| **`PME 3D-FFT`** | 21.095 s | 17.152 s | **1.23×** | **FFT (swapped axis)** |
+| `PME mesh` (total) | 73.281 s | 69.792 s | 1.05× | FFT + spread/gather/solve |
+| `PME solve Elec` | 9.116 s | 9.139 s | 1.00× | scalar (control) |
+| **`Force`** | 821.35 s | 819.97 s | 1.00× | **scalar kernels (90% of run)** |
+| `Neighbor search` | 6.731 s | 6.714 s | 1.00× | scalar (control) |
+
+The **`Force` row confirms the A/B** — 821.35 vs 819.97 s (0.17%, noise). RVV wins **~1.23×** on the isolated 3D-FFT step, but `Force` dominates the whole run.
+
+## Where GROMACS sits on the dilution spectrum
+
+| Probe | Axis | Isolated speedup | Whole-app effect |
+| ----- | ---- | ---------------: | ---------------- |
+| [DGEMM](../scientific-libs/dgemm.html) | BLAS | ~2.3× | (pure kernel) |
+| [QE](qe.html) (DFT SCF) | BLAS | ~1.5–2.0× on BLAS routines | ~1.2–1.3× (FFT half untouched) |
+| [QE + FFTW](../scientific-libs/fftw.html) | FFT | ~1.02× on `fftw` | **~0%** (`FFTW_ESTIMATE`) |
+| **GROMACS** (this page) | **FFT** | **~1.23× on `PME 3D-FFT`** | small (`Force` = 90%) |
+
+GROMACS is the mirror of QE: QE speeds up BLAS and is diluted by FFT; GROMACS speeds up FFT and is diluted by scalar force kernels.
+
+## Gotchas
+
+- GROMACS mixed precision links **`libfftw3f.so.3`** (single precision), not the double-precision `libfftw3.so.3` from the QE/FFTW work.
+- Fixed `-nsteps` keeps FFT call count identical between backends (4002 here).
+- RVV `libfftw3f` build is slow to compile (~2 h on one X60); scalar ~15 min.
+
+## Reproduce
+
+```bash
+module load GROMACS/2026.2-foss-2025b
+./gen-water-box.sh
+bash run-gmx-fft-ab.sh v1
+```
+
+**Toolchain:** GROMACS 2026.2 / foss-2025b (EESSI), FFTW 3.3.10 `libfftw3f`, Orange Pi RV2 (SpaceMiT X60).

apps/qe.md

@@ -37,7 +37,18 @@ The stock vector backend aborts in `inverse_s` (overlap-matrix inversion / Lowdi
 
 ### Default band count (`nbnd=136`)
 
-**67.6 s → 57.0 s = 1.19×** overall (`calbec` 1.90×, `rdiaghg` 1.53×). BLAS routines speed up ~1.5–2.0×; the FFT half of the run does not move with a FlexiBLAS swap — see [FFTW](../scientific-libs/fftw.html) for RVV FFT tuning on the same hardware (**1.06–1.60×** under `FFTW_MEASURE`, plus **3–5×** from planner choice).
+**67.6 s → 57.0 s = 1.19×** overall (`calbec` 1.90×, `rdiaghg` 1.53×). BLAS routines speed up ~1.5–2.0×; the FFT half does not move with a FlexiBLAS swap. Swapping the FFT library directly ([FFTW](../scientific-libs/fftw.html) `run-qe-fft-ab.sh`) also yields **~0% end-to-end** — QE uses `FFTW_ESTIMATE` on thousands of small transforms, not the `MEASURE` planner where r5v wins **1.06–1.60×** in isolation.
+
+## FFT axis — RVV FFTW also ~0% end-to-end
+
+Serial `pw.x` on [Orange Pi RV2](../boards/RV2.html), BLAS pinned to scalar OpenBLAS, FFT swapped via `LD_PRELOAD` ([`run-qe-fft-ab.sh`](https://github.com/opensolvers/benchmarks/blob/main/fftw/run-qe-fft-ab.sh)):
+
+| Timer | Scalar FFTW | r5v FFTW | Speedup |
+| ----- | ----------: | -------: | ------: |
+| `fftw` (~45% of run) | 112.24 s | 110.09 s | 1.019× |
+| **`PWSCF` (total)** | **248.49 s** | **248.10 s** | **1.002×** |
+
+Energy bit-identical. A microbenchmark FFT win does not automatically become an application win — see also [GROMACS](gromacs.html) for the FFT-axis mirror ( **1.23×** on isolated `PME 3D-FFT`, diluted by scalar `Force`).
 
 ## Where QE sits on the BLAS-dilution spectrum
 
@@ -50,7 +61,7 @@ Same X60, patched RVV vs scalar:
 | [ELPA](../scientific-libs/elpa.html) (eigensolver) | ~1.58× | BLAS-3 + BLAS-2 tridiagonalization |
 | **QE** (full DFT SCF) | **~1.2–1.3×** | BLAS + ~40–50% FFT + MPI |
 
-Each step down adds more non-BLAS / latency-bound work, diluting the BLAS-3 peak. QE is the most realistic whole-application figure — and the reason a raw `dgemm` A/B *overstates* what a real code sees.
+Each step down adds more non-BLAS / latency-bound work, diluting the BLAS-3 peak. And on the X60 today, **neither BLAS nor FFT drop-in swaps meaningfully move a real QE SCF** — BLAS wins are capped by the FFT half; FFT wins evaporate under `FFTW_ESTIMATE`.
 
 ## Reproducing
 

boards/RV2.md

@@ -48,4 +48,8 @@ Full walkthrough: [EESSI/docs#819](https://github.com/EESSI/docs/pull/819) — *
 
 ## FFTW RVV
 
-See [FFTW](../scientific-libs/fftw.html) — FFTW 3.3.10 with the `r5v` RVV backend vs scalar on this board. Under `FFTW_MEASURE`, r5v wins **1.06–1.60×**; `FFTW_ESTIMATE` under-plans large transforms by **3–5×**.
+See [FFTW](../scientific-libs/fftw.html) — r5v wins **1.06–1.60×** in `tests/bench`, but **~0%** end-to-end in [Quantum ESPRESSO](../apps/qe.html) (`FFTW_ESTIMATE`). [GROMACS](../apps/gromacs.html) FFT swap: **1.23×** on isolated `PME 3D-FFT`.
+
+## ScaLAPACK
+
+See [ScaLAPACK](../scientific-libs/scalapack.html) — `PDSYEV` on 2×4 grid: stock RVV **hangs**; patched RVV **107.23 s** vs scalar **116.87 s** (**1.09×**).

scientific-libs/elpa.md

@@ -16,7 +16,7 @@ Benchmark source: [opensolvers/benchmarks/elpa](https://github.com/opensolvers/b
 
 The stock vector backend is faster than scalar but **wrong** — the same OpenBLAS 0.3.30 `gemv_n` NaN bug that breaks [HPL](../apps/hpl.html) and [Quantum ESPRESSO](../apps/qe.html). With the patch, RVV is **1.58×** faster than scalar (54.92 / 34.81) and numerically correct.
 
-Speedup is smaller than pure `dgemm` (~2.3×) because tridiagonalization is latency-bound BLAS-2 — by design.
+Speedup is smaller than pure `dgemm` (~2.3×) because tridiagonalization is latency-bound BLAS-2 — by design. For the pure-MPI end of the spectrum see [ScaLAPACK](scalapack.html) (`PDSYEV`, **1.09×** patched RVV, stock RVV **hangs**).
 
 ### Banana Pi BPI-F3 (cross-board confirmation)
 

scientific-libs/fftw.md

@@ -4,7 +4,7 @@
 
 Benchmark source: [opensolvers/benchmarks/fftw](https://github.com/opensolvers/benchmarks/tree/main/fftw) — `build-fftw-r5v.sh` and `bench-fftw-ab.sh`.
 
-Relevant to [Quantum ESPRESSO](../apps/qe.html): plane-wave DFT spends a large fraction on FFT (`vloc_psi`, `fftw` timers). A FlexiBLAS swap does not touch FFT — tuning FFTW separately is how that half of the run moves.
+Relevant to [Quantum ESPRESSO](../apps/qe.html) and [GROMACS](../apps/gromacs.html): real apps spend large fractions on FFT. A FlexiBLAS swap does not touch FFT — swap the library via `LD_PRELOAD` instead (see `run-qe-fft-ab.sh` in the benchmarks repo).
 
 ## Orange Pi RV2 (SpaceMiT X60, 1 thread)
 
@@ -51,11 +51,37 @@ The biggest lever on this hardware is **planner choice** — worth **3–5×**,
 
 A first pass with `estimate` and `-t 0.3` appeared to show a **2× RVV regression** at N=262144. With honest timing, r5v is faster under both planners (717 vs 664 MFLOPS under MEASURE). Pin the planner and use **≥1 s** timing before trusting any single FFT A/B point.
 
+## End-to-end — Quantum ESPRESSO gets ~0% from RVV FFTW
+
+The microbench shows real RVV codelets (**1.06–1.60×**). Does that survive into a real DFT run? [`run-qe-fft-ab.sh`](https://github.com/opensolvers/benchmarks/blob/main/fftw/run-qe-fft-ab.sh) swaps **only** the FFT library (`LD_PRELOAD`) on serial `pw.x`, with BLAS pinned to scalar OpenBLAS via FlexiBLAS.
+
+**Correctness:** total energy bit-identical across backends (2-atom **−14.57861334 Ry**; 64-atom **−506.67991945 Ry**).
+
+64-atom SCF, WALL seconds ([Orange Pi RV2](../boards/RV2.html)):
+
+| Timer | Scalar FFTW | r5v (RVV) FFTW | r5v speedup |
+| ----- | ----------: | -------------: | ----------: |
+| `fftw` | 112.24 | 110.09 | **1.019×** |
+| `vloc_psi` | 105.29 | 103.08 | 1.021× |
+| **`PWSCF` (total)** | **248.49** | **248.10** | **1.002×** |
+
+**~0.2% wall, ~1.9% inside `fftw`** — even though `fftw` is ~45% of runtime. Why the 1.6× micro-win evaporates:
+
+- **QE plans with `FFTW_ESTIMATE`, not `MEASURE`.** The RVV advantage in section (2) is largely a planner effect; under `estimate` the two libs are near-parity — and `estimate` is what QE uses (it cannot afford `MEASURE` across thousands of transient transforms).
+- **3260 small mixed-radix transforms**, not the cache-resident power-of-two sizes where RVV shines.
+
+On the X60, **neither the BLAS axis nor the FFT axis moves a real QE SCF** with today's drop-in vectorized libraries. Contrast [GROMACS](../apps/gromacs.html): swapping `libfftw3f` wins **1.23×** on the isolated `PME 3D-FFT` step (but `Force` = 90% of the run).
+
+## EasyBuild module
+
+`FFTW-3.3.10-GCC-14.3.0-r5v.eb` packages the r5v build as a reproducible EESSI module (`--enable-r5v`, pinned `-march=rv64imafdcv_zvl256b`). Experimental — tracked on [hmeiland/easybuild-easyconfigs#3](https://github.com/hmeiland/easybuild-easyconfigs/pull/3), not yet upstream-ready.
+
 ## Reproduce
 
 ```bash
 ./build-fftw-r5v.sh    # needs $HOME/fftw-r5v.tar.gz on the board
 ./bench-fftw-ab.sh     # writes fftw-proper.log
+./run-qe-fft-ab.sh     # QE FFT-axis A/B (LD_PRELOAD)
 ```
 
 **Gotcha (Orange Pi RV2):** `module load GCCcore/14.3.0` does not repath `gcc` — EESSI's compat GCC 13.4.0 keeps winning. Prepend the real GCC 14 bindir explicitly (see `build-fftw-r5v.sh`). On [Banana Pi F3](../boards/F3.html) the plain `module load GCC/14.3.0` repaths correctly.

scientific-libs/scalapack.md

@@ -0,0 +1,38 @@
+# ScaLAPACK
+
+[ScaLAPACK](https://www.netlib.org/scalapack/) `PDSYEV` — a distributed dense real-symmetric eigenproblem on a 2D block-cyclic process grid. A **pure-MPI** proxy for node-local BLAS performance and correctness.
+
+Benchmark source: [opensolvers/benchmarks/scalapack](https://github.com/opensolvers/benchmarks/tree/main/scalapack) — `scalapack_bench.c`.
+
+Companion to [ELPA](elpa.html): same problem class, different library and parallelism model. ELPA runs one rank with threaded BLAS; ScaLAPACK runs many MPI ranks each with single-threaded BLAS.
+
+## Orange Pi RV2 (SpaceMiT X60, 8 cores)
+
+`na=3000`, `nb=64`, **2×4** grid, 8 ranks × 1 thread. Backends swapped via FlexiBLAS — no rebuild.
+
+| Backend | Time | Result |
+| ------- | ---: | ------ |
+| Stock CVMFS 0.3.30, default RVV (`ZVL256B`) | — | **HANG** (killed at 200 s) |
+| Scalar (`RISCV64_GENERIC`) | 116.87 s | finite=1 |
+| Patched RVV (`gemv_n` fix) | **107.23 s** | finite=1 (identical eigenvalues) |
+
+The unpatched vector `gemv_n` NaN does not just corrupt the result — it **hangs** the solver, because `pdsyev`'s serial tridiagonal QR iteration never converges on NaN. The patched build is correct and **1.09×** faster than scalar.
+
+## Why the speedup is modest
+
+Versus ~2.4× for raw `dgemm` and ~1.58× for threaded [ELPA](elpa.html) on the same machine: at 8 ranks the node-local BLAS blocks are small, MPI communication plus the serial tridiagonal solve dominate wall time. The RVV-accelerable BLAS-3 fraction is small — the conservative, communication-bound end of the BLAS-backend spectrum.
+
+## Reproduce
+
+```bash
+module load ScaLAPACK/2.2.2-gompi-2025b-fb
+make
+
+OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 OPENBLAS_CORETYPE=RISCV64_GENERIC \
+  mpirun --bind-to core -np 8 ./scalapack_bench 3000 64 2
+
+OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 FLEXIBLAS=/path/to/patched/libopenblas.so \
+  mpirun --bind-to core -np 8 ./scalapack_bench 3000 64 2
+```
+
+**Toolchain:** ScaLAPACK 2.2.2 / gompi-2025b-fb (EESSI), OpenBLAS 0.3.30 with `gemv_n` fix.

website.md

@@ -19,16 +19,16 @@ Context file for structural and content decisions on [opensolvers.com](https://w
 
 1. **Home**
 2. **Boards** — VisionFive 2, OrangePi RV2, BananaPi F3
-3. **Apps** — HPL, Quantum ESPRESSO, ONNX Runtime
-4. **Scientific libs** — BLAS, DGEMM, NumPy, LAPACK, ELPA, MLAS, FFTW
+3. **Apps** — HPL, Quantum ESPRESSO, ONNX Runtime, GROMACS
+4. **Scientific libs** — BLAS, DGEMM, NumPy, LAPACK, ELPA, MLAS, FFTW, ScaLAPACK
 
 Nav config: `_config.yml` (`navigation`, `navigation_boards`, `navigation_apps`, `navigation_scientific_libs`). Rendered in `_includes/header.html`. Cayman theme requires `_layouts/default.html` override to include the header.
 
 ## Content sources
 
 | Source | Used for |
 |--------|----------|
-| [opensolvers/benchmarks](https://github.com/opensolvers/benchmarks) | BLAS (`dgemm/`), HPL (`hpl/`), ELPA (`elpa/`), QE (`qe/`), NumPy/LAPACK (`numpy/`), ONNX/MLAS (`onnx/`), FFTW RVV (`fftw/`) |
+| [opensolvers/benchmarks](https://github.com/opensolvers/benchmarks) | BLAS (`dgemm/`), HPL (`hpl/`), ELPA/ScaLAPACK (`elpa/`, `scalapack/`), QE/FFTW (`qe/`, `fftw/`), GROMACS (`gromacs/`), ONNX/MLAS (`onnx/`), NumPy (`numpy/`) |
 | [EESSI/docs#818](https://github.com/EESSI/docs/pull/818) | VisionFive 2 / U74 OpenBLAS + HPL |
 | [EESSI/docs#819](https://github.com/EESSI/docs/pull/819) | Orange Pi RV2 / X60 RVV `gemv_n` fix + HPL |
 | [easyconfigs#26436](https://github.com/easybuilders/easybuild-easyconfigs/pull/26436) | U74 OpenBLAS package |
@@ -44,6 +44,7 @@ Nav config: `_config.yml` (`navigation`, `navigation_boards`, `navigation_apps`,
 
 | Date | Decision |
 |------|----------|
+| 2026-07-18 | Sync from benchmarks: GROMACS app, ScaLAPACK lib, FFTW QE end-to-end (~0%), updated QE/elpa/RV2 |
 | 2026-07-17 | Add FFTW `r5v` RVV page from `benchmarks/fftw`; cross-links from QE and RV2 |
 | 2026-07-17 | Add MLAS (lib) and ONNX Runtime (app) pages from `benchmarks/onnx` |
 | 2026-07-15 | Vector box: add ZVL256B and ZVL128B subtitles |