Merge pull request #40 from opensolvers/site/benchmarks-hpl-blis-ime

Sync benchmarks PR #21–22: HPL-on-BLIS + IME1 scale-build

Author: hugomeiland

README.md

@@ -30,14 +30,14 @@ Each RISC-V board exposes several compute paths. We benchmark and tune them inde
 | **Custom** | Custom ISA extensions beyond standard RVV | X60 **IME** / **XsmtVdot** (`smt.vmadot`) int8 on [RV2](boards/RV2.html) / [F3](boards/F3.html); [ONNX Runtime](apps/onnx.html) int4 via MLAS |
 | **GPU** | Integrated Imagination GPUs (OpenCL / Vulkan) | **IMG BXE-4-32 MC1** on [VisionFive 2](boards/VisionFive2.html) (JH7110); **IMG BXE-2-32** on [RV2](boards/RV2.html) / [F3](boards/F3.html) (K1) — present on silicon, **GPGPU benchmarks not yet** |
 
-Recent highlights on the Orange Pi RV2 (SpaceMiT X60, RVV): fixing an OpenBLAS `gemv_n` bug restores correctness across [BLAS](scientific-libs/blas.html), [LAPACK](scientific-libs/lapack.html), [ELPA](scientific-libs/elpa.html), [ScaLAPACK](scientific-libs/scalapack.html), [HPL](apps/hpl.html), and [Quantum ESPRESSO](apps/qe.html). [BLIS](scientific-libs/blis.html) RVV assembly beats patched OpenBLAS **~1.29×** on single-thread DGEMM (N=4096). [FFTW `r5v`](scientific-libs/fftw.html) wins **1.06–1.60×** in isolation but **~0%** inside a real QE SCF; [GROMACS](apps/gromacs.html) sees **1.23×** on isolated `PME 3D-FFT` and **3.31×** whole-app with a hand-written RVV `Force` backend. ONNX `accuracy_level=4` unlocks **9–10×** int4 decode — [ONNX Runtime](apps/onnx.html) / [MLAS](scientific-libs/mlas.html).
+Recent highlights on the Orange Pi RV2 (SpaceMiT X60, RVV): fixing an OpenBLAS `gemv_n` bug restores correctness across [BLAS](scientific-libs/blas.html), [LAPACK](scientific-libs/lapack.html), [ELPA](scientific-libs/elpa.html), [ScaLAPACK](scientific-libs/scalapack.html), [HPL](apps/hpl.html), and [Quantum ESPRESSO](apps/qe.html). [BLIS](scientific-libs/blis.html) RVV assembly beats patched OpenBLAS **~1.29×** on single-thread DGEMM (N=4096), but HPL linked to BLIS is correct yet only **0.35–0.53×** OpenBLAS-RVV. [FFTW `r5v`](scientific-libs/fftw.html) wins **1.06–1.60×** in isolation but **~0%** inside a real QE SCF; [GROMACS](apps/gromacs.html) sees **1.23×** on isolated `PME 3D-FFT` and **3.31×** whole-app with a hand-written RVV `Force` backend. ONNX `accuracy_level=4` unlocks **9–10×** int4 decode — [ONNX Runtime](apps/onnx.html) / [MLAS](scientific-libs/mlas.html). llama.cpp IME1 scale-build opt: **+4.3%** pp512 microkernel, bit-exact ([RV2 IME](boards/RV2.html#ime-integer-matrix-extension)).
 
 ## Scientific libs
 
 Library-level probes — performance *and* numerical correctness:
 
 - **[BLAS](scientific-libs/blas.html)** — OpenBLAS improvements (U74 kernel, X60 `gemv_n` / TRSM fixes) and [`OpenBLAS/`](https://github.com/opensolvers/benchmarks/tree/main/OpenBLAS) verification (`bench_dgemm`, `difftest`, `verify_ctrsm`)
-- **[BLIS](scientific-libs/blis.html)** — FLAME BLIS `rv64iv` RVV vs patched OpenBLAS; **1.29×** DGEMM at N=4096 (1 thread)
+- **[BLIS](scientific-libs/blis.html)** — FLAME BLIS `rv64iv` RVV vs patched OpenBLAS; **1.29×** DGEMM at N=4096 (1 thread); HPL end-to-end **0.35–0.53×** OpenBLAS-RVV
 - **[NumPy](scientific-libs/numpy.html)** — `bench_blas.py` DGEMM and `eigvalsh` through the SciPy stack
 - **[LAPACK](scientific-libs/lapack.html)** — LAPACK path via NumPy `eigvalsh`
 - **[ELPA](scientific-libs/elpa.html)** — dense eigensolver (CP2K / VASP class workloads)
@@ -49,15 +49,15 @@ Library-level probes — performance *and* numerical correctness:
 
 End-to-end application benchmarks on the same boards and EESSI toolchain:
 
-- **[HPL](apps/hpl.html)** — High Performance Linpack; cross-board summary and A/B configs from [opensolvers/benchmarks](https://github.com/opensolvers/benchmarks)
+- **[HPL](apps/hpl.html)** — High Performance Linpack; OpenBLAS A/B + BLIS-linked validation from [opensolvers/benchmarks](https://github.com/opensolvers/benchmarks)
 - **[Quantum ESPRESSO](apps/qe.html)** — plane-wave DFT SCF (`pw.x`); whole-application BLAS backend A/B with per-routine timers
 - **[ONNX Runtime](apps/onnx.html)** — int4 `MatMulNBits` LLM decode; `accuracy_level=4` unlocks X60 IME (**9–10×**)
 - **[GROMACS](apps/gromacs.html)** — PME MD; FFT-axis **1.23×** on `PME 3D-FFT`; RVV `Force` backend **3.31×** whole-app
 
 ## Boards
 
 - **[StarFive VisionFive 2](boards/VisionFive2.html)** — JH7110 SoC, 4× SiFive U74 (`rv64gc`). U74 OpenBLAS tuning: HPL **3.13 → 5.28 GFLOP/s**.
-- **[Orange Pi RV2](boards/RV2.html)** — SpaceMiT K1, 8× X60 (RVV). Fixed OpenBLAS: HPL **FAILED (`nan`) → 10.53 GFLOP/s**; BLIS DGEMM **1.29×** vs OpenBLAS; GROMACS Force **3.31×**; ELPA **34.81 s** (vs 54.92 s scalar).
+- **[Orange Pi RV2](boards/RV2.html)** — SpaceMiT K1, 8× X60 (RVV). Fixed OpenBLAS: HPL **FAILED (`nan`) → 10.53 GFLOP/s**; BLIS DGEMM **1.29×** / HPL **0.35–0.53×**; IME1 scale-build **+4.3%**; GROMACS Force **3.31×**; ELPA **34.81 s** (vs 54.92 s scalar).
 - **[Banana Pi F3](boards/F3.html)** — same K1 / X60 SoC, **3.7 GB RAM**. HPL **FAILED (`nan`) → 11.52 GFLOP/s**; NumPy DGEMM up to **17.51 GFLOP/s** on patched RVV.
 
 Use the menu above to jump to a board, app, or scientific lib page.

apps/hpl.md

@@ -48,6 +48,30 @@ Same `xhpl`, backend swapped via FlexiBLAS. All **PASSED** with the fixed vector
 
 A squarer **2×4** grid beats 1×8 for peak throughput. `HPL_big.dat` needs ~6.6 GB RAM — tight on 8 GB boards.
 
+### HPL on BLIS — end-to-end validation
+
+Unlike the FlexiBLAS OpenBLAS A/B above, BLIS cannot be swapped at runtime on the RV2 (no FlexiBLAS). A dedicated `xhpl` is linked statically against RVV `libblis.a` (`Make.rv64_blis`, `build-hpl-blis.sh`). Source: [benchmarks/hpl](https://github.com/opensolvers/benchmarks/tree/main/hpl).
+
+This validates that RVV BLIS drives a real Linpack solve — not just square DGEMM — because HPL's panel path leans on `dgemv`/`dtrsv` (the same reason stock OpenBLAS NaN'd).
+
+| Config | BLIS GFLOP/s | Residual | vs patched OpenBLAS-RVV |
+| ------ | -----------: | -------- | ----------------------: |
+| `HPL.dat` (N=8000, 1×8) | 4.02 | 4.12e-03 PASSED | **0.35×** |
+| `HPL-sweep.dat` (N=20000, 2×4) | 5.57 | 3.39e-03 PASSED | **0.53×** |
+
+#### Full-memory process-grid sweep (N=25600)
+
+Largest square problem that fits (~5.2 GB). N=28672 OOMs on this 8 GB no-swap board (rank 7 signal 9).
+
+| Grid | GFLOP/s | Time | Residual |
+| ---- | ------: | ---: | -------- |
+| **2×4** | **5.87** | 1904.5 s | 3.37e-03 PASSED |
+| 1×8 | 5.84 | 1916.0 s | 2.84e-03 PASSED |
+| 4×2 | 5.23 | 2140.5 s | 4.07e-03 PASSED |
+| 8×1 | 4.31 | 2592.9 s | 5.75e-03 PASSED |
+
+**Takeaway:** correctness holds (no NaN, residual on par with OpenBLAS). Throughput trails patched-RVV OpenBLAS — HPL is panel-heavy (thin k=256, `dtrsm`, level-2), not the large square DGEMM where BLIS wins ~1.2–1.3× single-thread. Wide grids beat tall (8×1 −26% vs 2×4). See [BLIS](../scientific-libs/blis.html).
+
 ## VisionFive 2
 
 Scalar U74 — stock OpenBLAS uses a generic kernel; the U74-tuned build lifts HPL **3.13 → 5.28 GFLOP/s** (**1.69×**). Walkthrough: [EESSI/docs#818](https://github.com/EESSI/docs/pull/818).

apps/onnx.md

@@ -4,7 +4,7 @@
 
 Benchmark source: [opensolvers/benchmarks/onnx](https://github.com/opensolvers/benchmarks/tree/main/onnx) — Llama-7B-proportioned FFN stack, `patch_accuracy_level.py`, and `onnxruntime_perf_test` runners.
 
-Library-level kernel numbers: [MLAS](../scientific-libs/mlas.html). Raw IME microbenchmarks: [Orange Pi RV2](../boards/RV2.html#ime-integer-matrix-extension).
+Library-level kernel numbers: [MLAS](../scientific-libs/mlas.html). Raw IME microbenchmarks: [Orange Pi RV2](../boards/RV2.html#ime-integer-matrix-extension). Related (llama.cpp, not ORT): IME1 scale-build prefill patch **+4.3%** on the isolated Q4_0 GEMM — see [RV2 IME](../boards/RV2.html#ime1-scale-build-prefill-optimization-llamacpp).
 
 ## The workload
 

boards/F3.md

@@ -14,7 +14,7 @@ Cross-board confirmation from [opensolvers/benchmarks](https://github.com/openso
 
 Same K1 / X60 silicon as the Orange Pi RV2 — each chip has SpaceMiT's **IME** int8 matrix unit (`smt.vmadot`) on **cluster 0** (cores 0–3, **512 KB shared L2**). The IME is separate from RVV: one instruction accumulates a **4×4 int32** tile from **4×8 int8** operands.
 
-Detailed `ime-bench` numbers were measured on the [RV2](RV2.html) (peak **~42 GOP/s** int8 GEMM vs ~5 GOP/s RVV int8, **~7–8×**). The BPI-F3 has the same IME hardware; pin to core 0 when running [benchmarks/ime](https://github.com/opensolvers/benchmarks/tree/main/ime). End-to-end int4 inference: [ONNX Runtime](../apps/onnx.html) and [MLAS](../scientific-libs/mlas.html).
+Detailed `ime-bench` numbers were measured on the [RV2](RV2.html) (peak **~42 GOP/s** int8 GEMM vs ~5 GOP/s RVV int8, **~7–8×**; IME1 scale-build **+4.3%** on llama.cpp Q4_0). The BPI-F3 has the same IME hardware; pin to core 0 when running [benchmarks/ime](https://github.com/opensolvers/benchmarks/tree/main/ime). End-to-end int4 inference: [ONNX Runtime](../apps/onnx.html) and [MLAS](../scientific-libs/mlas.html).
 
 ## HPL
 

boards/RV2.md

@@ -28,6 +28,21 @@ Microbenchmarks in [opensolvers/benchmarks/ime](https://github.com/opensolvers/b
 
 Peak **~42 GOP/s** single-core — vs ~5 GOP/s for a straightforward RVV int8 path. End-to-end int4 LLM decode through [ONNX Runtime](../apps/onnx.html) and isolated [MLAS](../scientific-libs/mlas.html) kernel rates use the same IME hardware; see also [papers/x60-ime-block-scale-optimization](https://github.com/opensolvers/benchmarks/blob/main/papers/x60-ime-block-scale-optimization.md) in the benchmarks repo.
 
+### IME1 scale-build prefill optimization (llama.cpp)
+
+llama.cpp's block-scaled Q4_0 kernel (`gemm_kernel_i8i4`) pays a per-block FP scale tax (~31–37% vs raw `s8s8s32`). Patch [`llama-ime1-scalebuild-opt.patch`](https://github.com/opensolvers/benchmarks/blob/main/ime/llama-ime1-scalebuild-opt.patch) rebuilds `As×Bs` scales with `vfmul.vv` (`LOAD_SCALE_4x16_FP16_OPT`) instead of the masked `vfmul.vf` chain.
+
+Isolated interleaved A/B on this board (30 rounds, cluster-0 pinned, performance governor, pp512 = 512³) — [benchmarks/ime](https://github.com/opensolvers/benchmarks/tree/main/ime):
+
+| | stock | scaleopt |
+| --- | ---: | ---: |
+| median GOP/s | 22.50 | **23.47** |
+| range | 22.4–22.7 | 23.2–23.6 |
+
+**+4.3%** (+0.97 GOP/s); scaleopt faster in **30/30** rounds; zero overlap; paired t = 50.3. Bit-exact (`sum`/`sumsq`/`max` identical across three shapes).
+
+The kernel win is real; end-to-end `llama-bench` pp512 on this multi-tenant part is **not resolvable above ±15–20% noise** (see the paper linked above). Decode (`tg`) uses the untouched M1/GEMV path.
+
 ## HPL via EESSI
 
 See also the [HPL app overview](../apps/hpl.html).
@@ -60,6 +75,8 @@ Full walkthrough: [EESSI/docs#819](https://github.com/EESSI/docs/pull/819) — *
 
 See [BLIS](../scientific-libs/blis.html) — FLAME BLIS `rv64iv` with hand-written RVV assembly vs patched OpenBLAS on the same `bench_dgemm.c`. Single-thread DGEMM at N=4096: **2.95 vs 2.28 GFLOP/s** (**1.29×** BLIS); 8-thread OpenBLAS still leads at large N. TRSM: **2400 cases, 0 fails** on BLIS.
 
+End-to-end [HPL on BLIS](../apps/hpl.html#hpl-on-blis--end-to-end-validation): all **PASSED**, but **0.35–0.53×** OpenBLAS-RVV (best full-memory **5.87 GFLOP/s** at N=25600, 2×4) — the DGEMM win does not carry to Linpack.
+
 ## FFTW RVV
 
 See [FFTW](../scientific-libs/fftw.html) — r5v wins **1.06–1.60×** in `tests/bench`, but **~0%** end-to-end in [Quantum ESPRESSO](../apps/qe.html) (`FFTW_ESTIMATE`).

scientific-libs/blis.md

@@ -10,7 +10,7 @@ See also [BLAS (OpenBLAS)](blas.html) for the OpenBLAS fixes and verification su
 
 ## Why link, not FlexiBLAS
 
-Other BLAS-axis benchmarks ([HPL](../apps/hpl.html), [NumPy](numpy.html), [QE](../apps/qe.html)) swap backends at runtime via FlexiBLAS. **FlexiBLAS is not installed on the RV2**, so this A/B links the same unchanged `bench_dgemm.c` against each library in turn — still one variable (the BLAS implementation), identical `-O3 -march=rv64imafdcv_zvl256b`.
+Other BLAS-axis benchmarks ([HPL](../apps/hpl.html), [NumPy](numpy.html), [QE](../apps/qe.html)) swap backends at runtime via FlexiBLAS. **FlexiBLAS is not installed on the RV2**, so this A/B links the same unchanged `bench_dgemm.c` against each library in turn — still one variable (the BLAS implementation), identical `-O3 -march=rv64imafdcv_zvl256b`. The same constraint applies end-to-end: [HPL on BLIS](../apps/hpl.html#hpl-on-blis--end-to-end-validation) builds a dedicated `xhpl` against static `libblis.a`.
 
 ## Build & run
 
@@ -53,6 +53,12 @@ BLIS `061c2eb` (`rv64iv`, OpenMP) vs patched RVV OpenBLAS `0.3.33.dev` (`zvl128b
 - **8 threads:** OpenBLAS scales slightly better (BLIS **0.80–0.89×**). Both get ~3.5–5× from 8 cores; BLIS's OpenMP path leaves headroom vs OpenBLAS threading.
 - **Correctness first:** both backends numerically identical on DGEMM; BLIS TRSM verified independently.
 
+## End-to-end: HPL on BLIS
+
+Linking HPL against the same RVV `libblis.a` ([HPL app](../apps/hpl.html#hpl-on-blis--end-to-end-validation)): all configs **PASSED**, but BLIS is **0.35–0.53×** patched OpenBLAS-RVV (4.02 / 5.57 GFLOP/s; full-memory best **5.87** at N=25600, 2×4).
+
+The 1T square-DGEMM advantage does **not** predict Linpack: HPL hits skinny rank-k / `dtrsm` / `dgemv` shapes where BLIS RVV trails. Details and grid sweep: [benchmarks/hpl](https://github.com/opensolvers/benchmarks/tree/main/hpl).
+
 ## References
 
 - BLIS RISC-V config: [`config_registry`](https://github.com/flame/blis/blob/master/config_registry) (`rv64iv`, `sifive_rvv`)

scientific-libs/mlas.md

@@ -4,7 +4,7 @@
 
 Benchmark source: [opensolvers/benchmarks/onnx](https://github.com/opensolvers/benchmarks/tree/main/onnx) — [`bench_qnbit_mlas.cpp`](https://github.com/opensolvers/benchmarks/blob/main/onnx/bench_qnbit_mlas.cpp) links directly against `libonnxruntime_mlas.a` and calls the same entry points as ORT's `matmul_nbits.cc`.
 
-See also the [ONNX Runtime app](../apps/onnx.html) for end-to-end int4 LLM-FFN inference and the [IME microbenchmarks](../boards/RV2.html#ime-integer-matrix-extension) on the Orange Pi RV2.
+See also the [ONNX Runtime app](../apps/onnx.html) for end-to-end int4 LLM-FFN inference and the [IME microbenchmarks](../boards/RV2.html#ime-integer-matrix-extension) on the Orange Pi RV2. Related (llama.cpp path): [IME1 scale-build +4.3%](../boards/RV2.html#ime1-scale-build-prefill-optimization-llamacpp).
 
 ## What it probes
 

website.md

@@ -44,7 +44,7 @@ Catalog: `_data/videos.yml` (newest first). Rendered by `_includes/video-grid.ht
 
 | Source | Used for |
 |--------|----------|
-| [opensolvers/benchmarks](https://github.com/opensolvers/benchmarks) | OpenBLAS (`OpenBLAS/`), **BLIS (`BLIS/`)**, HPL (`hpl/`), ELPA/ScaLAPACK (`elpa/`, `scalapack/`), QE/FFTW (`qe/`, `fftw/`), GROMACS (`gromacs/` incl. `rvv-backend/`), ONNX/MLAS (`onnx/`), NumPy (`numpy/`) |
+| [opensolvers/benchmarks](https://github.com/opensolvers/benchmarks) | OpenBLAS (`OpenBLAS/`), **BLIS (`BLIS/`)**, HPL (`hpl/` incl. BLIS-linked validation), ELPA/ScaLAPACK (`elpa/`, `scalapack/`), QE/FFTW (`qe/`, `fftw/`), GROMACS (`gromacs/` incl. `rvv-backend/`), ONNX/MLAS (`onnx/`), NumPy (`numpy/`), **IME (`ime/` scale-build opt + paper)** |
 | [EESSI/docs#818](https://github.com/EESSI/docs/pull/818) | VisionFive 2 / U74 OpenBLAS + HPL |
 | [EESSI/docs#819](https://github.com/EESSI/docs/pull/819) | Orange Pi RV2 / X60 RVV `gemv_n` fix + HPL |
 | [easyconfigs#26436](https://github.com/easybuilders/easybuild-easyconfigs/pull/26436) | U74 OpenBLAS package |
@@ -60,6 +60,7 @@ Catalog: `_data/videos.yml` (newest first). Rendered by `_includes/video-grid.ht
 
 | Date | Decision |
 |------|----------|
+| 2026-07-19 | Sync benchmarks PR #21–22: HPL-on-BLIS end-to-end (0.35–0.53× OpenBLAS); IME1 scale-build **+4.3%** pp512 |
 | 2026-07-19 | Publish X60 gemv_n / HPL NaN video (`W_-8cKA-CCU`) — `_data/videos.yml`, homepage, RV2, HPL, BLAS |
 | 2026-07-19 | Sync benchmarks PR #19–20: add **BLIS** page; expand **GROMACS** with RVV Force backend (3.31×) and 2026.3 autovec lever |
 | 2026-07-19 | Nav: OpenSolvers title links home; Home/Videos/YouTube on separate row from Apps |