diff --git a/BLIS/README.md b/BLIS/README.md new file mode 100644 index 0000000..81dc4ac --- /dev/null +++ b/BLIS/README.md @@ -0,0 +1,147 @@ +# BLIS on RISC-V - build, validation & performance vs OpenBLAS + +Builds [BLIS](https://github.com/flame/blis) (BLAS-like Library Instantiation +Software, from the FLAME group) for the **SpaceMiT X60 / K1** (Orange Pi RV2 / +BPI-F3, `rv64gcv`, RVV 1.0, VLEN=256) under the [EESSI](https://www.eessi.io/) +GCC 14.3.0 toolchain, verifies it is numerically correct, and A/Bs its BLAS +throughput against OpenBLAS. + +BLIS ships **hand-written RVV assembly** level-3 microkernels under +[`kernels/rviv/3/`](https://github.com/flame/blis/tree/master/kernels/rviv/3) +(dynamic-VLEN via `get_vlenb()`), selected by the `rv64iv` config target - so +this is a real vector-vs-vector comparison, not vector-vs-scalar. + +| File | What it does | +|---|---| +| `build-blis.sh` | Clones + configures (`rv64iv`, `--enable-cblas --enable-blas`) + builds BLIS with EESSI GCC 14.3.0. Emits `libblis.a` + CBLAS/BLAS symbols. | +| `run-ab.sh` | Links the backend-agnostic `bench_dgemm.c` against BLIS **and** OpenBLAS, sweeps sizes × threads, prints GFLOP/s side-by-side + a ratio. | +| `bench_dgemm.c` | Square `C=A*B` (level-3 `dgemm`, 3 reps), reports GFLOP/s, prints `C[0]` so a NaN/garbage backend is visible. *(shared with [`../OpenBLAS/`](../OpenBLAS))* | +| `difftest.c` | `dlopen`s a BLAS `.so` (or link a `.a`) and runs level-1/2/3 routines on fixed inputs, printing `sum / sumsq / nan / inf` per routine for differential comparison against a reference. | +| `verify_ctrsm.c` | Full-parameter TRSM correctness sweep (residual-checked); confirms BLIS's triangular solve matches the reference. | + +## Why link, not FlexiBLAS + +The other BLAS-axis dirs ([`../OpenBLAS`](../OpenBLAS), [`../hpl`](../hpl), +[`../numpy`](../numpy)) A/B by swapping the backend at runtime via FlexiBLAS. +**FlexiBLAS is not installed on the RV2**, so here the A/B is done by *linking* +the same unchanged `bench_dgemm.c` against each library in turn. This still +honours the repo principle - change exactly one variable (the BLAS +implementation), hold source and compiler flags constant. + +## Build + +```bash +# on the Orange Pi RV2 (native build); ~10-20 min for a full BLIS build +./build-blis.sh # installs to $HOME/blis-install +# or: ./build-blis.sh /custom/prefix +``` + +`build-blis.sh` forces EESSI **GCC 14.3.0** ahead of the EESSI compat GCC 13.4.0 +(see the gotcha below) and configures the RVV 1.0 target with OpenMP threading: + +```bash +./configure --prefix=$PREFIX --enable-cblas --enable-blas \ + --enable-threading=openmp rv64iv +make -j$(nproc) && make install +``` + +> `rv64iv` = RVV 1.0 with **dynamic VLEN detection** (correct for the X60's +> VLEN=256). Do **not** use `sifive_rvv`, which defaults to VLEN=128. +> `--enable-threading=openmp` is required for BLIS to use all 8 cores; without +> it DGEMM stays single-threaded (~2.7 GFLOP/s regardless of thread count). + +## Run + +```bash +# performance A/B (BLIS vs OpenBLAS) +BLIS_PREFIX=$HOME/blis-install \ +OPENBLAS_LIB=$HOME/trsm-pr5830/libopenblas.a \ + ./run-ab.sh +# tune the sweep: SIZES="1024 2048 4096" THREADS="1 8" ./run-ab.sh + +# correctness of the BLIS build (threaded lib needs -fopenmp + the GCC14 flags) +GCC14=/cvmfs/dev.eessi.io/riscv/versions/2025.06-001/software/linux/riscv64/generic/software/GCCcore/14.3.0 +gcc -O3 -march=rv64imafdcv_zvl256b -fopenmp verify_ctrsm.c -o verify_ctrsm \ + -I$HOME/blis-install/include/blis -L$GCC14/lib -B$GCC14/lib \ + $HOME/blis-install/lib/libblis.a -lm -lpthread +OMP_NUM_THREADS=8 ./verify_ctrsm # exit 0 = all pass +``` + +## A/B methodology + +- **One variable**: identical `bench_dgemm.c`, identical `-O3 + -march=rv64imafdcv_zvl256b`, only the linked BLAS differs. +- **OpenBLAS baseline**: the patched RVV OpenBLAS from this repo's work + (`$HOME/trsm-pr5830/libopenblas.a`, `ZVL128B`/`ZVL256B` RVV kernels). +- **Threads**: 1 (single-core kernel efficiency) and 8 (full X60 parallel). +- **Correctness first**: `verify_ctrsm` / `difftest` must pass before any speed + number is trusted - a fast NaN is worthless. + +## Results + +Measured on the Orange Pi RV2 (SpaceMiT X60, 8 cores, VLEN=256), BLIS +`061c2eb` (`rv64iv`, OpenMP threading) vs the patched RVV OpenBLAS +`0.3.33.dev` (`zvl128bp`), both `-O3 -march=rv64imafdcv_zvl256b`, EESSI +GCC 14.3.0. DGEMM GFLOP/s (square, 3 reps, best): + +| threads | N | BLIS | OpenBLAS | BLIS/OpenBLAS | +|--:|--:|--:|--:|--:| +| 1 | 1024 | 1.99 | 2.13 | 0.93x | +| 1 | 2048 | 2.73 | 2.25 | 1.21x | +| 1 | 4096 | 2.95 | 2.28 | 1.29x | +| 8 | 1024 | 8.93 | 10.13 | 0.88x | +| 8 | 2048 | 9.60 | 10.83 | 0.89x | +| 8 | 4096 | 9.55 | 11.94 | 0.80x | + +Takeaways: + +- **Single thread, large N**: BLIS's hand-written RVV assembly gemm microkernel + (`kernels/rviv/3/`) **beats OpenBLAS by ~20-30%** once the problem is big + enough to amortize packing (N>=2048). At N=1024 the two are within noise. +- **8 threads**: OpenBLAS scales slightly better (BLIS reaches 0.80-0.89x). Both + get ~3.5-5x from 8 cores; BLIS's OpenMP path leaves some headroom vs + OpenBLAS's threading. +- `C[0]=245.24` was identical for both backends at every size - no NaN/garbage. + +Correctness (`verify_ctrsm`, BLIS `rv64iv`): **2400 cases, 0 fails, +worst_resid=2.55e-07** (single- and 8-thread). BLIS's triangular solve is +numerically correct on the X60 - unlike the stock OpenBLAS `_rvv_v1` TRSM kernel +this repo's [`../OpenBLAS`](../OpenBLAS) work localized. + +## Gotcha - `module load` does not repath `gcc` on the Orange Pi RV2 + +Same trap as [`../fftw/README.md`](../fftw/README.md): `module load +GCCcore/14.3.0` returns rc=0 but does **not** put GCC 14 first on `PATH` - the +EESSI compat-layer GCC 13.4.0 keeps winning. Both scripts here force it +explicitly, *before* any `set -e` (the lmod `module` function reads unbound vars +and would kill a strict shell): + +```bash +GCC14=/cvmfs/dev.eessi.io/riscv/versions/2025.06-001/software/linux/riscv64/generic/software/GCCcore/14.3.0 +export PATH="$GCC14/bin:$PATH" +export LD_LIBRARY_PATH="$GCC14/lib64:$GCC14/lib:$LD_LIBRARY_PATH" +``` + +## Gotcha - GCC14 runtime libs are off the default linker search path + +The shared `libblis.so` link (and any `-fopenmp` link against the static lib) +fails with the EESSI GCC 14.3.0 unless you point the driver at `$GCC14/lib`, +where its `libgcc_s`, `libatomic`, and `libgomp.spec` actually live: + +``` +/usr/bin/ld: cannot find -lgcc_s / -latomic # → needs -L$GCC14/lib +gcc: fatal error: cannot read spec file 'libgomp.spec' # → needs -B$GCC14/lib +``` + +Both scripts pass `-L$GCC14/lib -B$GCC14/lib -Wl,-rpath,$GCC14/lib` on every +link. `-L` resolves the libs, `-B` lets the gcc driver read `libgomp.spec` for +`-fopenmp`, and the rpath lets the resulting binary find `libgomp.so.1` at run +time. (The static-linked `bench_dgemm` still needs the rpath because BLIS's +OpenMP calls pull in `libgomp` dynamically.) + +## References + +- BLIS RISC-V config: [`config_registry`](https://github.com/flame/blis/blob/master/config_registry) (`rv64iv`, `sifive_rvv`) +- RVV assembly gemm kernels: [`kernels/rviv/3/`](https://github.com/flame/blis/tree/master/kernels/rviv/3) +- BLIS RISC-V CI: [`ci/do_riscv.sh`](https://github.com/flame/blis/blob/master/ci/do_riscv.sh) +- Prior RISC-V work: PR [#737](https://github.com/flame/blis/pull/737) (SiFive X280), [#832](https://github.com/flame/blis/pull/832) (generic RVV, VLEN-tunable), [#868](https://github.com/flame/blis/pull/868) (SG2042, +49% sdgemm) diff --git a/BLIS/bench_dgemm.c b/BLIS/bench_dgemm.c new file mode 100644 index 0000000..9c04b4f --- /dev/null +++ b/BLIS/bench_dgemm.c @@ -0,0 +1,36 @@ +/* + * bench_dgemm.c - minimal DGEMM throughput benchmark. + * + * Times a square double-precision C = A*B (level-3 BLAS dgemm, 3 reps) and + * reports GFLOP/s. Links against any BLAS that provides the Fortran dgemm_ + * symbol; use FlexiBLAS (or OPENBLAS_CORETYPE / OPENBLAS_NUM_THREADS) to select + * and A/B the backend without recompiling. Prints C[0] so a NaN/garbage backend + * is immediately visible. + * + * Build: gcc -O2 bench_dgemm.c -o bench_dgemm -lflexiblas (or -lopenblas) + * Run: OPENBLAS_NUM_THREADS=8 ./bench_dgemm [N] # default N=4096 + * A/B: scalar : OPENBLAS_CORETYPE=RISCV64_GENERIC ./bench_dgemm + * vector : FLEXIBLAS=/path/to/libopenblas.so ./bench_dgemm + * SPDX-License-Identifier: MIT + */ +#include +#include +#include +extern void dgemm_(const char*,const char*,const int*,const int*,const int*, + const double*,const double*,const int*,const double*,const int*, + const double*,double*,const int*); +static double now(){struct timespec t;clock_gettime(CLOCK_MONOTONIC,&t);return t.tv_sec+t.tv_nsec*1e-9;} +int main(int argc,char**argv){ + int n=argc>1?atoi(argv[1]):4096; + double *A=malloc(8L*n*n),*B=malloc(8L*n*n),*C=malloc(8L*n*n); + if(!A||!B||!C){printf("alloc fail\n");return 1;} + for(long i=0;i<(long)n*n;i++){A[i]=(i%7)*0.1;B[i]=(i%5)*0.2;C[i]=0;} + double al=1.0,be=0.0; + dgemm_("N","N",&n,&n,&n,&al,A,&n,B,&n,&be,C,&n); + int reps=3; double t0=now(); + for(int r=0;r/dev/null || true +echo "cblas symbols present?" +nm -g "$PREFIX"/lib/libblis.a 2>/dev/null | grep -c -E "cblas_dgemm|cblas_dtrsm" || true +echo +echo "BLIS built. Next: ./run-ab.sh (bench vs OpenBLAS) and ./verify (correctness)." diff --git a/BLIS/difftest.c b/BLIS/difftest.c new file mode 100644 index 0000000..0d304f8 --- /dev/null +++ b/BLIS/difftest.c @@ -0,0 +1,111 @@ +/* + * difftest.c - differential BLAS correctness test (single-threaded). + * + * dlopen()s a given BLAS .so and calls a spread of level-1/2/3 routines on + * fixed pseudo-random inputs, printing sum / sumsq / NaN-count / Inf-count for + * each. Point it at two backends (or one backend under two OPENBLAS_CORETYPE + * settings) and diff the output to localize a numerically broken kernel. + * + * This is the test that isolated the OpenBLAS 0.3.30 RISC-V vector gemv_n bug: + * on a SpaceMiT X60 the stock DYNAMIC_ARCH build's RVV (ZVL256B) cblas_dgemv + * returns NaN (VFILL_ZERO zeroes an uninitialized vector register) while + * dgemm/dtrsm are correct - which is exactly why HPL (leans on dgemv in panel + * factorization) NaNs while a standalone dgemm benchmark looks fine. Forcing + * OPENBLAS_CORETYPE=RISCV64_GENERIC, or using a fixed/patched RVV build, makes + * dgemv match the scalar reference bit-for-bit (nan=0). + * + * Build: gcc -O2 difftest.c -o difftest -ldl -lm + * Run: ./difftest /path/to/libopenblas.so + * OPENBLAS_CORETYPE=RISCV64_GENERIC ./difftest /path/to/libopenblas.so + * Requires a backend exposing the cblas_* symbols (OpenBLAS does). + * SPDX-License-Identifier: MIT + */ +#define _GNU_SOURCE +#include +#include +#include +#include +#include + +enum {ColMajor=102,RowMajor=101,NoTrans=111,Trans=112,Upper=121,Lower=122, + Left=141,Right=142,NonUnit=131,Unit=132}; + +typedef void (*dgemm_t)(int,int,int,int,int,int,double,const double*,int,const double*,int,double,double*,int); +typedef void (*dtrsm_t)(int,int,int,int,int,int,int,double,const double*,int,double*,int); +typedef void (*dgemv_t)(int,int,int,int,double,const double*,int,const double*,int,double,double*,int); +typedef void (*dscal_t)(int,double,double*,int); +typedef void (*daxpy_t)(int,double,const double*,int,double*,int); +typedef size_t (*idamax_t)(int,const double*,int); +typedef double (*dnrm2_t)(int,const double*,int); +typedef double (*ddot_t)(int,const double*,int,const double*,int); +typedef void (*setnt_t)(int); + +static double rnd(long i){ + unsigned long x=(unsigned long)i*2654435761UL+1013904223UL; + x^=x>>13; x*=0xD1B54A32D192ED03UL; x^=x>>29; + return ((double)(x>>40)/(double)(1UL<<24))*2.0-1.0; /* [-1,1) */ +} + +static void report(const char*name,const double*v,long n){ + double s=0,s2=0; long nan=0,inf=0; + for(long i=0;ij?rnd((long)j*tm+i)*0.1:0.0); + T[(size_t)j*tm+i]=v; } + double*B=malloc((size_t)tm*tn*sizeof(double)); + for(long i=0;i<(long)tm*tn;i++)B[i]=rnd(i+13); + cblas_dtrsm(ColMajor,Left,Lower,NoTrans,NonUnit,tm,tn,1.0,T,tm,B,tm); report("dtrsm",B,(long)tm*tn); + + printf("DONE\n"); + return 0; +} diff --git a/BLIS/run-ab.sh b/BLIS/run-ab.sh new file mode 100755 index 0000000..76bda01 --- /dev/null +++ b/BLIS/run-ab.sh @@ -0,0 +1,63 @@ +#!/bin/bash +# run-ab.sh - DGEMM performance A/B: BLIS vs OpenBLAS on the SpaceMiT X60 / K1. +# +# No FlexiBLAS on this board, so the A/B is done by LINKING the same +# backend-agnostic bench_dgemm.c against each library in turn - one variable +# changed (the BLAS implementation), everything else held constant. Both builds +# use the same compiler flags and the same source. +# +# Usage: +# BLIS_PREFIX=$HOME/blis-install \ +# OPENBLAS_LIB=$HOME/trsm-pr5830/libopenblas.a \ +# ./run-ab.sh +# +# Env: +# BLIS_PREFIX BLIS install prefix (expects $BLIS_PREFIX/lib/libblis.a) [required] +# OPENBLAS_LIB path to an OpenBLAS static lib or .so [required] +# SIZES space-separated N list (default: 512 1024 2048 4096) +# THREADS space-separated thread counts (default: 1 8) + +# --- EESSI GCC 14.3.0 (force ahead of EESSI compat gcc 13; see ../README.md) --- +GCC14=/cvmfs/dev.eessi.io/riscv/versions/2025.06-001/software/linux/riscv64/generic/software/GCCcore/14.3.0 +export PATH="$GCC14/bin:$PATH" +export LD_LIBRARY_PATH="$GCC14/lib64:$GCC14/lib:$LD_LIBRARY_PATH" + +set -euo pipefail + +BLIS_PREFIX=${BLIS_PREFIX:-$HOME/blis-install} +OPENBLAS_LIB=${OPENBLAS_LIB:-$HOME/trsm-pr5830/libopenblas.a} +SIZES=${SIZES:-"512 1024 2048 4096"} +THREADS=${THREADS:-"1 8"} +CFLAGS="-O3 -march=rv64imafdcv_zvl256b -fopenmp" +# GCC14's runtime libs (libgcc_s, libatomic, libgomp.spec) live in $GCC14/lib, +# off the default search path; -L finds the libs, -B lets the driver read +# libgomp.spec for -fopenmp. Without these the OpenMP-threaded link fails. +LDPATH="-L$GCC14/lib -B$GCC14/lib -Wl,-rpath,$GCC14/lib" + +BLIS_LIB="$BLIS_PREFIX/lib/libblis.a" +[ -f "$BLIS_LIB" ] || { echo "ERROR: BLIS lib not found: $BLIS_LIB (run ./build-blis.sh)"; exit 1; } +[ -f "$OPENBLAS_LIB" ] || { echo "ERROR: OpenBLAS lib not found: $OPENBLAS_LIB"; exit 1; } + +echo "=== toolchain ==="; gcc --version | head -1 +echo "BLIS : $BLIS_LIB" +echo "OpenBLAS : $OPENBLAS_LIB" + +# --- build two binaries from the SAME source, differing only in the BLAS lib --- +# BLIS static lib needs -lm -lpthread; OpenBLAS static likewise (+ gfortran runtime). +gcc $CFLAGS bench_dgemm.c -o /tmp/bench_blis $LDPATH "$BLIS_LIB" -lm -lpthread +gcc $CFLAGS bench_dgemm.c -o /tmp/bench_openblas $LDPATH "$OPENBLAS_LIB" -lm -lpthread -lgfortran + +echo +printf "%-8s %-8s | %-14s | %-14s | %-8s\n" threads N BLIS_GFLOPs OpenBLAS_GFLOPs ratio +echo "-----------------------------------------------------------------------" +for th in $THREADS; do + export OPENBLAS_NUM_THREADS=$th BLIS_NUM_THREADS=$th OMP_NUM_THREADS=$th + for n in $SIZES; do + bo=$(/tmp/bench_blis "$n" 2>&1 | grep -oE "DGEMM=[0-9.]+" | cut -d= -f2) + oo=$(/tmp/bench_openblas "$n" 2>&1 | grep -oE "DGEMM=[0-9.]+" | cut -d= -f2) + ratio=$(awk -v b="$bo" -v o="$oo" 'BEGIN{if(o>0)printf "%.2fx", b/o; else print "n/a"}') + printf "%-8s %-8s | %-14s | %-14s | %-8s\n" "$th" "$n" "$bo" "$oo" "$ratio" + done +done +echo +echo "ratio = BLIS / OpenBLAS (>1 means BLIS faster). Check C[0] in raw output for NaN sanity." diff --git a/BLIS/verify_ctrsm.c b/BLIS/verify_ctrsm.c new file mode 100644 index 0000000..aeb7d97 --- /dev/null +++ b/BLIS/verify_ctrsm.c @@ -0,0 +1,101 @@ +/* + * verify_ctrsm.c - CTRSM (complex single) numerical-correctness sweep. + * + * Solves op(A) X = alpha B (Left) or X op(A) = alpha B (Right) for a triangular + * A across the full parameter space - {Left,Right} x {Upper,Lower} x + * {NoTrans,Trans,ConjTrans} x {NonUnit,Unit} x a size grid - then recomputes + * op(A)*X (or X*op(A)) and checks the max residual against the original B. + * Used to validate the RISC-V RVV (_rvv_v1) TRSM kernels: the VLEN-agnostic fix + * in OpenMathLib/OpenBLAS#5928 passes 0 fails here on both ZVL128B (GEMM_UNROLL_M + * =8) and ZVL256B (=16); the pre-fix kernels fail on ZVL128B (unroll 8 != the + * VSETVL_MAX=16 the kernel tiled packed-A by). + * + * The S/D/Z variants are structurally identical - swap the type, the cblas_?trsm + * call, and the element generator (real vs complex). + * + * Build: gcc -O2 verify_ctrsm.c -o verify_ctrsm -I -lopenblas -lm + * Run: ./verify_ctrsm # exit 0 = all pass, prints "N cases, F fails" + * SPDX-License-Identifier: MIT + */ +#include +#include +#include +#include +#include "cblas.h" + +typedef float _Complex cd; +static double frand(void){ return (double)rand()/RAND_MAX - 0.5; } +static cd crand(void){ return frand() + frand()*I; } + +static void make_tri(cd *A, int n, int uplo_upper, int unit){ + for(int j=0;jj)) v = 0.3*crand(); + else v = 0.0; + A[i+j*n]=v; + } +} + +static cd opA(int trans,int uplo,int is_unit,const cd*A,int lda,int r,int c){ + /* returns element (r,c) of op(A) given stored triangular A */ + int sr,sc; + if(trans==CblasNoTrans){ sr=r; sc=c; } + else if(trans==CblasTrans){ sr=c; sc=r; } + else { sr=c; sc=r; } /* ConjTrans handled by caller via conj */ + cd a; + if(sr==sc) a = is_unit ? (1.0+0.0*I) : A[sr+sc*lda]; + else if((uplo==CblasUpper && srsc)) a=A[sr+sc*lda]; + else a=0.0; + if(trans==CblasConjTrans) a = conj(a); + return a; +} + +static double resid(int side,int uplo,int trans,int is_unit,int m,int n, + const cd*A,int lda,const cd*X,const cd*B0,int ldb){ + double maxr=0.0; + if(side==CblasLeft){ + for(int i=0;imaxr) maxr=d; + } + } else { + for(int i=0;imaxr) maxr=d; + } + } + return maxr; +} + +int main(void){ + int grid[]={1,2,4,7,8,9,10,12,15,23}; + int ng=sizeof(grid)/sizeof(grid[0]); + int sides[]={CblasLeft,CblasRight}, uplos[]={CblasUpper,CblasLower}; + int transs[]={CblasNoTrans,CblasTrans,CblasConjTrans}, diags[]={CblasNonUnit,CblasUnit}; + int fails=0,total=0; double worst=0.0; srand(999); + cd alpha=1.0+0.0*I; + for(int si=0;si<2;si++) for(int ui=0;ui<2;ui++) for(int ti=0;ti<3;ti++) for(int di=0;di<2;di++) + for(int im=0;imworst) worst=r; + if(!(r<=tol)||isnan(r)){ fails++; + if(fails<=30) + printf("FAIL side=%d uplo=%d trans=%d unit=%d m=%d n=%d resid=%g tol=%g\n", + side,uplo,trans,unit,m,n,r,tol); } + free(A);free(B);free(B0); + } + printf("=== CTRSM: %d cases, %d fails, worst_resid=%g ===\n", total, fails, worst); + return fails?1:0; +} diff --git a/README.md b/README.md index 49a0aa4..871dbef 100644 --- a/README.md +++ b/README.md @@ -21,6 +21,7 @@ correctness (finite / bit-identical results) is checked in every case. | Dir | What it measures | Axis swapped | Kind | |---|---|---|---| | [`OpenBLAS/`](OpenBLAS) | OpenBLAS on RISC-V: DGEMM performance + differential correctness + TRSM sweep; localizes two broken RVV kernels (`gemv_n` NaN, `_rvv_v1` TRSM VLEN bug) | BLAS | microbench + verification | +| [`BLIS/`](BLIS) | BLIS (FLAME) RVV build on RISC-V + DGEMM performance A/B vs OpenBLAS | BLAS | microbench + verification | | [`numpy/`](numpy) | BLAS/LAPACK backend as seen through NumPy/SciPy | BLAS | application proxy | | [`hpl/`](hpl) | High-Performance Linpack, end-to-end | BLAS (FlexiBLAS) | application A/B | | [`elpa/`](elpa) | Dense real-symmetric eigensolver (ELPA, 1-stage) | BLAS | microbench | @@ -38,7 +39,7 @@ Several directories deliberately pair a **microbenchmark** with a **real-application** measurement of the *same* backend, because they often disagree — and that disagreement is the interesting result: -- **BLAS axis:** [`OpenBLAS`](OpenBLAS)/[`numpy`](numpy)/[`elpa`](elpa) (kernel level) +- **BLAS axis:** [`OpenBLAS`](OpenBLAS)/[`BLIS`](BLIS)/[`numpy`](numpy)/[`elpa`](elpa) (kernel level) → [`hpl`](hpl)/[`qe`](qe) (whole application). - **FFT axis:** [`fftw`](fftw) standalone microbench (RVV wins **1.06–1.60×**) → the same RVV FFTW dropped into a Quantum ESPRESSO SCF (**~0% end-to-end**,