Add MLAS and ONNX Runtime pages from benchmarks repo.

Document int4 MatMulNBits IME path on X60: 9–10× from accuracy_level=4, isolated MLAS kernel rates, nav and cross-links from homepage and board pages.

Co-authored-by: Cursor <cursoragent@cursor.com>

Author: hugomeiland

README.md

@@ -14,10 +14,10 @@ Each RISC-V board exposes several compute paths. We benchmark and tune them inde
 | ---- | ---------- | ---------------------- |
 | **Scalar** | Scalar ISA and portable C kernels — the correctness baseline | `rv64gc` on VisionFive 2; `OPENBLAS_CORETYPE=RISCV64_GENERIC`; U74 **4×4 DGEMM** tuning |
 | **Vector** | ISA vector extensions (RVV) in shared libs | OpenBLAS `RISCV64_ZVL256B` on SpaceMiT X60 (VLEN=256); `ZVL128B` where VLEN=128; the `gemv_n` bug we fixed |
-| **Specific** | Silicon-specific custom units beyond RVV | X60 **IME** (`smt.vmadot`) int8 on [RV2](boards/RV2.html) / [F3](boards/F3.html) |
+| **Specific** | Silicon-specific custom units beyond RVV | X60 **IME** (`smt.vmadot`) int8 on [RV2](boards/RV2.html) / [F3](boards/F3.html); [ONNX Runtime](apps/onnx.html) int4 via MLAS |
 | **GPU** | Discrete or integrated accelerators | On the roadmap — not yet in our RISC-V board benchmarks |
 
-Recent highlights on the Orange Pi RV2 (SpaceMiT X60, RVV): fixing an OpenBLAS `gemv_n` bug restores correctness across [BLAS](scientific-libs/blas.html), [LAPACK](scientific-libs/lapack.html), [ELPA](scientific-libs/elpa.html), [HPL](apps/hpl.html), and [Quantum ESPRESSO](apps/qe.html) — with patched RVV reaching **10.53 GFLOP/s** on Linpack, **1.58×** on a dense eigensolve, and **1.31×** on a 64-atom Si DFT SCF.
+Recent highlights on the Orange Pi RV2 (SpaceMiT X60, RVV): fixing an OpenBLAS `gemv_n` bug restores correctness across [BLAS](scientific-libs/blas.html), [LAPACK](scientific-libs/lapack.html), [ELPA](scientific-libs/elpa.html), [HPL](apps/hpl.html), and [Quantum ESPRESSO](apps/qe.html) — with patched RVV reaching **10.53 GFLOP/s** on Linpack, **1.58×** on a dense eigensolve, and **1.31×** on a 64-atom Si DFT SCF. On the IME path, setting `accuracy_level=4` on ONNX `MatMulNBits` nodes unlocks **9–10×** int4 LLM decode vs the fp32 fallback — see [ONNX Runtime](apps/onnx.html) and [MLAS](scientific-libs/mlas.html).
 
 ## Scientific libs
 
@@ -28,13 +28,15 @@ Library-level probes — performance *and* numerical correctness:
 - **[NumPy](scientific-libs/numpy.html)** — `bench_blas.py` DGEMM and `eigvalsh` through the SciPy stack
 - **[LAPACK](scientific-libs/lapack.html)** — LAPACK path via NumPy `eigvalsh`
 - **[ELPA](scientific-libs/elpa.html)** — dense eigensolver (CP2K / VASP class workloads)
+- **[MLAS](scientific-libs/mlas.html)** — ONNX Runtime QNBit int4 GEMM; isolated IME kernel rates on X60
 
 ## Apps
 
 End-to-end application benchmarks on the same boards and EESSI toolchain:
 
 - **[HPL](apps/hpl.html)** — High Performance Linpack; cross-board summary and A/B configs from [opensolvers/benchmarks](https://github.com/opensolvers/benchmarks)
 - **[Quantum ESPRESSO](apps/qe.html)** — plane-wave DFT SCF (`pw.x`); whole-application BLAS backend A/B with per-routine timers
+- **[ONNX Runtime](apps/onnx.html)** — int4 `MatMulNBits` LLM decode; `accuracy_level=4` unlocks X60 IME (**9–10×**)
 
 ## Boards
 

_config.yml

@@ -19,6 +19,8 @@ navigation_apps:
     url: /apps/hpl.html
   - title: Quantum ESPRESSO
     url: /apps/qe.html
+  - title: ONNX Runtime
+    url: /apps/onnx.html
 
 navigation_scientific_libs:
   - title: BLAS
@@ -31,3 +33,5 @@ navigation_scientific_libs:
     url: /scientific-libs/lapack.html
   - title: ELPA
     url: /scientific-libs/elpa.html
+  - title: MLAS
+    url: /scientific-libs/mlas.html

apps/onnx.md

@@ -0,0 +1,69 @@
+# ONNX Runtime
+
+[ONNX Runtime](https://onnxruntime.ai/) int4 **`MatMulNBits`** inference on SpaceMiT X60 — a whole-graph LLM decode probe through ORT's MLAS `SQNBit` path and the X60 **IME** (`smt.vmadot`).
+
+Benchmark source: [opensolvers/benchmarks/onnx](https://github.com/opensolvers/benchmarks/tree/main/onnx) — Llama-7B-proportioned FFN stack, `patch_accuracy_level.py`, and `onnxruntime_perf_test` runners.
+
+Library-level kernel numbers: [MLAS](../scientific-libs/mlas.html). Raw IME microbenchmarks: [Orange Pi RV2](../boards/RV2.html#ime-integer-matrix-extension).
+
+## The workload
+
+8-layer stack of FFN/projection blocks: `hidden=4096`, `ffn=11008`, **16 `MatMulNBits` nodes** (com.microsoft), 4-bit symmetric weights, `block_size=32` — the big quantised GEMMs in LLM decode. Measured with `onnxruntime_perf_test` (`-m times`, `-r 8`) at decode shape **M=1**.
+
+## Before / after — one missing attribute
+
+ORT selects the int4 compute variant in `matmul_nbits.cc`:
+
+| `accuracy_level` | MLAS path | X60 backend |
+| ---------------- | --------- | ----------- |
+| **4** | `SQNBIT_CompInt8` | IME `smt.vmadot` kernel |
+| anything else | `SQNBIT_CompFp32` | *no kernel registered* → generic fp32 dequant + SGEMM fallback |
+
+The generated model had **zero** `accuracy_level` attributes across all 16 nodes. Setting `accuracy_level=4` at generation time (or via [`patch_accuracy_level.py`](https://github.com/opensolvers/benchmarks/blob/main/onnx/patch_accuracy_level.py)) selects the IME path — **no kernel code change**.
+
+### End-to-end latency (Orange Pi RV2 / X60)
+
+| Configuration | Before (CompFp32 fallback) | After (IME CompInt8) | Speedup |
+| ------------- | -------------------------: | -------------------: | ------: |
+| Single thread (`-x1`) | 31,956 ms | **3,522 ms** | **9.1×** |
+| 8 threads (`-x8`) | 6,074 ms | **590 ms** | **10.3×** |
+
+- Peak RSS: ~1023 MB → **842 MB** (no fp32 dequant buffers).
+- x1→x8 scaling on the fixed path: **~6×** across 8 cores.
+
+## How it was confirmed
+
+- **Probe:** `fprintf` in both IME kernel paths — at M=1, **neither fired** on the broken model; the kernel never ran.
+- **Roofline:** X60 STREAM triad = 3.85 GB/s (1T) / ~10.6 GB/s (8T); int4 B traffic ≈ 361 MB/inference → bandwidth floor ~0.09 s (x1). Measured latency was **178–341× above** the floor — wrong code path, not slow kernel.
+- **Bytes:** `grep -c accuracy_level model.onnx` → 0 across 16 nodes.
+
+After the fix, the probe fires and first inference drops from ~32 s to ~4.5 s.
+
+## Kernel micro-opt was a regression
+
+The original goal was an M<4 RVV gemv fast path inside the IME `SQ4BitGemmKernel_CompInt8`. Once the correct path executed, fair A/B at M=1 showed:
+
+| Kernel (M=1) | Latency |
+| ------------ | ------: |
+| Hand-written RVV fast path | 4,521 ms |
+| Stock IME `smt.vmadot` tile | **3,522 ms** |
+
+The bespoke path was **28% slower** and was reverted. **Config beat kernels** — one attribute gave 9–10×; the hand kernel gave −28%.
+
+## Reproduce
+
+```sh
+python3 patch_accuracy_level.py int4_ffn.onnx int4_ffn_acc4.onnx
+
+onnxruntime_perf_test -e cpu -I -m times -r 8 -x 1 int4_ffn_acc4.onnx
+onnxruntime_perf_test -e cpu -I -m times -r 8 -x 8 int4_ffn_acc4.onnx
+```
+
+**Toolchain:** ONNX Runtime 1.29.0, `foss/2025b`, X60 `smt.vmadot` MLAS backend, `-march=rv64gcv_zvl256b_zfh_zvfh`.
+
+## Takeaways
+
+1. **Prove the code runs before optimising it** — a one-shot probe would have saved the kernel-tuning detour.
+2. **Roofline early** — traffic math pointed at "wrong path" in minutes.
+3. **Comments aren't evidence** — grep the artifact, not the intent.
+4. **Config beats kernels** — `accuracy_level=4` → 9–10×; hand-tuned RVV → −28%.

boards/F3.md

@@ -8,7 +8,7 @@ Cross-board confirmation from [opensolvers/benchmarks](https://github.com/openso
 
 Same K1 / X60 silicon as the Orange Pi RV2 — each chip has SpaceMiT's **IME** int8 matrix unit (`smt.vmadot`) on **cluster 0** (cores 0–3, **512 KB shared L2**). The IME is separate from RVV: one instruction accumulates a **4×4 int32** tile from **4×8 int8** operands.
 
-Detailed `ime-bench` numbers were measured on the [RV2](RV2.html) (peak **~42 GOP/s** int8 GEMM vs ~5 GOP/s RVV int8, **~7–8×**). The BPI-F3 has the same IME hardware; pin to core 0 when running [benchmarks/ime](https://github.com/opensolvers/benchmarks/tree/main/ime). Useful for int8 / quantised inference stacks (e.g. llama.cpp's `GGML_CPU_RISCV64_SPACEMIT` backend).
+Detailed `ime-bench` numbers were measured on the [RV2](RV2.html) (peak **~42 GOP/s** int8 GEMM vs ~5 GOP/s RVV int8, **~7–8×**). The BPI-F3 has the same IME hardware; pin to core 0 when running [benchmarks/ime](https://github.com/opensolvers/benchmarks/tree/main/ime). End-to-end int4 inference: [ONNX Runtime](../apps/onnx.html) and [MLAS](../scientific-libs/mlas.html).
 
 ## HPL
 

boards/RV2.md

@@ -16,7 +16,7 @@ Microbenchmarks in [opensolvers/benchmarks/ime](https://github.com/opensolvers/b
 | 768×768×512 | ~5.2 GOP/s | **42 GOP/s** (peak) | **8.1×** |
 | 1024×1024×512 | 5.2 GOP/s | **32 GOP/s** | 6.2× |
 
-Peak **~42 GOP/s** single-core — vs ~5 GOP/s for a straightforward RVV int8 path. llama.cpp ships an X60 IME backend (`GGML_CPU_RISCV64_SPACEMIT`) for block-scaled Q4 inference; see also [papers/x60-ime-block-scale-optimization](https://github.com/opensolvers/benchmarks/blob/main/papers/x60-ime-block-scale-optimization.md) in the benchmarks repo.
+Peak **~42 GOP/s** single-core — vs ~5 GOP/s for a straightforward RVV int8 path. End-to-end int4 LLM decode through [ONNX Runtime](../apps/onnx.html) and isolated [MLAS](../scientific-libs/mlas.html) kernel rates use the same IME hardware; see also [papers/x60-ime-block-scale-optimization](https://github.com/opensolvers/benchmarks/blob/main/papers/x60-ime-block-scale-optimization.md) in the benchmarks repo.
 
 ## HPL via EESSI
 

scientific-libs/mlas.md

@@ -0,0 +1,45 @@
+# MLAS
+
+[MLAS](https://github.com/microsoft/onnxruntime/tree/main/onnxruntime/core/mlas) (Microsoft Linear Algebra Subprograms) is ONNX Runtime's in-tree linear-algebra layer. On RISC-V SpaceMiT X60 it exposes an **IME** (`smt.vmadot`) backend for int4 **QNBit** GEMM — the kernel behind ONNX Runtime's `MatMulNBits` nodes.
+
+Benchmark source: [opensolvers/benchmarks/onnx](https://github.com/opensolvers/benchmarks/tree/main/onnx) — [`bench_qnbit_mlas.cpp`](https://github.com/opensolvers/benchmarks/blob/main/onnx/bench_qnbit_mlas.cpp) links directly against `libonnxruntime_mlas.a` and calls the same entry points as ORT's `matmul_nbits.cc`.
+
+See also the [ONNX Runtime app](../apps/onnx.html) for end-to-end int4 LLM-FFN inference and the [IME microbenchmarks](../boards/RV2.html#ime-integer-matrix-extension) on the Orange Pi RV2.
+
+## What it probes
+
+[`../ime`](https://github.com/opensolvers/benchmarks/tree/main/ime) benchmarks raw `smt.vmadot` int8 GEMM. MLAS sits one layer up: **4-bit symmetric weights**, `BlkLen=32`, external per-block scales — the packing and compute path ORT uses for quantised LLM weights.
+
+The harness calls:
+
+- `MlasQNBitGemmPackQuantBData` — pack quantised B (nbits=4, no zero-point),
+- `MlasQNBitGemmBatch<float>` — run the GEMM,
+
+with `thread_pool=nullptr` so results are **single-thread kernel rates**, not ORT graph throughput.
+
+## Single-thread kernel rates (X60, one core)
+
+| Shape (M×K×N) | Kernel rate | packed B |
+| ------------- | ----------: | -------: |
+| 1 × 4096 × 11008 | **0.47 GOP/s** | 22.5 MB |
+| 1 × 11008 × 4096 | **0.39 GOP/s** | 22.5 MB |
+| 1 × 4096 × 4096 | **0.48 GOP/s** | 8.4 MB |
+
+These isolate kernel efficiency. Production [ONNX](../apps/onnx.html) inference fans the same kernel across 8 cores (~**6×** scaling at M=1 decode).
+
+## RISC-V pack gotcha
+
+The X60 IME backend registers only the plain `SQ4BitGemmPackQuantBData` (`memcpy`), not the x64-style `...AndBlkSum` variant. Scales stay external and are passed at compute time via `QuantBScale`. A second "finalize" pack call with `QuantBData=nullptr` (x64 recipe) `memcpy`s from `NULL` and segfaults — the harness does a single data-only pack.
+
+## Kernel tuning note
+
+Once the correct ORT compute path was confirmed (`accuracy_level=4` → `SQNBIT_CompInt8`), a hand-written RVV gemv "fast path" inside the IME kernel was **28% slower** than the stock `smt.vmadot` tile path at M=1 and was reverted. Config beat micro-optimisation — see [ONNX app](../apps/onnx.html).
+
+## Reproduce
+
+```sh
+make board CXX=$GCC14/bin/g++
+LD_LIBRARY_PATH=$GCC14/lib64:$LD_LIBRARY_PATH ./qnbit-mlas-bench 1 4096 11008 50
+```
+
+**Toolchain:** ONNX Runtime 1.29.0, `foss/2025b`, X60 `smt.vmadot` (XsmtVdot v1.0), `-march=rv64gcv_zvl256b_zfh_zvfh`.

website.md

@@ -19,16 +19,16 @@ Context file for structural and content decisions on [opensolvers.com](https://w
 
 1. **Home**
 2. **Boards** — VisionFive 2, OrangePi RV2, BananaPi F3
-3. **Apps** — HPL, Quantum ESPRESSO
-4. **Scientific libs** — BLAS, DGEMM, NumPy, LAPACK, ELPA
+3. **Apps** — HPL, Quantum ESPRESSO, ONNX Runtime
+4. **Scientific libs** — BLAS, DGEMM, NumPy, LAPACK, ELPA, MLAS
 
 Nav config: `_config.yml` (`navigation`, `navigation_boards`, `navigation_apps`, `navigation_scientific_libs`). Rendered in `_includes/header.html`. Cayman theme requires `_layouts/default.html` override to include the header.
 
 ## Content sources
 
 | Source | Used for |
 |--------|----------|
-| [opensolvers/benchmarks](https://github.com/opensolvers/benchmarks) | BLAS verification (`dgemm/`), HPL configs (`hpl/`), ELPA (`elpa/`), QE (`qe/`), NumPy/LAPACK (`numpy/`) |
+| [opensolvers/benchmarks](https://github.com/opensolvers/benchmarks) | BLAS (`dgemm/`), HPL (`hpl/`), ELPA (`elpa/`), QE (`qe/`), NumPy/LAPACK (`numpy/`), ONNX/MLAS (`onnx/`) |
 | [EESSI/docs#818](https://github.com/EESSI/docs/pull/818) | VisionFive 2 / U74 OpenBLAS + HPL |
 | [EESSI/docs#819](https://github.com/EESSI/docs/pull/819) | Orange Pi RV2 / X60 RVV `gemv_n` fix + HPL |
 | [easyconfigs#26436](https://github.com/easybuilders/easybuild-easyconfigs/pull/26436) | U74 OpenBLAS package |
@@ -44,6 +44,7 @@ Nav config: `_config.yml` (`navigation`, `navigation_boards`, `navigation_apps`,
 
 | Date | Decision |
 |------|----------|
+| 2026-07-17 | Add MLAS (lib) and ONNX Runtime (app) pages from `benchmarks/onnx` |
 | 2026-07-15 | Vector box: add ZVL256B and ZVL128B subtitles |
 | 2026-07-15 | Compute-backends diagram: RV64GC / IME subtitles; U74 DGEMM under Scalar |
 | 2026-07-15 | Homepage: compute-backends illustration (Scalar / Vector / Specific / GPU) |