[MLAS] Optimize RISC-V RVV SGEMM kernel performance

[zejianzhang1982]

This PR optimizes the SGEMM kernel for RISC-V Vector Extension (RVV) with the following improvements:

1. Use vfloat32m1_t instead of vfloat32m4_t
   - vfloat32m1_t occupies only 1 vector register group instead of 4
   - Reduces register pressure, allowing more accumulators without spilling to stack
2. 4 accumulators per row to hide FMACC latency
   - FMACC (fused multiply-add) typically has 3-5 cycle latency on RISC-V processors
   - By alternating between 4 independent accumulators, subsequent operations don't wait for previous results
   - This effectively hides the pipeline latency
3. 8x K-loop unrolling for better instruction-level parallelism
   - Reduces loop overhead (branch, counter update) by approximately 75%
   - Provides more opportunities for the compiler/instruction scheduler to reorder instructions
4. Software prefetching for next iteration data
   - Uses __builtin_prefetch to hint the CPU to load next iteration's data into cache
   - Helps hide memory latency for larger matrices
   -
   The existing RVV SGEMM kernel implementation uses vfloat32m4_t with a single accumulator per row. While this design is VLEN-agnostic and portable across different RISC-V
   implementations, it doesn't fully utilize the available vector registers to hide FMACC latency, resulting in suboptimal performance.

Problem:

• RISC-V has 32 vector registers available
• vfloat32m4_t uses 4 register groups, leaving limited room for multiple accumulators
• Single accumulator per row means each FMACC must wait for the previous one to complete (data dependency)

Solution:

• Switch to vfloat32m1_t which uses only 1 register group
• This allows 4 accumulators per row (8 total for 2 rows) within the 32 register limit
• Process 4 columns at a time (fixed block size) for consistent performance

Performance Results on SG2044 (RISC-V RVV 1.0):

Test model: silero_vad.onnx

Metric	Before Optimization	After Optimization	Improvement
Total inference time	800,458 us	679,217 us	15.1% faster
Average inference	854 us	724 us	15.2% faster
Min inference	830 us	699 us	15.8% faster
Max inference	1,541 us	1,441 us	6.5% faster

The optimization provides consistent performance improvement across all metrics, with approximately 15% speedup in average inference time for real-world audio processing workloads.

---

[zejianzhang1982]

@microsoft-github-policy-service agree company=“ZTE Corporation”

[Copilot]

Pull request overview

This PR updates the RISC-V RVV SGEMM micro-kernel in MLAS to improve performance via increased accumulator parallelism, deeper K-loop unrolling, and software prefetching.

Changes:

• Reworks the RVV SGEMM kernel to use vfloat32m1_t with 4 independent accumulators per row.
• Adds 8× unrolling in the K loop and introduces __builtin_prefetch to improve ILP and reduce memory latency.
• Changes row dispatch to process 2 rows at a time using the new optimized kernel.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[hariharans29]

Summary

This swaps the existing VLEN-agnostic RVV SGEMM kernel (which uses vfloat32m4_t and runtime vsetvl_e32m4) for a fixed 4-column block kernel using vfloat32m1_t, with 4 accumulators per row, 8x K unrolling, and __builtin_prefetch. Reports 15% speedup on SG2044 / silero_vad. Net diff +255/−132.

Substantive concerns

1. VLEN-agnosticism is silently dropped. The old kernel let __riscv_vsetvl_e32m4(remaining_n_block) adapt to whatever VLEN the hardware has — VLEN=128 gets 16 lanes per m4, VLEN=256 gets 32, etc. The new kernel hardcodes kBlockSize = 4 and uses __riscv_vsetvl_e32m1(kBlockSize). On VLEN=128 (SG2044) m1 = 4 floats and vl == 4, which is the only case the rest of the code (e.g., float Row0Block_arr[4], the (CountN & 2) / (CountN & 1) partial-store branches, the B += kBlockSize stride) is valid for. On VLEN ≥ 256 hardware (T-Head, future SiFive parts) the kernel will set vl = 4 and leave half-or-more of each m1 register idle. The comment block at the top of the file claiming portability ("not tied to a fixed VLEN such as 128 or 256 bits") is also removed but the new comment doesn't acknowledge the trade-off. Please either:

   • state in the PR description and the file header that this kernel is now specialized for VLEN=128 (with a static_assert or runtime check), and add a separate VLEN-agnostic fallback for VLEN ≠ 128, or
   • parameterize kBlockSize on runtime VLEN so wider-VLEN cores actually benefit.

   As written, this is a perf win on SG2044 and a regression in generality.

2. Lost the 3-row and 4-row tile paths. Old MlasGemmFloatKernelRvvDispatchRows had specialized tiles for Rows == 1/2/3/4. New code drops the 3-row and 4-row variants entirely:

   while (CountM >= 2) { ProcessTwoRows = true; ... }
   if (CountM == 1) { ProcessTwoRows = false; ... }

   A 4-row tile shares each B-vector load across 4 A rows, halving B memory traffic vs two back-to-back 2-row tiles. The benchmark is silero_vad, which is heavy on small-M (often M=1 / GEMV-shaped) operators, so this regression won't show up there. For tall-M workloads (anything CNN-ish) this could be slower. Please post a second benchmark on a non-GEMV workload (e.g., ResNet50 inference, or any model with non-trivial M for the FC/Conv layers) before dropping the 4-row tile, and consider keeping the 4-row variant.

3. Numerical change. Splitting one accumulator into four and reducing via (Acc0 + Acc1) + (Acc2 + Acc3) produces different last-bit float results from the original serial accumulation. Probably acceptable for SGEMM (and matches what most production kernels do), but please confirm onnxruntime_test_all / the MLAS GEMM unit tests still pass with default tolerances on SG2044.

4. k_shift underflow. size_t k_shift = CountK * kPackedCountN - kPackedCountN; underflows to a huge value if CountK == 0. The current MLAS callers always pass CountK >= 1, but add an explicit assert(CountK > 0) or compute as kPackedCountN * (CountK > 0 ? CountK - 1 : 0).

5. No new test coverage. This is fine if the existing TestSGEMM cases cover the relevant (M, N, K) shapes — please confirm the shape matrix exercises CountN values 1, 2, 3, 5, 6, 7, 9, 15, 17 (i.e., the partial-store branches (CountN & 2) / (CountN & 1) on top of the 4-column stride) on the RVV path.

Nits

• The variable countb reads as a typo; block_index_in_packed_tile or tile_block_idx would be clearer.
• if (!ProcessTwoRows) { UNREFERENCED_PARAMETER(lda); UNREFERENCED_PARAMETER(ldc); } — ProcessTwoRows is a template non-type bool, this should be if constexpr (!ProcessTwoRows) for parity with the surrounding if constexpr (AlphaIsOne) style.
• Hardcoded prefetch distance +8 is reasonable but worth a // tuned for SG2044 comment so future tuners know it's empirical.
• __builtin_prefetch is GCC/Clang only. Not actually a portability issue since this file is RISC-V only and MSVC isn't a target, but consider wrapping behind a helper macro for cleanliness.
• The trailing-blank-line removals in MlasGemmFloatKernelRvvDispatch / MlasGemmFloatKernelRvv are pure noise; please revert to keep the diff focused.

Bottom line

Optimization direction (more accumulators, unrolling, prefetch) is sound and the perf number is credible for VLEN=128 RVV1.0. Two things would change my verdict from "needs work" to "LGTM":

1. Either acknowledge VLEN=128 specialization explicitly with a guard/fallback, or generalize kBlockSize to runtime VLEN.
2. A second benchmark on a non-GEMV-dominated model to confirm dropping the 4-row tile isn't a regression on real CNN workloads.