CUDA: route single-column f32 mul_mat through mmvf (transpose-free operand swap)

[roberteg16]

Summary

A mul_mat of a single-column f32 weight (ne01 == 1) against a wide activation (ne11 > MMVF_MAX_BATCH_SIZE) currently misses the in-house mmvf vector kernel — because ne11 (the token count) is the batch dimension and exceeds the 8-column cap — and falls back to an f32 cuBLAS/rocBLAS GEMM. That GEMM's 32×32 macro-tile computes just one useful output column out of 32, so it runs at a few percent of peak.

Since the output is a vector, the operands can be swapped transpose-free: the activation becomes the matrix and the single-column weight becomes the lone F32 vector, with dst reinterpreted as [ne11, 1] (same contiguous floats, only ne/nb relabeled — no data movement). The op then runs as a proper GEMV through ggml_cuda_mul_mat_vec_f.

The change lives entirely in the ggml_cuda_mul_mat dispatch (ggml/src/ggml-cuda/ggml-cuda.cu).

Where this shows up

These single-column f32 weights are exactly the small, unquantized control-signal projections that stay F32 in a GGUF (routers/gates), e.g. the shared-expert gate (ffn_gate_inp_shexp, weight f32[n_embd, 1]) in Qwen3.5/3.6 MoE.

Measured improvement (Qwen3.6-35B-A3B, gfx1151, prefill pp128)

Per-op (kernel selection is deterministic, so this is the apples-to-apples comparison):

shared-expert gate (weight f32[2048,1], ×40 layers)	before	after
kernel	rocBLAS Cijk_…MT32x32x8 Sgemm	mul_mat_vec_f (mmvf)
per instance	~78.4 µs	~3.8 µs
total (40 instances)	~3135 µs	~156 µs

~20× on this op, ~3.0 ms saved across the prefill. The router (N=256) and delta-net α/β (N=32) correctly stay on their existing path — they aren't vector-shaped, so the swap doesn't (and shouldn't) apply.

Correctness / tests

Adds test-backend-ops regression cases: m=1 with n ∈ {1,7,8,9,16,128,512} at k=2048, sweeping across MMVF_MAX_BATCH_SIZE so both the direct mmvf path (n ≤ 8) and the new swapped path (n > 8) are validated against the CPU reference.

• test-backend-ops test -o MUL_MAT: 1141/1141 passed on the HIP backend; all 7 new cases OK.

Notes / follow-ups

• The guard requires src0->type == F32 because the weight lands in mmvf's F32-only vector slot (ggml_cuda_should_use_mmvf only validates the matrix operand, so this must be checked explicitly).
• Restricted to plain 2-D matmuls (ne2 == 1 && ne3 == 1) and contiguous operands, which keeps the dst reinterpretation trivially correct.
• Cross-backend opportunity: the same pattern exists in Metal (GEMV↔GEMM via ne11 > ne11_mm_min) and Vulkan (mul_mat_vec_max_cols = 8). The optimization concept applies there too, but the transpose-free dst relabel is a kernel-dispatch detail, so each backend would need its own analogous snippet + measurement — left as follow-up rather than a fragile frontend rewrite.

[mgehre-amd]

Nice! Could you please try to upstream this?