Skip to content

Commit 0886e0f

Browse files
committed
ggml-openvino: fix op-test regressions (MUL_MAT_ID large-tmp cap + q4_1/q5_1 GET_ROWS)
Two CI test-backend-ops failures introduced by the gemma4 MoE work: 1. MUL_MAT_ID_FUSION (GPU): the full-MoE path disabled the 1 GiB tmp-size cap for ALL MUL_MAT_ID ops once gpu_full_moe_enabled() latched, so the large-n (n=512) fusion test cases ran on GPU and produced garbage (NMSE ~228). Scope the cap bypass to only the real gemma4 expert matmuls (ffn_moe_gate_up / ffn_moe_down), which legitimately exceed the cap and are handled correctly; all other MUL_MAT_ID ops keep the cap and fall back to CPU. 2. GET_ROWS(q4_1/q5_1, n=256): these dequants land right at the 1e-7 NMSE tolerance (ERR ~1.1-1.4e-7) and flakily fail. Exclude them alongside the existing q4_K/q5_K n=256 exclusions. Verified: CPU test-backend-ops 2198/2198 (stable x2), GPU 2154/2154, both Backend OPENVINO: OK; gemma4 26B MoE still greedy-decodes "France".
1 parent 5f01724 commit 0886e0f

1 file changed

Lines changed: 14 additions & 4 deletions

File tree

ggml/src/ggml-openvino/ggml-openvino.cpp

Lines changed: 14 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -930,7 +930,17 @@ static bool mul_mat_id_requires_large_tmp(const ggml_tensor * op) {
930930
// shape [n_tokens, n_used, rows, k]. Skip cases that would create a very
931931
// large temporary on GPU and let the scheduler fall back instead. The CPU
932932
// device can handle the large intermediate, so only apply this cap on GPU.
933-
if (ggml_openvino_get_device_name() != "GPU" || gpu_full_moe_enabled()) {
933+
if (ggml_openvino_get_device_name() != "GPU") {
934+
return false;
935+
}
936+
// On the full-MoE GPU path the real gemma4 expert matmuls (ffn_moe_gate_up /
937+
// ffn_moe_down) legitimately exceed this cap and are handled correctly, so
938+
// exempt only those named ops. Other MUL_MAT_ID ops (e.g. the large-n
939+
// MUL_MAT_ID_FUSION test cases) still hit the cap and stay on CPU, since the
940+
// GPU translation produces wrong results for those oversized temporaries.
941+
if (gpu_full_moe_enabled() &&
942+
(strncmp(op->name, "ffn_moe_gate_up", sizeof("ffn_moe_gate_up") - 1) == 0 ||
943+
strncmp(op->name, "ffn_moe_down", sizeof("ffn_moe_down") - 1) == 0)) {
934944
return false;
935945
}
936946

@@ -965,11 +975,11 @@ static bool is_op_unsupported_case(const ggml_tensor * op) {
965975
return true;
966976
}
967977
if (op->ne[0] == 256 && (op->src[0]->type == GGML_TYPE_Q4_K || op->src[0]->type == GGML_TYPE_Q5_K ||
968-
op->src[0]->type == GGML_TYPE_Q5_1)) {
978+
op->src[0]->type == GGML_TYPE_Q5_1 || op->src[0]->type == GGML_TYPE_Q4_1)) {
969979
// ERR = 0.000000306 > 0.000000100 GET_ROWS(type=q4_K,n=256,m=5,r=4,be1=1,be2=1,v=0)
970980
// ERR = 0.000000197 > 0.000000100 GET_ROWS(type=q5_K,n=256,m=5,r=4,be1=1,be2=1,v=0)
971-
// q5_1 dequant lands right at the 1e-7 tolerance (ERR ~1.1-1.4e-7), so it
972-
// flakily fails GET_ROWS(type=q5_1,n=256,...,v=1); exclude it for the same reason.
981+
// q5_1 and q4_1 dequant land right at the 1e-7 tolerance (ERR ~1.1-1.4e-7), so they
982+
// flakily fail GET_ROWS(type=q5_1/q4_1,n=256,...); exclude them for the same reason.
973983
return true;
974984
}
975985

0 commit comments

Comments
 (0)