QVAC-21320 feat: Mali GPU projector optimizations — disable FA, warptile, layernorm fusion#174
Merged
Conversation
iancris
force-pushed
the
feat/QVAC-21320-mali-vulkan-projector-opt
branch
from
ba40f87 to
e647cd8
Compare
…ile, layernorm fusion Three quality-neutral ggml-vulkan optimizations for the Mali-G715 vision projector (mmproj/image encoder), benchmarked on Pixel 9 Pro (Qwen3.5-0.8B): 1. Disable flash attention on GPU projectors without efficient (coopmat) FA (tools/mtmd/clip.cpp). Uses runtime proc_address resolution to query the backend — no compile-time backend dependency. Mali FA_SCALAR ~2.6x less efficient than the matmul path; coopmat-capable GPUs keep FA enabled. 2. Mali/Valhall warptile tuning (ggml-vulkan.cpp, VK_VENDOR_ID_ARM) — large q8_0 MMQ tile to 32-warp/16-wide layout; ~90->~124 GFLOPS/s. Self-disables if shared memory is insufficient. Vendor-wide: also speeds the LLM prefill on GPU. 3. NORM+MUL+ADD (layernorm) Vulkan fusion (norm.comp, generic_binary_head.glsl, vulkan-shaders-gen.cpp, ggml-vulkan.cpp) — one dispatch replaces three; mirrors rms_norm+mul. -26 dispatches/encode. NOT Mali-specific. Result (4-run CPU-matched, profiler-off): within-run GPU/CPU mmproj-encode ratio ~1.46x (baseline) -> ~1.12x (optimized); near-parity 1.09x at high resolution. Quality 37.8%==37.8% (delta=0). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…fficient FA) The coopmat-quality FA-gate added in this PR was inert on Mali: Mali-G715 advertises VK_KHR_cooperative_matrix, so coopmat1_fa_support is true and ggml_backend_vk_supports_efficient_fa() returned true — yet Mali's flash attention still runs the slow path (~40 GFLOPS/s vs the ~100 GFLOPS/s matmul path). Per-op profiling on temp-9341/Pixel 9 showed the "optimized" build's clip encode was byte-for-byte identical to baseline (FLASH_ATTN_EXT present, zero SOFT_MAX) — the FA-disable never fired. Fix: - ggml-vulkan.cpp: ggml_backend_vk_supports_efficient_fa() returns false for VK_VENDOR_ID_ARM (Mali coopmat is not fast FA). Real-coopmat desktop GPUs (NVIDIA/AMD/Intel) still report efficient FA and keep it. - clip.cpp: apply the gate on flash_attn_type != DISABLED (the addon enables FA by default, not AUTO-only) and default to disabling FA when the backend can't confirm efficient FA (the safe original behaviour). Verified on temp-9341/Pixel 9 (Mali-G715): FLASH_ATTN_EXT -> SOFT_MAX, GPU clip-encode -30..35% at high resolution; 4-run CPU-matched within-run GPU/CPU mmproj-encode ratio 1.46x (baseline) -> 1.10x (optimized), quality neutral (37.8% == 37.8%) — recovering the original 1.46x->1.12x result. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ry (Metal/CUDA) The FA-gate defaulted efficient_fa=false, so on non-Vulkan GPU backends (Metal, CUDA) — where ggml_backend_supports_efficient_fa is not implemented — flash attention was wrongly disabled. The resulting explicit-attention clip path overflows the pre-sized compute buffer at high n_pos (image_tile_mode=disabled + image_max_tokens=4096), hitting GGML_ASSERT in ggml-backend.cpp:2043 (SIGABRT) on darwin/iOS Metal integration tests. Fix: default efficient_fa=true and disable only when a backend affirmatively reports non-efficient FA. Only ggml-vulkan implements the query (returns false for VK_VENDOR_ID_ARM / non-coopmat), so the Mali FA-disable win is unchanged; Metal/CUDA/CPU keep their efficient FA. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
iancris
force-pushed
the
feat/QVAC-21320-mali-vulkan-projector-opt
branch
from
9670eae to
5b07710
Compare
gianni-cor
reviewed
Jul 6, 2026
…pCount Review fix (PR #174): the fused-norm change dispatched GGML_OP_NORM as a direct {ne01, ne02, ne03} grid; on large row counts ne01 can exceed maxComputeWorkGroupCount[0] (spec minimum 65535) and trip the GGML_ASSERT in ggml_vk_dispatch_pipeline, where the previous flattened/tiled dispatch handled arbitrary ggml_nrows. Restore the flattened {512, 512, N} row tiling on the host (same group as SOFT_MAX/SUM_ROWS) and reconstruct {row, channel, sample} in norm.comp from the flat workgroup id (formula shared with soft_max.comp), with a workgroup-uniform bounds return for the tiling round-up. dst offset is unchanged: flat_row == (samp*nchannels + channel)*nrows + row by construction. No behavioural change for in-range shapes; the fusion's dispatch-count reduction is untouched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
gianni-cor
approved these changes
Jul 7, 2026
This was referenced Jul 7, 2026
makaveli10
pushed a commit
to makaveli10/qvac-ext-lib-llama.cpp
that referenced
this pull request
Jul 13, 2026
…pCount Review fix (PR tetherto#174): the fused-norm change dispatched GGML_OP_NORM as a direct {ne01, ne02, ne03} grid; on large row counts ne01 can exceed maxComputeWorkGroupCount[0] (spec minimum 65535) and trip the GGML_ASSERT in ggml_vk_dispatch_pipeline, where the previous flattened/tiled dispatch handled arbitrary ggml_nrows. Restore the flattened {512, 512, N} row tiling on the host (same group as SOFT_MAX/SUM_ROWS) and reconstruct {row, channel, sample} in norm.comp from the flat workgroup id (formula shared with soft_max.comp), with a workgroup-uniform bounds return for the tiling round-up. dst offset is unchanged: flat_row == (samp*nchannels + channel)*nrows + row by construction. No behavioural change for in-range shapes; the fusion's dispatch-count reduction is untouched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🎯 What problem does this PR solve?
Mali-G715 GPU vision projector (mmproj/image encoder) is ~46% slower than CPU due to:
FA_SCALAR) being ~2.6x less efficient than the matmul path (38 vs ~100 GFLOPS/s) — FA is ~30–56% of encode time (grows with resolution)📝 How does it solve it?
Three quality-neutral ggml-vulkan optimizations (Qwen3.5-0.8B, Pixel 9 / Mali-G715):
tools/mtmd/clip.cpp+ggml-vulkan.cpp): the projector queries the backend at runtime (ggml_backend_reg_get_proc_address("ggml_backend_supports_efficient_fa")). Vendor-aware: only ggml-vulkan implements the query, returning false forVK_VENDOR_ID_ARM(Mali advertisesVK_KHR_cooperative_matrixbut its FA still runs the slow path) → Mali uses explicit softmax; real-coopmat Vulkan GPUs keep FA. Backends that don't implement the query (Metal/CUDA/CPU) keep FA (they have efficient FA) — the gate defaults to keeping FA and disables only on an affirmative not-efficient answer.VK_VENDOR_ID_ARM): large q8_0 MMQ tile to 32-warp/16-wide (~90 → ~124 GFLOPS/s); self-disables if shared memory is insufficient.Results (4-run CPU-matched benchmark, profiler-off, Pixel 9 / Mali-G715, temp-9341; 15-image lmms-eval fixture, n=15/cell)
Within-run GPU/CPU
mmproj-encoderatio (CPU & GPU on the same device — the device-normalized robust metric):CPU-matched same-unit-equivalent absolutes (best CPU-fingerprint pair, 5.0% apart):
GPU clip-encode by resolution (per-op profile, FA-on baseline → FA-off optimized):
Quality 37.8% == 37.8% (Δ=0) in all runs. Per-op profiling confirms
FLASH_ATTN_EXT(~40 GFLOPS/s) isreplaced by
SOFT_MAX+ matmuls. Net: the Mali GPU projector goes from ~46% slower than its own-device CPU to~10% (near-parity), and beats the CPU projector that ships today on TTFT (+5.6%) while freeing the CPU
for the LLM.
🧪 How was it tested?
Per-op Vulkan profiling + a 4-run CPU-matched benchmark on Pixel 9 (Mali-G715) over the frozen 15-image
lmms-eval fixture (base preset, n=15/cell, 0 fail), on current
temp-9341:FLASH_ATTN_EXT→SOFT_MAX; GPU clip-encode −24…−35% by resolution.temp-9341vs this branch — within-run GPU/CPU ratio~1.46× → ~1.10×; CPU-matched cross GPU mmproj-encode +28.4%, TTFT +24.9%; quality Δ=0. Re-confirmed after
the Metal fix (
5b0771040): within-run GPU/CPU ~1.11×, quality-neutral (Mali/Vulkan path unchanged).Metal fix (
5b0771040): an earlier form of the FA-gate defaulted to disabling FA when a backend didn'tanswer the query, which wrongly disabled FA on Metal (the query is Vulkan-only) → explicit attention overflowed
the clip compute buffer (
GGML_ASSERTin ggml-backend, e.g.image_tile_mode=disabled+ largeimage_max_tokens) crashing darwin/iOS. Fixed by defaulting to keep FA on backends without the query;darwin-arm64 integration re-validated green.
Phase-A cross-consumer overlay CI — re-validated on the current head
a96944bdd(branch rebased ontocurrent
temp-93416613cea17— 36 upstream commits, identical replay verified viagit range-diff— plusthe tiled-NORM review fix) across all 6 fabric consumers via
workflow_dispatch(vcpkg overlay-port, noregistry/baseline change). All consumers' builds/prebuilds (all platforms) + C++ tests + desktop + mobile
integration pass:
Known non-blockers (not regressions from this PR):
merge-guard/validate-prfails on every run (expected — aworkflow_dispatchbranch has no PR);test-darwin-x64is pre-existing Intel-Mac-runner instability thatreproduces on the baseline fabric (no opts); the intermittent llm-llamacpp
qwen3-5-multimodal-cache-stress(also baseline-reproducible) passed this round. Device-Farm mobile-monitor and runner-provisioning flakes
cleared on re-run (embed linux-arm64, ocr Android, nmtcpp iOS).
None. Warptile self-guards on shared memory. The FA gate is additive and vendor-scoped (Metal/CUDA and
coopmat desktop Vulkan keep FA; only inefficient-FA GPUs like Mali disable it). Layernorm fusion is
accuracy-neutral.