mtmd, vulkan : Gemma 4 E2B encoder token-count + Android Vulkan build + Mali attention speedup (QVAC-21361) by Alok-Ranjan23 · Pull Request #172 · tetherto/qvac-fabric-llm.cpp

Alok-Ranjan23 · 2026-06-25T14:42:08Z

What

Fixes and an on-device optimization for the Gemma 4 E2B vision encoder on the Vulkan backend (Android / Pixel 9 Pro, Arm Mali).

Correctness

mtmd — Gemma 4 encoder token-count rounding (tools/mtmd/clip.cpp).
clip_n_output_tokens computed the post-pool token count for the
GEMMA3 / GEMMA4V / IDEFICS3 / INTERNVL / NEMOTRON_V2_VL / LLAMA4 projectors as
floor(W/p · H/p / s²), but the encoder graph pools per dimension:
floor(W/p/s) · floor(H/p/s). When a side isn't divisible by n_merge the two
disagree, the predicted count mismatches the actual graph output, and the run
aborts in clip_image_batch_encode ("Invalid number of output tokens").
Now rounds per dimension — identical for divisible inputs, correct otherwise.

Build

vulkan — Android Vulkan build is broken out of the box (ggml/src/ggml-vulkan/CMakeLists.txt).
The Android NDK ships only the C Vulkan headers (vulkan/vulkan.h); it provides
neither the C++ bindings (vulkan/vulkan.hpp) nor SPIRV-Headers
(spirv/unified1/spirv.hpp). Fix:
- find_package(SPIRV-Headers) is QUIET; when the target is missing the build
  sources pinned, header-only SPIRV-Headers from the build tree's _deps
  (offline), falling back to the upstream repo.
- on Android, the C++ Vulkan headers (vulkan/vulkan.hpp) are likewise taken from
  the pinned _deps headers and added as an include dir only (the NDK already
  defines the Vulkan::Headers target, so it is not redefined).

Optimization (Mali / Pixel 9 Pro)

The encoder's dominant kernel shifts from FFN matmul to attention as the token
budget grows, and on Mali the Vulkan flash-attn path is scalar (no
cooperative-matrix), so it is comparatively slow at short sequence lengths.

vulkan — bake the on-device-best scalar flash-attn tile config for Mali
(ggml/src/ggml-vulkan/ggml-vulkan.cpp).
Use D_split=4, Br=4 on the short-sequence path; GGML_VK_FA_BR /
GGML_VK_FA_DSPLIT allow overriding for tuning. Output is unchanged
(cosine ~0.99998 vs the generic config).
mtmd/clip — budget-aware attention selection on the AUTO path
(tools/mtmd/clip.cpp).
On Mali, AUTO now selects the explicit (mul_mat) attention path for short
sequences — where it is faster than scalar flash-attn — and flash-attn for
long sequences, where the explicit path's N² score matrix would OOM. The
decision is per-image, gated by n_patches with a memory-aware cutoff, and
guarded by an OOM-to-flash-attn rebuild fallback (always correct). This is
default-on for Mali only; other GPUs keep the prior behavior. Override the
crossover via MTMD_CLIP_AUTO_FA_MIN_KV.

Tooling (debug-only, env-gated)

tools/mtmd/debug/mtmd-debug.cpp: the per-op eval callback now sits behind
MTMD_DEBUG_CB_EVAL so encoder timing is clean by default.
tools/mtmd/mtmd.cpp: optional in-process flash-attn config sweep + embedding
accuracy check (cosine / max-abs-err) used to drive the on-device tuning above.

Why

The token-count abort makes the Gemma 4 encoder unusable for any image whose
patch grid isn't a multiple of n_merge. The build gap blocks cross-compiling the
Vulkan backend for Android at all. With those resolved, the attention-selection
work targets the actual hotspot identified for QVAC-21361.

Verification

Android arm64 + Vulkan (NDK r28b, -DGGML_VULKAN=ON): clean configure +
build of llama-mtmd-cli / llama-mtmd-debug and libggml-vulkan.so, with no
system SPIRV-Headers — the pinned _deps headers supply them.
Token-count fix: non-aligned sizes no longer trip the sanity check; divisible
inputs unchanged (no regression).
On-device (Pixel 9 Pro, Mali-G715), encoder-only: AUTO correctly picks the
explicit path at 196 tokens and flash-attn at 784 tokens (no OOM); embeddings
unchanged (cosine ~0.99998).

Encoder timing — before vs after (Pixel 9 Pro, Mali-G715)

Baseline = original generic flash-attn; after = this PR's AUTO selection + Mali tile tuning.

Tokens (grid)	AUTO pick (Mali)	Baseline FA (ms)	This PR (ms)	Speedup
196 (14²)	explicit (mul_mat)	9587	8034	~16.2%
784 (28²)	flash-attn (tuned)	83946	74479	~11.3%

Embeddings match the baseline to cosine ~0.99998 in both cases.

Notes

Off-Mali behavior is unchanged: the budget-aware path is gated to Mali devices.
Runtime still links the device's ABI-stable libvulkan.so; the fetched headers
are compile-time only.

Made with Cursor

…uild (QVAC-21361) mtmd: clip_n_output_tokens computed Gemma3/Gemma4V/IDEFICS3/InternVL/Nemotron/ Llama4 output tokens as floor(W/p * H/p / s^2), which disagrees with the encoder's per-dimension 2D pooling floor(W/p/s) * floor(H/p/s). When a side is not divisible by n_merge the predicted count mismatched the actual graph output and aborted in clip_image_batch_encode. Round per dimension; identical for divisible inputs, correct otherwise. vulkan: the Android NDK provides only the C Vulkan headers, so the build failed on vulkan/vulkan.hpp and spirv/unified1/spirv.hpp. find_package(SPIRV-Headers) is now QUIET on Android, and the Android branch fetches pinned, header-only Vulkan-Headers + SPIRV-Headers via FetchContent so the Android Vulkan build works out of the box. Co-authored-by: Cursor <cursoragent@cursor.com>

…alysis (QVAC-21361) Device-independent analysis to guide Pixel 9 Pro (Mali) kernel optimization: encoder op coverage (no CPU fallbacks, graph splits = 1), exact matmul/attention kernel shapes, an analytical FLOP breakdown, and how the dominant kernel shifts from FFN matmul to attention as the token budget grows. Co-authored-by: Cursor <cursoragent@cursor.com>

…oid headers The qvac-fabric vcpkg port already provides the Android Vulkan C++ bindings (downloads Vulkan-Headers into ggml/src/ggml-vulkan/vulkan_cpp_wrapper) and SPIRV-Headers (vcpkg dependency). The earlier FetchContent-based change here overlapped that mechanism and, under the vcpkg port build, left ggml-vulkan.cpp unable to find <vulkan/vulkan.hpp>, breaking the arm64-android build (verified in qvac CI on the llm-llamacpp overlay PR). Revert ggml/src/ggml-vulkan/CMakeLists.txt to the temp-9341 baseline so the port keeps owning Android header provisioning. The standalone-build convenience, if wanted, can be proposed separately. Only the mtmd token-count fix remains. Co-authored-by: Cursor <cursoragent@cursor.com>

…n (QVAC-21361) Optimize the Gemma 4 E2B vision encoder on Arm Mali (Vulkan scalar flash-attn, no cooperative-matrix), measured on Pixel 9 Pro: * vulkan: bake the on-device-best scalar flash-attn tile config (D_split=4, Br=4) for Mali on the short-sequence path, plus GGML_VK_FA_BR / GGML_VK_FA_DSPLIT env overrides for tuning. Output unchanged (cosine ~0.99998 vs the generic config). * mtmd/clip: budget-aware attention selection on the AUTO path. On Mali, AUTO now picks the explicit (mul_mat) path for short sequences, where it is faster, and flash-attn for long ones, where explicit OOMs. The choice is per-image, gated by n_patches with a memory-aware cutoff, and guarded by an OOM-to-flash-attn rebuild fallback. Default-on for Mali only (other GPUs keep prior behavior); override via MTMD_CLIP_AUTO_FA_MIN_KV. * vulkan/CMake: source pinned Vulkan / SPIRV-Headers from the build tree's _deps (offline) for the Android NDK build, falling back to the upstream repos when absent. * mtmd-debug, mtmd: gate the per-op eval callback behind MTMD_DEBUG_CB_EVAL (clean encoder timing by default) and add an in-process flash-attn config sweep + embedding accuracy check used for the on-device tuning. Co-authored-by: Cursor <cursoragent@cursor.com>

Remove the device-independent op-coverage / hotspot writeup from this PR; it will be tracked separately rather than in the encoder fix/optimization PR. Co-authored-by: Cursor <cursoragent@cursor.com>

…QVAC-21361) The in-process flash-attn sweep used POSIX setenv/unsetenv, which MSVC does not provide, breaking every Windows CI leg at mtmd.cpp compile. Wrap them in mtmd_debug_setenv/unsetenv helpers that use _putenv_s on _WIN32 (empty value unsets) and setenv/unsetenv elsewhere. Debug-only path; no behavior change. Co-authored-by: Cursor <cursoragent@cursor.com>

…(QVAC-21361) - clip: use ceil for IDEFICS3/NEMOTRON_V2_VL token count (build_patch_merge_permute pads dims up), keep floor for the pool/reshape projectors - clip: cache Mali/env/device-memory inputs for the AUTO flash-attn decision so they don't run per image; force FA across the sequential OOM fallback; atoi -> strtol - vulkan: reject zero / out-of-range GGML_VK_FA_* overrides before they reach dispatch - vulkan/cmake: pin fetched Vulkan/SPIRV headers to commit SHAs; header-only FetchContent_MakeAvailable instead of deprecated FetchContent_Populate (CMake 4.x) Co-authored-by: Cursor <cursoragent@cursor.com>

…an (QVAC-21361) As a static lib, the PRIVATE link to SPIRV-Headers::SPIRV-Headers leaked into the installed ggml-targets export, breaking downstream find_package(ggml) on platforms whose Vulkan SDK ships the config target (Windows). Wrap it in BUILD_INTERFACE: since it is only needed to compile ggml-vulkan.cpp. Co-authored-by: Cursor <cursoragent@cursor.com>

@olyasir

…clamped FA overrides Addresses @olyasir's review on PR #172: - [blocker] CMakeLists: the Android <vulkan/vulkan.hpp> fallback checked only a build-tree _deps path the vcpkg port never populates, so a port build fell through to a live FetchContent git clone at configure time and prepended a second, differently-pinned copy BEFORE the port's staged headers (the exact conflict the earlier revert fixed). Now prefer the port's provisioned vulkan_cpp_wrapper/include first; only fall back to _deps, then FetchContent, for standalone NDK builds — no clone or shadowing under the port. - [perf] ggml-vulkan: get_fa_tuning_params_scalar runs on the FA dispatch hot path (per layer, per token). The ~9 GGML_VK_FA_* getenv() lookups (and GGML_VK_FA_DEBUG) are now parsed ONCE into a thread-safe function-local static and only cheaply applied per call. - [security] ggml-vulkan: env tile/split overrides were bounded only by UINT32_MAX, so a value like 4294967295 made Br+1 wrap to 0 in the uint32 shared-memory size math and slip past ggml_vk_flash_attn_scalar_shmem_support. Tile dims are now clamped to 1..1024 at parse time, keeping that math from wrapping and dispatch params sane. Verified: clean Android arm64 + Vulkan build (NDK r28b) of llama-mtmd-cli. Co-authored-by: Cursor <cursoragent@cursor.com>

The cached FA-override parse lambda is capture-less, so MSVC rejects using the block-scope constexpr FA_OVERRIDE_MAX inside it (error C3493: cannot be implicitly captured). GCC/Clang accept it, but the Windows Vulkan CI build failed. Make FA_OVERRIDE_MAX `static constexpr` (static storage duration → no capture required); still a compile-time constant, no behavior change. Co-authored-by: Cursor <cursoragent@cursor.com>

…d headers Per review (Gianfranco): for the addon build there is no spirv-headers issue — the qvac-fabric vcpkg port already depends on spirv-headers and stages Vulkan-Headers into vulkan_cpp_wrapper, and the temp-9341 baseline CMake already consumes both (find_package(SPIRV-Headers) + the ANDROID vulkan_cpp_wrapper include). The QUIET/FetchContent fallbacks only served a standalone (non-vcpkg) Android NDK build, which is not the addon path and is not exercised in fabric CI. Restore ggml-vulkan/CMakeLists.txt to the temp-9341 baseline. This also removes the FetchContent Android fallback olyasir flagged as a blocker (no more configure-time clone / port-header shadowing). The token-count fix and the Mali attention optimization (clip.cpp, ggml-vulkan.cpp, mtmd.cpp) are unchanged. Co-authored-by: Cursor <cursoragent@cursor.com>

…changes Per review (Gianfranco: "check if every change is really needed"): remove the on-device tuning/benchmark scaffolding that was used to *find* the config but is not needed to ship. - ggml-vulkan.cpp: drop the GGML_VK_FA_* env-override block, GGML_VK_FA_F16ACC, and the FA-CAPS/FA-TUNE debug prints (and their includes). Keep only the baked result of that sweep: the Arm Mali scalar FA tile tuning (D_split=4, Br=4). This also removes the env-override surface olyasir's perf/security comments were about. - mtmd.cpp: revert the MTMD_DEBUG_FA_SWEEP / accuracy-check / encode-bench debug harness to baseline. - mtmd-debug.cpp: revert the MTMD_DEBUG_CB_EVAL gating to baseline. Net PR is now just the correctness fix (clip.cpp token-count) + the Mali optimization (clip.cpp budget-aware attention + OOM fallback; ggml-vulkan.cpp FA tile tuning). ggml-vulkan.cpp syntax-checked with the Android/Vulkan flags. Co-authored-by: Cursor <cursoragent@cursor.com>

@olyasir

…clamped FA overrides Addresses @olyasir's review on PR tetherto#172: - [blocker] CMakeLists: the Android <vulkan/vulkan.hpp> fallback checked only a build-tree _deps path the vcpkg port never populates, so a port build fell through to a live FetchContent git clone at configure time and prepended a second, differently-pinned copy BEFORE the port's staged headers (the exact conflict the earlier revert fixed). Now prefer the port's provisioned vulkan_cpp_wrapper/include first; only fall back to _deps, then FetchContent, for standalone NDK builds — no clone or shadowing under the port. - [perf] ggml-vulkan: get_fa_tuning_params_scalar runs on the FA dispatch hot path (per layer, per token). The ~9 GGML_VK_FA_* getenv() lookups (and GGML_VK_FA_DEBUG) are now parsed ONCE into a thread-safe function-local static and only cheaply applied per call. - [security] ggml-vulkan: env tile/split overrides were bounded only by UINT32_MAX, so a value like 4294967295 made Br+1 wrap to 0 in the uint32 shared-memory size math and slip past ggml_vk_flash_attn_scalar_shmem_support. Tile dims are now clamped to 1..1024 at parse time, keeping that math from wrapping and dispatch params sane. Verified: clean Android arm64 + Vulkan build (NDK r28b) of llama-mtmd-cli. Co-authored-by: Cursor <cursoragent@cursor.com>

Alok-Ranjan23 and others added 2 commits June 25, 2026 14:34

Alok-Ranjan23 requested a review from a team as a code owner June 25, 2026 14:42

Alok-Ranjan23 marked this pull request as draft June 25, 2026 14:55

Alok-Ranjan23 mentioned this pull request Jun 25, 2026

QVAC-21361 test: validate fabric PR #172 (Gemma 4 encoder fixes) via llm-llamacpp overlay [DO NOT MERGE] tetherto/qvac#2889

Closed

github-actions Bot added documentation Improvements or additions to documentation mtmd labels Jun 26, 2026

Alok-Ranjan23 and others added 2 commits June 30, 2026 21:30

docs : drop Gemma 4 encoder analysis from this PR (QVAC-21361)

085133a

Remove the device-independent op-coverage / hotspot writeup from this PR; it will be tracked separately rather than in the encoder fix/optimization PR. Co-authored-by: Cursor <cursoragent@cursor.com>

github-actions Bot added Vulkan ggml labels Jun 30, 2026

Alok-Ranjan23 changed the title ~~mtmd, vulkan : fix Gemma 4 E2B encoder token count + Android Vulkan build (QVAC-21361)~~ mtmd, vulkan : Gemma 4 E2B encoder token-count + Android Vulkan build + Mali attention speedup (QVAC-21361) Jun 30, 2026

Alok-Ranjan23 marked this pull request as ready for review July 1, 2026 08:47

Alok-Ranjan23 requested a review from olyasir July 1, 2026 08:49

Alok-Ranjan23 assigned iancris Jul 1, 2026

Alok-Ranjan23 mentioned this pull request Jul 1, 2026

QVAC-21361 test[vcpkg]: overlay-validate fabric PR#172 (Gemma 4 encoder) across consumers [DO NOT MERGE] tetherto/qvac#2979

Closed

iancris approved these changes Jul 2, 2026

View reviewed changes

olyasir reviewed Jul 2, 2026

View reviewed changes

Comment thread ggml/src/ggml-vulkan/CMakeLists.txt Outdated

olyasir reviewed Jul 2, 2026

View reviewed changes

Comment thread ggml/src/ggml-vulkan/ggml-vulkan.cpp Outdated

olyasir reviewed Jul 2, 2026

View reviewed changes

Comment thread ggml/src/ggml-vulkan/ggml-vulkan.cpp Outdated

iancris mentioned this pull request Jul 2, 2026

feat: add vlm-benchmark skill (on-device VLM optimization A/B) tetherto/qvac#3020

Merged

Alok-Ranjan23 requested a review from gianni-cor July 2, 2026 13:06

Alok-Ranjan23 and others added 2 commits July 2, 2026 14:39

gianni-cor approved these changes Jul 3, 2026

View reviewed changes

gianni-cor merged commit 20195f5 into temp-9341 Jul 3, 2026
36 of 54 checks passed

gianni-cor deleted the feat/QVAC-21361-gemma4-encoder-vulkan-fixes-9341 branch July 15, 2026 18:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

mtmd, vulkan : Gemma 4 E2B encoder token-count + Android Vulkan build + Mali attention speedup (QVAC-21361)#172

mtmd, vulkan : Gemma 4 E2B encoder token-count + Android Vulkan build + Mali attention speedup (QVAC-21361)#172
gianni-cor merged 12 commits into
temp-9341from
feat/QVAC-21361-gemma4-encoder-vulkan-fixes-9341

Alok-Ranjan23 commented Jun 25, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

Alok-Ranjan23 commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Correctness

Build

Optimization (Mali / Pixel 9 Pro)

Tooling (debug-only, env-gated)

Why

Verification

Encoder timing — before vs after (Pixel 9 Pro, Mali-G715)

Notes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Alok-Ranjan23 commented Jun 25, 2026 •

edited

Loading