Metal: Gemma 4 E4B Q8_0 decode regression (~13%) between b9730 and b10219; qwen models unaffected

[treyio]

Summary

Between builds b9730 and b10219, decode (token generation) speed for Gemma 4 E4B Q8_0 on Apple Silicon / Metal dropped by ~13% (pooled median across 12 measurements, two independent sessions). Qwen-family models measured on the same two binaries, same machine, same method show no regression — the largest (Qwen3.6-35B-A3B) is ~5% faster on b10219. This points at a gemma-family-specific decode-path change somewhere in the b9730 → b10219 window.

Environment

• Hardware: Apple M5, 24 GB unified memory
• OS: macOS 27.0 (build 26A5388g)
• Backend: Metal, full offload (-ngl 99)
• Builds compared:
  • b9730 (commit e475fa2b5) — Homebrew build
  • b10219 (commit c629da565) — official release binary llama-b10219-bin-macos-arm64.tar.gz

Model (public artifact)

From https://huggingface.co/unsloth/gemma-4-E4B-it-GGUF:

• gemma-4-E4B-it-Q8_0.gguf — 8,192,953,472 bytes, sha256 f8854aa4480df62585a279e7ca0a881554fc18a41c59c4f62642d16a2ae47012
• mmproj sidecar mmproj-BF16.gguf — 991,552,320 bytes, sha256 ee01cba03fd9c71ea2ea722225d24a84f72e7197714367e550ef705ef8851bc6

Reproduction

Identical llama-server command on both builds (only the binary differs); both builds accept this flag set verbatim:

llama-server \
  --host 127.0.0.1 --port 8080 --threads-http 16 \
  --model gemma-4-E4B-it-Q8_0.gguf \
  --mmproj mmproj-BF16.gguf \
  --no-mmap --ctx-size 16384 --n-gpu-layers 99 --parallel 1 \
  --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 --cache-ram 0 \
  --temperature 1 --top-k 64 --top-p 0.95 --min-p 0.05

Measurement: POST /completion with a fixed plain-text prompt — the sentence below repeated 60× (the server reports prompt_n = 1142) — with "n_predict": 256, "temperature": 0, "cache_prompt": false so every rep pays full prefill. pp/tg tok/s are read from the response timings (prompt_per_second / predicted_per_second). 3 reps per session, two sessions per build.

The study of language models on consumer hardware reveals interesting tradeoffs between memory, bandwidth, and compute. 

curl -s http://127.0.0.1:8080/completion -d "{
  \"prompt\": \"$(printf 'The study of language models on consumer hardware reveals interesting tradeoffs between memory, bandwidth, and compute. %.0s' {1..60})\",
  \"n_predict\": 256, \"temperature\": 0, \"cache_prompt\": false
}" | jq .timings

Results — decode (tg), tok/s, all 12 reps

build	session 1 reps	session 2 reps	pooled median
b9730	14.89 · 15.23 · 12.41	14.17 · 13.94 · 17.22	14.53
b10219	12.79 · 11.32 · 12.54	13.04 · 12.97 · 9.88	12.66

Pooled median delta: −12.9%. Five of six b9730 reps are ≥ 13.94 tok/s while all six b10219 reps are ≤ 13.04 tok/s.

Prefill (pp) moved less and inconsistently between sessions (session medians 455.8 → 414.5 and 533.5 → 524.8 tok/s); the clear, replicated signal is decode.

Validity note: the machine was not fully idle in either session, but the two sessions had opposite background-load asymmetries — in session 1 the b10219 cell ran at higher background load, in session 2 the b9730 cell did (substantially so) and still measured faster. The gap's direction and rough size replicated under both conditions.

Control — other models, same two binaries, same machine, same method

• Qwen3.6-35B-A3B (UD-IQ2_M): tg +5.1%, pp +8.9% on b10219 (matched load)
• Qwen3.5-4B (Q8_0): tg +3.1% on b10219
• Qwen3.5-9B (Q4_K_M): parity within measurement noise on replication
• Gemma 4 12B QAT (UD-Q4_K_XL, same family): leaned slower on b10219 in 2 of 3 draws, though never cleanly beyond noise

The clean-to-faster qwen results argue against a generic binary/toolchain difference (b9730 tested as a Homebrew build vs. b10219 as the official release binary) explaining the E4B delta.

Possibly related

• ~3.8x generation speed regression on Apple M4 (Metal) between b8680 and master (d12cc3d) with Gemma 4 26B A4B IQ4_XS #21655 (closed) — earlier large Gemma decode regression on Apple Metal (M4, Gemma 4 26B A4B)
• Eval bug: gemma4-assistant MTP draft model fails to load — "invalid vector subscript" (regression: works on b9553, broken on b9702/b9717) #24795 (open) — gemma4-assistant regression in the b9553 → b9702 window (load failure, not speed)

Follow-ups

Happy to run more on request: llama-bench pairs on both builds, a release-tag bisect of the b9730 → b10219 window on this machine, or testing candidate patches.

[nikwen]

If you could bisect which version caused the regression, that would help a lot in finding the underlying issue.

[axiom-of-choice]

If you could bisect which version caused the regression, that would help a lot in finding the underlying issue.

I can help with the bisection here :)

[axiom-of-choice]

Cross-checking on different hardware: no regression on M2 Pro (32 GB), macOS 26.5.2.

Same model and sha256, llama-bench with generation only, alternating the two builds
back to back in one session so both see the same thermal state:

         b9730          b10219
run 1    28.47 ± 0.56   28.76 ± 0.47
run 2    27.49 ± 0.84   27.64 ± 2.02
run 3    27.44 ± 1.42   27.91 ± 0.74
run 4    28.68 ± 0.41   27.69 ± 1.38

The intervals overlap and the sign flips between runs, so there's nothing near the
13% you measured.

Worth flagging how easy this was to get wrong: measuring the two builds one after
another, rebuilding in between, first gave me 28.08 vs 25.41, which looks like your
regression. Re-measuring the good build then gave 24.87, below the bad one. That was
the machine heating up during the rebuild, not the commit. Separate build dirs and
alternating runs made it go away.

So on M2 Pro this looks specific to your setup rather than the Metal decode path
generally, which might mean M5 or macOS 27 rather than the b9730..b10219 window.
Happy to run any specific commit from that range if it helps.