Eval bug: Gemma 4 generates <unused> tokens in infinite loop

[CSCSoftware]

Bug Description

Gemma 4 models generate an infinite stream of <unused> tokens (Token ID 14 = <unused8>) on the Vulkan backend, both with GPU offloading and CPU-only. No valid text is produced — the model runs until MaxTokens is exhausted.

This happens despite having all known Gemma 4 fixes applied:

• Tokenizer fix (fix: Gemma4 tokenizer #21343)
• Template parser fix (Gemma 4 template parser fixes #21326)
• Custom newline split (llama: add custom newline split for Gemma 4 #21406)
• Byte token handling (vocab : add byte token handling to BPE detokenizer for Gemma4 #21488)
• Logit softcapping (llama-model: read final_logit_softcapping for Gemma 4 #21390)
• Plus the F32 MoE precision patch from PR models : set gemma 4 FFN MoE prec to F32 #21506 (manually applied)

Environment

• OS: Windows 11 Pro (10.0.26100)
• GPU: NVIDIA (24 GB VRAM), Vulkan backend
• llama.cpp: Built from 94ca829b6 (master, 2026-04-06) + PR models : set gemma 4 FFN MoE prec to F32 #21506 patch
• Model: gemma-4-E2B-it-Q4_K_M.gguf from unsloth/gemma-4-E2B-it-GGUF
• Build: CMake + MSVC + GGML_VULKAN=ON

Steps to Reproduce

1. Build llama.cpp from current master with Vulkan enabled
2. Load gemma-4-E2B-it-Q4_K_M.gguf with -ngl 99
3. Send any prompt (e.g., "Hello")
4. Observe: model generates ~18000+ tokens of <unused8> (token id=14) without producing any readable text or hitting EOG

Diagnostic Data

Token sampling output (first 10 tokens):

Token[0] id=14
Token[1] id=14
Token[2] id=14
Token[3] id=14
Token[4] id=14
Token[5] id=14
Token[6] id=14
Token[7] id=14
Token[8] id=14
Token[9] id=14

Token 14 in Gemma 4 vocab = <unused8>.

Generation stats: 183 tok/s, 18432 tokens generated, 44.7 seconds — no EOG token emitted.

Additional Testing

• CPU-only (-ngl 0): Same result — generates [multimodal] tokens (id=5) in an infinite loop. Also broken.
• PR models : set gemma 4 FFN MoE prec to F32 #21506 applied (F32 MoE FFN precision): No improvement on either CPU or Vulkan.
• Ollama: Same model works correctly in Ollama (which uses its own llama.cpp fork), producing valid responses.

Init Logs (successful)

Model handle: OK
Vocab size: 262144
Layers: 35
Context size: 32768
Gemma tokens: start_of_turn=105, end_of_turn=106, bos=2
System tokens: 11
Initialization complete!

Model loads correctly, context/sampler/batch all initialized — the issue is purely in inference/sampling.

Related Issues

• Eval bug: Gemma 4 generates <unused24> tokens #21321 — Gemma 4 generates <unused24> tokens (CUDA, partially fixed by models : set gemma 4 FFN MoE prec to F32 #21506)
• fix: Gemma4 tokenizer #21343 — Gemma 4 tokenizer fix (merged, applied here)
• models : set gemma 4 FFN MoE prec to F32 #21506 — F32 MoE precision (open, manually applied, does not fix this)

The root cause may be Vulkan-specific numerical precision issues beyond what #21506 addresses, or a different code path in the Vulkan compute shaders.

[pwilkin]

@CSCSoftware does #21451 help in this case?

[fchamieh]

@CSCSoftware does #21451 help in this case?

I am still having the same issue even with latest b8702. Tried both Vulkan and Cuda!

[KYmidnight]

I encounter the exact same issue on a strix halo 128gb using vulkan and llama.cpp. So it's not just CSCSoftware and fchamieh's hardware. I am running fedora 43.

[HDembinski]

I am getting the same with CUDA. I am on Windows, but run inside WSL. I am using one of the unsloth quants.

[smebberson]

I'm also encountering the same issue on a strix halo 128gb using vulkan and llama.cpp (built locally). I'm running Ubuntu 24.04 LTS. I'm using ggml-org/gemma-4-31B-it-GGUF:Q8_0 from Hugging Face.

[ggerganov]

If you are having issues with the Vulkan backend, please post exact commands, model, logs and hardware that reproduce the problem. Something like this report for example:

#21622 (comment)

[smebberson]

This is how I reproduce it:

llama-cli -hf ggml-org/gemma-4-31B-it-GGUF:Q8_0 --n-gpu-layers 60 --ctx-size 32768 --no-mmap --flash-attn on --jinja --reasoning-budget 0

The screenshot shows the llama.cpp version and device information.

[ggerganov]

cc @0cc4m - I can't reproduce on DGX Spark. Might be related to strix halo specifically.

@smebberson Btw, is there a reason why you use -ngl 60? You have plenty of memory to offload all 61 layers.

[smebberson]

@ggerganov - oh, it was just a left over from testing different flags. In one of the issues here someone said it helped to alleviate the unused24 issue by leaving one on the CPU.

[hayesmason335-hub]

I am also experiencing this issue on my setup. Here are the details for reproduction: Hardware: [Describe your GPU/CPU, e.g., NVIDIA RTX 4090 / Intel i9] OS: [e.g., Windows 11 / Ubuntu 24.04] Model: [e.g., gemma-4-27b-it-Q4_K_M.gguf] Command: [Insert the exact llama-cli or main command used]

…

On Thu, Apr 9, 2026, 7:02 AM Scott Mebberson ***@***.***> wrote: *smebberson* left a comment (ggml-org/llama.cpp#21516) <#21516 (comment)> @ggerganov <https://github.com/ggerganov> - oh, it was just a left over from testing different flags. In one of the issues here someone said it helped to alleviate the unused24 issue by leaving one on the CPU. — Reply to this email directly, view it on GitHub <#21516 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/B7GFMJN73OVCFEZV2AWRE4D4U57OHAVCNFSM6AAAAACXOGW566VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DEMJTGYZTEMJQGM> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[0cc4m]

I've tested on DGX Spark and Strix Halo and can't reproduce the problem on either. They both work fine with Vulkan. @jeffbolznv Do you see anything?

[jeffbolznv]

Is this the same as #21321?

[0cc4m]

Maybe, but that was a CUDA fusion bug. Is it possible there's a similar problem in the Vulkan fusion code?

[jeffbolznv]

It's possible, I've tried to catch memory overlap stuff in a general way but it's possible there's a bug.

[smebberson]

So.... this works?!

llama-cli -hf unsloth/gemma-4-31B-it-GGUF:Q8_0 --n-gpu-layers 99 --ctx-size 32768 --no-mmap --flash-attn on --jinja --reasoning-budget 0

The original model I was using is ggml-org/gemma-4-31B-it-GGUF:Q8_0, but I switched to unsloth/gemma-4-31B-it-GGUF:Q8_0 and it works?