Eval bug: Generation loop for Qwen 3.5 397B \ 122B after recent changes

[drrros]

Name and Version

3bd9aa1

Operating systems

Linux

GGML backends

CUDA

Hardware

Epyc 9274f + 3*RTX 4000 Pro Blackwell

Models

Qwen 3.5 397\122B (both in mxfp4 from unsloth)

Problem description & steps to reproduce

After recent changes (1-2 days max) I see some strange looping issues - I use claude code and on latest release of llama.cpp after 1-4 skill usage\tool usage model stucks in token generation loop (It can be seen in nvtop for example, as claude code does not shows message in streaming mode). I'm not sure whether it's thinking loop or it's generating something. Rolled back to commit 506200c and issue is gone. I will try to bissect to particular commit, but it would take quite some time as loading model takes about 20 min. for me (nmap is off for better prompt processing performance).

First Bad Commit

No response

Relevant log output

Logs look like regular ones, just processing and generating

Logs

[52265] slot update_slots: id  2 | task 72394 | n_tokens = 64619, memory_seq_rm [64619, end)
[52265] slot update_slots: id  2 | task 72394 | 8192 tokens since last checkpoint at 54384, creating new checkpoint during processing at position 66666
[52265] slot update_slots: id  2 | task 72394 | prompt processing progress, n_tokens = 66666, batch.n_tokens = 2048, progress = 0.943730
[52265] slot update_slots: id  2 | task 72394 | created context checkpoint 10 of 512 (pos_min = 64618, pos_max = 64618, n_tokens = 64619, size = 186.329 MiB)
[52265] slot update_slots: id  2 | task 72394 | n_tokens = 66666, memory_seq_rm [66666, end)
[52265] slot update_slots: id  2 | task 72394 | prompt processing progress, n_tokens = 68593, batch.n_tokens = 1928, progress = 0.971008
[52265] slot update_slots: id  2 | task 72394 | n_tokens = 68593, memory_seq_rm [68593, end)
[52265] slot update_slots: id  2 | task 72394 | prompt processing progress, n_tokens = 70637, batch.n_tokens = 2045, progress = 0.999943
[52265] slot update_slots: id  2 | task 72394 | created context checkpoint 11 of 512 (pos_min = 68592, pos_max = 68592, n_tokens = 68593, size = 186.329 MiB)
[52265] slot update_slots: id  2 | task 72394 | n_tokens = 70637, memory_seq_rm [70637, end)
[52265] reasoning-budget: activated, budget=2147483647 tokens
[52265] slot init_sampler: id  2 | task 72394 | init sampler, took 6.73 ms, tokens: text = 70641, total = 70641
[52265] slot update_slots: id  2 | task 72394 | prompt processing done, n_tokens = 70641, batch.n_tokens = 5
[52265] slot update_slots: id  2 | task 72394 | created context checkpoint 12 of 512 (pos_min = 70636, pos_max = 70636, n_tokens = 70637, size = 186.329 MiB)
[52265] srv  log_server_r: done request: POST /v1/messages 127.0.0.1 200
srv  log_server_r: done request: POST /v1/messages 192.168.0.61 200
srv    operator(): http client error: Connection handling canceled
[52265] srv          stop: cancel task, id_task = 72394
[52265] slot      release: id  2 | task 72394 | stop processing: n_tokens = 74571, truncated = 0
srv    operator(): http client error: Connection handling canceled
[52265] srv          stop: cancel task, id_task = 72393
[52265] slot      release: id  0 | task 72393 | stop processing: n_tokens = 4299, truncated = 0
[52265] srv  update_slots: all slots are idle
^C[52265] srv    operator(): operator(): cleaning up before exit...
srv    operator(): operator(): cleaning up before exit...

[JohannesGaessler]

If at all possible, please do a git bisect and identify the exact commit that caused the problem.

[johndoe606]

I have a similar issue with Gemma 4 outputting infinite looping of <unused49> when at high context (like summarizing a large txt file) after this specific commit:

c5ce4bc

The broken state of the model persists until restart of llama-server too, brand new prompts start emitting this even when it was working fine on smaller prompts.

disabling cuda graphs or reverting this commit fixes the issue for me.

[am17an]

@johndoe606 can you give me steps to reproduce?

[VONVONONE]

+1 all my 3 instant has same issues, work for normal chat but not agentic task i am running qwen 3.5
edit it seem not affect on qwen 3.5 397b a17b but loop on smaller model

[johndoe606]

@johndoe606 can you give me steps to reproduce?

I'm running this quant of 26BA4B:

https://huggingface.co/bartowski/google_gemma-4-26B-A4B-it-GGUF/blob/main/google_gemma-4-26B-A4B-it-Q4_K_L.gguf

llama-server --no-mmap --flash-attn on --fit on --parallel 1 --kv-unified --ctx-size 32768 --cache-ram 0 --swa-checkpoints 3 --fit-target 512 --temp 1.0 --top-k 64 --top-p 0.95 --min-p 0.0 -m google_gemma-4-26B-A4B-it-Q4_K_L.gguf

ANY file around 25K token on the webui with any prompt (like "summarize this")

Leads to this:

Works fine with

export GGML_CUDA_DISABLE_GRAPHS=1

Or on a forked branch of master with that commit reverted.

[VONVONONE]

work fine for me qwen 3.5 397b but not Qwen3.5-35B-A3B

[am17an]

I'm trying your command. Also, can you also try with newer quants, there have a couple of other fixes in the past few days

https://huggingface.co/unsloth/gemma-4-26B-A4B-it-GGUF/tree/main

[johndoe606]

I am on a metered connection so frankly I'll wait this one out for others to chime in then.

[ggerganov]

@johndoe606 I can't reproduce neither on DGX Spark, nor RTX 5090.

Can you confirm if the following command works for you:

./bin/llama-cli -m google_gemma-4-26B-A4B-it-Q4_K_L.gguf --no-mmap --flash-attn on --fit on --parallel 1 --ctx-size 32768 --cache-ram 0 --swa-checkpoints 3 --fit-target 512 --temp 1.0 --top-k 64 --top-p 0.95 --min-p 0.0

Here is my output:

If this command does not reproduce the issue, can you find a specific text file from the repo that reproduces the problem with llama-cli as I showed in the screenshot?

Also, make sure you do a clean build, just in case.

[drrros]

c5ce4bc - bisected to this commit aswell

[johndoe606]

Can you confirm if the following command works for you:

It doesn't work:

I used your own build from CI this time (I had downloaded it before to make sure it wasn't something on my end while compiling, I also do a git clean -fxd nuke when I build myself):

version: 8705 (09343c019)
built with Clang 19.1.5 for Windows x86_64

After running with export GGML_CUDA_DISABLE_GRAPHS=1:

[ggerganov]

Ok, thank you @johndoe606. I have found a way to reproduce.

@am17an This is the way to trigger the issue on RTX 5090:

./bin/llama-cli -hf bartowski/google_gemma-4-26B-A4B-it-GGUF:Q4_K_L --no-mmap --flash-attn on --fit on --parallel 1 --ctx-size 32768 --cache-ram 0 --swa-checkpoints 3 --fit-target 512 --temp 1.0 --top-k 64 --top-p 0.95 --min-p 0.0 -ngl 10

Adding the -ngl 10 causes some extra splits which make it easier to reproduce the problem that @JohannesGaessler noticed in #21472 (comment)