Event bug: Gibberish output after b8734

[bchtrue]

Name and Version

version: 8734 (ddf03c6)

Operating systems

Linux

Which llama.cpp modules do you know to be affected?

llama-server

Command line

Problem description & steps to reproduce

After b8734 gibberish output on CUDA builds (61 architecture 1080 ti cards), but the same version show the same gibberish output without GPU offload also (if I'm not wrong)

I tried to find where bug begin, but at least found that b8730 working find and above. b8740 give gibberish output.

First Bad Commit

not sure something after version: 8734 (ddf03c6)

Relevant log output

No response

[SharkWipf]

I haven't bisected yet, but I can confirm GLM 5.0 Q4_k_xl with multi-GPU offloading has started producing gibberish every prompt where it worked fine 1-2 days ago.

Launch command:
./build/bin/llama-server -m /home/sebastiaan/models/GLM-5-UD-Q4_K_XL/GLM-5-UD-Q4_K_XL-00001-of-00010.gguf --keep -1 --offline --no-context-shift --flash-attn on --jinja --temp 0.7 --top-p 1.0 --min-p 0.01 --kv-unified --cache-ram 32768 --ctx-checkpoints 256 --backend-sampling --parallel 1 -dev CUDA1,CUDA0 --tensor-split 1,0 --n-gpu-layers 9999 -ot (?:^|\.)blk\.(?:(?:[4-9]|1[0-9]|20))\..*ffn_(gate|up|down)_exps\.=CUDA0 -ot (?:^|\.)blk\.(?:(?:2[1-9]|3[0-9]|4[0-9]|5[0-9]|6[0-9]|7[0-8]))\..*ffn_(gate|up|down)_exps\.=CPU --fit off --no-kv-offload --no-op-offload --threads 64 --main-gpu 1 --cache-type-k q4_1 --cache-type-v q4_1 -b 2048 -ub 2048 --port 8124

[am17an]

-nkvo was broken but it was fixed with #21736

[SharkWipf]

Oh yeah, it was -nkvo. Didn't even mean to be using that, I ran an old version of my startup command.
Can confirm:

• Issue does not happen without -nkvo
• Issue does not happen on latest master with -nkvo

[Atinoda]

This one caught me out too - can confirm that the output is nonsense when split across two P40s in b8589 or b8757 but is working fine in b8720. Generation works fine on a single P40 on all the stated versions.