Environment
- GPU: RTX 4090
- OS: CachyOS
- CUDA: 13.3.1-1
- Branch: feature/turboquant-kv-cache
Repro flags
--cache-type-k q8_0 --cache-type-v turbo4 --flash-attn on --parallel 1
Bisected to
Commit 77ab7e988 ("turbo4: correct Lloyd-Max centroids (KLD/PPL fix), drop dead rnorm 68->66B (4.125bpw parity), wide-V decode config, perplexity size_t overflow fix (32K KLD)")
Root cause
In ggml/src/ggml-cuda/fattn-vec.cuh, the V_rows_per_thread calculation dropped GGML_TYPE_TURBO4_0 from the 4-rows-per-thread condition, causing a stride mismatch in the V-cache decode path:
- constexpr int V_rows_per_thread = V_is_unquantized ? ((type_V == GGML_TYPE_TURBO3_0 || type_V == GGML_TYPE_TURBO2_0) ? 4 : 2*cpy_ne) : 4;
+ constexpr int V_rows_per_thread = V_is_unquantized ? ((type_V == GGML_TYPE_TURBO3_0 || type_V == GGML_TYPE_TURBO2_0 || type_V == GGML_TYPE_TURBO4_0) ? 4 : 2*cpy_ne) : 4;
Before (broken, at 77ab7e9+)
curl -s http://127.0.0.1:8090/completion -d '{"prompt":"Which model are you?","n_predict":100}' | jq .
{
"content": " The question sounds so simple, but as it was, for me, \"it”m the way, you can never predict it\" \n\"i can't be there\" and \"i'm sorry\" have thebsites, a person's personal, 21446.21466-15-05.19398035 12 990367 1,00080278,\nA"
}After (fixed)
{
"content": " What are the key differences between the models in the same family? I'm an AI assistant developed by Tongyi Lab, and my name is Qwen. I am the latest large language model in the Qwen series, with a strong focus on dialogue understanding, code writing, logical reasoning, multimodal processing capabilities, and more. \n\nAs for the differences among models in the same family, they typically include:\n\n1. **Parameter Scale**: Different models within the same family may have different numbers of parameters"
}Model: Qwen3-Coder-30B-A3B-Instruct-UD-Q4_K_XL.gguf. Same fix confirmed clean across all turbo4-using configs (qwen3-coder, qwen3-coder-dense, qwen3-architect, qwen3-fast).
Environment
Repro flags
--cache-type-k q8_0 --cache-type-v turbo4 --flash-attn on --parallel 1
Bisected to
Commit
77ab7e988("turbo4: correct Lloyd-Max centroids (KLD/PPL fix), drop dead rnorm 68->66B (4.125bpw parity), wide-V decode config, perplexity size_t overflow fix (32K KLD)")Root cause
In
ggml/src/ggml-cuda/fattn-vec.cuh, theV_rows_per_threadcalculation droppedGGML_TYPE_TURBO4_0from the 4-rows-per-thread condition, causing a stride mismatch in the V-cache decode path:Before (broken, at 77ab7e9+)
curl -s http://127.0.0.1:8090/completion -d '{"prompt":"Which model are you?","n_predict":100}' | jq .
{ "content": " The question sounds so simple, but as it was, for me, \"it”m the way, you can never predict it\" \n\"i can't be there\" and \"i'm sorry\" have thebsites, a person's personal, 21446.21466-15-05.19398035 12 990367 1,00080278,\nA" }After (fixed)
{ "content": " What are the key differences between the models in the same family? I'm an AI assistant developed by Tongyi Lab, and my name is Qwen. I am the latest large language model in the Qwen series, with a strong focus on dialogue understanding, code writing, logical reasoning, multimodal processing capabilities, and more. \n\nAs for the differences among models in the same family, they typically include:\n\n1. **Parameter Scale**: Different models within the same family may have different numbers of parameters" }Model: Qwen3-Coder-30B-A3B-Instruct-UD-Q4_K_XL.gguf. Same fix confirmed clean across all turbo4-using configs (qwen3-coder, qwen3-coder-dense, qwen3-architect, qwen3-fast).