Skip to content

cuda: fix abort on mixed f16/bf16 + q8_0 KV cache (default build)#226

Open
zPirx wants to merge 1 commit into
TheTom:feature/turboquant-kv-cachefrom
zPirx:fix/mixed-kv-vec-instances-v2
Open

cuda: fix abort on mixed f16/bf16 + q8_0 KV cache (default build)#226
zPirx wants to merge 1 commit into
TheTom:feature/turboquant-kv-cachefrom
zPirx:fix/mixed-kv-vec-instances-v2

Conversation

@zPirx

@zPirx zPirx commented Jul 19, 2026

Copy link
Copy Markdown

Overview

The runtime kernel selector allows mixed f16/bf16 and q8_0 KV pairs, but in the default build (GGML_CUDA_FA_ALL_QUANTS=OFF) the four vec instances are not compiled and the dispatch list does not list them. The selector picks the vec kernel, dispatch finds no case, and it hits GGML_ABORT at fattn.cu:469 on the first decode token. Prefill runs fine, so pp-only testing misses it.

Reproduce with -fa 1 -ctk f16 -ctv q8_0 (or reversed): prefill runs, crash on the first decode token. RTX 4090 (sm_89), CUDA 12.8, base 471fb4e. A print before the abort showed D_Q=256 K=f16 V=q8_0.

Fix: add the four instance files to the default CMake else() list and the four cases to the non-ALL_QUANTS #else block. ALL_QUANTS=ON already had them.

Same missing-instance abort described in ggml-org#21526 for the (turbo*, f16) pairs; this PR covers the f16/bf16 + q8_0 pairs in the default build, which were still missing.

Additional information

After the fix, both combos decode instead of aborting (base 471fb4e, commit 53cd9f4):

=== DECODE K=f16 V=q8_0 ===
| model                          |       size |     params | backend    | ngl | type_v |  fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -----: | --: | --------------: | -------------------: |
| qwen35 27B Q4_K - Medium       |  16.67 GiB |    27.32 B | CUDA       |  99 |   q8_0 |   1 |           tg128 |         38.51 ± 0.15 |
| qwen35 27B Q4_K - Medium       |  16.67 GiB |    27.32 B | CUDA       |  99 |   q8_0 |   1 |  tg128 @ d32768 |         33.60 ± 0.01 |
=== DECODE K=q8_0 V=f16 ===
| model                          |       size |     params | backend    | ngl | type_k |  fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -----: | --: | --------------: | -------------------: |
| qwen35 27B Q4_K - Medium       |  16.67 GiB |    27.32 B | CUDA       |  99 |   q8_0 |   1 |           tg128 |         38.74 ± 0.17 |
| qwen35 27B Q4_K - Medium       |  16.67 GiB |    27.32 B | CUDA       |  99 |   q8_0 |   1 |  tg128 @ d32768 |         35.05 ± 0.01 |

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: YES - AI helped locate the missing instances and helped me write this description clearly (not a native speaker). I ran all builds and tests myself, verified the results and take responsibility for the change.

The runtime kernel selector allows mixed f16/bf16 and q8_0 KV pairs,
but in the default build (GGML_CUDA_FA_ALL_QUANTS=OFF) the four vec
instances are not compiled and the dispatch list does not list them.
The selector picks the vec kernel, dispatch finds no case, and it
hits GGML_ABORT at fattn.cu:469 on the first decode token. Prefill
runs fine, so pp-only testing misses it.

Add the four instance files to the default CMake else() list and the
four cases to the non-ALL_QUANTS #else block. ALL_QUANTS=ON already
had them.

Tested on RTX 4090, Qwen3.6-27B, -fa 1 -ctk f16 -ctv q8_0 and the
reversed pair: both decode at ~38.7 t/s instead of aborting.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant