cuda: fix abort on mixed f16/bf16 + q8_0 KV cache (default build)#226
Open
zPirx wants to merge 1 commit into
Open
cuda: fix abort on mixed f16/bf16 + q8_0 KV cache (default build)#226zPirx wants to merge 1 commit into
zPirx wants to merge 1 commit into
Conversation
The runtime kernel selector allows mixed f16/bf16 and q8_0 KV pairs, but in the default build (GGML_CUDA_FA_ALL_QUANTS=OFF) the four vec instances are not compiled and the dispatch list does not list them. The selector picks the vec kernel, dispatch finds no case, and it hits GGML_ABORT at fattn.cu:469 on the first decode token. Prefill runs fine, so pp-only testing misses it. Add the four instance files to the default CMake else() list and the four cases to the non-ALL_QUANTS #else block. ALL_QUANTS=ON already had them. Tested on RTX 4090, Qwen3.6-27B, -fa 1 -ctk f16 -ctv q8_0 and the reversed pair: both decode at ~38.7 t/s instead of aborting.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
The runtime kernel selector allows mixed f16/bf16 and q8_0 KV pairs, but in the default build (GGML_CUDA_FA_ALL_QUANTS=OFF) the four vec instances are not compiled and the dispatch list does not list them. The selector picks the vec kernel, dispatch finds no case, and it hits GGML_ABORT at fattn.cu:469 on the first decode token. Prefill runs fine, so pp-only testing misses it.
Reproduce with -fa 1 -ctk f16 -ctv q8_0 (or reversed): prefill runs, crash on the first decode token. RTX 4090 (sm_89), CUDA 12.8, base 471fb4e. A print before the abort showed D_Q=256 K=f16 V=q8_0.
Fix: add the four instance files to the default CMake else() list and the four cases to the non-ALL_QUANTS #else block. ALL_QUANTS=ON already had them.
Same missing-instance abort described in ggml-org#21526 for the (turbo*, f16) pairs; this PR covers the f16/bf16 + q8_0 pairs in the default build, which were still missing.
Additional information
After the fix, both combos decode instead of aborting (base 471fb4e, commit 53cd9f4):
Requirements