Skip to content

Commit e3cce57

Browse files
AnbeeldAnbeeld
committed
cuda: make default FA pairs q-cache focused
Change the no-flag CUDA/HIP/MUSA FlashAttention vec default from the old Turbo-focused set to Bee's 62-pair q/KVarN fallback policy. Keep Turbo/TCQ in HALF/ALL builds, derive default runtime support from bit tiers, update docs/tests, and add diagnostics that point default-build users to q/KVarN or larger FA build modes.
1 parent cb83fe7 commit e3cce57

12 files changed

Lines changed: 450 additions & 234 deletions

File tree

‎AGENTS.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -22,13 +22,13 @@ Prebuilt Windows binaries (CUDA 12.4/13.1) are on the releases page. Otherwise b
2222
```bash
2323
# Linux (GCC + CUDA)
2424
cmake -B build -DGGML_CUDA=ON -DGGML_NATIVE=ON \
25-
-DGGML_CUDA_FA=ON -DGGML_CUDA_FA_HALF_QUANTS=ON \
25+
-DGGML_CUDA_FA=ON \
2626
-DCMAKE_BUILD_TYPE=Release
2727
cmake --build build -j
2828

2929
# Windows (MSVC + CUDA)
3030
cmake -B build -DGGML_CUDA=ON -DGGML_NATIVE=ON ^
31-
-DGGML_CUDA_FA=ON -DGGML_CUDA_FA_HALF_QUANTS=ON ^
31+
-DGGML_CUDA_FA=ON ^
3232
-DCMAKE_BUILD_TYPE=Release
3333
cmake --build build --config Release --parallel
3434

@@ -37,7 +37,7 @@ cmake -B build -DGGML_METAL=ON -DCMAKE_BUILD_TYPE=Release
3737
cmake --build build -j
3838
```
3939

40-
`GGML_CUDA_FA_HALF_QUANTS=ON` is the recommended CUDA FlashAttention build mode for this fork. It compiles the useful asymmetric K/V cache type half-matrix plus f16 fallback pairs needed by TurboQuant/TCQ dequant paths: 103 FA vec K/V pairs instead of 169 in ALL_QUANTS. `GGML_CUDA_FA_ALL_QUANTS=ON` remains available for the full K/V matrix and arbitrary asymmetric cache-type combinations. These two flags are mutually exclusive. Add `-DCMAKE_CUDA_ARCHITECTURES=86` for RTX 3090, or `-DCMAKE_CUDA_ARCHITECTURES=89` for RTX 4090, if cross-compiling or building in CI without a GPU.
40+
Without `GGML_CUDA_FA_HALF_QUANTS` or `GGML_CUDA_FA_ALL_QUANTS`, the CUDA FlashAttention build compiles Bee's q/KVarN-fallback default: 62 K>=V vec pairs with fp cache types capped at q5 and q8/q6 capped at q4. This default leaves TurboQuant/TCQ FA pairs out; use `GGML_CUDA_FA_HALF_QUANTS=ON` for the larger 208-pair Turbo/TCQ-capable half-matrix, or `GGML_CUDA_FA_ALL_QUANTS=ON` for the full 361-pair matrix. These two flags are mutually exclusive. Add `-DCMAKE_CUDA_ARCHITECTURES=86` for RTX 3090, or `-DCMAKE_CUDA_ARCHITECTURES=89` for RTX 4090, if cross-compiling or building in CI without a GPU.
4141

4242
Key binaries: `build/bin/llama-server`, `build/bin/llama-cli`, `build/bin/llama-bench`, `build/bin/llama-perplexity`.
4343

‎CLAUDE.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -22,13 +22,13 @@ Prebuilt Windows binaries (CUDA 12.4/13.1) are on the releases page. Otherwise b
2222
```bash
2323
# Linux (GCC + CUDA)
2424
cmake -B build -DGGML_CUDA=ON -DGGML_NATIVE=ON \
25-
-DGGML_CUDA_FA=ON -DGGML_CUDA_FA_HALF_QUANTS=ON \
25+
-DGGML_CUDA_FA=ON \
2626
-DCMAKE_BUILD_TYPE=Release
2727
cmake --build build -j
2828

2929
# Windows (MSVC + CUDA)
3030
cmake -B build -DGGML_CUDA=ON -DGGML_NATIVE=ON ^
31-
-DGGML_CUDA_FA=ON -DGGML_CUDA_FA_HALF_QUANTS=ON ^
31+
-DGGML_CUDA_FA=ON ^
3232
-DCMAKE_BUILD_TYPE=Release
3333
cmake --build build --config Release --parallel
3434

@@ -37,7 +37,7 @@ cmake -B build -DGGML_METAL=ON -DCMAKE_BUILD_TYPE=Release
3737
cmake --build build -j
3838
```
3939

40-
`GGML_CUDA_FA_HALF_QUANTS=ON` is the recommended CUDA FlashAttention build mode for this fork. It compiles the useful asymmetric K/V cache type half-matrix plus f16 fallback pairs needed by TurboQuant/TCQ dequant paths: 103 FA vec K/V pairs instead of 169 in ALL_QUANTS. `GGML_CUDA_FA_ALL_QUANTS=ON` remains available for the full K/V matrix and arbitrary asymmetric cache-type combinations. These two flags are mutually exclusive. Add `-DCMAKE_CUDA_ARCHITECTURES=86` for RTX 3090, or `-DCMAKE_CUDA_ARCHITECTURES=89` for RTX 4090, if cross-compiling or building in CI without a GPU.
40+
Without `GGML_CUDA_FA_HALF_QUANTS` or `GGML_CUDA_FA_ALL_QUANTS`, the CUDA FlashAttention build compiles Bee's q/KVarN-fallback default: 62 K>=V vec pairs with fp cache types capped at q5 and q8/q6 capped at q4. This default leaves TurboQuant/TCQ FA pairs out; use `GGML_CUDA_FA_HALF_QUANTS=ON` for the larger 208-pair Turbo/TCQ-capable half-matrix, or `GGML_CUDA_FA_ALL_QUANTS=ON` for the full 361-pair matrix. These two flags are mutually exclusive. Add `-DCMAKE_CUDA_ARCHITECTURES=86` for RTX 3090, or `-DCMAKE_CUDA_ARCHITECTURES=89` for RTX 4090, if cross-compiling or building in CI without a GPU.
4141

4242
Key binaries: `build/bin/llama-server`, `build/bin/llama-cli`, `build/bin/llama-bench`, `build/bin/llama-perplexity`.
4343

‎README.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -343,13 +343,13 @@ Building from source with `-DGGML_NATIVE=ON` *may* result in a *tiny* bit better
343343
```bash
344344
# Linux (GCC + CUDA)
345345
cmake -B build -DGGML_CUDA=ON -DGGML_NATIVE=ON \
346-
-DGGML_CUDA_FA=ON -DGGML_CUDA_FA_HALF_QUANTS=ON \
346+
-DGGML_CUDA_FA=ON \
347347
-DCMAKE_BUILD_TYPE=Release
348348
cmake --build build -j
349349

350350
# Windows (MSVC + CUDA)
351351
cmake -B build -DGGML_CUDA=ON -DGGML_NATIVE=ON ^
352-
-DGGML_CUDA_FA=ON -DGGML_CUDA_FA_HALF_QUANTS=ON ^
352+
-DGGML_CUDA_FA=ON ^
353353
-DCMAKE_BUILD_TYPE=Release
354354
cmake --build build --config Release --parallel
355355

@@ -358,7 +358,7 @@ cmake -B build -DGGML_METAL=ON -DCMAKE_BUILD_TYPE=Release
358358
cmake --build build -j
359359
```
360360

361-
`GGML_CUDA_FA_HALF_QUANTS=ON` is the recommended CUDA FlashAttention build mode. It compiles the useful asymmetric K/V cache type half-matrix plus f16 fallback pairs needed by TurboQuant/TCQ dequant paths: 103 FA vec K/V pairs instead of 169 in ALL_QUANTS. Use `GGML_CUDA_FA_ALL_QUANTS=ON` only when you need the full K/V matrix or arbitrary asymmetric cache-type combinations. These flags are mutually exclusive.
361+
Without `GGML_CUDA_FA_HALF_QUANTS` or `GGML_CUDA_FA_ALL_QUANTS`, the CUDA FlashAttention build compiles Bee's q/KVarN-fallback default: 62 K>=V vec pairs with fp cache types capped at q5 and q8/q6 capped at q4. This default leaves TurboQuant/TCQ FA pairs out; use `GGML_CUDA_FA_HALF_QUANTS=ON` for the larger 208-pair Turbo/TCQ-capable half-matrix, or `GGML_CUDA_FA_ALL_QUANTS=ON` for the full 361-pair matrix. These flags are mutually exclusive.
362362

363363
### Other Backends
364364

‎docs/build.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -297,8 +297,8 @@ The following compilation options are also available to tweak performance:
297297
| GGML_CUDA_FORCE_MMQ | Boolean | false | Force the use of custom matrix multiplication kernels for quantized models instead of FP16 cuBLAS even if there is no int8 tensor core implementation available (affects V100, CDNA and RDNA3+). MMQ kernels are enabled by default on GPUs with int8 tensor core support. With MMQ force enabled, speed for large batch sizes will be worse but VRAM consumption will be lower. |
298298
| GGML_CUDA_FORCE_CUBLAS | Boolean | false | Force the use of FP16 cuBLAS instead of custom matrix multiplication kernels for quantized models. There may be issues with numerical overflows (except for V100, CDNA and RDNA4 which use FP32 compute type by default) and memory use will be higher. Prompt processing may become faster on recent datacenter GPUs (the custom kernels were tuned primarily for RTX 3000/4000). |
299299
| GGML_CUDA_PEER_MAX_BATCH_SIZE | Positive integer | 128 | Maximum batch size for which to enable peer access between multiple GPUs. Peer access requires either Linux or NVLink. When using NVLink enabling peer access for larger batch sizes is potentially beneficial. |
300-
| GGML_CUDA_FA_ALL_QUANTS | Boolean | false | Compile CUDA FlashAttention vec kernels for the full supported K/V cache type matrix: 169 pairs for the 13-type universe. This is useful for arbitrary asymmetric cache-type combinations. Mutually exclusive with GGML_CUDA_FA_HALF_QUANTS. |
301-
| GGML_CUDA_FA_HALF_QUANTS | Boolean | false | Compile the recommended CUDA FlashAttention vec K/V cache pair set for this fork: the useful K>=V half-matrix plus f16 fallback pairs needed by TurboQuant/TCQ dequant paths. This compiles 103 pairs instead of 169 in GGML_CUDA_FA_ALL_QUANTS. Types are ranked from higher precision to lower: f16 > bf16 > q8_0 > q6_0 > q5_1 > q5_0 > turbo4 > q4_1 > q4_0 > turbo3_tcq > turbo3 > turbo2_tcq > turbo2. Mutually exclusive with GGML_CUDA_FA_ALL_QUANTS. |
300+
| GGML_CUDA_FA_ALL_QUANTS | Boolean | false | Compile CUDA FlashAttention vec kernels for the full supported K/V cache type matrix: 361 pairs for the 19-type universe. This is useful for arbitrary asymmetric cache-type combinations. Mutually exclusive with GGML_CUDA_FA_HALF_QUANTS. |
301+
| GGML_CUDA_FA_HALF_QUANTS | Boolean | false | Compile the larger CUDA FlashAttention vec K/V cache pair set for this fork: the useful K>=V half-matrix plus f16 fallback pairs needed by TurboQuant/TCQ dequant paths. This compiles 208 pairs instead of 361 in GGML_CUDA_FA_ALL_QUANTS. Types are ranked from higher precision to lower: f16 > bf16 > q8_0 > q6_1 > q6_0 > q5_1 > q5_0 > turbo4_tcq > turbo4 > q4_1 > q4_0 > q3_1 > turbo3_tcq > turbo3 > q3_0 > q2_1 > turbo2_tcq > turbo2 > q2_0. Mutually exclusive with GGML_CUDA_FA_ALL_QUANTS. Without either flag, the default is 62 q/KVarN-fallback pairs: K>=V, no Turbo/TCQ, no mixed f16/bf16, fp capped at q5, and q8/q6 capped at q4. |
302302

303303
## MUSA
304304

‎docs/quickstart-gemma-4-31b-dflash.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -71,7 +71,7 @@ cmake -B build -DGGML_CUDA=ON -DGGML_NATIVE=ON \
7171
cmake --build build -j
7272
```
7373

74-
`GGML_CUDA_FA_HALF_QUANTS=ON` is the recommended CUDA FlashAttention build mode for these DFlash/TurboQuant quickstarts. It compiles the useful asymmetric K/V cache type half-matrix plus f16 fallback pairs needed by TurboQuant/TCQ dequant paths. Use `GGML_CUDA_FA_ALL_QUANTS=ON` only if you need the full 169-pair K/V matrix or arbitrary asymmetric cache-type combinations. These flags are mutually exclusive.
74+
`GGML_CUDA_FA_HALF_QUANTS=ON` is the recommended CUDA FlashAttention build mode for DFlash/TurboQuant quickstarts because it compiles the Turbo/TCQ-capable 208-pair half-matrix plus f16 fallback pairs needed by TurboQuant/TCQ dequant paths. If you are staying on standard q cache types or KVarN fallback types, the no-flag default compiles the smaller 62-pair q/KVarN-fallback set. Use `GGML_CUDA_FA_ALL_QUANTS=ON` only if you need the full 361-pair K/V matrix or arbitrary asymmetric cache-type combinations. These flags are mutually exclusive.
7575

7676
**macOS (Metal).**
7777

‎docs/quickstart-qwen36-dflash.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -71,7 +71,7 @@ cmake -B build -DGGML_CUDA=ON -DGGML_NATIVE=ON \
7171
cmake --build build -j
7272
```
7373

74-
`GGML_CUDA_FA_HALF_QUANTS=ON` is the recommended CUDA FlashAttention build mode for these DFlash/TurboQuant quickstarts. It compiles the useful asymmetric K/V cache type half-matrix plus f16 fallback pairs needed by TurboQuant/TCQ dequant paths. Use `GGML_CUDA_FA_ALL_QUANTS=ON` only if you need the full 169-pair K/V matrix or arbitrary asymmetric cache-type combinations. These flags are mutually exclusive.
74+
`GGML_CUDA_FA_HALF_QUANTS=ON` is the recommended CUDA FlashAttention build mode for DFlash/TurboQuant quickstarts because it compiles the Turbo/TCQ-capable 208-pair half-matrix plus f16 fallback pairs needed by TurboQuant/TCQ dequant paths. If you are staying on standard q cache types or KVarN fallback types, the no-flag default compiles the smaller 62-pair q/KVarN-fallback set. Use `GGML_CUDA_FA_ALL_QUANTS=ON` only if you need the full 361-pair K/V matrix or arbitrary asymmetric cache-type combinations. These flags are mutually exclusive.
7575

7676
**macOS (Metal).**
7777

‎ggml/src/ggml-cuda/CMakeLists.txt‎

Lines changed: 65 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -181,43 +181,74 @@ if (CUDAToolkit_FOUND)
181181
else()
182182
set(GGML_CUDA_FA_DEFAULT_KV_PAIRS
183183
f16:f16
184-
q4_0:q4_0
185-
q8_0:q8_0
184+
f16:q8_0
185+
f16:q6_1
186+
f16:q6_0
187+
f16:q5_1
188+
f16:q5_0
186189
bf16:bf16
187-
turbo2_0:turbo2_0
188-
turbo3_0:turbo3_0
189-
turbo4_0:turbo4_0
190-
turbo4_tcq:turbo4_tcq
191-
turbo3_tcq:turbo3_tcq
192-
turbo2_tcq:turbo2_tcq
193-
turbo2_0:q8_0
194-
turbo3_0:q8_0
195-
turbo4_0:q8_0
196-
q8_0:turbo2_0
197-
q8_0:turbo3_0
198-
q8_0:turbo4_0
199-
turbo4_0:turbo3_0
200-
turbo3_0:turbo4_0
201-
turbo2_0:turbo3_0
202-
turbo3_0:turbo2_0
203-
turbo4_tcq:q8_0
204-
turbo3_tcq:q8_0
205-
turbo2_tcq:q8_0
206-
q8_0:turbo4_tcq
207-
q8_0:turbo3_tcq
208-
q8_0:turbo2_tcq
209-
turbo4_tcq:turbo3_tcq
210-
turbo3_tcq:turbo4_tcq
211-
turbo4_tcq:turbo2_tcq
212-
turbo2_tcq:turbo4_tcq
213-
turbo3_tcq:turbo2_tcq
214-
turbo2_tcq:turbo3_tcq
215-
turbo4_0:turbo4_tcq
216-
turbo3_0:turbo4_tcq
217-
turbo4_0:turbo3_tcq
218-
turbo3_0:turbo3_tcq
190+
bf16:q8_0
191+
bf16:q6_1
192+
bf16:q6_0
193+
bf16:q5_1
194+
bf16:q5_0
195+
q8_0:q8_0
196+
q8_0:q6_1
197+
q8_0:q6_0
198+
q8_0:q5_1
199+
q8_0:q5_0
200+
q8_0:q4_1
201+
q8_0:q4_0
202+
q6_1:q6_1
203+
q6_1:q6_0
204+
q6_1:q5_1
205+
q6_1:q5_0
206+
q6_1:q4_1
207+
q6_1:q4_0
208+
q6_0:q6_0
209+
q6_0:q5_1
210+
q6_0:q5_0
211+
q6_0:q4_1
212+
q6_0:q4_0
213+
q5_1:q5_1
214+
q5_1:q5_0
215+
q5_1:q4_1
216+
q5_1:q4_0
217+
q5_1:q3_1
218+
q5_1:q3_0
219+
q5_0:q5_0
220+
q5_0:q4_1
221+
q5_0:q4_0
222+
q5_0:q3_1
223+
q5_0:q3_0
224+
q4_1:q4_1
225+
q4_1:q4_0
226+
q4_1:q3_1
227+
q4_1:q3_0
228+
q4_1:q2_1
229+
q4_1:q2_0
230+
q4_0:q4_0
231+
q4_0:q3_1
232+
q4_0:q3_0
233+
q4_0:q2_1
234+
q4_0:q2_0
235+
q3_1:q3_1
236+
q3_1:q3_0
237+
q3_1:q2_1
238+
q3_1:q2_0
239+
q3_0:q3_0
240+
q3_0:q2_1
241+
q3_0:q2_0
242+
q2_1:q2_1
243+
q2_1:q2_0
244+
q2_0:q2_0
219245
)
220246

247+
list(LENGTH GGML_CUDA_FA_DEFAULT_KV_PAIRS GGML_CUDA_FA_DEFAULT_PAIR_COUNT)
248+
if (NOT GGML_CUDA_FA_DEFAULT_PAIR_COUNT EQUAL 62)
249+
message(FATAL_ERROR "Default CUDA FA vec policy expected 62 pairs, got ${GGML_CUDA_FA_DEFAULT_PAIR_COUNT}")
250+
endif()
251+
221252
foreach(PAIR ${GGML_CUDA_FA_DEFAULT_KV_PAIRS})
222253
string(REPLACE ":" ";" PARTS "${PAIR}")
223254
list(GET PARTS 0 K_TYPE)

0 commit comments

Comments
 (0)