Skip to content

Commit b7b8dd0

Browse files
retroheimclaude
andcommitted
turbo Vulkan port: wire turbo KV cache into flash attention (Phase B)
Completes the turbo Vulkan port. turbo KV cache now runs end-to-end on Vulkan. Phase B wiring: - ggml-vulkan supports_op FLASH_ATTN_EXT: accept TURBO2/3/4_0 K/V (coopmat2-only, like Q1_0). fa_block_bytes already generic (ggml_type_size); TURBO_WHT op + SET_ROWS for turbo + cm2 dequantFuncTURBO3_0 survived earlier merges. - common/arg.cpp: re-added turbo2/3/4 to kv_cache_types (-ctk/-ctv turbo3). Status (AMD Radeon RX 9070 XT / RADV GFX1201): - turbo3 KV cache RUNS via llama-bench (tg64 @ d2048 = 12.2 t/s). - CORRECTNESS NOT VALIDATED on this GPU: RADV flash-attention is broken on RDNA4/GFX1201 for ALL types — test-backend-ops FLASH_ATTN_EXT 864/8453 pass, f16 KV perplexity = 9.4M with -fa vs 1.0092 without. Base (non-FA) Vulkan inference is correct (PPL 1.0092 == CPU); turbo weights validated. turbo KV correctness depends on FA, which this driver miscomputes for f16 too. -> Validate turbo KV on conformant-FA hardware (NVIDIA/CUDA or fixed RADV). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 1a859aa commit b7b8dd0

2 files changed

Lines changed: 8 additions & 0 deletions

File tree

common/arg.cpp

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -397,6 +397,9 @@ const std::vector<ggml_type> kv_cache_types = {
397397
GGML_TYPE_IQ4_NL,
398398
GGML_TYPE_Q5_0,
399399
GGML_TYPE_Q5_1,
400+
GGML_TYPE_TURBO2_0, // ik_llama port: TurboQuant 2-bit rotated KV
401+
GGML_TYPE_TURBO3_0, // ik_llama port: TurboQuant 3-bit rotated KV
402+
GGML_TYPE_TURBO4_0, // ik_llama port: TurboQuant 4-bit rotated KV
400403
};
401404

402405
static ggml_type kv_cache_type_from_str(const std::string & s) {

ggml/src/ggml-vulkan/ggml-vulkan.cpp

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16531,6 +16531,11 @@ static bool ggml_backend_vk_device_supports_op(ggml_backend_dev_t dev, const ggm
1653116531
case GGML_TYPE_Q4_0:
1653216532
return true;
1653316533
case GGML_TYPE_Q1_0:
16534+
case GGML_TYPE_TURBO2_0:
16535+
case GGML_TYPE_TURBO3_0:
16536+
case GGML_TYPE_TURBO4_0:
16537+
// ik_llama port: turbo KV cache types — coopmat2 FA only
16538+
// (flash_attn_cm2 decodes them via dequantFuncTURBO*).
1653416539
return coopmat2;
1653516540
default:
1653616541
return false;

0 commit comments

Comments
 (0)