Skip to content

Commit b37e5f0

Browse files
TheTomclaude
andcommitted
feat: Metal kernels for TurboQuant KV cache (turbo3, turbo4) PrismML-Eng#21
Added Metal shader implementations: - quantize_turbo3_0 / quantize_turbo4_0 (per-block quantization) - dequantize_turbo3_0 / dequantize_turbo4_0 (type4x4 and type4 variants) - kernel_set_rows_turbo template (128-element block size) - Flash attention instantiations for all dk/dv variants Added TURBO3_0/TURBO4_0 to Metal device SET_ROWS validation. Builds successfully. Testing with Qwen 3.5 35B-A3B MoE on M5 Max. Note: Initial version uses simplified quantization (no rotation matrix) for Metal compatibility. Full rotation requires custom kernel with extra buffer bindings — tracked for follow-up. Co-Authored-By: tturney@psyguard.ai Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 0b37448 commit b37e5f0

2 files changed

Lines changed: 282 additions & 0 deletions

File tree

ggml/src/ggml-metal/ggml-metal-device.m

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1266,6 +1266,8 @@ bool ggml_metal_device_supports_op(ggml_metal_device_t dev, const struct ggml_te
12661266
case GGML_TYPE_Q5_0:
12671267
case GGML_TYPE_Q5_1:
12681268
case GGML_TYPE_IQ4_NL:
1269+
case GGML_TYPE_TURBO3_0:
1270+
case GGML_TYPE_TURBO4_0:
12691271
return true;
12701272
default:
12711273
return false;

0 commit comments

Comments
 (0)