Skip to content

Commit 14d4810

Browse files
committed
Fix tq4_sdpa split-K dispatch for export/AOTI tracing
The split-K decode path was branching on runtime kv_len value via kv_len_t.item() >= threshold, which is data-dependent control flow that breaks torch.export/AOTI tracing. Fix: Make dispatch static by using N_KV (buffer size) instead of runtime kv_len value. The kv_len tensor is still used inside the kernel for bounds checking via tl.load on device (CUDA-graph safe). - Dispatch now based on static N_KV >= 256 (buffer size) - Removes incorrect guard_or_false usage with .item() - Preserves kv_len on-device usage in kernel for bounds - Maintains byte-identical behavior for qwen (kv_len fallback) - Eager correctness preserved
1 parent f786724 commit 14d4810

1 file changed

Lines changed: 4 additions & 1 deletion

File tree

backends/cuda/triton/kernels/tq4_sdpa.py

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -800,12 +800,15 @@ def tq4_sdpa(
800800

801801
# Split-K decode dispatch: L_q == 1 AND kv_len >= threshold (256)
802802
# Uses flash-decoding to partition KV across many CTAs for better occupancy
803+
# Dispatch is static (based on buffer size N_KV, not runtime kv_len value)
804+
# to be export/AOTI traceable. The kernel still uses kv_len on-device
805+
# via tl.load for bounds checking (CUDA-graph safe).
803806
_SPLITK_LKV_THRESHOLD = 256
804807
use_splitk = (
805808
N_Q == 1
806809
and HAS_KV_LEN
807810
and kv_len_t is not None
808-
and kv_len_t.item() >= _SPLITK_LKV_THRESHOLD
811+
and N_KV >= _SPLITK_LKV_THRESHOLD
809812
)
810813

811814
if use_splitk:

0 commit comments

Comments
 (0)