Skip to content

Commit 1090d51

Browse files
howard0suCopilot
andcommitted
gemma4: add BSA sparse-FA prefill path + unified flash_prefill_forward dispatch
- Implement gemma4_prefill_bsa() for per-layer BSA prefill using flash_prefill_forward for SWA layers (head_dim=128) with dense FA fallback for full-attention layers (head_dim=256). - Write KV cache during Graph A (ring-buffer aware for SWA layers). - Add GGML_ASSERT guard for swa_size > 0 before modulo operation. - Add flash_prefill_forward() unified dispatch to flashprefill.h that selects bf16/f16/q8 kernel based on compile flags + buffer type. - Simplify Qwen3 attention dispatch to use the unified function. - Remove duplicated ifdef boilerplate from both model implementations. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent de0e4c1 commit 1090d51

4 files changed

Lines changed: 760 additions & 68 deletions

File tree

dflash/src/flashprefill.h

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -90,6 +90,37 @@ int flash_prefill_forward_q8(
9090
ggml_type qkv_type,
9191
const FlashPrefillConfig & cfg);
9292

93+
// ── Unified dispatch ──────────────────────────────────────────────────────────
94+
// Picks the best available kernel at compile time + runtime buffer type:
95+
// BF16 buffers + sm_80 build → flash_prefill_forward_bf16
96+
// F16 buffers + Volta build → flash_prefill_forward_f16
97+
// otherwise → flash_prefill_forward_q8 (ggml FA fallback)
98+
//
99+
// Callers no longer need to duplicate the ifdef/dispatch boilerplate.
100+
inline int flash_prefill_forward(
101+
ggml_backend_t backend,
102+
const void * Q, const void * K, const void * V, void * O,
103+
int batch, int seq_len, int n_q_heads, int n_k_heads, int head_dim,
104+
float scale,
105+
ggml_type qkv_type,
106+
const FlashPrefillConfig & cfg)
107+
{
108+
#if defined(DFLASH27B_HAVE_FLASHPREFILL) || defined(DFLASH27B_HAVE_SM80_FLASHPREFILL)
109+
if (qkv_type == GGML_TYPE_BF16) {
110+
return flash_prefill_forward_bf16(Q, K, V, O,
111+
batch, seq_len, n_q_heads, n_k_heads, head_dim, scale, cfg);
112+
}
113+
#endif
114+
#if defined(DFLASH27B_HAVE_VOLTA_FLASHPREFILL) || defined(DFLASH27B_HAVE_PASCAL_FLASHPREFILL)
115+
if (qkv_type == GGML_TYPE_F16) {
116+
return flash_prefill_forward_f16(Q, K, V, O,
117+
batch, seq_len, n_q_heads, n_k_heads, head_dim, scale, cfg);
118+
}
119+
#endif
120+
return flash_prefill_forward_q8(backend, Q, K, V, O,
121+
batch, seq_len, n_q_heads, n_k_heads, head_dim, scale, qkv_type, cfg);
122+
}
123+
93124
#ifdef DFLASH27B_HAVE_BSA
94125
// Free BSA persistent device buffers (blockmask, head_mask_type, softmax_lse).
95126
// Safe to call any time; idempotent. Useful before unloading the drafter to

0 commit comments

Comments
 (0)