Skip to content

Commit 9fb1ee6

Browse files
committed
fix(v4-path-d): correct kkt BT/BC contract + with_state BV tiling (3/8 cells green; ffi-NULL unblocked)
The ffi-NULL fix (tilelang 168877b0) took Path D from 0/8 launched -> 8/8 launched, exposing two cppmega-side launch-config bugs (RULE #1 surfaced them; no fallback): - kkt_solve was launched BT=32,BC=16 but the FLA kkt kernel hard-codes a 4-sub-block decomposition requiring BT==4*BC -> wrong/NaN. Fixed BC=GDN_RUNTIME_BT//4=8 + pinned _NUM_WARPS=1 for the [8,8] GEMM (plumbed _NUM_WARPS through the lowering driver into from_ttir(num_warps=...)). - with_state initial-state chunk_h exceeded Apple's compute-function stack -> lowered GDN_CHUNK_H_METAL_SAFE_BV 16->8 (the predicted recompute_w_u-style BV tiling). Result: 3/8 cells green in isolation (fixed_k64_v32 0.018, custom_scale 0.009, with_state 0.019 -- named residual solved; kkt fix dropped fixed_k64 0.137->0.018). No fallback added; unsupported shapes still RAISE PathDRuntimeUnavailable. REMAINING (tilelang-side, separate fix): cells with chunks/seq>=3 or multi-head HV>=2 still NaN/wrong due to the TVM-FFI Metal bridge lazy-multi-launch buffer/producer-registry id()-reuse hazard (same metal_graph_sync issue) -- needs a tilelang bridge fix, not cppmega.
1 parent 24be790 commit 9fb1ee6

2 files changed

Lines changed: 45 additions & 2 deletions

File tree

cppmega_v4/_tilelang/linear_attention_path_d_real.py

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -876,6 +876,18 @@ def _lower_fla_kernel_uncached(
876876

877877
# Prefer ``from_ttir`` directly so we control the text-vs-mlir routing;
878878
# ``from_triton_kernel`` would re-capture and double the work.
879+
#
880+
# A stage may pin the cooperative-tensor warp count via a private
881+
# ``_NUM_WARPS`` constexpr (filtered from the TTIR signature above because
882+
# it is ``_``-prefixed). This is required for small-tile GEMMs such as the
883+
# GDN kkt_solve at BC=8: a [8, 8] block cannot be partitioned across the
884+
# lowering's default 4 warps (``m_warp * n_warp must equal num_warps``), so
885+
# the stage requests ``num_warps=1``. It is a launch-config override, not a
886+
# change to the lowered kernel body.
887+
num_warps_override = constexprs.get("_NUM_WARPS")
888+
num_warps_override = (
889+
int(num_warps_override) if num_warps_override is not None else None
890+
)
879891
try:
880892
from poc.triton_frontend import from_ttir, _walk_text_ttir
881893

@@ -887,6 +899,7 @@ def _lower_fla_kernel_uncached(
887899
name=primfunc_name,
888900
grid=grid,
889901
arg_buffer_shapes=arg_buffer_shapes,
902+
num_warps=num_warps_override,
890903
)
891904
return LowerResult(
892905
status="LOWERED_FULL",

cppmega_v4/_tilelang/path_d_runtime_adapter.py

Lines changed: 32 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,17 @@
3535
GDN_FIXED_BT = 64
3636
GDN_FIXED_BV = 32
3737
GDN_RUNTIME_BT = 32
38-
GDN_CHUNK_H_METAL_SAFE_BV = 16
38+
# chunk_delta_h (h_blockdim64) holds [64, BV] fp32 state accumulators (b_h1..)
39+
# thread-private. With an initial state loaded (USE_INITIAL_STATE, the
40+
# ``with_state`` shape) the extra h0 load + the BV-wide store path pushes the
41+
# per-thread footprint past Apple's compute-function stack budget at BV=16,
42+
# so newComputePipelineStateWithFunction fails ("Compute function exceeds
43+
# available stack space"). Tiling the V reduction to BV=8 (the smallest tile
44+
# Metal's GEMM allows -- N must be a multiple of 8) shrinks the state tile so
45+
# the pipeline-state creation fits, mirroring the recompute_w_u BK/BV tiling
46+
# below. BV=8 still launches and stays in parity for the no-initial-state
47+
# shapes, so it is used uniformly.
48+
GDN_CHUNK_H_METAL_SAFE_BV = 8
3949
GDN_FIXED_CHUNK_O_BV = 16
4050
# recompute_w_u (shared by GDN and KDA) materializes several BT*BK + BT*BV
4151
# THREAD-PRIVATE accumulators (b_w, b_u, b_kb, b_vb, b_A, ...). With the FLA
@@ -1926,7 +1936,27 @@ def _compile_gdn_runtime_stages(
19261936
"BT": GDN_RUNTIME_BT,
19271937
"IS_VARLEN": bool(is_varlen),
19281938
}
1929-
kkt_constexprs = {**runtime_private, **shared_dims}
1939+
# The FLA kkt_solve kernel hard-codes a 4-sub-block (BC x 4) decomposition
1940+
# of each BT-token chunk: it reads/inverts/stores blocks at column offsets
1941+
# 0, BC, 2*BC, 3*BC, so it is only correct when BT == 4 * BC. The default
1942+
# FLA pairing is BT=64 / BC=16. When this multi-stage launcher runs the
1943+
# whole GDN pipeline at the Metal-safe GDN_RUNTIME_BT (=32, because the
1944+
# chunk_o / chunk_h inter-chunk kernels overflow Apple's 32 KiB threadgroup
1945+
# memory at BT=64), the kkt stage MUST shrink BC to BT/4 so its 4 sub-blocks
1946+
# still tile exactly one BT chunk -- otherwise sub-blocks 2/3 read the next
1947+
# chunk's tokens and the (I+A)^{-1} merge writes past the [T, BT] A buffer,
1948+
# producing a wrong inverse that drifts across chunks (and NaNs past 2
1949+
# chunks). A [BC, BC] = [8, 8] GEMM cannot be split across the lowering's
1950+
# default 4 warps (m_warp * n_warp must equal num_warps), so the kkt stage
1951+
# also pins num_warps=1 for the small-BC tile. Both are launch-config
1952+
# corrections on the cppmega side; the lowered kernel itself is unchanged.
1953+
kkt_bc = max(1, GDN_RUNTIME_BT // 4)
1954+
kkt_constexprs = {
1955+
**runtime_private,
1956+
**shared_dims,
1957+
"BC": kkt_bc,
1958+
"_NUM_WARPS": 1 if kkt_bc < 16 else 4,
1959+
}
19301960
recompute_constexprs = {
19311961
**runtime_private,
19321962
**shared_dims,

0 commit comments

Comments
 (0)