Skip to content

Commit 83d953e

Browse files
committed
fix(path_c bwd): dseg — branchless unmasked T.Parallel body (CUDA barrier-in-mask scope bug, 2nd)
The T.Parallel(L*L) dseg rewrite STILL failed compile: 'cb_v' undefined. The TileLang shared-mem barrier pass inserts __syncthreads() between the global cb load and the shared dAcs_acc atomic; when that barrier lands INSIDE the 'if ss<ll' masked body it re-emits the if-guard per fragment WITHOUT hoisting the cb_v/dseg locals -> out of scope. The unmasked dC apply / DYX build blocks above survive the identical barrier ONLY because they carry no enclosing if. Fix: make the dseg body BRANCHLESS (no if mask) — compute cb_v/lmat/dt_s unconditionally (all in-bounds for any (ll,ss) in [0,L)^2; lmat for ss>ll is a bounded small positive, no overflow) and apply the strict-lower-tri (ss<ll) selection as a multiplicative 0/1 mask via T.if_then_else (a VALUE, not control flow). DYX[ll,ss]=0 already for ss>ll and the mask kills the ss==ll diagonal, so dseg=0 off the strict lower-tri -> the +/- scatter is a no-op there. Math IDENTICAL to the Metal prim. Now structurally identical to the proven dC apply block (unmasked T.Parallel + shared atomic_add). Metal prim untouched. RULE #1.
1 parent 0db3e88 commit 83d953e

1 file changed

Lines changed: 21 additions & 18 deletions

File tree

cppmega_mlx/nn/_tilelang/mamba3_chunked_backward_core.py

Lines changed: 21 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -661,27 +661,30 @@ def main(
661661
# dLmat[l,s] = sum_p dY[l,p]*x[s,p] == DYX[l,s] (reuse, zero recompute)
662662
# M[l,s] = cb*lmat*dt_s ; dseg = DYX[l,s]*M[l,s]
663663
# segsum-vjp: +dAcs_acc[l] over l>=s, -dAcs_acc[s] over s<l (strictly l>s).
664-
# Map the L*L (ll,ss) grid over T.Parallel (the SAME un-fragmented
665-
# thread-parallel idiom the dC apply / DYX build above use — a
666-
# lane-strided T.serial here gets its masked body split across the
667-
# shared-mem barriers TileLang inserts between the global cb load and the
668-
# atomic scatter, hoisting the locals out of scope). The dseg compute
669-
# lands in a thread-local; the +/- scatter (ll>ss mask) races on shared
670-
# dAcs_acc -> T.atomic_add. dseg=0 off the strict lower-tri (no-op adds).
664+
# Map the L*L (ll,ss) grid over T.Parallel with a BRANCHLESS body (NO
665+
# `if ss<ll` mask): TileLang inserts a __syncthreads() between the global
666+
# cb load and the shared-dAcs_acc atomic, and when that barrier lands
667+
# INSIDE an if-masked region it re-emits the guard per fragment WITHOUT
668+
# hoisting the locals (cb_v/dseg fall out of scope -> 'undefined'). The
669+
# unmasked dC apply / DYX build above survive the same barrier precisely
670+
# because they are NOT inside an if. So here cb_v/lmat/dt_s are computed
671+
# unconditionally (all in-bounds for any (ll,ss) in [0,L)^2) and the
672+
# strict-lower-tri (ss<ll) selection is a BRANCHLESS multiplicative mask:
673+
# T.if_then_else yields a value, not control flow. (DYX[ll,ss]=0 already
674+
# for ss>ll; the mask also kills the ss==ll diagonal -> dseg=0 off the
675+
# strict lower-tri, so the +/- scatter is a no-op there. Math IDENTICAL.)
671676
for ls in T.Parallel(L * L):
672677
ll = ls // L
673678
ss = ls % L
674-
dseg = T.alloc_local((1,), accum_dtype)
675-
dseg[0] = T.Cast(accum_dtype, 0)
676-
if ss < ll:
677-
cb_v = T.Cast(
678-
accum_dtype, cb[batch_idx, chunk_idx, group_idx, ll, ss]
679-
)
680-
lmat = T.exp2((dacs[ll] - dacs[ss]) * p)
681-
dt_s = T.Cast(accum_dtype, dt[batch_idx, head_idx, chunk_idx, ss])
682-
dseg[0] = DYX[ll, ss] * cb_v * lmat * dt_s
683-
T.atomic_add(dAcs_acc[ll], dseg[0])
684-
T.atomic_add(dAcs_acc[ss], -dseg[0])
679+
cb_v = T.Cast(accum_dtype, cb[batch_idx, chunk_idx, group_idx, ll, ss])
680+
lmat = T.exp2((dacs[ll] - dacs[ss]) * p)
681+
dt_s = T.Cast(accum_dtype, dt[batch_idx, head_idx, chunk_idx, ss])
682+
tri = T.if_then_else(
683+
ss < ll, T.Cast(accum_dtype, 1), T.Cast(accum_dtype, 0)
684+
)
685+
dseg = DYX[ll, ss] * cb_v * lmat * dt_s * tri
686+
T.atomic_add(dAcs_acc[ll], dseg)
687+
T.atomic_add(dAcs_acc[ss], -dseg)
685688
T.sync_threads()
686689

687690
for l in T.Parallel(L):

0 commit comments

Comments
 (0)