Skip to content

Commit 8344356

Browse files
committed
feat(path_c B2): batched large-tile tensor-core GEMM twin (CUDA mma.sync + Metal simdgroup)
NEW prims (flag-gated, the ONE path when on, RAISE on failure — RULE #1; serial/§27 prims kept byte-identical): * chunk_scan_combine_bwd_cuda_prim_gemm_batched (sm_121, fragment-C): grid (batch*nchunks, nheads//HEADS_PER_CTA); each CTA owns HEADS_PER_CTA heads and issues the 4 GEMM-able contractions (DYX, dC_off, dC_diag masked, dchunk_states transpose_A) as a HEAD-LOOP over a SHARED tall-M operand band (M = HEADS_PER_CTA*64), amortizing ldmatrix/staging/sync over the band — the P1/Tri-Dao lever the §27 single-64-tile twin (0.749x) lacked. dinp 3-index term + dz/dx/dD + dseg stay threaded (documented scope boundary). HEADS_PER_CTA default 4. * chunk_scan_combine_bwd_metal_gemm_prim_batched (simdgroup, C-in-frag): batches DYX+dchunk_states only (Apple 32KB pool); HEADS_PER_CTA default 2. Flags: CPPMEGA_PATH_C_B2_GEMM_BATCHED (cuda) / CPPMEGA_PATH_C_METAL_GEMM_BATCHED (metal), HEADS_PER_CTA via CPPMEGA_PATH_C_B2_HEADS_PER_CTA / CPPMEGA_PATH_C_METAL_HEADS_PER_CTA. Mutually exclusive with the other B2 flags (RAISE). Per-backend smem budget gate RAISES over-budget (gb10 99KB / Apple 32KB), no silent re-tile. z3 PROOF WIRED (RULE #1, fail-closed): _gemm_rewrite_proof.py gains b2_batched_dchunk/dcoff/dcdiag_contraction + b2_batched_tiling (m_blocks=HEADS_PER_CTA, m_stride=tile_m per-head-band single-writer disjointness). build dispatcher calls require_gemm_rewrite_proof before emitting the batched prim — RAISES GemmRewriteNotProven on sat/unknown/disabled. Verified locally: 3 rewrites discharge z3_proved=True, single_writer True; non-vacuous (overlap-band -> single_writer False, transpose bug -> maps False); fail-closed when z3 disabled. probe_chunked_backward_cuda_gb10.py: B2-BATCHED-AB arm (vs v1 + vs single-tile gemm, HEADS_PER_CTA-parametrized) for the gb10 measure (bs1 isolates tiling; bs4 = full recipe). 8-grad parity + tile microbench is the gb10/Test step (local tilelang import is broken; no heavy local python per 80GB guard). MEASURED win is EXTRAPOLATION until gb10 runs.
1 parent 2bd251d commit 8344356

3 files changed

Lines changed: 975 additions & 7 deletions

File tree

cppmega_mlx/nn/_tilelang/_gemm_rewrite_proof.py

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -547,6 +547,103 @@ def b2_dinp_contraction(z3, *, chunk_size: int, headdim: int):
547547
)
548548

549549

550+
def b2_batched_dchunk_contraction(z3, *, chunk_size: int, headdim: int, dstate: int):
551+
"""B2 BATCHED ``dchunk_states[p,n] = sum_l (decay[l]*dY[l,p])*C[l,n]`` (transpose_A).
552+
553+
The batched-large-tile prim stacks ``HEADS_PER_CTA`` heads along M; each head is
554+
a DISJOINT 64-row band of the tall opA tile and is GEMM'd against its own C
555+
operand (block-diagonal). This descriptor proves the PER-HEAD contraction (the
556+
operand maps + the transpose_A address arithmetic + the decay scale fold); the
557+
per-band single-writer disjointness is proved by the GemmTiling
558+
``m_blocks=HEADS_PER_CTA, m_stride=tile_m`` passed alongside (each band owns a
559+
contiguous 64-row slice, no overlap). Output rows = p (headdim), cols = n
560+
(dstate), reduction k = l (chunk). The decay ``sd[l]=exp2(dacs[l]*p)`` is a
561+
function of the reduction index l only, folded into the dY operand BEFORE the
562+
GEMM — modeled as an uninterpreted ``scale(l)`` that the GEMM folds identically.
563+
"""
564+
565+
scale = z3.Function("b2_sd", z3.IntSort(), z3.RealSort())
566+
return GemmContraction(
567+
name="B2_batched_dchunk_states",
568+
m_extent=headdim,
569+
n_extent=dstate,
570+
k_extent=chunk_size,
571+
a_addr_serial=lambda i, k: k * headdim + i, # dY[l, p] (transpose_A: k=l rows)
572+
a_addr_gemm=lambda i, k: k * headdim + i,
573+
b_addr_serial=lambda k, j: k * dstate + j, # C[l, n]
574+
b_addr_gemm=lambda k, j: k * dstate + j,
575+
scale_serial=lambda i, k, j: scale(k),
576+
scale_gemm=lambda i, k, j: scale(k),
577+
)
578+
579+
580+
def b2_batched_dcoff_contraction(z3, *, chunk_size: int, headdim: int, dstate: int):
581+
"""B2 BATCHED ``dC_off[l,n] = sum_p dY[l,p]*prev_states[p,n]`` (dense, un-transposed).
582+
583+
Per-head sub-GEMM inside the head-band. Output rows = l (chunk), cols = n
584+
(dstate), reduction k = p (headdim). No mask, no fold on the GEMM itself (the
585+
per-l state_decay is applied post-GEMM in the dC store). Address maps encode the
586+
un-transposed ``dY[l,p] @ prev_states[p,n]``.
587+
"""
588+
589+
return GemmContraction(
590+
name="B2_batched_dC_off",
591+
m_extent=chunk_size,
592+
n_extent=dstate,
593+
k_extent=headdim,
594+
a_addr_serial=lambda i, k: i * headdim + k, # dY[l, p]
595+
a_addr_gemm=lambda i, k: i * headdim + k,
596+
b_addr_serial=lambda k, j: k * dstate + j, # prev_states[p, n]
597+
b_addr_gemm=lambda k, j: k * dstate + j,
598+
)
599+
600+
601+
def b2_batched_dcdiag_contraction(z3, *, chunk_size: int, dstate: int):
602+
"""B2 BATCHED ``dC_diag[l,n] = sum_{s<=l} M[l,s]*B[s,n]`` (lower-tri masked).
603+
604+
Per-head sub-GEMM. The causal mask ``keep(i,j,k)=(k<=i)`` (s<=l) is applied to
605+
the A-operand fragment BEFORE the GEMM — the GEMM must fold the IDENTICAL
606+
predicate. Output rows = l (chunk), cols = n (dstate), reduction k = s (chunk).
607+
"""
608+
609+
return GemmContraction(
610+
name="B2_batched_dC_diag_lowertri",
611+
m_extent=chunk_size,
612+
n_extent=dstate,
613+
k_extent=chunk_size,
614+
a_addr_serial=lambda i, k: i * chunk_size + k, # M[l, s]
615+
a_addr_gemm=lambda i, k: i * chunk_size + k,
616+
b_addr_serial=lambda k, j: k * dstate + j, # B[s, n]
617+
b_addr_gemm=lambda k, j: k * dstate + j,
618+
mask_serial=lambda i, j, k: k <= i,
619+
mask_gemm=lambda i, j, k: k <= i,
620+
)
621+
622+
623+
def b2_batched_tiling(*, tile_m: int, tile_n: int, tile_k: int, heads_per_cta: int) -> "GemmTiling":
624+
"""The per-head-band tiling for the batched B2 GEMMs.
625+
626+
M is decomposed into ``heads_per_cta`` DISJOINT bands of ``tile_m`` rows each
627+
(``m_blocks=heads_per_cta, m_stride=tile_m`` so band b owns rows
628+
``[b*tile_m, b*tile_m+tile_m)`` — contiguous, non-overlapping). The
629+
single-writer obligation proves no two head-bands write the same output row
630+
(block-diagonal disjointness, the load-bearing race-freedom property of the
631+
head-accumulation tiling). N/K are single-tile (k_steps=1, n_blocks=1) at the
632+
prod 64 dims.
633+
"""
634+
635+
return GemmTiling(
636+
tile_m=tile_m,
637+
tile_n=tile_n,
638+
tile_k=tile_k,
639+
m_blocks=heads_per_cta,
640+
n_blocks=1,
641+
k_steps=1,
642+
m_stride=tile_m,
643+
n_stride=tile_n,
644+
)
645+
646+
550647
def f0_summary_contraction(z3, *, chunk_size: int, headdim: int, dstate: int):
551648
"""F0 ``summary_states[p,n] = sum_l decay[l]*dt[l]*x[l,p]*B[l,n]``.
552649

0 commit comments

Comments
 (0)