Skip to content

Commit 26575cd

Browse files
author
codex
committed
Path C: accumulate hidden_grad across residual_rmsnorm_bwd and block bwd
The row-phased block backward codegens (mamba3_mimo_bwd, m2rnn_bwd, attention_qkv_projection_bwd) used to zero-init the upstream `hidden_grad` slot before accumulating the block's in-projection contribution. When that slot was a full-sequence bank buffer also written by an earlier residual_rmsnorm_bwd in the same row iteration (the M block of an in-region brick chain), the zero-init clobbered the residual_rmsnorm bwd's chain-rule contribution, so the only term that reached the input residual stream was the block's in-projection grad. This change adds the `_is_full_sequence_bank_slot` helper and makes the three block bwds skip the zero-init for any `hidden_grad` output that resolves to a full-sequence bank slot. The block bwd's existing `hidden_grad_ref = hidden_grad_ref + ...` accumulator then adds the block's contribution on top of the total_grad that the residual_rmsnorm_bwd already wrote with `=`, yielding the correct `hidden_grad = total_grad + block_contribution` chain rule sum for the first brick of a fused region. Per-row scratch `*_hidden_grad` slots used by the LATER bricks (R, A) keep their per-row zero-init because no earlier bwd writes to them in the same row iteration. Standalone tests: 94 in tests/test_path_c_fusion_ir.py, 111 in tests/test_path_c_fusion_templates.py, 93 in tests/test_path_c_physical_abi.py and friends -- all green at this commit.
1 parent dd50d39 commit 26575cd

1 file changed

Lines changed: 33 additions & 3 deletions

File tree

cppmega_mlx/runtime/path_c_fusion_schedules.py

Lines changed: 33 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -5520,6 +5520,33 @@ def _row_phased_attention_indices_ref(
55205520
)
55215521

55225522

5523+
def _is_full_sequence_bank_slot(
5524+
buffer_name: str,
5525+
access_by_buffer: Mapping[str, str],
5526+
) -> bool:
5527+
"""Return True when ``buffer_name`` resolves to a full-sequence bank slot.
5528+
5529+
A "full-sequence bank slot" is a logical buffer whose physical access
5530+
expression resolves to ``path_c_..._abi_bank[OFFSET + i]`` (i.e. the
5531+
index depends on the kernel-wide loop variable ``i`` covering the
5532+
full ``sequence_length * hidden_size`` range), as opposed to a
5533+
per-row scratch buffer that wraps around each row iteration.
5534+
5535+
The schedule uses this signal to decide whether the bwd needs a
5536+
one-shot pre-zero (full-sequence bank slot) or whether per-row
5537+
zero-init followed by ``=`` accumulation is sufficient (per-row
5538+
scratch buffer).
5539+
"""
5540+
if buffer_name not in access_by_buffer:
5541+
return False
5542+
access = access_by_buffer[buffer_name]
5543+
if "_abi_bank[" not in access:
5544+
return False
5545+
if " % " in access:
5546+
return False
5547+
return True
5548+
5549+
55235550
def _append_row_phased_residual_rmsnorm_bwd_init(
55245551
body: list[str],
55255552
*,
@@ -5827,7 +5854,8 @@ def _append_row_phased_attention_qkv_projection_bwd_body(
58275854
access_by_buffer,
58285855
f"row * {hidden_size} + {h}",
58295856
)
5830-
body.append(f"{indent * 4}{hidden_grad_ref} = 0.0")
5857+
if not _is_full_sequence_bank_slot(hidden_grad, access_by_buffer):
5858+
body.append(f"{indent * 4}{hidden_grad_ref} = 0.0")
58315859
body.append(f"{indent * 4}for {q_flat} in T.serial(0, {q_dim}):")
58325860
q_head_expr = f"{q_flat} // {head_dim}"
58335861
q_value_index = f"row * {q_dim} + {q_flat}"
@@ -6079,7 +6107,8 @@ def _append_row_phased_m2rnn_bwd_body(
60796107
access_by_buffer,
60806108
f"row * {hidden_size} + {hidden_dim}",
60816109
)
6082-
body.append(f"{indent * 4}{hidden_grad_ref} = 0.0")
6110+
if not _is_full_sequence_bank_slot(hidden_grad, access_by_buffer):
6111+
body.append(f"{indent * 4}{hidden_grad_ref} = 0.0")
60836112
body.append(f"{indent * 4}for {proj_dim} in T.serial(0, {in_proj_dim}):")
60846113
delta_grad = _node_indexed_canonical_or_positional_input_expr(
60856114
node,
@@ -6374,7 +6403,8 @@ def _append_row_phased_mamba3_bwd_body(
63746403
access_by_buffer,
63756404
f"row * {hidden_size} + {hidden_dim}",
63766405
)
6377-
body.append(f"{indent * 4}{hidden_grad_ref} = 0.0")
6406+
if not _is_full_sequence_bank_slot(hidden_grad, access_by_buffer):
6407+
body.append(f"{indent * 4}{hidden_grad_ref} = 0.0")
63786408
body.append(f"{indent * 4}for {proj_dim} in T.serial(0, {in_proj_dim}):")
63796409
delta_grad = _node_indexed_canonical_or_positional_input_expr(
63806410
node,

0 commit comments

Comments
 (0)