Commit 26575cd
codex
Path C: accumulate hidden_grad across residual_rmsnorm_bwd and block bwd
The row-phased block backward codegens (mamba3_mimo_bwd, m2rnn_bwd,
attention_qkv_projection_bwd) used to zero-init the upstream
`hidden_grad` slot before accumulating the block's in-projection
contribution. When that slot was a full-sequence bank buffer also
written by an earlier residual_rmsnorm_bwd in the same row
iteration (the M block of an in-region brick chain), the
zero-init clobbered the residual_rmsnorm bwd's chain-rule
contribution, so the only term that reached the input residual
stream was the block's in-projection grad.
This change adds the `_is_full_sequence_bank_slot` helper and
makes the three block bwds skip the zero-init for any
`hidden_grad` output that resolves to a full-sequence bank slot.
The block bwd's existing `hidden_grad_ref = hidden_grad_ref +
...` accumulator then adds the block's contribution on top of the
total_grad that the residual_rmsnorm_bwd already wrote with `=`,
yielding the correct
`hidden_grad = total_grad + block_contribution` chain rule sum
for the first brick of a fused region.
Per-row scratch `*_hidden_grad` slots used by the LATER bricks
(R, A) keep their per-row zero-init because no earlier bwd writes
to them in the same row iteration.
Standalone tests: 94 in tests/test_path_c_fusion_ir.py,
111 in tests/test_path_c_fusion_templates.py, 93 in
tests/test_path_c_physical_abi.py and friends -- all green at
this commit.1 parent dd50d39 commit 26575cd
1 file changed
Lines changed: 33 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
5520 | 5520 | | |
5521 | 5521 | | |
5522 | 5522 | | |
| 5523 | + | |
| 5524 | + | |
| 5525 | + | |
| 5526 | + | |
| 5527 | + | |
| 5528 | + | |
| 5529 | + | |
| 5530 | + | |
| 5531 | + | |
| 5532 | + | |
| 5533 | + | |
| 5534 | + | |
| 5535 | + | |
| 5536 | + | |
| 5537 | + | |
| 5538 | + | |
| 5539 | + | |
| 5540 | + | |
| 5541 | + | |
| 5542 | + | |
| 5543 | + | |
| 5544 | + | |
| 5545 | + | |
| 5546 | + | |
| 5547 | + | |
| 5548 | + | |
| 5549 | + | |
5523 | 5550 | | |
5524 | 5551 | | |
5525 | 5552 | | |
| |||
5827 | 5854 | | |
5828 | 5855 | | |
5829 | 5856 | | |
5830 | | - | |
| 5857 | + | |
| 5858 | + | |
5831 | 5859 | | |
5832 | 5860 | | |
5833 | 5861 | | |
| |||
6079 | 6107 | | |
6080 | 6108 | | |
6081 | 6109 | | |
6082 | | - | |
| 6110 | + | |
| 6111 | + | |
6083 | 6112 | | |
6084 | 6113 | | |
6085 | 6114 | | |
| |||
6374 | 6403 | | |
6375 | 6404 | | |
6376 | 6405 | | |
6377 | | - | |
| 6406 | + | |
| 6407 | + | |
6378 | 6408 | | |
6379 | 6409 | | |
6380 | 6410 | | |
| |||
0 commit comments