Commit 097693e
committed
fix(path-c): time-chunk m2rnn_bwd too (not just mamba3) so direct-chain backward clears the Metal watchdog
The intra-segment time-chunking added in ac412fb switched ONLY the
mamba3_mimo_bwd reverse-time-scan backward segment to launcher_chunks
dispatch to stay under the macOS GPU watchdog. But m2rnn_bwd is an
equally long reverse-time recurrent scan and, in the direct-chain reverse
order, its segment runs BEFORE mamba3. Under the default grid_chunks
dispatch it ran as one monolithic Metal command buffer and tripped the
watchdog FIRST -> kIOGPUCommandBufferCallbackErrorTimeout at the
m2rnn_bwd segment, before mamba3 was ever reached.
Verified by running the real m04 --use-path-c-direct-chain-runtime route
on Metal: the direct-chain training runtime (PathCDirectFusionChainTraining
Runtime.value_and_grad) IS driven on the critical path by
CompiledPretrainingStep._eager_step (not eager) and the loss cotangent
bridge IS wired and ready; the run reached the backward route and timed
out at segment region ..._chain_10_11 / op=m2rnn_bwd. After this fix the
m2rnn_bwd segment is time-chunked (8 launches, ~3s, watchdog-clear) and the
standalone direct-chain executor test runs both recurrent backward scans
(m2rnn_bwd seg 3 + mamba3_mimo_bwd seg 5) to completion.
Add _TIME_CHUNKED_RECURRENT_BACKWARD_OPS = {m2rnn_bwd, mamba3_mimo_bwd}
and key the planner launcher_chunks switch on membership instead of the
hard-coded mamba3-only check. Per-row-INDEPENDENT backward ops
(attention_qkv_projection_bwd / sparse_mla_fp8_apply_bwd / *_rmsnorm_bwd)
stay grid_chunks. Update the standalone executor test to assert both
recurrent backward segments are time-chunked.
NOTE (separate pre-existing defect, NOT fixed here, now UNMASKED): at FULL
local_gb10_quarter size the mamba3_mimo_bwd launcher_chunks kernel crashes
the Metal compiler service (XPC_ERROR_CONNECTION_INTERRUPTED during
newComputePipelineState) in a cold process too; the m2rnn watchdog timeout
previously hid it. The guarded runtime RAISES loudly (RULE #1), so the m04
critical path still does not complete end-to-end at full scale.1 parent 1cd8289 commit 097693e
2 files changed
Lines changed: 67 additions & 27 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
137 | 137 | | |
138 | 138 | | |
139 | 139 | | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
140 | 164 | | |
141 | 165 | | |
142 | 166 | | |
| |||
14232 | 14256 | | |
14233 | 14257 | | |
14234 | 14258 | | |
14235 | | - | |
| 14259 | + | |
| 14260 | + | |
| 14261 | + | |
14236 | 14262 | | |
14237 | | - | |
| 14263 | + | |
14238 | 14264 | | |
14239 | 14265 | | |
14240 | | - | |
14241 | | - | |
| 14266 | + | |
| 14267 | + | |
14242 | 14268 | | |
14243 | | - | |
14244 | | - | |
14245 | | - | |
| 14269 | + | |
| 14270 | + | |
| 14271 | + | |
14246 | 14272 | | |
14247 | | - | |
14248 | | - | |
14249 | | - | |
14250 | | - | |
14251 | | - | |
| 14273 | + | |
| 14274 | + | |
| 14275 | + | |
| 14276 | + | |
| 14277 | + | |
| 14278 | + | |
| 14279 | + | |
| 14280 | + | |
| 14281 | + | |
| 14282 | + | |
| 14283 | + | |
14252 | 14284 | | |
14253 | 14285 | | |
14254 | 14286 | | |
14255 | 14287 | | |
14256 | | - | |
| 14288 | + | |
| 14289 | + | |
14257 | 14290 | | |
14258 | 14291 | | |
14259 | 14292 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3624 | 3624 | | |
3625 | 3625 | | |
3626 | 3626 | | |
3627 | | - | |
3628 | | - | |
| 3627 | + | |
| 3628 | + | |
| 3629 | + | |
3629 | 3630 | | |
3630 | | - | |
3631 | | - | |
3632 | | - | |
| 3631 | + | |
| 3632 | + | |
| 3633 | + | |
| 3634 | + | |
| 3635 | + | |
| 3636 | + | |
| 3637 | + | |
| 3638 | + | |
3633 | 3639 | | |
3634 | 3640 | | |
3635 | 3641 | | |
3636 | 3642 | | |
3637 | 3643 | | |
3638 | | - | |
3639 | | - | |
3640 | | - | |
3641 | | - | |
3642 | | - | |
3643 | | - | |
3644 | | - | |
| 3644 | + | |
| 3645 | + | |
| 3646 | + | |
| 3647 | + | |
| 3648 | + | |
| 3649 | + | |
| 3650 | + | |
| 3651 | + | |
3645 | 3652 | | |
3646 | 3653 | | |
3647 | 3654 | | |
3648 | 3655 | | |
3649 | | - | |
3650 | | - | |
| 3656 | + | |
| 3657 | + | |
3651 | 3658 | | |
3652 | 3659 | | |
3653 | 3660 | | |
| |||
0 commit comments