Skip to content

Commit 097693e

Browse files
committed
fix(path-c): time-chunk m2rnn_bwd too (not just mamba3) so direct-chain backward clears the Metal watchdog
The intra-segment time-chunking added in ac412fb switched ONLY the mamba3_mimo_bwd reverse-time-scan backward segment to launcher_chunks dispatch to stay under the macOS GPU watchdog. But m2rnn_bwd is an equally long reverse-time recurrent scan and, in the direct-chain reverse order, its segment runs BEFORE mamba3. Under the default grid_chunks dispatch it ran as one monolithic Metal command buffer and tripped the watchdog FIRST -> kIOGPUCommandBufferCallbackErrorTimeout at the m2rnn_bwd segment, before mamba3 was ever reached. Verified by running the real m04 --use-path-c-direct-chain-runtime route on Metal: the direct-chain training runtime (PathCDirectFusionChainTraining Runtime.value_and_grad) IS driven on the critical path by CompiledPretrainingStep._eager_step (not eager) and the loss cotangent bridge IS wired and ready; the run reached the backward route and timed out at segment region ..._chain_10_11 / op=m2rnn_bwd. After this fix the m2rnn_bwd segment is time-chunked (8 launches, ~3s, watchdog-clear) and the standalone direct-chain executor test runs both recurrent backward scans (m2rnn_bwd seg 3 + mamba3_mimo_bwd seg 5) to completion. Add _TIME_CHUNKED_RECURRENT_BACKWARD_OPS = {m2rnn_bwd, mamba3_mimo_bwd} and key the planner launcher_chunks switch on membership instead of the hard-coded mamba3-only check. Per-row-INDEPENDENT backward ops (attention_qkv_projection_bwd / sparse_mla_fp8_apply_bwd / *_rmsnorm_bwd) stay grid_chunks. Update the standalone executor test to assert both recurrent backward segments are time-chunked. NOTE (separate pre-existing defect, NOT fixed here, now UNMASKED): at FULL local_gb10_quarter size the mamba3_mimo_bwd launcher_chunks kernel crashes the Metal compiler service (XPC_ERROR_CONNECTION_INTERRUPTED during newComputePipelineState) in a cold process too; the m2rnn watchdog timeout previously hid it. The guarded runtime RAISES loudly (RULE #1), so the m04 critical path still does not complete end-to-end at full scale.
1 parent 1cd8289 commit 097693e

2 files changed

Lines changed: 67 additions & 27 deletions

File tree

cppmega_mlx/runtime/path_c_fusion_schedules.py

Lines changed: 46 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -137,6 +137,30 @@
137137
"mamba3_mimo_bwd",
138138
}
139139
)
140+
# Recurrent REVERSE-TIME-SCAN backward ops. Each is a single
141+
# ``for time_rev in T.serial(0, S)`` over the WHOLE sequence inside ONE
142+
# threadgroup. Encoded under the default grid_chunks dispatch as ONE Metal
143+
# command buffer that spans all S time steps, the scan exceeds the macOS GPU
144+
# watchdog window (kIOGPUCommandBufferCallbackErrorTimeout). These ops MUST be
145+
# switched to launcher_chunks so the reverse scan is driven as K command buffers
146+
# over TIME (one ``artifact()`` per row-window, reverse adjoint state carried
147+
# across launches through caller-owned buffers). The other row-phased backward
148+
# ops in _EXACT_ROW_PHASED_BACKWARD_OPS (attention_qkv_projection_bwd /
149+
# sparse_mla_fp8_apply_bwd) are per-row-INDEPENDENT (no carried reverse-time
150+
# state) so they fit one command buffer and stay grid_chunks.
151+
#
152+
# NOTE: the original intra-segment time-chunking (commit ac412fb) only covered
153+
# mamba3_mimo_bwd, but in the direct-chain reverse order the m2rnn_bwd segment
154+
# runs BEFORE mamba3 and is an equally long reverse-time scan -> it tripped the
155+
# watchdog first (verified: timeout at segment region ..._chain_10_11 /
156+
# op=m2rnn_bwd, before mamba3 was ever reached). Both recurrent backward ops
157+
# need the launcher split.
158+
_TIME_CHUNKED_RECURRENT_BACKWARD_OPS = frozenset(
159+
{
160+
"m2rnn_bwd",
161+
"mamba3_mimo_bwd",
162+
}
163+
)
140164
DESCRIPTOR_ROW_PHASED_BWD_SCRATCH_ABI_BUFFERS = frozenset(
141165
{
142166
"attention_hidden",
@@ -14232,28 +14256,37 @@ def plan_path_c_direct_fusion_chain_for_region(
1423214256
candidate_region,
1423314257
DESCRIPTOR_PHYSICAL_ABI_POLICY_DIRECT,
1423414258
)
14235-
# Intra-segment TIME-CHUNKING for the mamba3 mimo BACKWARD segment.
14259+
# Intra-segment TIME-CHUNKING for the recurrent reverse-time-scan
14260+
# BACKWARD segments (m2rnn_bwd AND mamba3_mimo_bwd; see
14261+
# _TIME_CHUNKED_RECURRENT_BACKWARD_OPS).
1423614262
#
14237-
# The mamba3 reverse-time recurrent scan is a single
14263+
# Each recurrent reverse scan is a single
1423814264
# `for time_rev in T.serial(0, S)` over the whole sequence inside ONE
1423914265
# threadgroup. Under the default grid_chunks dispatch that is ONE Metal
14240-
# command buffer spanning all S time steps -> ~7s GPU time -> macOS GPU
14241-
# watchdog kill (kIOGPUCommandBufferCallbackErrorTimeout).
14266+
# command buffer spanning all S time steps -> multi-second GPU time ->
14267+
# macOS GPU watchdog kill (kIOGPUCommandBufferCallbackErrorTimeout).
1424214268
#
14243-
# Switching JUST this segment to launcher_chunks makes its compiled
14244-
# kernel declare path_c_row_chunk_index / path_c_row_subchunk_index and
14245-
# emit the per-row reverse scan body
14269+
# Switching the segment to launcher_chunks makes its compiled kernel
14270+
# declare path_c_row_chunk_index / path_c_row_subchunk_index and emit
14271+
# the per-row reverse scan body
1424614272
# (`for time_rev in T.serial(row, row + 1)`), so the runtime can drive
14247-
# the reverse scan as K separate command buffers over TIME, carrying the
14248-
# reverse adjoint scan state (dh<->h0_grad, angle_grad<->
14249-
# mamba3_angle_grad_state) across launches via caller-owned buffers.
14250-
# Every other segment (forward + non-mamba3-bwd) keeps grid_chunks --
14251-
# only the long reverse scan needs the watchdog-safe launcher split.
14273+
# the reverse scan as K separate command buffers over TIME, carrying
14274+
# the reverse adjoint scan state across launches via caller-owned
14275+
# buffers. Every other segment (forward + per-row-INDEPENDENT backward
14276+
# ops like attention_qkv_projection_bwd / sparse_mla_fp8_apply_bwd /
14277+
# residual_rmsnorm_bwd) keeps grid_chunks -- only the long reverse
14278+
# scans need the watchdog-safe launcher split.
14279+
#
14280+
# The original time-chunking (commit ac412fb) covered ONLY
14281+
# mamba3_mimo_bwd; m2rnn_bwd runs FIRST in the reverse chain and tripped
14282+
# the watchdog before mamba3 was reached (verified timeout at region
14283+
# ..._chain_10_11 / op=m2rnn_bwd), so both recurrent ops are included.
1425214284
if (
1425314285
execution_phase == DESCRIPTOR_EXECUTION_STAGE_BACKWARD
1425414286
and direct_target.max_rows_per_launch is not None
1425514287
and any(
14256-
_path_c_descriptor_stage_node_op_name(node) == "mamba3_mimo_bwd"
14288+
_path_c_descriptor_stage_node_op_name(node)
14289+
in _TIME_CHUNKED_RECURRENT_BACKWARD_OPS
1425714290
for node in candidate_region.nodes
1425814291
)
1425914292
):

tests/test_m04_train_step.py

Lines changed: 21 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -3624,30 +3624,37 @@ def test_path_c_direct_chain_runtime_executor_runs_native_segments(
36243624
"ok",
36253625
"ok",
36263626
]
3627-
# The mamba3 mimo BACKWARD segment is TIME-CHUNKED (launcher_chunks): its
3628-
# kernel additionally declares the path_c_row_chunk_index /
3627+
# The recurrent reverse-time-scan BACKWARD segments are TIME-CHUNKED
3628+
# (launcher_chunks): both the m2rnn_bwd (segment 3) and mamba3_mimo_bwd
3629+
# (segment 5) reverse scans declare the path_c_row_chunk_index /
36293630
# path_c_row_subchunk_index / path_c_backward_stage_index scalar params so the
3630-
# runtime can split the reverse-time scan into watchdog-safe per-launch
3631-
# command buffers. That segment therefore carries 3 extra scalar args beyond
3632-
# the backward gate; every other segment keeps grid_chunks dispatch.
3631+
# runtime can split each reverse-time scan into watchdog-safe per-launch
3632+
# command buffers. Those segments therefore carry 3 extra scalar args beyond
3633+
# the backward gate; every other segment (forward + per-row-INDEPENDENT
3634+
# backward ops) keeps grid_chunks dispatch.
3635+
#
3636+
# m2rnn_bwd was added to the time-chunked set because in the reverse chain it
3637+
# runs BEFORE mamba3 and is an equally long reverse-time scan; under
3638+
# grid_chunks it tripped the macOS GPU watchdog before mamba3 was reached.
36333639
time_chunked_segment_indices = {
36343640
segment["index"]
36353641
for segment in payload["segments"]
36363642
if segment.get("time_chunk_launch_count")
36373643
}
3638-
assert time_chunked_segment_indices == {5}, time_chunked_segment_indices
3639-
mamba3_bwd_launch_count = next(
3640-
segment["time_chunk_launch_count"]
3641-
for segment in payload["segments"]
3642-
if segment["index"] == 5
3643-
)
3644-
assert mamba3_bwd_launch_count > 1
3644+
assert time_chunked_segment_indices == {3, 5}, time_chunked_segment_indices
3645+
for time_chunked_index in sorted(time_chunked_segment_indices):
3646+
recurrent_bwd_launch_count = next(
3647+
segment["time_chunk_launch_count"]
3648+
for segment in payload["segments"]
3649+
if segment["index"] == time_chunked_index
3650+
)
3651+
assert recurrent_bwd_launch_count > 1
36453652
assert [segment["kernel_arg_count"] for segment in payload["segments"]] == [
36463653
len(rb_segment["required_logical_buffers"])
36473654
# backward segments additionally pass the path_c_run_backward gate scalar
36483655
+ (1 if payload_segment["execution_phase"] == "backward" else 0)
3649-
# the time-chunked mamba3 backward also binds the launcher-chunk index
3650-
# scalars (path_c_row_chunk_index + path_c_row_subchunk_index +
3656+
# the time-chunked recurrent backward segments also bind the launcher-chunk
3657+
# index scalars (path_c_row_chunk_index + path_c_row_subchunk_index +
36513658
# path_c_backward_stage_index) so the runtime can select each launch.
36523659
+ (3 if payload_segment["index"] in time_chunked_segment_indices else 0)
36533660
for payload_segment, rb_segment in zip(

0 commit comments

Comments
 (0)