Skip to content

Commit 8407b19

Browse files
author
codex
committed
docs: update path_c blockers contract with B2 partial advance + Block A
Documents commit 26575cd's Fix B-1 (block bwd no longer clobbers residual chain), and adds a new "Block A" section that explains the structural divergence between eager and fused for the first brick of an in-region chain: the eager pre-block norm (layers.{first_in_region}.norm) is never applied in the fused region because layers.{N+1}.norm.weight is aliased to brick N's bridge weight (=> brick 0's norm has no home in the fused region). Includes: - Probe receipts showing delta_M sumabs differs by ~16x between eager (= mamba3(norm(hidden_entry))) and fused (= mamba3(hidden_entry)). - Two fix options at the TileLang surface layer: (1) inline entry-norm in the M-block fwd/bwd, no signature change; (2) new entry_rmsnorm op with descriptor + op_signature bump. The B6 bench gate stays blocked on Block A + remaining Block 2 numeric parity.
1 parent 26575cd commit 8407b19

1 file changed

Lines changed: 106 additions & 0 deletions

File tree

docs/upstream/_path_c_full_graph_runtime_blockers.md

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -198,3 +198,109 @@ removes the unsafe overlay and replaces it with the fail-closed
198198
warmup+eager path. The work above is what is left to actually beat
199199
path_b, and it lives at the TileLang descriptor + Metal codegen
200200
layer, exactly where the user asked the work to happen.
201+
202+
## Update 2026-05-23: B2 partial advance + structural Block A discovered
203+
204+
### Fix B-1 (committed, `26575cd`): block bwd no longer clobbers residual chain
205+
206+
The row-phased block backward emitters (`_append_row_phased_mamba3_bwd_body`,
207+
`_append_row_phased_m2rnn_bwd_body`,
208+
`_append_row_phased_attention_qkv_projection_bwd_body`) used to
209+
zero-init the input's `hidden_grad` slot at the start of every row
210+
iteration, then accumulate the block's in-projection contribution
211+
into the same slot with `+=`. When the input's `hidden_grad`
212+
output mapped to a **full-sequence bank slot** that the upstream
213+
`residual_rmsnorm_bwd` had already written with `=` in the same
214+
row iteration (specifically for the FIRST brick of an in-region
215+
chain), the zero-init clobbered the residual chain-rule contribution,
216+
so the only term that reached the input residual stream was the
217+
block's in-projection grad. The downstream eager prefix then saw a
218+
hidden_entry cotangent that was missing the chain through the
219+
fused region's residual bridges.
220+
221+
New helper `_is_full_sequence_bank_slot(buffer_name, access_by_buffer)`
222+
detects this case via the access string pattern
223+
`path_c_..._abi_bank[OFFSET + i]` (no `% H` modulo). The three
224+
block bwd emitters now skip the zero-init when this helper returns
225+
True, so the existing `hidden_grad_ref += block_contribution`
226+
accumulator adds onto whatever the residual_rmsnorm_bwd wrote with
227+
`=`. Per-row scratch `*_hidden_grad` slots (used by the LATER
228+
bricks in the same chain, e.g. R and A consuming bridge-normalized
229+
hidden) keep their per-row zero-init because no earlier bwd writes
230+
to them in the same row iteration.
231+
232+
This change is mathematically correct and bit-stable (no extra
233+
shared-memory sync, no FP order changes for `attention_out`); all
234+
tested suites stay green (94 + 111 + 93 + 66 + 401).
235+
236+
### Block A (NEW): first brick of the fused region is missing its pre-block norm
237+
238+
`_path_c_model_surfaces_from_bricks` initialises
239+
`context.route_hidden = initial_hidden` for the first brick. Brick
240+
N+1 in the chain reads `context.route_hidden = norm_{N+1}(hidden +
241+
delta_N)` from the inter-brick `residual_rmsnorm` bridge (which uses
242+
`layers.{N+1}.norm.weight` per the alias map). For brick **0** of
243+
the region, the M-block consumes raw `hidden` (the entry residual
244+
stream from the eager prefix) directly, **without** applying its
245+
own `layers.{first_in_region}.norm`.
246+
247+
The eager `HybridTinyBlock.route_delta` always does
248+
`x = self.norm(hidden); delta = block(x); return delta`. The fused
249+
region therefore feeds `mamba3(hidden)` instead of
250+
`mamba3(norm_first(hidden))` into its first brick, so:
251+
252+
* `local_gb10_quarter_brick_10_M_delta` is computed against a
253+
~16x-larger-magnitude input (`||hidden||` vs
254+
`||norm(hidden)||`).
255+
* `layers.10.block.D` and `layers.10.block.in_proj.weight`
256+
gradients pick up the inflated input as a multiplicative factor
257+
in the bwd chain rule, yielding relative-error
258+
~10^3 — 10^4 against eager (see
259+
`/tmp/path_c_blocker2_probe.py` receipts).
260+
* `layers.10.norm.weight` is never bound to any logical name in the
261+
fused region's ABI map; the alias loop in
262+
`path_c_parameter_logical_aliases` binds
263+
`layers.{i+1}.norm.weight` to brick `i`'s bridge weight for
264+
`i in [0, N-2]`, so `layers.{first_in_region}.norm.weight` lands
265+
on the bridge BEFORE the first brick — which is in the eager
266+
prefix, not in the fused region.
267+
268+
Confirming probe (`/tmp/probe_entry_norm.py`): with deterministic
269+
weights and a synthetic `hidden_entry`, the eager M-block produces
270+
`delta_M` with `sumabs=8.18`, while the fused-equivalent
271+
`mamba3(hidden_entry)` (no entry norm) produces `delta_M` with
272+
`sumabs=0.021`. The ratio matches `1 / inv_rms ~= 19` (and propagates
273+
multiplicatively through subsequent layers).
274+
275+
#### Fix layer (Block A)
276+
277+
The right place is the TileLang surface layer (lower than the
278+
cppmega_mlx app code, per the constraints in this file). Two
279+
ladder approaches:
280+
281+
1. **(Minimal-invasive) Add an inline entry-norm path inside the
282+
first brick's fwd codegen and a matching bwd accumulator into the
283+
bank `hidden_grad` slot.** Add a new real-ABI input
284+
`f"{first_brick.name}_entry_rmsnorm_weight"` and have
285+
`_emit_mamba3_model_brick_surfaces` consume it. The bwd accumulates
286+
`inv_rms * (entry_normed_hidden_grad * weight - hidden * dot *
287+
inv_rms^2 / D)` into `hidden_grad` (the bank slot we now fixed in
288+
Fix B-1 to accumulate). Pros: no new op signature, no
289+
`_MAMBA3_FP8_TRAIN_FWD_BWD_OP_SIGNATURE` churn, no acceptance
290+
profile invalidation. Cons: ad-hoc M-block fwd/bwd changes.
291+
292+
2. **(Structurally cleanest) Add a new `entry_rmsnorm` op-node** that
293+
the surface builder prepends to the brick chain, with its own
294+
fwd/bwd codegen and descriptor, and update the canonical op
295+
signature `_MAMBA3_FP8_TRAIN_FWD_BWD_OP_SIGNATURE` to
296+
`("entry_rmsnorm", "mamba3_mimo", "residual_rmsnorm", ...,
297+
"entry_rmsnorm_bwd")`. Wire `layers.{first_in_region}.norm.weight`
298+
into the alias loop as an additional candidate alias mapping to
299+
`f"{first_brick.path_c_brick_name}_entry_rmsnorm_weight"`. Pros:
300+
self-contained op, reuses the bridge codegen patterns. Cons:
301+
requires a new descriptor, an acceptance-profile bump, and the
302+
aliases mapping update.
303+
304+
Either fix is the next-step after `26575cd`. The B6 bench gate stays
305+
blocked until Block A lands plus the rest of Block 2 numeric parity
306+
passes on `local_gb10_quarter` tiny smoke.

0 commit comments

Comments
 (0)