Skip to content

Commit 359d3e9

Browse files
committed
Eliminate duplicated softmax recomputation in indexer loss.
By inserting `jax.lax.optimization_barrier` after the head aggregation step in `calculate_indexer_loss`, we force the compiler to reuse the head-aggregated intermediate tensor for the subsequent sequence reduction instead of recomputing the entire softmax pipeline from raw QK scores. TAG=agy CONV=5ff94b54-4171-4309-8704-6046df05eb13
1 parent 48c7da3 commit 359d3e9

1 file changed

Lines changed: 2 additions & 0 deletions

File tree

src/maxtext/layers/attention_mla.py

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1077,6 +1077,8 @@ def calculate_indexer_loss(
10771077

10781078
# Aggregate heads: [b, h, t, s] -> [b, t, s]
10791079
attention_probs = jnp.sum(attention_probs, axis=1)
1080+
# Force materialization and prevent fusion across this point to reuse the intermediate tensor
1081+
attention_probs = jax.lax.optimization_barrier(attention_probs)
10801082
# L1 normalize aggregated target distribution
10811083
attention_probs = attention_probs / (jnp.sum(attention_probs, axis=-1, keepdims=True) + EPS)
10821084

0 commit comments

Comments
 (0)