Skip to content

Commit 8387052

Browse files
committed
perf(v4): vectorize Engram doc-aware ngram gather (no Python batch loop)
cppmega_v4/nn/engram_v4.py::EngramV4Block._build_ngrams previously did a Python 'for b_i in range(B)' loop to gather token_ids at clamped positions per-batch when document_ids was provided. Replaced with one mx.take_along_axis call on flattened positions. For B=1 (typical decode) the change is a no-op; for larger batches this collapses B kernel dispatches into one. Verified: test_unified_superblock_v4.py — 9/9 (engram doc-ids tests include the boundary-crossing assertion).
1 parent 9854e55 commit 8387052

1 file changed

Lines changed: 7 additions & 5 deletions

File tree

cppmega_v4/nn/engram_v4.py

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -172,11 +172,13 @@ def _build_ngrams(
172172
# itself (the current token — guaranteed in-doc).
173173
current_pos = mx.broadcast_to(mx.arange(S)[None, :, None], (B, S, max_n))
174174
clamped_pos = mx.where(same_doc, positions[None, :, :], current_pos)
175-
# Gather token_ids at clamped positions: [B, S, max_n]
176-
ngrams_per_b = []
177-
for b_i in range(B):
178-
ngrams_per_b.append(mx.take(token_ids[b_i], clamped_pos[b_i], axis=0))
179-
return mx.stack(ngrams_per_b, axis=0)
175+
# Gather token_ids at clamped positions: [B, S, max_n].
176+
# Vectorized over batch via take_along_axis: tok_b expanded
177+
# to [B, S*max_n] for fancy-indexing along axis=1 with the
178+
# flattened clamped_pos.
179+
flat_pos = clamped_pos.reshape(B, S * max_n)
180+
ngrams_flat = mx.take_along_axis(token_ids, flat_pos, axis=1)
181+
return ngrams_flat.reshape(B, S, max_n)
180182
# No doc_ids: simple gather (with the >=0 clamp).
181183
ngrams = token_ids[:, positions]
182184
return ngrams

0 commit comments

Comments
 (0)