Skip to content

Commit 726bfe5

Browse files
authored
Fix lock signal scope for cross-GPU visibility in matmul_all_reduce (#477)
Use scope="sys" instead of scope="gpu" for the lock-signal atomic in _fused_matmul_all_reduce_kernel. The lock array lives on the symmetric heap mapped via IPC, so atomics consumed by remote ranks must use system scope for xGMI visibility. Fixes intermittent hangs on multi-GPU runs. Fixes #462 Co-Authored-By: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
1 parent 0f27599 commit 726bfe5

1 file changed

Lines changed: 1 addition & 1 deletion

File tree

iris/ops/matmul_all_reduce.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -132,7 +132,7 @@ def _fused_matmul_all_reduce_kernel(
132132
# Use atomic_xchg with release semantics to ensure memory ordering
133133
tile_id = pid_m * num_tiles_n + pid_n
134134
lock_ptr = locks + tile_id
135-
tl.atomic_xchg(lock_ptr, 1, sem="release", scope="gpu") # Release ensures prior stores visible
135+
tl.atomic_xchg(lock_ptr, 1, sem="release", scope="sys") # Release ensures prior stores visible to remote GPUs
136136

137137
# Create source view only when needed (aux_buffer is not None)
138138
src_view = iris.x.make_tensor_view(aux_buffer, M, N, stride_cm, stride_cn)

0 commit comments

Comments
 (0)