Skip to content

fix: avoid redundant device-to-device copy in NCCL weight-transfer receiver#3089

Open
dumko2001 wants to merge 1 commit into
PrimeIntellect-ai:mainfrom
dumko2001:fix/weight-transfer-avoid-redundant-clone
Open

fix: avoid redundant device-to-device copy in NCCL weight-transfer receiver#3089
dumko2001 wants to merge 1 commit into
PrimeIntellect-ai:mainfrom
dumko2001:fix/weight-transfer-avoid-redundant-clone

Conversation

@dumko2001

@dumko2001 dumko2001 commented Jul 20, 2026

Copy link
Copy Markdown

Summary

  • receive_state_dict (src/prime_rl/inference/vllm/worker/nccl.py) does concatenated[...].view(shape).clone() before yielding each tensor, and every consumer then copies that tensor straight into its destination parameter. So each weight is copied twice on-device on every sync.
  • Both consumers copy eagerly: load_weights_kernel calls param.copy_(tensor), and vLLM's load_weights uses default_weight_loader (param.data.copy_(...)). concatenated stays alive for the whole dtype group, so a plain view is valid until the consumer has copied from it and the clone is pure overhead.
  • Fix: yield concatenated[offset : offset + numel].view(shape) without .clone().

Verification

Microbenchmark of the split-and-consume pattern on an RTX 3090 (torch 2.8, CUDA 12.8), a 4 GB bf16 group in 2000 tensors:

clone + copy:  54.8 ms   ( 73 GB/s)
view + copy:   27.6 ms   (145 GB/s)

About 2x on the copy, scaling with checkpoint size (~0.7 s/sync at 100 GB). Peak memory is unchanged — the allocator reuses the transient clone — so this is bandwidth, not memory.


Note

Low Risk
Single-path performance change with unchanged copy semantics; views remain valid for the lifetime of the dtype-group buffer.

Overview
receive_state_dict in nccl.py no longer calls .clone() on each slice of the per-dtype concatenated broadcast buffer. It yields view(shape) slices only, with comments noting that load_weights_kernel and vLLM’s load_weights already copy_ into destination parameters before the next tensor is consumed, while concatenated stays alive for the whole dtype group.

The per-tensor try/finally that deleted each yielded tensor was removed as unnecessary with views.

Reviewed by Cursor Bugbot for commit 28db8ac. Bugbot is set up for automated code reviews on this repo. Configure here.

…ceiver

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@dumko2001
dumko2001 marked this pull request as ready for review July 20, 2026 11:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant