Skip to content

fix: clone tensors in recv_state_dict to avoid aliasing#261

Open
Abhishek21g wants to merge 1 commit into
PrimeIntellect-ai:mainfrom
Abhishek21g:abhishek/fix-recv-state-dict-aliasing
Open

fix: clone tensors in recv_state_dict to avoid aliasing#261
Abhishek21g wants to merge 1 commit into
PrimeIntellect-ai:mainfrom
Abhishek21g:abhishek/fix-recv-state-dict-aliasing

Conversation

@Abhishek21g

Copy link
Copy Markdown

Fixes #245

Problem

recv_state_dict receives tensor payloads into buffers derived from og_state_dict, then returns a state dict whose tensor values alias the caller's template objects (id match).

This affects live checkpoint recovery in checkpoint.py (recv_ckpt_from_peer), where optimizer/training state is received via recv_state_dict(..., self.optimizer.state_dict()).

Repro

Identity check from the issue thread — after receive, returned tensors share objects with og_state_dict:

received = recv_state_dict(pg, src_rank, og_state_dict)
assert id(received["optim_sates"]) != id(og_state_dict["optim_sates"])  # fails on main

Fix

Clone received CPU buffers into a fresh tensor list before _load_sendable_state_dict, so the returned dict owns independent tensors.

Also:

  • add test_recv_state_dict_no_tensor_aliasing (no distributed required)
  • fix send_state_dict / recv_state_dict call signatures in the skipped dist test
  • fix dist test to assert received values match the sender payload

Verification

uv run pytest tests/test_dist/test_send_state_dict.py::test_recv_state_dict_no_tensor_aliasing -v
uv run pytest tests/test_dist/test_send_state_dict.py::test_load_state_dict -v

Dist test (test_send_recv_state_dict) remains skipped (hang); happy to help unskip on a 2-process Linux host separately.

recv_state_dict copied received payloads into og_state_dict buffers
and returned state dict entries pointing at the same tensor objects.
Clone received CPU buffers before _load_sendable_state_dict so live
checkpoint recovery returns independent tensors.

Fixes PrimeIntellect-ai#245
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

send/recv state dict reuse the same tensor

1 participant