Skip to content

Commit 2cb062d

Browse files
mikasenghaasclaude
andcommitted
restore NIXL ModelExpress handshake signals in the version hooks
The merge of #3060 kept this branch's on_version_pending/on_new_version bodies and dropped main's ModelExpress status updates — the READY signal the NIXL trainer waits on before each broadcast and the INITIALIZING reset after it. Post-startup NIXL runs would hang in the handshake. Both signals are back, ordered as on main (READY pre-update, INITIALIZING post-update) around this branch's staleness/wake logic. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent a752e27 commit 2cb062d

1 file changed

Lines changed: 9 additions & 3 deletions

File tree

src/prime_rl/orchestrator/orchestrator.py

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -975,13 +975,16 @@ def train_needed(self) -> bool:
975975

976976
async def on_version_pending(self, step: int) -> None:
977977
"""``VersionObserver`` hook, fired *before* the engines pause for the
978-
weight update: record the newly published version (waking a held ship
979-
— filesystem holds release on publish), then drop everything already
978+
weight update: signal the NIXL trainer this side is ready for the
979+
transfer, record the newly published version (waking a held ship —
980+
filesystem holds release on publish), then drop everything already
980981
past the staleness cutoff — in-flight groups (their aborts must be
981982
processed while the engine is still stepping) and buffered survivors
982983
in the sink. The cutoff is the oldest generation version that can
983984
still train within ``max_off_policy_steps``, were it to ship in the
984985
batch being collected (which trains on policy v{progress.step - 1})."""
986+
if self.model_express is not None:
987+
await asyncio.to_thread(self.model_express.set_status, p2p_pb2.SOURCE_STATUS_READY)
985988
self.published_version = max(self.published_version, step)
986989
self.version_advanced.set()
987990
cutoff = (self.progress.step - 1) - self.config.max_off_policy_steps
@@ -990,7 +993,10 @@ async def on_version_pending(self, step: int) -> None:
990993

991994
async def on_new_version(self, step: int) -> None:
992995
"""``VersionObserver`` hook: the watcher just advanced ``policy.version``;
993-
wake a held ship in ``finalize_train_batch``."""
996+
reset the NIXL rendezvous for the next broadcast cycle and wake a held
997+
ship in ``finalize_train_batch``."""
998+
if self.model_express is not None:
999+
await asyncio.to_thread(self.model_express.set_status, p2p_pb2.SOURCE_STATUS_INITIALIZING)
9941000
self.version_advanced.set()
9951001

9961002
async def stop(self) -> None:

0 commit comments

Comments
 (0)