feat: Wire WeightSynchronizer into algorithm layer, replacing inline refit logic

[saumishr]

What does this PR do ?

Wire the WeightSynchronizer (from MR 1 / PR #2466) into the algorithm layer, replacing inline refit_policy_generation() calls and NEED_REFIT/POLICY_GENERATION_STALE flags with WeightSynchronizer method calls (sync_weights, mark_stale, is_stale).

Issues

Part of the modularity/interfaces initiative. Depends on PR #2466 (modularity/weight-sync-abc).

Usage

Before (old pattern):

# In setup()
state_dict_info = policy.prepare_refit_info()
policy_generation.prepare_refit_info(state_dict_info)
if not colocated_inference:
    futures_train = policy.init_collective(ip, port, world_size, ...)
    futures_inference = policy_generation.init_collective(ip, port, world_size, ...)
    ray.get(futures_train + futures_inference)

# In grpo_train()
NEED_REFIT = True
POLICY_GENERATION_STALE = True
...
if NEED_REFIT and POLICY_GENERATION_STALE:
    refit_policy_generation(policy, policy_generation, colocated_inference, timer=timer, kv_scales=...)
    POLICY_GENERATION_STALE = False
...
POLICY_GENERATION_STALE = True  # after training

After (new pattern):

# In setup()
weight_sync = create_weight_synchronizer(
    policy=policy, generation=policy_generation,
    generation_backend=backend, colocated=colocated_inference,
    train_cluster=train_cluster, inference_cluster=inference_cluster,
)
weight_sync.init_communicator()
# In grpo_train()
if weight_sync is not None and weight_sync.is_stale:
    weight_sync.sync_weights(timer=timer, kv_scales=...)
...
weight_sync.mark_stale()  # after training

Tests

Updated test_grpo.py and test_distillation.py to use weight_sync mocks

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
• Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

Stacked on #2466

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[saumishr]

/ok to test 657abc9

[saumishr]

/ok to test 616f2b0

[saumishr]

/ok to test 40f6d46

[terrykong]

@kajalj22 the L1 tests are timing out at 6hr. is something off? i did notice a lot of downloads where i expected the hf cache to be hit https://github.com/NVIDIA-NeMo/RL/actions/runs/26606449648/job/78418561361?pr=2467

[terrykong]

@saumishr given the nature of the change, could you run nightlies from these categories to make sure there's not some convergence issue due to the refit change #2467 (comment)

[saumishr]

/ok to test f303df0

[saumishr]

/ok to test 3555948

[saumishr]

/ok to test 196fd7c