fix(orchestrator): pin eval epochs to one policy version

[vineetjain96]

Summary

• pin each complete eval step to one live policy version
• drain pre-existing train work, dispatch the full eval cohort, then refill train capacity during the pinned eval tail
• retarget delayed watcher updates to the latest stable checkpoint after the eval lease is available
• fail fast when an eval configuration cannot complete and release its lease

Problem

Eval groups record the policy version and cache salt present when they are scheduled, but the live inference pool is not pinned to that version. The watcher can pause, update, and resume inference between requests or turns of the same eval step. A trajectory can therefore execute against multiple policies while being labeled as one version, and an eval cohort can contain mixed weights or hit provider failures while its model or adapter is replaced.

Holding the existing watcher update lock across eval exposes a related race. The watcher can select checkpoint N and then wait behind the eval lease while the trainer publishes newer checkpoints and deletes N. Loading the path selected before that wait then fails because it no longer exists.

Fix

Queue each due eval step and acquire the watcher lock in a supervised side task before adding its work to the dispatcher. Keeping lease acquisition off the bounded rollout consumer lets completed train and eval rollouts continue draining while a watcher already ahead of the trigger finishes its update. The lease remains held until every eval environment due at that step has returned; startup eval uses the same path, and failure and shutdown paths release it explicitly.

The dispatcher finishes any train group already being scheduled and drains the pre-existing train tail before starting eval. Once the complete eval cohort has been dispatched, it returns spare permits to train while the eval tail finishes under the pinned policy. Empty eval environments and group-scoring cohorts larger than max_inflight_rollouts are rejected up front because they could never complete the epoch and release the lease.

The watcher waits for checkpoint publication without holding the live-policy lock. If the originally requested checkpoint has already disappeared, it first retargets to a newer stable checkpoint. After acquiring the lock, it resolves the latest stable checkpoint again and uses that version consistently for the weight path, pending and completed observer callbacks, inference, and policy state.

Interval eligibility is unchanged. If another eval step becomes due while an epoch is active, it is queued and starts after the current epoch releases the lease. Eval sampling, scoring, and trainer data are unchanged.

Verification

• pytest -q tests/unit/orchestrator/test_orchestrator_setup.py — 10 passed
• pytest -q tests/unit/orchestrator -m 'not gpu' — 102 passed, 1 skipped
• ruff check --config=pyproject.toml .
• ruff format --check --config=pyproject.toml .
• git diff --check

---

Note

High Risk
Changes orchestrator scheduling, live inference weight updates, and eval correctness guarantees on a critical training path; mistakes could deadlock the pipeline or mis-label eval policy versions.

Overview
Pins each eval step to a single live policy version by holding the watcher’s update_lock from a background eval_trigger task until every env due at that step finishes. Due steps are queued (non-blocking on the rollout consumer); startup eval runs and completes its lease before the dispatcher and watcher start.

Dispatcher PREFER_EVAL now finishes open train groups, drains in-flight train work, dispatches the full eval cohort, then flips back to PREFER_TRAIN so spare capacity can run train while the pinned eval tail completes. Train scheduling stays off when train_scheduling_disabled.

WeightWatcher waits for checkpoint STABLE without the update lock, retargets if the requested step was deleted, and coalesces to the latest stable checkpoint after acquiring the lock.

Fail-fast on eval envs with no examples or group-scoring group_size above max_inflight_rollouts. EvalSource.eligible_envs peeks due envs without consuming startup trigger state. Docs and monitor-run skill describe the pinned-eval log line and overlapping train/eval inflight.

^{Reviewed by Cursor Bugbot for commit cbdc11c. Bugbot is set up for automated code reviews on this repo. Configure here.}