feat: speak the v1 episode wire (verifiers multi-agent restructure)

[hallerite]

Companion to PrimeIntellect-ai/verifiers#1939 (multi-agent v1). Do not merge before #1939 — deps/verifiers is pinned to the PR-branch head for now and should be re-pinned to the merge commit once it lands.

What changed in verifiers

• run_rollout now returns an Episode — the env-rollout atom: the task, a rollout-level errors list, and the run's traces (each stamped role/trainable) — instead of a single Trace. run_group survives only for the legacy (v0) bridge; a v1 server refuses it (sibling-dependent signals now score inside the env's own rollout).
• EnvServerConfig composes instead of inherits: the environment is a nested env block (env.taskset, per-seat settings like env.agent.harness, run limits like env.timeout/env.max_output_tokens/env.max_concurrent), with pool and the legacy id/args/extra_env_kwargs beside it. verifiers refuses the old flat taskset/harness keys with pointer messages.

What this PR changes

Multi-agent episodes are trainable

• Episode-shaped pipeline — Env.run_rollout yields one Rollout per trace, all stamped episode_id/episode_rollouts; the sinks finalize groups on complete episodes (singleton stamps keep single-agent and legacy counting bit-identical); the dispatcher's group bookkeeping runs in scheduling units so a fanned episode is one dispatch however many traces it returns. Train drops frozen (trainable=False) seats; episode-level errors fail every sibling (the episode is the failure atom, matching the env's own retries); eval keeps only the policy-played seats so a frozen judge can't poison env metrics.
• Algorithms declare the env shape they accept — Algorithm.multi_seat (False on every existing type). A multi-trace episode under a single-seat algorithm raises UntrainableEnvError, which fails the run: the mistake is categorical, so the dispatcher re-raises it instead of absorbing it into error markers (an all-refused stream would never fill a batch), and the main loop now surfaces any dead background component instead of polling an empty queue forever. A sole trainable seat beside frozen ones (the agentic-judge shape) is single-seat from the trainer's perspective and trains under plain grpo.
• algo.type = "hierarchical_grpo" — GRPO for multi-seat envs: each role baselines against whoever attempted the same prompt. A role fanned within an episode (proposer-solver's n solver attempts at one minted problem) baselines within its episode; a once-per-episode role (the proposer) across the group's episodes. Degenerates to plain GRPO on single-agent envs. configs/debug/v1/proposer_solver.toml is the runnable smoke.
• algo.type = "rae" — SPIRAL's Role-conditioned Advantage Estimation (arXiv:2506.24119): reward minus a per-role EMA baseline maintained across the run, the self-play estimator for zero-sum multi-seat games (a sibling-relative baseline would couple the opponents' credit). Second multi_seat algorithm — zero pipeline changes needed. Validated live on a self-play Kuhn-poker env (turn-per-run, standalone): 6 training steps, per-episode zero-sum payoffs on role-stamped decision traces.
• Grouping semantics live in the algorithm over the wire stamps (role/episode_id), deliberately not deepening the pipeline's group machinery.

Wire adaptation

• orchestrator/envs.py — unwraps episodes as above; an episode that failed before any trace completed becomes an error marker carrying the real task. run_group is the legacy-only route (the dispatcher already gates it on requires_group_scoring, which only the v0 bridge sets). Spawned servers receive the nested env block as their config.
• configs/orchestrator.py — drops the local is_legacy/env_id overrides (vf's EnvServerConfig provides them) and retargets the legacy-kwargs resolution at self.env.…; flat run-limit keys on an env entry get pointer refusals; the deprecated [[orchestrator.env]] shorthand combined with explicit train.envs is a hard conflict.
• Env entry lists renamed env → envs — [[orchestrator.train.envs]] / [[orchestrator.eval.envs]]: each entry wraps one nested env block, so the list reads as what it is and sub-tables avoid an env.env stutter. The deprecated [[orchestrator.env]] shorthand still translates (now onto the renamed field). No alias for the old list name — every entry's inner layout changes in this PR anyway, so it's one migration, not two.
• Config TOMLs (56 files) — taskset/harness/timeout/token limits move under the nested env table; prime-rl keys (name, ratio, sampling, group_size, algo, …) stay top-level.
• scripts/export_sft.py — handles both traces.jsonl shapes (episode-per-line eval runs AND flat trace-per-line trainer dumps / pre-episode runs); drops untrainable traces by default (--include-untrainable keeps them); malformed lines report file:line.
• Docs/skills — docs/configuration.md env-layout section, docs/algorithms.md + skills/configs/SKILL.md gain hierarchical_grpo and the multi-seat declaration rule, stale num_workers pointer in docs/inference.md.
• deps/verifiers + uv.lock — submodule bump to the fix env server task cancellation #1939 head; relock picks up its new example env package.
• tests/conftest.py — the zombie-cleanup pkill torchrun/VLLM sweep is CI-gated; it matched every such process on the machine, so any local pytest run killed live training runs beside it.

Behavior notes

• Single-agent training and eval are unchanged: a single-agent episode wraps exactly one trace with singleton episode stamps, so every downstream reader sees the same Rollout flow as before.
• batch_size counts traces (what the trainer consumes); group_size counts episodes per task (what the dispatcher schedules). On single-agent envs these coincide, as before.
• Known v1 limits (documented, deliberate): the permit counter paces dispatches, not traces (bound server-side fan-out with env.max_concurrent); the difficulty filter and per-env reward metrics mix roles on multi-agent envs; hierarchical_grpo has no length penalty yet.
• Free new knob: env.max_concurrent bounds a served worker's concurrent agent runs (an env's internal fan-out counts), riding the same nested config.

Verification

• uv run pytest tests/unit -m "not gpu" — 508 passed (includes test_load_configs over every TOML, the unwrap-seam contract, the episode counting rule, and the hierarchical partition math).
• ruff check / ruff format --check clean.
• E2E: 20-step single-env RL debug run and a 12-step two-train/two-eval multi-env run, both green end-to-end; multi-agent training smoke (proposer_solver.toml: proposer-solver + hierarchical_grpo on 2 GPUs) runs episodes through dispatch → episode-counted groups → hierarchical advantages → trainer steps; the same env under plain grpo fails the run in seconds with the UntrainableEnvError pointer (verified live).

🤖 Generated with Claude Code

---

Note

High Risk
Touches core rollout dispatch, group/batch accounting, and all RL configs—mis-counting episodes or env migration mistakes would break training silently or at scale; behavior change is gated on verifiers #1939 merge.

Overview
Adapts prime-rl to verifiers v1 episodes (one env rollout → multiple role-stamped traces) and restructures how environments are configured in TOML.

Orchestrator pipeline — run_rollout expands each episode into one Rollout per trace with shared episode_id / episode_rollouts. Train/eval sinks and group finalization use complete episodes (not raw rollout count). Train drops trainable=False seats; eval on multi-seat episodes keeps policy-played traces only. Multi-trace training under a single-seat algorithm raises UntrainableEnvError (fails the run; dispatcher re-raises). Algorithms gain multi_seat; new types hierarchical_grpo (per-role GRPO baselines) and rae (per-role EMA baselines for self-play).

Config — train.env / eval.env lists become train.envs / eval.envs. Each entry nests verifiers settings under env.* (env.taskset, env.agent.harness, run limits on env). Flat keys like timeout on the entry get validation pointers. ~56 TOMLs migrated; adds configs/debug/v1/proposer_solver.toml smoke.

Other — export_sft.py sniffs episode vs flat trace lines and drops untrainable traces by default; docs/skills updated; deps/verifiers + proposer-solver-v1 in lockfile; CI-only pytest zombie cleanup.

^{Reviewed by Cursor Bugbot for commit 984ad90. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 984ad90. Configure here.}

[cursor]

Eval progress counts traces not episodes

Medium Severity

Eval batch completion now uses complete_episodes, but batch_progress still reports len(pending_batches) against an episode-based batch_size_for. Multi-seat evals can show 100% progress while the epoch is only half finished, because each episode contributes several traces.

 

^{Reviewed by Cursor Bugbot for commit 984ad90. Configure here.}

[mikasenghaas]

Closing — superseded by the multi-agent v1 API that landed in verifiers (PrimeIntellect-ai/verifiers#1939); the prime-rl adoption will be redone against main.