fix(orchestrator): gate group-relative algorithms to a single trainable agent#3108
Open
hallerite wants to merge 1 commit into
Open
fix(orchestrator): gate group-relative algorithms to a single trainable agent#3108hallerite wants to merge 1 commit into
hallerite wants to merge 1 commit into
Conversation
hallerite
force-pushed
the
fix/gate-group-algos-to-one-trainable-seat
branch
from
July 22, 2026 11:30
5a66ae9 to
b68d05b
Compare
…le agent GRPO/MaxRL mean-center over the whole trainable cohort of a group. A multi-agent episode with several trainable agents would mix different agents' reward scales into one baseline — silently wrong credit. Refuse loudly instead: score_group now asserts the group's trainable traces share one agent name (assert_single_trainable_agent in algo/base.py). Single-trainable-agent multi-agent envs keep working unchanged — e.g. a trainable solver rewarded by a frozen agentic judge is exchangeable attempts by one agent, which is exactly what the baseline assumes. Echo inherits the gate via GRPO's score_group; per-rollout algorithms (opd/opsd/sft) are untouched. Cross-agent credit assignment stays open: implement your own algorithm for it. Verified: tests/unit/orchestrator/test_advantage.py + test_algorithms.py green, ruff clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
hallerite
force-pushed
the
fix/gate-group-algos-to-one-trainable-seat
branch
from
July 22, 2026 11:30
b68d05b to
388ac04
Compare
hallerite
changed the title
hallerite
marked this pull request as ready for review
July 22, 2026 13:22
| assert _max_rl(_make_group(rewards=[1.0, 1.0])) == pytest.approx([0.0, 0.0]) | ||
|
|
||
|
|
||
| def test_group_algos_refuse_multiple_trainable_agents(): |
|
|
||
| The default advantage is per-group reward minus per-group baseline (DR-GRPO without std normalization). For each prompt's group of `group_size` rollouts, every token in rollout $i$ receives advantage $s_i - \bar{s}$ where $\bar{s}$ is the group mean. | ||
|
|
||
| The group-relative algorithms (`grpo`, `max_rl`, `echo`) require the group's trainable traces to come from **one** agent — the baseline assumes exchangeable attempts by a single agent. Multi-agent envs work as long as exactly one agent is trainable (e.g. a trainable solver rewarded by a frozen agentic judge); a group spanning several trainable agent names is refused at scoring time. Credit assignment across multiple trainable agents is not something prime-rl prescribes — implement your own algorithm for it (see [Authoring an Algorithm](#authoring-an-algorithm)). |
Member
There was a problem hiding this comment.
do we have no such constraint for say opd?
Member
Author
There was a problem hiding this comment.
no, because for OPD, we just do a teacher prefill for each trainable trace
This was referenced Jul 24, 2026
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
GRPOAlgorithm.score_group/MaxRLAlgorithm.score_groupnow refuse a group whose trainable traces span more than oneagent_name(newassert_single_trainable_agenthelper inalgo/base.py). Echo inherits the gate through GRPO'sscore_group; per-rollout algorithms (opd/opsd/sft) are untouched.Why
Group-relative advantages assume the group is exchangeable attempts by one agent. After the multi-agent episode wire (#3104), a group's trainable survivors can span several agents, and mean-centering them together mixes different agents' reward scales into one baseline: silently wrong credit, no error.
The gate is deliberately the condition that matters, not "single-agent envs only": multi-agent envs with exactly one trainable agent keep working unchanged — e.g. a trainable solver rewarded by a frozen agentic judge. A group mixing trainable agent names is refused with an actionable message: mark the other agents untrainable in the env's
setup()(agents.<name>.trainable = False), or implement your own algorithm that assigns credit across agents.Cross-agent credit assignment is an open question prime-rl shouldn't answer silently in a base class — this refuses instead of guessing. If a well-founded scheme lands later, it arrives as a named
Algorithm, and this gate is exactly the code it replaces.Companion verifiers PR: PrimeIntellect-ai/verifiers#2101 (enforces standing at the source: an agent off the trainable model context refuses on training runs and is demoted on eval runs).
Test plan
uv run pytest tests/unit/orchestrator/test_advantage.py tests/unit/orchestrator/test_algorithms.py— green, including the newtest_group_algos_refuse_multiple_trainable_agents(mixed-agent group raises for both algos; same-agent multi-trace group still centers to zero).uv run ruff check/ruff format --checkclean on the touched files.🤖 Generated with Claude Code