refactor(algo): remove group_relative; stop SFT assigning a throwaway advantage

group_relative was a ClassVar on every advantage config whose only consumer was AlgorithmConfig.requires_group_advantage -> warn_group_size, a config-time warning for group-relative algorithms run with group_size=1. SFT also overrode score_group to assign group-normalized advantages it never trains on (the loss is CE), purely so the zero-advantage filter would drop uniform-reward groups -- overloading 'advantage' as a filter key.

Remove the group_relative ClassVars, the requires_group_advantage property, warn_group_size and its orchestrator call site, and the now-unused warnings import. SFT drops its score_group override (inherits the base no-op) and ships no advantage: the zero-advantage filter skips None, and SFT is ce-only so the trainer never requires one. SFT now trains on every sampled token; reward-based filtering is opt-in via an explicit filter.

The group_size=1 footgun is already caught by the runtime empty-batch guard (orchestrator warns on zero-trainable batches and aborts after repeated ones), which covers all causes, not just this prediction.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: hallerite

docs/algorithms.md

@@ -111,7 +111,7 @@ kwargs = { patterns = ["WARNING"] }
 def drop_warnings(rollout, *, patterns: list[str]) -> list[list[bool]]: ...
 ```
 
-Component compatibility is validated at config time: frozen-model sampling can only feed the `ce` loss component — the `rl` and `ref_kl` components need the live policy's own sampling logprobs for importance ratios — `opd` pointed at `"policy"` is rejected as degenerate (zero KL), `sft` without a frozen source is rejected (CE on the policy's own tokens is not a distillation target), and group-relative advantage with `group_size = 1` warns that every advantage collapses to zero.
+Component compatibility is validated at config time: frozen-model sampling can only feed the `ce` loss component — the `rl` and `ref_kl` components need the live policy's own sampling logprobs for importance ratios — `opd` pointed at `"policy"` is rejected as degenerate (zero KL), `sft` without a frozen source is rejected (CE on the policy's own tokens is not a distillation target). A group-relative algorithm with `group_size = 1` produces all-zero advantages; the resulting empty batch is caught at runtime (the orchestrator warns and aborts after repeated zero-trainable batches), not at config time.
 
 ### Per-Env Algorithms
 
@@ -283,7 +283,7 @@ The advantage strategy is the `advantage` component of the [algorithm](#the-algo
 | `reward` | `rl` | Advantage = raw reward, no baseline. |
 | `opd` | `ref_kl` | On-policy distillation: per-token reverse KL to a reference model (`model`, an inline frozen hosted model), evaluated in the trainer from shipped reference logprobs. No credit — rollouts keep `advantages = None` (advantage-based filters never fire) and ship no advantage stream; `group_size` only fans out sampling. |
 | `opsd` | `ref_kl` | SDFT: per-token reverse KL to a demo-conditioned reference. No credit — rollouts keep `advantages = None` (advantage-based filters never fire) and ship no advantage stream. |
-| `sft` | `ce` | Cross-entropy on the sampled tokens. The loss ignores advantages, but group-relative credit is still assigned so reward-based filtering keeps working. |
+| `sft` | `ce` | Cross-entropy on the sampled tokens. Assigns no advantage — trains on every sampled token. |
 | `custom` | `rl` | Your function (below); per-token advantages per rollout. |
 
 ### Default Advantage

packages/prime-rl-configs/src/prime_rl/configs/algorithm.py

@@ -25,7 +25,6 @@ class defaults are the vetted setting — ``type = "opd"`` with nothing else IS
 weights ship on the wire and the trainer just executes them.
 """
 
-import warnings
 from typing import Annotated, Any, ClassVar, Literal, TypeAlias
 
 from pydantic import AliasChoices, Field, model_validator
@@ -109,7 +108,6 @@ class GRPOAdvantageConfig(BaseConfig):
     consumed by the ``rl`` loss component on the rollout's action tokens."""
 
     action_loss_type: ClassVar[ActionLossType] = "rl"
-    group_relative: ClassVar[bool] = True
 
     length_penalty: LengthPenaltyConfig | None = None
     """Correctness-gated length penalty. ``tokens`` shapes by weighted token cost; ``turns`` shapes by trajectory turn count; None disables shaping. In mixed groups, lower-cost correct rollouts get amplified advantage (up to 2x), higher-cost correct rollouts are unchanged, incorrect untouched. In all-correct groups, below-average-cost rollouts get advantage in [0, 1], others get 0."""
@@ -187,7 +185,6 @@ class MaxRLAdvantageConfig(BaseConfig):
     zero-advantage filter drops it, matching the paper's K=0 convention)."""
 
     action_loss_type: ClassVar[ActionLossType] = "rl"
-    group_relative: ClassVar[bool] = True
 
 
 class RewardAdvantageConfig(BaseConfig):
@@ -196,7 +193,6 @@ class RewardAdvantageConfig(BaseConfig):
     ``rl`` loss component."""
 
     action_loss_type: ClassVar[ActionLossType] = "rl"
-    group_relative: ClassVar[bool] = False
 
 
 class OPDAdvantageConfig(BaseConfig):
@@ -210,7 +206,6 @@ class OPDAdvantageConfig(BaseConfig):
     only fans out sampling."""
 
     action_loss_type: ClassVar[ActionLossType] = "ref_kl"
-    group_relative: ClassVar[bool] = False
     model_role: ClassVar[str] = "teacher"
 
     model: ModelReference | None = None
@@ -235,7 +230,6 @@ class OPSDAdvantageConfig(BaseConfig):
     samples ship a neutral 0.0."""
 
     action_loss_type: ClassVar[ActionLossType] = "ref_kl"
-    group_relative: ClassVar[bool] = False
     model_role: ClassVar[str] = "teacher"
 
     model: ModelReference = "policy"
@@ -263,13 +257,12 @@ class OPSDAdvantageConfig(BaseConfig):
 
 class SFTAdvantageConfig(BaseConfig):
     type: Literal["sft"] = "sft"
-    """SFT distillation: cross-entropy on the sampled tokens. The ``ce``
-    loss component ignores scalar advantages, but group-relative scalars are still
-    assigned so reward-based filtering keeps working (the zero-advantage
-    filter drops uniform-reward groups)."""
+    """SFT distillation: cross-entropy on the sampled tokens. The ``ce`` loss
+    ignores advantages and SFT assigns none — it trains on every sampled token.
+    Reward-based filtering, if wanted, is an explicit filter, not smuggled
+    through an unused advantage stream."""
 
     action_loss_type: ClassVar[ActionLossType] = "ce"
-    group_relative: ClassVar[bool] = True
     source_role: ClassVar[str] = "teacher"
     """The sampling source is this algorithm's teacher — the frozen model
     whose tokens the policy trains on. Required: CE on the policy's own
@@ -283,7 +276,6 @@ class CustomAdvantageConfig(BaseConfig):
     each rollout's completion tokens."""
 
     action_loss_type: ClassVar[ActionLossType] = "rl"
-    group_relative: ClassVar[bool] = False
 
     import_path: str
     """Import path to the advantage function (e.g. ``my_module.my_advantage``)."""
@@ -345,12 +337,6 @@ class AlgorithmConfig(BaseConfig):
     """The per-token training signal: credit assignment and loss routing,
     fused. The ``type`` selects the algorithm."""
 
-    @property
-    def requires_group_advantage(self) -> bool:
-        """True when the advantage strategy assigns group-relative scalars,
-        i.e. degenerate with ``group_size=1``."""
-        return self.advantage.group_relative
-
     @model_validator(mode="after")
     def fold_model(self):
         """Fold the ``model`` shorthand into the component references.
@@ -414,15 +400,3 @@ def validate_component_compatibility(self):
                 "Use the 'sft' advantage to distill frozen-model tokens."
             )
         return self
-
-    def warn_group_size(self, group_size: int, env_name: str) -> None:
-        """Group-relative scoring with a single rollout per example collapses
-        every advantage to zero. Warn loudly — this is the classic footgun."""
-        if self.requires_group_advantage and group_size == 1:
-            warnings.warn(
-                f"Env '{env_name}' uses group-relative advantage ('{self.advantage.type}') with "
-                "group_size=1 — every advantage is 0 and (with the default zero-advantage filter) "
-                "no rollout will train. Set group_size >= 2 or a non-group-relative advantage "
-                "(e.g. advantage.type='reward').",
-                stacklevel=2,
-            )

packages/prime-rl-configs/src/prime_rl/configs/orchestrator.py

@@ -869,8 +869,6 @@ def resolve_batching(self):
         for env_cfg in self.train.env:
             if "group_size" not in env_cfg.model_fields_set:
                 env_cfg.group_size = self.group_size
-            assert env_cfg.algo is not None  # materialized by inherit_env_algorithms
-            env_cfg.algo.warn_group_size(env_cfg.group_size, env_cfg.resolved_name)
 
         # Resolve train env num_workers from max_inflight_rollouts
         for env_cfg in self.train.env:

src/prime_rl/orchestrator/algo/sft.py

@@ -1,22 +1,14 @@
 from __future__ import annotations
 
-from typing import TYPE_CHECKING
-
-from prime_rl.orchestrator.algo.advantage import assign_group_norm
 from prime_rl.orchestrator.algo.base import Algorithm
 
-if TYPE_CHECKING:
-    from prime_rl.orchestrator.types import RolloutView
-
 
 class SFTDistillAlgorithm(Algorithm):
     """Hard distillation. Needs a teacher: the frozen model that generates the
     rollouts (``sampling.source``); the policy trains with CE on its tokens.
 
-    The ``ce`` loss ignores credit, but group-relative advantages are still
-    assigned so reward-based filtering keeps working."""
+    Assigns no advantage — the ``ce`` loss ignores credit, and SFT trains on
+    every sampled token. Reward-based filtering, if wanted, is an explicit
+    filter, not smuggled through an unused advantage stream."""
 
     action_loss_type = "ce"
-
-    async def score_group(self, group: list[RolloutView]) -> None:
-        assign_group_norm(group, None)