feat: algorithm abstraction — named algorithm classes + inline frozen-model references (grpo, opd, sft_distill, self_distill, echo)

[hallerite]

Note

On verifiers-v1. Targets main after the vf-v1 ⇆ nano bridge (#2742) merged. Rollout is a vf.Trace subclass, samples are flat-token TrainingSample, scoring runs off vf.Trace branches/nodes, and the renderer is required. origin/main is merged in (latest: #2879 internal cleanup, #2880 eval-metric fix, #2881 dead-processor= removal, verifiers bumped to 62dc38a5); the branch is even with main and CI is green.

The two prior reach-arounds are now resolved on this branch:

1. OPD/OPSD prefill scoring — the hand-rolled compute_prefill_logprobs is gone; prefill logprobs go through InferencePool.score() (PrefillScorer).
2. echo granularity — upstream MessageNode.is_content landed, so echo weights only message-body tokens (scaffold excluded), with a whole-span fallback when attribution is absent. Pre-v1 parity restored.

Verification: verifiers pinned at submodule 62dc38a5 (has is_content, Trace.to_record(), Branch.input_len/output_len), no merge conflicts, ruff clean, imports resolve against the pinned verifiers, all CI green (unit + every integration suite). GPU smokes in a prior session confirmed all three loss components (rl / ce / ref_kl) train and learn.

---

What

Makes prime-rl's training algorithm a first-class, hackable abstraction — removes model roles from the pipeline, unifies every training signal under one concept (the advantage: credit assignment and loss routing, fused), and makes each algorithm a named runtime class that owns its methods. There is no registry and no separate "token scorer": there is the live policy — the only model prime-rl ever hosts — and per-env algorithms whose model references are either "policy" or an inline externally-hosted frozen endpoint.

An algorithm is a bundle of two components, configured under [orchestrator.algo] (per-env: algo = {...}) and resolved per env. algo.type names the algorithm — grpo, max_rl, opd, opsd, sft, echo — and each type's class defaults are its vetted setting; there is no separate preset layer:

1. Sampling — how train rollouts are produced: which model generates them, "policy" or an inline frozen endpoint. At runtime this builds the env's Sampler (orchestrator/sampler.py) — the pool rollouts come from, the liveness consequences (logprobs, prefix-cache salting, staleness), and the home of future sampling strategies (replay buffers, branching).
2. Advantage — the per-token training signal: one mapping from a finalized rollout to per-token (loss component, weight) pairs. Credit assignment computes the magnitude, loss routing picks the component — two coordinates of the same output, so they are one config component and one runtime object. Group-relative strategies compute scalars on the orchestrator and ship numbers; reference-KL strategies query a reference model as each rollout arrives and ship its prefill logprobs for the trainer to evaluate against the live policy. The strategy determines the action-token loss component (rl / ce / ref_kl) and what happens to env-provided observation tokens (masked out by default; echo trains on them with weighted CE — selected by message role via the renderer's per-token attribution, each role at its own alpha, tool-response bodies at λ=0.1 by default, optionally narrowed by a user-supplied per-rollout token filter).

The training loss is a sum of three components — rl (DPPO+KL or custom), ce (masked NLL), ref_kl (reverse KL to a reference as the PG signal) — each normalized by its own global token count:

$$\mathcal{L} = \tfrac{\sum \mathcal{L}_{rl}}{N_{rl}} + \tfrac{\sum \mathcal{L}_{ce}}{N_{ce}} + \tfrac{\sum \mathcal{L}_{ref_kl}}{N_{ref_kl}}$$

The orchestrator stamps per-token component weight streams (rl_weights / ce_weights / ref_kl_weights) plus the per-token advantage stream (advantages — there is no scalar advantage anywhere; uniform group credit is broadcast over completion tokens at assignment); a weight scales that component's per-token loss, 0.0 removes the token from the component's mask and denominator, and components may overlap on the same token (gradients sum). Per-component normalization means components never dilute each other: echo's observation tokens no longer shrink the rl term's effective per-token learning rate (previously both shared one global denominator, so the rl gradient scaled with the batch's obs/action ratio), and a supervised env packed next to a GRPO env doesn't soften its gradient.

[orchestrator.algo]
type = "opd"

[orchestrator.algo.teacher]   # opd's own field: the frozen model it scores against
name = "Qwen/Qwen3-32B"
base_url = ["http://localhost:8001/v1"]

The trainer is algorithm-blind (component weight streams ship per token on the wire; the trainer executes the three fixed components), and the orchestrator pipeline is too: dispatcher, train sink, and orchestrator call hooks on each env's Sampler + Algorithm objects and never branch on algorithm config or model roles.

The API — two seams to override

Writing an algorithm is subclassing Algorithm (orchestrator/algo/base.py) and overriding the hook(s) your signal needs. The whole override surface is small and identical for every algorithm — two declarations, a lifecycle hook, and two scoring hooks on one scope/timing ladder:

class Algorithm:
    # declaration — read by the pipeline, never branched on
    action_loss_type: ClassVar[ActionLossType] = "rl"   # which component action tokens feed: rl | ce | ref_kl

    async def setup(self) -> None: ...                   # connect pools to frozen models via self.connect(ref)

    async def score_rollout(self, rollout: Rollout)      -> None: ...   # one rollout, on arrival (model I/O here)
    async def score_group  (self, group:  list[Rollout]) -> None: ...   # the cohort, before filtering

The two hooks are one ladder of scope, the wider scope unlocked by a later barrier — so scope and timing coincide:

hook	input	fires	scope	model access
score_rollout	Rollout	as each rollout is tokenized	one rollout, no siblings	yes (self.policy_pool / self.teacher_pool)
score_group	list[Rollout]	on group completion, before filtering	the whole cohort	—

• score_rollout — rollout-local signals (raw reward, process rewards, echo's observation weighting) and the model-access seam: query a reference (OPD's frozen teacher, OPSD's live policy) and attach per-token logprobs as each rollout arrives. It runs before the pre-batch filters, so it pays reference compute on rollouts that may then be filtered — OPD/OPSD assign no advantage, so advantage filters never drop them anyway, and the serial main loop bounds in-flight scoring without an explicit cap.
• score_group — the group-relative seam: GRPO / MaxRL baselines, std-normalization. It runs before filtering on purpose, because advantage-based filters read the streams it writes.

Both hooks are async, so either may do I/O — a teacher or process-reward model at arrival, or a judge at group time whose verdict a pre-batch filter then reads; a hook that only does advantage math never awaits.

The input type is always Rollout (orchestrator/types.py) — which is the env's typed vf.Trace subclass, with prime-rl's training fields bolted on. A hook reads the trace directly and writes credit through one method:

rollout.reward       # the rollout's scalar reward (vf.Trace property)
rollout.nodes        # the message graph; rollout.num_turns / .task / .info / ... read straight off the trace
rollout.samples      # the TrainingSamples the renderer built (per branch)
rollout.env_name     # the producing env
rollout.assign_advantages(scalar | list[float])   # write the rl credit stream

assign_advantages takes either a scalar (broadcast over the rollout's trainable mask-True tokens — the common, uniform-credit case) or an explicit per-token list (aligned full-length to the samples' tokens). It writes the per-token advantages stream; advantages are per-token everywhere stored or shipped, the scalar is only a write-time convenience. For non-rl components an algorithm writes the sample stream directly — echo writes sample.ce_weights over is_content tokens at score_rollout time — at the stage where that data is valid. There's no read-only facade: this is a power-user extension surface, so a hook gets the whole Rollout (an earlier RolloutView handle was collapsed away — everyone reached through its .raw escape hatch anyway). Stage discipline (don't read is_filtered before filtering, don't read advantages before assigning) is a convention, not an enforced wall.

The pipeline drives the hooks through two non-virtual Algorithm methods named by the barrier they sit on — finalize_rollout / finalize_group — so the hooks answer "what do I override" and the finalize methods answer "when does the pipeline call me". Default hook bodies are no-ops: an algorithm that only needs a group baseline overrides score_group alone and inherits the rest.

The algorithm classes

Each algorithm is a named runtime class — the algorithm object is the algorithm. Dispatch is keyed on algo.type — it names the algorithm, and each config class's defaults are its vetted parameterization:

Each class overrides the scoring hook(s) its signal needs, on the scope/timing ladder score_rollout (one rollout, on arrival; async, model access) → score_group (the cohort, before filtering):

algo.type	Class	Component	hook(s)
grpo	GRPOAlgorithm	rl	score_group: group-norm scalars (optional length penalty)
max_rl	MaxRLAlgorithm	rl	score_group: mean-normalized group scalars (arXiv:2602.02710: unbiased for the order-group_size truncation of the max-likelihood objective)
echo	EchoAlgorithm	rl + ce	score_rollout: env-provided tokens selected by message role (roles.<role>.alpha, tool bodies @ 0.1 default), optional user token filter; score_group: group-norm scalars (inherited)
opd	OPDAlgorithm	ref_kl	score_rollout: own-context prefill under the teacher (a required frozen model; no credit — advantages=None, filters skip)
opsd	OPSDAlgorithm	ref_kl	score_rollout: demo-conditioned prefill under the live policy (self-distillation, no model to declare; no credit)
sft	SFTDistillAlgorithm	ce	score_group: group-norm (feeds reward-based filtering)

Reading a class top to bottom reads the algorithm — one module per algorithm under orchestrator/algo/ (grpo.py, opd.py, …, with the base class and its non-virtual finalize_* drivers in base.py and dispatch in the package __init__); writing your own is subclassing Algorithm and overriding the seam(s) your signal needs (see The API — two seams to override above). Duplication of orchestration between similar algorithms (OPD vs OPSD) is accepted so each class stays self-contained; shared math (group normalization, prefill alignment, length penalties) lives as plain functions in algo/advantage.py. Algorithms take exactly one host-owned resource beyond their config — the live policy pool (Algorithm(config, policy_pool), never closed by the algorithm). Anything else an algorithm needs it builds from its own config in setup(): opd connects its frozen teacher; opsd builds the renderer for its demonstration hint (the tokenizer is always the live policy's — self-distillation has no separate model). Text → token ids always goes through the renderer, the same path the policy's own prompts take.

One deliberate expressiveness trade: loss routing is not a free config axis (there is no algo.loss; you can't write opd + observation-CE in TOML) — routing variation is algorithm variation, expressed as a class. ECHO is a proper advantage type, not a flag on GRPO. Its selection surface is configurable where it matters: per-role alpha (roles.system/user/assistant/tool — setting any role replaces the whole table) and an optional filter hook (import_path + kwargs, called once per rollout as filter_fn(rollout, **kwargs) -> list[list[bool]], one keep-mask per trainable branch spanning that branch's token_ids; it can only narrow the role selection, never widen it) — the raw rollout exposes message text and sampling logprobs, so warning filters and low-probability filters are user code, no framework surface.

Extending it: code, not config-as-code

The loss is fixed; the signal is open — and you open it in code. There are exactly three loss components (rl / ce / ref_kl), each globally normalized, and the trainer is algorithm-blind — it runs those three and nothing else. action_loss_type is not a config field and has no "custom" value: it is a ClassVar declaration on each algorithm class (Literal["rl", "ce", "ref_kl"]) routing the action tokens into one of the three. You cannot configure a fourth loss component into existence.

A new credit-assignment scheme is a new named algorithm in the repo, not a config that points at an import path. Subclass Algorithm, assign credit in the scoring hook whose timing fits (score_rollout / score_group) via rollout.assign_advantages(...), add a typed *AlgoConfig to the discriminated union, and register the class in ALGORITHM_CLASSES. There is no custom advantage type and no advantage import_path — the config surface is a closed menu of vetted algorithms with typed, validated fields; extension is forking, and the authoring path is deliberately small (one score_group that builds rewards and assigns; shared math like efficiency_shaping lives in algo/advantage.py).

The one remaining import-path escape hatch is echo's optional token filter — and it is deliberately the weakest kind: it does not define an algorithm or touch the loss, it is a per-rollout boolean predicate (filter_fn(rollout, **kwargs) -> list[list[bool]], one keep-mask per trainable branch) that can only narrow the role-selected echo tokens, never widen them or assign credit. It parameterizes a vetted algorithm's data selection (drop tool-output warning lines, drop low-probability tokens), the same category as an env's reward function — env-specific data, not algorithm structure. The flip side is the hosted story: because every shipped algorithm is a named class plus per-token weight streams on the wire (data, not code), a hosted product exposes the vetted algorithm names and their knobs, disables that lone filter import-path, and still gets the full combinatorial reach over which streams get written without arbitrary code execution.

The algorithms

algo.type	Sampling	Loss
grpo (default)	policy	rl
max_rl (arXiv:2602.02710)	policy	rl
opd	policy	ref_kl (needs a frozen teacher)
sft	frozen model (sampling.source)	ce
opsd (SDFT, arXiv:2601.19897)	policy	ref_kl vs the live policy (self-distillation; no model to declare)
echo (ECHO)	policy	rl on actions + per-role α·ce on env-provided tokens

There is no preset layer: the type IS the algorithm, its class defaults ARE the vetted setting, and every key beyond type is visibly the user's own assembly (an earlier iteration had named atomic presets; with type-plus-defaults equal to the preset for every algorithm, the layer had nothing left to do and was deleted). Per-env algorithms compose in one run. Because a reference can be the literal "policy", opsd runs the SDFT paper's setting with zero extra deployments — that's its default.

Model references — no registry; roles are algorithm-local

prime-rl assumes it is never responsible for hosting any model other than the trainable policy. Everything else is an external OpenAI-compatible endpoint, declared inline where it's used:

• ModelReference = "policy" | FrozenModelConfig, where FrozenModelConfig is just the existing ClientConfig plus the served model's name — no new declaration scheme. base_url (or an elastic deployment) is required: a frozen reference with no endpoint fails at parse time.
• Each algorithm names its reference on the field where the model is actually used — there is no shared teacher slot and no model_role indirection. opd declares a required frozen teacher (typed FrozenModelConfig); sft samples from its sampling.source (a frozen model); opsd self-distills against the live policy and declares no model at all. The base config knows nothing about teachers. Validation errors name the field directly ("algorithm 'opd' ... teacher: Field required").
• Liveness is a property of the reference, not a role: policy-sourced rollouts get version-salted prefix caches, carry sampling logprobs, and age off-policy; frozen-sourced rollouts get a stable prefix cache, skip the logprobs knob, and are never off-policy-cancelled. The pipeline branches on liveness alone.
• Config-time validation: opd's teacher is typed FrozenModelConfig, so "policy" isn't even representable (the KL would be ≡ 0); frozen sampling can't feed an rl/ref_kl-type algorithm (no policy sampling logprobs for the importance ratio); sft without a frozen source is rejected (CE on the policy's own tokens is not a distillation target); and group-relative advantage with group_size=1 warns loudly.

Transport (orchestrator → trainer wire)

Everything an algorithm decides reaches the trainer as per-token streams on TrainingSample (src/prime_rl/transport/types.py) — there is no scalar advantage and no algorithm tag on the wire. The trainer reads the streams, not the algorithm. Five optional per-token arrays carry the whole training signal, each the full prompt + completion length:

field	what it carries	None means
advantages	the rl credit stream (one number per token)	no rl credit assigned — opd/opsd; legal only for samples with no live rl tokens, else the trainer raises
rl_weights	scales the rl (DPPO + KL) component per token	rl weight 1.0 on every trainable token
ce_weights	scales the ce (masked NLL) component per token	no ce component
ref_kl_weights	scales the ref_kl (reverse-KL-to-reference) component per token	no ref_kl component
ref_logprobs	the reference model's prefill logprobs, against which the trainer takes the KL	no reference

Three rules make the wire legible:

1. Per-token from end to end. Uniform group credit (GRPO's reward-minus-mean) is broadcast over the completion tokens at assignment time — the scalar never reaches the wire. A weight of 0.0 removes a token from its component's mask and its global denominator; components may overlap on a token (their gradients sum).
2. None = absent, and absent is the hot path. Plain GRPO ships exactly one extra stream (advantages, same order as completion_logprobs) and nothing else — the wire is as small as before this PR. ref_kl ships logprobs, not precomputed advantages, because the trainer evaluates the KL against the live policy each microbatch (a scalar couldn't capture that).
3. Streams materialize on demand at packing. The packer (trainer/batch.py) concatenates samples into a MicroBatch; when one packed sample carries a stream a neighbor lacks, the neighbor's span is backfilled with the component's identity fill (STREAM_FILL = {rl_weights: 1.0, ce_weights: 0.0, ref_kl_weights: 0.0}), so every per-token array stays aligned to input_ids. This is exactly what lets samples from different algorithms (e.g. a GRPO env packed next to an OPD env) share one micro-batch — membership is per token, not per bin. A post-pack assert pins every stream's length to len(input_ids).

The trainer then runs each component over its stream (rl mask = loss_mask & rl_weights != 0; ce/ref_kl mask = weights != 0) and sums the per-component means; the three global denominators [N_rl, N_ce, N_ref_kl] come from one batched all-reduce, so every rank issues the same collective and the no-stream path is identical to the old single-loss_scale math.

One field on TrainingSample is deliberately not on the wire: obs_spans, interleaving's provenance record for env-provided observation tokens ([completion_start, step_idx, step_prompt_start, length] per later-turn extension span). Algorithms that train on observations (echo) consume it at group time and write the ce_weights stream directly; it is cleared before transport.

How

• Configs (prime_rl.configs.algorithm): one algorithm discriminated union absorbs the former token scorers (logprobs/demo_logprobs → opd/opsd with a ModelReference) and loss routing (EchoAlgoConfig(GRPOAlgoConfig) carries the role table + filter); the action-token loss component is an action_loss_type class property of the algorithm, never configured; each variant declares its own reference field where the model is used (opd.teacher as a required FrozenModelConfig, sft.sampling.source; opsd none) — no algo.model shorthand and no model_role indirection.
• No preset machinery at all: env algorithm inheritance is a one-loop after-validator (envs without their own algo inherit the top-level one). Every AlgoConfig is built exactly once with everything in place: no preset tables, no name field, no merge machinery, no __pydantic_fields_set__ surgery.
• Runtime (the orchestrator/algo/ package + orchestrator/sampler.py):
  • Algorithm — base class with the two scoring hooks (score_rollout / score_group, detailed under The API above) plus setup/connect for frozen-model pools. Two non-virtual methods drive them and stamp the wire fields: finalize_rollout runs score_rollout as each rollout is tokenized (rollout-local credit and reference prefill — OPD/OPSD attach ref_logprobs here); finalize_group runs score_group then stamps each sample (the advantage stream — prompt-padded and sliced across samples — plus per-token loss routing into the declared component). setup() connects an InferencePool to the algorithm's inline frozen reference (connect_frozen_pool — client-side only: prime-rl connects and waits, never launches) and tracks connected_pools for shutdown. The live policy pool, by contrast, is handed to every algorithm as self.policy_pool (host-owned, never closed by the algorithm — opsd self-distills against it, others may judge against it or ignore it); connect() only ever opens frozen endpoints, so connected_pools is exactly the set the algorithm owns and must close. The named classes own their hook bodies outright; build_algorithm dispatches on algo.type.
  • Sampler — one per env, owns the rollout source: the generating pool (policy, or a frozen pool it connects in its own setup()), samples_from_live_policy, and sampling_args() (strips the logprobs knob for frozen endpoints). The dispatcher reads env.sampler for pool/liveness; the algorithm never sees sampling.
  • The pipeline has zero algorithm conditionals: the dispatcher derives cache-salt/aging from the sampler's liveness; the sink calls finalize_rollout per rollout (reference scoring runs inline here) and finalize_group per group; orchestrator setup is one asyncio.gather over both objects' setup().
• Wire (transport/types.py): the per-token streams laid out under Transport above. TrainingSample/MicroBatch carry the optional rl_weights / ce_weights / ref_kl_weights streams, the advantages stream, and ref_logprobs; absent streams keep the plain-GRPO wire minimal, and the packer backfills streams across heterogeneous bins (STREAM_FILL) so samples of different algorithms pack freely. The historical collapse of the scalar advantage + token_advantages into one per-token advantages stream, and teacher_logprobs → ref_logprobs, are in Breaking changes below.
• Trainer: compute_loss runs each component over its weight stream (rl mask = loss_mask & rl_weights != 0; ce/ref_kl mask = weights != 0) and sums the per-component means; the three global denominators come from one batched all-reduce ([N_rl, N_ce, N_ref_kl]), so every rank issues the same collective. The no-stream hot path is byte-identical to the old single-loss_scale math with zero extra device syncs. Mixed-algorithm packing fix: bins mixing ref-bearing (e.g. OPD) and ref-less (e.g. GRPO) samples now keep ref_logprobs position-aligned with input_ids (0.0 placeholders, backfill-or-pad like the sibling per-token arrays), with a regression test and a post-pack invariant assert that every per-token array on every micro batch matches len(input_ids).

Breaking changes (intentionally, no deprecation aliases)

• orchestrator.training_mode is deleted (was rl/opd/sft) → [orchestrator.algo] type = ....
• advantage.type → algo.type: the algo bundle is a discriminated union keyed on type directly — [orchestrator.algo] type = "grpo" (was [orchestrator.algo.advantage]). The unused advantage = {} env/top-level shorthand is deleted; opd declares a native algo.teacher field.
• The advantage type names the algorithm: group_norm → grpo, ref_kl → opd, demo_ref_kl → opsd, supervised → sft (config classes renamed to match). The preset name field is deleted — type-plus-defaults is the algorithm.
• Echo selection is a role table: observations / observation_weight are replaced by roles.<role>.alpha (tool bodies @ 0.1 default; setting any role replaces the table) plus an optional filter hook. Echo always requires orchestrator.renderer (the no-attribution "all" mode is gone).
• sft requires a frozen sampling source — CE on the policy's own tokens is rejected at validation (previously silently assemblable).
• [orchestrator.teacher] is deleted → each algorithm declares its frozen model inline where used: [orchestrator.algo.teacher] for opd, [orchestrator.algo.sampling.source] for sft (opsd needs none), with name + base_url.
• The policy sub-config is [orchestrator.model], flat: name / lora / vlm / trust_remote_code directly on it, the deployment under [orchestrator.model.client] — no more model.model stutter, in configs or resolved dumps (HostedModelConfig is now ModelConfig + a client field, mirroring FrozenModelConfig). The [orchestrator.policy] / [orchestrator.student] aliases, the [orchestrator.client] shorthand, and the flat-key re-nesting shim are deleted.
• algorithm.token_scorer is deleted — scorers are advantage strategies now (logprobs → ref_kl, demo_logprobs → demo_ref_kl).
• algo.loss does not exist — the action-token loss component derives from the advantage strategy, and observation-token routing is the echo advantage type (observation_weight).
• Advantage types: default → group_norm; none is deleted (the ref_kl-family strategies carry its no-scalar semantics; nothing else used it).
• Wire/trainer: TrainingSample.advantage (scalar) + token_advantages → one per-token advantages stream; AdvantageOutputs is deleted — algorithms assign credit in their scoring hooks via rollout.assign_advantages(scalar | list[float]) (scalar broadcast for uniform credit), and advantage-based filters/metrics derive from the streams (zero-advantage filter = all-zero stream; logged distributions = per-rollout means). TrainingSample.teacher_logprobs → ref_logprobs; the loss-type partition (loss_type / token_loss_types / token_loss_weights / loss_type_ids, LOSS_CORE_*) is replaced by the component weight streams rl_weights / ce_weights / ref_kl_weights (the partition was the degenerate disjoint case); the time/teacher_logprobs metric is removed (reference scoring now runs inline at rollout arrival, not as a distinct batch phase).
• configs/debug/training_modes/ → configs/debug/algorithms/ (incl. new self_distill.toml running SDFT against the live policy); CI integration configs migrated.

Configs are extra="forbid", so stale configs fail loudly at parse time with the exact unknown key.

Merged with main including #2720 (teacherless SFT). Its spelling is deliberately not carried forward: sft means distillation from a frozen teacher, and pointing it at "policy" is rejected at validation — CE on the policy's own rollouts is a different algorithm, and if wanted it gets its own advantage type/class rather than a degenerate spelling of sft.

Also merged main including #2641 (multiplexed trainer token export): the export machinery composes with the component weight streams — each exported record carries the per-token rl/ce/ref_kl arrays, and micro-batches carry run_id/run_step alongside the streams (training_mode stays deleted).

Note: this PR absorbed #2764 (named algorithm classes → two-component model → Sampler split), #2778 (MaxRL), and #2782 (echo per-role selection + filter hook); those branches were merged into this one and GitHub marked their PRs merged.

Deferred (next PRs)

The two-component model is built so each of these lands as its own PR without re-touching the wire:

• Sampling strategies behind the Sampler — today the Sampler answers one question (which pool, and is it live); next it absorbs within-env example iteration and group production, plus a sink→sampler observe() feedback edge at group finalization. In increasing order of machinery that unlocks: difficulty-pool curricula (per-example selection state fed back from group rewards), static dataset sources (supervised training from demonstrations instead of a frozen endpoint), and replay buffers / offline experience (a store between stamping and batch assembly — advantages and weight streams are already stamped-then-frozen at group finalization, which is the prerequisite).
• Trainer collapse — the three fixed components reduce to two cores (policy-gradient × supervised) composed with per-token factor streams. Dissolves the rl-slot-only custom-loss wart and lets the stability guards (importance correction, trust region) compose around custom losses instead of being replaced by them.
• Remaining ECHO surface — per-role α, arbitrary roles, and content/sampling-logprob filters shipped in this PR; still open: tool_names filtering (non-breaking optional field on the tool role; message_tool_names already rides in the attribution) and θ-dependent low-probability filters, which need a trainer-side per-component knob since the denominator collective happens pre-forward.
• Smaller: config knobs for component sums (per-env α/β/γ folded into the streams orchestrator-side — the wire and trainer already support overlap); opt-in weight-sum normalization (Σw denominators — a globally-folded λ cancels under it, so it pairs with a trainer-side coefficient); per-slot loss-fn configs for ce/ref_kl (the ref_kl trust region — one-sided today — becomes a deliberate choice there); barrier-blind sink (the group wait becomes a dependency declared by the algorithm; finalize_group → finalize); EMA/lagged reference endpoints; exposing frozen endpoints to envs as judge clients.

87c38c704 renames the config surface from advantage.type to algo.type per review: the algo bundle flattens into a discriminated union (each *AlgoConfig variant carries type + its params + the shared sampling/teacher on BaseAlgoConfig), the unused advantage = {} shorthand + its fold validator are dropped, and each algorithm declares its reference inline (opd.teacher, sft.sampling.source). Runtime Algorithm takes the algo config; build_algorithm dispatches on algo.type. 12 debug/CI TOMLs, docs/algorithms.md + training.md, and the unit tests migrated. Orchestrator+config unit tests pass (incl. the full TOML-load sweep) + a build_algorithm dispatch smoke over every algorithm type; ruff clean.

cf9a21a3b / f13119938 collapse the override surface from three scoring hooks to two: score_batch is removed and OPD/OPSD do their reference prefill in score_rollout (at arrival); finalize_rollout / finalize_group become non-virtual Algorithm methods (the cross-env finalize_batch is gone), and Algorithm is now constructed with just (config, policy_pool) — opsd builds its own hint renderer in setup() with the live policy's tokenizer (the model knob is gone; self-distillation has no separate model). The per-rollout max_concurrent scoring semaphore is dropped — the serial main loop already bounds in-flight reference calls — and with it the now-orphaned time/scoring metric. Docs + tests updated; orchestrator/config unit tests green.

🤖 Generated with Claude Code

---

Note

High Risk
Large breaking config and orchestrator/trainer contract change (per-token streams, deleted keys); mistakes in heterogeneous batches or frozen-model wiring could silently skew gradients until caught by tests.

Overview
Replaces the rl / opd / sft training-mode switch with [orchestrator.algo] type = … (grpo, max_rl, opd, opsd, sft, echo). Frozen teachers move inline (algo.teacher for OPD, algo.sampling.source for SFT); the trainable model is a flat [orchestrator.model] with [orchestrator.model.client] (no student / orchestrator.teacher).

Orchestrator runtime gains per-env Sampler (which pool generates rollouts) and Algorithm subclasses with score_rollout / score_group / score_batch, plus wire stamping of advantages, rl_weights / ce_weights / ref_kl_weights, and ref_logprobs. The dispatcher keys off policy vs frozen sampling for cache salting and off-policy cancellation; custom orchestrator.advantage and advantage.py are removed in favor of algorithm classes.

Trainer/docs/configs: loss is documented and wired as three globally normalized components; CI and configs/debug/training_modes/ become configs/debug/algorithms/ (new echo, max_rl, self_distill, mixed GRPO+OPD). Metrics rename time/teacher_logprobs → time/scoring.

^{Reviewed by Cursor Bugbot for commit c326868. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit a8dba67. Configure here.}

[mikasenghaas]

have to let @eexwhyzee know then

[mikasenghaas]

LFGTM