Future: off-policy / replay GRPO consumer for captured flywheel rollouts

[ProfSynapse]

Summary

The flywheel now captures token-faithful rollout data (completion token ids + per-token logprobs) but nothing consumes it for GRPO training yet. This issue tracks adding an off-policy / replay GRPO trainer that learns from logged rollouts, to be picked up when the upstream replay-GRPO tooling is more stable.

Deferred deliberately — off-policy RL is finicky and the upstream support is experimental. Filing now so the capture work has a documented destination.

Background

Plan #2 (docs/plans/proxy-token-capture-grpo-feed-plan.md, branch claude/polar-token-rollout-MTzCp) added opt-in capture at the logging proxy:

• InferenceLogRecord.completion_token_ids / completion_logprobs / prompt_token_ids (JSONL only; omitted when unset)
• FlywheelConfig.capture_token_ids and proxy_inject_logprobs (both default off)
• stager._format_grpo_example passes the token fields through into Datasets/flywheel/vNNN/grpo_training.jsonl

So the data substrate exists. What's missing is the trainer that reads it.

Why there's no consumer today

• The auto-retrain dispatch in shared/flywheel/orchestrator.py (_select_trainer) only branches to SFT and KTO — there is no grpo branch, so the staged grpo_training.jsonl is produced but never trained from (true even before the capture work).
• GRPO is on-policy: it samples completions from the current model, rewards them, and pushes toward the better ones. The flywheel's logged completions came from an older behavior policy, so reusing them is off-policy and requires importance sampling: weight each token by π_current(token) / π_behavior(token).
• The captured fields are exactly the ingredients for that:
  • completion_token_ids — exact tokens to re-score under the current model (no re-tokenization drift)
  • completion_logprobs — the behavior policy logprobs (the denominator of the importance ratio; can't be recovered later)
  • reward (from fitness_score) — the advantage signal

Proposed work (when ready)

1. Evaluate TRL's experimental replay-buffer GRPO (trl/experimental/grpo_with_replay_buffer/) — verify whether it accepts stored (tokens, behavior-logprobs, reward) and does the importance-sampling correction, the same way env_mask was verified for Plan Fix KTO Dataset True-False Alternation Pattern #1. If it fits, build on it rather than from scratch.
2. Add a flywheel GRPO replay training mode that reads grpo_training.jsonl and replays the stored rollouts through the off-policy GRPO loss (no fresh generation).
3. Wire _select_trainer in the orchestrator with an elif trainer == "grpo" branch pointing at that mode (and let auto consider it).
4. Staleness controls — cap the importance-sampling ratio and/or only replay recent logs, since logprobs from a much older checkpoint produce noisy ratios.
5. Docs + tests — mirror Plan Fix KTO Dataset True-False Alternation Pattern #1/Plan Claude JSONL data insertion strategy #2: a smoke run, schema checks, and a section in the fine-tuning skill / flywheel arch doc.

Acceptance criteria

• Decision recorded on whether to use TRL replay-buffer GRPO or a custom off-policy loss
• grpo_training.jsonl (with captured token ids + logprobs) trains a model end-to-end
• Orchestrator can dispatch GRPO retrain; auto mode behavior defined
• Staleness/ratio-cap policy implemented and configurable
• Tests + docs updated; existing SFT/KTO retrain paths unchanged

References

• docs/plans/proxy-token-capture-grpo-feed-plan.md
• docs/plans/token-faithful-grpo-rollout-plan.md (Plan Fix KTO Dataset True-False Alternation Pattern #1 — the on-policy faithful rollout, already implemented)
• shared/flywheel/orchestrator.py _select_trainer
• shared/flywheel/stager.py _format_grpo_example
• NVIDIA POLAR — arXiv:2605.24220