RFC: GRPO recipes

[burtenshaw]

This PR adds an RFC for framework-specific GRPO harness training recipes.

[sergiopaniego]

Hey, coming back to this one. I've opened #1028 and I'm leaving some comments here (iterated with CC) so we can iterate and merge.

The big change since this was filed is that the reference implementation already landed on the TRL side:

• trl/experimental/async_grpo ships the black-box (loop-owning) path: AsyncGRPOTrainer plus a HarnessRolloutWorker for OpenEnv session factories.
• examples/scripts/openenv/opencode.py is a complete runnable recipe: the real OpenCode agent in the loop, transparent proxy capturing per-turn token ids and logprobs, held-out verifier reward on DeepCoder tasks, vLLM weight sync, explicit hardware and serving requirements.
• And it is documented in prose: TRL's OpenEnv docs have a full "Training on harnesses" section covering the rollout semantics, the reward path (rollout_reward_fn), turn selection (train_turn_fn, agent_turn_fn) and the trace contract.
• On the OpenEnv side the supporting pieces are merged (opencode_env envs/opencode_env: OpenCode coding-agent harness environment #656, HF sandbox backend with the interception proxy Add HFSandboxBackend for the OpenCode harness #998), pi_env is coming as a second harness in Add pi_env: Pi coding-agent environment (HF sandbox + transparent-proxy logprobs) #999, and RFC 006: Agentic RL through Harness Interception — token-faithful traces for TRL #941 formalizes the interception contract.

I checked it against the "What The Recipe Must Specify" list and it covers every bullet, with one caveat about where the validation evidence lives (more below). So I'd say the two open questions are answered de facto: OpenCode is the first reference harness (Pi next), and coding tasks with a held-out verifier are the smoke and representative task.

With that, a few suggestions to get this merged:

1. Keep the recipe contract (the checklist) in this RFC, and add the template header (Status / Created / Authors / RFC ID) so it matches the other RFCs.
2. Let recipes live where they already live: each framework maintains its own in its repo, in its own idioms, and OpenEnv's deliverable is a curated index. We already do exactly this in docs/source/tutorials/index.md, which lists TRL, Unsloth and Inspect AI workflows and even links to a notebook hosted in the TRL repo. What was missing is one row: the black-box OpenCode + AsyncGRPO recipe was not discoverable from OpenEnv at all (the BrowserGym harness tutorial covers the earlier pattern where the trainer keeps sampling). That's what docs(tutorials): add black-box coding agent training tutorial (OpenCode + AsyncGRPO) #1028 adds.
3. Make the validation bar explicit with two statuses: runnable (the script runs end to end) and validated (a training run with metrics, before/after evals or a checkpoint). The OpenCode recipe has in fact been validated end to end on Qwen3 (Async grpo OpenEnv harness rollout trl#6420, results thread), but that evidence lives in a PR body and an X thread today, while the Wordle recipe keeps a results section next to the recipe itself. So the bar should also say where the evidence gets captured, and a status column in the index implements all of this with zero extra machinery.

One last thing worth stating in the text: the "no trajectory serialization format yet" non-goal points at #941, so this RFC effectively layers on top of the interception contract there. A one-line cross-reference would help the two merge in a sane order.