docs(tutorials): add black-box coding agent training tutorial (OpenCode + AsyncGRPO)

docs/source/_toctree.yml

@@ -52,6 +52,8 @@
     title: Evaluating with Inspect AI
   - local: tutorials/browsergym-harness
     title: RL Training with an Agentic Harness
+  - local: tutorials/opencode-agent-grpo
+    title: Coding Agent Training with TRL
   - local: tutorials/sft-warmup
     title: SFT Training with Environments
   title: Tutorials

docs/source/tutorials/index.md

@@ -26,5 +26,6 @@ Already familiar with the basics? These tutorials cover specific workflows in de
 | [RL Training with 2048](rl-training-2048.md) | Train a language model to play 2048 using GRPO. Covers game-state representation and reward shaping. | Yes | — |
 | [Evaluating agents with Inspect AI](evaluation-inspect.md) | Wrap an OpenEnv environment in an Inspect AI `Task`, run it via `InspectAIHarness`, and get a structured `EvalResult`. | No | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/OpenEnv/blob/main/examples/evaluation_inspect.ipynb) |
 | [BrowserGym Harness Rollouts](browsergym-harness.md) | Drive BrowserGym through the OpenEnv harness runtime when a trainer needs token sampling, logprobs, and reward assignment inside the training loop. | Yes | — |
+| [Training a Real Coding Agent](opencode-agent-grpo.md) | Train the actual OpenCode agent (black-box, loop-owning) with TRL's `AsyncGRPOTrainer`: a transparent proxy captures each turn's token ids and logprobs while the agent runs its own tool loop. | Yes | — |
 | [Collecting rollouts for supervised training](sft-warmup.md) | Run a teacher model to collect reward-labeled rollouts, filter them, and fine-tune a student with TRL's `SFTTrainer` as a warm-start for GRPO. | Yes | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/OpenEnv/blob/main/examples/sft_warmup.ipynb) |
 

docs/source/tutorials/opencode-agent-grpo.md

@@ -0,0 +1,62 @@
+# Coding Agent Training with TRL (OpenCode)
+
+This tutorial covers the black-box training path: training the actual
+[`opencode`](https://opencode.ai) coding agent, with its own planner, tools,
+context management, and stop condition, using TRL's experimental
+`AsyncGRPOTrainer`. The agent owns its loop, and OpenEnv captures what it did.
+
+> [!NOTE]
+> Three GRPO patterns, three tutorials. For a standard `reset()` / `step()`
+> flow where TRL drives the episode, see the
+> [Wordle GRPO tutorial](wordle-grpo.md). For harness rollouts where the
+> trainer still generates each turn (white-box), see the
+> [BrowserGym harness tutorial](browsergym-harness.md). Use this page when you
+> want to train a production agent as-is, without reimplementing its loop.
+
+## How It Works
+
+The full recipe lives in TRL. The moving pieces:
+
+1. Each rollout runs the agent inside an OpenEnv session created by
+   `OpenCodeSessionFactory` from
+   [`opencode_env`](https://github.com/huggingface/OpenEnv/tree/main/envs/opencode_env),
+   in `transparent_proxy` mode. A small proxy inside the sandbox forwards the
+   agent's `/v1/chat/completions` calls to your vLLM server and records each
+   turn's token ids and logprobs to a trace.
+2. When the agent stops, TRL's `HarnessRolloutWorker` reads the trace, rebuilds
+   the per-turn training rows from the recorded ids, and scores the final
+   workspace with the session's `verify()` method (a held-out verifier the
+   agent never sees).
+3. `AsyncGRPOTrainer` trains on those rows, propagating the rollout reward to
+   every trained token through the group-relative advantage. NCCL weight sync
+   keeps the vLLM server on the current policy, so the agent always samples
+   from the model being trained.
+
+Each rollout gets its own isolated session: one sandbox, one proxy port, one
+agent process. Three small functions adapt the recipe to your task:
+`rollout_reward_fn` (outcome to scalar reward), `train_turn_fn` (which turns
+receive gradient), and `agent_turn_fn` (which trace entries are real agent
+turns rather than auxiliary calls like title generation). All three are
+documented in
+[TRL's harness training guide](https://huggingface.co/docs/trl/openenv#training-on-harnesses-training-a-real-coding-agent-opencode).
+
+## Full Recipe
+
+The reference script trains on competitive-coding problems from
+`agentica-org/DeepCoder-Preview-Dataset`: the agent writes `solution.py`, and
+the verifier runs it against held-out tests, returning the fraction passed. It
+is self-contained, runs the agent in a local subprocess sandbox (no container
+setup needed), needs two GPUs (one serving the policy with vLLM, one
+training), and has been validated end to end on Qwen3 (see
+[huggingface/trl#6420](https://github.com/huggingface/trl/pull/6420)).
+Installation, the exact vLLM serving flags, and the run commands live next to
+the recipe in TRL:
+
+- [Training on harnesses](https://huggingface.co/docs/trl/openenv#training-on-harnesses-training-a-real-coding-agent-opencode)
+  in TRL's OpenEnv docs: rollout semantics, the reward path, turn selection,
+  and the trace contract.
+- [`examples/scripts/openenv/opencode.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/openenv/opencode.py)
+  in TRL: the complete, runnable script.
+- [`envs/opencode_env`](https://github.com/huggingface/OpenEnv/tree/main/envs/opencode_env):
+  the OpenEnv side, including the session factory, sandbox backends, and the
+  transparent interception proxy.