docs(tutorials): add black-box coding agent training tutorial (OpenCode + AsyncGRPO)

docs/source/_toctree.yml

@@ -52,6 +52,8 @@
     title: Evaluating with Inspect AI
   - local: tutorials/browsergym-harness
     title: RL Training with an Agentic Harness
+  - local: tutorials/opencode-agent-grpo
+    title: Training a Real Coding Agent
   - local: tutorials/sft-warmup
     title: SFT Training with Environments
   title: Tutorials

docs/source/tutorials/index.md

@@ -26,5 +26,6 @@ Already familiar with the basics? These tutorials cover specific workflows in de
 | [RL Training with 2048](rl-training-2048.md) | Train a language model to play 2048 using GRPO. Covers game-state representation and reward shaping. | Yes | — |
 | [Evaluating agents with Inspect AI](evaluation-inspect.md) | Wrap an OpenEnv environment in an Inspect AI `Task`, run it via `InspectAIHarness`, and get a structured `EvalResult`. | No | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/OpenEnv/blob/main/examples/evaluation_inspect.ipynb) |
 | [BrowserGym Harness Rollouts](browsergym-harness.md) | Drive BrowserGym through the OpenEnv harness runtime when a trainer needs token sampling, logprobs, and reward assignment inside the training loop. | Yes | — |
+| [Training a Real Coding Agent](opencode-agent-grpo.md) | Train the actual OpenCode agent (black-box, loop-owning) with TRL's `AsyncGRPOTrainer`: a transparent proxy captures each turn's token ids and logprobs while the agent runs its own tool loop. | Yes | — |
 | [Collecting rollouts for supervised training](sft-warmup.md) | Run a teacher model to collect reward-labeled rollouts, filter them, and fine-tune a student with TRL's `SFTTrainer` as a warm-start for GRPO. | Yes | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/OpenEnv/blob/main/examples/sft_warmup.ipynb) |
 

docs/source/tutorials/opencode-agent-grpo.md

@@ -0,0 +1,84 @@
+# Training a Real Coding Agent with GRPO (OpenCode)
+
+This tutorial covers the black-box training path: training the actual
+[`opencode`](https://opencode.ai) coding agent, with its own planner, tools,
+context management, and stop condition, using TRL's experimental
+`AsyncGRPOTrainer`. The agent owns its loop, and OpenEnv captures what it did.
+
+> [!NOTE]
+> Three GRPO patterns, three tutorials. For a standard `reset()` / `step()`
+> flow where TRL drives the episode, see the
+> [Wordle GRPO tutorial](wordle-grpo.md). For harness rollouts where the
+> trainer still generates each turn (white-box), see the
+> [BrowserGym harness tutorial](browsergym-harness.md). Use this page when you
+> want to train a production agent as-is, without reimplementing its loop.
+
+## How It Works
+
+The full recipe lives in TRL. The moving pieces:
+
+1. Each rollout runs the agent inside an OpenEnv session created by
+   `OpenCodeSessionFactory` from
+   [`opencode_env`](https://github.com/huggingface/OpenEnv/tree/main/envs/opencode_env),
+   in `transparent_proxy` mode. A small proxy inside the sandbox forwards the
+   agent's `/v1/chat/completions` calls to your vLLM server and records each
+   turn's token ids and logprobs to a trace.
+2. When the agent stops, TRL's `HarnessRolloutWorker` reads the trace, rebuilds
+   the per-turn training rows from the recorded ids, and scores the final
+   workspace with the session's `verify()` method (a held-out verifier the
+   agent never sees).
+3. `AsyncGRPOTrainer` trains on those rows, propagating the rollout reward to
+   every trained token through the group-relative advantage. NCCL weight sync
+   keeps the vLLM server on the current policy, so the agent always samples
+   from the model being trained.
+
+Each rollout gets its own isolated session: one sandbox, one proxy port, one
+agent process. Three small functions adapt the recipe to your task:
+`rollout_reward_fn` (outcome to scalar reward), `train_turn_fn` (which turns
+receive gradient), and `agent_turn_fn` (which trace entries are real agent
+turns rather than auxiliary calls like title generation). All three are
+documented in
+[TRL's harness training guide](https://huggingface.co/docs/trl/openenv#training-on-harnesses-training-a-real-coding-agent-opencode).
+
+## Run The Recipe
+
+The reference script trains on competitive-coding problems from
+`agentica-org/DeepCoder-Preview-Dataset`. The agent writes `solution.py`, and
+the verifier runs it against held-out tests, returning the fraction passed.
+
+```bash
+pip install trl trackio datasets
+pip install "openenv-opencode-env @ git+https://github.com/huggingface/OpenEnv.git#subdirectory=envs/opencode_env"
+```
+
+Serve the policy with tool calling, token ids, and NCCL weight sync enabled
+(one GPU), then train (a second GPU):
+
+```bash
+# Terminal 1
+CUDA_VISIBLE_DEVICES=0 VLLM_SERVER_DEV_MODE=1 vllm serve Qwen/Qwen3-4B-Instruct-2507 \
+    --host 0.0.0.0 --port 8000 \
+    --enable-auto-tool-choice --tool-call-parser hermes \
+    --logprobs-mode processed_logprobs \
+    --return-tokens-as-token-ids \
+    --weight-transfer-config '{"backend":"nccl"}'
+
+# Terminal 2
+CUDA_VISIBLE_DEVICES=1 python examples/scripts/openenv/opencode.py \
+    --model Qwen/Qwen3-4B-Instruct-2507 --vllm-url http://localhost:8000
+```
+
+The script is self-contained and runs the agent in a local subprocess sandbox,
+so no container setup is needed. The recipe has been validated end to end on
+Qwen3 (see [huggingface/trl#6420](https://github.com/huggingface/trl/pull/6420)).
+
+## Full Recipe
+
+- [Training on harnesses](https://huggingface.co/docs/trl/openenv#training-on-harnesses-training-a-real-coding-agent-opencode)
+  in TRL's OpenEnv docs: rollout semantics, the reward path, turn selection,
+  and the trace contract.
+- [`examples/scripts/openenv/opencode.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/openenv/opencode.py)
+  in TRL: the complete, runnable script.
+- [`envs/opencode_env`](https://github.com/huggingface/OpenEnv/tree/main/envs/opencode_env):
+  the OpenEnv side, including the session factory, sandbox backends, and the
+  transparent interception proxy.