From 6666243a677e8b0e26881812480ae48a171afaa1 Mon Sep 17 00:00:00 2001 From: sergiopaniego Date: Fri, 31 Jul 2026 12:10:51 +0200 Subject: [PATCH 1/2] docs(tutorials): add black-box coding agent training tutorial (OpenCode + AsyncGRPO) --- docs/source/_toctree.yml | 2 + docs/source/tutorials/index.md | 1 + docs/source/tutorials/opencode-agent-grpo.md | 62 ++++++++++++++++++++ 3 files changed, 65 insertions(+) create mode 100644 docs/source/tutorials/opencode-agent-grpo.md diff --git a/docs/source/_toctree.yml b/docs/source/_toctree.yml index e5d4f5807..71381e5a5 100644 --- a/docs/source/_toctree.yml +++ b/docs/source/_toctree.yml @@ -52,6 +52,8 @@ title: Evaluating with Inspect AI - local: tutorials/browsergym-harness title: RL Training with an Agentic Harness + - local: tutorials/opencode-agent-grpo + title: Training a Real Coding Agent - local: tutorials/sft-warmup title: SFT Training with Environments title: Tutorials diff --git a/docs/source/tutorials/index.md b/docs/source/tutorials/index.md index 98026486f..09e6959bf 100644 --- a/docs/source/tutorials/index.md +++ b/docs/source/tutorials/index.md @@ -26,5 +26,6 @@ Already familiar with the basics? These tutorials cover specific workflows in de | [RL Training with 2048](rl-training-2048.md) | Train a language model to play 2048 using GRPO. Covers game-state representation and reward shaping. | Yes | — | | [Evaluating agents with Inspect AI](evaluation-inspect.md) | Wrap an OpenEnv environment in an Inspect AI `Task`, run it via `InspectAIHarness`, and get a structured `EvalResult`. | No | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/OpenEnv/blob/main/examples/evaluation_inspect.ipynb) | | [BrowserGym Harness Rollouts](browsergym-harness.md) | Drive BrowserGym through the OpenEnv harness runtime when a trainer needs token sampling, logprobs, and reward assignment inside the training loop. | Yes | — | +| [Training a Real Coding Agent](opencode-agent-grpo.md) | Train the actual OpenCode agent (black-box, loop-owning) with TRL's `AsyncGRPOTrainer`: a transparent proxy captures each turn's token ids and logprobs while the agent runs its own tool loop. | Yes | — | | [Collecting rollouts for supervised training](sft-warmup.md) | Run a teacher model to collect reward-labeled rollouts, filter them, and fine-tune a student with TRL's `SFTTrainer` as a warm-start for GRPO. | Yes | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/OpenEnv/blob/main/examples/sft_warmup.ipynb) | diff --git a/docs/source/tutorials/opencode-agent-grpo.md b/docs/source/tutorials/opencode-agent-grpo.md new file mode 100644 index 000000000..4d55a1d4c --- /dev/null +++ b/docs/source/tutorials/opencode-agent-grpo.md @@ -0,0 +1,62 @@ +# Training a Real Coding Agent with GRPO (OpenCode) + +This tutorial covers the black-box training path: training the actual +[`opencode`](https://opencode.ai) coding agent, with its own planner, tools, +context management, and stop condition, using TRL's experimental +`AsyncGRPOTrainer`. The agent owns its loop, and OpenEnv captures what it did. + +> [!NOTE] +> Three GRPO patterns, three tutorials. For a standard `reset()` / `step()` +> flow where TRL drives the episode, see the +> [Wordle GRPO tutorial](wordle-grpo.md). For harness rollouts where the +> trainer still generates each turn (white-box), see the +> [BrowserGym harness tutorial](browsergym-harness.md). Use this page when you +> want to train a production agent as-is, without reimplementing its loop. + +## How It Works + +The full recipe lives in TRL. The moving pieces: + +1. Each rollout runs the agent inside an OpenEnv session created by + `OpenCodeSessionFactory` from + [`opencode_env`](https://github.com/huggingface/OpenEnv/tree/main/envs/opencode_env), + in `transparent_proxy` mode. A small proxy inside the sandbox forwards the + agent's `/v1/chat/completions` calls to your vLLM server and records each + turn's token ids and logprobs to a trace. +2. When the agent stops, TRL's `HarnessRolloutWorker` reads the trace, rebuilds + the per-turn training rows from the recorded ids, and scores the final + workspace with the session's `verify()` method (a held-out verifier the + agent never sees). +3. `AsyncGRPOTrainer` trains on those rows, propagating the rollout reward to + every trained token through the group-relative advantage. NCCL weight sync + keeps the vLLM server on the current policy, so the agent always samples + from the model being trained. + +Each rollout gets its own isolated session: one sandbox, one proxy port, one +agent process. Three small functions adapt the recipe to your task: +`rollout_reward_fn` (outcome to scalar reward), `train_turn_fn` (which turns +receive gradient), and `agent_turn_fn` (which trace entries are real agent +turns rather than auxiliary calls like title generation). All three are +documented in +[TRL's harness training guide](https://huggingface.co/docs/trl/openenv#training-on-harnesses-training-a-real-coding-agent-opencode). + +## Full Recipe + +The reference script trains on competitive-coding problems from +`agentica-org/DeepCoder-Preview-Dataset`: the agent writes `solution.py`, and +the verifier runs it against held-out tests, returning the fraction passed. It +is self-contained, runs the agent in a local subprocess sandbox (no container +setup needed), needs two GPUs (one serving the policy with vLLM, one +training), and has been validated end to end on Qwen3 (see +[huggingface/trl#6420](https://github.com/huggingface/trl/pull/6420)). +Installation, the exact vLLM serving flags, and the run commands live next to +the recipe in TRL: + +- [Training on harnesses](https://huggingface.co/docs/trl/openenv#training-on-harnesses-training-a-real-coding-agent-opencode) + in TRL's OpenEnv docs: rollout semantics, the reward path, turn selection, + and the trace contract. +- [`examples/scripts/openenv/opencode.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/openenv/opencode.py) + in TRL: the complete, runnable script. +- [`envs/opencode_env`](https://github.com/huggingface/OpenEnv/tree/main/envs/opencode_env): + the OpenEnv side, including the session factory, sandbox backends, and the + transparent interception proxy. From 5702e24447864842827fc7e36851e9e95e0de05c Mon Sep 17 00:00:00 2001 From: sergiopaniego Date: Fri, 31 Jul 2026 12:26:40 +0200 Subject: [PATCH 2/2] docs(tutorials): name TRL in the coding agent tutorial title --- docs/source/_toctree.yml | 2 +- docs/source/tutorials/opencode-agent-grpo.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/source/_toctree.yml b/docs/source/_toctree.yml index 71381e5a5..9337b0831 100644 --- a/docs/source/_toctree.yml +++ b/docs/source/_toctree.yml @@ -53,7 +53,7 @@ - local: tutorials/browsergym-harness title: RL Training with an Agentic Harness - local: tutorials/opencode-agent-grpo - title: Training a Real Coding Agent + title: Coding Agent Training with TRL - local: tutorials/sft-warmup title: SFT Training with Environments title: Tutorials diff --git a/docs/source/tutorials/opencode-agent-grpo.md b/docs/source/tutorials/opencode-agent-grpo.md index 4d55a1d4c..ce92d7774 100644 --- a/docs/source/tutorials/opencode-agent-grpo.md +++ b/docs/source/tutorials/opencode-agent-grpo.md @@ -1,4 +1,4 @@ -# Training a Real Coding Agent with GRPO (OpenCode) +# Coding Agent Training with TRL (OpenCode) This tutorial covers the black-box training path: training the actual [`opencode`](https://opencode.ai) coding agent, with its own planner, tools,