diff --git a/docs/source/_toctree.yml b/docs/source/_toctree.yml index e5d4f5807..71381e5a5 100644 --- a/docs/source/_toctree.yml +++ b/docs/source/_toctree.yml @@ -52,6 +52,8 @@ title: Evaluating with Inspect AI - local: tutorials/browsergym-harness title: RL Training with an Agentic Harness + - local: tutorials/opencode-agent-grpo + title: Training a Real Coding Agent - local: tutorials/sft-warmup title: SFT Training with Environments title: Tutorials diff --git a/docs/source/tutorials/index.md b/docs/source/tutorials/index.md index 98026486f..09e6959bf 100644 --- a/docs/source/tutorials/index.md +++ b/docs/source/tutorials/index.md @@ -26,5 +26,6 @@ Already familiar with the basics? These tutorials cover specific workflows in de | [RL Training with 2048](rl-training-2048.md) | Train a language model to play 2048 using GRPO. Covers game-state representation and reward shaping. | Yes | — | | [Evaluating agents with Inspect AI](evaluation-inspect.md) | Wrap an OpenEnv environment in an Inspect AI `Task`, run it via `InspectAIHarness`, and get a structured `EvalResult`. | No | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/OpenEnv/blob/main/examples/evaluation_inspect.ipynb) | | [BrowserGym Harness Rollouts](browsergym-harness.md) | Drive BrowserGym through the OpenEnv harness runtime when a trainer needs token sampling, logprobs, and reward assignment inside the training loop. | Yes | — | +| [Training a Real Coding Agent](opencode-agent-grpo.md) | Train the actual OpenCode agent (black-box, loop-owning) with TRL's `AsyncGRPOTrainer`: a transparent proxy captures each turn's token ids and logprobs while the agent runs its own tool loop. | Yes | — | | [Collecting rollouts for supervised training](sft-warmup.md) | Run a teacher model to collect reward-labeled rollouts, filter them, and fine-tune a student with TRL's `SFTTrainer` as a warm-start for GRPO. | Yes | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/OpenEnv/blob/main/examples/sft_warmup.ipynb) | diff --git a/docs/source/tutorials/opencode-agent-grpo.md b/docs/source/tutorials/opencode-agent-grpo.md new file mode 100644 index 000000000..f8ad51227 --- /dev/null +++ b/docs/source/tutorials/opencode-agent-grpo.md @@ -0,0 +1,84 @@ +# Training a Real Coding Agent with GRPO (OpenCode) + +This tutorial covers the black-box training path: training the actual +[`opencode`](https://opencode.ai) coding agent, with its own planner, tools, +context management, and stop condition, using TRL's experimental +`AsyncGRPOTrainer`. The agent owns its loop, and OpenEnv captures what it did. + +> [!NOTE] +> Three GRPO patterns, three tutorials. For a standard `reset()` / `step()` +> flow where TRL drives the episode, see the +> [Wordle GRPO tutorial](wordle-grpo.md). For harness rollouts where the +> trainer still generates each turn (white-box), see the +> [BrowserGym harness tutorial](browsergym-harness.md). Use this page when you +> want to train a production agent as-is, without reimplementing its loop. + +## How It Works + +The full recipe lives in TRL. The moving pieces: + +1. Each rollout runs the agent inside an OpenEnv session created by + `OpenCodeSessionFactory` from + [`opencode_env`](https://github.com/huggingface/OpenEnv/tree/main/envs/opencode_env), + in `transparent_proxy` mode. A small proxy inside the sandbox forwards the + agent's `/v1/chat/completions` calls to your vLLM server and records each + turn's token ids and logprobs to a trace. +2. When the agent stops, TRL's `HarnessRolloutWorker` reads the trace, rebuilds + the per-turn training rows from the recorded ids, and scores the final + workspace with the session's `verify()` method (a held-out verifier the + agent never sees). +3. `AsyncGRPOTrainer` trains on those rows, propagating the rollout reward to + every trained token through the group-relative advantage. NCCL weight sync + keeps the vLLM server on the current policy, so the agent always samples + from the model being trained. + +Each rollout gets its own isolated session: one sandbox, one proxy port, one +agent process. Three small functions adapt the recipe to your task: +`rollout_reward_fn` (outcome to scalar reward), `train_turn_fn` (which turns +receive gradient), and `agent_turn_fn` (which trace entries are real agent +turns rather than auxiliary calls like title generation). All three are +documented in +[TRL's harness training guide](https://huggingface.co/docs/trl/openenv#training-on-harnesses-training-a-real-coding-agent-opencode). + +## Run The Recipe + +The reference script trains on competitive-coding problems from +`agentica-org/DeepCoder-Preview-Dataset`. The agent writes `solution.py`, and +the verifier runs it against held-out tests, returning the fraction passed. + +```bash +pip install trl trackio datasets +pip install "openenv-opencode-env @ git+https://github.com/huggingface/OpenEnv.git#subdirectory=envs/opencode_env" +``` + +Serve the policy with tool calling, token ids, and NCCL weight sync enabled +(one GPU), then train (a second GPU): + +```bash +# Terminal 1 +CUDA_VISIBLE_DEVICES=0 VLLM_SERVER_DEV_MODE=1 vllm serve Qwen/Qwen3-4B-Instruct-2507 \ + --host 0.0.0.0 --port 8000 \ + --enable-auto-tool-choice --tool-call-parser hermes \ + --logprobs-mode processed_logprobs \ + --return-tokens-as-token-ids \ + --weight-transfer-config '{"backend":"nccl"}' + +# Terminal 2 +CUDA_VISIBLE_DEVICES=1 python examples/scripts/openenv/opencode.py \ + --model Qwen/Qwen3-4B-Instruct-2507 --vllm-url http://localhost:8000 +``` + +The script is self-contained and runs the agent in a local subprocess sandbox, +so no container setup is needed. The recipe has been validated end to end on +Qwen3 (see [huggingface/trl#6420](https://github.com/huggingface/trl/pull/6420)). + +## Full Recipe + +- [Training on harnesses](https://huggingface.co/docs/trl/openenv#training-on-harnesses-training-a-real-coding-agent-opencode) + in TRL's OpenEnv docs: rollout semantics, the reward path, turn selection, + and the trace contract. +- [`examples/scripts/openenv/opencode.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/openenv/opencode.py) + in TRL: the complete, runnable script. +- [`envs/opencode_env`](https://github.com/huggingface/OpenEnv/tree/main/envs/opencode_env): + the OpenEnv side, including the session factory, sandbox backends, and the + transparent interception proxy.