docs(tutorials): add black-box coding agent training tutorial (OpenCode + AsyncGRPO) - #1028
docs(tutorials): add black-box coding agent training tutorial (OpenCode + AsyncGRPO)#1028sergiopaniego wants to merge 2 commits into
Conversation
burtenshaw
added
documentation
Cursor
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Alignment Review Report
Scope: Docs-only PR — 3 files, +65 lines (docs/source/_toctree.yml, docs/source/tutorials/index.md, new docs/source/tutorials/opencode-agent-grpo.md). No Python/source changes.
Automated Checks
- Lint: N/A for this PR (no
.pyfiles changed). For reference I ran.claude/hooks/lint.shafter installinguv; it fails only on pre-existingruff formatdrift in ~20envs/*Python files (e.g.envs/opencode_env/*.py) that this PR does not touch — consistent with the baseline drift noted inAGENTS.md. (Side note, unrelated to this PR: the hook also reformats python code blocks insideenvs/**/README.mdbut only reverts.py, so it leaves.mdfiles dirty.) - Debug code: CLEAN for this PR.
.claude/hooks/check-debug.shlists manyprint()s and 3 TODOs, but all are pre-existing insrc/; none are in the changed docs files.
Doc-specific checks (all pass)
_toctree.ymlentrytutorials/opencode-agent-grporesolves to the new file. ✓- Internal links
wordle-grpo.mdandbrowsergym-harness.mdexist. ✓ - API references are accurate:
OpenCodeSessionFactory,transparent_proxymode, and sessionverify()all exist inenvs/opencode_env/harness.py, and the proxy description (/v1/chat/completionsinterception, per-turn ids + logprobs) matchesenvs/opencode_env/README.md. ✓ > [!NOTE]alert syntax matches the established convention across the other tutorials. ✓
Open RFCs Context
All RFCs are In Review except RFC 010 (Draft). The directly relevant one is RFC 005 — Agentic Harness Integration (In Review, @Darktex), which is exactly the area this tutorial documents (training a real coding-agent harness).
Tier 1: Fixes Required
None. No mechanical, lint, debug, or broken-reference issues in the changed files.
Tier 2: Alignment Discussion
Principle Conflicts
None identified. Notably the described reward path aligns with rewards inside environment (PRINCIPLES.md / RFC 004): the domain scoring lives in the session's held-out verify() inside the env, which "the agent never sees" — consistent with the reward-boundary and agents cannot reset invariants. Training the production agent as-is also supports minimize lifecycle deltas and minimize human-agent divergence.
RFC Conflicts
ALIGNMENT FLAG: Tutorial canonizes a harness-training design that diverges from the abstraction proposed in RFC 005 (In Review)
- Principle/RFC at stake: RFC 005 — Agentic Harness Integration (Status: In Review)
- The concern: RFC 005 proposes harness training via a
HarnessEnvironment(extendsMCPEnvironment) where eachstep()is one conversational turn the orchestrator drives, plus aHarnessAdapter,HarnessEvent/HarnessResponsetrajectory, and aRubriccomputing reward after each turn. This tutorial instead documents the shippedopencode_env+ TRL path: the agent owns its entire loop, atransparent_proxyintercepts LLM calls to capture token ids/logprobs, TRL'sHarnessRolloutWorkerrebuilds training rows from the trace, scoring uses the session'sverify(), and adaptation happens viarollout_reward_fn/train_turn_fn/agent_turn_fn. RFC 005 explicitly rejected the "eachstep()is the entire episode" model that this black-box path effectively uses. The divergence is not introduced by this PR (it already exists inopencode_env+ TRL) — the PR just makes it canonical in user-facing docs. Per the two-tier model, flagging so the team can reconcile: most likely by updating RFC 005 to reflect the shipped Session / proxy /HarnessRolloutWorkerdesign (or documenting both paths side by side). Non-blocking. - Suggested reviewer: @Darktex (RFC 005 author; also authored the reward-boundary lines in PRINCIPLES.md/INVARIANTS.md). cc @burtenshaw (tutorials/docs).
Summary
- 0 mechanical issues to fix (Tier 1)
- 0 principle conflicts
- 1 RFC conflict to discuss (RFC 005 divergence — non-blocking; docs describe shipped reality, RFC likely needs updating)
Overall a clear, well-structured tutorial that accurately reflects the opencode_env implementation and is nicely positioned against the other two GRPO tutorials. The only item for humans is the RFC 005 reconciliation above.
Sent by Cursor Automation: Pre-review
| This tutorial covers the black-box training path: training the actual | ||
| [`opencode`](https://opencode.ai) coding agent, with its own planner, tools, | ||
| context management, and stop condition, using TRL's experimental | ||
| `AsyncGRPOTrainer`. The agent owns its loop, and OpenEnv captures what it did. |
There was a problem hiding this comment.
ALIGNMENT FLAG (Tier 2, non-blocking) — RFC 005: Agentic Harness Integration (In Review)
This "agent owns its loop" black-box model diverges from RFC 005's proposed abstraction, where harness training goes through a HarnessEnvironment with per-turn step() + HarnessAdapter + Rubric. RFC 005 explicitly rejected the "one step() = whole episode" model that this path effectively uses.
The divergence already exists in the shipped opencode_env + TRL code — this tutorial just makes it canonical in the docs. Worth reconciling: probably update RFC 005 to reflect the shipped transparent_proxy / HarnessRolloutWorker / session-verify() design (or document both paths). cc @Darktex (RFC 005 author).
Summary
Adds a tutorial page for the black-box (loop-owning) training path: training the real OpenCode agent with TRL's experimental
AsyncGRPOTrainerand theopencode_envtransparent interception proxy. The recipe itself lives in TRL (docs section plusexamples/scripts/openenv/opencode.py), so this page explains when to use the pattern, how the pieces fit, and links out. It also adds the row to the tutorials index and the toctree entry.Until now this recipe was not discoverable from OpenEnv docs at all: the BrowserGym harness tutorial covers the white-box pattern where the trainer keeps sampling, and nothing pointed at the black-box path.
This is the index/discoverability piece discussed in RFC #924 (GRPO harness training recipes): recipes live in each framework's repo, OpenEnv curates the index.
Type of Change
Alignment Checklist
.claude/docs/PRINCIPLES.mdand this PR aligns with our principles.claude/docs/INVARIANTS.mdand no invariants are violated/pre-submit-pr(orbash .claude/hooks/lint.shand tests) and addressed all issuesRFC Status
Test Plan
_toctree.ymlparses (yaml.safe_load) and the new entry mirrors the existing tutorial entries.docs/trl/openenvharness section,examples/scripts/openenv/opencode.py) andenvs/opencode_env, all verified to exist.Claude Code Review
N/A (docs-only)
Note
Low Risk
Documentation-only changes with no runtime or API impact.
Overview
Adds discoverability for training the real OpenCode agent with TRL’s experimental
AsyncGRPOTrainer—the black-box path where the agent owns its loop and OpenEnv records token ids/logprobs via a transparent proxy.New page
opencode-agent-grpo.mdexplains when to use this pattern vs Wordle (TRL-drivenreset/step) and BrowserGym (white-box harness), walks through session factory → trace →HarnessRolloutWorker→ GRPO, and links to TRL’s harness docs,opencode.pyscript, andopencode_env. The tutorials index table and_toctree.ymlget a matching entry (“Coding Agent Training with TRL”).Reviewed by Cursor Bugbot for commit 5702e24. Bugbot is set up for automated code reviews on this repo. Configure here.