docs(tutorials): add black-box coding agent training tutorial (OpenCode + AsyncGRPO) - #1027
docs(tutorials): add black-box coding agent training tutorial (OpenCode + AsyncGRPO)#1027sergiopaniego wants to merge 1 commit into
Conversation
burtenshaw
added
documentation
Cursor
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Alignment Review Report
Two-tier alignment-review of this docs-only PR (3 files, +87): a new OpenCode + AsyncGRPO tutorial plus its _toctree.yml / index.md registration.
Automated Checks
- Lint: PASS (N/A to this diff). The PR changes only Markdown/YAML (0 Python files).
bash .claude/hooks/lint.sh(ruff/usort, Python-only) exits non-zero solely on ~21 pre-existingenvs/**/*.pyreformat candidates (chat_env, coding_tools_env, opencode_env, repl_env, …) — none are in this diff. (uvisn't preinstalled on the review VM; I installed it to run the hook.) - Debug code: CLEAN.
check-debug.shscans onlysrc/and reports pre-existingconsole.print/TODOitems incli/…, unrelated to this PR. The new docs contain none.
Open RFCs Context
- RFC 005 — Agentic Harness Integration (In Review): directly relevant; this tutorial teaches the harness / black-box training path. See RFC flag below.
- RFC 010 — ECHO env-token world modeling (Draft): tangentially related (token-role masking / "which turns receive gradient"), but not in conflict — the tutorial uses standard agent-token masking. Noted for context only; cc @thegovind if relevant.
- RFCs 000–004 are also "In Review" but foundational; no interaction with a docs-only tutorial.
Tier 1: Fixes Required
None. The page and its registration are well-formed and technically accurate:
_toctree.yml+tutorials/index.mdare updated consistently (same title/link, placed between BrowserGym and SFT).> [!NOTE]admonition and.mdcross-links match existing tutorial conventions.- OpenEnv-side claims verified against the code:
OpenCodeSessionFactory+mode="transparent_proxy"(envs/opencode_env/harness.py); the in-sandbox proxy capturing per-turncompletion_token_ids+per_token_logps(fetch_proxy_trace,sandbox/interception.py); the held-outOpenCodeSession.verify()(list_tools()returns[]); the pip packageopenenv-opencode-env+ subdirectory; and the white-box/black-box framing (run_white_box/run_black_box).
Tier 2: Alignment Discussion
Principle Conflicts
None identified — the taught design is aligned:
- Rewards in environment: reward is the session's held-out
verify();openenv.core.harnessenforces thatverifyonly forwards the env-produced reward (_resolve_env_rewardraises otherwise). - Agent isolation / agents-cannot-reset:
reset/step/state/closeare reserved and never exposed to the harness; the agent owns only its own tool loop.
RFC Conflicts
ALIGNMENT FLAG: Tutorial publishes an experimental, RFC-005-pending harness API as a first-class recipe
- Principle/RFC at stake: RFC 005 (Agentic Harness Integration) — In Review
- The concern: the page documents the shipped
openenv.core.harnessruntime (ResourceSession/ResourceSessionFactory+run_white_box/run_black_box, used byOpenCodeSessionFactory). That module's own docstring says it lives "outside the stableopenenv.corepackage surface while RFC 005 is still under review", and the shipped design has diverged from RFC 005's proposedHarnessConfig+HarnessAdapter(start/stop/inject_tools/send_message) +HarnessEnvironment(MCPEnvironment)per-turn-step()model. This PR does not introduce the divergence (it documents already-merged code), but two things merit a maintainer decision: (1) reconcile RFC 005 with the shippedResourceSession/ white-box / black-box design (the RFC reads stale); (2) optionally add a one-line "experimental / unstable API" caveat for the OpenEnv side — the page already flags TRL'sAsyncGRPOTraineras experimental, but notopenenv.core.harness. - Suggested reviewer: @Darktex (RFC 005 author); cc @sergiopaniego (PR author, also a committer on RFC 005)
Summary
- 0 mechanical issues to fix
- 1 alignment point for human review (RFC 005 reconciliation + optional experimental-API caveat)
- 0 hard RFC conflicts (RFC 010 noted as non-conflicting context)
Overall a clean, accurate, well-scoped docs PR. The only discussion item is that it surfaces an RFC-005-pending experimental API as a first-class tutorial — worth a maintainer note, not blocking.
Sent by Cursor Automation: Pre-review
| The full recipe lives in TRL. The moving pieces: | ||
|
|
||
| 1. Each rollout runs the agent inside an OpenEnv session created by | ||
| `OpenCodeSessionFactory` from |
There was a problem hiding this comment.
ALIGNMENT FLAG (Tier 2, non-blocking) — RFC 005 (Agentic Harness Integration, In Review).
This page documents the shipped openenv.core.harness runtime: OpenCodeSessionFactory → ResourceSession → run_white_box/run_black_box → verify(). That module's own docstring notes it lives "outside the stable openenv.core package surface while RFC 005 is still under review," and the shipped design has diverged from RFC 005's proposed HarnessConfig + HarnessAdapter (start/stop/inject_tools/send_message) + HarnessEnvironment(MCPEnvironment) per-turn-step() abstraction.
Not introduced by this PR (it documents already-merged code), but worth: (a) reconciling RFC 005 with the shipped ResourceSession design, and (b) optionally adding a one-line "experimental API" caveat here — the page already flags TRL's AsyncGRPOTrainer as experimental, but not the OpenEnv side. cc @Darktex (RFC 005 author), @sergiopaniego.
| turn's token ids and logprobs to a trace. | ||
| 2. When the agent stops, TRL's `HarnessRolloutWorker` reads the trace, rebuilds | ||
| the per-turn training rows from the recorded ids, and scores the final | ||
| workspace with the session's `verify()` method (a held-out verifier the |
There was a problem hiding this comment.
Well-aligned with the rewards-in-environment invariant: reward comes from the session's held-out verify(), and openenv.core.harness enforces it — VerifyResult.env_reward "must forward reward already produced inside the environment ... must not synthesize a new reward in the orchestration layer" and _resolve_env_reward raises if verify returns a mismatched reward. The agent also sees no orchestration tools (OpenCodeSession.list_tools() returns []). Nice.
Summary
Adds a tutorial page for the black-box (loop-owning) training path: training the real OpenCode agent with TRL's experimental
AsyncGRPOTrainerand theopencode_envtransparent interception proxy. The recipe itself lives in TRL (docs section plusexamples/scripts/openenv/opencode.py), so this page explains when to use the pattern, how the pieces fit, and links out. It also adds the row to the tutorials index and the toctree entry.Until now this recipe was not discoverable from OpenEnv docs at all: the BrowserGym harness tutorial covers the white-box pattern where the trainer keeps sampling, and nothing pointed at the black-box path.
This is the index/discoverability piece discussed in RFC #924 (GRPO harness training recipes): recipes live in each framework's repo, OpenEnv curates the index.
Type of Change
Alignment Checklist
.claude/docs/PRINCIPLES.mdand this PR aligns with our principles.claude/docs/INVARIANTS.mdand no invariants are violated/pre-submit-pr(orbash .claude/hooks/lint.shand tests) and addressed all issuesRFC Status
Test Plan
_toctree.ymlparses (yaml.safe_load) and the new entry mirrors the existing tutorial entries.docs/trl/openenvharness section,examples/scripts/openenv/opencode.py) andenvs/opencode_env, all verified to exist.Claude Code Review
N/A (docs-only)
Note
Low Risk
Documentation-only changes with no runtime or API impact.
Overview
Adds discoverability for the black-box GRPO path where the production OpenCode agent keeps its own tool loop and TRL trains from captured traces.
A new tutorial page (
opencode-agent-grpo.md) explains when to use this pattern versus Wordle-stylereset()/step()GRPO and the white-box BrowserGym harness, walks through the flow (OpenCodeSessionFactorytransparent proxy → trace →HarnessRolloutWorker→AsyncGRPOTrainerwith NCCL weight sync), and points to TRL’s runnable recipe plusopencode_env. The tutorials overview table and_toctree.ymlgain a Training a Real Coding Agent entry (GPU required, no Colab notebook).Reviewed by Cursor Bugbot for commit 98d20fb. Bugbot is set up for automated code reviews on this repo. Configure here.