Skip to content

Commit 72aa537

Browse files
abrichrclaude
abrichr
and
claude
authored
feat: add GiGPO anchor state computation to WAADesktopEnv (#122)
* feat: add GiGPO anchor state computation to WAADesktopEnv Add compute_anchor_state() function that produces a state key for GiGPO cross-rollout grouping. Uses a11y tree SHA256 hash (primary) with screenshot MD5 fallback. The state_key is included in the info dict from both reset() and step() so VAGEN/verl can use it for O(1) anchor grouping instead of recomputing perceptual hashes. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: clarify VAGEN vs verl-agent distinction in decision doc Add dated addendum (2026-03-16) correcting the earlier conflation of VAGEN and verl-agent as a single project. Key findings: VAGEN-Lite dropped Bi-Level GAE (only vanilla GRPO/PPO), GiGPO lives exclusively in verl-agent which uses its own env_base.py interface (not GymImageEnv), and our train_verl_e2e.py targets the wrong entry point. Outlines a corrected two-phase path: standalone GRPO first, then direct verl-agent integration if per-step credit is needed. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add comprehensive GRPO training research report Covers desktop RL landscape (30+ projects), per-step credit assignment alternatives (HCAPO recommended over GiGPO), scaling architectures (ComputerRL, DART-GUI), and synthetic environment feasibility (GUI-Genesis). Includes revised architecture recommendation: standalone GRPO + HCAPO first, then dense rewards + API-GUI hybrid, then async scaling. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: correct prioritization — validate GRPO before optimizing training math HCAPO and per-step credit are Phase 3 optimizations, not Phase 1. The bottleneck is rollout success rate (getting non-zero rewards), not loss computation. Dense partial-credit rewards and API-GUI hybrid actions directly increase gradient signal and should come first. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent f71c81f commit 72aa537

3 files changed

Lines changed: 441 additions & 0 deletions

File tree

‎docs/grpo_training_research_2026_03_17.md‎

Lines changed: 291 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,291 @@
1+
# GRPO Training Infrastructure: Comprehensive Research Report
2+
3+
**Date**: 2026-03-17
4+
**Scope**: Desktop RL training landscape, per-step credit assignment alternatives, scaling architectures, synthetic environment feasibility
5+
6+
---
7+
8+
## 1. Executive Summary
9+
10+
We evaluated 30+ open-source projects for desktop GUI RL training. Key findings:
11+
12+
1. **HCAPO is the recommended per-step credit method** — ~80-120 lines to add to our standalone GRPO trainer, no extra model, 8.3% compute overhead, no anchor states needed. Beats GiGPO on benchmarks.
13+
2. **VAGEN and verl-agent are separate projects** — our `train_verl_e2e.py` targets the wrong entry point. GiGPO lives in verl-agent, not VAGEN-Lite.
14+
3. **ComputerRL's API-GUI hybrid** and **DART-GUI's entropy filtering** are the highest-impact ideas to incorporate, without adopting either framework.
15+
4. **GUI-Genesis is not applicable** to Windows desktop tasks (mobile web only), but the dense reward concept is valuable.
16+
5. **Our WAADesktopEnv is the only OSS project wrapping WAA for RL training.**
17+
18+
---
19+
20+
## 2. VAGEN vs verl-agent Distinction
21+
22+
**Critical finding**: Our decision doc conflated two separate projects.
23+
24+
| | VAGEN (mll-lab-nu/VAGEN) | verl-agent (langfengQ/verl-agent) |
25+
|---|---|---|
26+
| **Purpose** | Environment framework | Training framework |
27+
| **Credit assignment** | Dropped Bi-Level GAE in Lite | **GiGPO** (what we want) |
28+
| **Env interface** | `GymImageEnv` (what we built) | Own `env_base.py` (different) |
29+
| **Desktop GUI env** | No | No |
30+
| **Entry point** | `vagen.main_ppo` | Own training scripts |
31+
| **Stars** | 431 | 1,700 |
32+
33+
**Impact**: Our `train_verl_e2e.py` targets `vagen.main_ppo` — wrong entry point for GiGPO. VAGEN-Lite only provides vanilla GRPO/PPO, which our standalone trainer already does.
34+
35+
**Corrected path**:
36+
- Phase 1: Standalone GRPO trainer (shipped in PR #55)
37+
- Phase 2: If per-step credit needed, consider HCAPO (standalone) before verl-agent (complex)
38+
39+
---
40+
41+
## 3. Per-Step Credit Assignment Methods
42+
43+
### 3.1 Comparison Table
44+
45+
| Criterion | GiGPO | **HCAPO** | iStar | HiPER |
46+
|-----------|-------|-----------|-------|-------|
47+
| **Can add to standalone trainer?** | No (needs verl-agent) | **YES** | Technically, but 2x memory | No (needs verl-agent) |
48+
| **Lines of code** | ~250 adapter + verl stack | **~80-120** | ~150-200 + 2nd model | 500+ + critic |
49+
| **Extra model in memory?** | No | **No** | YES (full policy copy) | YES (critic heads) |
50+
| **Extra compute per step** | Anchor hashing | **8.3% (1 fwd pass/traj)** | ~100% (PRM update) | ~50% (critic + GAE) |
51+
| **Anchor state grouping?** | YES (core limitation) | **No** | No | No |
52+
| **Works with VLMs?** | Yes | Untested (likely yes) | Yes (tested VL-7B) | Untested |
53+
| **Works with binary rewards?** | Yes | **Yes** | Yes | Yes |
54+
| **ALFWorld** | 90.8% | **91.4% (96.9% w/ smoothing)** | N/A | 97.4% |
55+
| **WebShop** | 75.2% | 73.8% | **86.5%** | 83.3% |
56+
| **Public code?** | Yes (verl-agent) | No | No | Yes (verl-agent fork) |
57+
58+
### 3.2 HCAPO: Recommended Approach
59+
60+
**How it works**: Uses the policy LLM itself as a post-hoc critic. After collecting a rollout, it scores each intermediate action conditioned on the final state (hindsight probability). This produces an importance weight for credit assignment.
61+
62+
**Key advantages over GiGPO for desktop tasks**:
63+
- **No anchor state problem**: GiGPO needs identical intermediate states across rollouts. Real desktop screenshots are almost never pixel-identical (mouse position, timing, anti-aliasing). GiGPO's step-level signal would likely be near-zero for WAA.
64+
- **No extra model**: Uses the same policy LLM — one extra non-autoregressive forward pass per trajectory.
65+
- **Do-no-harm mask**: Never penalizes actions in successful trajectories.
66+
- **Temporal smoothing**: Distributes credit to preparatory actions.
67+
68+
**Implementation plan** (~80-120 lines in openadapt-ml):
69+
1. `compute_hindsight_probabilities(model, processor, rollout, final_screenshot)` — ~40 lines
70+
2. `compute_hcapo_advantages(rollouts, hindsight_probs)` — ~30 lines
71+
3. Modifications to `_training_step()` — ~10 lines
72+
4. Config fields: `hcapo_enabled`, `hcapo_omega`, `hcapo_clip_min/max`, `hcapo_temp`, `hcapo_smoothing_alpha`
73+
74+
**Hyperparameters** (robust across benchmarks per authors): omega=1.0, C_min=0.8, C_max=1.2, T_temp=5.0, alpha=0.5.
75+
76+
### 3.3 iStar: Second Choice
77+
78+
Best absolute results (86.5% WebShop) and only method tested with VLMs on visual tasks. But requires **2x GPU memory** (separate PRM model), which is a dealbreaker on A10G (24GB). Only viable on larger GPUs (L40S 48GB+).
79+
80+
### 3.4 GiGPO: Deprioritized
81+
82+
The anchor state matching problem is fundamental for desktop GUI. Screenshots vary in mouse position, animation frames, and rendering artifacts even when the UI state is semantically identical. Our `compute_anchor_state()` (a11y tree hash) partially addresses this, but a11y trees are unreliable on WAA (UIA backend can't always find window elements during transitions).
83+
84+
---
85+
86+
## 4. Scaling Architectures: ComputerRL and DART-GUI
87+
88+
### 4.1 ComputerRL (Zhipu AI)
89+
90+
**Score**: 48.9% OSWorld (open-source SOTA)
91+
**Architecture**: gRPC-based distributed cluster, 1000+ concurrent qemu-in-docker Ubuntu VMs
92+
**License**: Apache 2.0
93+
94+
**Most valuable ideas**:
95+
96+
1. **API-GUI Hybrid Paradigm** — Agent can issue either GUI actions OR programmatic API calls. 103 auto-constructed APIs across LibreOffice Calc (27), Impress (22), Writer (19), Chrome (11), VS Code (12), VLC (12). Yielded **134% improvement** over GUI-only and 3-3.6x faster execution. **Directly applicable to WAA** via `/execute_windows` endpoint with `win32com`/PowerShell commands.
97+
98+
2. **Entropulse** — Alternating RL and SFT phases to prevent entropy collapse:
99+
- RL Phase 1 (180 steps): 31.9% → 42.0%
100+
- SFT on successful rollouts: entropy restored
101+
- RL Phase 2 (180 steps): → 45.8%
102+
- Simple to implement (~100 lines wrapper around training loop)
103+
104+
3. **OfficeWorld Benchmark** — 180 tasks across LibreOffice Calc/Writer/Impress, directly relevant to our Core4 tasks.
105+
106+
**Not recommended to adopt as platform**: Tightly coupled to OSWorld/Ubuntu and GLM-4 models. Only 8 git commits, sparse docs.
107+
108+
### 4.2 DART-GUI
109+
110+
**Score**: 42.13% OSWorld
111+
**Architecture**: Fully async 4-module design (env cluster, rollout service, data manager, trainer)
112+
**License**: Apache 2.0
113+
114+
**Most valuable ideas**:
115+
116+
1. **Step-wise GRPO with Entropy Filtering** — Train only on top 80% of steps by token entropy. Skips steps where the model is already confident; focuses compute on decision points. Complements (not replaces) per-step credit methods like HCAPO. **Effort: 1-2 weeks.**
117+
118+
2. **Adaptive Data Curation** (4 levels):
119+
- Task: Experience pool for hard tasks (pre-collect successful trajectories)
120+
- Trajectory: Dynamic rollout count (fewer for easy tasks)
121+
- Step: High-entropy filtering (see above)
122+
- Token: Truncated importance sampling (for async)
123+
124+
3. **Throughput gains**: Environment utilization 12.2% → 67.7% (5.5x) via rollout-wise scheduling. Even with single VM, avoid batch synchronization barriers.
125+
126+
4. **Async architecture pattern** — Critical when scaling to 10+ VMs. Not needed for single-VM validation.
127+
128+
**Not recommended to adopt as platform**: Tightly coupled to OSWorld/Ubuntu, UI-TARS model, Kubernetes, MySQL. Chinese Docker registry (Aliyun).
129+
130+
### 4.3 Integration Roadmap
131+
132+
| Priority | Idea | Source | Effort | Impact | Prerequisite |
133+
|----------|------|--------|--------|--------|--------------|
134+
| **P0** | Validate GRPO end-to-end on WAA | PR #55 | 1-2 weeks | Proves the pipeline works at all | None |
135+
| **P0** | Dense partial-credit rewards | GUI-Genesis concept | 1-2 weeks | Turns binary 0/1 into continuous 0-1 | GRPO working |
136+
| **P1** | API-GUI hybrid actions | ComputerRL | 2-3 weeks | 134% improvement, more rollouts succeed | GRPO working |
137+
| **P1** | Experience pool for hard tasks | DART-GUI | 1 week | Supplement sparse success signal | GRPO working |
138+
| **P2** | HCAPO per-step credit | Paper | 1-2 weeks | 7-14% over GRPO (untested on VLMs) | >20% rollout success rate |
139+
| **P2** | Entropy-based step filtering | DART-GUI | 1-2 weeks | Focus compute on decision points | Multiple training steps |
140+
| **P2** | Entropulse (RL/SFT alternation) | ComputerRL | 1 week | Prevent entropy collapse | Extended training runs |
141+
| **P3** | Async rollout architecture | DART-GUI | 4-6 weeks | Critical for 10+ VM scaling | Multi-VM pool |
142+
| **P3** | Auto-API construction | ComputerRL | 2 weeks/app | Generalize API-GUI to new apps | API-GUI validated |
143+
144+
---
145+
146+
## 5. GUI-Genesis: Synthetic Environments
147+
148+
**Paper**: arXiv:2602.14093 (Feb 2026)
149+
**Core idea**: Auto-synthesize Flask web apps that mirror real tasks, with code-native verifiable rewards.
150+
151+
### Assessment for OpenAdapt: **Not applicable in current form.**
152+
153+
- Generates mobile web apps (375x812), not desktop environments
154+
- No desktop benchmark results; transfer gap to Windows is unknown and likely large
155+
- Depends on Kimi k2 (proprietary LLM), no code released
156+
- Average trajectory length ~5.63 steps (ours: 15-30)
157+
158+
### What IS Useful
159+
160+
1. **Dense code-native rewards** — Instead of binary 0/1 from WAA `/evaluate`, define programmatic assertions that check intermediate state:
161+
- LibreOffice Calc: check cell values via UNO API
162+
- Writer: check font properties via document inspection
163+
- VS Code: check settings.json values
164+
- This gives continuous [0,1] rewards, enabling better credit assignment
165+
166+
2. **PC Agent-E** (arXiv:2505.13909) is more relevant — takes 312 real Windows trajectories, uses Claude to synthesize 9 alternative actions per step, creating trajectory trees. 141% relative improvement on WAA-V2. Closer to our use case.
167+
168+
---
169+
170+
## 6. Competitive Landscape
171+
172+
### Desktop RL Training Systems
173+
174+
| Project | Platform | RL Method | Best Score | Key Innovation |
175+
|---------|----------|-----------|------------|----------------|
176+
| **ComputerRL** | Ubuntu/OSWorld | GRPO + Entropulse | 48.9% OSWorld | API-GUI hybrid, 1000+ VMs |
177+
| **UI-TARS-2** | Win+Linux+Android | Multi-turn RLVR | **50.6% WAA** | Multi-platform (not fully open) |
178+
| **DART-GUI** | Ubuntu/OSWorld | Step-wise GRPO | 42.1% OSWorld | Async 4-module, 1.9x throughput |
179+
| **ARPO** | Ubuntu/OSWorld | GRPO + replay | 29.9% OSWorld | Experience replay buffer |
180+
| **ZeroGUI** | Ubuntu+Android | Online RL | +14% over base | Auto task gen + reward estimation |
181+
| **GUI-Genesis** | Synthetic Flask | GRPO | N/A (mobile) | Synthetic envs, code-native rewards |
182+
| **OpenAdapt** | **Windows/WAA** | **GRPO (+ HCAPO planned)** | Not yet | **Only OSS WAA RL training** |
183+
184+
### Key RL Training Frameworks
185+
186+
| Framework | Stars | GiGPO? | Multi-turn VLM? | Desktop Env? |
187+
|-----------|-------|--------|-----------------|--------------|
188+
| verl | 20k | No | Yes | No |
189+
| Agent Lightning | 15.5k | No | Yes | No |
190+
| OpenRLHF | 9.2k | No | Yes (OpenRLHF-M) | No |
191+
| verl-agent | 1.7k | **Yes** | Yes | No |
192+
| Agent-R1 | 1.3k | No | Yes | No |
193+
| AgentGym-RL | 639 | No | Yes | No |
194+
| VAGEN | 431 | No (dropped) | Yes | No |
195+
196+
---
197+
198+
## 7. Revised Architecture Recommendation
199+
200+
> **Principle**: Don't optimize the training math before validating that training works at all.
201+
> The bottleneck is rollout collection (2-10 min/episode on real VMs), not loss computation
202+
> (seconds). Per-step credit only matters when there are successful rollouts to learn from.
203+
204+
### Phase 1: Get GRPO Working End-to-End (Current → 1-2 weeks)
205+
- Run `validate_grpo_waa.py` phases 1-5 against real WAA VM (PR #55)
206+
- Get non-zero rewards: use easiest tasks, pre-trained model, short episodes (max_steps=5)
207+
- Fix whatever breaks (infra failures, OOM, reward always 0)
208+
- **Goal**: At least one training step with non-zero loss on a real WAA task
209+
- **Success criteria**: Checkpoint saved, loss non-zero, at least one rollout with reward > 0
210+
211+
### Phase 2: Make Rewards Less Sparse (1-3 weeks)
212+
- **Dense partial-credit rewards** — enhance WAA `/evaluate` to return continuous [0,1] scores
213+
via programmatic state checks (cell values, font properties, settings.json). This directly
214+
helps GRPO by giving more gradient signal — turns "all 0 vs all 0" groups into meaningful
215+
advantage estimates. **This is the single highest-impact change.**
216+
- **API-GUI hybrid actions** — let the agent issue programmatic `win32com`/PowerShell commands
217+
alongside GUI clicks (ComputerRL's 134% improvement). Reduces task difficulty, meaning more
218+
rollouts succeed, meaning GRPO has actual gradient signal to work with.
219+
- **Goal**: >20% of rollouts achieving non-zero reward on Core4 tasks
220+
221+
### Phase 3: Improve Training Efficiency (2-4 weeks, after Phase 2)
222+
- **HCAPO per-step credit** (~80-120 lines) — only valuable once we have a mix of successful
223+
and failed rollouts. HCAPO's 7.7-13.8% improvement over GRPO is meaningful only when GRPO
224+
itself is working. Note: untested with VLMs/screenshots — we'd be the first.
225+
- **Entropy-based step filtering** (DART-GUI) — train on top 80% of steps by token entropy,
226+
focusing compute on decision points. Cheap to add (~50 lines).
227+
- **Entropulse** (ComputerRL) — alternate RL/SFT phases to prevent entropy collapse during
228+
extended training. Only relevant after multiple RL phases (~100 lines).
229+
- **Experience pool** (DART-GUI) — pre-collect successful trajectories for hard tasks to
230+
supplement sparse online success signal.
231+
- **Goal**: Measurable improvement in sample efficiency over Phase 2 baseline
232+
233+
### Phase 4: Scaling (4-8 weeks, when needed)
234+
- Adopt DART-GUI async architecture pattern (reimplemented against our pool infra)
235+
- Scale to 10+ parallel WAA VMs
236+
- Only consider verl-agent if HCAPO proves insufficient and GiGPO anchors can be made reliable
237+
- **Goal**: Production-grade training at scale
238+
239+
### What to Drop
240+
- **VAGEN-Lite as training backend** — vanilla GRPO only, no advantage over standalone
241+
- **GiGPO anchor state approach** — unreliable for pixel-based desktop screenshots
242+
(mouse position, animation frames, anti-aliasing differ across rollouts even for
243+
semantically identical states; a11y tree hashing is unreliable during UI transitions)
244+
- **GUI-Genesis integration** — mobile web only, no code, proprietary LLM dependency
245+
- **HiPER** — requires verl-agent anyway, adds hierarchical complexity
246+
247+
### Prioritization Rationale
248+
249+
The research identified HCAPO as the best per-step credit method, but **per-step credit
250+
is a Phase 3 optimization, not a Phase 1 prerequisite**. Here's why:
251+
252+
1. If all 8 rollouts score 0, GRPO has zero gradient signal. HCAPO can't fix that —
253+
it redistributes credit *within* a trajectory, but the episode-level advantage is
254+
still zero when all rewards are equal.
255+
2. The published 7.7% WebShop improvement is relative to a working GRPO baseline on
256+
text environments with 5-10 step episodes. Transfer to 15-30 step visual desktop
257+
tasks is unproven.
258+
3. Our bottleneck is rollout success rate, not training math. Dense rewards and
259+
API-GUI actions directly increase the fraction of rollouts with non-zero reward,
260+
which is the prerequisite for any training algorithm to learn.
261+
262+
---
263+
264+
## 8. Sources
265+
266+
### Papers
267+
- ComputerRL: [arxiv.org/abs/2508.14040](https://arxiv.org/abs/2508.14040)
268+
- DART-GUI: [arxiv.org/abs/2509.23866](https://arxiv.org/abs/2509.23866)
269+
- GUI-Genesis: [arxiv.org/abs/2602.14093](https://arxiv.org/abs/2602.14093)
270+
- HCAPO: [arxiv.org/abs/2603.08754](https://arxiv.org/abs/2603.08754)
271+
- iStar: [arxiv.org/abs/2509.19199](https://arxiv.org/abs/2509.19199)
272+
- GiGPO: [arxiv.org/abs/2505.10978](https://arxiv.org/abs/2505.10978)
273+
- HiPER: [arxiv.org/abs/2602.16165](https://arxiv.org/abs/2602.16165)
274+
- VAGEN: [arxiv.org/abs/2510.16907](https://arxiv.org/abs/2510.16907)
275+
- PC Agent-E: [arxiv.org/abs/2505.13909](https://arxiv.org/abs/2505.13909)
276+
- UI-TARS-2: [arxiv.org/abs/2509.02544](https://arxiv.org/abs/2509.02544)
277+
- DigiRL: [arxiv.org/abs/2406.11896](https://arxiv.org/abs/2406.11896)
278+
279+
### Repositories
280+
- ComputerRL: [github.com/thudm/ComputerRL](https://github.com/thudm/ComputerRL)
281+
- DART-GUI: [github.com/Computer-use-agents/dart-gui](https://github.com/Computer-use-agents/dart-gui)
282+
- ARPO: [github.com/dvlab-research/ARPO](https://github.com/dvlab-research/ARPO)
283+
- ZeroGUI: [github.com/OpenGVLab/ZeroGUI](https://github.com/OpenGVLab/ZeroGUI)
284+
- verl: [github.com/verl-project/verl](https://github.com/verl-project/verl)
285+
- verl-agent: [github.com/langfengQ/verl-agent](https://github.com/langfengQ/verl-agent)
286+
- VAGEN: [github.com/mll-lab-nu/VAGEN](https://github.com/mll-lab-nu/VAGEN)
287+
- Agent Lightning: [github.com/microsoft/agent-lightning](https://github.com/microsoft/agent-lightning)
288+
- BrowserGym: [github.com/ServiceNow/BrowserGym](https://github.com/ServiceNow/BrowserGym)
289+
- OSWorld: [github.com/xlang-ai/OSWorld](https://github.com/xlang-ai/OSWorld)
290+
- PC Agent-E: [github.com/GAIR-NLP/PC-Agent-E](https://github.com/GAIR-NLP/PC-Agent-E)
291+
- HiPER: [github.com/JonP07/HiPER-agent](https://github.com/JonP07/HiPER-agent)

0 commit comments

Comments
 (0)