feat: add VAGEN/verl-agent environment adapter for VLM RL training

* feat: add VAGEN/verl-agent environment adapter for VLM RL training

WAADesktopEnv implements the GymImageEnv protocol from VAGEN, enabling
desktop GUI automation training with verl-agent's multi-turn VLM RL
pipeline (GiGPO, GRPO, PPO).

The adapter translates between openadapt-evals BenchmarkObservation
(PNG bytes + a11y tree) and VAGEN's observation format (obs_str +
multi_modal_input with PIL images).

- Async interface (reset/step/close/system_prompt)
- Action DSL parsing (CLICK, TYPE, KEY, SCROLL, WAIT, DONE)
- Fractional coordinate support (0.0-1.0)
- Lazy adapter initialization
- 21 tests passing with mock adapter
- Example VAGEN training config included

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* docs: add comprehensive verl-agent decision document

Records the full reasoning chain for choosing verl-agent/VAGEN:
- Framework comparison (TRL, standalone, verl-agent, VAGEN, OpenRLHF, Unsloth)
- Key insight: per-step verification via GiGPO for long-horizon GUI tasks
- TRL multi-turn VLM blocker (issues #5119, #5120)
- "Environment is the moat" strategic framing
- Architecture diagram and migration path

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: add verl-agent as optional dependency

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: vendor GymImageEnv base classes from VAGEN

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* docs: fact-check framework review in verl decision doc

Update Sections E (OpenRLHF), F (Unsloth), TRL, and comparison matrix
with accurate details from thorough review:

- OpenRLHF: document AgentTrainer multi-turn support and OpenRLHF-M fork
- Unsloth: nuanced assessment — single-turn VLM works, multi-turn text
  via ART works, but multi-turn VLM blocked by rollout_func issue (#3573)
- TRL: add note about OpenEnv/rollout_func for text models (VLM blocked)
- Comparison matrix: add Unsloth column with footnotes

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

Author: abrichr

configs/train_waa_vagen.yaml

@@ -0,0 +1,66 @@
+# VAGEN training config for WAA desktop automation
+#
+# This trains a VLM (e.g., Qwen2.5-VL-3B) to automate Windows desktop tasks
+# using GRPO/GiGPO via the verl-agent framework.
+#
+# Prerequisites:
+#   1. WAA server running (via SSH tunnel): ssh -L 5001:localhost:5050 azureuser@<VM_IP>
+#   2. VAGEN installed: pip install vagen
+#   3. Register env: add to vagen's env_registry.yaml:
+#        WAADesktop: openadapt_evals.adapters.verl_env.WAADesktopEnv
+#
+# Usage:
+#   python -m vagen.train --config configs/train_waa_vagen.yaml
+#
+# For mock testing (no VM):
+#   Set server_url to "mock" and use WAAMockAdapter internally
+
+# --- Model ---
+model:
+  name: Qwen/Qwen2.5-VL-3B-Instruct
+  # For larger models with LoRA:
+  # name: Qwen/Qwen2.5-VL-7B-Instruct
+  # lora:
+  #   r: 16
+  #   alpha: 32
+  #   target_modules: [q_proj, k_proj, v_proj, o_proj]
+
+# --- Environment ---
+envs:
+  - name: WAADesktop
+    n_envs: 8                    # Number of parallel environments (= GRPO group size)
+    data_source: waa
+    seed: [1, 100, 1]            # [start, end, step] for task selection
+    max_turns: 15                # Max actions per episode
+    response_length_per_turn: 512
+    config:
+      server_url: "http://localhost:5001"
+      task_id: "REPLACE_WITH_WAA_TASK_UUID"
+      max_steps: 15
+      evaluate_at_done: true
+      action_type: fractional    # VLM outputs normalized 0-1 coordinates
+
+# --- Training (GRPO) ---
+algorithm:
+  name: grpo                     # or "gigpo" for step-level advantages
+  kl_coef: 0.0                   # No KL penalty (DAPO/Open-Reasoner-Zero style)
+  epsilon: 0.2                   # PPO clip range (inactive with single epoch)
+  gamma: 1.0                     # No discounting for episodic tasks
+
+trainer:
+  total_epochs: 100
+  n_gpus_per_node: 2             # Minimum for VLM training
+  micro_batch_size: 4
+  gradient_accumulation_steps: 2
+
+# --- Rollout ---
+rollout:
+  temperature: 0.7
+  top_p: 0.95
+  mode: async                    # async sglang rollout for throughput
+
+# --- Logging ---
+logging:
+  project: openadapt-waa-rl
+  log_interval: 1
+  save_interval: 10

docs/verl_agent_decision.md

@@ -39,6 +39,7 @@
     RLEnvironment,
     RolloutStep,
 )
+from openadapt_evals.adapters.verl_env import WAADesktopEnv
 from openadapt_evals.adapters.waa import (
     WAAAdapter,
     WAAConfig,
@@ -69,6 +70,8 @@
     "WAAMockAdapter",
     "WAALiveAdapter",
     "WAALiveConfig",
+    # verl-agent / VAGEN integration
+    "WAADesktopEnv",
     # Task ID validation
     "SyntheticTaskError",
     "is_real_waa_task_id",

openadapt_evals/adapters/__init__.py

@@ -0,0 +1,10 @@
+"""Vendored pure-abstract base classes from VAGEN.
+
+These are copied from https://github.com/mll-lab-nu/VAGEN so that
+openadapt-evals can implement the GymImageEnv protocol without
+requiring the full VAGEN package (and its heavy transitive
+dependencies) to be installed.
+
+Only the abstract interface definitions are vendored here -- no
+concrete implementations or utilities.
+"""

openadapt_evals/adapters/_vendored/__init__.py

@@ -0,0 +1,40 @@
+# Vendored from https://github.com/mll-lab-nu/VAGEN
+# These are pure abstract base classes with no heavy dependencies.
+# Vendored to avoid requiring the full VAGEN installation.
+# Last synced: 2026-03-02
+
+from __future__ import annotations
+
+from abc import ABC, abstractmethod
+from typing import Any, Dict, Tuple
+
+
+class GymBaseEnv(ABC):
+    """
+    Abstract async environment API.
+    The handler does not assume any obs/data schema beyond what you return.
+
+    Contract:
+      - reset(seed) -> (obs, info)
+      - step(action_str) -> (obs, reward, done, info)
+    """
+
+    def __init__(self, env_config: Dict[str, Any]):
+        self.config = env_config
+
+    @abstractmethod
+    async def close(self) -> None:
+        """Async teardown."""
+        raise NotImplementedError
+
+    @abstractmethod
+    async def reset(self, seed: int):
+        raise NotImplementedError
+
+    @abstractmethod
+    async def step(self, action_str: str):
+        raise NotImplementedError
+
+    @abstractmethod
+    async def system_prompt(self):
+        raise NotImplementedError

openadapt_evals/adapters/_vendored/gym_base_env.py

@@ -0,0 +1,212 @@
+# Vendored from https://github.com/mll-lab-nu/VAGEN
+# These are pure abstract base classes with no heavy dependencies.
+# Vendored to avoid requiring the full VAGEN installation.
+# Last synced: 2026-03-02
+
+from __future__ import annotations
+
+from abc import abstractmethod
+from typing import Any, Dict, Tuple
+
+from .gym_base_env import GymBaseEnv
+
+
+class GymImageEnv(GymBaseEnv):
+    """
+    GymImageEnv is a base environment class that supports optional
+    **image-based multi-modal observations**, while keeping the same API
+    as GymBaseEnv.
+
+    --------------------------------------------------------------------
+    Observation Protocol
+    --------------------------------------------------------------------
+
+    WITH images
+    -------------------------------
+    If the environment returns images, the observation should follow:
+
+        obs = {
+            "obs_str": "... <image> ...",
+            "multi_modal_input": {
+                "<image>": [PIL.Image.Image, ...]
+            }
+        }
+
+    - Images are stored under obs["multi_modal_input"]["<image>"].
+    - "<image>" in obs_str is a placeholder indicating where each image
+      should appear in the prompt.
+    - The number of "<image>" in obs_str should match the number of
+      images in the list.
+
+    WITHOUT images:
+    ----------------------------------
+    Can simply use:
+
+        obs = {
+            "obs_str": "..."
+        }
+
+    - "multi_modal_input" is optional and may be omitted.
+    - obs_str should NOT contain "<image>" placeholders.
+
+
+    --------------------------------------------------------------------
+    Agent-Loop Rollout
+    --------------------------------------------------------------------
+    - sys      : system prompt (from system_prompt()).
+    - init_obs : observation from reset().
+    - step_obs : observation from step().
+    - res_i    : agent response at step i.
+
+    Concat mode (single growing context):
+        sys + init_obs + res_0 + step_obs_1 + res_1 + ...
+
+    Non-concat mode (step-wise independent contexts):
+        Step 0: sys + init_obs   + res_0
+        Step 1: sys + step_obs_1 + res_1
+        Step 2: sys + step_obs_2 + res_2
+
+    --------------------------------------------------------------------
+    Info
+    --------------------------------------------------------------------
+    The `info` dict returned by reset() and step() may include:
+        - success (bool): whether the task/episode is considered
+          successful, this will be used for wandb logging.
+    """
+
+    def __init__(self, env_config: Dict[str, Any]):
+        """
+        Initialize the environment.
+
+        Args:
+            env_config (Dict[str, Any]):
+                Environment configuration. The exact schema is defined by
+                the concrete environment implementation and/or GymBaseEnv.
+
+        Side effects:
+            - Calls GymBaseEnv.__init__(env_config).
+        """
+        super().__init__(env_config)
+
+    @abstractmethod
+    async def close(self) -> None:
+        """
+        Close the environment and release all resources.
+
+        This should clean up anything created by the environment, e.g.:
+        - windows / renderers
+        - subprocesses
+        - file handles
+        - GPU memory / models
+
+        Returns:
+            None
+        """
+        raise NotImplementedError
+
+    @abstractmethod
+    async def system_prompt(self) -> Dict[str, Any]:
+        """
+        Return the system-level prompt/observation for the environment.
+
+        Returns:
+            obs (Dict[str, Any]):
+                A dict representing the system prompt observation.
+
+                If returning images, it must follow:
+
+                    obs = {
+                        "obs_str": "... <image> ...",
+                        "multi_modal_input": {
+                            "<image>": [PIL.Image.Image, ...]
+                        }
+                    }
+
+                If returning no images, it must follow:
+
+                    obs = {
+                        "obs_str": "..."
+                    }
+        """
+        raise NotImplementedError
+
+    @abstractmethod
+    async def reset(self, seed: int) -> Tuple[Dict[str, Any], Dict[str, Any]]:
+        """
+        Reset the environment to the initial state.
+
+        Args:
+            seed (int):
+                Random seed used to initialize the environment
+
+        Returns:
+            obs (Dict[str, Any]):
+                The initial observation after reset.
+
+                If returning images, it must follow:
+
+                    obs = {
+                        "obs_str": "... <image> ...",
+                        "multi_modal_input": {
+                            "<image>": [PIL.Image.Image, ...]
+                        }
+                    }
+
+                If returning no images, it must follow:
+
+                    obs = {
+                        "obs_str": "..."
+                    }
+
+            info (Dict[str, Any]):
+                A dict containing any additional metadata about the reset,
+                e.g. debug information, episode identifiers, etc.
+        """
+        raise NotImplementedError
+
+    @abstractmethod
+    async def step(
+        self, action_str: str
+    ) -> Tuple[Dict[str, Any], float, bool, Dict[str, Any]]:
+        """
+        Execute one environment step using an agent-provided action.
+
+        Args:
+            action_str (str):
+                The action produced by the agent, in text form.
+
+        Returns:
+            obs (Dict[str, Any]):
+                The next observation after applying the action.
+
+                If returning images, it must follow:
+
+                    obs = {
+                        "obs_str": "... <image> ...",
+                        "multi_modal_input": {
+                            "<image>": [PIL.Image.Image, ...]
+                        }
+                    }
+
+                If returning no images, it must follow:
+
+                    obs = {
+                        "obs_str": "..."
+                    }
+
+            reward (float):
+                Scalar reward for the current step.
+
+            done (bool):
+                Whether the current episode has terminated after this
+                step.
+
+            info (Dict[str, Any]):
+                Additional step-level metadata.
+
+                Common optional keys:
+                    - success (bool): whether the task/episode is
+                      considered successful, typically used for logging
+                      (e.g. wandb).
+        """
+        raise NotImplementedError