fix(agent): replace manual string escaping with repr() and fix CU agent bugs (#83)

* fix(agent): replace manual string escaping with repr() and fix CU agent bugs

Five reliability fixes for eval runs:

1. Replace _escape_for_pyautogui() with repr() in _build_type_commands() -
   eliminates entire class of string-embedding bugs (newlines, tabs, quotes,
   unicode) using Python's own escaping mechanism

2. Fix drag coordinate field names: startCoordinate/endCoordinate (camelCase)
   → start_coordinate/coordinate (snake_case) per Claude computer_use API

3. Add _clamp_coord() to prevent (0,0) coordinates from triggering PyAutoGUI
   fail-safe, applied to click, drag, and mouse_move actions

4. Re-inject demo text at every step in tool_result messages to prevent
   context drift in demo-conditioned evaluation

5. Add command logging in WAALiveAdapter.step() for debugging

Also adds docs/eval_analysis_2026_03_02.md documenting ZS vs DC eval
results and literature review on demo-conditioning approaches.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: add multi-level demo format transform and fix tests

- Add scripts/transform_demo_format.py: transforms rigid {Observation,
  Intent, Action, Result} demos into adaptive {Think, Action, Expect}
  format with PLAN section (Option D from eval analysis)
  - LLM-assisted mode (default): uses vlm_call() for semantic transform
  - Rule-based mode (--no-llm): free, no API calls needed
  - Supports --dry-run for preview

- Fix tests for repr() escaping and coordinate clamping:
  - Remove TestEscapeForPyautogui (tests deleted function)
  - Update TestBuildTypeCommands for repr() output format
  - Add test_all_special_chars_produce_valid_python invariant test
  - Fix drag test to use snake_case field names
  - Fix coordinate edge test to expect clamped (0.005, 0.005)

- Regenerate uv.lock for consilium package name resolution

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* docs: add DC-multilevel eval results to analysis

DC-multilevel (new {Think, Action, Expect} + PLAN format) showed clear
improvement over DC-rigid: agent followed the plan, entered all headers
and years, typed correct formula, used drag-fill. Still scored 0.0 due
to premature task completion (finished 1/3 columns), but qualitatively
the best behavior across all three conditions.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

Author: abrichr

docs/eval_analysis_2026_03_02.md

@@ -215,43 +215,33 @@ def _is_failsafe_error(text: str) -> bool:
     return "failsafeexception" in lower or "fail-safe triggered" in lower
 
 
-def _escape_for_pyautogui(text: str) -> str:
-    """Escape text for embedding in a single-quoted Python string literal."""
-    return (
-        text
-        .replace("\\", "\\\\")
-        .replace("'", "\\'")
-        .replace("\t", "\\t")
-        .replace("\r", "")
-    )
-
-
 def _build_type_commands(text: str) -> str:
     """Build pyautogui command body to type text, handling embedded newlines.
 
-    ``pyautogui.write()`` cannot handle literal newline characters — the
-    generated Python command string becomes an unterminated string literal
-    when executed via ``exec()``.  This function splits the text on newlines
-    and interleaves ``pyautogui.write()`` with ``pyautogui.press('enter')``.
+    Uses ``repr()`` for string escaping instead of manual character-by-character
+    replacement.  This eliminates the entire class of escaping bugs (newlines,
+    tabs, quotes, unicode, null bytes, etc.) because ``repr()`` is Python's own
+    mechanism for producing valid string literals from any string content —
+    the same principle as parameterized SQL queries vs string concatenation.
+
+    Newlines are handled semantically: split into separate ``write()`` calls
+    with ``press('enter')`` between them, since the agent intends "press Enter."
 
     Returns:
         A pyautogui command body string (without ``import pyautogui;`` prefix).
         Callers must prepend the import themselves.
     """
+    text = text.replace("\r", "")
     segments = text.split("\n")
     if len(segments) == 1:
-        escaped = _escape_for_pyautogui(text)
-        return f"pyautogui.write('{escaped}', interval=0.02)"
+        return f"pyautogui.write({repr(text)}, interval=0.02)"
 
     commands: list[str] = []
     for i, seg in enumerate(segments):
-        # Skip empty trailing segment from a trailing newline
-        if seg or i < len(segments) - 1:
-            escaped = _escape_for_pyautogui(seg)
-            if escaped:
-                commands.append(f"pyautogui.write('{escaped}', interval=0.02)")
-            if i < len(segments) - 1:
-                commands.append("pyautogui.press('enter')")
+        if seg:
+            commands.append(f"pyautogui.write({repr(seg)}, interval=0.02)")
+        if i < len(segments) - 1:
+            commands.append("pyautogui.press('enter')")
     return "; ".join(commands) if commands else "pass"
 
 
@@ -575,6 +565,7 @@ def step(
 
         # Execute command via /execute_windows (has access to computer object)
         if command:
+            logger.info("Sending command to WAA: %r", command)
             try:
                 resp = requests.post(
                     f"{self.config.server_url}/execute_windows",

openadapt_evals/adapters/waa/live.py

@@ -78,6 +78,9 @@ class ClaudeComputerUseAgent(BenchmarkAgent):
 
     DEFAULT_MODEL = "claude-sonnet-4-6"
 
+    # Minimum normalized coordinate to avoid PyAutoGUI fail-safe (top-left corner)
+    _COORD_EPS = 0.005  # ~6px at 1280, ~4px at 720
+
     def __init__(
         self,
         api_key: str | None = None,
@@ -123,6 +126,16 @@ def __init__(
                 f"Demo provided ({len(self.demo)} chars) - persists across all steps"
             )
 
+    def _clamp_coord(self, x_norm: float, y_norm: float) -> tuple[float, float]:
+        """Clamp normalized coordinates away from (0,0) to avoid fail-safe."""
+        if x_norm < self._COORD_EPS and y_norm < self._COORD_EPS:
+            logger.warning(
+                f"Clamping near-zero coordinates ({x_norm:.4f}, {y_norm:.4f}) "
+                f"to ({self._COORD_EPS}, {self._COORD_EPS}) to avoid fail-safe"
+            )
+            return (self._COORD_EPS, self._COORD_EPS)
+        return (x_norm, y_norm)
+
     def reset(self) -> None:
         """Reset agent state between episodes."""
         self._messages = []
@@ -178,7 +191,17 @@ def act(
                 tool_result = self._build_tool_result(
                     screenshot_b64, self._last_tool_use_id
                 )
-                self._messages.append({"role": "user", "content": [tool_result]})
+                content: list[dict[str, Any]] = [tool_result]
+                # Re-inject demo at every step so it doesn't drift out of context
+                if self.demo:
+                    content.append({
+                        "type": "text",
+                        "text": (
+                            f"DEMONSTRATION (follow this pattern, you are at step "
+                            f"{self._step_count}):\n---\n{self.demo}\n---"
+                        ),
+                    })
+                self._messages.append({"role": "user", "content": content})
 
         # Loop: call API, and if Claude requests a screenshot/wait, send the
         # screenshot back and call again (up to MAX_INTERNAL_RETRIES times)
@@ -374,8 +397,9 @@ def _map_action(
             "triple_click",
         ):
             coord = tool_input.get("coordinate", [0, 0])
-            x_norm = coord[0] / self.display_width
-            y_norm = coord[1] / self.display_height
+            x_norm, y_norm = self._clamp_coord(
+                coord[0] / self.display_width, coord[1] / self.display_height
+            )
             ba_type = "click"
             raw["click_variant"] = action_type
             return BenchmarkAction(
@@ -414,25 +438,33 @@ def _map_action(
 
         # Drag action
         if action_type == "left_click_drag":
-            start = tool_input.get("startCoordinate", [0, 0])
-            end = tool_input.get("endCoordinate", [0, 0])
+            # Claude's computer_use API uses snake_case field names:
+            #   start_coordinate: [x, y]  (drag start)
+            #   coordinate: [x, y]        (drag end)
+            start = tool_input.get("start_coordinate", [0, 0])
+            end = tool_input.get("coordinate", [0, 0])
+            sx, sy = self._clamp_coord(
+                start[0] / self.display_width, start[1] / self.display_height
+            )
+            ex, ey = self._clamp_coord(
+                end[0] / self.display_width, end[1] / self.display_height
+            )
             return BenchmarkAction(
                 type="drag",
-                x=start[0] / self.display_width,
-                y=start[1] / self.display_height,
-                end_x=end[0] / self.display_width,
-                end_y=end[1] / self.display_height,
+                x=sx, y=sy, end_x=ex, end_y=ey,
                 raw_action=raw,
             )
 
         # Mouse move — treat as a click with no effect for BenchmarkAction
         if action_type == "mouse_move":
             coord = tool_input.get("coordinate", [0, 0])
+            x_norm, y_norm = self._clamp_coord(
+                coord[0] / self.display_width, coord[1] / self.display_height
+            )
             raw["is_mouse_move"] = True
             return BenchmarkAction(
                 type="click",
-                x=coord[0] / self.display_width,
-                y=coord[1] / self.display_height,
+                x=x_norm, y=y_norm,
                 raw_action=raw,
             )