|
| 1 | +--- |
| 2 | +name: desktop-control |
| 3 | +description: Background knowledge for droid-control workflows -- not invoked directly. Desktop-control driver mechanics for native GUI app automation via trycua cua-driver. |
| 4 | +user-invocable: false |
| 5 | +--- |
| 6 | + |
| 7 | +# Desktop-Control Driver |
| 8 | + |
| 9 | +The orchestrator routed you here. Use these mechanics to execute your plan. |
| 10 | + |
| 11 | +Drive native desktop GUI apps through upstream [trycua/cua](https://github.com/trycua/cua) `cua-driver`: enumerate apps and windows, snapshot accessibility trees, click/type/scroll by `element_index` or pixel coordinates, and verify by re-snapshot -- all without bringing the target to the foreground. |
| 12 | + |
| 13 | +## When to use |
| 14 | + |
| 15 | +- Automating a native desktop app (Finder, Notepad, System Settings, native editors) |
| 16 | +- Driving native dialogs and security/permission sheets that no DOM or PTY can reach |
| 17 | +- Visual QA of native UI: per-window screenshots, accessibility-tree assertions |
| 18 | + |
| 19 | +If the target is a terminal TUI, use **tuistory** or **true-input**. If it is a web page or an Electron app, use **agent-browser** -- CDP beats accessibility trees for anything Chromium-based. |
| 20 | + |
| 21 | +## Platform support |
| 22 | + |
| 23 | +| Platform | Upstream tier | Read | |
| 24 | +|---|---|---| |
| 25 | +| macOS | Production | [platforms/macos.md](platforms/macos.md) | |
| 26 | +| Windows | Production | [platforms/windows.md](platforms/windows.md) | |
| 27 | +| Linux | Pre-release (real caveats) | [platforms/linux.md](platforms/linux.md) | |
| 28 | + |
| 29 | +**Read the platform file for your target OS.** Each contains permissions, daemon launch, and platform-specific patterns and failure modes. |
| 30 | + |
| 31 | +## Prerequisites |
| 32 | + |
| 33 | +```bash |
| 34 | +# one-time install: per-user, no sudo/admin |
| 35 | +curl -fsSL https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/scripts/install.sh | bash |
| 36 | +# Windows (PowerShell): |
| 37 | +# irm https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/scripts/install.ps1 | iex |
| 38 | + |
| 39 | +cua-driver doctor # platform probes: permissions, daemon, accessibility plumbing |
| 40 | +cua-driver skills install # fetch the upstream skill pack to ~/.cua-driver/skills/cua-driver |
| 41 | +``` |
| 42 | + |
| 43 | +The upstream pack (`~/.cua-driver/skills/cua-driver/SKILL.md` + your platform's doc) is the deep reference -- full tool surface, window-state behavior matrix, forbidden-command lists -- and it updates with the binary. **Read it before any nontrivial workflow.** This atom owns the droid-control integration: routing, run isolation, delegation, evidence handoff. |
| 44 | + |
| 45 | +## Daemon lifecycle |
| 46 | + |
| 47 | +`element_index` workflows **require the daemon**. Without it each CLI invocation is a fresh process and the per-`(pid, window_id)` element cache dies between calls. |
| 48 | + |
| 49 | +```bash |
| 50 | +cua-driver serve # start the daemon (macOS needs the LaunchServices form -- see platforms/macos.md) |
| 51 | +cua-driver status # daemon + socket health |
| 52 | +cua-driver stop |
| 53 | +``` |
| 54 | + |
| 55 | +Permissions are checked and granted through the driver, not by hand-editing system settings (macOS-only gate; a no-op surface on Windows/Linux): |
| 56 | + |
| 57 | +```bash |
| 58 | +cua-driver permissions status # read-only; answers via the running daemon |
| 59 | +cua-driver permissions grant # attributed prompt flow -- the correct way to grant |
| 60 | +``` |
| 61 | + |
| 62 | +## Core loop |
| 63 | + |
| 64 | +Tool names are `snake_case` and invoked directly: `cua-driver <tool> '<json>'`. (`cua-driver call <tool>` is legacy; do not use it.) `cua-driver list-tools` for the inventory, `cua-driver describe <tool>` for any schema. |
| 65 | + |
| 66 | +Every workflow is Discover -> Observe -> Act -> Verify against an explicit `(pid, window_id)`: |
| 67 | + |
| 68 | +```bash |
| 69 | +cua-driver launch_app '{"name":"TextEdit"}' |
| 70 | +# -> {pid: 844, windows: [{window_id: 10725, ...}]} # list_windows only needed for long-lived pids |
| 71 | +cua-driver get_window_state '{"pid":844,"window_id":10725}' --screenshot-out-file "${RUN_DIR}/before.png" |
| 72 | +cua-driver click '{"pid":844,"window_id":10725,"element_index":14,"session":"'"${RUN_ID}"'-desktop"}' |
| 73 | +cua-driver get_window_state '{"pid":844,"window_id":10725}' --screenshot-out-file "${RUN_DIR}/after.png" |
| 74 | +``` |
| 75 | + |
| 76 | +**Snapshot before AND after every action.** The pre-action `get_window_state` resolves the `element_index` you are about to use -- indices are per-snapshot, per `(pid, window_id)`, and stale ones fail with `No cached AX state`. The post-action snapshot is the evidence the action landed; without it a silent no-op looks like success. |
| 77 | + |
| 78 | +Addressing-mode preference: |
| 79 | + |
| 80 | +1. **`element_index`** (default) -- semantic, works on hidden and backgrounded windows, no foreground change. |
| 81 | +2. **Pixel** `click '{"pid":N,"window_id":W,"x":X,"y":Y}'` -- for surfaces the tree does not reach (canvases, custom-drawn controls). Coordinates are window-local screenshot pixels, top-left origin. |
| 82 | +3. **Keyboard** (`press_key`, `hotkey`) and platform fallbacks -- last resort; see the platform files. |
| 83 | + |
| 84 | +## Run isolation (ground rule 5 -> cua sessions) |
| 85 | + |
| 86 | +cua sessions are the desktop equivalent of `tctl` session prefixes: a session owns its agent cursor, config overrides, and recording scope. Declare one per run, derived from the workflow's `RUN_ID`, and pass it on every action: |
| 87 | + |
| 88 | +```bash |
| 89 | +cua-driver start_session '{"session":"'"${RUN_ID}"'-desktop"}' |
| 90 | +# ... every action carries "session":"${RUN_ID}-desktop" ... |
| 91 | +cua-driver end_session '{"session":"'"${RUN_ID}"'-desktop"}' |
| 92 | +``` |
| 93 | + |
| 94 | +Parallel workers each declare their **own session** and pass `creates_new_application_instance: true` to `launch_app` so each gets its own window. The element cache is keyed on `(pid, window_id)` and the cursor on `session`, so isolated workers cannot collide. |
| 95 | + |
| 96 | +## Delegation |
| 97 | + |
| 98 | +`cua-driver` is on PATH -- workers need no `${DROID_PLUGIN_ROOT}` resolution. As with the other drivers, give capture workers **exact commands** with the parent's run scope baked in: |
| 99 | + |
| 100 | +``` |
| 101 | +Task prompt for a desktop capture worker: |
| 102 | + "Run these commands in order. Report screenshot paths and any errors. |
| 103 | + 1. cua-driver start_session '{"session":"1712345678-42-notepad"}' |
| 104 | + 2. cua-driver launch_app '{"name":"Notepad","creates_new_application_instance":true}' |
| 105 | + -> note the returned pid and window_id |
| 106 | + 3. cua-driver get_window_state '{"pid":<pid>,"window_id":<wid>}' --screenshot-out-file /tmp/droid-run-1712345678-42-xxxx/before.png |
| 107 | + 4. cua-driver type_text '{"pid":<pid>,"window_id":<wid>,"element_index":<text-area>,"text":"hello","session":"1712345678-42-notepad"}' |
| 108 | + 5. cua-driver get_window_state '{"pid":<pid>,"window_id":<wid>}' --screenshot-out-file /tmp/droid-run-1712345678-42-xxxx/after.png |
| 109 | + 6. cua-driver end_session '{"session":"1712345678-42-notepad"}'" |
| 110 | +``` |
| 111 | + |
| 112 | +## Evidence handoff |
| 113 | + |
| 114 | +| Proof type | How to capture | |
| 115 | +|---|---| |
| 116 | +| Window state | `get_window_state ... --screenshot-out-file ${RUN_DIR}/proof-N.png` (also keeps the PNG out of the tool response) | |
| 117 | +| Full display | `cua-driver screenshot '{"out_file":"'"${RUN_DIR}"'/screen.png"}'` | |
| 118 | +| Semantic assertions | `tree_markdown` from `get_window_state` (filter with `"query":"..."`) | |
| 119 | +| Video | `cua-driver recording start` / `recording stop` -> session-scoped `recording.mp4` | |
| 120 | + |
| 121 | +Hand PNG/mp4 paths to **compose** / **verify** like any other driver output. Keep raw tool output alongside screenshots whenever GUI behavior is the thing under test. |
| 122 | + |
| 123 | +## Critical rules |
| 124 | + |
| 125 | +1. **Never change the user's frontmost app.** If a command says activate, foreground, raise, or make key -- stop; the per-pid event paths exist precisely so you do not need it. Platform forbidden-lists live in the upstream pack. |
| 126 | +2. **Re-snapshot after every action and report what you observed**, not what you intended. An unchanged tree after an action is a finding, not a formality. |
| 127 | +3. **Destructive actions need explicit user intent.** Do not delete files, send messages, or submit forms unless the workflow asked for exactly that. |
0 commit comments