Skip to content

Commit 48dcb2e

Browse files
thymikeethymikee
committed
docs: simplify README overview
1 parent bda96f1 commit 48dcb2e

1 file changed

Lines changed: 52 additions & 154 deletions

File tree

‎README.md‎

Lines changed: 52 additions & 154 deletions
Original file line numberDiff line numberDiff line change
@@ -12,109 +12,57 @@
1212
[![CI](https://github.com/callstackincubator/agent-device/actions/workflows/ci.yml/badge.svg)](https://github.com/callstackincubator/agent-device/actions/workflows/ci.yml)
1313
[![License: MIT](https://img.shields.io/badge/license-MIT-black.svg)](LICENSE)
1414

15-
Device automation CLI for AI agents. Mobile, TV, and desktop apps.
15+
Mobile app verification for AI agents.
1616

17-
`agent-device` lets coding agents run real apps, inspect UI state, interact with visible elements, and collect debugging evidence through one CLI.
17+
A device automation CLI for real apps on iOS, Android, TV, and desktop. Agents get token-efficient snapshots, semantic refs, and evidence captured only when needed.
1818

19-
It is built around token-efficient accessibility snapshots, not pixel-first screenshots. Agents read compact UI trees, locate elements through refs like `@e3`, perform touch and text actions, and capture screenshots, video, logs, network, CPU/memory/perf, crash-related logs, and React profiles only when evidence is needed.
19+
`agent-device` lets coding agents open apps, inspect the current UI, interact with visible elements, and collect debugging evidence through one CLI. Use it when an agent needs to verify what actually happens on a device, not just reason about code.
2020

21-
Agents can ingest the current docs from [llms-full.txt](https://incubator.callstack.com/agent-device/llms-full.txt). The installed CLI help remains authoritative for exact command syntax.
21+
If you know Vercel's [agent-browser](https://github.com/vercel-labs/agent-browser), `agent-device` is the same idea for mobile, TV, and desktop apps.
2222

23-
## Agentic QA And Development
23+
It works with native iOS and Android apps, plus apps built with Expo, Flutter, and React Native, as long as the target can run on a supported device, simulator, emulator, or desktop environment.
2424

25-
- **Quality Assurance**: dogfood flows, validate PR builds, check accessibility coverage, and turn stable explorations into `.ad` e2e tests.
26-
- **Development**: build from specs, inspect real runtime behavior, and iterate until the UI matches the work.
25+
![agent-device demo showing Codex using agent-device to create a new contact in the iOS Contacts app from a simple prompt](./website/docs/public/agent-device-contacts.gif)
2726

28-
`agent-device` closes the agentic development loop: agents can write code, run the real app, verify the UI end-to-end, collect screenshots/videos/logs/perf evidence, and feed bugs, crashes, or performance findings back into the next fix iteration before a human reviews the PR.
27+
## Capabilities
2928

30-
![Sketch showing agent-device as the live app verification layer in the agentic development loop](./website/docs/public/agentic-development-loop.svg)
29+
- **Inspect** real app UI through compact accessibility snapshots, interactive refs like `@e3`, selectors, and React Native component trees.
30+
- **Interact** by opening apps, tapping, typing, scrolling, performing gestures, waiting, asserting state, handling alerts, and closing sessions.
31+
- **Capture evidence** with screenshots, videos, logs, traces, network traffic, performance samples, crash context, and React profiles.
32+
- **Replay workflows** by recording `.ad` scripts for local runs, CI, and repeatable e2e checks.
33+
- **Run across platforms** on iOS, Android, tvOS, Android TV, macOS, Linux, simulators, emulators, desktops, and physical devices.
3134

32-
If you know Vercel's [agent-browser](https://github.com/vercel-labs/agent-browser), this is the same idea for apps and devices.
35+
Unlike traditional mobile automation workflows around Appium, Detox, or Maestro, `agent-device` is optimized for AI agents that need to explore, verify, debug, profile, and then turn useful flows into replayable checks. It is a CLI, so teams can use it alongside existing mobile testing and CI workflows.
3336

34-
Use it for AI mobile testing, AI QA for React Native and Expo apps, iOS Simulator automation, Android Emulator automation, tvOS/Android TV checks, and desktop app verification from coding agents. Humans install and configure `agent-device`; agents run the workflows.
37+
## Use Cases
3538

36-
![agent-device demo showing Codex using agent-device to create a new contact in the iOS Contacts app from a simple prompt](./website/docs/public/agent-device-contacts.gif)
39+
- Verify mobile changes on real devices, simulators, and emulators before review or merge.
40+
- Give AI coding agents a real app feedback loop while they implement features.
41+
- Debug regressions with screenshots, logs, traces, network evidence, and crash context.
42+
- Profile performance issues with CPU/memory samples and React render profiles when needed.
43+
- Turn exploratory app interactions into replayable e2e checks for CI.
44+
- Use one agent workflow across native iOS, Android, Expo, Flutter, React Native, TV, and desktop apps.
3745

38-
Demo: Codex uses `agent-device` to inspect iOS Contacts through accessibility snapshots, interact with visible UI, and create a contact from a simple prompt.
46+
![Sketch showing agent-device as the live app verification layer in the agentic development loop](./website/docs/public/agentic-development-loop.svg)
3947

4048
## Quick Start
4149

42-
Install the CLI first:
50+
Install the CLI:
4351

4452
```bash
4553
npm install -g agent-device@latest
4654
agent-device --version
4755
agent-device help workflow
4856
```
4957

50-
The CLI help is the source of truth for agents and is shipped with the installed version. Skills are optional but recommended when your agent runtime supports them: they auto-route device, React DevTools, and dogfood tasks to the right `agent-device help <topic>` page and verify the CLI is new enough before acting.
51-
52-
If you install skills separately, keep the CLI on `agent-device >= 0.14.0`. Older CLIs do not include the workflow help topics that the router skills expect.
53-
54-
### AI Agent Entry Points
58+
The installed CLI help is the source of truth for agents. Start with `agent-device help workflow`, then follow the topic-specific help when a task needs dogfooding, debugging, replay, or React Native profiling.
5559

56-
- **Agent + terminal**: in Cursor, Codex, Claude Code, Windsurf, and similar clients, run `agent-device` in the integrated terminal. Start planning with `agent-device help workflow`; CLI help is authoritative.
57-
- **Skills or rules**: install the skill with `npx skills add callstackincubator/agent-device`, use the bundled [agent-device skill](skills/agent-device/SKILL.md), or mirror it as a thin project rule, so the agent checks the installed version and reads `agent-device help workflow` before acting. Use `agent-device help react-native` for React Native apps, overlays, Metro/Fast Refresh blockers, and routing to React DevTools or debugging evidence.
58-
- **MCP router**: use `agent-device mcp` when an MCP-aware client needs to discover the CLI package, install command, version check, and first help command. MCP is discovery-only; device automation still runs through terminal CLI commands.
60+
Prerequisites depend on the target platform: Node.js 22+, Xcode for iOS/tvOS/macOS targets, Android SDK + ADB for Android, and macOS Accessibility permission for desktop automation. See [Installation](https://incubator.callstack.com/agent-device/docs/installation) for platform setup.
5961

60-
For client-specific setup, see [AI Agent Setup](https://incubator.callstack.com/agent-device/docs/agent-setup). For agent-readable docs, use [llms-full.txt](https://incubator.callstack.com/agent-device/llms-full.txt).
61-
62-
### MCP Router
63-
64-
`agent-device` ships an official stdio MCP router for discovery-oriented clients. It exposes only a `status` tool that returns structured CLI handoff guidance: npm package name, installed version, CLI command name, install command, verify command, starting help command, and an explicit note that automation happens through the CLI.
65-
66-
MCP clients must not use this server as a device automation surface or generic shell runner. If the CLI is missing, agents should ask a human before installing or updating packages, then verify with `agent-device --version` and start with `agent-device help workflow`.
67-
68-
Paste one of these into clients that accept `mcpServers`, such as Cursor project `.cursor/mcp.json` or user-level MCP settings.
69-
70-
<details>
71-
<summary>Global install MCP config</summary>
72-
73-
```json
74-
{
75-
"mcpServers": {
76-
"agent-device": {
77-
"command": "agent-device",
78-
"args": ["mcp"]
79-
}
80-
}
81-
}
82-
```
83-
84-
</details>
85-
86-
<details>
87-
<summary>No global install MCP config</summary>
88-
89-
```json
90-
{
91-
"mcpServers": {
92-
"agent-device": {
93-
"command": "npx",
94-
"args": ["-y", "agent-device@<reviewed-version>", "mcp"]
95-
}
96-
}
97-
}
98-
```
99-
100-
</details>
101-
102-
Registry metadata uses MCP name `io.github.callstackincubator/agent-device`, npm package `agent-device`, stdio transport, `mcpName` package verification, `server.json`, and `smithery.yaml`.
62+
Try the basic loop:
10363

10464
```bash
105-
npm install -g agent-device@latest
106-
agent-device --version
107-
agent-device help
108-
```
109-
110-
`agent-device` performs a lightweight background upgrade check for interactive CLI runs and, when a newer package is available, suggests a global reinstall command. Updating the package also refreshes the bundled `skills/` shipped with the CLI.
111-
112-
Prerequisites: Node.js 22+, Xcode for iOS/tvOS/macOS targets, Android SDK + ADB for Android, and macOS Accessibility permission for desktop automation. See [Installation](https://incubator.callstack.com/agent-device/docs/installation).
113-
114-
Try the loop.
115-
116-
```bash
117-
# Find the app.
65+
# Find an app.
11866
agent-device apps --platform ios
11967
agent-device apps --platform android
12068

@@ -127,111 +75,61 @@ agent-device snapshot -i
12775
# @e2 [button] "Sign In"
12876
# @e3 [text-field] "Email"
12977

130-
# Act, capture a screenshot, and close.
131-
agent-device fill @e3 "test"
78+
# Act, capture evidence, and close.
79+
agent-device fill @e3 "test@example.com"
13280
agent-device screenshot ./artifacts/settings.png
13381
agent-device close
13482
```
13583

136-
Snapshots assign refs like `@e1`, `@e2`, and `@e3` to current-screen elements. Refs from the default snapshot are immediately actionable; for hidden content, scroll and re-snapshot.
137-
138-
### First 5 Minutes: Expo Test App
139-
140-
Use the bundled Expo fixture when you want a concrete first agent run with setup checks, screenshots, replay, and performance evidence. This path requires a repo checkout because `examples/test-app` and the `pnpm test-app:*` scripts are not included in the published npm package.
141-
142-
```bash
143-
git clone https://github.com/callstackincubator/agent-device.git
144-
cd agent-device
145-
```
146-
147-
First terminal:
84+
Snapshots assign refs like `@e1`, `@e2`, and `@e3` to elements on the current screen. Refs from the latest snapshot are immediately actionable; after scrolling or changing screens, take a fresh snapshot.
14885

149-
```bash
150-
pnpm test-app:install
151-
cd examples/test-app
152-
npx expo-doctor@latest
153-
cd ../..
154-
pnpm test-app:ios
155-
# or: pnpm test-app:android
156-
```
86+
## Next Steps
15787

158-
Then give your agent this prompt:
159-
160-
```text
161-
Use agent-device to dogfood the bundled Expo app and produce an evidence-backed report.
162-
163-
Setup:
164-
- Read `agent-device help workflow`, `agent-device help dogfood`, `agent-device help debugging`, and `agent-device help react-devtools` before planning commands.
165-
- Confirm the test app setup commands were run: `pnpm test-app:install`, `cd examples/test-app && npx expo-doctor@latest`, then `pnpm test-app:ios` or `pnpm test-app:android`.
166-
- If Metro prints an Expo URL, prefer opening the shell with that URL. On iOS use `agent-device open "Expo Go" <url> --platform ios`; on Android use the visible Expo/dev-client target or URL. Confirm the app UI with `snapshot -i`.
167-
168-
Run:
169-
- Create `./dogfood-output/screenshots`, `./dogfood-output/videos`, `./dogfood-output/traces`, `./dogfood-output/perf`, and `./dogfood-output/replays`.
170-
- Open a named session `expo-qa` and save a replay script to `./dogfood-output/replays/expo-test.ad`.
171-
- Use command shapes like `agent-device --session expo-qa open "Expo Go" <url> --platform ios --save-script ./dogfood-output/replays/expo-test.ad`, `agent-device --session expo-qa screenshot ./dogfood-output/screenshots/home.png`, `agent-device --session expo-qa perf --json > ./dogfood-output/perf/baseline.json`, and `agent-device --session expo-qa record start ./dogfood-output/videos/checkout.mp4`.
172-
- Capture a baseline `snapshot -i`, screenshot, and `perf --json` sample.
173-
- Exercise Home, Catalog, product detail, Checkout, and Settings. Re-snapshot after each mutation and use refs/selectors from fresh snapshots.
174-
- Capture at least one overlay-ref screenshot, one normal screenshot, one short video recording for a meaningful flow, logs marks around any issue, and trace output if a runtime symptom needs diagnostics.
175-
- Run focused performance checks: compare `perf --json` before and after a navigation or form flow; if React DevTools connects, use one bounded first-pass profile survey (`slow --limit 5`, `rerenders --limit 5`, and `timeline --limit 20` only when timing matters), then drill into a specific `@c` ref with `profile report`. If it cannot connect, include the status and continue.
176-
- Close the session so the `.ad` replay is written.
177-
178-
Report:
179-
- Write `./dogfood-output/report.md`.
180-
- Link every screenshot, video, trace, log path, replay file, and performance artifact you used.
181-
- Include setup results, platform/device, Expo doctor outcome, coverage, severity counts, findings with repro commands, and a short performance section summarizing startup/CPU/memory/frame-health or React profile findings.
182-
- If no issues are found, report covered flows and residual risk instead of claiming the app is bug-free.
183-
```
88+
- **Set up your agent**: run the CLI from Cursor, Codex, Claude Code, Windsurf, or another agent terminal. For skills, rules, MCP discovery, and client-specific setup, see [AI Agent Setup](https://incubator.callstack.com/agent-device/docs/agent-setup).
89+
- **Try the sample app**: clone the repo and run the bundled Expo fixture when you want a guided first dogfood run with screenshots, replay, and performance evidence. See [Quick Start](https://incubator.callstack.com/agent-device/docs/quick-start).
90+
- **Go deeper**: use [Commands](https://incubator.callstack.com/agent-device/docs/commands), [Replay & E2E](https://incubator.callstack.com/agent-device/docs/replay-e2e), and [Debugging & Profiling](https://incubator.callstack.com/agent-device/docs/debugging-profiling) for production workflows.
18491

18592
## Where To Run agent-device
18693

18794
| Path | Best for | Start with |
18895
| --- | --- | --- |
18996
| Local | Exploration, debugging, and development loops on simulators, emulators, physical devices, macOS apps, and Linux desktop targets. | Follow the Quick Start. |
190-
| CI/CD | Automated PR and merge validation with replay scripts and captured artifacts. | Start with the [EAS workflow template](https://github.com/callstackincubator/eas-agent-device/blob/main/.eas/workflows/agent-qa-mobile.yml). GitHub Actions template coming soon. |
97+
| CI/CD | Automated PR and merge validation with replay scripts and captured artifacts. | Try the [EAS workflow template](https://github.com/callstackincubator/eas-agent-device/blob/main/.eas/workflows/agent-qa-mobile.yml). GitHub Actions template coming soon. |
19198
| Cloud / remote execution | Linux runners, managed devices, and remote execution. | Use [Agent Device Cloud](https://agent-device.dev/cloud), see [Commands](https://incubator.callstack.com/agent-device/docs/commands) for remote profiles, or [contact Callstack](mailto:hello@callstack.com) for team-scale QA. |
19299

193-
## Capabilities
194-
195-
- **Platforms**: iOS, Android, tvOS, Android TV, macOS, and Linux. Real devices and simulators are supported.
196-
- **Agent-native UI model**: token-efficient accessibility snapshots, current-screen refs for exploration, selectors for durable replay, and skill-tested workflow guidance.
197-
- **Capture and debug**: screenshots, video, logs, network traffic, CPU/memory/performance data, crash-related logs, accessibility snapshots, and React render profiles.
198-
- **Produce**: replayable `.ad` scripts (recorded replay files that run locally or in CI), e2e test runs, snapshot and screenshot diffs, and debugging artifacts.
199-
- **React Native and Expo**: component tree inspection, props/state/hooks, and render profiling.
200-
- **MCP boundary**: discovery and help over MCP; app/device control through the CLI for explicit, auditable commands.
201-
- **License**: MIT. Free to use.
202-
203100
## How It Works
204101

205-
`agent-device` runs session-aware commands through platform backends: XCTest for iOS and tvOS, ADB plus the Android snapshot helper for Android, a local helper for macOS desktop automation, and AT-SPI for Linux desktop targets. See [Introduction](https://incubator.callstack.com/agent-device/docs/introduction) and [Commands](https://incubator.callstack.com/agent-device/docs/commands) for platform details.
102+
`agent-device` runs session-aware commands through platform backends: XCTest for iOS and tvOS, ADB plus the Android snapshot helper for Android, a local helper for macOS desktop automation, and AT-SPI for Linux desktop targets.
206103

207104
Node consumers can use the typed client and public subpaths for bridge integrations. `agent-device/android-adb` exposes the Android ADB provider contract, logcat/clipboard/keyboard/app helpers, and port reverse management.
208105

106+
## FAQ
107+
108+
### What is agent-device?
109+
110+
`agent-device` is a device automation CLI for AI mobile app testing. It lets AI agents verify real apps on iOS, Android, TV, desktop, simulators, emulators, and physical devices.
111+
112+
### Does it work with React Native, Expo, Flutter, and native apps?
113+
114+
Yes. `agent-device` works with native iOS and Android apps, Expo apps, Flutter apps, React Native apps, TV apps, and desktop apps that run on supported targets.
115+
116+
### How is it different from Appium, Detox, or Maestro?
117+
118+
Appium, Detox, and Maestro are traditional mobile automation frameworks. `agent-device` is optimized for AI agents that need to inspect app state, interact semantically, capture evidence, debug, profile, and turn useful explorations into replayable checks.
119+
209120
## Used By
210121

211122
Used by teams and developers at Callstack, Expensify, Shopify, Kindred, Total Wine & More, LegendList, HerLyfe, App & Flow, and more.
212123

213124
## Documentation
214125

215-
- [Installation](https://incubator.callstack.com/agent-device/docs/installation)
216-
- [AI Agent Setup](https://incubator.callstack.com/agent-device/docs/agent-setup)
217-
- [Typed Client](https://incubator.callstack.com/agent-device/docs/client-api)
218-
- [Commands](https://incubator.callstack.com/agent-device/docs/commands)
219-
- [Replay & E2E](https://incubator.callstack.com/agent-device/docs/replay-e2e)
220-
- [Security & Trust](https://incubator.callstack.com/agent-device/docs/security-trust)
221-
- [Known limitations](https://incubator.callstack.com/agent-device/docs/known-limitations)
222-
- [llms-full.txt](https://incubator.callstack.com/agent-device/llms-full.txt)
223-
224-
Agent integration:
225-
226-
- [agent-device skill](skills/agent-device/SKILL.md)
227-
- [dogfood skill](skills/dogfood/SKILL.md)
228-
- MCP router: `agent-device mcp`
229-
- [agent-device skill on ClawHub](https://clawhub.ai/okwasniewski/agent-device)
126+
- [Docs](https://incubator.callstack.com/agent-device/)
127+
- [Agent-readable docs](https://incubator.callstack.com/agent-device/llms-full.txt)
230128

231129
## Contributing
232130

233131
See [CONTRIBUTING.md](CONTRIBUTING.md).
234132

235133
## Made at Callstack
236134

237-
agent-device is open source and MIT licensed. Visit [agent-device.dev](https://agent-device.dev/), try the [EAS workflow template](https://github.com/callstackincubator/eas-agent-device/blob/main/.eas/workflows/agent-qa-mobile.yml), read the [incubator docs](https://incubator.callstack.com/agent-device/), or contact us at hello@callstack.com.
135+
`agent-device` is open source and MIT licensed. Visit [agent-device.dev](https://agent-device.dev/), try the [EAS workflow template](https://github.com/callstackincubator/eas-agent-device/blob/main/.eas/workflows/agent-qa-mobile.yml), read the [incubator docs](https://incubator.callstack.com/agent-device/), or contact us at hello@callstack.com.

0 commit comments

Comments
 (0)