Skip to content

Commit d6b8b12

Browse files
authored
docs: improve agent discoverability for observability (#431)
* 0.12.9 * docs: improve agent discoverability for observability * docs: add debugging and profiling guide
1 parent bb24567 commit d6b8b12

12 files changed

Lines changed: 135 additions & 16 deletions

File tree

README.md

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88

99
# agent-device
1010

11-
`agent-device` is a CLI for UI automation on iOS, tvOS, macOS, Android, and AndroidTV. It is designed for agent-driven workflows: inspect the UI, act on it deterministically, and keep that work session-aware and replayable.
11+
`agent-device` is a CLI for UI automation and app observability on iOS, tvOS, macOS, Android, and AndroidTV. It is built for agent-driven workflows: inspect the UI, interact deterministically, collect logs/network/perf evidence when behavior breaks, and keep the whole flow session-aware and replayable.
1212

1313
If you know Vercel's [agent-browser](https://github.com/vercel-labs/agent-browser), this project applies the same broad idea to mobile apps and devices.
1414

@@ -19,18 +19,26 @@ If you know Vercel's [agent-browser](https://github.com/vercel-labs/agent-browse
1919
- Give agents a practical way to understand mobile UI state through structured snapshots.
2020
- Keep automation flows token-efficient enough for real agent loops.
2121
- Make common interactions reliable enough for repeated automation runs.
22+
- Make debugging evidence easy to collect through logs, network inspection, and performance snapshots.
2223
- Keep automation grounded in sessions, selectors, and replayable flows instead of one-off scripts.
2324

2425
## Core Ideas
2526

2627
- Sessions: open a target once, interact within that session, then close it cleanly.
2728
- Snapshots: inspect the current accessibility tree in a compact form and get current-screen refs for exploration.
2829
- Refs vs selectors: use refs for discovery, use selectors for durable replay and assertions.
30+
- Observability: collect session logs, inspect recent HTTP traffic with `network dump`, and sample CPU/memory with `perf`.
2931
- Tests: run deterministic `.ad` scripts as a light e2e test suite.
3032
- Replay scripts: save `.ad` flows with `--save-script`, replay one script with `replay`, or run a folder/glob as a serial suite with `test`.
3133
`test` supports metadata-aware retries up to 3 additional attempts, per-test timeouts, flaky pass reporting, and runner-managed artifacts under `.agent-device/test-artifacts` by default. Each attempt writes `replay.ad` and `result.txt`; failed attempts also keep copied logs and artifacts when available.
3234
- Human docs vs agent skills: docs explain the system for people; skills provide compact operating guidance for agents.
3335

36+
## Complementary Tooling
37+
38+
Use `agent-device` for on-device UI automation, screenshots/recordings, app logs, network inspection, and performance snapshots.
39+
40+
When the task needs the React component tree, props, state, hooks, or render profiling, pair it with the complementary [`agent-react-devtools`](https://github.com/callstackincubator/agent-react-devtools) project. The two tools solve different layers of the same debugging workflow.
41+
3442
## Command Flow
3543

3644
The canonical loop is:

package.json

Lines changed: 12 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
{
22
"name": "agent-device",
3-
"version": "0.12.8",
4-
"description": "Unified control plane for physical and virtual devices via an agent-driven CLI.",
3+
"version": "0.12.9",
4+
"description": "Agent-driven CLI for mobile UI automation, network inspection, and performance diagnostics across iOS, Android, tvOS, and macOS.",
55
"license": "MIT",
66
"author": "Callstack",
77
"homepage": "https://agent-device.dev/",
@@ -128,11 +128,20 @@
128128
"agent",
129129
"device",
130130
"cli",
131+
"automation",
131132
"adb",
132133
"simctl",
133134
"devicectl",
134135
"ios",
135-
"android"
136+
"android",
137+
"tvos",
138+
"macos",
139+
"react-native",
140+
"observability",
141+
"diagnostics",
142+
"network",
143+
"profiling",
144+
"performance"
136145
],
137146
"dependencies": {
138147
"fast-xml-parser": "^5.5.10",

skills/agent-device/SKILL.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
---
22
name: agent-device
3-
description: Automates interactions for Apple-platform apps (iOS, tvOS, macOS) and Android devices. Use when navigating apps, taking snapshots/screenshots, tapping, typing, scrolling, or extracting UI info across mobile, TV, and desktop targets.
3+
description: Automates interactions for Apple-platform apps (iOS, tvOS, macOS) and Android devices. Use when navigating apps, taking snapshots/screenshots, tapping, typing, scrolling, extracting UI info, or collecting logs, network inspection, and perf snapshots across mobile, TV, and desktop targets.
44
---
55

66
# agent-device
@@ -71,3 +71,4 @@ Use this skill as a router with mandatory defaults. Read this file first. For no
7171
- Need desktop surfaces, menu bar behavior, or macOS-specific interaction rules: [references/macos-desktop.md](references/macos-desktop.md)
7272
- Need remote HTTP transport, `connect --remote-config`, or tenant leases on a remote macOS host: [references/remote-tenancy.md](references/remote-tenancy.md)
7373
This includes remote React Native runs where `agent-device` now prepares Metro locally and manages the local Metro companion tunnel automatically.
74+
- Need the React component tree, props, state, hooks, or render profiling: pair `agent-device` with the complementary [`agent-react-devtools`](https://github.com/callstackincubator/agent-react-devtools) project when available.

skills/agent-device/references/debugging.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,8 @@
44

55
Open this file when the task turns into failure triage, logs, network inspection, permission prompts, setup trouble, or unstable session behavior.
66

7+
If the debugging task needs the React component tree, props, state, hooks, or render profiling, pair `agent-device` with the complementary [`agent-react-devtools`](https://github.com/callstackincubator/agent-react-devtools) project instead of trying to infer those internals from the accessibility tree or app logs alone.
8+
79
## Main commands to reach for first
810

911
- `logs clear --restart`

skills/agent-device/references/exploration.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ Open this file when the app or screen is already running and you need to discove
2020
- User asks what is visible on screen: `snapshot`
2121
- User asks for exact text from a known target: `get text`
2222
- User asks you to tap, type, or choose an element: `snapshot -i`, then act
23+
- User asks for the React component tree, props/state/hooks, or render profiling: pair `agent-device` with the complementary [`agent-react-devtools`](https://github.com/callstackincubator/agent-react-devtools) project
2324
- React Native dev or debug build shows warning/error UI: capture enough evidence to identify it, dismiss it if it is not the requested behavior, then continue the flow and report it in the summary
2425
- The on-screen keyboard is blocking the next step: `keyboard dismiss`; on iOS do this only while an app session is active, and use `keyboard status|get` only on Android
2526
- UI does not expose the answer: say so plainly; do not browse or force the app into a new state unless asked

skills/agent-device/references/verification.md

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -129,5 +129,6 @@ agent-device perf --json
129129
- `startup` is command round-trip timing around `open`.
130130
- It is not true first-frame or first-interactive telemetry.
131131
- Android app sessions also expose `memory` (`dumpsys meminfo`) and `cpu` (`dumpsys cpuinfo`) snapshots when the session has an app package context.
132-
- Apple app sessions on macOS and iOS simulators also expose `memory` and `cpu` process snapshots when the session has an app bundle ID.
133-
- `fps` is still unavailable, and physical iOS devices still leave `memory` and `cpu` unavailable in this release.
132+
- Apple app sessions on macOS, iOS simulators, and physical iOS devices also expose `memory` and `cpu` process snapshots when the session has an app bundle ID.
133+
- On physical iOS devices, sampling uses a short `xcrun xctrace` Activity Monitor capture, so keep the device unlocked, connected, and the app active in the foreground while sampling.
134+
- `fps` is still unavailable in this release.

skills/dogfood/SKILL.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -166,6 +166,7 @@ agent-device --session {SESSION} close
166166
- Re-snapshot after any mutation (navigation, modal, list update, form submit).
167167
- Use `fill` for clear-then-type semantics; use `type` for incremental typing behavior checks.
168168
- Keep logs optional and targeted: enable/read app logs only when useful for diagnosis.
169+
- If the issue appears rooted in React internals rather than device/app runtime behavior, pair `agent-device` with the complementary [`agent-react-devtools`](https://github.com/callstackincubator/agent-react-devtools) project for component-tree or render-profiling inspection.
169170
- Never read source code of the app under test; findings must come from observed runtime behavior.
170171
- Write each issue immediately to avoid losing evidence.
171172
- Never delete screenshots/videos/report artifacts during a session.

website/docs/docs/_meta.json

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,11 @@
1414
"type": "file",
1515
"label": "Quick Start"
1616
},
17+
{
18+
"name": "debugging-profiling",
19+
"type": "file",
20+
"label": "Debugging & Profiling"
21+
},
1722
{
1823
"name": "client-api",
1924
"type": "file",
Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
---
2+
title: Debugging & Profiling
3+
---
4+
5+
# Debugging & Profiling
6+
7+
Use `agent-device` when the task moves past UI automation and you need runtime evidence from the app or device layer.
8+
9+
## What `agent-device` covers well
10+
11+
- Session app logs for targeted debugging windows
12+
- Network inspection from recent HTTP(s) entries in app logs via `network dump`
13+
- Performance snapshots with `perf` / `metrics`
14+
- Screenshots, recordings, and replayable repro flows
15+
16+
## What to use instead
17+
18+
If the task needs the React component tree, props, state, hooks, or render profiling, pair `agent-device` with the complementary [`agent-react-devtools`](https://github.com/callstackincubator/agent-react-devtools) project.
19+
20+
`agent-device` is centered on the device and app runtime layer. `agent-react-devtools` is the better fit for React internals.
21+
22+
## Fast path
23+
24+
```bash
25+
agent-device open MyApp --platform ios
26+
agent-device logs clear --restart
27+
agent-device network dump 25 --include headers
28+
agent-device perf --json
29+
agent-device logs path
30+
```
31+
32+
Use this flow when you need a clean repro window with logs, recent network activity, and a quick perf sample from the active app session.
33+
34+
## Core commands
35+
36+
### Logs
37+
38+
```bash
39+
agent-device logs start
40+
agent-device logs stop
41+
agent-device logs clear --restart
42+
agent-device logs path
43+
agent-device logs doctor
44+
agent-device logs mark "before submit"
45+
```
46+
47+
- Logging is off by default; enable it only for focused debugging windows.
48+
- Prefer `logs clear --restart` for clean repro loops.
49+
- Use `logs path` and then grep the file instead of loading whole logs into agent context.
50+
51+
### Network inspection
52+
53+
```bash
54+
agent-device network dump 25
55+
agent-device network dump 25 --include headers
56+
agent-device network dump 25 --include all
57+
```
58+
59+
- `network dump` parses recent HTTP(s) entries from the session app log.
60+
- `network log` is an alias for `network dump`.
61+
- Parsed results depend on what the app emits into the platform log backend.
62+
63+
### Performance snapshots
64+
65+
```bash
66+
agent-device perf --json
67+
agent-device metrics --json
68+
```
69+
70+
- `perf` returns session-scoped startup and, where supported, CPU and memory samples.
71+
- Startup is measured around the `open` command; it is not first-frame instrumentation.
72+
- CPU and memory availability depends on platform and whether the active session is bound to an app/package.
73+
74+
## Where to go deeper
75+
76+
- Full command reference: [Commands](/docs/commands)
77+
- Typed client observability APIs: [Typed Client](/docs/client-api)
78+
- Session behavior and lifecycle: [Sessions](/docs/sessions)

website/docs/docs/introduction.md

Lines changed: 14 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -9,22 +9,26 @@ title: Introduction
99
- Accessibility snapshots for UI understanding
1010
- Deterministic interactions (tap, type, scroll)
1111
- Session-aware workflows and replay
12+
- Session logs and network inspection for debugging broken flows
13+
- Performance snapshots with `perf`/`metrics`, including CPU and memory data where supported
1214

13-
If you know `agent-browser`, this is the mobile-native counterpart for iOS/Android UI automation.
15+
If you know `agent-browser`, this is the mobile-native counterpart for iOS/Android UI automation and app-level observability.
1416
For exploratory QA and bug-hunting workflows, see `skills/dogfood/SKILL.md` in this repository.
17+
For React component trees, props/state/hooks, and render profiling, pair it with the complementary [`agent-react-devtools`](https://github.com/callstackincubator/agent-react-devtools) project.
1518

1619
## What it’s good at
1720

18-
- Capturing structured UI state for LLMs
19-
- Driving common UI actions with refs or semantic selectors
20-
- Replaying flows for regression checks
21+
- Exploring and driving app flows on real devices and simulators
22+
- Collecting debugging evidence through logs, network traffic, screenshots, recordings, and performance snapshots
23+
- Replaying successful flows as lightweight regression checks
2124

2225
## Platform support highlights
2326

2427
- iOS core runner commands: `snapshot`, `snapshot --diff`, `diff snapshot`, `wait`, `click`, `fill`, `get`, `is`, `find`, `press`, `long-press`, `focus`, `type`, `scroll`, `back`, `home`, `rotate`, `app-switcher`, `open` (app), `close`, `screenshot`, `apps`, `appstate`, `install`, `install-from-source`, `reinstall`, `trigger-app-event`.
2528
- iOS `appstate` is session-scoped on the selected target device.
2629
- iOS/tvOS simulator-only: `settings`, `push`, `clipboard`.
2730
- Apple simulators and macOS desktop app sessions: `alert`, `pinch`.
31+
- Session diagnostics: `logs` and `network dump` are available for debugging active app sessions, with network inspection based on recent HTTP(s) entries captured in the session app log.
2832
- Session performance metrics: `perf`/`metrics` is available on iOS, macOS, and Android. Startup timing comes from `open` command round-trip duration. Android app sessions and Apple app sessions on macOS, iOS simulators, or connected iOS devices also expose CPU and memory snapshots when an app identifier is available in the session.
2933
- iOS `record` supports simulators and physical devices.
3034
- Simulators use native `simctl io ... recordVideo`.
@@ -43,6 +47,12 @@ For exploratory QA and bug-hunting workflows, see `skills/dogfood/SKILL.md` in t
4347
3. iOS uses XCTest runner for snapshots and input on simulators and physical devices.
4448
4. Android uses ADB-based tooling.
4549

50+
## Complementary React tooling
51+
52+
`agent-device` is intentionally centered on the device/app layer: UI automation, screenshots/recordings, app logs, network inspection, and performance sampling.
53+
54+
When a debugging workflow needs React internals such as the component tree, props, state, hooks, or render profiling, use the complementary `agent-react-devtools` project alongside `agent-device` rather than as a replacement.
55+
4656
## Example
4757

4858
```bash

0 commit comments

Comments
 (0)