Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 20 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,8 @@ The project is in early development and considered experimental. Pull requests a

## Features
- Platforms: iOS (simulator + physical device core automation) and Android (emulator + device).
- Core commands: `open`, `back`, `home`, `app-switcher`, `press`, `longpress`, `focus`, `type`, `fill`, `scroll`, `scrollintoview`, `wait`, `alert`, `screenshot`, `close`, `reinstall`.
- Inspection commands: `snapshot` (accessibility tree), `appstate`, `apps`, `devices`.
- Core commands: `open`, `back`, `home`, `app-switcher`, `press`, `long-press`, `focus`, `type`, `fill`, `scroll`, `scrollintoview`, `wait`, `alert`, `screenshot`, `close`, `reinstall`.
- Inspection commands: `snapshot` (accessibility tree), `diff snapshot` (structural baseline diff), `appstate`, `apps`, `devices`.
- Device tooling: `adb` (Android), `simctl`/`devicectl` (iOS via Xcode).
- Minimal dependencies; TypeScript executed directly on Node 22+ (no build step).

Expand All @@ -39,9 +39,11 @@ Use `press` as the canonical tap command; `click` is an equivalent alias.
```bash
agent-device open Contacts --platform ios # creates session on iOS Simulator
agent-device snapshot
agent-device diff snapshot # first run initializes baseline
agent-device press @e5
agent-device fill @e6 "John"
agent-device fill @e7 "Doe"
agent-device diff snapshot # subsequent runs compare against previous baseline
agent-device press @e3
agent-device close
```
Expand Down Expand Up @@ -135,8 +137,8 @@ agent-device swipe 540 1500 540 500 120 --count 8 --pause-ms 30 --pattern ping-p
## Command Index
- `boot`, `open`, `close`, `reinstall`, `home`, `back`, `app-switcher`
- `batch`
- `snapshot`, `find`, `get`
- `press` (alias: `click`), `focus`, `type`, `fill`, `longpress`, `swipe`, `scroll`, `scrollintoview`, `pinch`, `is`
- `snapshot`, `diff snapshot`, `find`, `get`
- `press` (alias: `click`), `focus`, `type`, `fill`, `long-press`, `swipe`, `scroll`, `scrollintoview`, `pinch`, `is`
- `alert`, `wait`, `screenshot`
- `trace start`, `trace stop`
- `settings wifi|airplane|location on|off`
Expand All @@ -149,6 +151,20 @@ Notes:
- iOS snapshots use XCTest on simulators and physical devices.
- Scope snapshots with `-s "<label>"` or `-s @ref`.
- If XCTest returns 0 nodes (e.g., foreground app changed), agent-device fails explicitly.
- `diff snapshot` uses the same snapshot flags and compares the current capture with the previous session baseline, then updates baseline.

Diff snapshots:
- Run `diff snapshot` once to initialize baseline for the current session.
- Run `diff snapshot` again after UI changes to get unified-style output (`-` removed, `+` added, unchanged context).
- Use `--json` to get `{ mode, baselineInitialized, summary, lines }`.

Efficient snapshot usage:
- Default to `snapshot -i` for iterative agent loops.
- Add `-s "<label>"` (or `-s @ref`) for screen-local work to reduce payload size.
- Add `-d <depth>` when lower tree levels are not needed.
- Re-snapshot after UI mutations before reusing refs.
- Use `diff snapshot` for low-noise structural change verification between adjacent states.
- Reserve `--raw` for troubleshooting and parser/debug investigations.

Flags:
- `--version, -V` print version and exit
Expand Down
11 changes: 11 additions & 0 deletions skills/agent-device/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,10 +64,16 @@ agent-device snapshot -c # Compact output
agent-device snapshot -d 3 # Limit depth
agent-device snapshot -s "Camera" # Scope to label/identifier
agent-device snapshot --raw # Raw node output
agent-device diff snapshot # Structural diff against previous session baseline
```

XCTest is the iOS snapshot engine: fast, complete, and no Accessibility permission required.

Snapshot diff notes:
- First `diff snapshot` call initializes baseline for the current session.
- Subsequent `diff snapshot` calls compare current UI to prior baseline and then update baseline.
- Use this for compact change tracking between adjacent UI states.

### Find (semantic)

```bash
Expand Down Expand Up @@ -241,6 +247,11 @@ agent-device apps --platform android --user-installed
- Snapshot refs are the core mechanism for interactive agent flows.
- Use selectors for deterministic replay artifacts and assertions (e.g. in e2e test workflows).
- Prefer `snapshot -i` to reduce output size.
- Prefer scoped snapshots (`-s "<label>"` or `-s @ref`) for screen-local tasks.
- Add `-d <depth>` when only upper tree levels matter; avoid full-tree snapshots by default.
- Use `diff snapshot` after mutations to detect structural changes with less output than full re-read.
- Refresh refs immediately after navigation/modal/list mutations before issuing next ref-targeted action.
- Use `--raw` only for debugging parser/tree edge-cases; avoid it for normal agent loops due to size.
- On iOS, snapshots use XCTest and do not require Accessibility permission.
- If XCTest returns 0 nodes (foreground app changed), treat it as an explicit failure and retry the flow/app state.
- `open <app|url> [url]` can be used within an existing session to switch apps or open deep links.
Expand Down
16 changes: 16 additions & 0 deletions skills/agent-device/references/snapshot-refs.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,22 @@ agent-device snapshot -i -s "Camera"
agent-device snapshot -i -s @e3
```

## Diff snapshots (structural)

Use `diff snapshot` when you need compact state-change visibility between nearby UI states:

```bash
agent-device diff snapshot # First run initializes baseline
agent-device press @e5
agent-device diff snapshot # Shows +/− structural lines vs prior baseline
```

Efficient pattern:
- Initialize once at a stable point.
- Mutate UI (`press`, `fill`, `swipe`).
- Run `diff snapshot` to confirm expected change shape.
- Re-run full/scoped `snapshot` only when you need fresh refs for next step selection.

## Troubleshooting

- Ref not found: re-snapshot.
Expand Down
96 changes: 96 additions & 0 deletions src/__tests__/cli-diff.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import { runCli } from '../cli.ts';
import type { DaemonRequest, DaemonResponse } from '../daemon-client.ts';

class ExitSignal extends Error {
public readonly code: number;

constructor(code: number) {
super(`EXIT_${code}`);
this.code = code;
}
}

type RunResult = {
code: number | null;
stdout: string;
stderr: string;
calls: Omit<DaemonRequest, 'token'>[];
};

async function runCliCapture(argv: string[]): Promise<RunResult> {
let stdout = '';
let stderr = '';
let code: number | null = null;
const calls: Array<Omit<DaemonRequest, 'token'>> = [];

const originalExit = process.exit;
const originalStdoutWrite = process.stdout.write.bind(process.stdout);
const originalStderrWrite = process.stderr.write.bind(process.stderr);

(process as any).exit = ((nextCode?: number) => {
throw new ExitSignal(nextCode ?? 0);
}) as typeof process.exit;
(process.stdout as any).write = ((chunk: unknown) => {
stdout += String(chunk);
return true;
}) as typeof process.stdout.write;
(process.stderr as any).write = ((chunk: unknown) => {
stderr += String(chunk);
return true;
}) as typeof process.stderr.write;

const sendToDaemon = async (req: Omit<DaemonRequest, 'token'>): Promise<DaemonResponse> => {
calls.push(req);
return {
ok: true,
data: {
mode: 'snapshot',
baselineInitialized: false,
summary: { additions: 1, removals: 1, unchanged: 1 },
lines: [
{ kind: 'unchanged', text: '@e2 [window]' },
{ kind: 'removed', text: ' @e3 [text] "67"' },
{ kind: 'added', text: ' @e3 [text] "134"' },
],
},
};
};

try {
await runCli(argv, { sendToDaemon });
} catch (error) {
if (error instanceof ExitSignal) code = error.code;
else throw error;
} finally {
process.exit = originalExit;
process.stdout.write = originalStdoutWrite;
process.stderr.write = originalStderrWrite;
}

return { code, stdout, stderr, calls };
}

test('diff snapshot renders human-readable unified diff text', async () => {
const result = await runCliCapture(['diff', 'snapshot']);
assert.equal(result.code, null);
assert.equal(result.calls.length, 1);
assert.match(result.stdout, /^@e2 \[window\]/m);
assert.match(result.stdout, /^- @e3 \[text\] "67"$/m);
assert.match(result.stdout, /^\+ @e3 \[text\] "134"$/m);
assert.match(result.stdout, /1 additions, 1 removals, 1 unchanged/);
assert.equal(result.stderr, '');
});

test('diff snapshot --json passes daemon payload through unchanged', async () => {
const result = await runCliCapture(['diff', 'snapshot', '--json']);
assert.equal(result.code, null);
assert.equal(result.calls.length, 1);
const payload = JSON.parse(result.stdout);
assert.equal(payload.success, true);
assert.equal(payload.data.mode, 'snapshot');
assert.equal(payload.data.baselineInitialized, false);
assert.equal(Array.isArray(payload.data.lines), true);
assert.equal(result.stderr, '');
});
7 changes: 6 additions & 1 deletion src/cli.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import { parseArgs, toDaemonFlags, usage, usageForCommand } from './utils/args.ts';
import { asAppError, AppError, normalizeError } from './utils/errors.ts';
import { formatSnapshotText, printHumanError, printJson } from './utils/output.ts';
import { formatSnapshotDiffText, formatSnapshotText, printHumanError, printJson } from './utils/output.ts';
import { readVersion } from './utils/version.ts';
import { pathToFileURL } from 'node:url';
import { sendToDaemon } from './daemon-client.ts';
Expand Down Expand Up @@ -189,6 +189,11 @@ export async function runCli(argv: string[], deps: CliDeps = DEFAULT_CLI_DEPS):
if (logTailStopper) logTailStopper();
return;
}
if (command === 'diff' && positionals[0] === 'snapshot') {
process.stdout.write(formatSnapshotDiffText((response.data ?? {}) as Record<string, unknown>));
if (logTailStopper) logTailStopper();
return;
}
if (command === 'get') {
const sub = positionals[0];
if (sub === 'text') {
Expand Down
1 change: 1 addition & 0 deletions src/core/__tests__/capabilities.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ test('core commands support iOS simulator, iOS device, and Android', () => {
'boot',
'click',
'close',
'diff',
'fill',
'find',
'focus',
Expand Down
1 change: 1 addition & 0 deletions src/core/capabilities.ts
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ const COMMAND_CAPABILITY_MATRIX: Record<string, CommandCapability> = {
click: { ios: { simulator: true, device: true }, android: { emulator: true, device: true, unknown: true } },
close: { ios: { simulator: true, device: true }, android: { emulator: true, device: true, unknown: true } },
fill: { ios: { simulator: true, device: true }, android: { emulator: true, device: true, unknown: true } },
diff: { ios: { simulator: true, device: true }, android: { emulator: true, device: true, unknown: true } },
find: { ios: { simulator: true, device: true }, android: { emulator: true, device: true, unknown: true } },
focus: { ios: { simulator: true, device: true }, android: { emulator: true, device: true, unknown: true } },
get: { ios: { simulator: true, device: true }, android: { emulator: true, device: true, unknown: true } },
Expand Down
99 changes: 99 additions & 0 deletions src/daemon/__tests__/snapshot-diff.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import { attachRefs, type RawSnapshotNode } from '../../utils/snapshot.ts';
import { buildSnapshotDiff } from '../snapshot-diff.ts';

function nodes(raw: RawSnapshotNode[]) {
return attachRefs(raw);
}

test('buildSnapshotDiff reports unchanged lines when snapshots are equal', () => {
const previous = nodes([
{ index: 0, depth: 0, type: 'XCUIElementTypeWindow' },
{ index: 1, depth: 1, type: 'XCUIElementTypeButton', label: 'Increment' },
]);
const current = nodes([
{ index: 0, depth: 0, type: 'XCUIElementTypeWindow' },
{ index: 1, depth: 1, type: 'XCUIElementTypeButton', label: 'Increment' },
]);

const diff = buildSnapshotDiff(previous, current);
assert.equal(diff.summary.additions, 0);
assert.equal(diff.summary.removals, 0);
assert.equal(diff.summary.unchanged, 2);
assert.deepEqual(diff.lines.map((line) => line.kind), ['unchanged', 'unchanged']);
});

test('buildSnapshotDiff reports added and removed lines', () => {
const previous = nodes([
{ index: 0, depth: 0, type: 'XCUIElementTypeWindow' },
{ index: 1, depth: 1, type: 'XCUIElementTypeStaticText', label: '67' },
{ index: 2, depth: 1, type: 'XCUIElementTypeButton', label: 'Increment' },
]);
const current = nodes([
{ index: 0, depth: 0, type: 'XCUIElementTypeWindow' },
{ index: 1, depth: 1, type: 'XCUIElementTypeStaticText', label: '134' },
{ index: 2, depth: 1, type: 'XCUIElementTypeButton', label: 'Increment' },
]);

const diff = buildSnapshotDiff(previous, current);
assert.equal(diff.summary.additions, 1);
assert.equal(diff.summary.removals, 1);
assert.equal(diff.summary.unchanged, 2);
assert.deepEqual(diff.lines.map((line) => line.kind), ['unchanged', 'removed', 'added', 'unchanged']);
});

test('buildSnapshotDiff treats value changes as remove plus add', () => {
const previous = nodes([{ index: 0, depth: 0, type: 'XCUIElementTypeTextField', label: 'Amount', value: '67' }]);
const current = nodes([{ index: 0, depth: 0, type: 'XCUIElementTypeTextField', label: 'Amount', value: '134' }]);

const diff = buildSnapshotDiff(previous, current);
assert.equal(diff.summary.additions, 1);
assert.equal(diff.summary.removals, 1);
assert.equal(diff.summary.unchanged, 0);
assert.deepEqual(diff.lines.map((line) => line.kind), ['removed', 'added']);
});

test('buildSnapshotDiff preserves surrounding context ordering', () => {
const previous = nodes([
{ index: 0, depth: 0, type: 'XCUIElementTypeWindow' },
{ index: 1, depth: 1, type: 'XCUIElementTypeStaticText', label: 'Count' },
{ index: 2, depth: 1, type: 'XCUIElementTypeStaticText', label: '67' },
{ index: 3, depth: 1, type: 'XCUIElementTypeButton', label: 'Increment' },
]);
const current = nodes([
{ index: 0, depth: 0, type: 'XCUIElementTypeWindow' },
{ index: 1, depth: 1, type: 'XCUIElementTypeStaticText', label: 'Count' },
{ index: 2, depth: 1, type: 'XCUIElementTypeStaticText', label: '134' },
{ index: 3, depth: 1, type: 'XCUIElementTypeButton', label: 'Increment' },
]);

const diff = buildSnapshotDiff(previous, current);
assert.equal(diff.lines[0]?.kind, 'unchanged');
assert.equal(diff.lines[1]?.kind, 'unchanged');
assert.equal(diff.lines[2]?.kind, 'removed');
assert.equal(diff.lines[3]?.kind, 'added');
assert.equal(diff.lines[4]?.kind, 'unchanged');
});

test('buildSnapshotDiff flatten option uses flat snapshot line shape', () => {
const previous = nodes([
{ index: 0, depth: 0, type: 'XCUIElementTypeWindow' },
{ index: 1, depth: 1, type: 'XCUIElementTypeOther', label: '335' },
{ index: 2, depth: 2, type: 'XCUIElementTypeStaticText', label: '335' },
]);
const current = nodes([
{ index: 0, depth: 0, type: 'XCUIElementTypeWindow' },
{ index: 1, depth: 1, type: 'XCUIElementTypeOther', label: '402' },
{ index: 2, depth: 2, type: 'XCUIElementTypeStaticText', label: '402' },
]);

const diff = buildSnapshotDiff(previous, current, { flatten: true });
assert.equal(diff.summary.additions, 2);
assert.equal(diff.summary.removals, 2);
const changed = diff.lines.filter((line) => line.kind !== 'unchanged');
assert.equal(changed.length, 4);
for (const line of changed) {
assert.equal(line.text.startsWith(' '), false);
}
});
Loading
Loading