Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ The project is in early development and considered experimental. Pull requests a
- Platforms: iOS (simulator + physical device core automation) and Android (emulator + device).
- Core commands: `open`, `back`, `home`, `app-switcher`, `press`, `long-press`, `focus`, `type`, `fill`, `scroll`, `scrollintoview`, `wait`, `alert`, `screenshot`, `close`, `reinstall`, `push`.
- Inspection commands: `snapshot` (accessibility tree), `diff snapshot` (structural baseline diff), `appstate`, `apps`, `devices`.
- Clipboard commands: `clipboard read`, `clipboard write <text>`.
- App logs: `logs path` returns session log metadata; `logs start` / `logs stop` stream app output; `logs clear` truncates session app logs; `logs clear --restart` resets and restarts stream in one step; `logs doctor` checks readiness; `logs mark` writes timeline markers.
- Device tooling: `adb` (Android), `simctl`/`devicectl` (iOS via Xcode).
- Minimal dependencies; TypeScript executed directly on Node 22+ (no build step).
Expand Down Expand Up @@ -147,6 +148,7 @@ agent-device scrollintoview @e42
- `alert`, `wait`, `screenshot`
- `trace start`, `trace stop`
- `logs path`, `logs start`, `logs stop`, `logs clear`, `logs clear --restart`, `logs doctor`, `logs mark` (session app log file for grep; iOS simulator + iOS device + Android)
- `clipboard read`, `clipboard write <text>` (iOS simulator + Android)
- `settings wifi|airplane|location on|off`
- `settings appearance light|dark|toggle`
- `settings faceid match|nonmatch|enroll|unenroll` (iOS simulator only)
Expand Down Expand Up @@ -333,6 +335,12 @@ App state:
- On iOS, `appstate` returns the currently tracked session app (`source: session`) and requires an active session on the selected device.
- `apps` includes default/system apps by default (use `--user-installed` to filter).

Clipboard:
- `clipboard read` returns current clipboard text.
- `clipboard write <text>` sets clipboard text (`clipboard write ""` clears it).
- Supported on Android emulator/device and iOS simulator.
- iOS physical devices currently return `UNSUPPORTED_OPERATION` for clipboard commands.

## Debug

- **App logs (token-efficient):** Logging is off by default in normal flows. Enable it on demand when debugging. With an active session, run `logs path` to get path + state metadata (e.g. `~/.agent-device/sessions/<session>/app.log`). Run `logs start` to stream app output to that file; use `logs stop` to stop. Run `logs clear` to truncate `app.log` (and remove rotated `app.log.N` files) before a new repro window. Run `logs doctor` for tool/runtime checks and `logs mark "step"` to insert timeline markers. Grep the file when you need to inspect errors (e.g. `grep -n "Error\|Exception" <path>`) instead of pulling full logs into context. Supported on iOS simulator, iOS physical device, and Android.
Expand Down
3 changes: 3 additions & 0 deletions skills/agent-device/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,8 @@ agent-device is visible 'id="anchor"'

```bash
agent-device appstate
agent-device clipboard read
agent-device clipboard write "token"
agent-device push <bundle|package> <payload.json|inline-json>
agent-device get text @e1
agent-device screenshot out.png
Expand All @@ -108,6 +110,7 @@ agent-device batch --steps-file /tmp/batch-steps.json --json
- Use refs for discovery, selectors for replay/assertions.
- Use `fill` for clear-then-type semantics; use `type` for focused append typing.
- iOS `appstate` is session-scoped; Android `appstate` is live foreground state.
- Clipboard helpers: `clipboard read` / `clipboard write <text>` are supported on Android and iOS simulators; iOS physical devices are not supported yet.
- iOS settings helpers are simulator-only; use `appearance light|dark|toggle` and faceid `match|nonmatch|enroll|unenroll`.
- `push` simulates notification delivery:
- iOS simulator uses APNs-style payload JSON.
Expand Down
92 changes: 92 additions & 0 deletions src/__tests__/cli-clipboard.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import { runCli } from '../cli.ts';
import type { DaemonRequest, DaemonResponse } from '../daemon-client.ts';

class ExitSignal extends Error {
public readonly code: number;

constructor(code: number) {
super(`EXIT_${code}`);
this.code = code;
}
}

type RunResult = {
code: number | null;
stdout: string;
stderr: string;
calls: Omit<DaemonRequest, 'token'>[];
};

async function runCliCapture(
argv: string[],
responder: (req: Omit<DaemonRequest, 'token'>) => Promise<DaemonResponse>,
): Promise<RunResult> {
let stdout = '';
let stderr = '';
let code: number | null = null;
const calls: Array<Omit<DaemonRequest, 'token'>> = [];

const originalExit = process.exit;
const originalStdoutWrite = process.stdout.write.bind(process.stdout);
const originalStderrWrite = process.stderr.write.bind(process.stderr);

(process as any).exit = ((nextCode?: number) => {
throw new ExitSignal(nextCode ?? 0);
}) as typeof process.exit;
(process.stdout as any).write = ((chunk: unknown) => {
stdout += String(chunk);
return true;
}) as typeof process.stdout.write;
(process.stderr as any).write = ((chunk: unknown) => {
stderr += String(chunk);
return true;
}) as typeof process.stderr.write;

const sendToDaemon = async (req: Omit<DaemonRequest, 'token'>): Promise<DaemonResponse> => {
calls.push(req);
return await responder(req);
};

try {
await runCli(argv, { sendToDaemon });
} catch (error) {
if (error instanceof ExitSignal) code = error.code;
else throw error;
} finally {
process.exit = originalExit;
process.stdout.write = originalStdoutWrite;
process.stderr.write = originalStderrWrite;
}

return { code, stdout, stderr, calls };
}

test('clipboard read prints clipboard text', async () => {
const result = await runCliCapture(['clipboard', 'read'], async () => ({
ok: true,
data: { action: 'read', text: 'otp-123456' },
}));

assert.equal(result.code, null);
assert.equal(result.calls.length, 1);
assert.equal(result.calls[0]?.command, 'clipboard');
assert.deepEqual(result.calls[0]?.positionals, ['read']);
assert.equal(result.stdout, 'otp-123456\n');
assert.equal(result.stderr, '');
});

test('clipboard write prints update confirmation', async () => {
const result = await runCliCapture(['clipboard', 'write', 'hello'], async () => ({
ok: true,
data: { action: 'write', textLength: 5 },
}));

assert.equal(result.code, null);
assert.equal(result.calls.length, 1);
assert.equal(result.calls[0]?.command, 'clipboard');
assert.deepEqual(result.calls[0]?.positionals, ['write', 'hello']);
assert.equal(result.stdout, 'Clipboard updated\n');
assert.equal(result.stderr, '');
});
15 changes: 15 additions & 0 deletions src/cli.ts
Original file line number Diff line number Diff line change
Expand Up @@ -290,6 +290,21 @@ export async function runCli(argv: string[], deps: CliDeps = DEFAULT_CLI_DEPS):
if (logTailStopper) logTailStopper();
return;
}
if (command === 'clipboard') {
const data = response.data as Record<string, unknown> | undefined;
const action = (positionals[0] ?? (typeof data?.action === 'string' ? data.action : '')).toLowerCase();
if (action === 'read') {
const text = typeof data?.text === 'string' ? data.text : '';
process.stdout.write(`${text}\n`);
if (logTailStopper) logTailStopper();
return;
}
if (action === 'write') {
process.stdout.write('Clipboard updated\n');
if (logTailStopper) logTailStopper();
return;
}
}
if (command === 'click' || command === 'press') {
const ref = (response.data as any)?.ref ?? '';
const x = (response.data as any)?.x;
Expand Down
2 changes: 1 addition & 1 deletion src/core/__tests__/capabilities.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ test('iOS simulator-only commands reject iOS devices and Android', () => {
});

test('simulator-only iOS commands with Android support reject iOS devices', () => {
for (const cmd of ['settings', 'push']) {
for (const cmd of ['settings', 'push', 'clipboard']) {
assert.equal(isCommandSupportedOnDevice(cmd, iosSimulator), true, `${cmd} on iOS sim`);
assert.equal(isCommandSupportedOnDevice(cmd, iosDevice), false, `${cmd} on iOS device`);
assert.equal(isCommandSupportedOnDevice(cmd, androidDevice), true, `${cmd} on Android`);
Expand Down
1 change: 1 addition & 0 deletions src/core/capabilities.ts
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ const COMMAND_CAPABILITY_MATRIX: Record<string, CommandCapability> = {
back: { ios: { simulator: true, device: true }, android: { emulator: true, device: true, unknown: true } },
boot: { ios: { simulator: true, device: true }, android: { emulator: true, device: true, unknown: true } },
click: { ios: { simulator: true, device: true }, android: { emulator: true, device: true, unknown: true } },
clipboard: { ios: { simulator: true }, android: { emulator: true, device: true, unknown: true } },
close: { ios: { simulator: true, device: true }, android: { emulator: true, device: true, unknown: true } },
fill: { ios: { simulator: true, device: true }, android: { emulator: true, device: true, unknown: true } },
diff: { ios: { simulator: true, device: true }, android: { emulator: true, device: true, unknown: true } },
Expand Down
31 changes: 30 additions & 1 deletion src/core/dispatch.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,13 +9,20 @@ import {
ensureAdb,
homeAndroid,
pushAndroidNotification,
readAndroidClipboardText,
setAndroidSetting,
snapshotAndroid,
writeAndroidClipboardText,
} from '../platforms/android/index.ts';
import { listIosDevices } from '../platforms/ios/devices.ts';
import { getInteractor, type RunnerContext } from '../utils/interactors.ts';
import { runIosRunnerCommand } from '../platforms/ios/runner-client.ts';
import { pushIosNotification, setIosSetting } from '../platforms/ios/index.ts';
import {
pushIosNotification,
readIosClipboardText,
setIosSetting,
writeIosClipboardText,
} from '../platforms/ios/index.ts';
import { isDeepLinkTarget } from './open-target.ts';
import type { RawSnapshotNode } from '../utils/snapshot.ts';
import type { CliFlags } from '../utils/command-schema.ts';
Expand Down Expand Up @@ -413,6 +420,28 @@ export async function dispatchCommand(
await appSwitcherAndroid(device);
return { action: 'app-switcher' };
}
case 'clipboard': {
const action = (positionals[0] ?? '').toLowerCase();
if (action !== 'read' && action !== 'write') {
throw new AppError('INVALID_ARGS', 'clipboard requires a subcommand: read or write');
}
if (action === 'read') {
if (positionals.length !== 1) {
throw new AppError('INVALID_ARGS', 'clipboard read does not accept additional arguments');
}
const text = device.platform === 'ios'
? await readIosClipboardText(device)
: await readAndroidClipboardText(device);
return { action, text };
}
if (positionals.length < 2) {
throw new AppError('INVALID_ARGS', 'clipboard write requires text (use "" to clear clipboard)');
}
const text = positionals.slice(1).join(' ');
if (device.platform === 'ios') await writeIosClipboardText(device, text);
else await writeAndroidClipboardText(device, text);
return { action, textLength: Array.from(text).length };
}
case 'settings': {
const [setting, state, target, mode, appBundleId] = positionals;
const permissionOptions =
Expand Down
152 changes: 152 additions & 0 deletions src/daemon/handlers/__tests__/session.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -612,6 +612,158 @@ test('appstate with explicit missing session returns SESSION_NOT_FOUND', async (
}
});

test('clipboard requires an active session or explicit device selector', async () => {
const sessionStore = makeSessionStore();
const response = await handleSessionCommands({
req: {
token: 't',
session: 'default',
command: 'clipboard',
positionals: ['read'],
flags: {},
},
sessionName: 'default',
logPath: path.join(os.tmpdir(), 'daemon.log'),
sessionStore,
invoke: noopInvoke,
});

assert.ok(response);
assert.equal(response?.ok, false);
if (response && !response.ok) {
assert.equal(response.error.code, 'INVALID_ARGS');
assert.match(response.error.message, /clipboard requires an active session or an explicit device selector/i);
}
});

test('clipboard read uses active session device', async () => {
const sessionStore = makeSessionStore();
const sessionName = 'ios-sim-session';
sessionStore.set(
sessionName,
makeSession(sessionName, {
platform: 'ios',
id: 'sim-1',
name: 'iPhone 17 Pro',
kind: 'simulator',
booted: true,
}),
);

const response = await handleSessionCommands({
req: {
token: 't',
session: sessionName,
command: 'clipboard',
positionals: ['read'],
flags: {},
},
sessionName,
logPath: path.join(os.tmpdir(), 'daemon.log'),
sessionStore,
invoke: noopInvoke,
ensureReady: async () => {},
dispatch: async (device, command, positionals) => {
assert.equal(device.id, 'sim-1');
assert.equal(command, 'clipboard');
assert.deepEqual(positionals, ['read']);
return { action: 'read', text: 'otp-123456' };
},
resolveTargetDevice: async () => {
throw new Error('resolveTargetDevice should not run');
},
});

assert.ok(response);
assert.equal(response?.ok, true);
if (response && response.ok) {
assert.equal(response.data?.platform, 'ios');
assert.equal(response.data?.action, 'read');
assert.equal(response.data?.text, 'otp-123456');
}
});

test('clipboard write supports explicit selector without active session', async () => {
const sessionStore = makeSessionStore();
const selectedDevice: SessionState['device'] = {
platform: 'android',
id: 'emulator-5554',
name: 'Pixel Emulator',
kind: 'emulator',
booted: true,
};

const response = await handleSessionCommands({
req: {
token: 't',
session: 'default',
command: 'clipboard',
positionals: ['write', 'hello', 'clipboard'],
flags: { platform: 'android', serial: 'emulator-5554' },
},
sessionName: 'default',
logPath: path.join(os.tmpdir(), 'daemon.log'),
sessionStore,
invoke: noopInvoke,
ensureReady: async () => {},
resolveTargetDevice: async () => selectedDevice,
dispatch: async (device, command, positionals) => {
assert.equal(device.id, 'emulator-5554');
assert.equal(command, 'clipboard');
assert.deepEqual(positionals, ['write', 'hello', 'clipboard']);
return { action: 'write', textLength: 15 };
},
});

assert.ok(response);
assert.equal(response?.ok, true);
if (response && response.ok) {
assert.equal(response.data?.platform, 'android');
assert.equal(response.data?.action, 'write');
assert.equal(response.data?.textLength, 15);
}
});

test('clipboard rejects unsupported iOS physical devices', async () => {
const sessionStore = makeSessionStore();
const sessionName = 'ios-device-session';
sessionStore.set(
sessionName,
makeSession(sessionName, {
platform: 'ios',
id: 'ios-device-1',
name: 'iPhone Device',
kind: 'device',
booted: true,
}),
);

const response = await handleSessionCommands({
req: {
token: 't',
session: sessionName,
command: 'clipboard',
positionals: ['read'],
flags: {},
},
sessionName,
logPath: path.join(os.tmpdir(), 'daemon.log'),
sessionStore,
invoke: noopInvoke,
ensureReady: async () => {},
dispatch: async () => {
throw new Error('dispatch should not run for unsupported targets');
},
});

assert.ok(response);
assert.equal(response?.ok, false);
if (response && !response.ok) {
assert.equal(response.error.code, 'UNSUPPORTED_OPERATION');
assert.match(response.error.message, /clipboard is not supported on this device/i);
}
});

test('open URL on existing iOS session clears stale app bundle id', async () => {
const sessionStore = makeSessionStore();
const sessionName = 'ios-session';
Expand Down
Loading
Loading