Skip to content

Commit 81f2b40

Browse files
XeonBloomfieldclaude
authored andcommitted
feat: add agent-device acp, a deterministic stdio ACP agent
Add an Agent Client Protocol (ACP) v1 agent mode so ACP clients such as Zed can drive devices from the agent panel. agent-device has no LLM, so the agent is deterministic: each prompt line is one agent-device command in CLI syntax, executed through the same AgentDeviceClient path as MCP tools. - Hand-rolled protocol layer in src/acp/ mirroring src/mcp/ (zero new runtime dependencies), reusing the MCP newline-delimited JSON-RPC stdio transport, payload queue, and error formatting. - Prompt lines are tokenized with the replay-script tokenizer and parsed with the real CLI parser; target flags (--platform, --device, --udid, --session, --state-dir) are sticky within an ACP session. - Command executions stream as tool_call/tool_call_update notifications with daemon progress forwarding; screenshots attach as inline images; errors keep the code/hint/supportedOn contract. - session/cancel is intercepted ahead of the serialized queue so it takes effect between command lines of a running prompt; the command already in flight runs to completion (documented v1 limitation). - Natural-language prompts are refused with guidance instead of guessed at; available commands are advertised per session after session/new. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent 0dcc1aa commit 81f2b40

23 files changed

Lines changed: 1662 additions & 17 deletions

CHANGELOG.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,9 @@
11
# Changelog
22

3+
## Unreleased
4+
5+
- Added `agent-device acp`: a deterministic stdio ACP (Agent Client Protocol) agent for Zed and other ACP clients. Prompt lines are agent-device commands in CLI syntax, executed through the same client path as MCP tools, with tool-call streaming, sticky per-session target flags, and inline screenshot images.
6+
37
## 0.15.0
48

59
- Breaking: `apps` discovery and public app-list helpers now default to user-installed apps. Use `--all` or `filter: 'all'` to include system/OEM apps.

CONTEXT.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,7 @@
4242
start-time/command reads, process listing, process-tree expansion, PID de-duplication, and
4343
best-effort signaling. It must not own domain cleanup policy such as browser ownership markers,
4444
runner lease reclamation, daemon takeover checks, or app-log PID metadata verification.
45-
- Command surface: catalog of public command identity, interface exposure, adapter policy, and shared command metadata across CLI, Node.js, MCP, and batch entrypoints.
45+
- Command surface: catalog of public command identity, interface exposure, adapter policy, and shared command metadata across CLI, Node.js, MCP, ACP, and batch entrypoints.
4646
- Daemon command registry: daemon-side source of truth for command route ownership and request-policy traits, including admission exemptions, session locking, selector validation, replay-scoped actions, recording invalidation, Android dialog guards, and request provider device resolution.
4747
- Runner command traits: per-command-type classification for iOS/macOS runner lifecycle behavior, distinct from the public command surface and daemon command registry. The Swift runner traits classify interaction, read-only, and runner-lifecycle axes for XCTest execution; Swift resolves the alert command as read-only only for its `get` action. The TypeScript runner command traits classify daemon-side runner send/recovery policy such as read-only retry routing, readiness probes, and recent-healthy-mutation preflight skips; the TypeScript table is command-type keyed and currently classifies alert as read-only for daemon retry policy. Each side keeps one source of truth keyed by runner command type.
4848
- Coordinate-first resolved element activation: iOS/macOS runner interaction pattern where a selector or text query resolves the semantic `XCUIElement`, then activation uses the element's resolved center coordinate when a frame is available. This keeps target selection semantic while avoiding `XCUIElement.tap()` post-action element re-resolution after normal navigation. tvOS remains focus/remote-driven.

README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -88,7 +88,7 @@ Snapshots come from the app's accessibility tree, so high-quality labels, roles,
8888

8989
## Next Steps
9090

91-
- **Set up your agent**: run the CLI from Cursor, Codex, Claude Code, Windsurf, or another agent terminal. For skills, rules, direct MCP tools, and client-specific setup, see [AI Agent Setup](https://oss.callstack.com/agent-device/docs/agent-setup).
91+
- **Set up your agent**: run the CLI from Cursor, Codex, Claude Code, Windsurf, or another agent terminal. For skills, rules, direct MCP tools (`agent-device mcp`), the ACP agent for Zed and other ACP clients (`agent-device acp`), and client-specific setup, see [AI Agent Setup](https://oss.callstack.com/agent-device/docs/agent-setup).
9292
- **Try the sample app**: clone the repo and run the bundled Expo fixture when you want a guided first dogfood run with screenshots, replay, and performance evidence. See [Quick Start](https://oss.callstack.com/agent-device/docs/quick-start).
9393
- **Go deeper**: use [Commands](https://oss.callstack.com/agent-device/docs/commands), [Replay & E2E](https://oss.callstack.com/agent-device/docs/replay-e2e), and [Debugging & Profiling](https://oss.callstack.com/agent-device/docs/debugging-profiling) for production workflows.
9494

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
import assert from 'node:assert/strict';
2+
import { tmpdir } from 'node:os';
3+
import { test } from 'vitest';
4+
import { applyStickyFlags, parsePromptCommandLines, type RunnableLine } from '../prompt-lines.ts';
5+
6+
const CWD = tmpdir();
7+
8+
function parseOne(text: string): ReturnType<typeof parsePromptCommandLines>[number] {
9+
const lines = parsePromptCommandLines(text, { cwd: CWD });
10+
assert.equal(lines.length, 1);
11+
return lines[0]!;
12+
}
13+
14+
test('splits prompt text into command lines, skipping blanks and comments', () => {
15+
const lines = parsePromptCommandLines('\n# comment\ndevices\n\nsnapshot -i\n', { cwd: CWD });
16+
assert.deepEqual(
17+
lines.map((line) => (line.kind === 'command' ? line.command : line.error)),
18+
['devices', 'snapshot'],
19+
);
20+
});
21+
22+
test('strips an optional leading agent-device token', () => {
23+
const line = parseOne('agent-device devices');
24+
assert.equal(line.kind, 'command');
25+
assert.equal((line as RunnableLine).command, 'devices');
26+
});
27+
28+
test('tokenizes quoted arguments with spaces like replay scripts', () => {
29+
const line = parseOne('fill "id:search" "hello world" --platform ios');
30+
assert.equal(line.kind, 'command');
31+
const runnable = line as RunnableLine;
32+
assert.deepEqual(runnable.positionals, ['id:search', 'hello world']);
33+
assert.equal(runnable.flags.platform, 'ios');
34+
});
35+
36+
test('rejects unknown commands with per-line guidance', () => {
37+
const line = parseOne('frobnicate');
38+
assert.equal(line.kind, 'error');
39+
assert.match((line as { error: string }).error, /Unknown command: frobnicate/);
40+
});
41+
42+
test('rejects CLI-only commands that are not automatable through ACP', () => {
43+
for (const command of ['mcp', 'acp', 'connect', 'auth']) {
44+
const line = parseOne(command);
45+
assert.equal(line.kind, 'error', `expected ${command} to be rejected`);
46+
}
47+
});
48+
49+
test('sticky flags fill in target selection and explicit flags win', () => {
50+
const bare = parseOne('snapshot -i') as RunnableLine;
51+
assert.equal(bare.kind, 'command');
52+
assert.deepEqual(bare.providedSticky, {});
53+
const merged = applyStickyFlags(bare, { platform: 'ios', session: 'demo' });
54+
assert.equal(merged.platform, 'ios');
55+
assert.equal(merged.session, 'demo');
56+
57+
const explicit = parseOne('snapshot -i --platform android') as RunnableLine;
58+
assert.equal(explicit.kind, 'command');
59+
assert.deepEqual(explicit.providedSticky, { platform: 'android' });
60+
assert.equal(applyStickyFlags(explicit, { platform: 'ios' }).platform, 'android');
61+
});
Lines changed: 235 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,235 @@
1+
import assert from 'node:assert/strict';
2+
import { tmpdir } from 'node:os';
3+
import { test } from 'vitest';
4+
import type { AgentDeviceClient } from '../../client/client-types.ts';
5+
import type { RequestProgressSink } from '../../daemon/request-progress.ts';
6+
import { AppError } from '../../kernel/errors.ts';
7+
import type { AcpSessionUpdate } from '../contracts.ts';
8+
import { createPromptRunner, type PromptRunnerDeps } from '../prompt-runner.ts';
9+
import type { AcpSessionState } from '../session-state.ts';
10+
11+
const FAKE_CLIENT = {} as AgentDeviceClient;
12+
13+
function makeSession(sticky = {}): AcpSessionState {
14+
return {
15+
id: 'sess-1',
16+
cwd: tmpdir(),
17+
sticky,
18+
cancelled: false,
19+
promptActive: true,
20+
toolCallCounter: 0,
21+
};
22+
}
23+
24+
function collectUpdates(): {
25+
updates: AcpSessionUpdate[];
26+
sendUpdate: (u: AcpSessionUpdate) => void;
27+
} {
28+
const updates: AcpSessionUpdate[] = [];
29+
return { updates, sendUpdate: (update) => updates.push(update) };
30+
}
31+
32+
function runnerWith(deps: Partial<PromptRunnerDeps>) {
33+
return createPromptRunner({
34+
createClient: async () => FAKE_CLIENT,
35+
readImageFile: () => undefined,
36+
...deps,
37+
});
38+
}
39+
40+
test('runs each prompt line as a tool call and summarizes the turn', async () => {
41+
const executed: string[] = [];
42+
const runner = runnerWith({
43+
runCommandLine: async ({ command }) => {
44+
executed.push(command);
45+
return { result: { ok: true }, cliOutput: { data: {}, text: `${command} done` } };
46+
},
47+
});
48+
const { updates, sendUpdate } = collectUpdates();
49+
50+
const stopReason = await runner.runPrompt({
51+
session: makeSession(),
52+
promptText: 'devices\nsnapshot -i',
53+
sendUpdate,
54+
});
55+
56+
assert.equal(stopReason, 'end_turn');
57+
assert.deepEqual(executed, ['devices', 'snapshot']);
58+
const kinds = updates.map((update) => update.sessionUpdate);
59+
assert.deepEqual(kinds, [
60+
'tool_call',
61+
'tool_call_update',
62+
'tool_call',
63+
'tool_call_update',
64+
'agent_message_chunk',
65+
]);
66+
const firstCall = updates[0] as Extract<AcpSessionUpdate, { sessionUpdate: 'tool_call' }>;
67+
assert.equal(firstCall.title, 'devices');
68+
assert.equal(firstCall.status, 'in_progress');
69+
assert.equal(firstCall.kind, 'read');
70+
const firstDone = updates[1] as Extract<AcpSessionUpdate, { sessionUpdate: 'tool_call_update' }>;
71+
assert.equal(firstDone.toolCallId, firstCall.toolCallId);
72+
assert.equal(firstDone.status, 'completed');
73+
assert.deepEqual(firstDone.rawOutput, { ok: true });
74+
assert.match(JSON.stringify(firstDone.content), /devices done/);
75+
const summary = updates.at(-1) as Extract<
76+
AcpSessionUpdate,
77+
{ sessionUpdate: 'agent_message_chunk' }
78+
>;
79+
assert.match(summary.content.type === 'text' ? summary.content.text : '', / devices/);
80+
});
81+
82+
test('command failures become failed tool calls with the full error contract', async () => {
83+
const runner = runnerWith({
84+
runCommandLine: async () => {
85+
throw new AppError('DEVICE_NOT_FOUND', 'No booted device.', { hint: 'Boot one first.' });
86+
},
87+
});
88+
const { updates, sendUpdate } = collectUpdates();
89+
90+
const stopReason = await runner.runPrompt({
91+
session: makeSession(),
92+
promptText: 'devices',
93+
sendUpdate,
94+
});
95+
96+
assert.equal(stopReason, 'end_turn');
97+
const failed = updates[1] as Extract<AcpSessionUpdate, { sessionUpdate: 'tool_call_update' }>;
98+
assert.equal(failed.status, 'failed');
99+
const text = JSON.stringify(failed.content);
100+
assert.match(text, /Error \(DEVICE_NOT_FOUND\): No booted device\./);
101+
assert.match(text, /Hint: Boot one first\./);
102+
});
103+
104+
test('prompts with no runnable command lines are refused without touching the daemon', async () => {
105+
let clientCreated = false;
106+
const runner = runnerWith({
107+
createClient: async () => {
108+
clientCreated = true;
109+
return FAKE_CLIENT;
110+
},
111+
runCommandLine: async () => {
112+
throw new Error('must not run');
113+
},
114+
});
115+
const { updates, sendUpdate } = collectUpdates();
116+
117+
const stopReason = await runner.runPrompt({
118+
session: makeSession(),
119+
promptText: 'please tap the login button',
120+
sendUpdate,
121+
});
122+
123+
assert.equal(stopReason, 'refusal');
124+
assert.equal(clientCreated, false);
125+
assert.equal(updates.length, 1);
126+
assert.equal(updates[0]!.sessionUpdate, 'agent_message_chunk');
127+
});
128+
129+
test('unparseable lines fail individually while runnable lines still execute', async () => {
130+
const executed: string[] = [];
131+
const runner = runnerWith({
132+
runCommandLine: async ({ command }) => {
133+
executed.push(command);
134+
return { result: {}, cliOutput: { data: {}, text: 'ok' } };
135+
},
136+
});
137+
const { updates, sendUpdate } = collectUpdates();
138+
139+
const stopReason = await runner.runPrompt({
140+
session: makeSession(),
141+
promptText: 'frobnicate\ndevices',
142+
sendUpdate,
143+
});
144+
145+
assert.equal(stopReason, 'end_turn');
146+
assert.deepEqual(executed, ['devices']);
147+
const failedCall = updates[0] as Extract<AcpSessionUpdate, { sessionUpdate: 'tool_call' }>;
148+
assert.equal(failedCall.status, 'failed');
149+
assert.match(JSON.stringify(failedCall.content), /Unknown command: frobnicate/);
150+
});
151+
152+
test('sticky flags from one line carry into later lines of the same session', async () => {
153+
const platforms: Array<string | undefined> = [];
154+
const session = makeSession();
155+
const runner = runnerWith({
156+
runCommandLine: async ({ flags }) => {
157+
platforms.push(flags.platform);
158+
return { result: {}, cliOutput: { data: {}, text: 'ok' } };
159+
},
160+
});
161+
const { sendUpdate } = collectUpdates();
162+
163+
await runner.runPrompt({
164+
session,
165+
promptText: 'apps --platform ios\nsnapshot -i',
166+
sendUpdate,
167+
});
168+
169+
assert.deepEqual(platforms, ['ios', 'ios']);
170+
assert.equal(session.sticky.platform, 'ios');
171+
});
172+
173+
test('cancellation between lines stops the turn with the cancelled stop reason', async () => {
174+
const session = makeSession();
175+
const executed: string[] = [];
176+
const runner = runnerWith({
177+
runCommandLine: async ({ command }) => {
178+
executed.push(command);
179+
session.cancelled = true;
180+
return { result: {}, cliOutput: { data: {}, text: 'ok' } };
181+
},
182+
});
183+
const { sendUpdate } = collectUpdates();
184+
185+
const stopReason = await runner.runPrompt({
186+
session,
187+
promptText: 'devices\nsnapshot -i',
188+
sendUpdate,
189+
});
190+
191+
assert.equal(stopReason, 'cancelled');
192+
assert.deepEqual(executed, ['devices']);
193+
});
194+
195+
test('screenshot results attach an inline image when the artifact is readable', async () => {
196+
const runner = runnerWith({
197+
runCommandLine: async () => ({
198+
result: { path: '/artifacts/screen.png' },
199+
cliOutput: { data: {}, text: 'Saved screenshot' },
200+
}),
201+
readImageFile: (path) =>
202+
path === '/artifacts/screen.png'
203+
? { type: 'image', data: 'aGk=', mimeType: 'image/png' }
204+
: undefined,
205+
});
206+
const { updates, sendUpdate } = collectUpdates();
207+
208+
await runner.runPrompt({ session: makeSession(), promptText: 'screenshot', sendUpdate });
209+
210+
const done = updates[1] as Extract<AcpSessionUpdate, { sessionUpdate: 'tool_call_update' }>;
211+
assert.equal(done.status, 'completed');
212+
const image = done.content?.find((entry) => entry.content.type === 'image');
213+
assert.ok(image, 'expected an inline image content block');
214+
});
215+
216+
test('daemon progress events stream as in_progress tool call updates', async () => {
217+
let progressSink: RequestProgressSink | undefined;
218+
const runner = runnerWith({
219+
createClient: async (_config, onProgress) => {
220+
progressSink = onProgress;
221+
return FAKE_CLIENT;
222+
},
223+
runCommandLine: async () => {
224+
progressSink?.({ type: 'command', status: 'progress', message: 'Booting simulator…' });
225+
return { result: {}, cliOutput: { data: {}, text: 'ok' } };
226+
},
227+
});
228+
const { updates, sendUpdate } = collectUpdates();
229+
230+
await runner.runPrompt({ session: makeSession(), promptText: 'boot', sendUpdate });
231+
232+
const progress = updates[1] as Extract<AcpSessionUpdate, { sessionUpdate: 'tool_call_update' }>;
233+
assert.equal(progress.status, 'in_progress');
234+
assert.match(JSON.stringify(progress.content), /Booting simulator/);
235+
});

0 commit comments

Comments
 (0)