Skip to content

Commit 66910f1

Browse files
authored
fix: remove Android ADB swipe fallbacks (#1243)
* fix: remove Android gesture swipe fallback * fix: tighten Android gesture review follow-up * fix: route Android touch actions through gesture helper * test: isolate Android touch provider fixture * test: drop Android swipe fallback assertions * test: provide semantic Android touch in provider scenarios * refactor: drop redundant Android touch planning code * fix: require viewport for Android touch providers * refactor: extract Android touch executor * docs: clarify Android planned touch seam * fix: complete Android gesture failure handling * refactor: tighten Android gesture review fixes * perf: avoid unnecessary Android viewport probes * fix: remove unused Android helper cache export * fix: close Android gesture contract gaps * test: cover max Android helper gesture timeout
1 parent 6c41638 commit 66910f1

24 files changed

Lines changed: 829 additions & 550 deletions

CONTEXT.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,12 @@
4848
- Coordinate-first resolved element activation: iOS/macOS runner interaction pattern where a selector or text query resolves the semantic `XCUIElement`, then activation uses the element's resolved center coordinate when a frame is available. This keeps target selection semantic while avoiding `XCUIElement.tap()` post-action element re-resolution after normal navigation. tvOS remains focus/remote-driven.
4949
- Interaction dispatch path: one concrete route an interaction command takes to the device (runtime selector/ref resolution, direct iOS selector, native ref via web clickRef, coordinate, maestro non-hittable fallback). Every path classifies every guarantee in the ADR 0011 registry.
5050
- Gesture plan: typed, platform-neutral normalization of one- or two-contact gesture intent into bounded pointer trajectories. Contact topology is separate from motion; two-contact intent remains pan/pinch/rotate/transform even when native injection shares one executor. See ADR 0013.
51+
- Android planned-touch executor: Android-local adapter seam that accepts `AndroidTouchPlan`—the
52+
platform-neutral `GesturePlan` plus Android's stationary long-press plan—and selects the paired
53+
provider-native touch/viewport adapter or bundled instrumentation-helper adapter. Scroll and
54+
long-press retain their command semantics and only share physical touch execution through this
55+
seam. Helper long-press executes its absolute stationary path without a viewport probe; provider
56+
long-press receives its paired provider-owned viewport. See ADR 0013.
5157
- Multi-touch geometry: the internal initial span and angle plus centroid translation, scale, and rotation used to build both contact trajectories. Geometry is viewport-aware and fails early when the requested motion cannot fit; it is not a public tuning surface.
5258
- Guarantee cell: one (dispatch path, guarantee) entry in `src/contracts/interaction-guarantees.ts`, classified as runtime/runner/delegated/inapplicable/waived. Completeness is a compile error; honesty is gate-tested.
5359
- Owned waiver: a `gap:`-prefixed waived cell carrying a `trackingIssue` URL. Waivers are diffable debt with an owner, never folklore.

android-multitouch-helper/src/main/java/com/callstack/agentdevice/multitouchhelper/MultiTouchInstrumentation.java

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -19,8 +19,8 @@
1919
public final class MultiTouchInstrumentation extends Instrumentation {
2020
private static final String PROTOCOL = "android-multitouch-helper-v1";
2121
private static final String HELPER_API_VERSION = "1";
22-
private static final int MIN_DURATION_MS = 16;
23-
private static final int MAX_DURATION_MS = 10_000;
22+
private static final int MIN_DURATION_MS = 0;
23+
private static final int MAX_DURATION_MS = 120_000;
2424
private Bundle arguments;
2525

2626
@Override
@@ -204,7 +204,8 @@ private static PointerPath[] readPointers(JSONArray pointers, String kind, int d
204204
throw new IllegalArgumentException("Planned sample coordinates must be finite");
205205
}
206206
long offsetMs = (long) rawOffsetMs;
207-
if (offsetMs <= previousOffsetMs) {
207+
if (offsetMs < previousOffsetMs
208+
|| (offsetMs == previousOffsetMs && durationMs != 0)) {
208209
throw new IllegalArgumentException("Planned sample offsets must be strictly increasing");
209210
}
210211
parsed[sampleIndex] = new PointerSample(offsetMs, (float) x, (float) y);
@@ -307,7 +308,7 @@ private static MotionEvent motionEvent(
307308

308309
private static int requireDuration(int value) {
309310
if (value < MIN_DURATION_MS || value > MAX_DURATION_MS) {
310-
throw new IllegalArgumentException("durationMs must be between 16 and 10000");
311+
throw new IllegalArgumentException("durationMs must be between 0 and 120000");
311312
}
312313
return value;
313314
}

docs/adr/0013-unified-gesture-plans.md

Lines changed: 18 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -15,8 +15,9 @@ Two-contact geometry also differed by platform. Android used a fixed radius whil
1515
radius from the app frame. Neither path validated every planned point against the active
1616
interaction viewport before injection.
1717

18-
`scroll` remains separate because it owns viewport/edge traversal, content-state verification, and
19-
runner fallback policy rather than a single physical gesture.
18+
`scroll` remains a separate command because it owns viewport/edge traversal and content-state
19+
verification rather than a single physical gesture. Its Android physical movement still lowers to
20+
the Android planned-touch executor.
2021

2122
## Decision
2223

@@ -54,11 +55,16 @@ no established automation use case justifies a public tuning surface.
5455

5556
Platform adapters consume the canonical plan:
5657

57-
- Android sends the plan to provider-native touch injection when available, otherwise to the
58-
bundled instrumentation helper. The helper injects the exact planned pointer samples. A
59-
two-contact plan never falls back to `adb input swipe`; issue #690 separately owns removal of the
60-
existing one-contact fallback. The snapshot helper is stopped before local gesture
61-
instrumentation because Android permits only one instrumentation owner of `UiAutomation`.
58+
- Android's `executeAndroidTouchPlan` adapter seam sends planned touch, including gesture plans plus
59+
the physical movement for scroll and long-press, to provider-native touch injection when
60+
available, otherwise to the bundled instrumentation helper. The helper injects the exact planned
61+
pointer samples. A stationary long-press needs no viewport on the helper path; the executor adds
62+
the paired provider-owned viewport only for provider-native touch. Android touch execution never
63+
falls back to `adb input swipe`. Public scroll durations below one 16 ms planner frame normalize
64+
to that physical minimum and report the executed duration. Scroll evidence reports absolute
65+
injected coordinates against zero-origin extents that include the viewport offset. The snapshot helper is stopped
66+
before local gesture instrumentation because Android permits only one instrumentation owner of
67+
`UiAutomation`.
6268
- iOS converts every planned point to native orientation and feeds the exact arrays to the existing
6369
private XCTest event bridge. macOS lowers a one-contact plan to its drag executor and tvOS lowers
6470
it to remote direction. Core admission and the Apple adapter both consume the same shared
@@ -93,6 +99,9 @@ selectors or refs and therefore cannot claim element-targeting guarantees.
9399
- One-finger pan remains the default and explicit two-finger pan retains pan intent.
94100
- The active viewport is resolved for each gesture, so rotation, keyboard, and window changes do
95101
not use stale geometry.
102+
- On bare ADB, Android scroll and long-press require the bundled touch helper and `UiAutomation`;
103+
helper installation or runtime failure is surfaced directly rather than degrading to an
104+
approximate `adb input swipe`.
96105
- Pointer plans are larger than scalar requests but bounded by duration and the 16 ms sample
97106
cadence; deleting duplicate scalar executors offsets the package cost.
98107

@@ -106,4 +115,5 @@ selectors or refs and therefore cannot claim element-targeting guarantees.
106115
fling/swipe as a quick directional throw and pan as deliberate timed translation.
107116
- Add `two-finger-pan`: rejected because pointer count is topology, not a new motion intent.
108117
- Expose span/angle controls: rejected until a concrete automation use case needs them.
109-
- Consolidate scroll: rejected because edge/content verification and fallback policy are distinct.
118+
- Consolidate scroll command semantics: rejected because edge/content verification is distinct;
119+
only its Android physical touch execution is shared.

src/core/command-descriptor/__tests__/timeout-policy.test.ts

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -127,6 +127,7 @@ test('request envelopes deviating from the default are bounded, reviewed sets',
127127
install: 180_000,
128128
reinstall: 180_000,
129129
install_source: 180_000,
130+
longpress: 210_000,
130131
test: 'unbounded',
131132
};
132133
for (const descriptor of commandDescriptors) {

src/core/command-descriptor/registry.ts

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -703,7 +703,13 @@ export const RAW_COMMAND_DESCRIPTORS = [
703703
daemon: { route: 'interaction', replayScopedAction: true, androidBlockingDialogGuard: true },
704704
dispatch: {},
705705
capability: { apple: APPLE_SIM_AND_DEVICE, android: ANDROID_ALL, linux: LINUX_DEVICE },
706-
timeoutPolicy: interactionTimeoutPolicy('longpress'),
706+
timeoutPolicy: {
707+
...SETTLE_FLAG_PRESERVE_DAEMON_TIMEOUT_POLICY,
708+
// Android's cold path may inspect/install the helper, hand off a running
709+
// snapshot helper, hold for 120 seconds, then use 15 seconds of helper
710+
// completion overhead. Keep that complete route inside the envelope.
711+
envelopeMs: 210_000,
712+
},
707713
postActionObservation: postActionObservation('longpress'),
708714
batchable: true,
709715
},

src/core/interactors/android.ts

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -17,9 +17,9 @@ import {
1717
typeAndroid,
1818
} from '../../platforms/android/input-actions.ts';
1919
import {
20-
performGestureAndroid,
20+
executeAndroidTouchPlan,
2121
readAndroidGestureViewport,
22-
} from '../../platforms/android/multitouch-helper.ts';
22+
} from '../../platforms/android/touch-executor.ts';
2323
import {
2424
readAndroidClipboardText,
2525
writeAndroidClipboardText,
@@ -53,7 +53,7 @@ export function createAndroidInteractor(device: DeviceInfo): Interactor {
5353
type: (text, delayMs) => typeAndroid(device, text, delayMs),
5454
fill: (x, y, text, delayMs) => fillAndroid(device, x, y, text, delayMs),
5555
scroll: (direction, options) => scrollAndroid(device, direction, options),
56-
performGesture: (plan) => performGestureAndroid(device, plan),
56+
performGesture: (plan) => executeAndroidTouchPlan(device, plan),
5757
gestureViewport: () => readAndroidGestureViewport(device),
5858
screenshot: (outPath, options) => screenshotAndroid(device, outPath, options),
5959
snapshot: async (options) => {

src/daemon/__tests__/recording-gestures.test.ts

Lines changed: 21 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -108,26 +108,36 @@ test('scroll augmentation preserves explicit reference frame from platform resul
108108
assert.equal(augmented.y2, 240);
109109
});
110110

111-
test('scroll augmentation preserves explicit pixel travel coordinates', () => {
111+
test('scroll visualization preserves absolute travel in its zero-origin reference frame', () => {
112112
const session = makeSession();
113113
session.snapshot = undefined;
114114

115115
const augmented = augmentScrollVisualizationResult(session, 'scroll', ['down'], {
116116
direction: 'down',
117117
pixels: 240,
118-
x1: 201,
119-
y1: 557,
120-
x2: 201,
121-
y2: 317,
122-
referenceWidth: 402,
123-
referenceHeight: 874,
118+
x1: 211,
119+
y1: 577,
120+
x2: 211,
121+
y2: 337,
122+
referenceWidth: 412,
123+
referenceHeight: 894,
124124
}) as Record<string, unknown>;
125125

126-
assert.equal(augmented.x1, 201);
127-
assert.equal(augmented.y1, 557);
128-
assert.equal(augmented.x2, 201);
129-
assert.equal(augmented.y2, 317);
126+
recordTouchVisualizationEvent(session, 'scroll', ['down'], augmented, {}, 1_500);
127+
128+
assert.equal(augmented.x1, 211);
129+
assert.equal(augmented.y1, 577);
130+
assert.equal(augmented.x2, 211);
131+
assert.equal(augmented.y2, 337);
130132
assert.equal(augmented.pixels, 240);
133+
const event = session.recording?.gestureEvents[0];
134+
assert.equal(event?.kind, 'scroll');
135+
assert.equal(event?.referenceWidth, 412);
136+
assert.equal(event?.referenceHeight, 894);
137+
assert.equal(event?.x, 211);
138+
assert.equal(event?.y, 577);
139+
assert.equal(event?.x2, 211);
140+
assert.equal(event?.y2, 337);
131141
});
132142

133143
test('gesture recording prefers native runner timing when available', () => {

src/platforms/android/__tests__/input-actions.test.ts

Lines changed: 100 additions & 80 deletions
Original file line numberDiff line numberDiff line change
@@ -1,50 +1,120 @@
11
import { test } from 'vitest';
22
import assert from 'node:assert/strict';
33
import { promises as fs } from 'node:fs';
4-
import os from 'node:os';
5-
import path from 'node:path';
64
import {
75
fillAndroid,
6+
longPressAndroid,
87
rotateAndroid,
98
scrollAndroid,
10-
swipeAndroid,
119
typeAndroid,
1210
} from '../input-actions.ts';
13-
import type { DeviceInfo } from '../../../kernel/device.ts';
1411
import { AppError } from '../../../kernel/errors.ts';
1512
import { withScriptedAdb } from '../../../__tests__/test-utils/mocked-binaries.ts';
13+
import { ANDROID_EMULATOR } from '../../../__tests__/test-utils/index.ts';
14+
import { withAndroidAdbProvider, type AndroidTouchInjector } from '../adb-executor.ts';
1615

17-
test('scrollAndroid supports explicit pixel travel distance', async () => {
18-
await withScriptedAdb(
19-
'agent-device-android-scroll-pixels-',
20-
[
21-
'#!/bin/sh',
22-
'printf "%s\\n" "$@" >> "$AGENT_DEVICE_TEST_ARGS_FILE"',
23-
'if [ "$1" = "-s" ]; then',
24-
' shift',
25-
' shift',
26-
'fi',
27-
'if [ "$1" = "shell" ] && [ "$2" = "wm" ] && [ "$3" = "size" ]; then',
28-
' echo "Physical size: 1080x1920"',
29-
' exit 0',
30-
'fi',
31-
'exit 0',
32-
'',
33-
].join('\n'),
34-
async ({ argsLogPath, device }) => {
35-
const result = await scrollAndroid(device, 'down', { pixels: 240, durationMs: 120 });
36-
const args = await fs.readFile(argsLogPath, 'utf8');
16+
test('scrollAndroid plans explicit pixel travel through semantic touch injection', async () => {
17+
const touchCalls: Parameters<AndroidTouchInjector>[0][] = [];
18+
const result = await withAndroidAdbProvider(
19+
{
20+
exec: async () => {
21+
throw new Error('adb must not run');
22+
},
23+
gestureViewport: async () => ({ x: 10, y: 20, width: 1080, height: 1920 }),
24+
touch: async (request) => {
25+
touchCalls.push(request);
26+
return { injected: true };
27+
},
28+
},
29+
{ serial: ANDROID_EMULATOR.id },
30+
async () => await scrollAndroid(ANDROID_EMULATOR, 'down', { pixels: 240, durationMs: 120 }),
31+
);
32+
33+
assert.equal(touchCalls.length, 1);
34+
const touch = touchCalls[0]!;
35+
assert.equal(touch.intent, 'pan');
36+
assert.deepEqual(touch.pointers[0]?.samples[0]?.point, { x: 550, y: 1100 });
37+
assert.deepEqual(touch.pointers[0]?.samples.at(-1)?.point, { x: 550, y: 860 });
38+
assert.equal(result.pixels, 240);
39+
assert.equal(result.durationMs, 120);
40+
assert.equal(result.referenceWidth, 1090);
41+
assert.equal(result.referenceHeight, 1940);
42+
assert.equal(result.x1, 550);
43+
assert.equal(result.y1, 1100);
44+
assert.equal(result.x2, 550);
45+
assert.equal(result.y2, 860);
46+
assert.equal(result.backend, 'provider-native-touch');
47+
assert.equal(result.injected, true);
48+
});
3749

38-
assert.match(args, /shell\ninput\nswipe\n540\n1080\n540\n840\n120\n/);
39-
assert.doesNotMatch(args, /uiautomator|dump/);
40-
assert.equal(result.pixels, 240);
41-
assert.equal(result.durationMs, 120);
42-
assert.equal(result.referenceWidth, 1080);
43-
assert.equal(result.referenceHeight, 1920);
50+
test('scrollAndroid accepts sub-frame public durations at the Android planner minimum', async () => {
51+
const touchCalls: Parameters<AndroidTouchInjector>[0][] = [];
52+
const results = await withAndroidAdbProvider(
53+
{
54+
exec: async () => {
55+
throw new Error('adb must not run');
56+
},
57+
gestureViewport: async () => ({ x: 0, y: 0, width: 1080, height: 1920 }),
58+
touch: async (request) => {
59+
touchCalls.push(request);
60+
},
4461
},
62+
{ serial: ANDROID_EMULATOR.id },
63+
async () => {
64+
const outputs: Record<string, unknown>[] = [];
65+
for (const durationMs of [0, 15]) {
66+
outputs.push(await scrollAndroid(ANDROID_EMULATOR, 'down', { durationMs }));
67+
}
68+
return outputs;
69+
},
70+
);
71+
72+
assert.deepEqual(
73+
touchCalls.map((call) => call.durationMs),
74+
[16, 16],
75+
);
76+
assert.deepEqual(
77+
results.map((result) => result.durationMs),
78+
[16, 16],
4579
);
4680
});
4781

82+
test('longPressAndroid sends a stationary semantic touch plan', async () => {
83+
const touchCalls: Parameters<AndroidTouchInjector>[0][] = [];
84+
const result = await withAndroidAdbProvider(
85+
{
86+
exec: async () => {
87+
throw new Error('adb must not run');
88+
},
89+
gestureViewport: async () => ({ x: 10, y: 20, width: 300, height: 500 }),
90+
touch: async (request) => {
91+
touchCalls.push(request);
92+
},
93+
},
94+
{ serial: ANDROID_EMULATOR.id },
95+
async () => await longPressAndroid(ANDROID_EMULATOR, 30, 40, 750),
96+
);
97+
98+
assert.deepEqual(touchCalls, [
99+
{
100+
topology: 'single',
101+
intent: 'longPress',
102+
durationMs: 750,
103+
viewport: { x: 10, y: 20, width: 300, height: 500 },
104+
pointers: [
105+
{
106+
pointerId: 0,
107+
samples: [
108+
{ offsetMs: 0, point: { x: 30, y: 40 } },
109+
{ offsetMs: 750, point: { x: 30, y: 40 } },
110+
],
111+
},
112+
],
113+
},
114+
]);
115+
assert.equal(result.backend, 'provider-native-touch');
116+
});
117+
48118
test('rotateAndroid locks auto-rotate and sets user rotation', async () => {
49119
await withScriptedAdb(
50120
'agent-device-android-rotate-landscape-left-',
@@ -59,56 +129,6 @@ test('rotateAndroid locks auto-rotate and sets user rotation', async () => {
59129
);
60130
});
61131

62-
test('swipeAndroid invokes adb input swipe with duration', async () => {
63-
const tmpDir = await fs.mkdtemp(path.join(os.tmpdir(), 'agent-device-swipe-test-'));
64-
const adbPath = path.join(tmpDir, 'adb');
65-
const argsLogPath = path.join(tmpDir, 'args.log');
66-
await fs.writeFile(
67-
adbPath,
68-
'#!/bin/sh\nprintf "%s\\n" "$@" > "$AGENT_DEVICE_TEST_ARGS_FILE"\nexit 0\n',
69-
'utf8',
70-
);
71-
await fs.chmod(adbPath, 0o755);
72-
73-
const previousPath = process.env.PATH;
74-
const previousArgsFile = process.env.AGENT_DEVICE_TEST_ARGS_FILE;
75-
process.env.PATH = `${tmpDir}${path.delimiter}${previousPath ?? ''}`;
76-
process.env.AGENT_DEVICE_TEST_ARGS_FILE = argsLogPath;
77-
78-
const device: DeviceInfo = {
79-
platform: 'android',
80-
id: 'emulator-5554',
81-
name: 'Pixel',
82-
kind: 'emulator',
83-
booted: true,
84-
};
85-
86-
try {
87-
await swipeAndroid(device, 10, 20, 30, 40, 250);
88-
const args = (await fs.readFile(argsLogPath, 'utf8')).trim().split('\n').filter(Boolean);
89-
assert.deepEqual(args, [
90-
'-s',
91-
'emulator-5554',
92-
'shell',
93-
'input',
94-
'swipe',
95-
'10',
96-
'20',
97-
'30',
98-
'40',
99-
'250',
100-
]);
101-
} finally {
102-
process.env.PATH = previousPath;
103-
if (previousArgsFile === undefined) {
104-
delete process.env.AGENT_DEVICE_TEST_ARGS_FILE;
105-
} else {
106-
process.env.AGENT_DEVICE_TEST_ARGS_FILE = previousArgsFile;
107-
}
108-
await fs.rm(tmpDir, { recursive: true, force: true });
109-
}
110-
});
111-
112132
test('typeAndroid chunks ASCII input text for shell fallback', async () => {
113133
await withScriptedAdb(
114134
'agent-device-android-type-ascii-chunked-',

0 commit comments

Comments
 (0)