feat(android): one persistent automation helper owning snapshot + viewport + canonical injection

[thymikee]

Closes #1275.

What

One persistent Android automation helper now owns snapshot capture, gesture viewport resolution, and canonical one-/two-pointer plan injection. The separate one-shot multitouch helper APK is deleted — atomic replacement, no legacy fallback, no second production path.

• Java (android/snapshot-helper): touch scheduling/injection is extracted into focused classes instead of growing SnapshotInstrumentation — TouchPlan (payload parse/validate), TouchPlanInjector (paced injectInputEvent with cancel-on-failure), PointerEventSchedule (moved verbatim from the multitouch helper, including the per-step waitForDispatch semantics), GestureViewportReader. SnapshotInstrumentation gains only dispatch: one-shot -e mode snapshot|viewport|gesture, and viewport <id> / gesture <id> <payloadBase64> commands on the existing session socket protocol. helperApiVersion bumps to 2; the response protocol literal, APK/package/runner identity, and artifact names are unchanged (no orphan installs, no release-pipeline churn).
• TS: new src/platforms/android/touch-helper.ts replaces multitouch-helper.ts/multitouch-helper-install.ts. A live snapshot-helper session executes touch commands directly over its socket; without one, the same APK runs one-shot. Gestures never start or stop the session — the forced stopAndroidSnapshotHelperSessionForDevice() before every local gesture (the instrumentation churn measured in refactor: replace Maestro compat with typed direct engine #1217 validation) is gone. Install rides the shared ensureAndroidSnapshotHelper (sha-256 verified, version-coded).
• Error semantics: a structured helper ok=false propagates and leaves the session alive; a transport-level session failure stops the session and propagates without a one-shot retry for gestures (events may already be partially injected — double-injection is worse than a loud error). Viewport reads are idempotent and do retry one-shot after a session error.
• Backend evidence: gestures now report backend: "android-helper" with helperTransport: "persistent-session" | "instrumentation" (previously android-multitouch-helper).
• ADR 0013 amended: the "snapshot helper is stopped before local gesture instrumentation" clause is replaced by the shared-helper model.
• scripts/check-affected android-helpers rule now also covers android/ime-helper/ (was previously untracked by any build-ownership rule).

Prototype gate (issue requirement) — measured on a live emulator

Identical scripted sequence (open → snapshot → 5 scroll passes w/ snapshot each → short pan → 2 snapshots → 5 s pan → pinch/rotate/two-finger pan/transform → close) against the repo test app (gesture-canary screen), branch vs origin/main baseline, same emulator (Pixel 9 Pro XL API 37). The multitouch APK was uninstalled from the device before the branch run.

1. Touch request latency (command wall time):

gesture	main	branch
short pan (300 ms)	2184 ms	1048 ms
pinch	1955 ms	987 ms
rotate	2081 ms	1015 ms
two-finger pan	2519 ms	1347 ms
transform	2721 ms	1467 ms
scroll pass	2.2–4.6 s	1.0–1.3 s

2. Touch → two stable snapshots without process restart: on main the first snapshot after a gesture reports helperSessionReused: false (forced session restart — the churn); on branch both report helperSessionReused: true. Post-gesture snapshots: ~1.5 s (main) → ~0.8 s (branch).

3. Five consecutive scrollable passes: all snapshots non-empty and correct on both variants.

4. Long one-pointer pan (5 s): green on both. One gate sample showed helper-elapsed 8.3 s on the session transport; three follow-up repeats measured 5.4–6.0 s, matching one-shot (6.0 s) and main (5.8 s) — one-off emulator noise, not a transport property.

5. Canary parity (plans and app-observable effects unchanged): injectedEvents identical per gesture kind on main and branch (pan 21, 5 s pan 315, pinch 23, rotate 23, two-finger pan 48, transform 54 — the planner is untouched and the injector was ported verbatim including upstream's waitForDispatch/cancel-timing changes). App canaries flip identically (pan/pinch/rotate changed yes), final transform state equal within recognizer frame noise (scale 0.65 both; rotate 60° vs 58°).

The one-shot transport was additionally live-verified with sessions disabled (AGENT_DEVICE_ANDROID_SNAPSHOT_HELPER_SESSION=0): gestures report helperTransport: "instrumentation", helperApiVersion: 2, canaries flip — with the multitouch APK absent.

Verification

• pnpm typecheck, pnpm lint, pnpm check:tooling, pnpm check:fallow clean; unit suite 462 files / 4092 tests green (one known CPU-contention flake in runtime-hints.test.ts, passes in isolation).
• New touch-helper.test.ts pins: one-shot arg/payload shape (android-touch-plan-v1), timeout budget max(45 s, durationMs + 15 s), result envelope, session gesture with no am instrument call and session left alive, ok=false keeps the session, malformed/stale session response stops the session without one-shot gesture retry, viewport one-shot fallback after session error.
• Helper APK build runs the moved PointerEventScheduleTest (upstream waitForDispatch expectations) as part of package:android-snapshot-helper.

Notes for review

• Subsumes perf(android): reuse touch-helper preparation across scroll viewport and injection #1250 (touch-prep reuse); coordinates with Harden Android snapshot helper so stock UIAutomator fallback can be removed #1253 (this PR does not touch the snapshot capture path or its stock fallback).
• The APK keeps the android-snapshot-helper name/identity; renaming the artifact family to "automation helper" was deliberately deferred to avoid release-pipeline and installed-package churn in the same change.
• Limrun/WebDriver providers are untouched: provider-native touch still short-circuits in touch-executor.ts before any helper involvement.

[github-actions]

PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-07-16 15:04 UTC

[github-actions]

Size Report

Metric	Base	Current	Diff
JS raw	1.7 MB	1.7 MB	+1.3 kB
JS gzip	557.5 kB	557.8 kB	+287 B
npm tarball	673.0 kB	673.2 kB	+201 B
npm unpacked	2.4 MB	2.4 MB	+525 B

Startup median (7 runs, lower is better):

Scenario	Base	Current	Diff
CLI --version	21.6 ms	22.2 ms	+0.6 ms
CLI --help	44.0 ms	45.6 ms	+1.5 ms

Top changed chunks:

Chunk	Raw diff	Gzip diff
dist/src/internal/daemon.js	+1.6 kB	+381 B

[thymikee]

Review findings at 0b033f19db61d1294a179dfe2acde703edc06221:

1. Blocking: stop the persistent helper before viewport one-shot fallback. runAndroidSnapshotHelperSessionTouchCommand intentionally keeps the session alive for a structured ok=false response, but readAndroidTouchHelperViewport catches every error and immediately starts one-shot instrumentation. The still-live helper may retain UiAutomation, so the fallback can contend with it and mask the original structured failure. Stop the session before retrying one-shot, and add a regression for a structured ok=false viewport response.

2. Please split the new ownership boundaries before merge. SnapshotInstrumentation.java is now 908 lines and touch-helper.test.ts is 674 lines. The linked issue explicitly required focused extraction rather than growing the snapshot instrumentation. Extract touch command/response dispatch and split one-shot tests from the persistent-session socket harness.

3. Update the checked-in helper contract: the implementation reports helperApiVersion=2, while android/snapshot-helper/README.md still documents version 1.

The architecture direction, no-double-injection policy, 23 green checks, and live emulator measurements otherwise look good.

[thymikee]

All three findings addressed, CI green (23/23):

1. Viewport session-stop ordering — fixed in 5961b92: a structured ok=false session viewport response now stops the session before the one-shot am instrument retry runs (one instrumentation may own UiAutomation). Pinned by the new regression test a structured ok=false viewport response stops the session before the one-shot retry, which asserts inside the one-shot exec that the session is already dead.

2. Extraction + test split (ab1743c) — touch dispatch is out of SnapshotInstrumentation.java (908 → 803 lines): TouchCommandHandler.java (72 lines; one-shot + session viewport/gesture population, parameterized by the connected UiAutomation), SessionResponseWriter.java (54 lines; session response bundle/ok/error encoding + header sanitization), and HelperProtocol.java (10 lines; shared protocol/apiVersion/outputFormat constants). Wire format is byte-identical — same header keys, values, and error shapes; TouchPlan/TouchPlanInjector/PointerEventSchedule/GestureViewportReader untouched. The test file split mirrors it: touch-helper.test.ts (354 lines; normalize/parse/one-shot gesture + viewport + result envelope), touch-helper-session.test.ts (356 lines; persistent-session transport incl. the fake-session harness), shared device/plan/install-probe fixtures moved to touch-helper.fixtures.ts (80 lines). All 19 tests preserved.

3. README — android/snapshot-helper/README.md now documents helper API v2: the one-shot -e mode snapshot|viewport|gesture modes, the android-touch-plan-v1 payload shape (kind swipe|transform, durationMs, strictly-increasing offsetMs samples from 0 to durationMs), the persistent-session socket commands (snapshot/viewport/gesture/quit <requestId>) with headers-only viewport/gesture responses, and the note that the response protocol literal stays android-snapshot-helper-v1.

Post-refactor live smoke on emulator-5556 (rebuilt APK, throwaway state dir): snapshot --json → backend android-helper, helperApiVersion 2, helperTransport persistent-session (sessionReused true on warm session); gesture pinch 0.5 672 1190 --json → backend android-helper, helperTransport persistent-session, helperKind transform, injectedEvents 23.

🤖 Addressed by Claude Code

[thymikee]

Re-review of current head ab1743c46: the previous viewport fallback, file-split, and helper API documentation findings are addressed, but this head is still not merge-ready.

• P1: prepareAndroidTouchHelper installs/replaces the snapshot-helper APK but does not stop an already-running persistent session when install.installed === true. The snapshot path explicitly does this in src/platforms/android/snapshot.ts; without it, a gesture can be sent to the old process after a version/SHA replacement, or fail with no gesture retry. Please invalidate the session after a replacement and add a regression with a live fake session.
• P2: GestureViewportReader.read never recycles the AccessibilityWindowInfo objects returned by UiAutomation.getWindows(), including its early-return paths. This is now a repeated persistent-session operation, so copy the bounds and recycle every window in a finally.
• P3: ADR 0002 still says touch synthesis must stop the persistent snapshot helper, while ADR 0013 now specifies shared-helper touch execution. Please update ADR 0002 so the architecture sources do not conflict.

All current CI checks are green; the remaining gaps are lifecycle/resource correctness not exercised by those checks.

[thymikee]

All three re-review findings addressed in 9c13215, CI green (23/23):

P1 — session invalidation after APK replacement. prepareAndroidTouchHelper now mirrors the snapshot path: when ensureAndroidSnapshotHelper reports install.installed, the persistent session started against the previous binary is stopped before any touch command, so the gesture runs one-shot against the fresh install instead of the stale session socket. Regression (live fake session): an APK replacement stops the stale session and the gesture runs one-shot in touch-helper-session.test.ts — an outdated-install probe (new outdatedVersionAdb fixture) drives a real reinstall, then asserts the session socket received no gesture command, helperTransport === 'instrumentation', and the session is gone. Verified it fails without the fix (gesture reached the old socket as persistent-session).

P2 — AccessibilityWindowInfo recycling. GestureViewportReader.read now does a single pass over getWindows() copying the active/focused-application bounds and first-application fallback bounds into locals, recycles every window (including non-application ones) in a finally, then applies the unchanged precedence: active/focused app bounds → root-in-active-window bounds → fallback app bounds → IllegalStateException. APK rebuilt; Java PointerEventScheduleTest passes.

P3 — ADR alignment. ADR 0002's touch-synthesis paragraph is amended (2026-07, issue #1275, consistent with ADR 0013) to the shared-helper model: one bundled helper owns snapshot capture, gesture viewport resolution, and planned-touch injection; a live persistent session executes touch commands directly over its socket, one-shot otherwise. The old stop-before-gesture requirement remains as historical context.

Post-fix live smoke on emulator-5556 with the rebuilt APK: first snapshot triggered a real replacement (installReason: mismatched — which exercised the new invalidation live), then snapshot -i --json → backend android-helper, helperApiVersion 2, helperTransport persistent-session (sessionReused: true); gesture pinch 0.5 672 1190 --json → persistent-session, kind transform, injectedEvents: 23 — viewport resolution through the recycling reader working on the session path.

🤖 Addressed by Claude Code

[thymikee]

Re-review finding on current head 9c132152c (blocking readiness): the consolidated touch path does not honor the Android provider's snapshotHelperArtifact.

captureAndroidUiHierarchy() resolves options.helperArtifact ?? adbProvider.snapshotHelperArtifact, but prepareAndroidTouchHelper() always calls the local bundled-artifact resolver. For an ADB-backed provider that supplies the helper artifact and has no native touch override, snapshots can use the provider artifact while scroll/gesture/longpress fallback tries to resolve/install a different local artifact (or fails when it is unavailable). That violates #1275's single-helper ownership contract.

Please resolve touch/viewport from adbProvider.snapshotHelperArtifact when present, using the same artifact contract as snapshots, and add a provider regression with ADB + snapshotHelperArtifact and no native touch implementation.

Two small docs corrections can ride the same update: ADR 0002 should explicitly exempt non-idempotent gesture failures from one-shot retry, and the helper README should describe a persistent process with one short-lived socket connection per request (the server closes each accepted connection).

[thymikee]

Addressed in ff6031b, CI green (23/23):

Provider artifact precedence. prepareAndroidTouchHelper now resolves the helper artifact exactly like snapshot capture: the scoped adbProvider.snapshotHelperArtifact when present, otherwise the bundled resolver (its strict unavailable error preserved). That single artifact drives both the install decision (ensureAndroidSnapshotHelper) and the instrumentationRunner used for one-shot commands, so an ADB-backed provider that supplies a helper artifact but no native touch override runs snapshots and gestures against the same helper. Regression: a provider-supplied snapshotHelperArtifact overrides the bundled artifact for touch in touch-helper.test.ts — a provider with snapshotHelperArtifact (distinct packageName/runner) and no native touch pins the provider packageName on the install probe, the provider apkPath on the install call, the provider runner on the am instrument args, and that the bundled resolver is never invoked. Verified it fails without the fix.

Doc riders. ADR 0002 now states explicitly that one-shot retry after a failed session command applies only to idempotent reads (viewport, and only after the failed session is stopped); gesture injection is not idempotent — events may already be partially injected — so it surfaces directly. Helper README session transport corrected: a persistent process serving one short-lived socket connection per request (the server closes each accepted connection), not a single long-lived connection.

Live emulator smoke re-run on the same head: snapshot -i → apiVersion 2, persistent-session; gesture pinch 0.5 672 1190 → persistent-session, injectedEvents: 23.

🤖 Addressed by Claude Code

[thymikee]

Current-head re-review (ff6031b7f): provider artifact resolution and both docs findings are improved, but one blocking crossover remains.

prepareAndroidTouchHelper() now selects adbProvider.snapshotHelperArtifact, yet it only stops an existing helper session when installation occurs. Persistent sessions are keyed by device and runAndroidSnapshotHelperSessionTouchCommand() does not compare helper identity. If artifact A already owns a live session, then a provider supplies already-current artifact B for that device, no install occurs and the gesture is sent to A's session despite B being selected. Snapshot capture already invalidates on session identity mismatch.

Please apply the same identity guard before touch/viewport session reuse and add a regression with: live session from A + provider artifact B reported current + no install; assert A is not used and B's runner handles the command.

[thymikee]

Addressed in 3d0aad9, CI green (23/23; the iOS Smoke Tests run needed one rerun for an unrelated testAlertResolutionCannotBypassRequestedDeadline wall-clock flake — the commit touches only Android TS).

Helper-identity guard on touch session reuse. The persistent session record now stores its helper binary identity (packageName, runner, helperVersion, helperVersionCode — the same values that feed the snapshot session identity), and runAndroidSnapshotHelperSessionTouchCommand takes the requesting command's identity from the selected artifact. Mismatch (packageName/runner always; version/versionCode when both sides define them) stops the live session and returns undefined, so the touch command runs one-shot against the selected artifact — gestures still never start sessions, and the next snapshot restarts one with the right artifact. Matching identity reuses the session exactly as before. Snapshot identity string and snapshot behavior are unchanged.

Regression: a provider artifact that mismatches the live session helper stops it and runs one-shot in touch-helper-session.test.ts — live fake session from artifact A (bundled fixture identity), then a gesture through an ADB provider supplying already-current artifact B (distinct packageName/runner, installReason: 'current', no install): A's socket receives zero gesture commands, A's session is stopped, the one-shot am instrument args end with B's runner, helperTransport === 'instrumentation'. Stash-verified it fails without the guard (gesture reached A's socket as persistent-session).

Live emulator smoke re-run proves matching-identity reuse intact: snapshot -i → apiVersion 2, persistent-session; gesture pinch 0.5 672 1190 → persistent-session, injectedEvents: 23.

🤖 Addressed by Claude Code

[thymikee]

Current-head re-review (3d0aad965): the package/runner crossover is fixed, but same-version binary crossover remains.

The persistent-session identity stores package, runner, version, and versionCode, but not the artifact SHA. ensureAndroidSnapshotHelper() memoizes install decisions by SHA without evicting older SHA entries. After same-package/version artifact B replaces artifact A, a later command selecting A can receive its old cached current result; the new identity guard then considers B's live session compatible and sends A's command to B.

This is a supported case: the install path already replaces same-version helpers when APK bytes differ. Include the artifact checksum in the live-session identity (and requested touch identity), invalidate stale cached decisions as needed, and add the missing regression: A and B share package/runner/version/versionCode but have different SHA; after B owns the live session, selecting A must not reuse B.

[thymikee]

Addressed in 4ec5915, CI green (23/23, no reruns needed):

sha256 joins the helper identity end-to-end. AndroidSnapshotHelperCaptureOptions gains helperSha256 (snapshot capture passes artifact.manifest.sha256 alongside version/versionCode), the session record stores it, createSessionIdentity includes it — snapshot session reuse is now sha-aware too, consistent with the install path's existing same-version sha check — and the touch identity guard compares it with the same both-defined rule (touchSessionHelperIdentity passes the selected artifact's sha). A same-version different-bytes crossover now stops the live session and runs the command one-shot; matching sha reuses as before.

Install-memo eviction. rememberInstalledSnapshotHelper now evicts every other cached decision for the same deviceKey+packageName (any versionCode/sha, mirroring forgetAndroidSnapshotHelperInstall's prefix matching) before recording, so installing B invalidates A's stale current memo and a later command selecting A re-inspects the device (sha check) instead of skipping straight to reuse.

Regressions, both stash-verified to fail without their fix:

• a same-version artifact with a different sha stops the live session and runs one-shot (touch-helper-session.test.ts) — B owns the live fake session (same packageName/runner/version/versionCode, different sha); a gesture selecting A sends zero commands to B's socket, stops the session, and completes one-shot (helperTransport: 'instrumentation').
• installing a same-version different-sha helper evicts the stale install memo (snapshot-helper.test.ts) — A cached as current, B installed over it, selecting A again re-inspects and reinstalls (reason: 'mismatched', installed: true) instead of serving the stale memo.

Live emulator smoke: snapshot -i → apiVersion 2, persistent-session; gesture pinch 0.5 672 1190 → persistent-session, injectedEvents: 23 — matching-sha reuse intact.

🤖 Addressed by Claude Code

[thymikee]

Reviewed exact head 4ec5915858c19e9a8044a7acf5cdf411a2b7cf69.

Code review is clean. The APK SHA now participates end-to-end in persistent helper identity (snapshot session identity and touch-session reuse), and recording a confirmed/current install decision evicts competing memo entries for the same device + package. That closes the same-version, different-bytes A → B → A crossover. Both new regressions are revert-sensitive: without SHA identity the gesture reaches the wrong live session, and without memo eviction the final A selection incorrectly returns the stale cached decision.

One residual merge-readiness requirement remains under the repository device-evidence policy: the current live evidence proves matching-SHA helper reuse, but it does not exercise a real emulator crossover where A and B share package/runner/version/versionCode and differ only in APK bytes. Please capture that A → B → A run before merge, proving each selected artifact is installed/re-inspected and no command crosses into the other live helper session.

The coordinator will handle ready-for-human policy/labeling.