feat: unify gesture planning and multi-touch execution (#1212)

* feat: unify gesture planning and multi-touch execution

* fix: correct unified gesture helper behavior

* refactor: tighten unified gesture architecture

* fix: preserve gesture routing contracts

* test: account for fresh gesture viewport

* refactor: remove retired gesture series

* fix: preserve example app navigation targets

* test: reconcile unified gestures with helper ownership

* docs: update Android helper gesture protocol

* fix: refresh Maestro percentage swipe frames

* refactor: remove stale Maestro frame cache

* fix: harden unified gesture execution

* fix: model gesture viewport in providers

* refactor: remove legacy gesture paths

* fix: remove unused swipe preset parser

* refactor: tighten unified gesture boundaries

* fix: close gesture review gaps

* fix: preserve gesture compatibility contracts

* fix: preserve multi-touch recording semantics

* fix: refresh Apple runner state after app relaunch

* test: lock Apple fling fallback route

* fix: close Apple runner review gaps

* refactor: tighten unified gesture seams

* refactor: consolidate gesture planning policy

* fix: preserve swipe response compatibility

* fix: keep gesture lab aligned with replay coordinates

Author: thymikee

.github/workflows/ios.yml

@@ -31,6 +31,7 @@ jobs:
       AGENT_DEVICE_STATE_DIR: ${{ github.workspace }}/.tmp/agent-device-state
       AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH: ${{ github.workspace }}/.tmp/ios-runner-derived
       AGENT_DEVICE_IOS_PREPARE_TIMEOUT_MS: '420000'
+      AGENT_DEVICE_XCUITEST_INCLUDE_UNIT_TESTS: '1'
     steps:
       - name: Checkout
         uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
@@ -56,6 +57,17 @@ jobs:
           runtime-version: ${{ env.IOS_RUNTIME_VERSION }}
           preferred-device-name: iPhone 17 Pro
 
+      - name: Run iOS runner gesture and lifecycle regressions
+        run: |
+          XCTESTRUN_PATH="$(find "$AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH/Build/Products" -maxdepth 1 -name '*.xctestrun' -print -quit)"
+          test -n "$XCTESTRUN_PATH"
+          xcodebuild test-without-building \
+            -xctestrun "$XCTESTRUN_PATH" \
+            -destination "platform=iOS Simulator,id=${{ steps.ios-simulator.outputs.simulator-udid }}" \
+            -only-testing:AgentDeviceRunnerUITests/RunnerTests/testSinglePointerFlingFallsBackToXCTestCoordinateDragWhenPrivateSynthesisFails \
+            -only-testing:AgentDeviceRunnerUITests/RunnerTests/testMissingBundleCommandInvalidatesCompleteCachedTargetState \
+            -only-testing:AgentDeviceRunnerUITests/RunnerTests/testCachedTargetInvalidationClearsProcessBoundState
+
       - name: Prepare iOS runner
         run: |
           pnpm clean:daemon

AGENTS.md

@@ -61,6 +61,7 @@ Single-context repo. Read `CONTEXT.md` for domain language and testing/architect
   - prefer deep modules over mechanical splits: extract when it improves locality for a concept callers already need, not just to reduce line count.
 - Before finalizing a code change, do one tightening pass over touched and directly adjacent areas: drop obsolete code, redundant tests, stale helpers/fixtures, and needless duplication made unnecessary by the change.
 - Prefer existing helpers. Add a helper only when it reduces real repetition or clarifies domain behavior.
+- Prefer composition at platform boundaries: public aliases normalize into shared primitives, and providers contribute transport/device bindings instead of cloning interaction runtimes.
 - When adding new guidance, examples, schemas, or command metadata, decide whether it belongs in the command surface, CLI grammar, CLI help, MCP projection, or daemon runtime before editing.
 - Prefer updating existing domain vocabulary in `CONTEXT.md` when naming a new durable module concept. Do not coin parallel names in docs, tests, and code.
 

CONTEXT.md

@@ -47,6 +47,8 @@
 - Runner command traits: per-command-type classification for iOS/macOS runner lifecycle behavior, distinct from the public command surface and daemon command registry. The Swift runner traits classify interaction, read-only, and runner-lifecycle axes for XCTest execution; Swift resolves the alert command as read-only only for its `get` action. The TypeScript runner command traits classify daemon-side runner send/recovery policy such as read-only retry routing, readiness probes, and recent-healthy-mutation preflight skips; the TypeScript table is command-type keyed and currently classifies alert as read-only for daemon retry policy. Each side keeps one source of truth keyed by runner command type.
 - Coordinate-first resolved element activation: iOS/macOS runner interaction pattern where a selector or text query resolves the semantic `XCUIElement`, then activation uses the element's resolved center coordinate when a frame is available. This keeps target selection semantic while avoiding `XCUIElement.tap()` post-action element re-resolution after normal navigation. tvOS remains focus/remote-driven.
 - Interaction dispatch path: one concrete route an interaction command takes to the device (runtime selector/ref resolution, direct iOS selector, native ref via web clickRef, coordinate, maestro non-hittable fallback). Every path classifies every guarantee in the ADR 0011 registry.
+- Gesture plan: typed, platform-neutral normalization of one- or two-contact gesture intent into bounded pointer trajectories. Contact topology is separate from motion; two-contact intent remains pan/pinch/rotate/transform even when native injection shares one executor. See ADR 0013.
+- Multi-touch geometry: the internal initial span and angle plus centroid translation, scale, and rotation used to build both contact trajectories. Geometry is viewport-aware and fails early when the requested motion cannot fit; it is not a public tuning surface.
 - Guarantee cell: one (dispatch path, guarantee) entry in `src/contracts/interaction-guarantees.ts`, classified as runtime/runner/delegated/inapplicable/waived. Completeness is a compile error; honesty is gate-tested.
 - Owned waiver: a `gap:`-prefixed waived cell carrying a `trackingIssue` URL. Waivers are diffable debt with an owner, never folklore.
 - Parity table: golden JSON fixture under `contracts/fixtures/` consumed by both vitest and the runner's gated Swift tests, so a cross-language rule (e.g. tap-point policy) cannot drift silently. Change the rule only via the table.

android-multitouch-helper/README.md

@@ -18,7 +18,16 @@ sh ./scripts/build-android-multitouch-helper.sh "$VERSION" .tmp/android-multitou
 ## Run
 
 ```sh
-PAYLOAD="$(printf '%s' '{"kind":"transform","x":672,"y":1500,"dx":80,"dy":-40,"scale":1.8,"degrees":35,"durationMs":700}' | base64)"
+PAYLOAD_JSON='{
+  "protocol":"android-multitouch-helper-v1",
+  "kind":"transform",
+  "durationMs":32,
+  "pointers":[
+    {"pointerId":0,"samples":[{"offsetMs":0,"x":120,"y":180},{"offsetMs":16,"x":130,"y":175},{"offsetMs":32,"x":140,"y":170}]},
+    {"pointerId":1,"samples":[{"offsetMs":0,"x":120,"y":260},{"offsetMs":16,"x":135,"y":270},{"offsetMs":32,"x":150,"y":280}]}
+  ]
+}'
+PAYLOAD="$(printf '%s' "$PAYLOAD_JSON" | base64)"
 adb install -r -t ".tmp/android-multitouch-helper/agent-device-android-multitouch-helper-$VERSION.apk"
 adb shell am instrument -w \
   -e payloadBase64 "$PAYLOAD" \
@@ -30,12 +39,19 @@ adb shell am instrument -w \
 The APK emits instrumentation result records using
 `agentDeviceProtocol=android-multitouch-helper-v1`.
 
+Before planning a gesture, the runtime invokes the same instrumentation with
+`-e mode viewport`. That read-only mode waits for `UiAutomation` window state and returns the
+active application window bounds, falling back to the active accessibility root when Android does
+not expose interactive-window metadata. It does not use or cache the physical display size.
+
 Successful results include:
 
 - `ok=true`
 - `helperApiVersion=1`
-- `kind` (`swipe`, `pinch`, `rotate`, or `transform`)
+- `kind` (`swipe` for one planned pointer path or `transform` for two)
 - `injectedEvents`
 - `elapsedMs`
 
+Viewport results use `kind=viewport` plus `x`, `y`, `width`, and `height`.
+
 Failures return `ok=false`, `errorType`, and `message`.