architecture: 3-layer device control (introspection/interaction/flow) + Session Arbiter to end multi-runner foreground contention

[Lykhoyda]

Summary

This session repeatedly stalled driving a single iOS flow because multiple control planes competed for one simulator with nothing serializing them. This proposes a 3-layer device-control architecture + a stateful Session Arbiter in the cdp-bridge, with a small shippable Phase 1 that prevents the root cause.

Related: #186 (maestro-mcp interop), #194 (iOS verification UX / recovery loops), #199 (native-log-first / clearState gotchas), #201 (maestro_run --app-file gap).

Root cause (observed via ps aux)

Three iOS XCUITest-family runners + an orphaned daemon were alive at once:

• RnFastRunner.app — the in-tree XCTest runner that should own iOS (PR feat(rn-device): iOS-first MVP — replace external agent-device iOS surface #164 / D1219) ✅
• AgentDeviceRunner.app + AgentDeviceRunnerUITests-Runner.app — stale legacy upstream, still alive ❌
• agent-device/.../daemon.js + ~/.agent-device/daemon.{json,lock} — orphaned from a prior day ❌
• RN_DEVICE_KILL_LEGACY — unset (the documented kill-switch never fired)

The legacy AgentDeviceRunner (a separate app) stole foreground from the app-under-test → iOS paused its JS thread → CDP wedged (cdp_status → "JS thread paused"). The agent then misattributed it to "the XCTest tier," fled to raw maestro-runner CLI, burned ~7 recovery attempts per wedge with no backstop, and drifted across three tool surfaces (plugin device_*/maestro_run, standalone maestro MCP, raw CLI) for one job.

The defect was duplicate interaction runners, not any single tool. pkill -f AgentDeviceRunner + clearing the daemon lock (done manually) immediately restored RnFastRunner ownership and un-wedged CDP.

North-star: three layers by capability

Layer 3 · FLOW-REPLAY      maestro-runner (Golang + WDA) — authored .yaml E2E / CI only.
                           NOT a primitive tap driver.
Layer 2 · INTERACTION      iOS: RnFastRunner (in-tree XCTest) / Android: agent-device.
  (primitive UI drive)     behind device_* + cdp_interact (fiber path). ONE runner/platform.
                           Injects in-process → keeps app-under-test foregrounded.
Layer 1 · INTROSPECTION    CDP/Hermes (read-mostly): store/network/component-tree/mmkv/native.
  (internal-state reads)   Never the way to DRIVE the UI.

Contracts: one mechanism per capability tier; L1+L2 complementary (drive with XCTest, assert with CDP — they coexist once the legacy thief is gone); L3 whole-flow only; legacy agent-device retired on iOS (Android-only).

Why Layer 2 (RnFastRunner) earns its own existence (it was debated)

maestro could almost absorb L2, but can't match: (1) sub-second primitives — maestro's unit is a flow (~3–5 s WDA/parse/startup per run) vs RnFastRunner's persistent in-process rig (~200 ms/IPC); (2) @ref-handle interaction tied to a live snapshot vs maestro's per-command selector re-resolution (which is what broke when a refactor changed testIDs this session); (3) fiber-tree-resolved taps (cdp_interact bypasses AutoFill/keyboard-drop, reaches non-a11y components) — maestro only sees the a11y tree; (4) shared in-process session with CDP — a device_press then immediate cdp_store_state is one serialized handoff; maestro is a separate process and can't interleave a tap with a CDP read at sub-call granularity. The drive-with-L2/assert-with-L1 per-step loop needs L2 in-process.

RnFastRunner's fragility this session was not inherent — it was the legacy runner fighting it. Fix is single-ownership enforcement, not removal.

Session Arbiter (in cdp-bridge — already long-lived, already owns CDP)

DeviceSessionArbiter
  state: platform · foregroundOwner('introspection'|'interaction'|'flow'|null) · runnerHealth
  acquire(plane)        → ensure app foreground, block conflicting plane, return lease
  release(plane)        → free the floor
  ensureSingleRunner()  → kill stale AgentDeviceRunner + agent-device daemon, clear lock files
  recoverWedge()        → foreground app-under-test, re-probe (BOUNDED: 1/call, N/session)

• Boot-time (SessionStart): ensureSingleRunner() — pkill -f AgentDeviceRunner, kill orphaned daemon, rm ~/.agent-device/daemon.{json,lock}, RN_DEVICE_KILL_LEGACY=1 default-on (opt-out via =0).
• Per-call: tools call acquire(plane). CDP reads that detect JS-paused trigger recoverWedge() (foreground + reprobe) before returning the wedge error — instead of the caller rediscovering the fix over 7 attempts.

Why the bridge (not a hook or daemon): it's the only component both long-lived and already owning a device session. A hook can do the boot kill but can't serialize across later calls. A separate daemon re-introduces the orphaned-lock hazard that caused this. The bridge already is the L1 chokepoint.

Failure handling

Failure (observed)	Detection	Recovery
Stale runner steals foreground	>1 iOS runner	kill legacy at boot; per-call single-owner assert
CDP wedge (JS paused)	probe timeout	foreground app + reprobe once (bounded), then error
Plane race (CDP read mid-flow)	conflicting lease	serialize; refuse mid-flow CDP read with a clear message
clearState needs --app-file (#201)	flow has clearState + iOS	auto-resolve .app, inject --app-file — no CLI escape
Orphaned daemon lock	boot file check	clear stale locks before any device call

recoverWedge() is bounded (no unbounded loops); the bridge never silently swaps planes.

Migration sequencing

• Phase 1 (small, shippable now): ensureSingleRunner() in SessionStart (auto-kill legacy + lock cleanup + RN_DEVICE_KILL_LEGACY=1 default-on) + fix bug: maestro_run can't pass --app-file, so clearState:true flows on iOS force a raw-CLI escape #201 (buildArgs auto-resolve --app-file for clearState). Prevents ~80% of the observed friction alone.
• Phase 2: introduce DeviceSessionArbiter; wire cdp_status → recoverWedge(), device_*/maestro_run → acquire(plane).
• Phase 3: formalize the 3-layer contracts; consolidate the dual Maestro surfaces (enhancement: rn-dev-agent + maestro-mcp interop — driver conflict, connection persistence, runFlow allowlist, flow-drift #186).

Out of scope (YAGNI)

Separate arbiter daemon (re-creates the orphaned-lock hazard); retiring RnFastRunner (debated, rejected — agent needs L2's tight loop); multi-session/CI fan-out arbitration.

---

Designed via a structured brainstorming session. Filed issue-first (no cache hand-edits).