feat: replay --save-script for self-updating e2e tests; add it assertions; add DSL with selectors (#24)

* feat:  for self-healing e2e tests;  assertions

* rework ad scripts

Author: thymikee

AGENTS.md

@@ -14,6 +14,9 @@ Minimal operating guide for AI coding agents in this repo.
 - Use daemon session flow for interactions (`open` before interactions, `close` after).
 - Do not remove shared snapshot/session model behavior without full migration.
 - If Swift runner code changes, run `pnpm build:xcuitest`.
+- Do not add command logic to `daemon.ts` — it is a thin router. Use handler modules.
+- Use `inferFillText` and `uniqueStrings` from `src/daemon/action-utils.ts`. Do not duplicate.
+- Use `evaluateIsPredicate` from `src/daemon/is-predicates.ts` for assertion logic. Do not inline.
 
 ## Architecture In One Screen
 
@@ -24,14 +27,18 @@ Minimal operating guide for AI coding agents in this repo.
 2. Daemon client transport:
    - `src/daemon-client.ts`
 3. Daemon server bootstrap/router:
-   - `src/daemon.ts`
+   - `src/daemon.ts` — thin router only, delegates to handler modules
 4. Daemon command families:
    - session/apps/appstate/open/close/replay: `src/daemon/handlers/session.ts`
+   - click/fill/get/is: `src/daemon/handlers/interaction.ts`
    - snapshot/wait/alert/settings: `src/daemon/handlers/snapshot.ts`
    - semantic find actions: `src/daemon/handlers/find.ts`
    - record/trace: `src/daemon/handlers/record-trace.ts`
 5. Daemon shared domain:
    - session state + logs: `src/daemon/session-store.ts`
+   - selector DSL (parse, resolve, build): `src/daemon/selectors.ts`
+   - `is` predicate evaluation: `src/daemon/is-predicates.ts`
+   - shared action helpers (inferFillText, uniqueStrings): `src/daemon/action-utils.ts`
    - snapshot tree shaping + label resolution: `src/daemon/snapshot-processing.ts`
    - handler context helpers: `src/daemon/context.ts`, `src/daemon/device-ready.ts`, `src/daemon/app-state.ts`
 6. Platform dispatch/backends:
@@ -55,34 +62,42 @@ Do not read both iOS and Android paths unless issue is explicitly cross-platform
 
 - `session list`, `devices`, `apps`, `appstate`, `open`, `close`, `replay`:
   - `src/daemon/handlers/session.ts`
+- `click`, `fill`, `get`, `is`:
+  - `src/daemon/handlers/interaction.ts`
 - `snapshot`, `wait`, `alert`, `settings`:
   - `src/daemon/handlers/snapshot.ts`
 - `find ...`:
   - `src/daemon/handlers/find.ts`
 - `record start|stop`, `trace start|stop`:
   - `src/daemon/handlers/record-trace.ts`
-- `click`, `fill`, `get`, generic passthrough:
-  - `src/daemon.ts` (remaining fallback logic)
+- Generic passthrough (press, scroll, type, etc.):
+  - `src/daemon.ts` (fallback after all handlers return null)
 
 ## Capability Source Of Truth
 
 - Command/device support must come from `src/core/capabilities.ts`.
 - Do not scatter new support checks across handlers.
 
+## Selector System Rules
+
+All interaction commands (`click`, `fill`, `get`, `is`) and `wait` accept selectors in addition to `@ref`.
+The selector pipeline is: **parse → resolve → act → record selectorChain → heal on replay**.
+
+- Selector DSL lives in `src/daemon/selectors.ts`. Do not duplicate parsing/matching logic elsewhere.
+- `buildSelectorChainForNode` generates fallback chains stored in action results. Always call it after resolving a node for an interaction — it powers replay healing.
+- When adding a new interaction command that targets a UI element: support both `@ref` and selector input, record `selectorChain`, and update replay healing (`healReplayAction` + `collectReplaySelectorCandidates` in `session.ts`).
+- When adding a new selector key: update `SelectorKey` type, `ALL_KEYS`/`TEXT_KEYS`/`BOOLEAN_KEYS` sets, `matchesTerm`, and `isSelectorToken` — all in `selectors.ts`.
+- When adding a new `is` predicate: update `IsPredicate` type and `evaluateIsPredicate` in `is-predicates.ts`, not in the handler.
+- `daemon.ts` must stay a thin router. Do not add command logic there — use the appropriate handler module.
+
 ## Testing Strategy
 
 ### Test placement policy
 
 - Unit tests are colocated with source files under `src/**`.
 - Use `__tests__` folders colocated with the related source folder.
 - The `test/**` tree is integration-only (including smoke integration tests).
-
-### Unit tests (default for all refactors, colocated)
-
-- `src/core/__tests__/capabilities.test.ts`
-- `src/daemon/handlers/__tests__/find.test.ts`
-- `src/daemon/__tests__/snapshot-processing.test.ts`
-- `src/daemon/__tests__/session-store.test.ts`
+- Example: tests for `src/daemon/selectors.ts` go in `src/daemon/__tests__/selectors.test.ts`.
 
 Add/extend colocated unit tests in the same PR for touched module logic.
 

README.md

@@ -33,9 +33,11 @@ npx agent-device open SampleApp
 
 ## Quick Start
 
+Use refs for agent-driven exploration and normal automation flows.
+
 ```bash
 agent-device open Contacts --platform ios # creates session on iOS Simulator
-agent-device snapshot                      
+agent-device snapshot
 agent-device click @e5
 agent-device fill @e6 "John"
 agent-device fill @e7 "Doe"
@@ -75,7 +77,7 @@ Coordinates:
 ## Command Index
 - `open`, `close`, `home`, `back`, `app-switcher`
 - `snapshot`, `find`, `get`
-- `click`, `focus`, `type`, `fill`, `press`, `long-press`, `scroll`, `scrollintoview`
+- `click`, `focus`, `type`, `fill`, `press`, `long-press`, `scroll`, `scrollintoview`, `is`
 - `alert`, `wait`, `screenshot`
 - `trace start`, `trace stop`
 - `settings wifi|airplane|location on|off`
@@ -117,14 +119,45 @@ Sessions:
 - If a session is already open, `open <app>` switches the active app and updates the session app bundle.
 - `close` stops the session and releases device resources. Pass an app to close it explicitly, or omit to just close the session.
 - Use `--session <name>` to manage multiple sessions.
-- Session logs are written to `~/.agent-device/sessions/<session>-<timestamp>.ad`.
-- With `--record-json`, JSON logs are written to `~/.agent-device/sessions/<session>-<timestamp>.json` by default.
+- Session scripts are written to `~/.agent-device/sessions/<session>-<timestamp>.ad` when recording is enabled with `--save-script`.
+- Deterministic replay is `.ad`-based; use `replay --update` (`-u`) to update selector drift and rewrite the replay file in place.
 
 Find (semantic):
 - `find <text> <action> [value]` finds by any text (label/value/identifier) using a scoped snapshot.
 - `find text|label|value|role|id <value> <action> [value]` for specific locators.
 - Actions: `click` (default), `fill`, `type`, `focus`, `get text`, `get attrs`, `wait [timeout]`, `exists`.
 
+Assertions:
+- `is` predicates: `visible`, `hidden`, `exists`, `editable`, `selected`, `text`.
+- `is text` uses exact equality.
+
+Replay update:
+- `replay <path>` runs deterministic replay from `.ad` scripts.
+- `replay -u <path>` attempts selector updates on failures and atomically rewrites the same file.
+- Refs are the default/core mechanism for interactive agent flows.
+- Update targets: `click`, `fill`, `get`, `is`, `wait`.
+- Selector matching is a replay-update internal: replay parses `.ad` lines into actions, tries them, snapshots on failure, resolves a better selector, then rewrites that failing line.
+
+Update examples:
+
+```sh
+# Before (stale selector)
+click "id=\"old_continue\" || label=\"Continue\""
+
+# After replay -u (rewritten in place)
+click "id=\"auth_continue\" || label=\"Continue\""
+```
+
+```sh
+# Before (ref-based action from discovery)
+snapshot -i -c -s "Continue"
+click @e13 "Continue"
+
+# After replay -u (upgraded to selector-based action)
+snapshot -i -c -s "Continue"
+click "id=\"auth_continue\" || label=\"Continue\""
+```
+
 Android fill reliability:
 - `fill` clears the current value, then enters text.
 - `type` enters text into the focused field without clearing.

skills/agent-device/SKILL.md

@@ -5,6 +5,8 @@ description: Automates mobile and simulator interactions for iOS and Android dev
 
 # Mobile Automation with agent-device
 
+For agent-driven exploration: use refs. For deterministic replay scripts: use selectors.
+
 ## Quick start
 
 ```bash
@@ -26,9 +28,9 @@ npx -y agent-device
 ## Core workflow
 
 1. Open app or just boot device: `open [app]`
-2. Snapshot: `snapshot` to get full XCTest accessibility tree snapshot
+2. Snapshot: `snapshot` to get refs from accessibility tree
 3. Interact using refs (`click @ref`, `fill @ref "text"`)
-4. Re-snapshot after navigation or UI changes
+4. Re-snapshot after navigation/UI changes
 5. Close session when done
 
 ## Commands
@@ -109,6 +111,8 @@ agent-device home
 agent-device app-switcher
 agent-device wait 1000
 agent-device wait text "Settings"
+agent-device is visible 'id="settings_anchor"'  # selector assertions for deterministic checks
+agent-device is text 'id="header_title"' "Settings"
 agent-device alert get
 ```
 
@@ -120,6 +124,16 @@ agent-device get attrs @e1
 agent-device screenshot out.png
 ```
 
+### Deterministic replay and updating
+
+```bash
+agent-device open App --save-script   # Save session script (.ad) on close
+agent-device replay ./session.ad      # Run deterministic replay from .ad script
+agent-device replay -u ./session.ad   # Update selector drift and rewrite .ad script in place
+```
+
+`replay` reads `.ad` recordings.
+
 ### Trace logs (AX/XCTest)
 
 ```bash
@@ -142,7 +156,8 @@ agent-device apps --platform android --user-installed
 ## Best practices
 
 - Pinch (`pinch <scale> [x y]`) is supported on iOS simulators and Android; scale > 1 zooms in, < 1 zooms out. On Android, pinch uses multi-touch `sendevent` injection.
-- Always snapshot right before interactions; refs invalidate on UI changes.
+- Snapshot refs are the core mechanism for interactive agent flows.
+- Use selectors for deterministic replay artifacts and assertions (e.g. in e2e test workflows).
 - Prefer `snapshot -i` to reduce output size.
 - On iOS, `xctest` is the default and does not require Accessibility permission.
 - If XCTest returns 0 nodes (foreground app changed), agent-device falls back to AX when available.
@@ -153,6 +168,7 @@ agent-device apps --platform android --user-installed
 - Use `fill` when you want clear-then-type semantics.
 - Use `type` when you want to append/enter text without clearing.
 - On Android, prefer `fill` for important fields; it verifies entered text and retries once when IME reorders characters.
+- If using deterministic replay scripts, use `replay -u` during maintenance runs to update selector drift in replay scripts. Use plain `replay` in CI.
 
 ## References
 

skills/agent-device/references/session-management.md

@@ -14,9 +14,18 @@ Sessions isolate device context. A device can only be held by one session at a t
 - Name sessions semantically.
 - Close sessions when done.
 - Use separate sessions for parallel work.
+- For deterministic replay scripts, prefer selector-based actions and assertions.
+- Use `replay -u` to update selector drift during maintenance.
 
 ## Listing sessions
 
 ```bash
 agent-device session list
 ```
+
+## Replay within sessions
+
+```bash
+agent-device replay ./session.ad --session auth
+agent-device replay -u ./session.ad --session auth
+```

skills/agent-device/references/snapshot-refs.md

@@ -1,8 +1,8 @@
-# Snapshot + Refs Workflow (Mobile)
+# Snapshot Refs and Selectors (Mobile)
 
 ## Purpose
 
-Refs let agents interact without repeating full UI trees. Snapshot -> refs -> click/fill.
+Refs are useful for discovery/debugging. For deterministic scripts, use selectors.
 
 ## Snapshot
 
@@ -21,17 +21,25 @@ App: com.apple.Preferences
   @e3 [button] "Privacy & Security"
 ```
 
-## Using refs
+## Using refs (discovery/debug)
 
 ```bash
 agent-device click @e2
 agent-device fill @e5 "test"
 ```
 
+## Using selectors (deterministic)
+
+```bash
+agent-device click 'id="camera_row" || label="Camera" role=button'
+agent-device fill 'id="search_input" editable=true' "test"
+agent-device is visible 'id="camera_settings_anchor"'
+```
+
 ## Ref lifecycle
 
-Refs become invalid when UI changes (navigation, modal, dynamic list updates).
-Always re-snapshot after any transition.
+Refs can become invalid when UI changes (navigation, modal, dynamic list updates).
+Re-snapshot after transitions if you keep using refs.
 
 ## Scope snapshots
 
@@ -47,3 +55,8 @@ agent-device snapshot -i -s @e3
 - Ref not found: re-snapshot.
 - AX returns Simulator window: restart Simulator and re-run.
 - AX empty: verify Accessibility permission or use `--backend xctest` (XCTest is more complete).
+
+## Replay note
+
+- Prefer selector-based actions in recorded `.ad` replays.
+- Use `agent-device replay -u <path>` to update selector drift and rewrite replay scripts in place.

skills/agent-device/references/video-recording.md

@@ -12,7 +12,7 @@ agent-device record start ./recordings/ios.mov
 
 # Perform actions
 agent-device open App
-agent-device snapshot
+agent-device snapshot -i
 agent-device click @e3
 agent-device close
 
@@ -30,7 +30,7 @@ agent-device record start ./recordings/android.mp4
 
 # Perform actions
 agent-device open App
-agent-device snapshot
+agent-device snapshot -i
 agent-device click @e3
 agent-device close
 

src/cli.ts

@@ -93,6 +93,12 @@ export async function runCli(argv: string[]): Promise<void> {
           return;
         }
       }
+      if (command === 'is') {
+        const predicate = (response.data as any)?.predicate ?? 'assertion';
+        process.stdout.write(`Passed: is ${predicate}\n`);
+        if (logTailStopper) logTailStopper();
+        return;
+      }
       if (command === 'click') {
         const ref = (response.data as any)?.ref ?? '';
         const x = (response.data as any)?.x;

src/core/capabilities.ts

@@ -25,6 +25,7 @@ const COMMAND_CAPABILITY_MATRIX: Record<string, CommandCapability> = {
   find: { ios: { simulator: true }, android: { emulator: true, device: true, unknown: true } },
   focus: { ios: { simulator: true }, android: { emulator: true, device: true, unknown: true } },
   get: { ios: { simulator: true }, android: { emulator: true, device: true, unknown: true } },
+  is: { ios: { simulator: true }, android: { emulator: true, device: true, unknown: true } },
   home: { ios: { simulator: true }, android: { emulator: true, device: true, unknown: true } },
   'long-press': { ios: { simulator: true }, android: { emulator: true, device: true, unknown: true } },
   open: { ios: { simulator: true }, android: { emulator: true, device: true, unknown: true } },

src/core/dispatch.ts

@@ -33,10 +33,11 @@ export type CommandFlags = {
   snapshotScope?: string;
   snapshotRaw?: boolean;
   snapshotBackend?: 'ax' | 'xctest';
+  saveScript?: boolean;
   noRecord?: boolean;
-  recordJson?: boolean;
   appsFilter?: 'launchable' | 'user-installed' | 'all';
   appsMetadata?: boolean;
+  replayUpdate?: boolean;
 };
 
 export async function resolveTargetDevice(flags: CommandFlags): Promise<DeviceInfo> {