|
| 1 | +# Snapshot Traversal Contract |
| 2 | + |
| 3 | +Each platform backend (Swift/XCTest, Android/UIAutomator, Python/AT-SPI2) produces |
| 4 | +a flat array of `RawSnapshotNode` objects. Despite different accessibility frameworks, |
| 5 | +all backends must conform to this shared contract so that the downstream pipeline |
| 6 | +(filtering, ref assignment, presentation) works identically. |
| 7 | + |
| 8 | +## Output schema |
| 9 | + |
| 10 | +Every node in the `nodes` array must include: |
| 11 | + |
| 12 | +| Field | Type | Required | Notes | |
| 13 | +|---------------|-----------------------------------|----------|------------------------------------------| |
| 14 | +| `index` | `number` | yes | Sequential, 0-based, pre-order DFS | |
| 15 | +| `type` | `string` | yes | Normalized role (see table below) | |
| 16 | +| `role` | `string` | no | Raw platform role for debugging | |
| 17 | +| `label` | `string \| undefined` | no | Accessible name or description | |
| 18 | +| `value` | `string \| undefined` | no | Text content or numeric value | |
| 19 | +| `rect` | `{x, y, width, height} \| undef` | no | Screen-absolute bounding rect | |
| 20 | +| `enabled` | `boolean \| undefined` | no | | |
| 21 | +| `selected` | `boolean \| undefined` | no | | |
| 22 | +| `hittable` | `boolean \| undefined` | no | Can receive pointer/touch events | |
| 23 | +| `depth` | `number` | yes | Tree depth (root = 0) | |
| 24 | +| `parentIndex` | `number \| undefined` | no | Index of parent node; undefined for roots | |
| 25 | + |
| 26 | +Platform-specific fields (optional, passed through): |
| 27 | +- `pid`, `appName`, `windowTitle` (Linux, macOS desktop) |
| 28 | +- `identifier`, `subrole` (iOS/macOS) |
| 29 | +- `resourceId`, `className` (Android) |
| 30 | + |
| 31 | +## Traversal rules |
| 32 | + |
| 33 | +| Parameter | Default | Description | |
| 34 | +|-----------------|---------|----------------------------------------------| |
| 35 | +| `maxNodes` | 1500 | Stop traversal after this many nodes | |
| 36 | +| `maxDepth` | 12 | Do not descend beyond this tree depth | |
| 37 | +| `maxApps` | 24 | Desktop only: max top-level apps to traverse | |
| 38 | + |
| 39 | +- Traversal order is **pre-order depth-first**. |
| 40 | +- `index` values are assigned in traversal order (0, 1, 2, …). |
| 41 | +- `parentIndex` points to the containing node's `index`. |
| 42 | +- When `maxNodes` is reached, set `truncated: true` in the result. |
| 43 | +- Backends should skip defunct/inaccessible subtrees gracefully. |
| 44 | + |
| 45 | +## Surface semantics |
| 46 | + |
| 47 | +| Surface | Behavior | |
| 48 | +|------------------|-------------------------------------------------------| |
| 49 | +| `app` | Snapshot the target application's UI tree | |
| 50 | +| `frontmost-app` | Snapshot the focused/frontmost application (desktop) | |
| 51 | +| `desktop` | Snapshot all visible applications on the desktop | |
| 52 | +| `menubar` | macOS only: snapshot the system menu bar | |
| 53 | + |
| 54 | +## Normalized role types |
| 55 | + |
| 56 | +All backends must map platform-specific roles to these normalized strings. |
| 57 | +The canonical mapping is maintained in: |
| 58 | +- **iOS/macOS**: `ios-runner/…/SnapshotTraversal.swift` → `normalizedSnapshotType` |
| 59 | +- **Android**: `src/platforms/android/ui-hierarchy.ts` → `normalizeAndroidType` |
| 60 | +- **Linux**: `src/platforms/linux/role-map.ts` → `normalizeAtspiRole` |
| 61 | + |
| 62 | +Common normalized types: `Button`, `StaticText`, `TextField`, `TextArea`, |
| 63 | +`CheckBox`, `RadioButton`, `Switch`, `ComboBox`, `Tab`, `TabList`, |
| 64 | +`Menu`, `MenuItem`, `MenuBar`, `List`, `ListItem`, `Table`, `Cell`, `Row`, |
| 65 | +`Tree`, `TreeItem`, `Group`, `Window`, `Dialog`, `Alert`, `ScrollArea`, |
| 66 | +`ScrollBar`, `Slider`, `ProgressBar`, `Image`, `Link`, `Separator`, |
| 67 | +`Toolbar`, `StatusBar`, `Tooltip`, `Application`, `Heading`. |
| 68 | + |
| 69 | +Unmapped roles should be PascalCased (e.g., `"color chooser"` → `"ColorChooser"`). |
| 70 | + |
| 71 | +## Adding a new platform |
| 72 | + |
| 73 | +1. Implement a snapshot function returning `{ nodes: RawSnapshotNode[], truncated: boolean }`. |
| 74 | +2. Map platform roles to the normalized types listed above. |
| 75 | +3. Add unit tests verifying role normalization and node schema conformance. |
| 76 | +4. Wire into `snapshot-capture.ts` (`captureSnapshotData`) and `dispatch.ts`. |
0 commit comments