Skip to content

Commit 1ba6d20

Browse files
committed
docs: add cross-platform snapshot traversal contract
Document the shared schema, traversal rules, surface semantics, and normalized role types that all snapshot backends (Swift, Python, Android) must conform to. This serves as the single source of truth when adding or modifying platform backends. https://claude.ai/code/session_01H9hrmueNF5pcBM8JeX81mT
1 parent 484614d commit 1ba6d20

1 file changed

Lines changed: 76 additions & 0 deletions

File tree

src/platforms/SNAPSHOT_CONTRACT.md

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
# Snapshot Traversal Contract
2+
3+
Each platform backend (Swift/XCTest, Android/UIAutomator, Python/AT-SPI2) produces
4+
a flat array of `RawSnapshotNode` objects. Despite different accessibility frameworks,
5+
all backends must conform to this shared contract so that the downstream pipeline
6+
(filtering, ref assignment, presentation) works identically.
7+
8+
## Output schema
9+
10+
Every node in the `nodes` array must include:
11+
12+
| Field | Type | Required | Notes |
13+
|---------------|-----------------------------------|----------|------------------------------------------|
14+
| `index` | `number` | yes | Sequential, 0-based, pre-order DFS |
15+
| `type` | `string` | yes | Normalized role (see table below) |
16+
| `role` | `string` | no | Raw platform role for debugging |
17+
| `label` | `string \| undefined` | no | Accessible name or description |
18+
| `value` | `string \| undefined` | no | Text content or numeric value |
19+
| `rect` | `{x, y, width, height} \| undef` | no | Screen-absolute bounding rect |
20+
| `enabled` | `boolean \| undefined` | no | |
21+
| `selected` | `boolean \| undefined` | no | |
22+
| `hittable` | `boolean \| undefined` | no | Can receive pointer/touch events |
23+
| `depth` | `number` | yes | Tree depth (root = 0) |
24+
| `parentIndex` | `number \| undefined` | no | Index of parent node; undefined for roots |
25+
26+
Platform-specific fields (optional, passed through):
27+
- `pid`, `appName`, `windowTitle` (Linux, macOS desktop)
28+
- `identifier`, `subrole` (iOS/macOS)
29+
- `resourceId`, `className` (Android)
30+
31+
## Traversal rules
32+
33+
| Parameter | Default | Description |
34+
|-----------------|---------|----------------------------------------------|
35+
| `maxNodes` | 1500 | Stop traversal after this many nodes |
36+
| `maxDepth` | 12 | Do not descend beyond this tree depth |
37+
| `maxApps` | 24 | Desktop only: max top-level apps to traverse |
38+
39+
- Traversal order is **pre-order depth-first**.
40+
- `index` values are assigned in traversal order (0, 1, 2, …).
41+
- `parentIndex` points to the containing node's `index`.
42+
- When `maxNodes` is reached, set `truncated: true` in the result.
43+
- Backends should skip defunct/inaccessible subtrees gracefully.
44+
45+
## Surface semantics
46+
47+
| Surface | Behavior |
48+
|------------------|-------------------------------------------------------|
49+
| `app` | Snapshot the target application's UI tree |
50+
| `frontmost-app` | Snapshot the focused/frontmost application (desktop) |
51+
| `desktop` | Snapshot all visible applications on the desktop |
52+
| `menubar` | macOS only: snapshot the system menu bar |
53+
54+
## Normalized role types
55+
56+
All backends must map platform-specific roles to these normalized strings.
57+
The canonical mapping is maintained in:
58+
- **iOS/macOS**: `ios-runner/…/SnapshotTraversal.swift``normalizedSnapshotType`
59+
- **Android**: `src/platforms/android/ui-hierarchy.ts``normalizeAndroidType`
60+
- **Linux**: `src/platforms/linux/role-map.ts``normalizeAtspiRole`
61+
62+
Common normalized types: `Button`, `StaticText`, `TextField`, `TextArea`,
63+
`CheckBox`, `RadioButton`, `Switch`, `ComboBox`, `Tab`, `TabList`,
64+
`Menu`, `MenuItem`, `MenuBar`, `List`, `ListItem`, `Table`, `Cell`, `Row`,
65+
`Tree`, `TreeItem`, `Group`, `Window`, `Dialog`, `Alert`, `ScrollArea`,
66+
`ScrollBar`, `Slider`, `ProgressBar`, `Image`, `Link`, `Separator`,
67+
`Toolbar`, `StatusBar`, `Tooltip`, `Application`, `Heading`.
68+
69+
Unmapped roles should be PascalCased (e.g., `"color chooser"``"ColorChooser"`).
70+
71+
## Adding a new platform
72+
73+
1. Implement a snapshot function returning `{ nodes: RawSnapshotNode[], truncated: boolean }`.
74+
2. Map platform roles to the normalized types listed above.
75+
3. Add unit tests verifying role normalization and node schema conformance.
76+
4. Wire into `snapshot-capture.ts` (`captureSnapshotData`) and `dispatch.ts`.

0 commit comments

Comments
 (0)