Skip to content

Commit 13b3d4f

Browse files
authored
fix: demote non-unique ids from writer identity/selector chain (#1269) (#1272)
* fix(replay): demote non-unique ids from writer identity/selector chain (#1269) Android list-row GET replays bind the wrong row because the recorder uses the non-unique framework resource id `android:id/title` (matchCount 11 on Settings root) as primary identity; positional drift then makes the identity verifier correctly refuse with `identity-mismatch`. Demote an id from identity whenever it matches more than one node in the record-time tree (capture-time uniqueness, not an `android:id/*` namespace check — a reused RN FlatList testID hits the same class on iOS). Applied in both places a recorded id feeds identity: - `computeTargetEvidence` (session-target-evidence.ts): the `target-v1` identity tuple falls back to role+label when the id's own capture-time match count exceeds one, reusing the existing `filterIdentitySet` domain machinery (an empty ancestry degrades it to a plain id scan). - `buildSelectorChainForNode` (selectors/build.ts): the recorded selector chain omits a non-unique id rather than leading with it. Every writer call site (get/press/fill recording, plus the divergence-suggestion path) now passes the record-time tree so the check has something to count against; omitting it preserves prior behavior for isolated-node callers (tests). Resolver-side `resolveSelectorChain` and live press/fill resolution are untouched per ADR 0012 (disclosed-not-changed disambiguation) — this is writer/replay-scoped only. Amends ADR 0012 decision 3: an id may serve as identity (and lead the selector chain) only when it uniquely denotes the target in the record-time tree. Adds fixtures: an Android duplicated-`android:id/title` list (the measured repro) and an iOS/RN duplicated-testID FlatList shape, both demoted and still verifying via the now-selective label; a regression case confirming an already-unique id is unaffected. Out of scope: the Android list-*press* class (matchCount 12, label-less `role="linearlayout"` container with no id at all to demote) needs a separate design decision — deriving identity from the labeled descendant. Tracked as a follow-up, not attempted here. * fix(replay): unify the id-uniqueness predicate across both writer sites (#1269 review) Address the maintainer review on #1272: 1. ONE shared uniqueness predicate. The two demotion sites were counting id matches with DIFFERENT semantics — `demoteNonUniqueId` via `filterIdentitySet` (NFC + 256-byte cap, and a broken-parent-walk exclusion), `selectableId` via a raw `normalizeSelectorText` scan (trim, no NFC/cap, no exclusion) — so the identity tuple and the selector chain could disagree and half-demote (id gone from one, kept in the other). Extract `idMatchCountInTree(nodes, id)` in target-identity-node.ts, counting over the canonical `readNodeLocalIdentity` id the replay verifier keys on, with no ancestry/parent-walk exclusion. Both `demoteNonUniqueId` and `selectableId` now call it. Corrects the inaccurate "vacuously-true / plain id scan" comment. Cross-invariant test (build.test.ts): for the same node+tree, evidence.id === undefined iff the built chain has no id= clause — across demoted, unique, and a non-NFC (decomposed vs precomposed) edge case. Verified it fails under the old raw-scan and passes under the unified predicate. 2. End-to-end reorder proof (session-replay-target-classification.test.ts): record against a tree whose rows share android:id/title, then classify against a DIFFERENT replay tree where the shared-id rows reorder — the demoted role+label identity rebinds the correct row (verified, matchCount 1) while `id="android:id/title"` resolves ambiguously (null). This pins the FDR 1.0 -> 0 mechanism, not just record-time demotion. 3. Removed the conflated "20/20 clean" live-number comment from the unit test; it now states the mechanism (role+label selectivity) instead. Behavior for the already-clean unique-id path is unchanged: for ordinary ascii ids the canonical count equals the old raw count. The only outcomes that change are the edge cases the old split mishandled (non-NFC, broken parent walk) — where demotion is the correct result. The kept clause still emits the chain's own normalizeSelectorText id string, so unique ids lead the chain exactly as before. * fix(replay): thread record-time tree through the extracted suggestion helper Rebase-conflict resolution against origin/main. #1217 (typed direct Maestro engine) extracted `buildReplayDivergenceSuggestionForNode` out of `resolveSuggestionCandidate` and added a second caller in `session-replay-maestro-failure.ts`. My #1269 change had added `nodes` to the `buildSelectorChainForNode` call that #1217 moved into the extracted helper, so after rebase the helper referenced an out-of-scope `nodes`. Thread the record-time tree as a required `nodes` param on the helper and pass it from BOTH callers (each already has it in scope). This keeps the non-unique-id demotion applied wherever a divergence/repair suggestion chain is built — now including the typed-Maestro suggestion path — with no behavior change for the already-unique-id case.
1 parent e58cbcd commit 13b3d4f

11 files changed

Lines changed: 610 additions & 6 deletions

docs/adr/0012-interactive-replay.md

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,11 @@ Accepted (2026-07-10); partially implemented (last updated 2026-07-13). See [Mig
1919
filters, not scopings; and `screen.refs` is ranked within the byte cap (foreign-window dismiss targets
2020
ahead of app content; mass-covered app nodes surfaced rather than emptied) instead of sliced in document
2121
order, so a captured overlay is never buried past the cap (#1264).
22+
- Decision 3 amendment, id demotion under non-unique capture-time match — a recorded id no longer serves
23+
as identity, or leads the selector chain, unless it is unique in the record-time tree; a non-unique id
24+
(a shared Android framework resource id such as `android:id/title`, or a reused RN `FlatList` `testID`)
25+
falls back to role+label in both `computeTargetEvidence`'s `target-v1` tuple and
26+
`buildSelectorChainForNode`'s chain (#1269).
2227

2328
**Accepted but NOT yet implemented** (this amendment; tracked by #1235 — repair-transaction lifecycle):
2429
the R7 repair-transaction keep-alive and its distinct `resume.repairSessionHeld` signal, the ARMED →
@@ -316,6 +321,14 @@ or, when the recording carries no `id`, when their normalized roles are equal an
316321
are equal (label absent on both sides counts as equal; label present on exactly one side is a mismatch).
317322
A recorded `id` never matches a node without that id.
318323

324+
> **Amendment (#1269).** An id may serve as identity (and lead the selector chain) only when it uniquely
325+
> denotes the target in the record-time tree: the writer computes the id's own capture-time match count
326+
> (independent of ancestry) and, whenever more than one node in the record-time tree carries the recorded
327+
> id, demotes it — falling back to role+label exactly as an unrecorded id already does. A shared
328+
> Android framework resource id (`android:id/title`, present on every titled list row) is the measured
329+
> case, but the rule is capture-time uniqueness, not an `android:id/*` namespace heuristic: a reused RN
330+
> `FlatList` `testID` hits the same demotion on iOS.
331+
319332
**Ancestry.** The chain is the nearest **K = 8** ancestors of the target, ordered **leaf→root** (nearest
320333
ancestor first), each entry `{ role, label? }` under the same normalization (`role` may be the empty
321334
string when the node has no type; `label` is omitted when empty). Truncation drops entries from the

src/commands/interaction/runtime/resolution.ts

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -403,6 +403,7 @@ function describeResolvedInteractionNode(
403403
node,
404404
selectorChain: buildSelectorChainForNode(node, runtime.backend.platform, {
405405
action: action === 'fill' ? 'fill' : 'click',
406+
nodes,
406407
}),
407408
refLabel: resolveRefLabel(node, nodes),
408409
...describeNonHittableTarget(node, action),

src/commands/interaction/runtime/selector-read.ts

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -210,6 +210,7 @@ export const getCommand: RuntimeCommand<GetCommandOptions, GetCommandResult> = a
210210
);
211211
const selectorChain = buildSelectorChainForNode(resolved.node, runtime.backend.platform, {
212212
action: 'get',
213+
nodes: capture.snapshot.nodes,
213214
});
214215
const target = { kind: 'ref' as const, ref: `@${resolved.ref}` };
215216
const preActionNodes = capture.snapshot.nodes;
@@ -233,6 +234,7 @@ export const getCommand: RuntimeCommand<GetCommandOptions, GetCommandResult> = a
233234

234235
const selectorChain = buildSelectorChainForNode(resolved.node, runtime.backend.platform, {
235236
action: 'get',
237+
nodes: resolved.capture.snapshot.nodes,
236238
});
237239

238240
if (options.property === 'attrs') {

src/daemon/__tests__/session-target-evidence.test.ts

Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -466,3 +466,126 @@ test('computeTargetEvidence: a parent cycle records unverifiable, never verified
466466
assert.ok(evidence);
467467
assert.equal(evidence.verification, 'unverifiable');
468468
});
469+
470+
// ---------------------------------------------------------------------------
471+
// #1269: ADR 0012 decision 3 amendment — an id is identity only when it
472+
// uniquely denotes the target in the record-time tree. A shared framework
473+
// resource id (Android's `android:id/title`, present on every titled list
474+
// row) is not selective: the recorded ancestry is label-less generic
475+
// containers (linearlayout/recyclerview/framelayout) contributing no
476+
// disambiguation, so an id-led identity set spans every row and replay binds
477+
// the wrong one under positional drift. The fix demotes the id back to
478+
// role+label whenever it matches more than one node at record time — the
479+
// same fallback an unrecorded id already uses.
480+
// ---------------------------------------------------------------------------
481+
482+
function androidSettingsListFixture(): SnapshotNode[] {
483+
return toSnapshotNodes([
484+
{ index: 0, type: 'FrameLayout', depth: 0 },
485+
{ index: 1, type: 'RecyclerView', depth: 1, parentIndex: 0 },
486+
// Each row is a label-less LinearLayout wrapper (real Android list
487+
// shape) whose title TextView shares one framework resource id.
488+
{ index: 2, type: 'LinearLayout', depth: 2, parentIndex: 1 },
489+
{
490+
index: 3,
491+
type: 'TextView',
492+
identifier: 'android:id/title',
493+
label: 'Network & internet',
494+
rect: { x: 0, y: 100, width: 300, height: 48 },
495+
depth: 3,
496+
parentIndex: 2,
497+
},
498+
{ index: 4, type: 'LinearLayout', depth: 2, parentIndex: 1 },
499+
{
500+
index: 5,
501+
type: 'TextView',
502+
identifier: 'android:id/title',
503+
label: 'Connected devices',
504+
rect: { x: 0, y: 148, width: 300, height: 48 },
505+
depth: 3,
506+
parentIndex: 4,
507+
},
508+
{ index: 6, type: 'LinearLayout', depth: 2, parentIndex: 1 },
509+
{
510+
index: 7,
511+
type: 'TextView',
512+
identifier: 'android:id/title',
513+
label: 'Apps',
514+
rect: { x: 0, y: 196, width: 300, height: 48 },
515+
depth: 3,
516+
parentIndex: 6,
517+
},
518+
]);
519+
}
520+
521+
test('computeTargetEvidence: a shared Android framework id (matchCount 3 > 1) is demoted to role+label, and still verifies via the now-selective label', () => {
522+
const nodes = androidSettingsListFixture();
523+
const winner = findByLabel(nodes, 'Network & internet');
524+
const evidence = computeTargetEvidence({ node: winner, preActionNodes: nodes });
525+
assert.ok(evidence);
526+
assert.equal(
527+
evidence.id,
528+
undefined,
529+
'the non-unique android:id/title must not be recorded as identity',
530+
);
531+
assert.equal(evidence.role, 'textview');
532+
assert.equal(evidence.label, 'Network & internet');
533+
assert.deepEqual(evidence.ancestry[0], { role: 'linearlayout' });
534+
// With the id demoted, role+label uniquely isolates this row among the
535+
// three sharing the id, so the record-time self-check still verifies via
536+
// the now-selective label rather than the non-selective id.
537+
assert.equal(evidence.verification, 'verified');
538+
});
539+
540+
test('computeTargetEvidence: an RN FlatList reusing one testID across rows (iOS/RN, matchCount > 1) is demoted the same way', () => {
541+
// The class is not Android-specific: an RN `FlatList` `renderItem` that
542+
// assigns the SAME testID to every row reproduces one shared accessibility
543+
// identifier across siblings on iOS too. Capture-time uniqueness — not an
544+
// `android:id/*` namespace check — is the rule.
545+
const nodes = toSnapshotNodes([
546+
{ index: 0, type: 'Application', depth: 0 },
547+
{ index: 1, type: 'ScrollView', identifier: 'contacts-list', depth: 1, parentIndex: 0 },
548+
{
549+
index: 2,
550+
type: 'Cell',
551+
identifier: 'contact-row',
552+
label: 'Ada Lovelace',
553+
rect: { x: 0, y: 0, width: 320, height: 60 },
554+
depth: 2,
555+
parentIndex: 1,
556+
},
557+
{
558+
index: 3,
559+
type: 'Cell',
560+
identifier: 'contact-row',
561+
label: 'Grace Hopper',
562+
rect: { x: 0, y: 60, width: 320, height: 60 },
563+
depth: 2,
564+
parentIndex: 1,
565+
},
566+
{
567+
index: 4,
568+
type: 'Cell',
569+
identifier: 'contact-row',
570+
label: 'Katherine Johnson',
571+
rect: { x: 0, y: 120, width: 320, height: 60 },
572+
depth: 2,
573+
parentIndex: 1,
574+
},
575+
]);
576+
const winner = findByLabel(nodes, 'Grace Hopper');
577+
const evidence = computeTargetEvidence({ node: winner, preActionNodes: nodes });
578+
assert.ok(evidence);
579+
assert.equal(evidence.id, undefined, 'the shared testID must not be recorded as identity');
580+
assert.equal(evidence.role, 'cell');
581+
assert.equal(evidence.label, 'Grace Hopper');
582+
assert.equal(evidence.verification, 'verified');
583+
});
584+
585+
test('computeTargetEvidence: a unique id is still recorded as identity (already-clean case unchanged)', () => {
586+
const nodes = toolbarFixture();
587+
const winner = findByLabel(nodes, 'Save');
588+
const evidence = computeTargetEvidence({ node: winner, preActionNodes: nodes });
589+
assert.ok(evidence);
590+
assert.equal(evidence.id, 'save');
591+
});

src/daemon/handlers/__tests__/session-replay-target-classification.test.ts

Lines changed: 98 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,10 @@
11
import { test } from 'vitest';
22
import assert from 'node:assert/strict';
3-
import type { SnapshotNode } from '../../../kernel/snapshot.ts';
3+
import type { RawSnapshotNode, SnapshotNode } from '../../../kernel/snapshot.ts';
44
import type { TargetAnnotationV1 } from '../../../replay/target-identity.ts';
5+
import { computeTargetEvidence } from '../../session-target-evidence.ts';
6+
import { buildSelectorChainForNode } from '../../../selectors/build.ts';
7+
import { parseSelectorChain, resolveSelectorChain } from '../../../selectors/index.ts';
58
import { classifyReplayTarget } from '../session-replay-target-classification.ts';
69
import {
710
bottomTabsRealCaptureFixture,
@@ -464,3 +467,97 @@ test('classifyReplayTarget: document-order determinism for equal rect centers',
464467
});
465468

466469
// ---------------------------------------------------------------------------
470+
// #1269 end-to-end mechanism (Ask 2): the load-bearing proof that id-demotion
471+
// makes replay clean — not just that the id is dropped at record time. Record
472+
// against a tree whose rows share `android:id/title`, then replay against a
473+
// DIFFERENT tree where those shared-id rows REORDER (positions drift). The
474+
// demoted role+label identity binds the correct row; the id path would not.
475+
// ---------------------------------------------------------------------------
476+
477+
function androidSharedIdListFixture(
478+
rows: { id: string; label: string; y: number }[],
479+
): SnapshotNode[] {
480+
const raw: RawSnapshotNode[] = [
481+
{ index: 0, type: 'FrameLayout', depth: 0 },
482+
{ index: 1, type: 'RecyclerView', depth: 1, parentIndex: 0 },
483+
];
484+
rows.forEach((row, position) => {
485+
const wrapperIndex = 2 + position * 2;
486+
const titleIndex = wrapperIndex + 1;
487+
raw.push({ index: wrapperIndex, type: 'LinearLayout', depth: 2, parentIndex: 1 });
488+
raw.push({
489+
index: titleIndex,
490+
type: 'TextView',
491+
identifier: row.id,
492+
label: row.label,
493+
rect: { x: 0, y: row.y, width: 300, height: 48 },
494+
depth: 3,
495+
parentIndex: wrapperIndex,
496+
});
497+
});
498+
return toSnapshotNodes(raw);
499+
}
500+
501+
test('#1269 e2e: a demoted shared-id row rebinds by role+label after the shared-id rows reorder on replay', () => {
502+
const ANDROID = 'android' as const;
503+
// Record-time Settings root: three rows all carrying the framework id.
504+
const recordNodes = androidSharedIdListFixture([
505+
{ id: 'android:id/title', label: 'Network & internet', y: 100 },
506+
{ id: 'android:id/title', label: 'Connected devices', y: 148 },
507+
{ id: 'android:id/title', label: 'Apps', y: 196 },
508+
]);
509+
const target = recordNodes.find((node) => node.label === 'Connected devices')!;
510+
511+
// The writer demotes the non-unique id in BOTH the tuple and the chain.
512+
const recorded = computeTargetEvidence({ node: target, preActionNodes: recordNodes })!;
513+
assert.equal(recorded.id, undefined);
514+
assert.equal(recorded.verification, 'verified');
515+
const chain = buildSelectorChainForNode(target, ANDROID, { action: 'get', nodes: recordNodes });
516+
assert.ok(!chain.some((entry) => entry.startsWith('id=')));
517+
const token = chain.join(' || '); // the recorded selector positional the replay loop re-resolves
518+
519+
// Replay-time tree: a new conditional row appears at the top and the
520+
// shared-id rows are in a DIFFERENT document/viewport order. The recorded
521+
// row is now third, at a drifted position.
522+
const replayNodes = androidSharedIdListFixture([
523+
{ id: 'android:id/title', label: 'Wi-Fi', y: 100 },
524+
{ id: 'android:id/title', label: 'Apps', y: 148 },
525+
{ id: 'android:id/title', label: 'Connected devices', y: 196 },
526+
{ id: 'android:id/title', label: 'Network & internet', y: 244 },
527+
]);
528+
const expected = replayNodes.find((node) => node.label === 'Connected devices')!;
529+
530+
const result = classifyReplayTarget({
531+
recorded,
532+
token,
533+
nodes: replayNodes,
534+
platform: ANDROID,
535+
refLabel: undefined,
536+
requireRect: true,
537+
allowDisambiguation: true,
538+
});
539+
assertVerified(result, { winnerRef: expected.ref, matchCount: 1 });
540+
541+
// Contrast — the reorder is genuinely hostile to the id path: the shared id
542+
// matches all four rows (no unique bind → resolution refuses), while the
543+
// demoted role+label resolves the correct row uniquely. This is the
544+
// FDR 1.0 → 0 difference the demotion buys.
545+
const idResolved = resolveSelectorChain(
546+
replayNodes,
547+
parseSelectorChain('id="android:id/title"'),
548+
{
549+
platform: ANDROID,
550+
requireRect: true,
551+
requireUnique: true,
552+
},
553+
);
554+
assert.equal(idResolved, null); // non-unique: refuses to bind
555+
const labelResolved = resolveSelectorChain(
556+
replayNodes,
557+
parseSelectorChain('role="textview" label="Connected devices"'),
558+
{ platform: ANDROID, requireRect: true, requireUnique: true },
559+
);
560+
assert.equal(labelResolved?.node.ref, expected.ref);
561+
});
562+
563+
// ---------------------------------------------------------------------------

src/daemon/handlers/session-replay-divergence.ts

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -536,6 +536,7 @@ function resolveSuggestionCandidate(params: {
536536
return {
537537
suggestion: buildReplayDivergenceSuggestionForNode({
538538
node: resolved.node,
539+
nodes,
539540
session,
540541
action,
541542
basis,
@@ -548,15 +549,18 @@ function resolveSuggestionCandidate(params: {
548549

549550
export function buildReplayDivergenceSuggestionForNode(params: {
550551
node: SnapshotNode;
552+
/** The record-time tree the node came from, for #1269 non-unique-id demotion in the chain. */
553+
nodes: readonly SnapshotNode[];
551554
session: SessionState;
552555
action: ReplayReportAction;
553556
basis: ReplayDivergenceSuggestionBasis;
554557
sanitize: DivergenceFieldSanitizer;
555558
}): ReplayDivergenceSuggestion {
556-
const { node, session, action, basis, sanitize } = params;
559+
const { node, nodes, session, action, basis, sanitize } = params;
557560
const selectorChain = buildSelectorChainForNode(node, session.device.platform, {
558561
action:
559562
action.command === 'fill' ? 'fill' : isTouchTargetCommand(action.command) ? 'click' : 'get',
563+
nodes,
560564
});
561565
const role = formatRole(node.type ?? 'Element');
562566
const label = displayLabel(node, role);

src/daemon/handlers/session-replay-maestro-failure.ts

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -210,6 +210,7 @@ function collectTypedMaestroSuggestions(params: {
210210
).map(({ node, basis }) =>
211211
buildReplayDivergenceSuggestionForNode({
212212
node,
213+
nodes: params.nodes,
213214
session: params.session,
214215
action: params.action,
215216
basis,

src/daemon/session-target-evidence.ts

Lines changed: 29 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,11 @@
1717
import type { SnapshotNode } from '../kernel/snapshot.ts';
1818
import { resolveRectCenter } from '../utils/rect-center.ts';
1919
import { findNearestScrollableContainer } from './snapshot-presentation/tree.ts';
20-
import { readNodeLocalIdentity, siblingOrdinal } from '../replay/target-identity-node.ts';
20+
import {
21+
idMatchCountInTree,
22+
readNodeLocalIdentity,
23+
siblingOrdinal,
24+
} from '../replay/target-identity-node.ts';
2125
import {
2226
classifyTargetBindingMatch,
2327
matchesAncestryPrefix,
@@ -45,7 +49,7 @@ export function computeTargetEvidence(
4549
const { node, preActionNodes: nodes } = capture;
4650
if (typeof node.index !== 'number') return undefined;
4751
const byIndex = buildIndexMap(nodes);
48-
const identity = boundedLocalIdentity(node);
52+
const identity = demoteNonUniqueId(boundedLocalIdentity(node), nodes);
4953
const ancestryWalk = buildAncestryChain(node, byIndex, TARGET_ANNOTATION_MAX_ANCESTRY);
5054
const fullAncestry = ancestryWalk.chain;
5155
const sibling = computeSiblingOrdinal(nodes, node);
@@ -237,6 +241,29 @@ export function filterIdentitySet(
237241
});
238242
}
239243

244+
/**
245+
* ADR 0012 decision 3 amendment (#1269): an id is identity only when it
246+
* uniquely denotes the target in the record-time tree. `boundedLocalIdentity`
247+
* reads a node's id unconditionally, but a shared framework resource id
248+
* (Android's `android:id/title` matching every list row is the measured
249+
* case — #1269) is not selective: on replay the id-led identity set spans
250+
* every row, position drifts, and verification correctly refuses a
251+
* confident bind. `idMatchCountInTree` — the SAME predicate
252+
* `buildSelectorChainForNode` uses for the selector chain — counts nodes
253+
* sharing this canonical id across the whole tree; when more than one, fall
254+
* back to role+label, exactly the identity an unrecorded id already computes.
255+
* Both sites sharing one predicate is what keeps the tuple and the chain from
256+
* disagreeing (demoting one but not the other). The rule is capture-time
257+
* uniqueness, not an id-namespace heuristic: a reused RN `FlatList` `testID`
258+
* hits the same demotion on iOS.
259+
*/
260+
function demoteNonUniqueId(identity: LocalIdentity, nodes: readonly SnapshotNode[]): LocalIdentity {
261+
if (identity.id === undefined) return identity;
262+
if (idMatchCountInTree(nodes, identity.id) <= 1) return identity;
263+
const { role, label } = identity;
264+
return { role, ...(label !== undefined ? { label } : {}) };
265+
}
266+
240267
function computeDisambiguationDomain(params: {
241268
nodes: readonly SnapshotNode[];
242269
byIndex: Map<number, SnapshotNode>;

0 commit comments

Comments
 (0)