Skip to content

feat: add Android snapshot helper#454

Merged
thymikee merged 13 commits intomainfrom
feat/android-snapshot-helper
Apr 27, 2026
Merged

feat: add Android snapshot helper#454
thymikee merged 13 commits intomainfrom
feat/android-snapshot-helper

Conversation

@thymikee
Copy link
Copy Markdown
Contributor

@thymikee thymikee commented Apr 26, 2026

Summary

Add a provider-neutral Android snapshot helper APK with release packaging, manifest/checksum support, public helper APIs, npm-bundled helper artifact resolution, and helper-first Android snapshot fallback metadata.

The bundled helper is now the default Android snapshot path when the npm package includes android-snapshot-helper/dist: the first snapshot verifies/installs the APK if missing or outdated, and any install/capture failure falls back to stock UIAutomator with androidSnapshot.fallbackReason. The earlier env-var gate and its now-obsolete config module were removed; direct helperArtifact injection remains for tests and provider integrations.

Follow-up hardening in this branch keeps helper APK releases on a persistent debug signing key, marks that keystore as binary, bundles the helper APK/manifest into npm packages during prepack, allowlists manifest install args before executing provider ADB, validates manifest SHA-256 shape up front, caps manifest/APK downloads, cleans owned helper cache directories, documents the stable release contract and -t install requirement, preserves and decodes XML attribute control whitespace, removes the CodeQL-flagged attribute regex path, captures interactive Android window roots for keyboards/system overlays with active-window fallback metadata, tolerates disappearing accessibility windows during helper traversal, resolves sdkmanager explicitly in CI/release workflows, keeps the helper subpath visible to Fallow, removes duplicate helper packaging work from CI, and splits the TypeScript helper implementation into artifact/install/capture/types modules behind the same public facade.

The public parseAndroidSnapshotHelperSnapshot() convenience wrapper was dropped; providers can compose parseAndroidSnapshotHelperOutput() with parseAndroidSnapshotHelperXml() when they need to parse raw instrumentation output.

Touched files: 36. Scope stayed within Android snapshot helper implementation, packaging/release workflow, snapshot metadata plumbing, tests, and docs.

Follow-up issue: #455 tracks install-result caching, standalone release APK smoke, toast capture investigation, and secondary-display investigation.

Validation

  • pnpm format
  • pnpm lint
  • pnpm typecheck
  • pnpm build
  • pnpm test:unit src/platforms/android/__tests__/snapshot-helper.test.ts src/platforms/android/__tests__/snapshot.test.ts
  • pnpm test:smoke
  • Full Android replay suite with bundled helper backend: AGENT_DEVICE_STATE_DIR=<tmp> node --experimental-strip-types src/bin.ts test test/integration/replays/android --retries 2 --report-junit test/artifacts/replays-android-full-helper.junit.xml (6 passed, 0 failed)
  • Post-replay Android snapshot metadata check reported androidSnapshot.backend = "android-helper", captureMode = "interactive-windows", windowCount = 2, nodeCount = 91
  • pnpm test:unit src/platforms/android/__tests__/index.test.ts src/platforms/android/__tests__/snapshot-helper.test.ts src/platforms/android/__tests__/snapshot.test.ts src/__tests__/client-shared.test.ts
  • pnpm test:unit src/platforms/android/__tests__/snapshot-helper.test.ts src/platforms/android/__tests__/snapshot.test.ts src/__tests__/client-shared.test.ts
  • pnpm test:unit src/__tests__/client-metro-packaged.test.ts
  • pnpm package:android-snapshot-helper
  • pnpm package:android-snapshot-helper:npm
  • npm pack --dry-run --json --ignore-scripts verified npm tarball includes helper APK + manifest and excludes .idsig
  • bundled helper artifact resolution smoke via node --experimental-strip-types
  • pnpm check:fallow --base origin/main
  • pnpm check:quick
  • pnpm check:tooling
  • pnpm check:unit
  • git diff --check
  • APK signer cert inspected with apksigner verify --print-certs
  • pnpm test-app:install
  • pnpm test-app:typecheck
  • Live Android smoke on Pixel_9_Pro_XL AVD with examples/test-app via Expo Go and helper artifact:
    • launcher snapshot reported androidSnapshot.backend = "android-helper", waitForIdleTimeoutMs = 500, nodeCount = 69
    • Home scenario from SkillGym: dismiss-notice press + diff snapshot -i
    • Catalog scenario from SkillGym: home-open-catalog, fill id="catalog-search" with tart, helper snapshot saw filtered catalog content
    • Settings scenario from SkillGym: load-diagnostics, helper snapshot saw retry-diagnostics

Known gap: no standalone release APK install smoke outside Expo Go; helper APK itself was built and exercised on the emulator above.

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Apr 26, 2026

PR Preview Action v1.8.1

QR code for preview link

🚀 View preview at
https://callstackincubator.github.io/agent-device/pr-preview/pr-454/

Built to branch gh-pages at 2026-04-27 19:43 UTC.
Preview will be ready when the GitHub Pages deployment is complete.

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f8e9c309db

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +84 to +92
keytool -genkeypair \
-keystore "$KEYSTORE" \
-storepass android \
-keypass android \
-alias androiddebugkey \
-keyalg RSA \
-keysize 2048 \
-validity 10000 \
-dname "CN=Android Debug,O=Android,C=US" >/dev/null 2>&1
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Use a persistent signing key for helper APK releases

This build script generates a new keystore for every run and signs the APK with that ephemeral key. Android package replacement requires the same signing certificate, so devices that already have an older helper installed will reject later install -r -t updates with signature mismatch errors, which means ensureAndroidSnapshotHelper cannot upgrade in place and will repeatedly fall back to stock snapshots. Keep a stable release keystore (or uninstall before install) so helper updates remain installable.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in b42b80e: the build now signs with a checked-in helper debug keystore instead of generating a new key per build. The runtime uninstall/retry path for INSTALL_FAILED_UPDATE_INCOMPATIBLE remains as an extra recovery path.

Comment on lines +204 to +205
default:
xml.append(character);
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve control whitespace when writing XML attributes

The default branch writes raw characters directly into XML attributes, including newline/tab/carriage-return characters. In XML attribute parsing those characters are normalized, so multiline Android text/content descriptions lose their original structure (for example line breaks becoming spaces), which can change snapshot text and break text-based matching/assertions. Encode these whitespace control characters as entities to preserve fidelity.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in b42b80e: the XML writer now entity-escapes newline, carriage return, and tab characters in attributes so parsed snapshot text keeps those control-whitespace characters instead of letting XML attribute normalization collapse them.

@thymikee thymikee force-pushed the feat/android-snapshot-helper branch from f8e9c30 to 69a4dae Compare April 26, 2026 23:31
@thymikee thymikee force-pushed the feat/android-snapshot-helper branch from 69a4dae to 55e7da9 Compare April 27, 2026 01:07
@thymikee thymikee merged commit d77a921 into main Apr 27, 2026
17 checks passed
@thymikee thymikee deleted the feat/android-snapshot-helper branch April 27, 2026 20:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(android): investigate custom snapshot helper for stale UIAutomator hierarchies

1 participant