Skip to content

Maestro conformance oracle: three generated layers replacing the hand-typed parser fixture #1274

Description

@thymikee

Context

PR #1217 (typed direct Maestro engine, ADR 0015) ships with a small parser-regression fixture (scripts/maestro-conformance*, now running in the vitest unit project). That harness is deliberately narrow: 5 hand-authored flows compared at parsed-IR level against a hand-typed transcription of Maestro 2.5.1's command model. It proves parser self-consistency, not conformance — during #1217 development, four bug classes (percent rounding/rejection, target-swipe direction default, retry-cap semantics, settle-loop ordering) all slipped past it and were found by multi-hour live suite runs instead.

ADR 0015 commits to a real three-layer oracle as follow-up. This issue tracks it.

The three layers

  1. Parser normalization, generated from upstream. Drive the pinned Maestro JVM parser (maestro-orchestra at tag v2.5.1 / a4c7c95f) over a corpus — starting with upstream's own e2e/demo_app/.maestro/*.yaml and maestro-test resource flows — and emit normalized fixtures. Classify every flow: parses-identically / explicitly-rejected / MISMATCH. The mismatch list is the parity backlog (this would have caught optional, pasteText, assertTrue mechanically).
  2. Semantic vectors, generated by a JVM harness. A small Kotlin harness that calls the actual upstream functions (relative-point conversion, ElementCoordinateUtil, swipe-endpoint math, retry clamping, settle constants) against the pinned checkout and dumps JSON vectors. Hand-derived vectors are explicitly out — hand transcription is the failure mode this replaces.
  3. App-observable differential scenarios. A small set of device flows run through both real Maestro and agent-device, comparing outcomes (settle behavior, tap retries, warned-vs-failed classification for optional). Opt-in/scheduled, not unit CI.

Wiring

  • Layers 1–2 regenerate only on explicit upstream-pin updates; the checked-in generated fixtures verify deterministically in normal CI (same pattern as the layering-guard node --test job). No Java in normal CI.
  • Layer 3 runs scheduled (like perf-nightly), not per-PR.
  • Verify the pinned upstream artifact SHA-256s at regeneration time (the current harness parses but never checks them).

Acceptance

  • The four refactor: replace Maestro compat with typed direct engine #1217 bug classes above each have a failing-before/passing-after fixture.
  • Every command in support-matrix.ts is either fixture-backed or explicitly listed as unverified.
  • Documented intentional deviations (hierarchy-vs-screenshot animation wait, horizontal-swipe presets, condition-poll-as-stabilization, ambiguity strictness under optional) are encoded as expected-divergence entries, not silent mismatches.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions