You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
PR #1217 (typed direct Maestro engine, ADR 0015) ships with a small parser-regression fixture (scripts/maestro-conformance*, now running in the vitest unit project). That harness is deliberately narrow: 5 hand-authored flows compared at parsed-IR level against a hand-typed transcription of Maestro 2.5.1's command model. It proves parser self-consistency, not conformance — during #1217 development, four bug classes (percent rounding/rejection, target-swipe direction default, retry-cap semantics, settle-loop ordering) all slipped past it and were found by multi-hour live suite runs instead.
ADR 0015 commits to a real three-layer oracle as follow-up. This issue tracks it.
The three layers
Parser normalization, generated from upstream. Drive the pinned Maestro JVM parser (maestro-orchestra at tag v2.5.1 / a4c7c95f) over a corpus — starting with upstream's own e2e/demo_app/.maestro/*.yaml and maestro-test resource flows — and emit normalized fixtures. Classify every flow: parses-identically / explicitly-rejected / MISMATCH. The mismatch list is the parity backlog (this would have caught optional, pasteText, assertTrue mechanically).
Semantic vectors, generated by a JVM harness. A small Kotlin harness that calls the actual upstream functions (relative-point conversion, ElementCoordinateUtil, swipe-endpoint math, retry clamping, settle constants) against the pinned checkout and dumps JSON vectors. Hand-derived vectors are explicitly out — hand transcription is the failure mode this replaces.
App-observable differential scenarios. A small set of device flows run through both real Maestro and agent-device, comparing outcomes (settle behavior, tap retries, warned-vs-failed classification for optional). Opt-in/scheduled, not unit CI.
Wiring
Layers 1–2 regenerate only on explicit upstream-pin updates; the checked-in generated fixtures verify deterministically in normal CI (same pattern as the layering-guard node --test job). No Java in normal CI.
Layer 3 runs scheduled (like perf-nightly), not per-PR.
Verify the pinned upstream artifact SHA-256s at regeneration time (the current harness parses but never checks them).
Context
PR #1217 (typed direct Maestro engine, ADR 0015) ships with a small parser-regression fixture (
scripts/maestro-conformance*, now running in the vitest unit project). That harness is deliberately narrow: 5 hand-authored flows compared at parsed-IR level against a hand-typed transcription of Maestro 2.5.1's command model. It proves parser self-consistency, not conformance — during #1217 development, four bug classes (percent rounding/rejection, target-swipe direction default, retry-cap semantics, settle-loop ordering) all slipped past it and were found by multi-hour live suite runs instead.ADR 0015 commits to a real three-layer oracle as follow-up. This issue tracks it.
The three layers
maestro-orchestraat tag v2.5.1 / a4c7c95f) over a corpus — starting with upstream's owne2e/demo_app/.maestro/*.yamlandmaestro-testresource flows — and emit normalized fixtures. Classify every flow: parses-identically / explicitly-rejected / MISMATCH. The mismatch list is the parity backlog (this would have caughtoptional,pasteText,assertTruemechanically).ElementCoordinateUtil, swipe-endpoint math, retry clamping, settle constants) against the pinned checkout and dumps JSON vectors. Hand-derived vectors are explicitly out — hand transcription is the failure mode this replaces.optional). Opt-in/scheduled, not unit CI.Wiring
node --testjob). No Java in normal CI.Acceptance
support-matrix.tsis either fixture-backed or explicitly listed as unverified.optional) are encoded as expected-divergence entries, not silent mismatches.