feat: declare layer-3 divergences and schedule the differential

Layer 3 ran both engines for the first time (29504440599) and immediately found a
real engine bug. Blocking the measurement instrument on repairing what it just
measured inverts the dependency, so layer 3 now gets the contract layer 1 already
had: every divergence is a decision on the record.

Adds `knownDivergence: { reason, tracking }` to the scenario type — the layer-3
twin of FLOW_DIVERGENCES. A declared divergence keeps the run green; only
UNDECLARED ones fail. Two rules stop that from rotting, both enforced
mechanically rather than by prose discipline:

  - `tracking` is required and must be a real issue URL (run.test.ts), because a
    declaration with nothing behind it is how "temporarily expected" becomes
    permanent without anyone deciding to.
  - a stale declaration FAILS: if a declared-divergent scenario starts passing,
    the run goes red until the declaration is removed. The fix PR must delete it,
    and the differential then enforces the gap stays closed — the oracle is the
    acceptance test for its own findings.

Declared:
  - settle-after-tap  -> #1299. Our scrollUntilVisible times out finding
    home-open-form where Maestro 2.5.1 scrolls to it and passes. Real engine
    correctness bug in an advertised command, found by this differential. Blocks
    bug class 4's device detector until fixed.
  - tap-retry-if-no-change -> #1300. The invariant caught the scenario being
    vacuous: both engines pass but tapRetries was 0, so retryIfNoChange never
    ran. Needs an inert fixture control; a scenario defect, not an engine one.

Proven green on both engines and enforced now: percent-swipe,
optional-warned-not-failed — the latter is real device-verified warned-vs-failed
parity.

With declarations in place the differential is green, so the schedule goes in
(cron 05:00) per #1274. A green run still prints what it is not proving.

Author: thymikee

.github/workflows/conformance-differential.yml

@@ -6,16 +6,16 @@ name: Conformance Differential
 # replay trace. This is the only layer that can catch settle-loop ordering, which
 # has no reflectable upstream constant.
 #
-# DISPATCH-ONLY, DELIBERATELY. This path has never executed end-to-end: nothing
-# else in this repo builds or installs the Expo fixture app, so the build/install
-# steps below are new and unproven. A nightly cron that fails at 05:00 every day
-# teaches us nothing and trains people to ignore it. Run it manually, watch it,
-# fix what breaks, and only then add:
-#   schedule:
-#     - cron: "0 5 * * *"
-# Tracked as the follow-up to #1274; layers 1-2 verify per-PR without any device.
+# Scheduled: this path is proven end-to-end (run 29504440599 executed both
+# engines against the real fixture app). Scenarios that currently diverge are
+# DECLARED via `knownDivergence` in differential/scenarios.ts with a tracking
+# issue, so the schedule stays green on known gaps and fails only on undeclared
+# ones — the same contract layer 1 has. A declaration that stops reproducing also
+# fails, so a fix cannot land while leaving the oracle blind.
 
 on:
+  schedule:
+    - cron: "0 5 * * *"
   workflow_dispatch:
     inputs:
       only:

docs/adr/0015-direct-maestro-engine.md

@@ -137,11 +137,17 @@ so a hand edit to a captured command or constant fails the build; the scheduled
 job then re-runs the harness against the pinned jars and fails on any byte difference, which is what makes
 "generated from upstream" enforced rather than documented. Layer 3 runs a small set of app-observable
 differential scenarios (settle ordering, tap retries, optional warned-vs-failed) through both engines
-against the real fixture app, pinning the Maestro CLI to the same version as layers 1-2; it is
-dispatch-only until its first supervised run, and never per-PR. The four #1217 regressions (percent
+against the real fixture app, pinning the Maestro CLI to the same version as layers 1-2; it is scheduled,
+never per-PR. Layer 3 carries the same declared-divergence contract as layer 1: a scenario that currently
+fails declares a `knownDivergence` with a required tracking issue, the schedule stays green on that known
+gap, and only undeclared divergences fail. A declaration that stops reproducing also fails, so the fix PR
+must remove it and the differential becomes the acceptance test for its own findings. This exists because
+the instrument must not be blocked on repairing what it just measured. The four #1217 regressions (percent
 rounding/rejection, target-swipe direction default, retry-cap semantics, settle-loop ordering) each have a
-dedicated fixture; bug class 4's detector is an engine-side timing invariant, unit-tested against
-synthetic traces, that only executes on the device path.
+dedicated fixture. Percent truncation's runtime half is pinned by a pure unit test rather than a device
+scenario, because a one-pixel delta is not app-observable. Bug class 4's detector is an engine-side timing
+invariant, unit-tested against synthetic traces, that executes on the device path — currently blocked by a
+declared `scrollUntilVisible` divergence (#1299) that the differential itself found on its first working run.
 
 Documented intentional deviations are encoded as expected-divergence entries rather than silent
 mismatches: the hierarchy-vs-screenshot animation wait (constants match, mechanism differs), the

scripts/maestro-conformance/README.md

@@ -54,11 +54,25 @@ exist on no device — so a device run against them would fail before exercising
 any runtime behavior, making the settle detector silently vacuous. A test
 enforces the separation, and the workflow hard-fails if the app is not installed.
 
-**Layer 3 is dispatch-only and has never executed end-to-end.** Nothing else in
-this repo builds or installs the Expo fixture app, so those steps are new and
-unproven. Run it manually, watch it, fix what breaks, and only then add the cron
-— a nightly job that fails at 05:00 every day teaches nothing and trains people
-to ignore it.
+### Declared divergences (`knownDivergence`)
+
+Layer 3 has the same contract as layer 1: **every divergence is a decision on the
+record.** When the differential catches a real engine bug, the instrument does not
+block on repairing what it just measured — the scenario declares it with a
+`knownDivergence: { reason, tracking }`, the scheduled run stays green on that
+known gap, and only *undeclared* divergences fail.
+
+Two rules keep that from rotting, both enforced by `run.test.ts` / the runner
+rather than by good intentions:
+
+- **`tracking` is required** and must be a real issue URL. A declared divergence
+  with nothing behind it is how "temporarily expected" silently becomes permanent.
+- **A stale declaration fails.** If a declared-divergent scenario starts passing,
+  the run goes red until the declaration is removed — so the fix PR must delete
+  it, and the oracle then enforces that the gap stays closed. The differential is
+  the acceptance test for its own findings.
+
+A green run still prints what it is *not* proving.
 
 This matters most for **bug class 4** (settle ordering), whose 200ms × 10 loop has
 no reflectable upstream constant, so layer 3 is its only home. Its detector is the
@@ -121,11 +135,23 @@ This resolves the pinned jars, **verifies their SHA-256 against
 refresh the vendored corpus flows and their `manifest.json` `sha256`s, regenerate,
 and reconcile any new divergences.
 
-## Layer 3 (scheduled / opt-in)
+## Layer 3 (device)
 
 ```sh
 pnpm maestro:conformance:differential -- --platform ios --out-dir .tmp/diff
 ```
 
-Runs on the `conformance-differential` workflow (scheduled). `--dry-run`
-validates the scenario registry without a device.
+Runs scheduled on the `conformance-differential` workflow, and on demand via
+`workflow_dispatch`. `--dry-run` validates the scenario registry without a device.
+
+The workflow builds and installs the fixture app, verifies its bundle id, pins
+the Maestro CLI to the same version as layers 1-2, and passes `--maestro` so the
+flow routes through the compat engine.
+
+**Investigate locally, not through CI.** A device iteration in CI is ~40 minutes;
+`--only` plus a local simulator is minutes:
+
+```sh
+pnpm test-app:install && pnpm --dir examples/test-app exec expo run:ios --configuration Release
+pnpm maestro:conformance:differential -- --platform ios --only settle-after-tap --trace-root .agent-device
+```

scripts/maestro-conformance/differential/run.test.ts

@@ -46,6 +46,26 @@ test('no differential scenario points at the parse-only layer-1 corpus', () => {
   }
 });
 
+// A declared divergence without an issue behind it is how "temporarily expected"
+// becomes permanent without anyone deciding to. Layer 1 requires `unsupported`
+// on every we-reject entry for the same reason; enforce the twin here, because
+// the lesson of this whole arc is that prose discipline does not survive contact
+// with a two-day debugging session.
+test('every knownDivergence carries a tracking issue', () => {
+  const problems: string[] = [];
+  for (const scenario of DIFFERENTIAL_SCENARIOS) {
+    const declared = scenario.knownDivergence;
+    if (!declared) continue;
+    if (!/^https:\/\/github\.com\/.+\/issues\/\d+$/.test(declared.tracking ?? '')) {
+      problems.push(`${scenario.id}: knownDivergence.tracking must be a GitHub issue URL`);
+    }
+    if (!declared.reason || declared.reason.length < 20) {
+      problems.push(`${scenario.id}: knownDivergence.reason must explain what it blocks`);
+    }
+  }
+  assert.deepEqual(problems, [], problems.join('\n'));
+});
+
 test('every device flow targets the fixture app the workflow installs', () => {
   for (const scenario of DIFFERENTIAL_SCENARIOS) {
     const body = fs.readFileSync(path.join(CONFORMANCE_DIR, scenario.flow), 'utf8');

scripts/maestro-conformance/differential/run.ts

@@ -90,6 +90,16 @@ export type ScenarioReport = {
   outcomeDiverged: boolean;
   /** Engine-side invariants over agent-device's own timing trace. */
   invariants: InvariantResult[];
+  /**
+   * ok                  — behaved as expected.
+   * failed              — an UNDECLARED divergence. The signal this exists for.
+   * known-divergence    — failed exactly as declared; tracked, so the run stays green.
+   * stale-declaration   — passed while still declared divergent: remove the
+   *                       declaration (the gap is closed). Fails, so a fix cannot
+   *                       land while leaving the oracle blind to a regression.
+   */
+  status: 'ok' | 'failed' | 'known-divergence' | 'stale-declaration';
+  tracking?: string;
   failed: boolean;
 };
 
@@ -138,6 +148,20 @@ function runScenario(scenario: DifferentialScenario, options: RunnerOptions): Sc
     ? evaluateInvariants(trace ? readTrace(trace) : [], scenario.engineInvariants)
     : [];
   const invariantFailed = invariants.some((result) => result.status !== 'held');
+  const misbehaved = outcomeDiverged || invariantFailed;
+
+  // A declared divergence is an expected, tracked gap: it keeps the run green so
+  // the oracle is not blocked on the engine bug it just found. But a declaration
+  // that no longer reproduces FAILS — the fix PR has to delete it, which is what
+  // turns the differential into the acceptance test for its own findings.
+  const declared = scenario.knownDivergence;
+  const status = declared
+    ? misbehaved
+      ? ('known-divergence' as const)
+      : ('stale-declaration' as const)
+    : misbehaved
+      ? ('failed' as const)
+      : ('ok' as const);
 
   return {
     id: scenario.id,
@@ -146,7 +170,9 @@ function runScenario(scenario: DifferentialScenario, options: RunnerOptions): Sc
     agentDevice,
     outcomeDiverged,
     invariants,
-    failed: outcomeDiverged || invariantFailed,
+    status,
+    ...(declared ? { tracking: declared.tracking } : {}),
+    failed: status === 'failed' || status === 'stale-declaration',
   };
 }
 
@@ -158,11 +184,15 @@ function main(argv: readonly string[]): void {
   if (options.dryRun) {
     for (const scenario of scenarios) {
       const invariants = scenario.engineInvariants?.length ?? 0;
+      const declared = scenario.knownDivergence
+        ? `\tdeclared-divergence=${scenario.knownDivergence.tracking}`
+        : '';
       console.log(
-        `${scenario.id}\t${scenario.flow}\texpect=${scenario.expect}\tinvariants=${invariants}`,
+        `${scenario.id}\t${scenario.flow}\texpect=${scenario.expect}\tinvariants=${invariants}${declared}`,
       );
     }
-    console.log(`\n${scenarios.length} scenario(s) validated.`);
+    const known = scenarios.filter((scenario) => scenario.knownDivergence).length;
+    console.log(`\n${scenarios.length} scenario(s) validated, ${known} declared divergence(s).`);
     return;
   }
 
@@ -176,13 +206,26 @@ function main(argv: readonly string[]): void {
   }
 
   for (const report of reports) {
-    const status = report.failed ? 'FAILED' : 'ok';
     console.log(
-      `${status.padEnd(7)} ${report.id} maestro=${report.maestro.outcome} agent-device=${report.agentDevice.outcome}`,
+      `${report.status.padEnd(17)} ${report.id} maestro=${report.maestro.outcome} agent-device=${report.agentDevice.outcome}`,
     );
     for (const result of report.invariants) {
       console.log(`        invariant ${result.status}: ${result.detail}`);
     }
+    if (report.status === 'known-divergence') {
+      console.log(`        declared divergence, tracked: ${report.tracking}`);
+    }
+    if (report.status === 'stale-declaration') {
+      console.log(
+        `        passed while declared divergent — remove knownDivergence (${report.tracking}) so this stays enforced`,
+      );
+    }
+  }
+
+  // Keep declared gaps visible: a green run must still say what it is not proving.
+  const known = reports.filter((report) => report.status === 'known-divergence');
+  if (known.length > 0) {
+    console.log(`\n${known.length} declared divergence(s), not enforced: ${known.map((r) => r.id).join(', ')}`);
   }
 
   const failed = reports.filter((report) => report.failed);

scripts/maestro-conformance/differential/scenarios.ts

@@ -25,6 +25,27 @@ export const DIFFERENTIAL_APP_ID = 'com.callstack.agentdevicelab';
 
 export type DifferentialOutcome = 'pass' | 'fail';
 
+/**
+ * A divergence this scenario is currently EXPECTED to exhibit.
+ *
+ * This is the layer-3 twin of layer 1's FLOW_DIVERGENCES, and it exists for the
+ * same reason: the oracle's contract is that every divergence is a decision on
+ * the record, never a silent one. When the differential catches a real engine
+ * bug, the instrument must not be blocked on repairing what it just measured —
+ * declare it, file it, and let the scheduled run stay green on the known gap
+ * while still failing on anything undeclared.
+ *
+ * `tracking` is REQUIRED and enforced by run.test.ts. A declared divergence with
+ * no issue behind it is exactly how "temporarily expected" becomes permanent
+ * without anyone deciding to make it so.
+ */
+export type KnownDivergence = {
+  /** Why this scenario currently fails, and what it blocks. */
+  reason: string;
+  /** Issue tracking the fix. Required — see above. */
+  tracking: string;
+};
+
 export type DifferentialScenario = {
   id: string;
   /** The #1217 bug class this scenario guards, when applicable. */
@@ -39,6 +60,12 @@ export type DifferentialScenario = {
   engineInvariants?: Invariant[];
   /** What a divergence would indicate. */
   divergenceMeans: string;
+  /**
+   * Declared, tracked failure. The run stays green while this holds — but it
+   * FAILS if the scenario starts passing, so the fix PR must remove the
+   * declaration and the oracle enforces that the gap stays closed.
+   */
+  knownDivergence?: KnownDivergence;
 };
 
 export const DIFFERENTIAL_SCENARIOS: DifferentialScenario[] = [
@@ -61,6 +88,11 @@ export const DIFFERENTIAL_SCENARIOS: DifferentialScenario[] = [
     ],
     divergenceMeans:
       'agent-device settled in a different order than upstream (sleep-before vs sleep-after capture) or never latches within the shared budget.',
+    knownDivergence: {
+      reason:
+        'Caught by this differential on its first working run: our scrollUntilVisible times out finding home-open-form where Maestro 2.5.1 scrolls to it and passes (maestro=pass, agent-device=fail). The flow cannot reach its tapOn step, so bug class 4 has no working device detector until that engine bug is fixed. Declared rather than chased inline — the instrument does not block on repairing what it just measured.',
+      tracking: 'https://github.com/callstack/agent-device/issues/1299',
+    },
   },
   {
     id: 'percent-swipe',
@@ -97,6 +129,11 @@ export const DIFFERENTIAL_SCENARIOS: DifferentialScenario[] = [
       },
     ],
     divergenceMeans: 'agent-device fails a tap upstream completes (or vice versa) under retry-if-no-change.',
+    knownDivergence: {
+      reason:
+        'The invariant caught this scenario being vacuous: both engines pass, but tapRetries was 0, so retryIfNoChange never ran. Tapping the fixture app title does not hold the hierarchy signature still (its ScreenTitle carries a dynamic cart badge), so the no-change path is never forced. Needs an inert fixture control — a defect in the scenario, not the engine.',
+      tracking: 'https://github.com/callstack/agent-device/issues/1300',
+    },
   },
   {
     id: 'optional-warned-not-failed',