Skip to content

Fix e2e on 11.12 - #560

Open
YogendraShelke wants to merge 16 commits into
mx/11.12.xfrom
update-screenshot-11.12
Open

Fix e2e on 11.12#560
YogendraShelke wants to merge 16 commits into
mx/11.12.xfrom
update-screenshot-11.12

Conversation

@YogendraShelke

@YogendraShelke YogendraShelke commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes three defects in IntroScreen's slide-position handling, all rooted in how SwipeableContainer reconciled activeIndex (React state) with the FlashList's real scroll offset. The previous code used an isInitializing flag flipped by a double requestAnimationFrame, which guessed at timing instead of tracking what was actually requested.

Problems Fixed

Problem 1

Reopening a page showed slide 1 instead of the slide in the Active slide attribute, and fired a spurious On slide change back to slide 1

Root cause

activeIndex initialized to 0 while initialScrollIndex rendered the attribute's slide, so state and screen disagreed on mount. The effect only re-scrolled when slide !== activeIndex, so a matching-but-wrong state was never

Fix

Seed activeIndex from the attribute via a useState initializer, and force the positioning scroll once on the first pass where width is known (hasAppliedInitialScroll), independent of whether activeIndex already matches


Problem 2

The initial slide was reached with an animation, briefly leaving slide content unreachable for TalkBack/VoiceOver

Root cause

goToSlide always animated. activeIndex applies immediately and drives accessibilityElementsHidden / importantForAccessibility, so the slide exposed to screen readers was off-screen until the animation landed

Fix

goToSlide(pageNum, animated = true); the initial jump passes animated: false


Problem 3

Using Next / Previous / pagination could snap back to slide 1 and report an extra slide change

Root cause

onMomentumScrollEnd fires for programmatic scrolls too, reporting the offset the list is leaving rather than the one it's heading to. That stale offset overwrote activeIndex and re-fired onSlideChange. A drag flag alone can't disambiguate: it's only cleared inside onMomentumScrollEnd, so a drag that produces no momentum event leaves it stuck true, and the next programmatic scroll's stale event is misread as a user swipe

Fix

Track pendingScrollTarget — the slide goToSlide actually requested. onMomentumScrollEnd ignores any event whose index disagrees with that target, clearing it once reached. onScrollBeginDrag clears the target so a user taking over is always honoured, which also prevents the target latching when a scroll emits no momentum event

@YogendraShelke
YogendraShelke requested a review from a team as a code owner July 23, 2026 07:23
@YogendraShelke
YogendraShelke force-pushed the update-screenshot-11.12 branch 2 times, most recently from 6eed515 to b6be27f Compare July 24, 2026 11:46
@YogendraShelke YogendraShelke changed the title chore: inrease maestro driver setup timeout Fix e2e on 11.12 Jul 27, 2026
@YogendraShelke
YogendraShelke force-pushed the update-screenshot-11.12 branch from c9a59ea to 1ac6b5e Compare July 27, 2026 12:14
@YogendraShelke
YogendraShelke force-pushed the update-screenshot-11.12 branch from 1ac6b5e to eda1e26 Compare July 27, 2026 12:14
YogendraShelke and others added 8 commits July 27, 2026 17:50
chore: fix tests

chore: update ios test timeout

chore: inrease maestro driver setup timeout

chore: address flaky tests

chore: fix tests

chore: revert

chore: revert
…et index

Flows navigate the Widgets menu by tapping a single letter, which filters and
re-renders the list, then immediately tap the widget name. Nothing waited for
that render, so the second tap raced it. Maestro's implicit element lookup
usually absorbed the gap, but on a loaded CI runner it did not:

  Tap on "B"... COMPLETED
  Tap on "Bottom Sheet"... FAILED
  Element not found: Text matching regex: Bottom Sheet

which then passed on retry with the same build. Three bottom-sheet flows hit
this in one run.

Insert an extendedWaitUntil on the widget name between the two taps, matching
the pattern Modal_basic_non_native.yaml already used for its "Open" button.
71 flows across 20 widgets carried the race; all are now guarded.

Verified: all 82 flow files still parse, all 71 index-taps are followed by a
wait, and no flow ended up with a duplicated consecutive wait.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
restart_simulator is the recovery path for a wedged Maestro/XCTest driver
("iOS driver not ready in time"), but it did not restart anything. It ran
`simctl shutdown all || true`, discarding the result, and prepare_ios.sh then
reported:

    Monitoring boot status for iPhone 17 Pro (...).
    Device already booted, nothing to do.

The device never went down, so the shard got an app reinstall on top of the
same wedged runtime and ran degraded from that point on. Seen in a run where
the smoke check lost its driver and a later assertion then timed out.

Changes:
- Kill the XCTest runner before shutting down, not after: while it holds the
  device, shutdown can fail.
- Target SIMULATOR_DEVICE_ID when known, falling back to `all`.
- Poll until the device reports Shutdown (bounded at 30s) and emit a
  ::warning:: if it never does, instead of silently continuing.
- Drop the blind `sleep 10`. Targeted `simctl shutdown <udid>` is synchronous
  (measured ~3s to reach Shutdown), so the sleep only padded every retry.

Verified against a real simulator: booted device reaches Shutdown in ~3.4s
(was ~13s); an already-shutdown device logs the benign CoreSimulator 405 and
continues; the unset-UDID path falls back to `all`. Both return exit 0.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…creenshots

20 flows waited before takeScreenshot by watching for text that by design never
appears, with optional: true:

    - extendedWaitUntil:
         visible: randText  # Any random text that does not exist in the UI
         optional: true
         timeout: 15000

That is a sleep, not a wait. It ALWAYS burns the full timeout (it can never
succeed early) and never confirms the UI settled, so it is both slow and flaky.
It cost 325s (5.4 min) per full run PER PLATFORM, and still was not enough: a
bottom-sheet screenshot came in at 14.7% similarity against a 95% threshold
because the sheet was mid-expand when captured, then passed on retry.

Replace with waitForAnimationToEnd, which polls until the screen stops changing
-- returning as soon as the image decode / sheet animation finishes, and still
capping at the timeout. Already used by animation-native, so no new concept.

Each site keeps its existing timeout as the cap, so no flow gets a smaller
budget than before; the saving comes from returning early rather than from
lowering the ceiling.

Verified: no randText remains anywhere, all 82 flow files parse, every
waitForAnimationToEnd has a timeout, all 20 timeouts carried over unchanged,
and the assertVisible count is unchanged (99) so no real assertion was dropped
along with the stale comments.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
NativePipeline already had concurrency + cancel-in-progress, but Build,
UnitTests, Release and ShaCheck had none — so every push to a PR left the
previous run of each still going. On GitHub Free all of them share one
20-concurrent-job cap with NativePipeline's ~10 test shards, so stale runs
were not just noise: they delayed the run for the commit under review.

cancel-in-progress is gated on pull_request. A push run on main / mx-* is the
recorded CI result for that commit (and Release.yml's push path publishes),
so those are allowed to finish; only superseded PR runs get cancelled.

Not added to MarketplaceRelease.yml (tag-triggered publish — never cancel),
NightlyDispatcher.yml (scheduled), the workflow_dispatch-only workflows, or
PullRequestLabeler.yml (seconds long).

fix: make native tests cancellable

ci: make cancelling the native pipeline actually stop it

Clicking "Cancel workflow" appeared to do nothing for minutes. The reason is
that `if: always()` is exempt from cancellation by design: a step or job with
always() keeps running (and a job with it can even start) after the run is
cancelled. The pipeline used always() in eleven places, including every upload
inside archive-test-results — so on cancel each of the ~10 in-flight shards
first uploaded five artifact sets nobody was waiting for.

Replaced all of them with `!cancelled()`, which is equivalent for the cases we
care about — it is true on success AND on failure, so failing shards still
archive their screenshots/logs/videos/diffs, and jobs with skipped
dependencies still run — but is false on cancellation. A shard that trips
timeout-minutes is not "cancelled" in this sense, so timeout artifacts survive.

Not fixed here: the maestro bash harness does not trap SIGINT/SIGTERM, so a
shard mid-flow still runs to the runner's force-kill. Cancellation is much
faster, not instant.
@YogendraShelke
YogendraShelke force-pushed the update-screenshot-11.12 branch from 062e66f to 238a920 Compare July 27, 2026 12:47
…mits

The Android IntroScreen flow failed at its last assertion: after re-entering
the page, NEXT moved slide 2 -> 3 and the attribute correctly became 3, then
immediately flipped to 1, so "Active slide: 3" was never visible. The failure
video shows the value on screen going 2 -> 3 -> 1 with no input in between.

Cause: once the programmatic scroll to slide 3 settles, Android's list emits a
SECOND onScrollBeginDrag + onMomentumScrollEnd pair, and that momentum event
carries the offset the list moved AWAY from (0). The existing pendingScrollTarget
guard had already been cleared by the first, legitimate momentum end, and
wasUserDragging was true because the settling list — not a finger — began the
drag. So the event was treated as a swipe back to slide 1 and rewrote the
attribute. This is the same class of bug as the earlier phantom-momentum fix,
one step later in the sequence.

Fix: remember the slide a programmatic scroll settled on, and ignore a momentum
end that moves more than one page from activeIndex while that memory is set.
The list is pagingEnabled, so a genuine drag moves exactly one page; a
multi-page jump right after a scroll settled can only be that scroll's tail.
One-page swipes, including a swipe straight after a programmatic scroll, still
report normally (covered by existing tests).

Regression test added; it fails with setValue receiving [3, 1] and a spurious
onSlideChange(0, 2) when the guard is removed.

Revert "fix(e2e): scroll the alphabet-index target into view before tapping it"

This reverts commit 31e9fd78c2fb70fb0e88b33b2d9bf8a983c4db51.

fix(e2e): scroll the alphabet-index target into view before tapping it

FloatingActionButton.yaml failed on iOS with

  Tap on "F"... COMPLETED
  Assert that "Floating action button" is visible... FAILED

then passed on retry with the same build. The flow already had the 10s
extendedWaitUntil added in 20d3d52, and that wait is what timed out — so
this is not the render race that commit fixed; nothing was going to change in
second 11 and a longer timeout would not have helped.

extendedWaitUntil: visible requires the element to be IN THE VIEWPORT, not
merely present in the hierarchy. Tapping the index scrolls to the letter's
section, and under F the entries are Feedback then Floating action button; if
the scroll settles with Feedback at the bottom edge, the target is rendered
just below the fold and `visible` stays false forever. The retry's scroll
landed a few pixels differently and it passed.

Use scrollUntilVisible instead: it succeeds immediately when the element is
already in view and scrolls to it when it is not, so it covers both the
original render race and the off-screen case. centerElement keeps the
subsequent tap away from the screen edge.

Applied to the 55 flows whose index letter hosts more than one widget entry
(B, C, F, I, L, P, R, S) — the case where the target can be pushed below the
fold. The 16 flows on a sole-entry letter keep the cheaper wait.

Verified: all 82 flows still parse, all 71 index taps remain guarded (55
scrollUntilVisible + 16 extendedWaitUntil, 0 unguarded), and the pinned 2.6.1
binary accepts every field used — confirmed it rejects an unknown property, so
that parse is a real schema check.

ci(e2e): archive maestro's debug output for failed flows

Maestro writes a debug bundle per run — failure screenshot, maestro.log
(which records the element Bounds each selector matched) and
commands-<flow>.json — to ~/.maestro/tests/<timestamp>/. That is outside the
workspace, so none of it was ever uploaded, and it is exactly what
distinguishes "element never rendered" from "element rendered off-screen
below the fold" when an assertVisible/extendedWaitUntil fails. Without it
those failures can only be diagnosed by inference.

Redirect it into the workspace with --debug-output (+ --flatten-debug-output,
which its own help text recommends for CI) one dir per flow, and archive it.
Kept only for FAILED flows, mirroring the existing video policy, so a green
shard uploads nothing — each bundle is ~300KB.

Also gitignore maestro/debug and maestro/videos: both are produced into the
workspace by local runs too. The existing *.log rule covered maestro.log but
not the bundle's screenshots or commands JSON.

Verified --debug-output and --flatten-debug-output against the pinned 2.6.1
binary, and drove run_tests with a stub to confirm a failing flow's bundle is
kept while a passing flow's is discarded.

fix(maestro): recover from iOS driver/accessibility faults instead of hanging

A shard could die with, and never recover from:

  Exception in thread "main" UnknownFailure(errorResponse=Request for
  viewHierarchy failed, code: 500, body: {"errorMessage":"Error getting
  element frame kAXErrorInvalidUIElement","code":"internal"})

Traced through the pinned 2.6.1 jars, this is unrecoverable by design.
XCTestDriverClient.handleExceptions maps a 500 to a typed error only for
four known message strings and otherwise throws UnknownFailure;
XCTestIOSDevice.execute catches only AppCrash/OperationTimeout/Unreachable,
and IOSDriver.runDeviceCall catches only the matching IOSDeviceErrors.
UnknownFailure matches none, so it escapes to main — hence "Exception in
thread main" rather than a failed step, and hence a flow-level `retry:`
block cannot help: the CLI is already dead, no command failed. kAXError*
means the accessibility element reference is stale, so every later query
against that driver fails identically.

Recovery therefore has to live in the shell. Add run_maestro(), which:

  - runs `maestro test` under a wall-clock watchdog (MAESTRO_FLOW_TIMEOUT,
    480s) so a JVM that died in main but is held open by non-daemon threads
    can't eat the 30-minute job budget, and
  - classifies a FAILED run as infrastructure (exit 90) when the output
    carries a driver/accessibility signature, so callers can tell "the
    device broke" from "the widget is wrong".

run_tests then resets the device and gives the flow one clean attempt with
a freshly reinstalled driver, instead of recording a widget failure and
marching every remaining flow through the same wedged driver. All three
call sites (flows, smoke check, retries) now go through run_maestro, and
the duplicated device-reset branches collapse into reset_device().

Only non-zero exits are reclassified, so a flow that passed after Maestro's
own internal retry logged such an error still counts as a pass.

fix(maestro): actually apply the iOS driver startup timeout

MAESTRO_DRIVER_STARTUP_TIMEOUT was passed to maestro as `--env`, but
LocalXCTestInstaller.getStartupTimeout() reads it with System.getenv() —
`--env` only fills Maestro's per-flow variable map, which that function
never consults. So every value we passed was silently ignored and the
Kotlin runCatching kept falling back to its built-in 120000 default.
Export it instead (and raise it to 240000), and drop the three no-op
`--env MAESTRO_DRIVER_STARTUP_TIMEOUT` arguments.

Also stop reinstalling the XCUITest driver once per flow. Maestro's
TestCommand.reinstallDriver defaults to true, so each `maestro test`
process runs restartXCTestRunner (uninstall + install + start) — and that
install is the window that throws IOSDriverTimeoutException on a 3-vCPU
macOS runner. The smoke check still installs the driver fresh at the start
of every shard and retries still reinstall via restart_simulator, so only
the first-pass flows reuse it. MAESTRO_REUSE_DRIVER=false restores the
old behaviour.
`mktemp -t maestro-run` works on BSD/macOS, which appends the random suffix
itself, but GNU coreutils requires at least 6 X's in the template. So on the
Linux Android runners every flow printed

  mktemp: too few X's in template 'maestro-run'
  '': No such file or directory (os error 2)

— log_file was empty, so the tee redirection failed and maestro never ran.
iOS was unaffected, which is why this passed local and macOS testing.

Use an explicit XXXXXX template, which both implementations accept, and fall
back to a PID-based path if mktemp fails for any other reason: artifact
plumbing must never be what decides whether the tests run.

Audited the rest of the commands added in this series (tr -c, grep -qE) —
both POSIX, and already used by the pre-existing recording code that runs on
Android.

Verified by stubbing a GNU-strict mktemp on PATH: reproduces the CI error with
the old form and run_maestro completes with the new one.

fix(intro-screen-native): catch the trailing drag whatever order it arrives in

The Android e2e flow failed deterministically (both the first attempt and the
retry) at `Assert that "Active slide: 3" is visible`. The CI video shows the
widget reach slide 3, then snap back to slide 1 and increment the change
counter, so the trailing drag the previous guard was written for still got
through.

It got through because the guard required evidence that a programmatic scroll
had landed (settledScrollTarget), and that evidence is not always recorded: the
settling list can emit its trailing scrollBeginDrag BEFORE the scroll's own
momentum end, and onScrollBeginDrag clears the pending target, so the branch
that would have set settledScrollTarget never runs. The guard was then skipped
and the stale offset was treated as a swipe.

Drop the bookkeeping and rely on the geometry instead. The list is
pagingEnabled, so a drag settles at most one page from where it began; a
momentum end further away than that cannot be a finger regardless of what came
before it. Covered by a test reproducing the CI ordering, which fails without
the change.
@YogendraShelke
YogendraShelke force-pushed the update-screenshot-11.12 branch from cb57696 to 2ea757e Compare July 27, 2026 17:43
The ios-app job paid npm i + pod install + xcodebuild on every run, even for a
widget-only change that cannot affect the native app. That is the most expensive
step in the pipeline and it runs on a scarce macOS runner.

It is safe to skip because the Mendix JS bundle is a RESOURCE of the .app rather
than compiled input: in nativeTemplate.xcodeproj the `Bundle` entry is a folder
reference (lastKnownFileType = folder) in the Resources phase, there is no
react-native-xcode.sh phase (unlike a stock RN app the template never builds a
bundle itself, it only copies what mxbuild produced), and AppDelegate resolves it
at launch through MendixAppDelegate.getJSBundleFile(). Widget .mpks are compiled
into the JS bundle by mxbuild, never into the iOS binary. So on a hit we restore
the previous .app, replace its bundle folder, and skip the build entirely; the
job's uploaded artifact is unchanged, so prepare_ios.sh and the test shards are
untouched.

Two fixes to the caching this replaces, which cached DerivedData but still built
every time:

- The old key hashed ios/**/*.swift and Podfile.lock but not
  native_dependencies.json. add-native-dependencies.js merges that file into
  native-template/package.json and the Podfile autolinks from there, so
  `pod install` REGENERATES Podfile.lock — the committed lock is not the
  effective one and a new native dependency did not change the key.
- restore-keys: ${{ runner.os }}-ios-build- restored the newest entry whatever
  changed, mixing stale products with fresh pods. A partial-input change must
  rebuild, so the new key has no restore-keys.

The key also pins the resolved native-template commit (nt_branch can be a moving
branch, so the name is not a content identity) and the Xcode version, since the
runner image updates it independently of our inputs. A hit that does not contain
the exact .app prepare_ios.sh installs, with a bundle inside it, warns and falls
back to a full build rather than shipping a half-swapped app.

Caches only the Products dir, not the DerivedData root: it is what we upload and
a fraction of the size, so it does not evict the AVD/dist caches.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants