Fix e2e on 11.12 - #560
Open
YogendraShelke wants to merge 16 commits into
Open
Conversation
YogendraShelke
force-pushed
the
update-screenshot-11.12
branch
2 times, most recently
from
July 24, 2026 11:46
6eed515 to
b6be27f
Compare
YogendraShelke
force-pushed
the
update-screenshot-11.12
branch
from
July 27, 2026 12:14
c9a59ea to
1ac6b5e
Compare
YogendraShelke
force-pushed
the
update-screenshot-11.12
branch
from
July 27, 2026 12:14
1ac6b5e to
eda1e26
Compare
chore: fix tests chore: update ios test timeout chore: inrease maestro driver setup timeout chore: address flaky tests chore: fix tests chore: revert chore: revert
…et index Flows navigate the Widgets menu by tapping a single letter, which filters and re-renders the list, then immediately tap the widget name. Nothing waited for that render, so the second tap raced it. Maestro's implicit element lookup usually absorbed the gap, but on a loaded CI runner it did not: Tap on "B"... COMPLETED Tap on "Bottom Sheet"... FAILED Element not found: Text matching regex: Bottom Sheet which then passed on retry with the same build. Three bottom-sheet flows hit this in one run. Insert an extendedWaitUntil on the widget name between the two taps, matching the pattern Modal_basic_non_native.yaml already used for its "Open" button. 71 flows across 20 widgets carried the race; all are now guarded. Verified: all 82 flow files still parse, all 71 index-taps are followed by a wait, and no flow ended up with a duplicated consecutive wait. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
restart_simulator is the recovery path for a wedged Maestro/XCTest driver
("iOS driver not ready in time"), but it did not restart anything. It ran
`simctl shutdown all || true`, discarding the result, and prepare_ios.sh then
reported:
Monitoring boot status for iPhone 17 Pro (...).
Device already booted, nothing to do.
The device never went down, so the shard got an app reinstall on top of the
same wedged runtime and ran degraded from that point on. Seen in a run where
the smoke check lost its driver and a later assertion then timed out.
Changes:
- Kill the XCTest runner before shutting down, not after: while it holds the
device, shutdown can fail.
- Target SIMULATOR_DEVICE_ID when known, falling back to `all`.
- Poll until the device reports Shutdown (bounded at 30s) and emit a
::warning:: if it never does, instead of silently continuing.
- Drop the blind `sleep 10`. Targeted `simctl shutdown <udid>` is synchronous
(measured ~3s to reach Shutdown), so the sleep only padded every retry.
Verified against a real simulator: booted device reaches Shutdown in ~3.4s
(was ~13s); an already-shutdown device logs the benign CoreSimulator 405 and
continues; the unset-UDID path falls back to `all`. Both return exit 0.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…creenshots
20 flows waited before takeScreenshot by watching for text that by design never
appears, with optional: true:
- extendedWaitUntil:
visible: randText # Any random text that does not exist in the UI
optional: true
timeout: 15000
That is a sleep, not a wait. It ALWAYS burns the full timeout (it can never
succeed early) and never confirms the UI settled, so it is both slow and flaky.
It cost 325s (5.4 min) per full run PER PLATFORM, and still was not enough: a
bottom-sheet screenshot came in at 14.7% similarity against a 95% threshold
because the sheet was mid-expand when captured, then passed on retry.
Replace with waitForAnimationToEnd, which polls until the screen stops changing
-- returning as soon as the image decode / sheet animation finishes, and still
capping at the timeout. Already used by animation-native, so no new concept.
Each site keeps its existing timeout as the cap, so no flow gets a smaller
budget than before; the saving comes from returning early rather than from
lowering the ceiling.
Verified: no randText remains anywhere, all 82 flow files parse, every
waitForAnimationToEnd has a timeout, all 20 timeouts carried over unchanged,
and the assertVisible count is unchanged (99) so no real assertion was dropped
along with the stale comments.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
NativePipeline already had concurrency + cancel-in-progress, but Build, UnitTests, Release and ShaCheck had none — so every push to a PR left the previous run of each still going. On GitHub Free all of them share one 20-concurrent-job cap with NativePipeline's ~10 test shards, so stale runs were not just noise: they delayed the run for the commit under review. cancel-in-progress is gated on pull_request. A push run on main / mx-* is the recorded CI result for that commit (and Release.yml's push path publishes), so those are allowed to finish; only superseded PR runs get cancelled. Not added to MarketplaceRelease.yml (tag-triggered publish — never cancel), NightlyDispatcher.yml (scheduled), the workflow_dispatch-only workflows, or PullRequestLabeler.yml (seconds long). fix: make native tests cancellable ci: make cancelling the native pipeline actually stop it Clicking "Cancel workflow" appeared to do nothing for minutes. The reason is that `if: always()` is exempt from cancellation by design: a step or job with always() keeps running (and a job with it can even start) after the run is cancelled. The pipeline used always() in eleven places, including every upload inside archive-test-results — so on cancel each of the ~10 in-flight shards first uploaded five artifact sets nobody was waiting for. Replaced all of them with `!cancelled()`, which is equivalent for the cases we care about — it is true on success AND on failure, so failing shards still archive their screenshots/logs/videos/diffs, and jobs with skipped dependencies still run — but is false on cancellation. A shard that trips timeout-minutes is not "cancelled" in this sense, so timeout artifacts survive. Not fixed here: the maestro bash harness does not trap SIGINT/SIGTERM, so a shard mid-flow still runs to the runner's force-kill. Cancellation is much faster, not instant.
YogendraShelke
force-pushed
the
update-screenshot-11.12
branch
from
July 27, 2026 12:47
062e66f to
238a920
Compare
…mits The Android IntroScreen flow failed at its last assertion: after re-entering the page, NEXT moved slide 2 -> 3 and the attribute correctly became 3, then immediately flipped to 1, so "Active slide: 3" was never visible. The failure video shows the value on screen going 2 -> 3 -> 1 with no input in between. Cause: once the programmatic scroll to slide 3 settles, Android's list emits a SECOND onScrollBeginDrag + onMomentumScrollEnd pair, and that momentum event carries the offset the list moved AWAY from (0). The existing pendingScrollTarget guard had already been cleared by the first, legitimate momentum end, and wasUserDragging was true because the settling list — not a finger — began the drag. So the event was treated as a swipe back to slide 1 and rewrote the attribute. This is the same class of bug as the earlier phantom-momentum fix, one step later in the sequence. Fix: remember the slide a programmatic scroll settled on, and ignore a momentum end that moves more than one page from activeIndex while that memory is set. The list is pagingEnabled, so a genuine drag moves exactly one page; a multi-page jump right after a scroll settled can only be that scroll's tail. One-page swipes, including a swipe straight after a programmatic scroll, still report normally (covered by existing tests). Regression test added; it fails with setValue receiving [3, 1] and a spurious onSlideChange(0, 2) when the guard is removed. Revert "fix(e2e): scroll the alphabet-index target into view before tapping it" This reverts commit 31e9fd78c2fb70fb0e88b33b2d9bf8a983c4db51. fix(e2e): scroll the alphabet-index target into view before tapping it FloatingActionButton.yaml failed on iOS with Tap on "F"... COMPLETED Assert that "Floating action button" is visible... FAILED then passed on retry with the same build. The flow already had the 10s extendedWaitUntil added in 20d3d52, and that wait is what timed out — so this is not the render race that commit fixed; nothing was going to change in second 11 and a longer timeout would not have helped. extendedWaitUntil: visible requires the element to be IN THE VIEWPORT, not merely present in the hierarchy. Tapping the index scrolls to the letter's section, and under F the entries are Feedback then Floating action button; if the scroll settles with Feedback at the bottom edge, the target is rendered just below the fold and `visible` stays false forever. The retry's scroll landed a few pixels differently and it passed. Use scrollUntilVisible instead: it succeeds immediately when the element is already in view and scrolls to it when it is not, so it covers both the original render race and the off-screen case. centerElement keeps the subsequent tap away from the screen edge. Applied to the 55 flows whose index letter hosts more than one widget entry (B, C, F, I, L, P, R, S) — the case where the target can be pushed below the fold. The 16 flows on a sole-entry letter keep the cheaper wait. Verified: all 82 flows still parse, all 71 index taps remain guarded (55 scrollUntilVisible + 16 extendedWaitUntil, 0 unguarded), and the pinned 2.6.1 binary accepts every field used — confirmed it rejects an unknown property, so that parse is a real schema check. ci(e2e): archive maestro's debug output for failed flows Maestro writes a debug bundle per run — failure screenshot, maestro.log (which records the element Bounds each selector matched) and commands-<flow>.json — to ~/.maestro/tests/<timestamp>/. That is outside the workspace, so none of it was ever uploaded, and it is exactly what distinguishes "element never rendered" from "element rendered off-screen below the fold" when an assertVisible/extendedWaitUntil fails. Without it those failures can only be diagnosed by inference. Redirect it into the workspace with --debug-output (+ --flatten-debug-output, which its own help text recommends for CI) one dir per flow, and archive it. Kept only for FAILED flows, mirroring the existing video policy, so a green shard uploads nothing — each bundle is ~300KB. Also gitignore maestro/debug and maestro/videos: both are produced into the workspace by local runs too. The existing *.log rule covered maestro.log but not the bundle's screenshots or commands JSON. Verified --debug-output and --flatten-debug-output against the pinned 2.6.1 binary, and drove run_tests with a stub to confirm a failing flow's bundle is kept while a passing flow's is discarded. fix(maestro): recover from iOS driver/accessibility faults instead of hanging A shard could die with, and never recover from: Exception in thread "main" UnknownFailure(errorResponse=Request for viewHierarchy failed, code: 500, body: {"errorMessage":"Error getting element frame kAXErrorInvalidUIElement","code":"internal"}) Traced through the pinned 2.6.1 jars, this is unrecoverable by design. XCTestDriverClient.handleExceptions maps a 500 to a typed error only for four known message strings and otherwise throws UnknownFailure; XCTestIOSDevice.execute catches only AppCrash/OperationTimeout/Unreachable, and IOSDriver.runDeviceCall catches only the matching IOSDeviceErrors. UnknownFailure matches none, so it escapes to main — hence "Exception in thread main" rather than a failed step, and hence a flow-level `retry:` block cannot help: the CLI is already dead, no command failed. kAXError* means the accessibility element reference is stale, so every later query against that driver fails identically. Recovery therefore has to live in the shell. Add run_maestro(), which: - runs `maestro test` under a wall-clock watchdog (MAESTRO_FLOW_TIMEOUT, 480s) so a JVM that died in main but is held open by non-daemon threads can't eat the 30-minute job budget, and - classifies a FAILED run as infrastructure (exit 90) when the output carries a driver/accessibility signature, so callers can tell "the device broke" from "the widget is wrong". run_tests then resets the device and gives the flow one clean attempt with a freshly reinstalled driver, instead of recording a widget failure and marching every remaining flow through the same wedged driver. All three call sites (flows, smoke check, retries) now go through run_maestro, and the duplicated device-reset branches collapse into reset_device(). Only non-zero exits are reclassified, so a flow that passed after Maestro's own internal retry logged such an error still counts as a pass. fix(maestro): actually apply the iOS driver startup timeout MAESTRO_DRIVER_STARTUP_TIMEOUT was passed to maestro as `--env`, but LocalXCTestInstaller.getStartupTimeout() reads it with System.getenv() — `--env` only fills Maestro's per-flow variable map, which that function never consults. So every value we passed was silently ignored and the Kotlin runCatching kept falling back to its built-in 120000 default. Export it instead (and raise it to 240000), and drop the three no-op `--env MAESTRO_DRIVER_STARTUP_TIMEOUT` arguments. Also stop reinstalling the XCUITest driver once per flow. Maestro's TestCommand.reinstallDriver defaults to true, so each `maestro test` process runs restartXCTestRunner (uninstall + install + start) — and that install is the window that throws IOSDriverTimeoutException on a 3-vCPU macOS runner. The smoke check still installs the driver fresh at the start of every shard and retries still reinstall via restart_simulator, so only the first-pass flows reuse it. MAESTRO_REUSE_DRIVER=false restores the old behaviour.
`mktemp -t maestro-run` works on BSD/macOS, which appends the random suffix itself, but GNU coreutils requires at least 6 X's in the template. So on the Linux Android runners every flow printed mktemp: too few X's in template 'maestro-run' '': No such file or directory (os error 2) — log_file was empty, so the tee redirection failed and maestro never ran. iOS was unaffected, which is why this passed local and macOS testing. Use an explicit XXXXXX template, which both implementations accept, and fall back to a PID-based path if mktemp fails for any other reason: artifact plumbing must never be what decides whether the tests run. Audited the rest of the commands added in this series (tr -c, grep -qE) — both POSIX, and already used by the pre-existing recording code that runs on Android. Verified by stubbing a GNU-strict mktemp on PATH: reproduces the CI error with the old form and run_maestro completes with the new one. fix(intro-screen-native): catch the trailing drag whatever order it arrives in The Android e2e flow failed deterministically (both the first attempt and the retry) at `Assert that "Active slide: 3" is visible`. The CI video shows the widget reach slide 3, then snap back to slide 1 and increment the change counter, so the trailing drag the previous guard was written for still got through. It got through because the guard required evidence that a programmatic scroll had landed (settledScrollTarget), and that evidence is not always recorded: the settling list can emit its trailing scrollBeginDrag BEFORE the scroll's own momentum end, and onScrollBeginDrag clears the pending target, so the branch that would have set settledScrollTarget never runs. The guard was then skipped and the stale offset was treated as a swipe. Drop the bookkeeping and rely on the geometry instead. The list is pagingEnabled, so a drag settles at most one page from where it began; a momentum end further away than that cannot be a finger regardless of what came before it. Covered by a test reproducing the CI ordering, which fails without the change.
YogendraShelke
force-pushed
the
update-screenshot-11.12
branch
from
July 27, 2026 17:43
cb57696 to
2ea757e
Compare
The ios-app job paid npm i + pod install + xcodebuild on every run, even for a
widget-only change that cannot affect the native app. That is the most expensive
step in the pipeline and it runs on a scarce macOS runner.
It is safe to skip because the Mendix JS bundle is a RESOURCE of the .app rather
than compiled input: in nativeTemplate.xcodeproj the `Bundle` entry is a folder
reference (lastKnownFileType = folder) in the Resources phase, there is no
react-native-xcode.sh phase (unlike a stock RN app the template never builds a
bundle itself, it only copies what mxbuild produced), and AppDelegate resolves it
at launch through MendixAppDelegate.getJSBundleFile(). Widget .mpks are compiled
into the JS bundle by mxbuild, never into the iOS binary. So on a hit we restore
the previous .app, replace its bundle folder, and skip the build entirely; the
job's uploaded artifact is unchanged, so prepare_ios.sh and the test shards are
untouched.
Two fixes to the caching this replaces, which cached DerivedData but still built
every time:
- The old key hashed ios/**/*.swift and Podfile.lock but not
native_dependencies.json. add-native-dependencies.js merges that file into
native-template/package.json and the Podfile autolinks from there, so
`pod install` REGENERATES Podfile.lock — the committed lock is not the
effective one and a new native dependency did not change the key.
- restore-keys: ${{ runner.os }}-ios-build- restored the newest entry whatever
changed, mixing stale products with fresh pods. A partial-input change must
rebuild, so the new key has no restore-keys.
The key also pins the resolved native-template commit (nt_branch can be a moving
branch, so the name is not a content identity) and the Xcode version, since the
runner image updates it independently of our inputs. A hit that does not contain
the exact .app prepare_ios.sh installs, with a bundle inside it, warns and falls
back to a full build rather than shipping a half-swapped app.
Caches only the Products dir, not the DerivedData root: it is what we upload and
a fraction of the size, so it does not evict the AVD/dist caches.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes three defects in IntroScreen's slide-position handling, all rooted in how SwipeableContainer reconciled activeIndex (React state) with the FlashList's real scroll offset. The previous code used an isInitializing flag flipped by a double requestAnimationFrame, which guessed at timing instead of tracking what was actually requested.
Problems Fixed
Problem 1
Reopening a page showed slide 1 instead of the slide in the Active slide attribute, and fired a spurious On slide change back to slide 1
Root cause
activeIndex initialized to 0 while initialScrollIndex rendered the attribute's slide, so state and screen disagreed on mount. The effect only re-scrolled when slide !== activeIndex, so a matching-but-wrong state was never
Fix
Seed activeIndex from the attribute via a useState initializer, and force the positioning scroll once on the first pass where width is known (hasAppliedInitialScroll), independent of whether activeIndex already matches
Problem 2
The initial slide was reached with an animation, briefly leaving slide content unreachable for TalkBack/VoiceOver
Root cause
goToSlide always animated. activeIndex applies immediately and drives accessibilityElementsHidden / importantForAccessibility, so the slide exposed to screen readers was off-screen until the animation landed
Fix
goToSlide(pageNum, animated = true); the initial jump passes animated: false
Problem 3
Using Next / Previous / pagination could snap back to slide 1 and report an extra slide change
Root cause
onMomentumScrollEnd fires for programmatic scrolls too, reporting the offset the list is leaving rather than the one it's heading to. That stale offset overwrote activeIndex and re-fired onSlideChange. A drag flag alone can't disambiguate: it's only cleared inside onMomentumScrollEnd, so a drag that produces no momentum event leaves it stuck true, and the next programmatic scroll's stale event is misread as a user swipe
Fix
Track pendingScrollTarget — the slide goToSlide actually requested. onMomentumScrollEnd ignores any event whose index disagrees with that target, clearing it once reached. onScrollBeginDrag clears the target so a user taking over is always honoured, which also prevents the target latching when a scroll emits no momentum event