chore: Mitigate iOS device test timeouts#5063
Conversation
#skip-changelog
jamescrosswell
changed the title
Semver Impact of This PR⚪ None (no version bump detected) 📋 Changelog PreviewThis is how your changes will appear in the changelog. This PR will not appear in the changelog. 🤖 This preview updates automatically when you update the PR. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #5063 +/- ##
==========================================
- Coverage 74.01% 74.00% -0.01%
==========================================
Files 499 499
Lines 18065 18065
Branches 3518 3518
==========================================
- Hits 13370 13369 -1
- Misses 3836 3841 +5
+ Partials 859 855 -4 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
![sentry-warden[bot]](https://avatars.githubusercontent.com/in/2755838?s=60&v=4){kind=small}
| return [PSCustomObject]@{ | ||
| Udid = $selected.Device.udid | ||
| Name = $selected.Device.name | ||
| DeviceType = $selected.Device.deviceTypeIdentifier | ||
| Runtime = $runtimeKey | ||
| } |
There was a problem hiding this comment.
Breaking change: Get-IosSimulatorUdid return type change breaks ios.Tests.ps1
The function Get-IosSimulatorUdid now returns a PSCustomObject instead of a string UDID. However, integration-test/ios.Tests.ps1 calls this function at line 11 and uses $simulator directly with xcrun simctl install, xcrun simctl uninstall, xcrun simctl spawn, and xcrun simctl launch commands which expect a string UDID. This will cause runtime failures in the iOS integration tests because PowerShell will coerce the object to a string representation instead of the actual UDID value.
Verification
Read scripts/device-test-utils.ps1 to see the return type change from string to PSCustomObject. Read integration-test/ios.Tests.ps1 which calls Get-IosSimulatorUdid at line 11 and uses $simulator directly at lines 43, 53-54, 68-69, 70-74 with xcrun simctl commands. Confirmed ios.Tests.ps1 is not in the list of files modified in this PR, meaning it won't be updated to handle the new return type.
Identified by Warden code-review · 5YQ-FP3
| Start-Sleep -Seconds 5 # give the daemon time to re-initialise | ||
|
|
||
| # Create a brand-new simulator with the same device type & runtime | ||
| $newUdid = (& xcrun simctl create $SimInfo.Name $SimInfo.DeviceType $SimInfo.Runtime).Trim() |
There was a problem hiding this comment.
Missing error check on xcrun simctl create can cause cascading failures with invalid UDID
Line 23 captures the output of xcrun simctl create but does not check $LASTEXITCODE. If the create command fails (e.g., invalid DeviceType/Runtime, or resource exhaustion), the error message text becomes the $newUdid value. Subsequent operations at lines 27-28 will attempt to boot this garbage string as a UDID, and line 32 returns it to the caller, causing hard-to-diagnose CI failures.
Verification
Read Reset-IosSimulator function completely. Traced flow from xcrun simctl create at line 23 through simctl boot at line 27 and return at line 32. Confirmed no $LASTEXITCODE check or try/catch around the create operation. Compared with other functions in the file which also lack such checks.
Identified by Warden find-bugs · 4YX-2ML
|
Admitting defeat... |
1 participant
Problem
The iOS device tests workflow is frequently timing out and failing in CI. ... e.g. this run:
After rerunning it (manually) once or twice it usually succeeds.
Attempted solution
This PR attemts to mitigate this by doing a couple of things:
xcrun simctl bootstatus -b, which waits until the simulator is truly operational (not just in "Booted" state). The errors show the simulator as already "booted" butxcrun simctl installstill hangs — so maybe the simulator was only partially initialized when running these tests previously.device-test.ps1, so the YAML needs just one step instead of a continue-on-error + duplicate retry pattern.The retry count has also been increased to 3 (from 2) and to account for this + the fact that the retries have been consolidated into thes ps1 script, the timeout has been increased from 40mins (for an individual attempt) to 120 minutes (for all 3 attempts combined). That's a massive timeout, but frequently if a human being wasn't monitoring the CI runs previously, it might be as much as 24 hours before someone manually reran the iOS device tests (something I've been doing personally many times a day across multiple different branches, with no small amount of swearing accompanying the chore).
In any case - 🤞🏻
#skip-changelog