Skip to content

Unflake the iOS/JIT/MCP sim tests that flip under VM load #296

Description

@obj-p

Context

The Bazel merge-queue bar (chunk D) runs bazel test //... in a macOS VM. A small set of sim/JIT tests flip pass/fail between runs under VM scheduling jitter — different tests fail each run, which is flakiness, not regressions. The bar now masks this with --flaky_test_attempts=3 (commit on mergequeue-vz-kit); this issue tracks fixing the tests so the retries can eventually be removed.

Observed flaky tests (one or two per run, varying)

  1. MCPIntegrationTests — daemon-hosted app server / preview_snapshot: timed out waiting for the first streamed frame (30s). Improved by Fix #292: poll preview_elements for render-readiness instead of a fixed sleep #295 (render-readiness polling) and the 12-vCPU bump, but still timing-sensitive.
  2. PreviewsJITLinkTests:IOSPreviewE2ETests.literalEditReSeedsSameObjectWithoutRecompile: a literal-edit hot reload where the snapshot still showed the old string — a propagation race between the edit and the capture.
  3. PreviewsJITLinkTests:PreviewsJITLinkTests.perEditRecompileIsBelowWholeModuleCompile: a performance assertion (splitMs < wholeMs) that loses under VM load. Comparing two compile timings is inherently noisy in a shared VM.
  4. Earlier (pre-display-fix) bridgeRecordsWindowFrameOnMoveAndResize failed due to the VM's retina logical size; fixed by dropping the display to 110ppi. Listed for completeness — likely resolved.

Directions

  • Propagation/timing tests (1, 2): replace remaining fixed waits with readiness polling (as Fix #292: poll preview_elements for render-readiness instead of a fixed sleep #295 did for preview_elements); poll the rendered content for the expected value before asserting, with a generous bound.
  • Perf assertions (3): these do not belong as hard gates in a shared VM. Either widen the threshold, convert to informational (log the numbers, do not assert ordering), or tag them to skip in the VM gate. Track the jit_latency numbers as a trend instead of a pass/fail.
  • Once the tests are deterministic in the VM, drop --flaky_test_attempts from the bar.

Found while landing the Bazel bar (chunk D). Related: #293 (cache), #294 (parallel sandboxes).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions