Skip to content

Commit aa4d46b

Browse files
obj-pclaude
obj-p
and
claude
authored
tests+ci: regress guard tranche 3 — tool rows P01/B01/B02/B03/F01/L04 (#437)
* tests+ci: regress guard tranche 3 — tool rows P01/B01/B02/B03/F01/L04 Adds RegressToolGuardTests, a manual-tagged sibling swift_test compiling the shared CLIIntegrationTests harness, for the regress-matrix rows that need tooling the required gate's deterministic target must not depend on: generator scripts (P01 cold-build heartbeat, B02/B03 XCFramework renders, F01 classified slice error), a Bazel fetch of the fixture's pinned rules (B01, via an explicit PATH pin — bazelisk is invisible in bazel's scrubbed test env), and an iOS simulator (L04's deterministic simctl-terminate crash/respawn trigger, dedicated device index 9). Tool preconditions skip locally but fail under PREVIEWSMCP_REQUIRE_DEDICATED_SIM so CI can never silently skip a row. The new non-required regress-tools job in ci.yml runs the target on the same labels (needs: ci keeps the serial runner working the required gate first) to build a standing flake record; promotion into the required gate is dropping the manual tag. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * tests: fold gate findings — shared row asserts, one artifact pass, gate accessor /simplify fold: assertRenders/assertFails/cleanSlate/fixture move to RegressRowAsserts (compiled into both guard targets via HARNESS_SRCS) so the two suites cannot drift; the three binary-framework rows share one per-process artifact generation; the fail-on-gate-vs-skip-locally policy gets its general name (RequiredGateEnforcement.enforced) instead of riding the sim-specific flag, which now delegates to it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * tests+ci: review fixes — flock release on gate throw, job timeout margin Adversarial review fold: provisionSimulator releases the host-wide sim flock on the RequiredDeviceUnavailable throw path (a leaked LOCK_EX self-deadlocks the remaining iOS rows for 10 minutes each); regress-tools timeout-minutes rises to 90 so a hung test hits bazel's 3600s timeout (step failure -> diagnostics step) before job cancellation would skip it; doc notes for the artifact task's .serialized dependency and CoreSimulatorHygiene's actual lock contract. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
1 parent 3f10884 commit aa4d46b

8 files changed

Lines changed: 552 additions & 101 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -191,3 +191,94 @@ jobs:
191191
pkill -9 -f '\.xctest' || true
192192
killall -9 Simulator || true
193193
xcrun simctl shutdown all || true
194+
195+
# Non-required signal for the regress-matrix rows that need fixture artifact
196+
# generation, a Bazel fetch of a fixture's pinned rules, or an iOS simulator
197+
# (//previewsmcp/Tests/CLIIntegrationTests:RegressToolGuardTests, tagged
198+
# `manual` so the required `ci` job's `//...` expansion never runs it). Runs
199+
# on the same triggers to build a standing flake record; promote into the
200+
# required gate by dropping the `manual` tag once that record is clean. This
201+
# check is NOT in the ruleset's required contexts — a red here never blocks
202+
# auto-merge. `needs: ci` keeps the single serial runner working the required
203+
# gate first, so this job never delays a merge.
204+
regress-tools:
205+
needs: ci
206+
if: >-
207+
github.event_name != 'pull_request' ||
208+
github.event.label.name == 'ci' ||
209+
github.event.label.name == 'merge'
210+
runs-on: [self-hosted, macOS, ARM64]
211+
# Above the target's `eternal` (3600s) bazel test timeout, like `ci`'s
212+
# 120 vs its heaviest target: a hung test must hit bazel's timeout (a
213+
# step failure(), which runs the diagnostics step) before the job-level
214+
# cancellation, which would skip it.
215+
timeout-minutes: 90
216+
permissions:
217+
contents: read
218+
checks: read
219+
env:
220+
DEVELOPER_DIR: /Applications/Xcode-26.2.0.app/Contents/Developer
221+
# Same required-gate coverage signal as `ci`: a host that cannot
222+
# provision the dedicated simulator (or is missing bazel) FAILS the row
223+
# instead of silently skipping it (RequiredGateEnforcement.enforced in
224+
# TestSupport, read by RegressToolGuardTests.requireTool and the sim
225+
# resolver).
226+
PREVIEWSMCP_REQUIRE_DEDICATED_SIM: "1"
227+
steps:
228+
# Dedup like `ci`: a `merge` label after a green `ci`-label run skips
229+
# the second identical pass.
230+
- name: Skip suite if this commit is already green
231+
id: dedup
232+
if: github.event_name == 'pull_request' && github.event.label.name == 'merge'
233+
env:
234+
GH_TOKEN: ${{ github.token }}
235+
SHA: ${{ github.event.pull_request.head.sha }}
236+
run: |
237+
if gh api "repos/$GITHUB_REPOSITORY/commits/$SHA/check-runs" \
238+
--jq '.check_runs[] | select(.name=="regress-tools" and .conclusion=="success") | .id' \
239+
| grep -q .; then
240+
echo "skip=true" >> "$GITHUB_OUTPUT"
241+
echo "A green regress-tools check already exists for $SHA — skipping the suite."
242+
fi
243+
244+
- if: steps.dedup.outputs.skip != 'true'
245+
uses: actions/checkout@v7
246+
247+
# Same clean-host reset as `ci` — this job boots simulators too.
248+
- name: Reset simulator and daemon state
249+
if: steps.dedup.outputs.skip != 'true'
250+
run: |
251+
pkill -9 -f 'previewsmcp serve' || true
252+
pkill -9 -f 'previewsmcp run' || true
253+
pkill -9 -x PreviewAgent || true
254+
pkill -9 -f '\.xctest' || true
255+
killall -9 Simulator || true
256+
xcrun simctl shutdown all || true
257+
launchctl kickstart -k "gui/$(id -u)/com.apple.CoreSimulator.CoreSimulatorService" 2>/dev/null \
258+
|| launchctl kickstart -k "system/com.apple.CoreSimulator.CoreSimulatorService" 2>/dev/null || true
259+
xcrun simctl delete unavailable || true
260+
261+
- name: Test (regress tool-guard rows)
262+
if: steps.dedup.outputs.skip != 'true'
263+
run: >-
264+
bazel test //previewsmcp/Tests/CLIIntegrationTests:RegressToolGuardTests
265+
--test_output=errors
266+
267+
- name: Failure diagnostics
268+
if: failure()
269+
run: |
270+
xcrun simctl list devices booted || true
271+
pgrep -fl "previewsmcp|simctl" || true
272+
for f in "${TMPDIR:-/tmp}"previewsmcp-test-*.log /tmp/previewsmcp-test-*.log; do
273+
[ -f "$f" ] && { echo "=== $f (tail)"; tail -100 "$f"; }
274+
done || true
275+
276+
- name: Reap leaked processes and simulators
277+
if: always()
278+
run: |
279+
pkill -9 -f 'previewsmcp serve' || true
280+
pkill -9 -f 'previewsmcp run' || true
281+
pkill -9 -x PreviewAgent || true
282+
pkill -9 -f '\.xctest' || true
283+
killall -9 Simulator || true
284+
xcrun simctl shutdown all || true

‎docs/ci.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,19 @@ tier failing still blocks the merge through the same required `ci` check. The
3333
`integration` tag is only a selection label; existing `exclusive` tags and
3434
runtime simulator locks continue to control scheduling and isolation.
3535

36+
## The non-required `regress-tools` signal
37+
38+
A second job, `regress-tools`, runs
39+
`//previewsmcp/Tests/CLIIntegrationTests:RegressToolGuardTests` — the
40+
regress-matrix guard rows that need fixture artifact generation, a Bazel fetch
41+
of a fixture's pinned rules, or an iOS simulator. The target is tagged
42+
`manual`, so neither required tier's `bazel test //...` expansion ever runs
43+
it; only this job does. The check is deliberately NOT in the ruleset's
44+
required contexts: it runs on the same labels to build a standing flake
45+
record, and a red never blocks auto-merge. `needs: ci` keeps the single serial
46+
runner working the required gate first. Promote the target into the required
47+
gate by dropping its `manual` tag once the record is clean.
48+
3649
## Normal flow
3750

3851
1. Open the PR. Nothing runs.

‎previewsmcp/Tests/CLIIntegrationTests/BUILD.bazel‎

Lines changed: 57 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,22 @@
11
load("@rules_swift//swift:swift.bzl", "swift_test")
22

3+
# Shared subprocess/daemon harness. CLIIntegrationTests picks these up via
4+
# its glob; RegressToolGuardTests compiles them again as its own sources.
5+
HARNESS_SRCS = [
6+
"CLIRunner.swift",
7+
"CoreSimulatorHygiene.swift",
8+
"DaemonTestLock.swift",
9+
"PipeBuffer.swift",
10+
"RegressRowAsserts.swift",
11+
]
12+
313
swift_test(
414
name = "CLIIntegrationTests",
515
timeout = "eternal",
6-
srcs = glob(["**/*.swift"]),
16+
srcs = glob(
17+
["**/*.swift"],
18+
exclude = ["RegressToolGuardTests.swift"],
19+
),
720
copts = [
821
"-swift-version",
922
"6",
@@ -34,3 +47,46 @@ swift_test(
3447
visibility = ["//visibility:public"],
3548
deps = ["//previewsmcp/Tests/TestSupport"],
3649
)
50+
51+
# Guards for regress-matrix rows that need build tooling the deterministic
52+
# target above must not depend on: fixture artifact generation, a Bazel
53+
# fetch of the fixture's pinned rules, or an iOS simulator. Tagged `manual`
54+
# so `bazel test //...` (both required-gate tiers) never expands to it; the
55+
# non-required `regress-tools` job in .github/workflows/ci.yml names it
56+
# explicitly. Promote into the required gate by dropping `manual` once its
57+
# flake record is clean.
58+
swift_test(
59+
name = "RegressToolGuardTests",
60+
timeout = "eternal",
61+
srcs = HARNESS_SRCS + ["RegressToolGuardTests.swift"],
62+
copts = [
63+
"-swift-version",
64+
"6",
65+
],
66+
data = [
67+
"//:ios_jit_resources",
68+
"//:orc_rt_archive",
69+
"//previewsmcp/PreviewAgent",
70+
"//previewsmcp/cli:previewsmcp",
71+
],
72+
env = {
73+
"PREVIEWSMCP_AGENT": "$(rlocationpath //previewsmcp/PreviewAgent)",
74+
"PREVIEWSMCP_BINARY": "$(rlocationpath //previewsmcp/cli:previewsmcp)",
75+
"PREVIEWSMCP_IOS_JIT_DIR": "$(rlocationpath //:ios_jit_resources)",
76+
"PREVIEWSMCP_ORC_RT": "$(rlocationpath //:orc_rt_archive)",
77+
# Bazel scrubs PATH to /bin:/usr/bin:/usr/local/bin in test actions,
78+
# which hides Homebrew's bazelisk from the CLI's `/usr/bin/env bazel`
79+
# (the reason SnapshotCommandTests' bazel test tool-gate skips under
80+
# `bazel test`). B01 must not skip, so this target pins an explicit
81+
# PATH that includes /opt/homebrew/bin.
82+
"PATH": "/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin",
83+
},
84+
tags = [
85+
"exclusive",
86+
"integration",
87+
"local",
88+
"manual",
89+
],
90+
visibility = ["//visibility:public"],
91+
deps = ["//previewsmcp/Tests/TestSupport"],
92+
)

‎previewsmcp/Tests/CLIIntegrationTests/CoreSimulatorHygiene.swift‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -16,10 +16,11 @@ import os
1616
/// shut every simulator down and bounce CoreSimulatorService so the next boot
1717
/// starts from a clean service. Call while holding `SimulatorTestLock` (which
1818
/// serializes sim-booting tests host-wide, so this reset cannot kill another
19-
/// workspace's live run — #336) and `DaemonTestLock`, which serializes the
20-
/// sim-touching suites WITHIN this target (its flock path is per-target);
21-
/// across targets within one invocation the guard is this target's
22-
/// `exclusive` tag.
19+
/// workspace's live run — #336) and with no live daemon session in this
20+
/// process — either inside `DaemonTestLock.run` (whose teardown fences the
21+
/// daemon dead before releasing) or between run blocks, where that fence
22+
/// has already run; across targets within one invocation the guard is this
23+
/// target's `exclusive` tag.
2324
enum CoreSimulatorHygiene {
2425
private static let didReset = OSAllocatedUnfairLock(initialState: false)
2526

0 commit comments

Comments
 (0)