Commit fd69244
authored
fix(ci): improve
* fix(ci): build Windows + macOS mypyc wheels natively; harden smoke install
v0.47.0 release (run 26351509901) failed every Windows + macOS test-wheels
job because `build-wheels-mypyc` ran on ubuntu-latest, where cibuildwheel
silently ignores `CIBW_ARCHS_WINDOWS` / `CIBW_ARCHS_MACOS`. Only Linux
wheels have ever been published; the test step then fell back to PyPI
(`uv pip install --find-links` permits index fallback when no local wheel
matches), installed pure-Python 0.46.3, and `mypyc_smoke.py --require-compiled`
(added in PR #446) correctly reported every module as interpreted and exit 1.
Changes
- publish.yml: split build-wheels-mypyc into a per-OS matrix
(ubuntu × macos × windows) × (3.10-3.14). QEMU gated to Linux only.
Linux + macOS keep three-stage PGO; macOS PGO blocks finally execute.
Windows runs plain mypyc (MSVC PGO is not validated here).
Artifacts now named `wheels-mypyc-<os>-py<ver>`.
- publish.yml + test-build.yml test-wheels: install with `--no-index`
so missing wheels fail loudly. Assert `sqlspec.__version__` matches
the release tag (or pyproject.toml on PRs) before running smoke.
- test-build.yml: mirror the per-OS matrix; PR-default (`test_matrix=subset`)
stays ubuntu-only for fast feedback; `workflow_dispatch test_matrix=full`
exercises all 15 build legs + 9 test legs.
- All workflows: enable `setup-uv@v7` cache keyed on `uv.lock`.
- publish.yml + test-build.yml: cache pip downloads + the pinned
`HATCH_MYPYC_BUILD_DIR` per (OS, python). PGO profile data is not cached.
- ci.yml: cache the pre-pulled docker image tarball
(`docker save`/`docker load`) keyed on the ci.yml hash.
Beads: sqlspec-tt2h (epic), sqlspec-8xun (T1) / sqlspec-1jv0 (T2)
/ sqlspec-2xh7 (T3) / sqlspec-qr1u (T4) / sqlspec-b5mq (T5)
/ sqlspec-v9mb (T6)
* fix(ci): fall back to pyproject.toml for publish.yml version assert
`Resolve expected version` reads `github.event.release.tag_name`, which is
empty when publish.yml is triggered via workflow_dispatch. The empty value
made the install string `sqlspec==` (malformed) and broke manual reruns.
Fall back to the version in pyproject.toml when no release tag is present.
The `release: published` path is unchanged.
* ci: auto-promote test-build.yml to full matrix on build-infra PRs
Subset mode (linux x py3.12) stays the default for fast PR feedback, but a
preflight detect-mode job now promotes the build + test matrix to full
(3 OS x 5 py builds, 3 OS x 3 py tests) when the PR diff touches any of:
.github/workflows/*.yml
pyproject.toml
uv.lock
tools/scripts/pgo_training.py
tools/scripts/mypyc_smoke.py
That's the exact set of files where a matrix-shape regression like the
0.47.0 release (Linux-only wheels published) could land. workflow_dispatch
test_matrix=full still works as an explicit override.
* fix(ci): use {project} placeholder in CIBW_BEFORE_BUILD on macOS
cibuildwheel runs Linux builds inside a container that mounts the source
at /project. macOS and Windows run natively on the host with no such
mount, so `/project/build-constraints.txt` and `/project/tools/...` do
not exist there. The macOS PGO BEFORE_BUILD block had four such literals;
it was latent until the per-OS matrix split started actually executing
on a macOS runner, where pip would fail with "Could not find file".
Switch every `/project/` inside CIBW_BEFORE_BUILD_{LINUX,MACOS} to the
cibuildwheel `{project}` placeholder, which resolves to /project inside
the Linux container and to the host project path on macOS. Same change
mirrored into test-build.yml.
The PyPI publish URL (https://pypi.org/project/sqlspec/) is unrelated
and untouched.
* fix(ci): drop PGO from macOS + Windows; Linux keeps three-stage PGO
macOS PGO consistently failed at `llvm-profdata merge` because
pgo_training.py's workloads report "filters not installed" (the script's
optional filter deps aren't present in the cibuildwheel BEFORE_BUILD env)
→ instrumented binary never emits .profraw files → merge has no input.
Rather than debug the filter import path, drop PGO from macOS + Windows:
both now use the base CIBW_BEFORE_BUILD (install hatch-mypyc + hatchling)
and a plain mypyc env with HATCH_BUILD_HOOKS_ENABLE + MYPYC_* flags.
macOS still pins HATCH_MYPYC_BUILD_DIR so the mypyc compile cache
(actions/cache, T5) still hits across runs.
Linux retains the full three-stage PGO build unchanged — that's where
the perf benefit measured in CI is and stays.
* fix(ci): unpin macOS mypyc build dir; pre-warm Defender on Windows
macOS:
hatch_mypyc.plugin.initialize() calls os.mkdir(BUILD_DIR/build) — not
os.makedirs — so the parent must already exist. The Linux PGO
BEFORE_BUILD creates it explicitly; the (now removed) macOS PGO
BEFORE_BUILD did the same. With both gone, pinning
HATCH_MYPYC_BUILD_DIR=/tmp/sqlspec-mypyc-build pointed hatch-mypyc at
a missing parent and the build crashed with FileNotFoundError. Unpin
the var — hatch-mypyc falls back to its default per-project temp dir,
which it creates as needed. Also drop macOS from the actions/cache
step since the pinned path is no longer used there.
Windows:
virtualenv's interpreter discovery times out at ~10s on the freshly
extracted nuget-cpython python.exe because Windows Defender real-time
protection scans the binary on its first invocation. Add Defender
exclusions for cibuildwheel's cache dir, the runner temp dir, and
python.exe via a Windows-only pre-step. -ErrorAction SilentlyContinue
so future tamper-protection changes don't break the job.
* ci: trim mypyc wheel matrix to python 3.12 across all 3 OSes
15 build legs (3 OS × 5 py) was too much CI footprint for an initial
release that's first turning on Windows + macOS. Trim to 3 build legs
(linux py3.12, macos py3.12, windows py3.12) + 3 test legs. Users on
other Python versions install the pure-Python sqlspec-*-py3-none-any.whl
fallback that build-wheels-standard still produces.
Same trim mirrored into test-build.yml (full mode is now identical to
subset's py3.12 baseline for python-version; only the os list differs).
Expand back to 3.10–3.14 once py3.12 is known stable across all 3 OSes.
* ci: split matrices — full sweep on release, single-py on PR (configurable)
publish.yml (release):
Restore full sweep — 3 OS × 5 py for builds (3.10–3.14), 3 OS × 3 py
for test-wheels (3.10/3.12/3.14 sampling matches the pre-trim shape).
Every published release exercises every supported Python.
test-build.yml (PR / workflow_dispatch):
Stay on a single Python version per run (default 3.12). Add a new
`python_version` workflow_dispatch input with a choice of 3.10–3.14
for manual overrides when you want to validate a specific interpreter.
Build + test matrices read the input, falling back to '3.12' for
pull_request triggers (where workflow_dispatch inputs are empty).
detect-mode's OS auto-promotion (linux subset vs full 3-OS) is
unchanged — orthogonal to the python-version choice.
* ci: cancel in-flight runs on the same ref to stop wasting runner minutes
Pre-existing gap (main never had concurrency either): pushing a new
commit to a PR branch left the previous Tests And Linting + Test Build
Configuration runs queued/running alongside the new ones. With several
quick pushes during a debugging session that compounded into 3–5
parallel runs per workflow per branch.
Add concurrency: { group: ${{ github.workflow }}-${{ github.ref }},
cancel-in-progress: true } to ci.yml, test-build.yml, docs.yml.
NOT applied to publish.yml: a release upload mid-cancel could leave
PyPI in a partial state (some wheels uploaded, others not). Releases
stay all-or-nothing as before — see the `needs:` chain on
publish-release. pgo-validate.yml and pr-title.yml left alone too
(manual-only / quick lint, low impact).
* ci(windows): exclude wheelhouse + workspace from Defender
Defender's real-time scan was holding the freshly-built .whl file handle
when actions/upload-artifact@v7 opened it for chunked upload, causing
the step to silently terminate after pre-flight validation with no
error logged. macOS used the same action version and succeeded.
Extend the pre-warm exclusion list to cover GITHUB_WORKSPACE and the
wheelhouse output directory so the artifact file isn't being scanned
when the action reads it.
* ci(test-build): drop aarch64 from PR validation matrix
aarch64 cross-build under QEMU + PGO three-stage hangs for >1h per leg
on PR runners. publish.yml already covers the full cross-arch matrix
at release time, so PR validation is pure duplication.
Strip QEMU setup and force host-arch-only builds: x86_64 on Linux,
arm64 on macos-latest (now Apple Silicon native), AMD64 on Windows.
* ci(publish): build aarch64 on native ARM runner instead of QEMU
The aarch64 cross-build under QEMU + PGO three-stage hangs for hours
per leg, putting every Linux publish job at risk of the 6-hour cap.
GitHub now offers ubuntu-24.04-arm (native ARM runner, free for
public repos), making QEMU emulation unnecessary.
Add ubuntu-24.04-arm to the build matrix and make CIBW_ARCHS_LINUX
conditional on which runner is executing. Drop the QEMU setup step.
Drop macOS x86_64 (macos-latest is Apple Silicon native; x86_64
would need Rosetta and isn't worth the build minutes — pure-Python
wheel covers Intel Macs). Add ubuntu-24.04-arm to test-wheels so
the ARM wheel is actually validated before publish.
Build matrix grows from 15 → 20 legs but each leg now runs at native
speed (~10–20 min) instead of QEMU-emulated hours.
* ci(test-wheels): resolve transitive deps from PyPI
Splitting the install into two steps preserves the no-silent-fallback
guarantee for sqlspec itself while letting runtime deps like
mypy-extensions, sqlglot, etc. resolve from PyPI.
Step 1: --no-index --no-deps --find-links — sqlspec wheel MUST come
from the local artifact bundle (this is the safety check).
Step 2: same install without --no-index — fills in transitive runtime
deps from PyPI; sqlspec itself is already installed and won't reresolve.mypyc build testing and failure detection (#461)1 parent af9c0ab commit fd69244
4 files changed
Lines changed: 302 additions & 74 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
9 | 16 | | |
10 | 17 | | |
11 | 18 | | |
| |||
14 | 21 | | |
15 | 22 | | |
16 | 23 | | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
17 | 27 | | |
18 | 28 | | |
19 | 29 | | |
| |||
41 | 51 | | |
42 | 52 | | |
43 | 53 | | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
44 | 57 | | |
45 | 58 | | |
46 | 59 | | |
| |||
58 | 71 | | |
59 | 72 | | |
60 | 73 | | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
61 | 77 | | |
62 | 78 | | |
63 | 79 | | |
| |||
75 | 91 | | |
76 | 92 | | |
77 | 93 | | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
78 | 97 | | |
79 | 98 | | |
80 | 99 | | |
| |||
98 | 117 | | |
99 | 118 | | |
100 | 119 | | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
101 | 123 | | |
102 | 124 | | |
103 | 125 | | |
104 | 126 | | |
105 | 127 | | |
106 | 128 | | |
107 | 129 | | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
108 | 141 | | |
| 142 | + | |
109 | 143 | | |
110 | 144 | | |
111 | 145 | | |
| |||
121 | 155 | | |
122 | 156 | | |
123 | 157 | | |
| 158 | + | |
| 159 | + | |
124 | 160 | | |
125 | 161 | | |
126 | 162 | | |
| |||
230 | 266 | | |
231 | 267 | | |
232 | 268 | | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
233 | 272 | | |
234 | 273 | | |
235 | 274 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
11 | 18 | | |
12 | 19 | | |
13 | 20 | | |
| |||
16 | 23 | | |
17 | 24 | | |
18 | 25 | | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
19 | 29 | | |
20 | 30 | | |
21 | 31 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
15 | 15 | | |
16 | 16 | | |
17 | 17 | | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
18 | 21 | | |
19 | 22 | | |
20 | 23 | | |
| |||
40 | 43 | | |
41 | 44 | | |
42 | 45 | | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
43 | 49 | | |
44 | 50 | | |
45 | 51 | | |
| |||
57 | 63 | | |
58 | 64 | | |
59 | 65 | | |
60 | | - | |
61 | | - | |
| 66 | + | |
| 67 | + | |
62 | 68 | | |
| 69 | + | |
63 | 70 | | |
| 71 | + | |
64 | 72 | | |
65 | 73 | | |
66 | 74 | | |
67 | 75 | | |
68 | 76 | | |
69 | | - | |
70 | | - | |
71 | | - | |
72 | | - | |
73 | | - | |
74 | 77 | | |
75 | 78 | | |
76 | 79 | | |
77 | 80 | | |
78 | 81 | | |
79 | 82 | | |
80 | 83 | | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
81 | 87 | | |
82 | 88 | | |
83 | 89 | | |
84 | 90 | | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
85 | 125 | | |
86 | 126 | | |
87 | 127 | | |
88 | 128 | | |
89 | 129 | | |
90 | 130 | | |
91 | | - | |
92 | 131 | | |
93 | 132 | | |
94 | 133 | | |
95 | | - | |
96 | | - | |
97 | | - | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
98 | 138 | | |
99 | 139 | | |
100 | | - | |
101 | 140 | | |
102 | 141 | | |
103 | | - | |
104 | | - | |
105 | | - | |
106 | | - | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
107 | 146 | | |
108 | | - | |
109 | 147 | | |
110 | 148 | | |
111 | | - | |
| 149 | + | |
112 | 150 | | |
113 | 151 | | |
114 | 152 | | |
115 | 153 | | |
116 | 154 | | |
117 | 155 | | |
118 | | - | |
| 156 | + | |
119 | 157 | | |
120 | 158 | | |
121 | 159 | | |
| |||
125 | 163 | | |
126 | 164 | | |
127 | 165 | | |
128 | | - | |
| 166 | + | |
129 | 167 | | |
130 | 168 | | |
131 | 169 | | |
| |||
135 | 173 | | |
136 | 174 | | |
137 | 175 | | |
138 | | - | |
139 | | - | |
140 | | - | |
141 | | - | |
142 | | - | |
143 | | - | |
144 | | - | |
145 | | - | |
146 | | - | |
147 | | - | |
148 | | - | |
149 | | - | |
150 | | - | |
151 | | - | |
152 | | - | |
153 | | - | |
154 | | - | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
155 | 183 | | |
156 | 184 | | |
157 | | - | |
158 | 185 | | |
159 | | - | |
160 | 186 | | |
161 | | - | |
| 187 | + | |
162 | 188 | | |
163 | 189 | | |
164 | 190 | | |
165 | 191 | | |
166 | 192 | | |
167 | 193 | | |
168 | 194 | | |
169 | | - | |
| 195 | + | |
170 | 196 | | |
171 | 197 | | |
172 | | - | |
173 | 198 | | |
174 | 199 | | |
175 | 200 | | |
176 | 201 | | |
177 | 202 | | |
178 | 203 | | |
179 | | - | |
| 204 | + | |
180 | 205 | | |
181 | 206 | | |
182 | 207 | | |
183 | 208 | | |
184 | 209 | | |
185 | 210 | | |
186 | 211 | | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
187 | 227 | | |
188 | 228 | | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
189 | 232 | | |
190 | 233 | | |
191 | 234 | | |
| |||
203 | 246 | | |
204 | 247 | | |
205 | 248 | | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
206 | 253 | | |
| 254 | + | |
207 | 255 | | |
208 | 256 | | |
209 | | - | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
210 | 260 | | |
211 | 261 | | |
212 | 262 | | |
| 263 | + | |
213 | 264 | | |
214 | 265 | | |
215 | | - | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
216 | 269 | | |
217 | 270 | | |
218 | 271 | | |
| |||
228 | 281 | | |
229 | 282 | | |
230 | 283 | | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
231 | 287 | | |
232 | 288 | | |
233 | 289 | | |
| |||
0 commit comments