Commit 30a7057
fix(docker): hermetic wheel closure for s390x/ppc64le multiplatform builds (#5287)
* fix(docker): full hermetic wheel closure for s390x/ppc64le multiplatform builds
Multiplatform Docker Build was red on main for s390x (alembic unsatisfiable —
the old s390x wheel image only carried curated native wheels, no pure-Python
alembic/tiktoken) and ppc64le (tiktoken source build failing, no wheel image
at all). amd64/arm64 are unaffected (PyPI manylinux wheels exist).
Replaces the curated native-wheel list with a full dependency closure derived
from uv.lock (uv export --no-default-groups), so Containerfile.lite's
--no-index install resolves every pure-Python and native package without
PyPI egress. Wheel images are content-addressed by sha256(uv.lock) and built
inline as a needs: dependency in docker-multiplatform.yml (pull-or-build,
idempotent), replacing the standalone s390x-wheels.yml nightly workflow.
Containerfile.lite's wheel stage is now arch-agnostic (WHEELS_REF build-arg,
empty for amd64/arm64) instead of unconditionally pulling a s390x-only image,
and the per-arch install branching collapses into one hermetic/fallback path.
Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>
* fix(docker): build job skipping entirely on PRs due to needs:wheels gate
GitHub's default needs condition requires success(), and treats a skipped
dependency the same as failed. The wheels job intentionally skips on
pull_request events, which was causing the entire build job (including
amd64, which is supposed to build on every PR) to skip too instead of just
the s390x/ppc64le wheel-dependent path.
Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>
* fix(docker): address PR #5287 review feedback on wheel image fallback
- Push a per-arch `latest` tag alongside the lock-hash tag for wheel
images, so a fallback target exists for both the build job and
local builds (brian-hussey).
- Decouple the `build` job from `needs: wheels`; it now probes the
registry for the lock-matched wheel image and falls back to
`latest` if the lock-hash image isn't published yet, instead of
blocking every arch on `wheels` finishing (brian-hussey).
- Move the WHEELS_REF default onto the ARG declaration instead of a
shell-style fallback on the FROM line (brian-hussey).
- Add `uv.lock` and `infra/wheels/**` to docker-required.yml's
paths-ignore so it doesn't race docker-multiplatform.yml on PRs
that only touch those paths (madhu-mohan-jaishankar).
Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>
* fix(docker): omit empty WHEELS_REF build-arg to let Containerfile default apply
Passing --build-arg WHEELS_REF= (empty) overrides Containerfile.lite's
ARG WHEELS_REF=${UBI_MINIMAL} default with a blank value, since Docker
only applies an ARG default when the build-arg isn't passed at all.
This broke the amd64 build with "base name (${WHEELS_REF}) should not
be blank". Build the build-args block conditionally so the line is
omitted entirely when WHEELS_REF is unset.
Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>
* fix(docker): restore needs: wheels after verifying live-PyPI fallback is broken
Decoupling build from wheels (probe-and-fall-back-to-latest, no needs)
seemed like the right tradeoff per review feedback, but workflow_dispatch
run 27754997156 proved the live-PyPI fallback Containerfile.lite uses when
no wheel image exists is not viable for these archs:
- s390x: psycopg-binary has no PyPI wheels for s390x at all -> unsatisfiable
- ppc64le: tiktoken needs a Rust compiler to build from sdist, which the
builder stage doesn't install -> "can't find Rust compiler"
Both failed on the very first cold-lock-hash run, before the wheels job
(which took ~35min to build from scratch) had published anything for the
probe to find. There's nothing to gracefully degrade to, so blocking on
wheels is required. Keeping the latest-tag push, the ARG-default fix, and
the conditional build-arg fix - those are independently correct.
Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>
* fix(docker): trim build job comment to describe current behavior, not history
Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>
* fix(docker): default WHEELS_REF to arch-specific latest wheel image
Containerfile.lite's ARG WHEELS_REF defaulted to UBI_MINIMAL, so a plain
local `docker build` for s390x/ppc64le got no wheel closure and fell
through to the live-PyPI install path - which doesn't work for those
archs (no s390x wheel for psycopg-binary, no Rust toolchain for tiktoken's
sdist build on ppc64le).
Default is now arch-aware: WHEELS_REF=ghcr.io/ibm/mcp-context-forge-wheels:${TARGETARCH}-latest.
These images are public (verified anonymous pull). To make the tag always
resolve, the `wheels` job now builds for all four arches instead of just
s390x/ppc64le - amd64/arm64 don't need the closure for their own install
(PyPI has full coverage there) but building it is cheap, mostly downloading
existing wheels rather than compiling (~50s locally vs ~35min cold for
s390x/ppc64le), and it makes those builds hermetic too.
Bumped the Containerfile.lite syntax pin from docker/dockerfile:1.7 to 1.12:
verified locally that TARGETARCH resolves empty when referenced in a global
ARG default before the first FROM on syntax 1.4-1.10 (a buildkit frontend
bug), fixed from 1.11 onward.
Verified locally: default resolves to the right per-arch tag, explicit
--build-arg override still takes precedence, and an arch with no published
wheel image fails with a clear "not found" rather than silently resolving
to something blank.
Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>
* fix(docker): revert arch-aware WHEELS_REF default, scope back to s390x/ppc64le
The previous commit defaulted WHEELS_REF to ghcr.io/.../wheels:${TARGETARCH}-latest
for all four arches, intended to make local builds "just work" without flags.
This broke CI immediately: the PR-gating amd64 build now depended on an
amd64-latest tag that didn't exist yet (wheels job had never built one, and
PR events never run it), failing with "not found".
Confirmed by direct testing that Dockerfile ARG defaults can't branch per
arch - no nested ${VAR_${TARGETARCH}} indirection support ("unsupported
modifier ($) in substitution"). A single default can't safely differ
between "amd64/arm64 must stay self-sufficient" and "s390x/ppc64le want the
latest wheel image" - it's all four or none.
Reverted WHEELS_REF default to UBI_MINIMAL (matches original safe
behavior, no GHCR dependency for the PR-gating amd64/arm64 builds) and
reverted the wheels job matrix back to s390x/ppc64le only - building
amd64/arm64 wheel images served no purpose once nothing defaults to
consuming them.
Local s390x/ppc64le builds that want the prebuilt closure without a
lock-hash tag can pass it explicitly:
--build-arg WHEELS_REF=ghcr.io/ibm/mcp-context-forge-wheels:s390x-latest
Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>
* fix(docker): restore arch-aware WHEELS_REF default, push resolution into the pipeline
Brian's feedback: defaulting to UBI_MINIMAL makes `COPY --from=wheels
/wheels /tmp/wheels` copy an empty directory, and gives local builds no
benefit from the prebuilt closure. The earlier revert traded that away to
fix a real bug (PR-gating amd64 build failing with "amd64-latest: not
found"), but the bug was fixable without abandoning the arch-aware default.
Containerfile.lite's `ARG WHEELS_REF` default is back to
ghcr.io/.../wheels:${TARGETARCH}-latest for all four arches. What changes
is where the pipeline gets its safety net: the `build` job's "Compute wheel
image ref" step now runs for every arch (previously s390x/ppc64le only)
and always resolves WHEELS_REF explicitly - lock-hash tag, else `latest`
tag, else (cold-start only, before `wheels` has published anything for an
arch) the literal UBI_MINIMAL value read from Containerfile.lite. CI no
longer depends on Containerfile.lite's bare default ever being reached;
that default is now reserved for genuine local `docker build` with no
flags, which is exactly where it's wanted.
wheels job matrix is back to all four arches so the default has something
to resolve against on amd64/arm64 too.
Verified locally: default build pulls the real wheel image (136 wheels,
not an empty dir); explicit UBI_MINIMAL override still resolves cleanly
for the cold-start path.
Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>
* fix(docker): pin WHEELS_REF in FIPS security scan to avoid cp311/cp312 mismatch
The FIPS build step in docker-scan.yml overrides PYTHON_VERSION=3.11 but
never set WHEELS_REF, so it picked up Containerfile.lite's default
amd64-latest closure, which is built against PYTHON_VERSION=3.12 wheels.
uv rejected the cp312-tagged psycopg[c] wheel under the 3.11 venv,
failing the build.
Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>
* fix(deps): bump python-multipart to >=0.0.30 for GHSA-5rvq-cxj2-64vf
0.0.29 is vulnerable to a denial-of-service via malformed multipart
form-data boundaries (CVE-2024-53981 / GHSA-5rvq-cxj2-64vf, High).
Grype flagged this in the wheel-closure-based amd64 image build,
failing the Docker Security Scan workflow.
Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>
* fix(packaging): include infra/wheels/Containerfile and dockerignore in sdist
check-manifest flagged infra/wheels/Containerfile and
infra/wheels/Containerfile.dockerignore as tracked in VCS but missing
from the sdist, since MANIFEST.in had no rule covering them.
Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>
* fix(packaging): drop stale infra/s390x/Containerfile from MANIFEST.in
infra/s390x/Containerfile was removed when the s390x/ppc64le builds
moved to the shared infra/wheels/Containerfile closure; the MANIFEST.in
include for the old path is now dead.
Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>
* chore(deps): drop GHSA reference from python-multipart pin comment
Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>
---------
Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>
Co-authored-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>1 parent ada6ced commit 30a7057
12 files changed
Lines changed: 255 additions & 210 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
28 | 28 | | |
29 | 29 | | |
30 | 30 | | |
| 31 | + | |
| 32 | + | |
31 | 33 | | |
32 | 34 | | |
33 | 35 | | |
| |||
37 | 39 | | |
38 | 40 | | |
39 | 41 | | |
| 42 | + | |
| 43 | + | |
40 | 44 | | |
41 | 45 | | |
42 | 46 | | |
| |||
54 | 58 | | |
55 | 59 | | |
56 | 60 | | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
57 | 145 | | |
58 | 146 | | |
59 | 147 | | |
60 | 148 | | |
61 | | - | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
62 | 159 | | |
63 | 160 | | |
64 | 161 | | |
| |||
124 | 221 | | |
125 | 222 | | |
126 | 223 | | |
| 224 | + | |
| 225 | + | |
127 | 226 | | |
128 | | - | |
| 227 | + | |
129 | 228 | | |
130 | 229 | | |
131 | 230 | | |
132 | 231 | | |
133 | 232 | | |
134 | 233 | | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
135 | 264 | | |
136 | 265 | | |
137 | 266 | | |
| |||
149 | 278 | | |
150 | 279 | | |
151 | 280 | | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
152 | 288 | | |
153 | 289 | | |
154 | 290 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
25 | 25 | | |
26 | 26 | | |
27 | 27 | | |
| 28 | + | |
| 29 | + | |
28 | 30 | | |
29 | 31 | | |
30 | 32 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
174 | 174 | | |
175 | 175 | | |
176 | 176 | | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
177 | 182 | | |
178 | 183 | | |
179 | 184 | | |
180 | 185 | | |
181 | 186 | | |
182 | 187 | | |
| 188 | + | |
183 | 189 | | |
184 | 190 | | |
185 | 191 | | |
| |||
This file was deleted.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
52 | 52 | | |
53 | 53 | | |
54 | 54 | | |
| 55 | + | |
55 | 56 | | |
56 | 57 | | |
57 | 58 | | |
| |||
0 commit comments