Skip to content

Commit 30a7057

Browse files
msureshkumar88Suresh Kumar Moharajan
andauthored
fix(docker): hermetic wheel closure for s390x/ppc64le multiplatform builds (#5287)
* fix(docker): full hermetic wheel closure for s390x/ppc64le multiplatform builds Multiplatform Docker Build was red on main for s390x (alembic unsatisfiable — the old s390x wheel image only carried curated native wheels, no pure-Python alembic/tiktoken) and ppc64le (tiktoken source build failing, no wheel image at all). amd64/arm64 are unaffected (PyPI manylinux wheels exist). Replaces the curated native-wheel list with a full dependency closure derived from uv.lock (uv export --no-default-groups), so Containerfile.lite's --no-index install resolves every pure-Python and native package without PyPI egress. Wheel images are content-addressed by sha256(uv.lock) and built inline as a needs: dependency in docker-multiplatform.yml (pull-or-build, idempotent), replacing the standalone s390x-wheels.yml nightly workflow. Containerfile.lite's wheel stage is now arch-agnostic (WHEELS_REF build-arg, empty for amd64/arm64) instead of unconditionally pulling a s390x-only image, and the per-arch install branching collapses into one hermetic/fallback path. Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com> * fix(docker): build job skipping entirely on PRs due to needs:wheels gate GitHub's default needs condition requires success(), and treats a skipped dependency the same as failed. The wheels job intentionally skips on pull_request events, which was causing the entire build job (including amd64, which is supposed to build on every PR) to skip too instead of just the s390x/ppc64le wheel-dependent path. Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com> * fix(docker): address PR #5287 review feedback on wheel image fallback - Push a per-arch `latest` tag alongside the lock-hash tag for wheel images, so a fallback target exists for both the build job and local builds (brian-hussey). - Decouple the `build` job from `needs: wheels`; it now probes the registry for the lock-matched wheel image and falls back to `latest` if the lock-hash image isn't published yet, instead of blocking every arch on `wheels` finishing (brian-hussey). - Move the WHEELS_REF default onto the ARG declaration instead of a shell-style fallback on the FROM line (brian-hussey). - Add `uv.lock` and `infra/wheels/**` to docker-required.yml's paths-ignore so it doesn't race docker-multiplatform.yml on PRs that only touch those paths (madhu-mohan-jaishankar). Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com> * fix(docker): omit empty WHEELS_REF build-arg to let Containerfile default apply Passing --build-arg WHEELS_REF= (empty) overrides Containerfile.lite's ARG WHEELS_REF=${UBI_MINIMAL} default with a blank value, since Docker only applies an ARG default when the build-arg isn't passed at all. This broke the amd64 build with "base name (${WHEELS_REF}) should not be blank". Build the build-args block conditionally so the line is omitted entirely when WHEELS_REF is unset. Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com> * fix(docker): restore needs: wheels after verifying live-PyPI fallback is broken Decoupling build from wheels (probe-and-fall-back-to-latest, no needs) seemed like the right tradeoff per review feedback, but workflow_dispatch run 27754997156 proved the live-PyPI fallback Containerfile.lite uses when no wheel image exists is not viable for these archs: - s390x: psycopg-binary has no PyPI wheels for s390x at all -> unsatisfiable - ppc64le: tiktoken needs a Rust compiler to build from sdist, which the builder stage doesn't install -> "can't find Rust compiler" Both failed on the very first cold-lock-hash run, before the wheels job (which took ~35min to build from scratch) had published anything for the probe to find. There's nothing to gracefully degrade to, so blocking on wheels is required. Keeping the latest-tag push, the ARG-default fix, and the conditional build-arg fix - those are independently correct. Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com> * fix(docker): trim build job comment to describe current behavior, not history Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com> * fix(docker): default WHEELS_REF to arch-specific latest wheel image Containerfile.lite's ARG WHEELS_REF defaulted to UBI_MINIMAL, so a plain local `docker build` for s390x/ppc64le got no wheel closure and fell through to the live-PyPI install path - which doesn't work for those archs (no s390x wheel for psycopg-binary, no Rust toolchain for tiktoken's sdist build on ppc64le). Default is now arch-aware: WHEELS_REF=ghcr.io/ibm/mcp-context-forge-wheels:${TARGETARCH}-latest. These images are public (verified anonymous pull). To make the tag always resolve, the `wheels` job now builds for all four arches instead of just s390x/ppc64le - amd64/arm64 don't need the closure for their own install (PyPI has full coverage there) but building it is cheap, mostly downloading existing wheels rather than compiling (~50s locally vs ~35min cold for s390x/ppc64le), and it makes those builds hermetic too. Bumped the Containerfile.lite syntax pin from docker/dockerfile:1.7 to 1.12: verified locally that TARGETARCH resolves empty when referenced in a global ARG default before the first FROM on syntax 1.4-1.10 (a buildkit frontend bug), fixed from 1.11 onward. Verified locally: default resolves to the right per-arch tag, explicit --build-arg override still takes precedence, and an arch with no published wheel image fails with a clear "not found" rather than silently resolving to something blank. Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com> * fix(docker): revert arch-aware WHEELS_REF default, scope back to s390x/ppc64le The previous commit defaulted WHEELS_REF to ghcr.io/.../wheels:${TARGETARCH}-latest for all four arches, intended to make local builds "just work" without flags. This broke CI immediately: the PR-gating amd64 build now depended on an amd64-latest tag that didn't exist yet (wheels job had never built one, and PR events never run it), failing with "not found". Confirmed by direct testing that Dockerfile ARG defaults can't branch per arch - no nested ${VAR_${TARGETARCH}} indirection support ("unsupported modifier ($) in substitution"). A single default can't safely differ between "amd64/arm64 must stay self-sufficient" and "s390x/ppc64le want the latest wheel image" - it's all four or none. Reverted WHEELS_REF default to UBI_MINIMAL (matches original safe behavior, no GHCR dependency for the PR-gating amd64/arm64 builds) and reverted the wheels job matrix back to s390x/ppc64le only - building amd64/arm64 wheel images served no purpose once nothing defaults to consuming them. Local s390x/ppc64le builds that want the prebuilt closure without a lock-hash tag can pass it explicitly: --build-arg WHEELS_REF=ghcr.io/ibm/mcp-context-forge-wheels:s390x-latest Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com> * fix(docker): restore arch-aware WHEELS_REF default, push resolution into the pipeline Brian's feedback: defaulting to UBI_MINIMAL makes `COPY --from=wheels /wheels /tmp/wheels` copy an empty directory, and gives local builds no benefit from the prebuilt closure. The earlier revert traded that away to fix a real bug (PR-gating amd64 build failing with "amd64-latest: not found"), but the bug was fixable without abandoning the arch-aware default. Containerfile.lite's `ARG WHEELS_REF` default is back to ghcr.io/.../wheels:${TARGETARCH}-latest for all four arches. What changes is where the pipeline gets its safety net: the `build` job's "Compute wheel image ref" step now runs for every arch (previously s390x/ppc64le only) and always resolves WHEELS_REF explicitly - lock-hash tag, else `latest` tag, else (cold-start only, before `wheels` has published anything for an arch) the literal UBI_MINIMAL value read from Containerfile.lite. CI no longer depends on Containerfile.lite's bare default ever being reached; that default is now reserved for genuine local `docker build` with no flags, which is exactly where it's wanted. wheels job matrix is back to all four arches so the default has something to resolve against on amd64/arm64 too. Verified locally: default build pulls the real wheel image (136 wheels, not an empty dir); explicit UBI_MINIMAL override still resolves cleanly for the cold-start path. Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com> * fix(docker): pin WHEELS_REF in FIPS security scan to avoid cp311/cp312 mismatch The FIPS build step in docker-scan.yml overrides PYTHON_VERSION=3.11 but never set WHEELS_REF, so it picked up Containerfile.lite's default amd64-latest closure, which is built against PYTHON_VERSION=3.12 wheels. uv rejected the cp312-tagged psycopg[c] wheel under the 3.11 venv, failing the build. Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com> * fix(deps): bump python-multipart to >=0.0.30 for GHSA-5rvq-cxj2-64vf 0.0.29 is vulnerable to a denial-of-service via malformed multipart form-data boundaries (CVE-2024-53981 / GHSA-5rvq-cxj2-64vf, High). Grype flagged this in the wheel-closure-based amd64 image build, failing the Docker Security Scan workflow. Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com> * fix(packaging): include infra/wheels/Containerfile and dockerignore in sdist check-manifest flagged infra/wheels/Containerfile and infra/wheels/Containerfile.dockerignore as tracked in VCS but missing from the sdist, since MANIFEST.in had no rule covering them. Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com> * fix(packaging): drop stale infra/s390x/Containerfile from MANIFEST.in infra/s390x/Containerfile was removed when the s390x/ppc64le builds moved to the shared infra/wheels/Containerfile closure; the MANIFEST.in include for the old path is now dead. Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com> * chore(deps): drop GHSA reference from python-multipart pin comment Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com> --------- Signed-off-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com> Co-authored-by: Suresh Kumar Moharajan <suresh.kumar.m@ibm.com>
1 parent ada6ced commit 30a7057

12 files changed

Lines changed: 255 additions & 210 deletions

.github/workflows/docker-multiplatform.yml

Lines changed: 138 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,8 @@ on:
2828
- 'mcpgateway/**'
2929
- 'plugins/**'
3030
- 'pyproject.toml'
31+
- 'uv.lock'
32+
- 'infra/wheels/**'
3133
- '.github/workflows/docker-multiplatform.yml'
3234
pull_request:
3335
types: [opened, synchronize, ready_for_review]
@@ -37,6 +39,8 @@ on:
3739
- 'mcpgateway/**'
3840
- 'plugins/**'
3941
- 'pyproject.toml'
42+
- 'uv.lock'
43+
- 'infra/wheels/**'
4044
- '.github/workflows/docker-multiplatform.yml'
4145
merge_group:
4246
branches: ["main"]
@@ -54,11 +58,104 @@ env:
5458
IMAGE_NAME: ${{ github.repository }}
5559

5660
jobs:
61+
# ---------------------------------------------------------------
62+
# Build the full dependency-closure wheel image for every arch, so
63+
# Containerfile.lite's `ARG WHEELS_REF` default (${TARGETARCH}-latest)
64+
# resolves on all four without a flag. Content-addressed by
65+
# sha256(uv.lock) and pull-or-build idempotent, so unchanged deps are a
66+
# cache hit. Rides the same non-PR events as the platform builds below
67+
# (push/merge_group/workflow_dispatch).
68+
# ---------------------------------------------------------------
69+
wheels:
70+
if: github.event_name != 'pull_request'
71+
name: Build wheels (${{ matrix.suffix }})
72+
strategy:
73+
fail-fast: false
74+
matrix:
75+
include:
76+
- platform: linux/amd64
77+
runner: ubuntu-24.04
78+
suffix: amd64
79+
- platform: linux/arm64
80+
runner: ubuntu-24.04-arm
81+
suffix: arm64
82+
- platform: linux/s390x
83+
runner: ubuntu-24.04-s390x
84+
suffix: s390x
85+
- platform: linux/ppc64le
86+
runner: ubuntu-24.04-ppc64le
87+
suffix: ppc64le
88+
runs-on: ${{ matrix.runner }}
89+
timeout-minutes: 120
90+
permissions:
91+
contents: read
92+
packages: write
93+
94+
steps:
95+
- name: Checkout code
96+
uses: actions/checkout@93cb6efe18208431cddfb8368fd83d5badbf9bfd # v5
97+
with:
98+
persist-credentials: false
99+
100+
- name: Compute image ref
101+
id: ref
102+
run: |
103+
WHEELS_IMAGE_LC=$(echo "${{ env.IMAGE_NAME }}-wheels" | tr '[:upper:]' '[:lower:]')
104+
LOCK_HASH=$(sha256sum uv.lock | cut -c1-16)
105+
echo "image=${{ env.REGISTRY }}/${WHEELS_IMAGE_LC}:${{ matrix.suffix }}-${LOCK_HASH}" >> "${GITHUB_OUTPUT}"
106+
echo "latest=${{ env.REGISTRY }}/${WHEELS_IMAGE_LC}:${{ matrix.suffix }}-latest" >> "${GITHUB_OUTPUT}"
107+
108+
- name: Set up Docker Buildx
109+
uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3
110+
111+
- name: Log in to GHCR
112+
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
113+
with:
114+
registry: ${{ env.REGISTRY }}
115+
username: ${{ github.actor }}
116+
password: ${{ secrets.GITHUB_TOKEN }}
117+
118+
- name: Check for existing wheel image
119+
id: check
120+
run: |
121+
if docker manifest inspect "${{ steps.ref.outputs.image }}" >/dev/null 2>&1; then
122+
echo "Wheel image ${{ steps.ref.outputs.image }} already exists, skipping build"
123+
echo "exists=true" >> "${GITHUB_OUTPUT}"
124+
else
125+
echo "exists=false" >> "${GITHUB_OUTPUT}"
126+
fi
127+
128+
# Push both the lock-hash tag (exact reproducible reference) and a
129+
# `latest` tag per arch (fallback for local builds and for `build`
130+
# below when the lock-hash image isn't published yet).
131+
- name: Build and push wheel image
132+
if: steps.check.outputs.exists != 'true'
133+
uses: docker/build-push-action@10e90e3645eae34f1e60eeb005ba3a3d33f178e8 # v6
134+
with:
135+
context: .
136+
file: infra/wheels/Containerfile
137+
platforms: ${{ matrix.platform }}
138+
push: true
139+
tags: |
140+
${{ steps.ref.outputs.image }}
141+
${{ steps.ref.outputs.latest }}
142+
cache-from: type=gha,scope=wheels-${{ matrix.suffix }}
143+
cache-to: type=gha,mode=max,scope=wheels-${{ matrix.suffix }}
144+
57145
# ---------------------------------------------------------------
58146
# Build each platform in parallel
59147
# ---------------------------------------------------------------
60148
build:
61-
if: github.event_name != 'pull_request' || !github.event.pull_request.draft
149+
# Waits on `wheels`: s390x/ppc64le have no live-PyPI fallback (psycopg-binary
150+
# has no s390x wheels, tiktoken needs a Rust compiler the builder stage lacks
151+
# to build from sdist on ppc64le), so the matching wheel image must exist
152+
# before building against it. `wheels` skips the rebuild via `docker manifest
153+
# inspect` when the lock hash is unchanged, so the wait is ~2-3 min in steady
154+
# state; only a real dependency change pays the full build time. `always() &&
155+
# needs.wheels.result != 'failure'` tolerates `wheels` being skipped entirely
156+
# on PR events.
157+
if: always() && needs.wheels.result != 'failure' && (github.event_name != 'pull_request' || !github.event.pull_request.draft)
158+
needs: wheels
62159
name: Build ${{ matrix.suffix }}
63160
strategy:
64161
fail-fast: false
@@ -124,14 +221,46 @@ jobs:
124221
if: github.event_name != 'pull_request' || matrix.build_on_pr
125222
uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3
126223

224+
# Login is needed both for pushing the final image (push/workflow_dispatch)
225+
# and for probing the wheel-image registry below.
127226
- name: Log in to GHCR
128-
if: github.event_name == 'push' || github.event_name == 'workflow_dispatch'
227+
if: github.event_name != 'pull_request' || matrix.build_on_pr
129228
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
130229
with:
131230
registry: ${{ env.REGISTRY }}
132231
username: ${{ github.actor }}
133232
password: ${{ secrets.GITHUB_TOKEN }}
134233

234+
# Resolve WHEELS_REF explicitly for every arch so the pipeline never
235+
# relies on Containerfile.lite's bare `ARG WHEELS_REF` default (which
236+
# points at a GHCR tag for local-build convenience). Prefer the
237+
# lock-matched tag (exact reproducible closure); if it isn't published
238+
# yet (cold lock-hash, `wheels` job still running), fall back to the
239+
# per-arch `latest` wheel image. If neither exists (first-ever run,
240+
# before `wheels` has published anything for this arch), fall back to
241+
# the literal UBI_MINIMAL default so the build doesn't depend on a
242+
# GHCR tag that doesn't exist yet - Containerfile.lite's install step
243+
# then falls through to a live PyPI install, compiling anything
244+
# missing itself.
245+
- name: Compute wheel image ref
246+
if: github.event_name != 'pull_request' || matrix.build_on_pr
247+
run: |
248+
WHEELS_IMAGE_LC=$(echo "${{ env.IMAGE_NAME }}-wheels" | tr '[:upper:]' '[:lower:]')
249+
LOCK_HASH=$(sha256sum uv.lock | cut -c1-16)
250+
LOCK_TAG="${{ env.REGISTRY }}/${WHEELS_IMAGE_LC}:${{ matrix.suffix }}-${LOCK_HASH}"
251+
LATEST_TAG="${{ env.REGISTRY }}/${WHEELS_IMAGE_LC}:${{ matrix.suffix }}-latest"
252+
if docker manifest inspect "${LOCK_TAG}" >/dev/null 2>&1; then
253+
echo "Using lock-matched wheel image: ${LOCK_TAG}"
254+
echo "WHEELS_REF=${LOCK_TAG}" >> "${GITHUB_ENV}"
255+
elif docker manifest inspect "${LATEST_TAG}" >/dev/null 2>&1; then
256+
echo "Lock-matched wheel image not published yet; falling back to ${LATEST_TAG}"
257+
echo "WHEELS_REF=${LATEST_TAG}" >> "${GITHUB_ENV}"
258+
else
259+
UBI_MINIMAL_DEFAULT=$(grep -m1 '^ARG UBI_MINIMAL=' Containerfile.lite | cut -d= -f2-)
260+
echo "No wheel image available yet for ${{ matrix.suffix }}; falling back to ${UBI_MINIMAL_DEFAULT}, Containerfile.lite will install from PyPI"
261+
echo "WHEELS_REF=${UBI_MINIMAL_DEFAULT}" >> "${GITHUB_ENV}"
262+
fi
263+
135264
- name: Extract metadata
136265
if: github.event_name != 'pull_request' || matrix.build_on_pr
137266
id: meta
@@ -149,6 +278,13 @@ jobs:
149278
context: .
150279
file: Containerfile.lite
151280
platforms: ${{ matrix.platform }}
281+
# WHEELS_REF is always set by "Compute wheel image ref" above when
282+
# this step runs (lock tag, latest tag, or UBI_MINIMAL fallback).
283+
# The `|| ''` guards the case the step itself didn't run (same
284+
# `if` gate skips both), so the build-arg is simply omitted and
285+
# Containerfile.lite's own default applies.
286+
build-args: |
287+
${{ env.WHEELS_REF && format('WHEELS_REF={0}', env.WHEELS_REF) || '' }}
152288
153289
push: ${{ github.event_name == 'push' || github.event_name == 'workflow_dispatch' }}
154290
load: ${{ github.event_name == 'pull_request' || github.event_name == 'merge_group' }}

.github/workflows/docker-required.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,8 @@ on:
2525
- 'mcpgateway/**'
2626
- 'plugins/**'
2727
- 'pyproject.toml'
28+
- 'uv.lock'
29+
- 'infra/wheels/**'
2830
- '.github/workflows/docker-multiplatform.yml'
2931

3032
permissions:

.github/workflows/docker-scan.yml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -174,12 +174,18 @@ jobs:
174174
push: false
175175
load: true
176176
tags: ${{ env.IMAGE_NAME }}:fedramp-validate
177+
# WHEELS_REF pinned to UBI_MINIMAL: this step overrides PYTHON_VERSION to
178+
# 3.11, but the published amd64 wheel closure is built against the
179+
# default PYTHON_VERSION (3.12, see infra/wheels/Containerfile), so its
180+
# psycopg[c] wheel carries a cp312 tag that uv rejects under a 3.11 venv.
181+
# Pinning here skips the hermetic closure and installs from PyPI instead.
177182
build-args: |
178183
ENABLE_FIPS=true
179184
PYTHON_VERSION=3.11
180185
UBI_BASE=registry.access.redhat.com/ubi9/ubi:latest
181186
NODEJS_IMAGE=registry.access.redhat.com/ubi9/nodejs-20:latest
182187
UBI_MINIMAL=registry.access.redhat.com/ubi9/ubi-minimal@sha256:6ea809bdd8164f8b2b607e38f01b14c374a15bdfbc74fcd0155161512dd4e00e
188+
WHEELS_REF=registry.access.redhat.com/ubi9/ubi-minimal@sha256:6ea809bdd8164f8b2b607e38f01b14c374a15bdfbc74fcd0155161512dd4e00e
183189
184190
- name: FedRAMP post-build compliance validation
185191
run: |

.github/workflows/s390x-wheels.yml

Lines changed: 0 additions & 117 deletions
This file was deleted.

.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,7 @@ parts/
5252
sdist/
5353
var/
5454
wheels/
55+
!infra/wheels/
5556
*.egg-info/
5657
.installed.cfg
5758
*.egg

0 commit comments

Comments
 (0)