Commit 51e3d4c
Bhanu Teja Goshikonda
fix(pytorch): opt 2.13 GPU configs out of wheel-cache upload
Add a per-config `build.skip_wheel_cache_upload` toggle and enable it on the
two 2.13 GPU configs. The wheel-cache upload path in
scripts/ci/build/pytorch_runtime/lib/upload_wheels.sh runs
`docker buildx build --target wheel-export` which re-executes the entire
Dockerfile from scratch to reach a scratch stage that COPYs the built wheels
out of the flash-attn and TE builder stages. On 2.13 that concurrent recompile
(flash-attn 2.8.3 source build + transformer-engine 2.16.1) exceeds the CI
runner cgroup memory limit and gets OOM-killed with SIGKILL / exit code 137,
before post_build.sh's `|| true` can catch it.
Evidence from PR CI attempt 2 (both GPU builds, same signature):
upload_wheels.sh: line 45: NNN Killed docker buildx build \
--progress=plain --target wheel-export \
--output type=local,dest=${EXPORT_DIR} -f ${DOCKERFILE} .
##[error]Process completed with exit code 137.
The primary image build+push succeeded before this step in both jobs; the
wheel cache is a build-time optimization that has no active consumer for 2.13
yet (WHEEL_CACHE_HIT was already false). Skipping keeps GPU builds green until
a larger CI runner class is provisioned.
- resolve_build_args.py: add skip_wheel_cache_upload to RESERVED_KEYS so it
doesn't leak into Docker as a --build-arg.
- post_build.sh: short-circuit before invoking upload_wheels.sh when the flag
is set.
- 2.13-{ec2,sagemaker}-cuda.yml: set build.skip_wheel_cache_upload: "true"
with an inline comment explaining the opt-out. CPU configs unaffected.1 parent bef8cb8 commit 51e3d4c
4 files changed
Lines changed: 24 additions & 1 deletion
File tree
- .github
- actions/build-image
- config/image/pytorch
- scripts/ci/build/pytorch_runtime
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
27 | | - | |
| 27 | + | |
28 | 28 | | |
29 | 29 | | |
30 | 30 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
29 | 29 | | |
30 | 30 | | |
31 | 31 | | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
32 | 36 | | |
33 | 37 | | |
34 | 38 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
30 | 30 | | |
31 | 31 | | |
32 | 32 | | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
33 | 37 | | |
34 | 38 | | |
35 | 39 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
33 | 33 | | |
34 | 34 | | |
35 | 35 | | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
36 | 51 | | |
37 | 52 | | |
38 | 53 | | |
| |||
0 commit comments