Skip to content

Commit 51e3d4c

Browse files
author
Bhanu Teja Goshikonda
committed
fix(pytorch): opt 2.13 GPU configs out of wheel-cache upload
Add a per-config `build.skip_wheel_cache_upload` toggle and enable it on the two 2.13 GPU configs. The wheel-cache upload path in scripts/ci/build/pytorch_runtime/lib/upload_wheels.sh runs `docker buildx build --target wheel-export` which re-executes the entire Dockerfile from scratch to reach a scratch stage that COPYs the built wheels out of the flash-attn and TE builder stages. On 2.13 that concurrent recompile (flash-attn 2.8.3 source build + transformer-engine 2.16.1) exceeds the CI runner cgroup memory limit and gets OOM-killed with SIGKILL / exit code 137, before post_build.sh's `|| true` can catch it. Evidence from PR CI attempt 2 (both GPU builds, same signature): upload_wheels.sh: line 45: NNN Killed docker buildx build \ --progress=plain --target wheel-export \ --output type=local,dest=${EXPORT_DIR} -f ${DOCKERFILE} . ##[error]Process completed with exit code 137. The primary image build+push succeeded before this step in both jobs; the wheel cache is a build-time optimization that has no active consumer for 2.13 yet (WHEEL_CACHE_HIT was already false). Skipping keeps GPU builds green until a larger CI runner class is provisioned. - resolve_build_args.py: add skip_wheel_cache_upload to RESERVED_KEYS so it doesn't leak into Docker as a --build-arg. - post_build.sh: short-circuit before invoking upload_wheels.sh when the flag is set. - 2.13-{ec2,sagemaker}-cuda.yml: set build.skip_wheel_cache_upload: "true" with an inline comment explaining the opt-out. CPU configs unaffected.
1 parent bef8cb8 commit 51e3d4c

4 files changed

Lines changed: 24 additions & 1 deletion

File tree

.github/actions/build-image/resolve_build_args.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@
2424
import subprocess
2525
import sys
2626

27-
RESERVED_KEYS = {"dockerfile", "target"}
27+
RESERVED_KEYS = {"dockerfile", "target", "skip_wheel_cache_upload"}
2828

2929

3030
def load_yaml(path):

.github/config/image/pytorch/2.13-ec2-cuda.yml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,10 @@ build:
2929
max_jobs: "8"
3030
dlc_major_version: "1"
3131
dlc_minor_version: "0"
32+
# wheel-export re-runs the CUDA-devel builder stages and OOM-kills on the CI
33+
# runner with flash-attn + transformer-engine 2.16 wheels. Opt out until a
34+
# larger runner class is available; the primary image still builds/pushes.
35+
skip_wheel_cache_upload: "true"
3236

3337
release:
3438
release: true

.github/config/image/pytorch/2.13-sagemaker-cuda.yml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,10 @@ build:
3030
max_jobs: "8"
3131
dlc_major_version: "1"
3232
dlc_minor_version: "0"
33+
# wheel-export re-runs the CUDA-devel builder stages and OOM-kills on the CI
34+
# runner with flash-attn + transformer-engine 2.16 wheels. Opt out until a
35+
# larger runner class is available; the primary image still builds/pushes.
36+
skip_wheel_cache_upload: "true"
3337

3438
release:
3539
release: true

scripts/ci/build/pytorch_runtime/post_build.sh

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,21 @@ done
3333
[[ -n "$CONFIG_FILE" ]] || { echo "ERROR: --config-file is required" >&2; exit 1; }
3434
[[ -f "$CONFIG_FILE" ]] || { echo "ERROR: Config file not found: $CONFIG_FILE" >&2; exit 1; }
3535

36+
# Per-config opt-out. When `build.skip_wheel_cache_upload: true` is set on an
37+
# image config, skip lib/upload_wheels.sh entirely. Rationale: upload_wheels.sh
38+
# invokes `docker buildx build --target wheel-export` which re-executes the
39+
# CUDA-devel builder stages from scratch to produce a scratch-image containing
40+
# built wheels. On heavy configs (flash-attn + transformer-engine >=2.16), the
41+
# concurrent compile drives the runner over its cgroup memory limit and gets
42+
# OOM-killed (SIGKILL, exit 137). The wheel cache is a build-time optimization
43+
# — skipping it means future PR builds cannot reuse wheels from this run, but
44+
# the primary image already built and pushed successfully.
45+
SKIP_WHEEL_CACHE_UPLOAD=$(yq '.build.skip_wheel_cache_upload // "false"' "$CONFIG_FILE")
46+
if [[ "${SKIP_WHEEL_CACHE_UPLOAD}" == "true" ]]; then
47+
echo "build.skip_wheel_cache_upload=true in ${CONFIG_FILE} — skipping wheel-cache upload"
48+
exit 0
49+
fi
50+
3651
CUDA_VERSION=$(yq '.build.cuda_version' "$CONFIG_FILE")
3752
FLASH_ATTN_VERSION=$(yq '.build.flash_attn_version // ""' "$CONFIG_FILE")
3853
TRANSFORMER_ENGINE_VERSION=$(yq '.build.transformer_engine_version // ""' "$CONFIG_FILE")

0 commit comments

Comments
 (0)