Skip to content

Commit eb84c58

Browse files
author
Bhanu Teja Goshikonda
committed
fix(pytorch): reduce 2.13 GPU max_jobs to 4 to fit wheel-export in RAM
The wheel-cache upload path in scripts/ci/build/pytorch_runtime/lib/upload_wheels.sh runs `docker buildx build --target wheel-export`, which re-executes the entire Dockerfile to reach a scratch stage that COPYs the built wheels out of the flash-attn and TE builder stages. With MAX_JOBS=8, the concurrent flash-attn 2.8.3 + transformer-engine 2.16.1 source builds during that re-execution exceed the CI runner cgroup memory limit and get OOM-killed (SIGKILL / exit code 137, PR #6365 attempt 2). Halving MAX_JOBS to 4 roughly halves peak compile RAM, keeping the re-execution under the runner ceiling at the cost of a slower flash-attn / TE compile step. CPU configs unaffected.
1 parent bef8cb8 commit eb84c58

2 files changed

Lines changed: 2 additions & 2 deletions

File tree

.github/config/image/pytorch/2.13-ec2-cuda.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ build:
2626
nccl_version: "2.30.7-1"
2727
efa_version: "1.49.0"
2828
gdrcopy_version: "2.6"
29-
max_jobs: "8"
29+
max_jobs: "4"
3030
dlc_major_version: "1"
3131
dlc_minor_version: "0"
3232

.github/config/image/pytorch/2.13-sagemaker-cuda.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@ build:
2727
nccl_version: "2.30.7-1"
2828
efa_version: "1.49.0"
2929
gdrcopy_version: "2.6"
30-
max_jobs: "8"
30+
max_jobs: "4"
3131
dlc_major_version: "1"
3232
dlc_minor_version: "0"
3333

0 commit comments

Comments
 (0)