feat(pytorch): add PyTorch 2.13 currency scaffold#6365
Open
bhanutejagk wants to merge 3 commits into
Open
Conversation
Add DLC image configs, workflow callers, and release schedule entries
for PyTorch 2.13.0 on cu130 (Amazon Linux 2023). Covers ec2 and
sagemaker variants across cuda and cpu devices.
- docker/pytorch/2.13/{cuda,cpu}/{pyproject.toml,uv.lock} pinned to
torch 2.13.0, torchvision 0.28.0, torchaudio 2.11.0 (held; upstream
audio has no 2.12/2.13 tag yet)
- 4 image configs and 4 workflow callers (pr + autorelease x ec2/sagemaker)
- autorelease crons: 30 19 UTC ec2, 00 23 UTC sagemaker on Tue/Thu
Align 2.13 currency image with newer upstream releases that were
available when scaffolding but held back from the initial 2.12-copy.
- nccl 2.26.2 -> 2.30.7-1: matches torch 2.13's bundled NCCL pin
(torch 2.13 requires source builds to use NCCL >= 2.23 and pins its
bundled runtime NCCL to 2.30)
- deepspeed 0.18.8 -> 0.19.2: 0.19.x train carries the torch >= 2.11
compatibility patches; 0.18.8 predates them
- transformer_engine 2.12.0 -> 2.16.1: security patch + cuDNN 9.18.1
requirement in the latest stable train (2.12 is 4 months old)
- efa 1.47.0 -> 1.49.0: latest EFA installer (libfabric 2.4.0amzn5.0,
efa driver 3.1.0, OFI NCCL Plugin 1.20.0)
- gdrcopy 2.4.4 -> 2.6: latest stable; new optional DMA-BUF backend
requires cuda driver 13.3+ but falls back to gdrdrv (current path)
- pyproject deepspeed==0.18.8 -> 0.19.2 mirrors the config bump
- uv.lock regenerated in 2.13/{cuda,cpu}
cryptography>=49.0.0 bump was attempted but reverted: mlflow>=3.9.0
caps cryptography<49 in the sagemaker extra. Revisit when mlflow's
upper cap loosens.
CPU configs unchanged (no gpu-only pins to bump).
The wheel-cache upload path in scripts/ci/build/pytorch_runtime/lib/upload_wheels.sh runs `docker buildx build --target wheel-export`, which re-executes the entire Dockerfile to reach a scratch stage that COPYs the built wheels out of the flash-attn and TE builder stages. With MAX_JOBS=8, the concurrent flash-attn 2.8.3 + transformer-engine 2.16.1 source builds during that re-execution exceed the CI runner cgroup memory limit and get OOM-killed (SIGKILL / exit code 137, PR #6365 attempt 2). Halving MAX_JOBS to 4 roughly halves peak compile RAM, keeping the re-execution under the runner ceiling at the cost of a slower flash-attn / TE compile step. CPU configs unaffected.
bhanutejagk
force-pushed
the
pytorch-2.13-currency
branch
from
51e3d4c to
eb84c58
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Scaffold for the PyTorch 2.13 currency release on the V2 pipeline. Adds DLC image configs, workflow callers, per-version pyproject/uv locks, and release-schedule entries for PyTorch 2.13.0 on cu130 (Amazon Linux 2023), covering
ec2andsagemakercustomer types acrosscudaandcpudevice types.No shared Dockerfile changes —
docker/pytorch/Dockerfile.{cuda,cpu}remain shared across all versions; per-version pins live underdocker/pytorch/2.13/{cuda,cpu}/.Version pins
2.13.00.28.02.11.0(held)pytorch/audiohas no 2.12 or 2.13 release tag; latest is v2.11.0. Matches the 2.12 DLC pin.13.0.2(cu130)download.pytorch.org/whl/*with a complete torch + torchvision + torchaudio wheel set for the pinned versions (cu132 has torch+vision but no torchaudio).3.12uv lockre-resolved cleanly in both2.13/cudaand2.13/cpu.Files added
docker/pytorch/2.13/{cuda,cpu}/{pyproject.toml,uv.lock}(4 files).github/config/image/pytorch/2.13-{ec2,sagemaker}-{cuda,cpu}.yml(4 configs).github/workflows/pytorch.{pr-2.13-cuda,pr-2.13-cpu,autorelease-2.13-ec2,autorelease-2.13-sagemaker}.yml(4 callers).github/release-schedule.yml— 2 new entries (Tue/Thu, 30-min grid,lightGPU class)Autorelease cron slots
Placed on Tue/Thu inside the documented 16:00–23:00 UTC release window, on the existing 00/30-minute grid, with a free 30-min buffer around each pick:
pytorch.autorelease-2.13-ec2.yml30 19 * * 2,4pytorch.autorelease-2.13-sagemaker.yml00 23 * * 2,4Both entries in
release-schedule.ymlmatch theon: schedule:blocks inside the workflow files (_prcheck.release-schedule.ymlshould pass).Test plan
pytorch.pr-2.13-cuda.ymltriggers on this PR (paths match.github/config/image/pytorch/2.13-*-cuda.yml,docker/pytorch/2.13/cuda/**) — build + sanity + security + single-gpu + efa + telemetry + unit + sagemakerpytorch.pr-2.13-cpu.ymltriggers on this PR — build + sanity + security + telemetry + unit_prcheck.release-schedule.ymlvalidates the cron correspondence)schedule+workflow_dispatchonly); they will fire on the first Tue/Thu after mergeworkflow_dispatchsmoke ofpytorch.autorelease-2.13-ec2.ymlafter merge to confirm the release path