[release] Add repackage_rocm_libnuma_fix.py for ROCm import-torch deadlock (#189110)#8256
Draft
atalman wants to merge 1 commit into
Draft
[release] Add repackage_rocm_libnuma_fix.py for ROCm import-torch deadlock (#189110)#8256atalman wants to merge 1 commit into
atalman wants to merge 1 commit into
Conversation
|
@atalman is attempting to deploy a commit to the Meta Open Source Team on Vercel. A member of the Team first needs to authorize it. |
meta-cla
Bot
added
the
CLA Signed
atalman
force-pushed
the
repackage-rocm-libnuma-fix
branch
4 times, most recently
from
a786244 to
05d659a
Compare
…dlock (#189110)
Repackages already-published ROCm torch wheels on the TEST channel to fix the
intermittent `import torch` hang on no-GPU/no-kfd hosts (rocSHMEM NUMAWrapper
dlopen("libnuma.so") -> exit() -> rocprofiler atexit deadlock). For each wheel
under whl/test/<rocm> it downloads, adds bare-name OS-dep symlink aliases in
torch/lib (libnuma.so -> libnuma.so.1, ...) so the dlopen resolves, re-uploads
under the same key to S3 (pytorch) and R2 (pytorch-downloads), and verifies the
alias landed in the wheel + RECORD. RECORD regenerated via auditwheel.
Processes ONE wheel at a time (download -> repackage -> upload -> delete) so
peak disk stays at ~one wheel. --work-dir routes ALL scratch (our mkdtemp AND
auditwheel's unpack, via tempfile.tempdir) to a chosen filesystem -- important
since /tmp is often a small tmpfs. Fails fast with guidance if <25GB free.
Defaults to release 2.13 for rocm7.1/rocm7.2. --wheel <path-or-filename> does a
single wheel (downloads if not local); --output-dir writes locally, no upload.
Modeled on release/repackage_torchaudio_cu130_to_cu132.py.
atalman
force-pushed
the
repackage-rocm-libnuma-fix
branch
from
05d659a to
85df54a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Repackages already-published ROCm
torchwheels on the test channel to fix the intermittentimport torchhang on no-GPU / no-kfd hosts (pytorch/pytorch#189110).For each wheel under
whl/test/<rocm>it:torch/lib— addslibnuma.so→libnuma.so.1(andlibdrm/libdrm_amdgpu/libdw/libelf/libtinfo/libgomp) as symlinks. No new binary content is bundled — just an alias to the file already in the wheel. This makes rocSHMEM'sNUMAWrapperdlopen("libnuma.so")succeed instead ofexit()-ing at load (which trips a racy rocprofiler-sdk atexit deadlock). RECORD is regenerated by auditwheel.pytorch) and R2 (pytorch-downloads) withchecksum-sha256metadata.Defaults: version
2.13.0, bothrocm7.1androcm7.2, test channel only.Mirrors the CD-side fix in
pytorch/pytorch.ci/manywheel/repair_wheel.py(pytorch/pytorch#189114), for wheels already in the wild. Modeled onrelease/repackage_torchaudio_cu130_to_cu132.py.Note: the filename is unchanged but content/sha256 changes, so regenerate the PEP 503 index (
s3_management/manage_v2.py) after uploading so the#sha256=fragments match.