torch-distributed-hanging-solutions.md

Diagnosing Hangings and Deadlocks in Multi-Node Multi-GPU Python Programs

Moved to Debugging PyTorch Programs.

Helper tools

Moved to Debugging PyTorch Programs.

Approaches to diagnosing multi-gpu hanging / deadlocks

Moved to Debugging PyTorch Programs.

py-spy

Moved to Debugging PyTorch Programs.

multi-process py-spy

Moved to Debugging PyTorch Programs.

multi-node py-spy via srun

Moved to Debugging PyTorch Programs.

multi-node py-spy via pdsh

Moved to Debugging PyTorch Programs.

multi-node py-spy via ds_ssh

Moved to Debugging PyTorch Programs.

Network-level hanging

Moved to Debugging PyTorch Programs.

Isolate problematic GPUs

Moved to Debugging PyTorch Programs.

python trace

Moved to Debugging PyTorch Programs.

NicerTrace

Moved to Debugging PyTorch Programs.

Working with generated trace files

Moved to Debugging PyTorch Programs.

good old print

Moved to Debugging PyTorch Programs.

core files

Moved to Debugging PyTorch Programs.

Code loops

Moved to Debugging PyTorch Programs.

Hardware-specific issues

Moved to Debugging PyTorch Programs.

AMD/ROCm hangs or slow with IOMMU enabled

Moved to Debugging PyTorch Programs.