Skip to content

Commit 4bb47eb

Browse files
mithun-pensandosaadrahim
mithun-pensando
and
saadrahim
authored
Installation document describing the steps to install Enroot/Pyxis on a (#58)
* Installation document describing the steps to install Enroot/Pyxis on a SLURM cluster with AMD GPUs * Update and rename enroot-pyxis-installation.txt to enroot-pyxis-installation.md * Update _toc.yml.in * Rename docs/enroot-pyxis-installation.md to docs/container-runtime/enroot-pyxis-installation.md * Update index.md * Update index.md * Update enroot-pyxis-installation.md --------- Co-authored-by: Saad Rahim (AMD) <44449863+saadrahim@users.noreply.github.com>
1 parent 213c510 commit 4bb47eb

3 files changed

Lines changed: 191 additions & 0 deletions

File tree

‎docs/container-runtime/enroot-pyxis-installation.md‎

Lines changed: 189 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,189 @@
1+
# Enroot and Pyxis
2+
3+
[Enroot](https://github.com/NVIDIA/enroot) and [Pyxis](https://github.com/NVIDIA/pyxis) are tools created to run containerized AI/HPC workloads on SLURM. These tools can now be used on a SLURM cluster with AMD GPUs to make them run efficiently and achieve isolation for these GPUs. Traditional runtimes like Docker/Podman bring additional overhead such as daemons, root privileges and extra storage layers. With Enroot, users can convert Docker images into a simple unpacked filesystem tree and run containers as a regular Linux process. Also, with Enroot and Pyxis, each job is granted exclusive GPU device files which prevents jobs from accidentally accessing the same GPU device.
4+
5+
This guide provides the steps to install enroot/pyxis on a SLURM cluster as well as examples to run containerized images isolating specific AMD GPUs on Ubuntu.
6+
7+
## Installation
8+
9+
Pre-requisites:
10+
Make sure SLURM is already installed and the cluster is up and running.
11+
Since GPUs are used with enroot and pyxis, the /etc/slurm/gres.conf should be configured with the correct renderD IDs.
12+
13+
Sample gres.conf file :
14+
```bash
15+
Name=gpu Type=rocm File=/dev/dri/renderD128
16+
Name=gpu Type=rocm File=/dev/dri/renderD136
17+
Name=gpu Type=rocm File=/dev/dri/renderD144
18+
Name=gpu Type=rocm File=/dev/dri/renderD152
19+
Name=gpu Type=rocm File=/dev/dri/renderD160
20+
Name=gpu Type=rocm File=/dev/dri/renderD168
21+
Name=gpu Type=rocm File=/dev/dri/renderD176
22+
Name=gpu Type=rocm File=/dev/dri/renderD184
23+
```
24+
Enroot Installation:
25+
```bash
26+
#Check requirements
27+
curl -fSsL -O https://github.com/NVIDIA/enroot/releases/download/v3.5.0/enroot-check_3.5.0_$(uname -m).run
28+
chmod +x enroot-check_*.run
29+
./enroot-check_*.run --verify
30+
./enroot-check_*.run
31+
```
32+
33+
Install enroot through a package
34+
```bash
35+
arch=$(dpkg --print-architecture)
36+
curl -fSsL -O https://github.com/NVIDIA/enroot/releases/download/v3.5.0/enroot_3.5.0-1_${arch}.deb
37+
curl -fSsL -O https://github.com/NVIDIA/enroot/releases/download/v3.5.0/enroot+caps_3.5.0-1_${arch}.deb
38+
sudo apt install -y ./*.deb
39+
```
40+
Validate Enroot installation
41+
```bash
42+
enroot import docker://rocm/pytorch:latest
43+
enroot create rocm+pytorch+latest.sqsh
44+
enroot start rocm+pytorch+latest rocm-smi
45+
ENROOT_RESTRICT_DEV=y enroot start rocm+pytorch+latest rocm-smi
46+
```
47+
Reference: https://github.com/NVIDIA/enroot/blob/master/doc/installation.md
48+
49+
## Steps to build and install Pyxis
50+
Install the following packages
51+
52+
```bash
53+
sudo apt update
54+
sudo apt install -y devscripts
55+
sudo apt install -y debhelper
56+
```
57+
Create a deb package
58+
```bash
59+
git clone https://github.com/NVIDIA/pyxis
60+
cd pyxis
61+
git checkout v0.20.0
62+
make orig
63+
CPPFLAGS="-I/usr/local/slurm-24.05.5.1/include" LDFLAGS="-L/usr/local/slurm-24.05.5.1/lib" make deb
64+
```
65+
After this step, nvslurm-plugin-pyxis_0.20.0-1_amd64.deb will be created in the same directory.
66+
67+
## Steps to install pyxis deb on all the compute nodes and also the slurm head-node
68+
Install the same pyxis deb package on the headnode and all the compute nodes.
69+
While installing pyxis on the headnode/controller node, it will throw error that enroot is not installed but we can ignore this error since we need not have enroot on the head-bode.
70+
71+
```bash
72+
sudo dpkg -i ./nvslurm-plugin-pyxis_0.20.0-1_amd64.deb
73+
sudo mkdir /etc/slurm/plugstack.conf.d
74+
sudo ln -s /usr/share/pyxis/pyxis.conf /etc/slurm/plugstack.conf.d/pyxis.conf
75+
sudo touch /etc/slurm/plugstack.conf
76+
echo "include /etc/slurm/plugstack.conf.d/*" | sudo tee -a /etc/slurm/plugstack.conf
77+
```
78+
79+
Restart slurmd on the compute
80+
```bash
81+
sudo systemctl restart slurmd
82+
```
83+
Restart slurmd and slurmctld on all nodes
84+
```bash
85+
sudo systemctl restart slurmd
86+
sudo systemctl restart slurmctld
87+
```
88+
### Test with SLURM
89+
90+
Following shows 4 isolated AMD GPUs running a containerized image rocm/pytorch
91+
92+
```bash
93+
ubuntu@node-4:~$ srun --gres=gpu:4 --container-image=docker://rocm/pytorch:latest rocm-smi
94+
pyxis: importing docker image: docker://rocm/pytorch:latest
95+
pyxis: imported docker image: docker://rocm/pytorch:latest
96+
============================================ ROCm System Management Interface ============================================
97+
98+
====================================================== Concise Info ======================================================
99+
100+
Device Node IDs Temp Power Partitions SCLK MCLK Fan Perf PwrCap VRAM% GPU%
101+
102+
(DID, GUID) (Junction) (Socket) (Mem, Compute, ID)
103+
104+
==========================================================================================================================
105+
106+
0 2 0x74a1, 28851 45.0°C 137.0W NPS1, SPX, 0 123Mhz 900Mhz 0% auto 750.0W 0% 0%
107+
108+
1 3 0x74a1, 43178 41.0°C 133.0W NPS1, SPX, 0 124Mhz 900Mhz 0% auto 750.0W 0% 0%
109+
110+
2 4 0x74a1, 32898 44.0°C 133.0W NPS1, SPX, 0 124Mhz 900Mhz 0% auto 750.0W 0% 0%
111+
112+
3 5 0x74a1, 22683 40.0°C 136.0W NPS1, SPX, 0 124Mhz 900Mhz 0% auto 750.0W 0% 0%
113+
114+
==========================================================================================================================
115+
116+
================================================== End of ROCm SMI Log ===================================================
117+
```
118+
119+
Following shows isolating 2 AMD GPUs running a containerized image rocm/pytorch
120+
121+
```bash
122+
ubuntu@node-4:~$ srun --gres=gpu:2 --container-image=docker://rocm/pytorch:latest rocm-smi
123+
pyxis: importing docker image: docker://rocm/pytorch:latest
124+
pyxis: imported docker image: docker://rocm/pytorch:latest
125+
============================================ ROCm System Management Interface ============================================
126+
127+
====================================================== Concise Info ======================================================
128+
129+
Device Node IDs Temp Power Partitions SCLK MCLK Fan Perf PwrCap VRAM% GPU%
130+
131+
(DID, GUID) (Junction) (Socket) (Mem, Compute, ID)
132+
133+
==========================================================================================================================
134+
135+
0 2 0x74a1, 28851 45.0°C 137.0W NPS1, SPX, 0 123Mhz 900Mhz 0% auto 750.0W 0% 0%
136+
137+
1 3 0x74a1, 43178 41.0°C 133.0W NPS1, SPX, 0 124Mhz 900Mhz 0% auto 750.0W 0% 0%
138+
139+
==========================================================================================================================
140+
141+
================================================== End of ROCm SMI Log ===================================================
142+
```
143+
144+
Following command runs a test.py script to different torch.cuda variables
145+
146+
```bash
147+
ubuntu@node-4:~$ srun --gres=gpu:2 --container-image=docker://rocm/pytorch:latest --container-mounts="$HOME:/home/$MY_USER" python3 /home/$MY_USER/test.py
148+
pyxis: importing docker image: docker://rocm/pytorch:latest
149+
pyxis: imported docker image: docker://rocm/pytorch:latest
150+
--- PyTorch CUDA Status ---
151+
torch.cuda.is_available(): True
152+
torch.cuda.device_count(): 2
153+
154+
Visible Devices (from CUDA_VISIBLE_DEVICES env var):
155+
CUDA_VISIBLE_DEVICES=0,1
156+
Detected Devices:
157+
Device 0: AMD Instinct MI300X
158+
Capability: (9, 4)
159+
Memory (GB): 191.98
160+
Device 1: AMD Instinct MI300X
161+
Capability: (9, 4)
162+
Memory (GB): 191.98
163+
--- End PyTorch CUDA Status ---
164+
165+
The following command can be used to save the image locally to use for subsequent runs
166+
ubuntu@node-4:~$ srun --gres=gpu:8 --container-image=docker://rocm/pytorch:latest --container-save=/var/lib/ubuntu/enroot/rocm+pytorch+latest.sqsh rocm-smi
167+
ubuntu@node-4:~$ srun --gres=gpu:8 --container-image=/var/lib/ubuntu/enroot/rocm+pytorch+latest.sqsh rocm-smi
168+
169+
Following command shows the usage of --exclusive directive with 2 GPUS requested.
170+
root@head-node:~# srun --exclusive --gres=gpu:2 --container-image=./rocm+ubuntu.sqsh --pty bash
171+
root@valid-prawn:/# rocm-smi
172+
============================================ ROCm System Management Interface ============================================
173+
174+
====================================================== Concise Info ======================================================
175+
176+
Device Node IDs Temp Power Partitions SCLK MCLK Fan Perf PwrCap VRAM% GPU%
177+
178+
(DID, GUID) (Junction) (Socket) (Mem, Compute, ID)
179+
180+
==========================================================================================================================
181+
182+
0 2 0x74a1, 28851 35.0°C 142.0W NPS1, SPX, 0 132Mhz 900Mhz 0% auto 750.0W 0% 0%
183+
184+
1 3 0x74a1, 44463 34.0°C 136.0W NPS1, SPX, 0 134Mhz 900Mhz 0% auto 750.0W 0% 0%
185+
186+
==========================================================================================================================
187+
188+
================================================== End of ROCm SMI Log ===================================================
189+
```

‎docs/index.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,4 +17,5 @@ This documentation site provides information about the AMD Container Toolkit, wh
1717
- [Migration Guide](container-runtime/migration-guide.rst)
1818
- [Developer Guide](container-runtime/developer-guide.rst)
1919
- [Docker Compose](container-runtime/docker-compose.rst)
20+
- [Enroot Pyxis Installation](container-runtime/enroot-pyxis-installation.md)
2021
- [Release Notes](container-runtime/release-notes.rst)

‎docs/sphinx/_toc.yml.in‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,5 +18,6 @@ subtrees:
1818
title: Developer Guide
1919
- file: container-runtime/docker-compose.rst
2020
title: Docker Compose Usage
21+
- file: container-runtime/enroot-pyxis-installation.md
2122
- file: container-runtime/release-notes.rst
2223
title: Release Notes

0 commit comments

Comments
 (0)