Skip to content

Commit 099b3e3

Browse files
authored
Update 02_jupyter_with_conda_singularity.mdx
removed some prompting so commands will be easier to copy for users.
1 parent 71a9810 commit 099b3e3

1 file changed

Lines changed: 34 additions & 27 deletions

File tree

docs/hpc/09_ood/02_jupyter_with_conda_singularity.mdx

Lines changed: 34 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -57,7 +57,8 @@ The above code automatically makes your environment look for the default shared
5757
### Prepare Overlay File
5858
First, start a job to work on the compute nodes.
5959
```bash
60-
[NetID@log-1 my_env]$ sbatch --cpus-per-task=2 --mem=10GB --time=04:00:00 --wrap "sleep infinity"
60+
# From a login node:
61+
sbatch --cpus-per-task=2 --mem=10GB --time=04:00:00 --wrap "sleep infinity"
6162

6263
# wait to be assigned a node then SSH to the node
6364
```
@@ -66,16 +67,16 @@ Once you SSH to the node, you can begin setting up the environment.
6667
:::note Files will not properly compile on the login nodes as they have different Red Hat OS images than the compute nodes. All compilation and package installation must be done on the compute nodes.
6768

6869
```bash
69-
[NetID@log-1 ~]$ mkdir /scratch/$USER/my_env
70-
[NetID@log-1 ~]$ cd /scratch/$USER/my_env
71-
[NetID@log-1 my_env]$ cp -rp /share/apps/overlay-fs-ext3/overlay-15GB-500K.ext3.gz .
72-
[NetID@log-1 my_env]$ gunzip overlay-15GB-500K.ext3.gz
70+
mkdir /scratch/$USER/my_env
71+
cd /scratch/$USER/my_env
72+
cp -rp /share/apps/overlay-fs-ext3/overlay-15GB-500K.ext3.gz .
73+
gunzip overlay-15GB-500K.ext3.gz
7374
```
7475
Above we used the overlay file `overlay-15GB-500K.ext3.gz` which will contain all of the installed packages. There are more optional overlay files. You can find instructions on the following pages: [Singularity with Conda](../07_containers/03_singularity_with_conda.md), [Squash File System and Singularity](../07_containers/04_squash_file_system_and_singularity.md).
7576

7677
### Launch Singularity Environment for Installation
7778
```bash
78-
[NetID@log-1 ~]$ singularity exec --fakeroot --overlay /scratch/$USER/my_env/overlay-15GB-500K.ext3:rw /share/apps/images/cuda12.3.2-cudnn9.0.0-ubuntu-22.04.4.sif /bin/bash
79+
singularity exec --fakeroot --overlay /scratch/$USER/my_env/overlay-15GB-500K.ext3:rw /share/apps/images/cuda12.3.2-cudnn9.0.0-ubuntu-22.04.4.sif /bin/bash
7980
```
8081
Above we used the Singularity OS image `cuda12.3.2-cudnn9.0.0-ubuntu-22.04.4.sif` which provides the base operating system environment for the conda environment. There are other Singularity OS images available at `/share/apps/images`
8182

@@ -84,6 +85,7 @@ Launching Singularity with the `--overlay` flag mounts the overlay file to a new
8485
:::warning
8586
Be sure that you have the Singularity prompt (`Singularity>`) and that `/ext3` is available before the next step.
8687
```bash
88+
# Example of what it should look like inside Singularity:
8789
Singularity> ls -lah /ext3
8890
total 8.5K
8991
drwxrwxr-x. 2 root root 4.0K Oct 19 10:01 .
@@ -93,17 +95,19 @@ drwx------. 29 root root 8.0K Oct 19 10:01 ..
9395

9496
### Install Miniforge to Overlay File
9597
```bash
96-
Singularity> wget --no-check-certificate https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh
97-
Singularity> sh Miniforge3-Linux-x86_64.sh -b -p /ext3/miniforge3
98+
# Make sure you have the Singularity> prompt!
99+
wget --no-check-certificate https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh
100+
sh Miniforge3-Linux-x86_64.sh -b -p /ext3/miniforge3
98101
```
99102
Next, create a wrapper script at `/ext3/env.sh`:
100103
```bash
101-
Singularity> touch /ext3/env.sh
102-
Singularity> echo '#!/bin/bash' >> /ext3/env.sh
103-
Singularity> echo 'unset -f which' >> /ext3/env.sh
104-
Singularity> echo 'source /ext3/miniforge3/etc/profile.d/conda.sh' >> /ext3/env.sh
105-
Singularity> echo 'export PATH=/ext3/miniforge3/bin:$PATH' >> /ext3/env.sh
106-
Singularity> echo 'export PYTHONPATH=/ext3/miniforge3/bin:$PATH' >> /ext3/env.sh
104+
# Make sure you have the Singularity> prompt!
105+
touch /ext3/env.sh
106+
echo '#!/bin/bash' >> /ext3/env.sh
107+
echo 'unset -f which' >> /ext3/env.sh
108+
echo 'source /ext3/miniforge3/etc/profile.d/conda.sh' >> /ext3/env.sh
109+
echo 'export PATH=/ext3/miniforge3/bin:$PATH' >> /ext3/env.sh
110+
echo 'export PYTHONPATH=/ext3/miniforge3/bin:$PATH' >> /ext3/env.sh
107111
```
108112
Your `/ext3/env.sh` file should now contain the following:
109113
```bash
@@ -117,22 +121,25 @@ The wrapper script will activate your conda environment, to which you will be in
117121

118122
Next, activate your conda environment with the following:
119123
```bash
120-
Singularity> source /ext3/env.sh
124+
# Make sure you have the Singularity> prompt!
125+
source /ext3/env.sh
121126
```
122127

123128
### Install Packages to Miniforge Environment
124129
Now that your environment is activated, you can update and install packages:
125130
```bash
126-
Singularity> conda config --remove channels defaults
127-
Singularity> conda update -n base conda -y
128-
Singularity> conda clean --all --yes
129-
Singularity> conda install pip --yes
130-
Singularity> conda install ipykernel --yes # Note: ipykernel is required to run as a kernel in the Open OnDemand Jupyter Notebooks
131+
# Make sure you have the Singularity> prompt!
132+
conda config --remove channels defaults
133+
conda update -n base conda -y
134+
conda clean --all --yes
135+
conda install pip --yes
136+
conda install ipykernel --yes # Note: ipykernel is required to run as a kernel in the Open OnDemand Jupyter Notebooks
131137
```
132138
To confirm that your environment is appropriately referencing your Miniforge installation, try out the following:
133139
```bash
134-
Singularity> unset which
135-
Singularity> which conda
140+
# Make sure you have the Singularity> prompt!
141+
unset which
142+
which conda
136143
# output: /ext3/miniforge3/bin/conda
137144

138145
Singularity> which python
@@ -169,13 +176,13 @@ After it is running, you’ll be redirected to a compute node. From there, run s
169176
To create a kernel named my_env copy the template files to your home directory.
170177
Please note that kernel_template on Greene is stored under `/share/apps/mypy/src/kernel_template`
171178
```bash sh
172-
Singularity> mkdir -p ~/.local/share/jupyter/kernels
173-
Singularity> cd ~/.local/share/jupyter/kernels
174-
Singularity> cp -R /share/apps/kernel_template ./my_env # this should be the name of your Singularity env
179+
mkdir -p ~/.local/share/jupyter/kernels
180+
cd ~/.local/share/jupyter/kernels
181+
cp -R /share/apps/kernel_template ./my_env # this should be the name of your Singularity env
175182

176-
Singularity> cd ./my_env
183+
cd ./my_env
177184

178-
Singularity> ls
185+
ls
179186
#kernel.json logo-32x32.png logo-64x64.png python # files in the ~/.local/share/jupyter/kernels directory
180187
```
181188

0 commit comments

Comments
 (0)