diff --git a/docs/hpc/07_containers/01_intro.md b/docs/hpc/07_containers/01_intro.md index 56d5ae9b6b..9e5c38d7d5 100644 --- a/docs/hpc/07_containers/01_intro.md +++ b/docs/hpc/07_containers/01_intro.md @@ -1,12 +1,14 @@ # Custom Applications with Containers -## What is Singularity +## What is Singularity? Singularity is a container based Linux kernel workspace that works just like Docker. You can run pre-built programs in containers without having to worry about the pre-install environment. You can even run Docker containers with Singularity. Please see the [Singularity and Docker](https://docs.sylabs.io/guides/4.3/user-guide/singularity_and_docker.html) documentation by Syslabs for details about all the ways Singularity supports Docker. For a detailed introduction on Singularity, visit their [official site](https://apptainer.org/documentation/). +The Singularity we use is actually an alias for the name Apptainer. When we transition to the new Torch cluster you will only see reference to Apptainer. The reason for this is that there were two container projects with the name Singularity. The original free and open-source (FOSS) project and a closed-source corporate fork. The FOSS version has renamed itself to Apptainer to avoid confusion and this is the version we use. + ## Why do we use Singularity There are multiple reasons to use Singularity on the HPC clusters: -- **Security**: Singularity provides a layer of security as it does not require any root access on our clusters. This makes it safer against malware and bad scripts that might jeopardize the outer system. Thus we only support Singularity on our clusters(there are not other options such as Kubernetes or Docker on our clusters right now). -- **Containerization**: Singularity will run all your images(packaged and pre-built programs) inside of its containers, each container works like a small vm. They contain all the required environment and files of a single Linux kernel and you don't have to worry about any pre-installation nonsense. +- **Security**: Singularity provides a layer of security as it does not require any root access on our clusters. This makes it safer against malware and bad scripts that might jeopardize the outer system. Thus we only support Singularity on our clusters (there are not other options such as Kubernetes or Docker on our clusters right now). +- **Containerization**: Singularity will run all your images (packaged and pre-built programs) inside of its containers, each container works like a small vm. They contain all the required environment and files of a single Linux kernel and you don't have to worry about any pre-installation nonsense. - **Inter-connectivity**: Containers are able to talk to each other, as well as the home system, so while each container has its own small space, they are still a part of a big interconnected structure. Thus enabling you to connect your programs. - **Accessibility**: Probably the most important feature of all, Singularity allows you to run your program in 2 to 3 simple steps, as shown below. @@ -26,7 +28,7 @@ $ singularity pull $ singularity build .simg:rw # the image name can be a local image or an image from a hub ``` -We add the :rw tag at the end of the .simg to explicitly give it "read and write" permissions while building. +We add the `:rw` tag at the end of the `.simg` file to explicitly give it "read and write" permissions while building. ![singularity2](./static/singularity2.png) @@ -40,9 +42,9 @@ $ singularity run .simg:ro $ ./.simg:ro ``` -Unlike in the build phase, we add the :ro tag which means "read only" - as we are now just executing the image, not building it, and thus do not need it to be written. +Unlike in the build phase, we add the `:ro` tag which means "read only" - as we are now just executing the image, not building it, and thus do not need it to be written. :::warning -Writing access causes the Singularity image to be locked and it can become inaccessible while it is in read/write mode, so read only mode is best for executing commands. +Write access causes the Singularity image to be locked and it can become inaccessible while it is in read/write mode, so read only mode is best for executing commands. ::: running this would yield a menu for output: @@ -56,8 +58,8 @@ singularity shell .simg:ro ``` ![singularity4](./static/singularity4.png) -::: +:::tip #### Run commands outside the container You can run commands for the container using exec arguments without actually going into the container ```sh @@ -68,6 +70,7 @@ $ singularity exec .simg:ro Example: ![singularity5](./static/singularity5.png) +::: That's it! Now you're good to go and can just use these simple steps to run singularity images and run your programs. @@ -76,13 +79,13 @@ For full information and documentation please visit [Singularity](https://sylabs ## How to Create a Singularity Container So what if you want to create an image from your container and save it for a rainy day? -The instructions are [here](https://apptainer.org/docs/user/latest/build_a_container.html) for your convenience, read through them to create your own Singularity container and package it into an image! +Apptainer documentation has [instructions for building containers](https://apptainer.org/docs/user/latest/build_a_container.html) for your convenience. Please read through them to create your own Singularity container and package it into an image! -For those that know how docker containers are built, you can build docker containers using the information here and upload them onto docker hub and pulling them using Singularity. Singularity supports all docker images! +Similarly, you can build docker containers using the information from [Docker's documentation](https://docs.docker.com/get-started/docker-concepts/building-images/). You can then upload them onto docker hub and pull them using Singularity. Singularity supports all docker images! ## Singularity vs Docker Why are there so many mentions of Docker? The reason is that Singularity is essentially the same as Docker and you don't need to relearn Singularity if you already have experience with Docker. Now let's get into some pros and cons between the two programs. -- Docker is more accepted commercially than Singularity. You can download and run Docker on your own computer with any operating system and build containers with ease while Singularity is used in a more academic setting. Singularity only supports Linux operating systems and cannot run on a Windows Linux Kernel(your Windows Ubuntu), so it is much more limited. +- Docker is more accepted commercially than Singularity. You can download and run Docker on your own computer with any operating system and build containers with ease while Singularity is used in a more academic setting. Singularity only supports Linux operating systems and cannot run on a Windows Linux Kernel (your Windows Ubuntu), so it is much more limited. - However, Docker requires root or admin access for the operating system it deploys on, and our clusters do not offer that access to any software that requires this criteria. Thus Docker is not available on the clusters and Singularity is. - A silver lining in all of this is that Singularity fully supports Docker images and you can do everything in Docker and push your image to Docker Hub and pull them on the clusters. Thus making sure that you don't need to relearn Singularity all over again and can just use it through the simplest of commands in this wiki. diff --git a/docs/hpc/07_containers/03_singularity_with_conda.md b/docs/hpc/07_containers/03_singularity_with_conda.md index bd739d56bb..dcb191c667 100644 --- a/docs/hpc/07_containers/03_singularity_with_conda.md +++ b/docs/hpc/07_containers/03_singularity_with_conda.md @@ -1,8 +1,5 @@ # Singularity with Conda -## What is Singularity? -Singularity is a free, cross-platform and open-source program that creates and executes containers on the HPC clusters. Containers are streamlined, virtualized environments for specific programs or packages. Singularity is an industry standard tool to utilize containers in HPC environments. Containers allow for the support of highly specific environments and further increase scientific reproducibility and portability. Using Singularity containers, researchers can work in the reproducible containerized environments of their choice can easily tailor them to their needs. - ## Using Singularity Overlays for Miniforge (Python & Julia) ### Preinstallation Warning :::warning @@ -35,49 +32,49 @@ The above code automatically makes your environment look for the default shared ### Miniforge Environment PyTorch Example [Conda environments](https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html) allow users to create customizable, portable work environments and dependencies to support specific packages or versions of software for research. Common conda distributions include Anaconda, Miniconda and Miniforge. Packages are available via "channels". Popular channels include "conda-forge" and "bioconda". In this tutorial we shall use [Miniforge](https://github.com/conda-forge/miniforge) which sets "conda-forge" as the package channel. Traditional conda environments, however, also create a large number of files that can cut into quotas. To help reduce this issue, we suggest using [Singularity](https://docs.sylabs.io/guides/4.1/user-guide/), a container technology that is popular on HPC systems. Below is an example of how to create a pytorch environment using Singularity and Miniforge. -Create a directory for the environment +Create a directory for the environment: ```bash [NetID@log-1 ~]$ mkdir /scratch//pytorch-example [NetID@log-1 ~]$ cd /scratch//pytorch-example ``` -Copy an appropriate gzipped overlay images from the overlay directory. You can browse available images to see available options +Copy an appropriate gzipped overlay images from the overlay directory. You can browse available images to see available options: ```bash [NetID@log-1 pytorch-example]$ ls /scratch/work/public/overlay-fs-ext3 ``` -In this example we use overlay-15GB-500K.ext3.gz as it has enough available storage for most conda environments. It has 15GB free space inside and is able to hold 500K files +In this example we use `overlay-15GB-500K.ext3.gz` as it has enough available storage for most conda environments. It has 15GB free space inside and is able to hold 500K files You can use another size as needed. ```bash [NetID@log-1 pytorch-example]$ cp -rp /scratch/work/public/overlay-fs-ext3/overlay-15GB-500K.ext3.gz . [NetID@log-1 pytorch-example]$ gunzip overlay-15GB-500K.ext3.gz ``` -Choose a corresponding Singularity image. For this example we will use the following image +Choose a corresponding Singularity image. For this example we will use the following image: ```bash /scratch/work/public/singularity/cuda12.1.1-cudnn8.9.0-devel-ubuntu22.04.2.sif ``` -For Singularity image available on nyu HPC greene, please check the singularity images folder +For Singularity image available on nyu HPC Greene, please check the singularity images folder: ```sh [NetID@log-1 pytorch-example]$ ls /scratch/work/public/singularity/ ``` For the most recent supported versions of PyTorch, please check the [PyTorch website](https://pytorch.org/get-started/locally/). -Launch the appropriate Singularity container in read/write mode (with the :rw flag) +Launch the appropriate Singularity container in read/write mode (with the :rw flag): ```sh singularity exec --overlay overlay-15GB-500K.ext3:rw /scratch/work/public/singularity/cuda12.1.1-cudnn8.9.0-devel-ubuntu22.04.2.sif /bin/bash ``` The above starts a bash shell inside the referenced Singularity Container overlaid with the 15GB 500K you set up earlier. This creates the functional illusion of having a writable filesystem inside the typically read-only Singularity container. -Now, inside the container, download and install miniforge to `/ext3/miniforge3` +Now, inside the container, download and install miniforge to `/ext3/miniforge3`: ```bash Singularity> wget --no-check-certificate https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh Singularity> bash Miniforge3-Linux-x86_64.sh -b -p /ext3/miniforge3 # rm Miniforge3-Linux-x86_64.sh # if you don't need this file any longer ``` -Next, create a wrapper script /ext3/env.sh using a text editor, like nano. +Next, create a wrapper script /ext3/env.sh using a text editor, like nano: ```sh Singularity> touch /ext3/env.sh Singularity> nano /ext3/env.sh @@ -99,7 +96,7 @@ Activate your conda environment with the following: Singularity> source /ext3/env.sh ``` -If you have the "defaults" channel enabled, please disable it with +If you have the "defaults" channel enabled, please disable it with: ```bash Singularity> conda config --remove channels defaults ``` @@ -133,22 +130,25 @@ Singularity> exit #### Install packages -You may now install packages into the environment with either the pip install or conda install commands. +You may now install packages into the environment with either the `pip install` or `conda install` commands. + +The login nodes restrict memory to 2GB per user, which may cause some large packages to crash. For this reason, please start an interactive job with adequate compute and memory resources to install packages: -First, start an interactive job with adequate compute and memory resources to install packages. The login nodes restrict memory to 2GB per user, which may cause some large packages to crash. ```sh [NetID@log-1 pytorch-example]$ srun --cpus-per-task=2 --mem=10GB --time=04:00:00 --pty /bin/bash # wait to be assigned a node +``` +After it is running, you’ll be redirected to a compute node. From there, run singularity to setup on conda environment, same as you were doing on login node: + +```sh [NetID@cm001 pytorch-example]$ singularity exec --overlay overlay-15GB-500K.ext3:rw /scratch/work/public/singularity/cuda12.1.1-cudnn8.9.0-devel-ubuntu22.04.2.sif /bin/bash Singularity> source /ext3/env.sh # activate the environment ``` -After it is running, you’ll be redirected to a compute node. From there, run singularity to setup on conda environment, same as you were doing on login node. - We will install PyTorch as an example: ```sh Singularity> pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu116 @@ -160,14 +160,14 @@ For the latest versions of PyTorch please check the [PyTorch website](https://py You can see the available space left on your image with the following commands: ```sh -find /ext3 | wc -l +Singularity> find /ext3 | wc -l # output: should be something like: 77674 -du -sh /ext3 +Singularity> du -sh /ext3 # output should be something like: 6.5G /ext3 ``` -Now, exit the Singularity container and then rename the overlay image. Typing 'exit' and hitting enter will exit the Singularity container if you are currently inside it. You can tell if you're in a Singularity container because your prompt will be different, such as showing the prompt 'Singularity>' +Now, exit the Singularity container and then rename the overlay image. Typing `exit` and hitting `enter` will exit the Singularity container if you are currently inside it. You can tell if you're in a Singularity container because your prompt will be different, such as showing the prompt `Singularity>`: ```sh Singularity> exit [NetID@cm001 pytorch-example]$ mv overlay-15GB-500K.ext3 my_pytorch.ext3 @@ -180,7 +180,7 @@ Singularity> exit #output: 2.7.1+cu126 ``` :::note - the end ':ro' addition at the end of the pytorch ext3 image starts the image in read-only mode. To add packages you will need to use ':rw' to launch it in read-write mode. + the end `:ro` addition at the end of the pytorch ext3 image starts the image in read-only mode. To add packages you will need to use `:rw` to launch it in read-write mode. ::: ### Using your Singularity Container in a SLURM Batch Job @@ -229,12 +229,12 @@ singularity exec --nv \ You will notice that the singularity exec command features the `--nv` flag - this flag is required to pass the CUDA drivers from a GPU to the Singularity container. -Run the run-test.SBATCH script +Run the run-test.SBATCH script: ```sh [NetID@log-1 pytorch-example]$ sbatch run-test.SBATCH ``` -Check your SLURM output for results, an example is shown below +Check your SLURM output for results, an example is shown below: ```sh [NetID@log-1 pytorch-example]$ cat slurm-3752662.out @@ -263,7 +263,7 @@ Notice that it saves over 3GB of storage in this case, though your results may v #### Use a squashFS Image for Running Jobs -You can use squashFS images similarly to the ext3 images. +You can use squashFS images similarly to the ext3 images: ```sh [NetID@log-1 pytorch-example]$ singularity exec --overlay /scratch//pytorch-example/my_pytorch.sqf:ro /scratch/work/public/singularity/cuda12.1.1-cudnn8.9.0-devel-ubuntu22.04.2.sif /bin/bash -c 'source /ext3/env.sh; python -c "import torch; print(torch.__file__); print(torch.__version__)"' @@ -275,7 +275,7 @@ You can use squashFS images similarly to the ext3 images. If the first ext3 overlay image runs out of space or you are using a squashFS conda environment, but need to install a new package inside, please copy another writable ext3 overlay image to work together. -Open the first image in read only mode +Open the first image in read only mode: ```sh [NetID@log-1 pytorch-example]$ cp -rp /scratch/work/public/overlay-fs-ext3/overlay-2GB-100K.ext3.gz . [NetID@log-1 pytorch-example]$ gunzip overlay-2GB-100K.ext3.gz @@ -295,25 +295,25 @@ Please also keep in mind that once the overlay image is opened in default read-w ### Julia Singularity Image Singularity can be used to set up a Julia environment. -Create a directory for your julia work, such as `/scratch//julia`, and then change to your working directory to it. An example is shown below: +Create a directory for your Julia work, such as `/scratch//julia`, and then change to your working directory to it. An example is shown below: ```sh [NetID@log-1 NetID]$ mkdir /home//julia [NetID@log-1 NetID]$ cd /home//julia ``` -Copy an overlay image, such as the 2GB 100K overlay, which generally has enough storage for Julia packages. Once copied, unzip to the same folder, rename to julia-pkgs.ext3 +Copy an overlay image, such as the 2GB 100K overlay, which generally has enough storage for Julia packages. Once copied, unzip to the same folder, rename to julia-pkgs.ext3: ```sh [NetID@log-1 julia]$ cp -rp /scratch/work/public/overlay-fs-ext3/overlay-2GB-100K.ext3.gz . [NetID@log-1 julia]$ gunzip overlay-2GB-100K.ext3.gz [NetID@log-1 julia]$ mv overlay-2GB-100K.ext3 julia-pkgs.ext3 ``` -Copy the following wrapper script in the directory +Copy the following wrapper script in the directory: ```sh [NetID@log-1 julia]$ cp -rp /share/apps/utils/julia-setup/* . ``` -Now launch writable Singularity overlay to install packages +Now launch writable Singularity overlay to install packages: ```sh [NetID@log-1 julia]$ module purge [NetID@log-1 julia]$ module load knitro/12.3.0 @@ -326,7 +326,7 @@ julia> Pkg.add("KNITRO") julia> Pkg.add("JuMP") ``` -Now exit from the container to launch a read only version to test (example below) +Now exit from the container to launch a read only version to test (example below): ```julia [NetID@log-1 julia]$ ~/julia/my-julia _ @@ -364,7 +364,7 @@ julia> @NLobjective(m, Min, x1*(1-x2^2)) julia> optimize!(m) ``` -You can make the above code into a julia script to test batch jobs. Save the following as test-knitro.jl +You can make the above code into a Julia script to test batch jobs. Save the following as `test-knitro.jl`: ```julia using Pkg using JuMP, KNITRO @@ -376,7 +376,7 @@ m = Model(with_optimizer(KNITRO.Optimizer)) optimize!(m) ``` -You can add additional packages with commands like the one below. +You can add additional packages with commands like the one below: :::warning Please do not install new packages when you have Julia jobs running, this may create issues with your Julia installation ::: @@ -384,7 +384,7 @@ Please do not install new packages when you have Julia jobs running, this may cr [NetID@log-1 julia]$ ~/julia/my-julia-writable -e 'using Pkg; Pkg.add(["Calculus", "LinearAlgebra"])' ``` -Run a SLURM job to test with the following sbatch command (e.g. julia-test.SBATCH) +Run a SLURM job to test with the following sbatch command (e.g. julia-test.SBATCH): ```bash #!/bin/bash @@ -407,7 +407,7 @@ Then run the command with the following: [NetID@log-1 julia]$ sbatch julia-test.SBATCH ``` -Once the job completes, check the SLURM output (example below) +Once the job completes, check the SLURM output (example below): ```sh [NetID@log-1 julia]$ cat slurm-1022969.out @@ -476,7 +476,7 @@ Time spent in evaluations (secs) = 0.00000 ### Using CentOS 8 for Julia (for Module Compatibility) Building on the previous Julia example, this will demonstrate how to set up a similar environment using the Singularity CentOS 8 image for additional customization. Using the CentOS 8 overlay allows for the loading of modules installed on Greene, such as Knitro 12.3.0 -Copy overlay image +Copy overlay image: ```sh [NetID@log-1 julia]$ cp -rp /scratch/work/public/overlay-fs-ext3/overlay-2GB-100K.ext3.gz . [NetID@log-1 julia]$ gunzip overlay-2GB-100K.ext3.gz @@ -487,20 +487,20 @@ Copy overlay image The path in this example is `/scratch//julia/julia-pkgs.ext3` ::: -To use modules installed into `/share/apps` you can make two directories +To use modules installed into `/share/apps` you can make two directories: ```sh [NetID@log-1 julia]$ mkdir julia-compiled julia-logs ``` :::note -Now, in this example, the absolute paths are as follows +Now, in this example, the absolute paths are as follows: ```sh /scratch//julia/julia-compiled /scratch//julia/julia-logs ``` ::: -To launch Singularity with overlay images in writable mode to install packages +Launch Singularity with overlay images in writable mode to install packages: ```sh [NetID@log-1 julia]$ singularity exec \ --overlay /scratch//julia/julia-pkgs.ext3 \ @@ -511,7 +511,7 @@ To launch Singularity with overlay images in writable mode to install packages /bin/bash ``` -Implement a wrapper script /ext3/env.sh +Implement a wrapper script /ext3/env.sh: ```bash #/bin/bash @@ -523,7 +523,7 @@ module load knitro/12.3.0 module load julia/1.5.3 ``` -Load julia via the wrapper script and check that it loads properly +Load Julia via the wrapper script and check that it loads properly: ```sh Singularity> source /ext3/env.sh Singularity> which julia @@ -532,7 +532,7 @@ Singularity> julia --version # example output: julia version 1.5.3 ``` -Run julia to install packages +Run Julia to install packages: ```julia Singularity> julia julia> using Pkg @@ -552,7 +552,7 @@ m = Model(with_optimizer(KNITRO.Optimizer)) optimize!(m) ``` -Now implement a wrapper script named julia into ~/bin, the overlay image is in readonly mode +Now implement a wrapper script named `julia` into ~/bin, the overlay image is in readonly mode: ```bash #!/bin/bash @@ -576,12 +576,12 @@ julia $args " ``` -Make the wrapper executable +Make the wrapper executable: ```sh [NetID@log-1 julia]$ chmod 755 ~/bin/julia ``` -Test your installation with a SLURM job example. The following code has been put into a file called test-julia-centos.SBATCH +Test your installation with a SLURM job example. The following code has been put into a file called test-julia-centos.SBATCH: ```bash #!/bin/bash @@ -601,14 +601,14 @@ Run the above with the following: [NetID@log-1 julia]$ sbatch test-julia-centos.SBATCH ``` -Read the output (example below) +Read the output (example below): ```sh [NetID@log-1 julia]$ cat slurm-764085.out ``` #### Installing New Julia Packages Later -Implement another writable julia-writable with overlay image writable in order to install new Julia packages later +Implement another writable julia-writable with overlay image writable in order to install new Julia packages later: ```sh [NetID@log-1 julia]$ cd /home//bin [NetID@log-1 julia]$ cp -rp julia julia-writable @@ -636,22 +636,22 @@ julia $args " ``` -Check the writable image +Check the writable image: ```sh [NetID@log-1 julia]$ which julia-writable #example output: ~/bin/julia-writable ``` -Install packages to the writable image +Install packages to the writable image: ```sh [NetID@log-1 julia]$ julia-writable -e 'using Pkg; Pkg.add(["Calculus", "LinearAlgebra"])' ``` -If you do not need host packages installed in `/share/apps`, you can work with Singularity OS image +If you do not need host packages installed in `/share/apps`, you can work with Singularity OS image: ```sh /scratch/work/public/singularity/ubuntu-20.04.1.sif ``` -download Julia installation package from [https://julialang-s3.julialang.org/bin/linux/x64/1.5/julia-1.5.3-linux-x86_64.tar.gz](https://julialang-s3.julialang.org/bin/linux/x64/1.5/julia-1.5.3-linux-x86_64.tar.gz) +Download Julia installation package from [https://julialang-s3.julialang.org/bin/linux/x64/1.5/julia-1.5.3-linux-x86_64.tar.gz](https://julialang-s3.julialang.org/bin/linux/x64/1.5/julia-1.5.3-linux-x86_64.tar.gz) -install Julia to `/ext3`, setup PATH properly. It will be easy to move to other servers in future. +Install Julia to `/ext3`, setup PATH properly. It will be easy to move to other servers in future.