diff --git a/docs/hpc/03_storage/06_best_practices.md b/docs/hpc/03_storage/06_best_practices.md index 3533e41235..9a0baa091b 100644 --- a/docs/hpc/03_storage/06_best_practices.md +++ b/docs/hpc/03_storage/06_best_practices.md @@ -49,4 +49,3 @@ If you create a conda or python environment in you home directory, this can eat ::: Please review the [Package Management section](../06_tools_and_software/01_software_on_greene.md#package-management-for-r-python--julia-and-conda-in-general) of the [Greene Software Page](../06_tools_and_software/01_software_on_greene.md). - diff --git a/docs/hpc/06_tools_and_software/01_software_on_greene.md b/docs/hpc/06_tools_and_software/01_software_on_greene.md index 77b886e7fb..d63c3da192 100644 --- a/docs/hpc/06_tools_and_software/01_software_on_greene.md +++ b/docs/hpc/06_tools_and_software/01_software_on_greene.md @@ -7,7 +7,7 @@ There are different types of software packages available - This includes the licensed software listed below - Singularity Containers - You can find those already built and ready to use, at location `/scratch/work/public/singularity/` - - For more information on running software with Singularity, [click here](../07_containers/01_intro.md). + - For more information on running software with Singularity, See our [Containers Intro](../07_containers/01_intro.md). - Python/R/Julia packages can be installed by a user If you need another linux program installed, please contact us at [hpc@nyu.edu](mailto:hpc@nyu.edu) @@ -22,7 +22,7 @@ Below is a list of modules and their associated functions: | Command | Function | |-----------------------------------|-----------------------------------------------------------------------| | module unload `` | unload a module | -| module show `` | see exactly what effect loading the module will have with | +| module show `` | see exactly what effect loading the module will have | | module purge | remove all loaded modules from your environment | | module load `` | load a module | | module whatis `` | find out more about a software package | @@ -32,10 +32,9 @@ Below is a list of modules and their associated functions: ## Package Management for R, Python, & Julia, and Conda in general -- [Conda environments (Python, R)](../06_tools_and_software/02_conda_environments.md) -- [Using virtual environments for Python](../06_tools_and_software/03_python_packages_with_virtual_environments.md) -- [Managing R packages with renv](../06_tools_and_software/04_r_packages_with_renv.md) -- [Singularity with Miniconda](../07_containers/03_singularity_with_conda.md) +- [Conda environments (Python, R)](../06_tools_and_software/04_conda_environments.mdx) +- [Using virtual environments for Python](../06_tools_and_software/02_python_packages_with_virtual_environments.mdx) +- [Managing R packages with renv](../06_tools_and_software/03_r_packages_with_renv.mdx) ## Examples of software usage on Greene Examples can be found under `/scratch/work/public/examples/` and include the following @@ -68,7 +67,7 @@ Schrödinger can be used for non-commercial, academic purposes ONLY. To load Schrodinger module execute ```sh -$ module load schrodinger/2021-1 +$ module load schrodinger/2024.4 ``` #### Using SCHRODINGER on NYU Lab Computers @@ -158,7 +157,7 @@ In order to check what Comsol licenses are available on Greene use `comsol_licen Several versions of COMSOL are available on the HPC cluster. To use COMSOL on the Greene HPC cluster, please load the relevant module in your batch job submission script: ```sh -module load comsol/5.6.0.280 +module load comsol/6.3 ``` To submit a COMSOL job in a parallel fashion, running on multiple processing cores, follow the steps below: 1. Create a directory on "scratch" as given below. @@ -182,7 +181,7 @@ Mathematica is a general computing environment with organizing algorithmic, visu To run Mathematica on the Greene HPC cluster, please load the relevant module in your batch job submission script: ```sh -module load mathematica/12.1.1 +module load mathematica/14.1.0 ``` :::note In the example below the module is loaded already in the sbatch script. @@ -254,8 +253,7 @@ MATLAB can be used for non-commercial, academic purposes. There are several versions of Matlab available on the cluster and the relevant version can be loaded. ```sh -module load matlab/2020b -module load matlab/2021a +module load matlab/2024b ``` In order to run MATLAB interactively on the cluster, [start an interactive slurm job](../05_submitting_jobs/01_slurm_submitting_jobs.md), load the matlab module and launch an interactive matlab session in the terminal. diff --git a/docs/hpc/06_tools_and_software/02_conda_environments.md b/docs/hpc/06_tools_and_software/02_conda_environments.md deleted file mode 100644 index 000b02edbb..0000000000 --- a/docs/hpc/06_tools_and_software/02_conda_environments.md +++ /dev/null @@ -1,260 +0,0 @@ -# Conda Environments (Python, R) - -## What is Conda? -Package, dependency and environment management for any language—Python, R, Ruby, Lua, Scala, Java, JavaScript, C/ C++, FORTRAN, and more. - -Please find more information at this link: [https://docs.conda.io/en/latest/](https://docs.conda.io/en/latest/) - -Conda provides a great way to install packages that are already compiled, so you don't need to go over the long compilation process. If a package you need is not available, you can install it (and compile it when needed) using pip (Python) or install.packages (R). - -:::note -Reproducibility: -One of the ways to ensure the reproducibility of your results is to have an independent conda environment in the directory of each project (one of the options shown below). This will also keep conda environment files away from your `/home/$USER` directory. -::: - -## Advantages/disadvantages of using Conda -### Advantages - -- A lot of pre-compiled packages (fast and easy to install) -- Note for Python: pip also offers pre-compiled packages (wheels). List can be found here https://pythonwheels.com. However, Conda has a significantly larger number of pre-compiled packages. -- Compiled packages use highly efficient Intel Math Kernel Library (MKL) library - -### Disadvantages - -- Conda does not take advantage of packages already installed in the system (while [virtualenv and venv](./03_python_packages_with_virtual_environments.md) do) -- As you will see below, you may need to do additional steps to keep track of all installed packages (including those installed by pip and/or install.packages) - -## Initializing Conda -Load anaconda module -```sh -module purge -module load anaconda3/2020.07 -``` - -Conda init can create problems with package installation, so we suggest using `source activate` instead of `conda activate`, even though conda activate is considered a best practice by the Anaconda developers. - -### Automatic deletion of your files -This page describes the installation of packages on /scratch. One has to remember, though, that files stored in the HPC scratch file system are subject to the HPC Scratch old file purging policy: Files on the /scratch file system that have not been accessed for 60 or more days will be purged (read more about [Data Management](../03_storage/01_intro_and_data_management.mdx). - -Thus you can consider the following options - -- Reinstall your packages if some of the files get deleted - - You can do this manually - - You can do this automatically. For example, within a workflow of a pipeline software like [Nextflow](https://www.nextflow.io/) -- Pay for "Research Project Space" - read more [here](../03_storage/05_research_project_space.mdx) -- Use Singularity and install packages within a corresponding overlay file - read more [here](../07_containers/03_singularity_with_conda.md) - -## Python -Load anaconda module -```sh -module purge -module load anaconda3/2020.07 -``` -:::tip -Keep your program/project in `/scratch` and create conda environment using '-p' parameter. This will keep all the files inside the project's directory, instead of putting in in your `/home/$USER` -::: - -```sh -conda create -p ./penv python=3 ## environment will be created in project directory -conda activate ./penv -``` -Also, you need to create a symbolic link, so conda will download files for packages to be installed into scratch, not your home directory. -```sh -mkdir /home//.conda -mkdir /scratch//conda_pkgs -ln -s /scratch//conda_pkgs /home//.conda/pkgs -``` -[Install pre-compiled packages available in conda](https://anaconda.org/anaconda/repo) -```sh -conda install -c anaconda pandas -``` - -Other packages may be installed (and compiled when needed) using pip -```sh -pip install -``` -:::note -Conda and packages install by default to `~/.local/lib/python` -::: - -If you did use `'pip install --user'` to install some packages (without conda or other virtual environment), they will be available in `~/.local/lib/python` - -:::warning -***The primary takeaway:*** - -Let say you have tornado v.6 installed in `~/.local/lib/python`, and tornado v.5 installed by `conda install`. - -When you will do `conda activate` you will have tornado v.6 available!! Not v.5!! - -(this behaviour is the same for packages installed by to `~/.local/lib/python` before or after you create your conda environment) - -`pip freeze` will give v.6 - -`conda list` will give v.5 - -***Solution*** - -To overcome this, do `export PYTHONNOUSERSITE=True` after conda activate -::: - -## R -Load anaconda module -```sh -module load anaconda3/2020.07 -``` -:::tip -Keep your program/project in `/scratch` and create conda environment using '-p' parameter. This will keep all the files inside the project's directory, instead of putting them in your `/home/$USER` -::: - -```sh -conda create -p ./renv r=3.5 ## environment will be created in project directory -## OR -conda create -c conda-forge -p ./penv r-base=3.6.3 ## environment will be created in project directory -conda activate ./renv -``` - -Install pre-compiled packages available in conda: - -[https://docs.anaconda.com/anaconda/packages/r-language-pkg-docs/](https://docs.anaconda.com/anaconda/packages/r-language-pkg-docs/) - -```sh -conda install -c r r-dplyr -``` - -Other packages may be installed (and compiled) using install.packages() -```sh -install.packages("") -``` - -## Reproducibility -Packages installed only using conda - -Save a list of packages (so you are able to report environment in publication, and to restore/reproduce env on another machine at any time) - -```sh -# save -conda list --export > requirements.txt -# restore -conda create -p ./penv --file requirements.txt -``` -:::note -This will not list packages installed by `pip` or `install.packages()` -::: - -If you installed extra packages using pip (Python) - -In this you can use -```sh -export PYTHONNOUSERSITE=True ## to ignore packages in ~/.local/lib/python -# save -conda list --export > conda_requirements.txt -pip freeze > pip_requirements.txt -# restore -conda create -p ./penv --file conda_requirements.txt -pip install -r pip_requirements.txt -``` - -:::note -Alternatively, you can use conda env export > all_requirements.txt, which will save both: packages installed by conda and by pip. -::: - -However, this may fail if your conda environment is created as a sub-directory of your project's directory (which we recommend) - -Installed extra packages using install.packages? (R) - -Usecase: You need packages not available in conda channels, and want to use install.packages. - -Command `conda list --export` will not include packages installed by "install.packages". So, do not use `conda install` at all. To have reproducibility in this case you need to use Conda and renv together, as described below - -Conda + pakcrat: specific version of R and install.packages (R) - -- use conda to install version of R you need -- do not use 'conda install' at all -- use renv -- install all the packages using install.packages -- use [renv as described here](../06_tools_and_software/04_r_packages_with_renv.md) to keep track of the environment - -In order for conda + renv to work, you need to add following steps: - -- After you activate conda AND before loading R - ```sh - export R_RENV_DEFAULT_LIBPATHS=/renv/lib/x86_64-conda_cos6-linux-gnu// - ``` -- Start R and execute - ```sh - .libPaths(c(.libPaths(), Sys.getenv("R_RENV_SYSTEM_LIBRARY"))) - ``` - -## Use conda env in a batch script -The part of the batch script which will call the command shall look like (replace `` to an appropriate value) - -### Python - -#### Single node -```bash -#!/bin/bash -#SBATCH --job-name=test -#SBATCH --nodes=1 -#SBATCH --cpus-per-task=1 -#SBATCH --ntasks-per-node=4 -#SBATCH --mem=8GB -#SBATCH --time=1:00:00 -module purge; -module load anaconda3/2020.07; -export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK; -source /share/apps/anaconda3/2020.07/etc/profile.d/conda.sh; -conda activate ./penv; -export PATH=./penv/bin:$PATH; -python python_script.py -``` - -#### Multiple nodes, using MPI -```sh -mpiexec --mca bash -c "module purge; -export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK; -module load anaconda3/2020.07; -source /share/apps/anaconda3/2020.07/etc/profile.d/conda.sh; -conda activate ./penv; -export PATH=./penv/bin:$PATH; -python python_script.py" -``` - -### R (conda packages only) -```bash -#!/bin/bash -#SBATCH --job-name=test -#SBATCH --nodes=1 -#SBATCH --cpus-per-task=1 -#SBATCH --ntasks-per-node=4 -#SBATCH --mem=8GB -#SBATCH --time=1:00:00 -module purge; -module load anaconda3/2020.07; -export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK; -source /share/apps/anaconda3/2020.07/etc/profile.d/conda.sh; -conda activate ./renv; -export PATH=./renv/bin:$PATH; -Rscript r_script.R -``` - -#### Multiple nodes, using MPI -```sh -mpiexec --mca bash -c "module purge; -export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK; -module load anaconda3/2020.07; -source /share/apps/anaconda3/2020.07/etc/profile.d/conda.sh; -conda activate ./renv; -export PATH=./renv/bin:$PATH; -Rscript r_script.R" -``` - -### R (conda with renv combination) - -In this case, when you use sbatch you would activate conda in sbatch script, and R script will pickup packages installed in renv -```sh -module purge -module load anaconda3/2020.07 -source /share/apps/anaconda3/2020.07/etc/profile.d/conda.sh -conda activate ./renv -Rscript test.R -``` diff --git a/docs/hpc/06_tools_and_software/03_python_packages_with_virtual_environments.md b/docs/hpc/06_tools_and_software/02_python_packages_with_virtual_environments.mdx similarity index 73% rename from docs/hpc/06_tools_and_software/03_python_packages_with_virtual_environments.md rename to docs/hpc/06_tools_and_software/02_python_packages_with_virtual_environments.mdx index a6cfacf883..ca6dcaada6 100644 --- a/docs/hpc/06_tools_and_software/03_python_packages_with_virtual_environments.md +++ b/docs/hpc/06_tools_and_software/02_python_packages_with_virtual_environments.mdx @@ -16,15 +16,16 @@ module load python/intel/3.8.6 ``` ## Automatic deletion of your files -This page describes the installation of packages on /scratch. One has to remember, though, that files stored in the HPC scratch file system are subject to the HPC Scratch old file purging policy: Files on the /scratch file system that have not been accessed for 60 or more days will be purged (read [more](../03_storage/01_intro_and_data_management.mdx)). +This page describes the installation of packages on /scratch. One has to remember, though, that files stored in the HPC scratch file system are subject to the HPC Scratch old file purging policy:
+**Files on the /scratch file system that have not been accessed for 60 or more days will be purged** (see [HPC Storage](../03_storage/01_intro_and_data_management.mdx) for details). Thus you can consider the following options - Reinstall your packages if some of the files get deleted - You can do this manually - - You can do this automatically. For example, within a workflow of a pipeline software like [Nextflow](https://www.nextflow.io/) -- Pay for "Research Project Space" - read more [here](../03_storage/05_research_project_space.mdx) -- Use Singularity and install packages within a corresponding overlay file - read more [here](../07_containers/03_singularity_with_conda.md) + - You can also do this automatically. For example, within a workflow of pipeline software like [Nextflow](https://www.nextflow.io/) +- Pay for "Research Project Space" - see details on the [Research Project Space](../03_storage/05_research_project_space.mdx) page. +- Use Singularity and install packages within a corresponding overlay file - See instructions on our [Singularity with Conda](../07_containers/03_singularity_with_conda.md) page. ## Create virtual environment It is advisable to create private environment inside the project directory. This boosts reproducibility and does not use space in `/home/$USER` @@ -34,13 +35,11 @@ It is advisable to create private environment inside the project directory. This Since Python 3.3, a subset of it has been integrated into the standard library under the venv module. -Note: you may need to install virtualenv first, if it is not yet installed ([instructions](https://virtualenv.pypa.io/en/latest/installation.html)) +You can create new virtual environment in two ways: -Now create new virtual environment in current directory - -- Empty -- OR +- empty - inherit all packages from those installed on HPC already (and available in PATH after you load python module) + ```sh ## created directory for your project and cd there mkdir /scratch/$USER/my_project @@ -54,15 +53,14 @@ virtualenv venv --system-site-packages ``` ### venv -[venv](https://docs.python.org/3/library/venv.html) is package shipped with Python3. It provides subset of options available in virtualenv tool ([link](https://virtualenv.pypa.io/en/latest/)). +[venv](https://docs.python.org/3/library/venv.html) is package shipped with Python. It provides subset of options available in virtualenv tool ([link](https://virtualenv.pypa.io/en/latest/)). ```sh -python3 -m venv venv +python -m venv venv ``` Create new virtual environment in current directory -- Empty -- OR +- empty - inherit all packages from those installed on HPC already (and available in PATH after you load python module) ```sh ## created directory for your project and cd there @@ -71,10 +69,10 @@ cd /scratch/$USER/my_project ##EMPTY ## (use venv command to create environment called "venv") -python3 -m venv venv +python -m venv venv ## Inhering all packages -python3 -m venv venv --system-site-packages +python -m venv venv --system-site-packages ``` ## Install packages. Keep things reproducible @@ -108,8 +106,8 @@ python python_script.py If you use mpi ```sh -mpiexec bash -c "module purge; - source venv/bin/activate; - export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK; - python python_script.py" +mpiexec bash -c "module purge; + source venv/bin/activate; + export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK; + python python_script.py" ``` diff --git a/docs/hpc/06_tools_and_software/04_r_packages_with_renv.md b/docs/hpc/06_tools_and_software/03_r_packages_with_renv.mdx similarity index 67% rename from docs/hpc/06_tools_and_software/04_r_packages_with_renv.md rename to docs/hpc/06_tools_and_software/03_r_packages_with_renv.mdx index 19b9e3e52e..461b608aeb 100644 --- a/docs/hpc/06_tools_and_software/04_r_packages_with_renv.md +++ b/docs/hpc/06_tools_and_software/03_r_packages_with_renv.mdx @@ -1,5 +1,19 @@ # R Packages with renv +## Is this needed: +:::tip +If you would like to use conda with `renv` you need to add following steps: + +- After you activate conda AND before loading R + ```sh + export R_RENV_DEFAULT_LIBPATHS=/renv/lib/x86_64-conda_cos6-linux-gnu// + ``` +- Start R and execute + ```sh + .libPaths(c(.libPaths(), Sys.getenv("R_RENV_SYSTEM_LIBRARY"))) + ``` +::: + You may use the renv R package to create a personal R Project environment for R packages. Documentation on renv can be found on the [RStudio site](https://rstudio.github.io/renv/). ## Setup @@ -7,21 +21,22 @@ Say your R code is in directory /scratch/$USER/projects/project1 ```sh cd /scratch/$USER/projects/project1 module purge -module load r/gcc/4.4.0 +module load r/gcc/4.5.0 R ``` ### Automatic deletion of your files -This page describes the installation of packages on /scratch. One has to remember, though, that files stored in the HPC scratch file system are subject to the HPC Scratch old file purging policy: Files on the `/scratch` file system that have not been accessed for 60 or more days will be purged (read [more](../03_storage/01_intro_and_data_management.mdx)). +This page describes the installation of packages on /scratch. One has to remember, though, that files stored in the HPC scratch file system are subject to the HPC Scratch old file purging policy:
+**Files on the `/scratch` file system that have not been accessed for 60 or more days will be purged.** Please see details at [HPC Storage](../03_storage/01_intro_and_data_management.mdx). Thus you can consider the following options: - Reinstall your packages if some of the files get deleted - You can do this manually - You can do this automatically. For example, within a workflow of a pipeline software like [Nextflow](https://www.nextflow.io/) -- Pay for "Research Project Space" - read more [here](../03_storage/05_research_project_space.mdx) -- Use Singularity and install packages within a corresponding overlay file - read more [here](../07_containers/04_squash_file_system_and_singularity.md) +- Pay for "Research Project Space" - Details available at [Research Project Space](../03_storage/05_research_project_space.mdx) +- Use Singularity and install packages within a corresponding overlay file - Details available at [Squash File System and Singularity](../07_containers/04_squash_file_system_and_singularity.md) ### Cache directory setup By default, renv will cache package installation files to your home directory (most likely either in `~/.local/share/renv` or `~/.cache/R/renv/` or something similar). @@ -32,13 +47,13 @@ To avoid filling up your home directory, we advise to set up path to alternative ```sh mkdir -p /scratch/$USER/.cache/R/renv ``` -- Create a file in you project directory named .Renviron and put the following in in the file. It is the R project directory (`/scratch/$USER/projects/project1`) in this example. +- Create a file in you project directory named `.Renviron` and put the following in in the file. It is the R project directory (`/scratch/$USER/projects/project1`) in this example. ```sh RENV_PATHS_ROOT=/scratch//.cache/R/renv ``` ### Init renv -The renv package is already installed for module `r/gcc/4.4.0`. You need to install it yourself if you use other R module version +The renv package is already installed for module `r/gcc/4.5.0`. You need to install it yourself if you use other R module version ```R ## Do this if renv is not available (already installed for r/gcc/4.4.0) # install.packages("renv") ## By default this will install renv package into a sub-directory within your home directory @@ -49,10 +64,10 @@ renv::init(".") - Restart R for renv to take effect. Once you start R, your renv environment will be loaded automatically. ```R -R version 4.4.0 (2024-04-24) -- "Puppy Cup" +R version 4.5.0 (2025-04-11) -- "How About a Twenty-Six" ... -* Project '/scratch/$USER/projects/project1' loaded. [renv 1.0.7] +* Project '/scratch/$USER/projects/project1' loaded. [renv 1.1.4] ``` ### Check @@ -141,42 +156,47 @@ Calling `renv::init()` will also write out the infrastructure necessary to autom If you’d like to initialize a project without attempting dependency discovery and installation – that is, you’d prefer to manually install the packages your project requires on your own – you can use `renv::init(bare = TRUE)` to initialize a project with an empty project library. ## Use with sbatch -When you launch a job with sbatch, R will check if there is renv directory, and if renv is on it will pick up packages, installed using renv in the current directory. +When you launch a job with sbatch, R will check if there is an `renv` directory, and if `renv` is on it will pick up packages, installed using `renv` in the current directory. -Before you launch sbatch job, you need to make sure your project renv environment is ready, as outlined in the previous section. +Before you launch an sbatch job, you need to make sure your project `renv` environment is ready, as outlined in the previous section. -### Store and Share your R Project's R version and R Package Versions -#### Reproduce Environment -If you already have file renv.lock or bundle file skip step 1 +## Store and Share your R Project's R version and R Package Versions +### Reproduce Environment +If you already have file `renv.lock` or bundle file skip step 1 1. In the original location (your own laptop for example) go to project directory and execute -(Make sure the whole path to project directory and names of your script files don't have empty spaces!) +:::warning +Make sure the whole path to project directory and names of your script files don't have empty spaces! +::: ```R R # install.packages("renv") ## if needed renv::init() renv::snapshot() ``` -2. Take file renv.lock and copy it to a new location for the project -3. At the new location - restore environment: go to directory of the project and execute. (Make sure version of R is the same) +2. Take file `renv.lock` and copy it to a new location for the project +3. At the new location - restore environment: go to directory of the project and execute. +:::warning +Make sure version of R is the same +::: ```R ## Reproduce environment module purge -module load r/gcc/4.4.0 +module load r/gcc/4.5.0 R renv::restore() renv::init() ``` -renv will install/compile what is needed on any system (Linux, Windows, etc). You can share your code with other researchers no matter what system they use. However, you should be careful that the same version of R is used between systems. - -#### What to save/publish/commit with Git -In order to have your work reproducible by you or/and others, save and/or commit your code in git, please including +`renv` will install/compile what is needed on any system (Linux, Windows, etc). You can share your code with other researchers no matter what system they use. However, you should be careful that the same version of R is used between systems, as mentioned above. -renv.lock (which lists all packages and versions that you use including the version of R) +### What to save/publish/commit with Git +In order to have your work reproducible by you and/or others: +- save and/or commit your code in git +- be sure to include `renv.lock` (which lists all packages and versions that you use including the version of R) -#### Migrating from Packrat -The renv package has replaced the now deprecated Packrat package. The `renv::migrate()` function makes it possible to migrate projects from Packrat to renv. See the ?migrate documentation for more details. In essence, calling `renv::migrate("")` will be enough to migrate the Packrat library and lockfile such that they can then be used by renv. +### Migrating from Packrat +The `renv` package has replaced the now deprecated Packrat package. The `renv::migrate()` function makes it possible to migrate projects from Packrat to `renv`. See the ?migrate documentation for more details. In essence, calling `renv::migrate("")` will be enough to migrate the Packrat library and lockfile such that they can then be used by renv. -### Useful links +## Useful links - [https://rstudio.github.io/renv/articles/renv.html](https://rstudio.github.io/renv/articles/renv.html) diff --git a/docs/hpc/06_tools_and_software/04_conda_environments.mdx b/docs/hpc/06_tools_and_software/04_conda_environments.mdx new file mode 100644 index 0000000000..72b3a970af --- /dev/null +++ b/docs/hpc/06_tools_and_software/04_conda_environments.mdx @@ -0,0 +1,349 @@ +# Conda Environments (Python, R) + +## What is Conda? +Package, dependency and environment management for any language—Python, R, Ruby, Lua, Scala, Java, JavaScript, C/ C++, FORTRAN, and more. + +Please find more information at [the official documentation page](https://docs.conda.io/en/latest/) + +Conda provides a great way to install packages that are already compiled, so you don't need to go over the long compilation process. If a package you need is not available, you can install it (and compile it when needed) using pip (Python) or install.packages (R). + +:::note +**Reproducibility**: +One of the ways to ensure the reproducibility of your results is to have an independent conda environment in the directory of each project (one of the options shown below). This will also keep conda environment files away from your `/home/$USER` directory. +::: + +## Advantages/disadvantages of using Conda +### Advantages + +- A lot of pre-compiled packages (fast and easy to install) +:::note +**For Python**: pip also offers pre-compiled packages (wheels). List can be found on [Python Wheels](https://pythonwheels.com). +However, Conda has a significantly larger number of pre-compiled packages. +::: +- Compiled packages use highly efficient Intel Math Kernel Library (MKL) library + +### Disadvantages + +- Conda does not take advantage of packages already installed in the system (while [virtualenv and venv](./02_python_packages_with_virtual_environments.mdx) do) +- As you will see below, you may need to do additional steps to keep track of all installed packages (including those installed by pip and/or install.packages) + +::::warning[Automatic deletion of your files] +This page describes the installation of packages on `/scratch`. One has to remember, though, that files stored in the HPC scratch file system are subject to the HPC Scratch old file purging policy:
+Files on the `/scratch` file system that have not been accessed for 60 or more days will be purged.
+You can read more this in [Data Management](../03_storage/01_intro_and_data_management.mdx). + +:::tip +Thus you can consider the following options: + +- Reinstall your packages if some of the files get deleted + - You can do this manually + - You can do this automatically. For example, within a workflow of a pipeline software like [Nextflow](https://www.nextflow.io/) +- Pay for "Research Project Space" - for details see [Research Project Space](../03_storage/05_research_project_space.mdx) +- Use Singularity and install packages within a corresponding overlay file - for details see [Singularity with Conda](../07_containers/03_singularity_with_conda.md) +::: +:::: + +## Initializing Conda +Load anaconda module +```sh +module purge +module load anaconda3/2024.02 +``` + +Conda init can create problems with package installation, so we suggest using `source activate` instead of `conda activate`, even though conda activate is considered a best practice by the Anaconda developers. + +There are two main ways to create conda environments. You can either define them with a name by using the `-n` or `--name` parameter or you can define them by full path (or prefix) using the `-p` or `--prefix` parameter. + +:::tip +NYU HPC recommends using prefix environments as a best practice because it makes reproducibility easier as we will see below. However, we do show how to use named environments with your `/scratch` space, so you don't fill up your `/home` space, if that's preferred. +::: + +## Prefix Conda Environments +You can store your program/project in `/scratch` and keep the conda environment with it by using the `-p` parameter. This will keep all the files inside the project's directory, instead of putting in in your `/home/$USER`.
+Here are details about how to do this for Python and R projects: + +### Python +#### 1. Load anaconda module and create local prefix environment +```sh +module purge +module load anaconda3/2024.02 +conda create -p ./penv python ## environment will be created in project directory +source activate ./penv +``` +#### 2. Create a symbolic link so conda will download files for packages to be installed into scratch, not your home directory. +```sh +mkdir /home//.conda +mkdir /scratch//conda_pkgs +ln -s /scratch//conda_pkgs /home//.conda/pkgs +``` +[Install pre-compiled packages available in conda](https://anaconda.org/anaconda/repo) +```sh +conda install -c anaconda pandas +``` + +#### 3. Other packages may be installed (and compiled when needed) using pip +```sh +pip install +``` +:::note +Conda and packages install by default to `~/.local/lib/python`, so if you did use `'pip install --user'` to install some packages (without conda or other virtual environment), they will be available in `~/.local/lib/python` +::: + +::::warning +:::info[The primary takeaway:] + +Let say you have tornado v.6 installed in `~/.local/lib/python`, and tornado v.5 installed by `conda install`. + +When you will do `source activate` you will have tornado v.6 available!! Not v.5!! + +(this behaviour is the same for packages installed by to `~/.local/lib/python` before or after you create your conda environment) + +`pip freeze` will give v.6 + +`conda list` will give v.5 +::: +:::tip[Solution] + +To overcome this, do `export PYTHONNOUSERSITE=True` after source activate +::: +:::: + +### R +#### 1. Load anaconda module and create local prefix environment +```sh +module load anaconda3/2024.02 +conda create -p ./renv r=3.5 ## environment will be created in project directory +## OR +conda create -c conda-forge -p ./renv r-base=3.6.3 ## environment will be created in project directory +source activate ./renv +``` + +#### 2. Install pre-compiled packages available in conda: + +[https://docs.anaconda.com/anaconda/packages/r-language-pkg-docs/](https://docs.anaconda.com/anaconda/packages/r-language-pkg-docs/) + +```sh +conda install -c r r-dplyr +``` + +#### 3. Other packages may be installed (and compiled) using install.packages() +```sh +install.packages("") +``` + +## Named Conda Environments + +You can create a named conda environment with a command like: +```bash +conda create --name +``` +You'll need to replace `` with the name of your environment. + +We defined where the prefix environment is created in the creation command, so where is the named environment created for named conda environments? + +We can find this out by using the `info` command with conda: +```bash +$ conda info +``` +This should output something like: +```bash + active environment : None + user config file : /home/NetID/.condarc + populated config files : + conda version : 24.1.2 + conda-build version : 24.1.2 + python version : 3.11.7.final.0 + solver : libmamba (default) + virtual packages : __archspec=1=cascadelake + __conda=24.1.2=0 + __glibc=2.34=0 + __linux=5.14.0=0 + __unix=0=0 + base environment : /share/apps/anaconda3/2024.02 (read only) + conda av data dir : /share/apps/anaconda3/2024.02/etc/conda + conda av metadata url : None + channel URLs : https://repo.anaconda.com/pkgs/main/linux-64 + https://repo.anaconda.com/pkgs/main/noarch + https://repo.anaconda.com/pkgs/r/linux-64 + https://repo.anaconda.com/pkgs/r/noarch + package cache : /share/apps/anaconda3/2024.02/pkgs + /home/NetID/.conda/pkgs + envs directories : /home/NetID/.conda/envs + /share/apps/anaconda3/2024.02/envs + platform : linux-64 + user-agent : conda/24.1.2 requests/2.31.0 CPython/3.11.7 Linux/5.14.0-284.86.1.el9_2.x86_64 rhel/9.2 glibc/2.34 solver/libmamba conda-libmamba-solver/24.1.0 libmambapy/1.5.6 aau/0.4.3 c/U4Vmh2Rw4idTcwqzLhNX1g s/2MYchdSIR4EFcYeOthelfQ + UID:GID : 402570:402570 + netrc file : None + offline mode : False +``` + +You can see in this output the default location for your `package cache` and your `envs directories`. They're currently set to subdirectories of our home directory. If we create named environments with these setting we will very quickly fill up our limited home directory space. The solution to this is to create or modify our personal conda config file to change these settings. You can see the setting above for `user config file` tells us the location of the file where we can make changes to these defaults. If the file doesn't exist you'll need to create it. You may also need to change the config file location if your output from `conda info` differs from the above. +```bash +$ nano /home/NetID/.condarc +``` +Now add the following lines to your config file: +```bash +envs_dirs: + - /scratch/NetID/conda_envs +pkgs_dirs: + - /scratch/NetID/conda_pkgs +always_copy: true +``` +You'll need to create those directories if they don't exist: +```bash +$ mkdir /scratch/NetID/conda_envs +$ mkdir /scratch/NetID/conda_pkgs +``` +Don't forget to change `NetID` above to your `NetID`. + +You should now see that the `package cache` and `envs directories` entries have changed to your `/scratch` space when you run `conda info` and packages and environments will be saved on your `/scratch` space for all named conda environments. + +## Reproducibility +### Packages installed only using conda + +Save a list of packages (so you are able to report the environment in a publication, and to restore/reproduce the environment on another machine at any time) + +```bash +# save +conda list --export > requirements.txt +# restore +conda create -p ./penv --file requirements.txt +``` +:::note +This will not list packages installed by `pip` or `install.packages()` +::: + + +### Packages installed using conda and `pip` (Python) + +In this case you can use: +```bash +export PYTHONNOUSERSITE=True ## to ignore packages in ~/.local/lib/python +# save +conda list --export > conda_requirements.txt +pip freeze > pip_requirements.txt +# restore +conda create -p ./penv --file conda_requirements.txt +pip install -r pip_requirements.txt +``` + + +::::tip +Alternatively, you can use `conda env export > all_requirements.txt`, which will save both: packages installed by conda and `pip`. +:::warning +However, this may fail if your conda environment is created as a sub-directory of your project's directory (which we recommend) +::: +:::: + +### Packages installed using `install.packages` (R) + +The command `conda list --export` will not include packages installed by `install.packages`. So, only use `conda install` to install R and use `renv` to maintain information about packages installed by `install.packages`. + +#### `renv` + +Please see details of using `renv` with conda for reproducibilty on [R packages with `renv`](./03_r_packages_with_renv.mdx). + +## Use conda env in a batch script +The part of the batch script that will call the command should look like: +### Python + +#### Single node +```bash +#!/bin/bash +#SBATCH --job-name=test +#SBATCH --nodes=1 +#SBATCH --cpus-per-task=1 +#SBATCH --ntasks-per-node=4 +#SBATCH --mem=8GB +#SBATCH --time=1:00:00 +module purge; +module load anaconda3/2024.02; +export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK; +source /share/apps/anaconda3/2020.07/etc/profile.d/conda.sh; +export PATH_TO_ENV= +source activate $PATH_TO_ENV; +export PATH=$PATH_TO_ENV/bin:$PATH; +python test.py + +``` +You'll need to replace `` with the full path to your penv directory. It's probably something like `/scratch/NetID/conda_tests/penv` + +#### Multiple nodes, using MPI +```sh +#!/bin/bash +#SBATCH --job-name=test +#SBATCH --nodes=1 +#SBATCH --cpus-per-task=1 +#SBATCH --ntasks-per-node=4 +#SBATCH --mem=8GB +#SBATCH --time=1:00:00 +export PATH_TO_ENV="" +module purge; +module load openmpi/gcc/4.1.6; +mpiexec bash -c "module purge; +export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK; +module load anaconda3/2024.02; +source /share/apps/anaconda3/2024.02/etc/profile.d/conda.sh; +source activate "$PATH_TO_ENV"; +export PATH="$PATH_TO_ENV/bin:$PATH"; +python test.py" +``` +Again, you'll need to replace `` above with the full path to your penv directory. + +### R (conda packages only) +```bash +#!/bin/bash +#SBATCH --job-name=test +#SBATCH --nodes=1 +#SBATCH --cpus-per-task=1 +#SBATCH --ntasks-per-node=4 +#SBATCH --mem=8GB +#SBATCH --time=1:00:00 +module purge; +module load anaconda3/2024.02; +export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK; +source /share/apps/anaconda3/2024.02/etc/profile.d/conda.sh; +export PATH_TO_ENV=; +source activate $PATH_TO_ENV; +export PATH=$PATH_TO_ENV/bin:$PATH; +Rscript r_script.R +``` +You'll again need to replace `` above with the full path to your renv directory. + +#### Multiple nodes, using MPI +```sh +#!/bin/bash +#SBATCH --job-name=test +#SBATCH --nodes=2 +#SBATCH --cpus-per-task=2 +#SBATCH --ntasks-per-node=4 +#SBATCH --mem=8GB +#SBATCH --time=1:00:00 +module purge; +module load anaconda3/2024.02; +export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK; +source /share/apps/anaconda3/2024.02/etc/profile.d/conda.sh; +export PATH_TO_ENV=""; +source activate $PATH_TO_ENV; +export PATH=$PATH_TO_ENV/bin:$PATH; +Rscript test.R +``` +You'll again need to replace `` above with the full path to your renv directory. + +### R (conda with renv combination) + +In this case, when you use sbatch you would activate conda in sbatch script, and R script will pickup packages installed in renv +```sh +#!/bin/bash +#SBATCH --job-name=test +#SBATCH --nodes=2 +#SBATCH --cpus-per-task=2 +#SBATCH --ntasks-per-node=4 +#SBATCH --mem=8GB +#SBATCH --time=1:00:00module purge +module load anaconda3/2024.02 +source /share/apps/anaconda3/2024.02/etc/profile.d/conda.sh +source activate +Rscript test.R +``` +You'll again need to replace `` above with the full path to your renv directory. diff --git a/docs/hpc/06_tools_and_software/05_sqlite_handling_large_structured_data.md b/docs/hpc/06_tools_and_software/05_sqlite_handling_large_structured_data.md index 6df7b27e94..cb932bf643 100644 --- a/docs/hpc/06_tools_and_software/05_sqlite_handling_large_structured_data.md +++ b/docs/hpc/06_tools_and_software/05_sqlite_handling_large_structured_data.md @@ -23,7 +23,7 @@ It is better in this case to request smaller amount of RAM and read data (effici - You are not limited by RAM any longer - Compared to other file formats SQLite is very good in selecting certain lines (especially if you use indexing) -- You can use familiar dplyr syntax or execute SQL queries directly +- You can use familiar `dplyr` syntax or execute SQL queries directly - [dplyr](https://dplyr.tidyverse.org/) is an interface for working with data in a database, not for modifying remote tables. - [DBI package](https://dbi.r-dbi.org/) allows to both read and modify tables - SQLite is [actually faster for common data analysis tasks](https://www.sqlite.org/speed.html) than other popular databases. @@ -44,26 +44,30 @@ It is better in this case to request smaller amount of RAM and read data (effici ## Command line (CLI) example Create environment ```sh -mkdir projects/sqlite-test -cd projects/sqlite-test -conda create -p ./cenv -conda activate ./cenv -conda install -y sqlite +$ mkdir projects/sqlite-test +$ cd projects/sqlite-test +$ conda create -p ./cenv +$ source activate ./cenv +$ conda install -y sqlite ``` Then [follow this SQLite example](https://sqlite.org/cli.html). ```sh -sqlite3 db_file.sqlite -create table tbl1(one varchar(10), two smallint); -insert into tbl1 values('hello!',10); -insert into tbl1 values('goodbye', 20); -select * from tbl1; +$ sqlite3 db_file.sqlite +sqlite> create table tbl1(one varchar(10), two smallint); +sqlite> insert into tbl1 values('hello!', 10); +sqlite> insert into tbl1 values('goodbye', 20); +sqlite> select * from tbl1; +hello!|10 +goodbye|20 ``` Now Close session (Ctrl-D). Reopen session to check if changes are saved ```sh -sqlite3 db_file.sqlite -select * from tbl1; +$ sqlite3 db_file.sqlite +sqlite> select * from tbl1; +hello!|10 +goodbye|20 ``` ## R example @@ -75,15 +79,15 @@ conda will install pre-compiled packages. Which is good (faster) and bad (not fu ::: :::tip -Alternative: install packages to a local directory or use renv as described in [R Packages with renv](./04_r_packages_with_renv.md) +Alternative: install packages to a local directory or use renv as described in [R Packages with renv](./03_r_packages_with_renv.mdx) ```sh mkdir /scratch/$USER/projects/myTempProject cd /scratch/$USER/projects/myTempProject -module load anaconda3/2020.07 +module load anaconda3/2024.02 -conda create -p ./cenv -c conda-forge r=4.1 -conda activate ./cenv +conda create -p ./cenv -c conda-forge r=4.5 +source activate ./cenv conda install -c r r-rsqlite conda install -c r r-tidyverse conda install -c conda-forge r-remotes @@ -144,7 +148,7 @@ df_temp <- df_con %>% filter( row_number() %in% c(1, 3) ) %>% collect Save as feather ```R -feather::write_feather(df_temp, paste0("file_", ind, ".feather")) +feather::write_feather(df_temp, "my_data.feather") ``` ### Alternative: read csv file to SQLite directly @@ -155,9 +159,39 @@ conda install -c conda-forge r-sqldf R library(sqldf) ## create data file -sqldf("attach allData as new") -## read file directly from csv to sqlite -read.csv.sql(file = "test.tab", sql = "create table states_data as select * from file", dbname = "allData") +# sqldf("attach allData as new") + +# make csv file for this example +write.csv(df_con, "df_con.csv", row.names = FALSE) + +# read file directly from csv to sqlite +read.csv.sql(file = "df_con.csv", sql = "create table states_data as select * from file", dbname = "allData") + +# verify data in data frame +dbListTables(con) +[1] "fl" "sqlite_stat1" "sqlite_stat4" "states_data" + +df_con_sd <- tbl(con, "states_data") +df_con_sd +# Source: table<`states_data`> [?? x 19] +# Database: sqlite 3.50.1 [/scratch/netID/myTempProject/allData] + year month day dep_time sched_dep_time dep_delay arr_time sched_arr_time + + 1 2013 1 1 517 515 2 830 819 + 2 2013 1 1 533 529 4 850 830 + 3 2013 1 1 542 540 2 923 850 + 4 2013 1 1 544 545 -1 1004 1022 + 5 2013 1 1 554 600 -6 812 837 + 6 2013 1 1 554 558 -4 740 728 + 7 2013 1 1 555 600 -5 913 854 + 8 2013 1 1 557 600 -3 709 723 + 9 2013 1 1 557 600 -3 838 846 +10 2013 1 1 558 600 -2 753 745 +# ℹ more rows +# ℹ 11 more variables: arr_delay , carrier , flight , +# tailnum , origin , dest , air_time , distance , +# hour , minute , time_hour +# ℹ Use `print(n = ...)` to see more rows ``` ## UI for SQLite - SQLiteStudio diff --git a/docs/hpc/07_containers/03_singularity_with_conda.md b/docs/hpc/07_containers/03_singularity_with_conda.md index 742af02b54..bd884f5a8b 100644 --- a/docs/hpc/07_containers/03_singularity_with_conda.md +++ b/docs/hpc/07_containers/03_singularity_with_conda.md @@ -287,7 +287,7 @@ pip install tensorboard ``` :::note -[Click here](../06_tools_and_software/02_conda_environments.md) for information on how to configure your conda environment. +[Click here](../06_tools_and_software/04_conda_environments.mdx) for information on how to configure your conda environment. ::: Please also keep in mind that once the overlay image is opened in default read-write mode, the file will be locked. You will not be able to open it from a new process. Once the overlay is opened either in read-write or read-only mode, it cannot be opened in RW mode from other processes either. For production jobs to run, the overlay image should be open in read-only mode. You can run many jobs at the same time as long as they are run in read-only mode. In this ways, it will protect the computation software environment, software packages are not allowed to change when there are jobs running.