diff --git a/docs/hpc/05_submitting_jobs/01_slurm_submitting_jobs.md b/docs/hpc/05_submitting_jobs/01_slurm_submitting_jobs.md index c782d504be..62a493d82f 100644 --- a/docs/hpc/05_submitting_jobs/01_slurm_submitting_jobs.md +++ b/docs/hpc/05_submitting_jobs/01_slurm_submitting_jobs.md @@ -1,690 +1,71 @@ -# Slurm: Submitting Jobs +# Submitting Jobs on Torch - -## Batch vs Interactive Jobs - -- HPC workloads are usually better suited to *batch processing* than *interactive* workflows. -- A batch job is sent to the system when submitted with an **sbatch** command. -- The working pattern we are all familiar with is *interactive* - where we type ( or click ) something interactively, and the computer performs the associated action. Then we type ( or click ) the next thing. -- Comments at the start of the script, which match a special pattern ( `#SBATCH` ) are read as Slurm options. - -### Challenges of Interactive Work - -There is a reason why GUIs are less common in HPC environments: **point-and-click** is **necessarily interactive**. In HPC environments (*as we'll see in section 3*) work is scheduled in order to allow exclusive use of the shared resources. On a busy system there may be several hours wait between when you submit a job and when the resources become available, so a reliance on user interaction is not viable. In Unix, commands need not be run interactively at the prompt, you can write a sequence of commands into a file to be run as a script, either manually (for sequences you find yourself repeating frequently) or by another program such as the batch system. - -:::tip -The job might not start immediately, and might take hours or days, so we prefer a *batch* approach: - -- Plan the sequence of commands which will perform the actions we need and write the commands into a script. - -You can now run the script interactively, which is a great way to save effort if i frequently use the same workflow, or ... -- Submit the script to a batch system, to run on dedicated resources when they become available. +:::tip Beginner tutorial available +If you are new to using HPC resources and would like to learn about the principles of using the `SLURM` scheduler for submitting batch jobs, please refer to [this section](../13_tutorial_intro_hpc/04_scheduler_fundamentals.mdx). This section focuses on the specifics of the Torch cluster and assumes familiarity with the tutorial. ::: -### Job Output - -- The batch system writes stdout and stderr from a job to a file named for example *"slurm-12345.out"* - - You can change either stdout or stderr using sbatch options. -- While a job is running, it caches the stdout an stderr in the job working directory. -- You can use redirection to send output of a specific command into a file. - -### Writing and Submitting a Job - -There are two aspects to a batch job script: -- A set of *SBATCH* directives describing the resources required and other information about the job. -- The script itself, comprised of commands to set up and perform the computations without additional user interaction. - -### A Simple Job Example - -A typical batch script on an NYU HPC cluster looks something like these two examples: - -```bash -#!/bin/bash -#SBATCH --nodes=1 -#SBATCH --ntasks-per-node=1 -#SBATCH --cpus-per-task=1 -#SBATCH --time=5:00:00 -#SBATCH --mem=2GB -#SBATCH --job-name=myTest -#SBATCH --mail-type=END -#SBATCH --mail-user=bob.smith@nyu.edu -#SBATCH --output=slurm_%j.out -#SBATCH --error=slurm_%j.err - - -module purge -module load stata/17.0 -RUNDIR=$SCRATCH/my_project/run-${SLURM_JOB_ID/.*} -mkdir -p $RUNDIR - -DATADIR=$SCRATCH/my_project/data -cd $RUNDIR -stata -b do $DATADIR/data_0706.do -``` - -```bash -#!/bin/bash -#SBATCH --nodes=1 -#SBATCH --ntasks-per-node=1 -#SBATCH --cpus-per-task=1 -#SBATCH --time=5:00:00 -#SBATCH --mem=2GB -#SBATCH --job-name=myTest -#SBATCH --mail-type=END -#SBATCH --mail-user=bob.smith@nyu.edu -#SBATCH --output=slurm_%j.out -#SBATCH --error=slurm_%j.err - -module purge - -SRCDIR=$HOME/my_project/code -RUNDIR=$SCRATCH/my_project/run-${SLURM_JOB_ID/.*} -mkdir -p $RUNDIR - -cd $SLURM_SUBMIT_DIR -cp my_input_params.inp $RUNDIR - -cd $RUNDIR -module load fftw/intel/3.3.9 -$SRCDIR/my_exec.exe < my_input_params.inp -``` - -We'll work through them more closely in a moment. -You submit the job with *sbatch*: - -```sh -[NetID@log-1 ~]$ sbatch myscript.sh -``` - -And monitor it's progress with: - -```sh -[NetID@log-1 ~]$ squeue -u $USER -``` - -**What just happened ?** Here's an annotated version of the first script: - -```sh -#!/bin/bash -# This line tells the shell how to execute this script, and is unrelated -# to SLURM. - -# at the beginning of the script, lines beginning with "#SBATCH" are read by -# SLURM and used to set queueing options. You can comment out a SBATCH -# directive with a second leading #, eg: -##SBATCH --nodes=1 - -# we need 1 node, will launch a maximum of one task and use one cpu for the task: -#SBATCH --nodes=1 -#SBATCH --ntasks-per-node=1 -#SBATCH --cpus-per-task=1 - -# we expect the job to finish within 5 hours. If it takes longer than 5 -# hours, SLURM can kill it: -#SBATCH --time=5:00:00 - -# we expect the job to use no more than 2GB of memory: -#SBATCH --mem=2GB - -# we want the job to be named "myTest" rather than something generated -# from the script name. This will affect the name of the job as reported -# by squeue: -#SBATCH --job-name=myTest - -# when the job ends, send me an email at this email address. -#SBATCH --mail-type=END -#SBATCH --mail-user=bob.smith@nyu.edu - -# both standard output and standard error are directed to the same file. -# It will be placed in the directory I submitted the job from and will -# have a name like slurm_12345.out -#SBATCH --output=slurm_%j.out - -# once the first non-comment, non-SBATCH-directive line is encountered, SLURM -# stops looking for SBATCH directives. The remainder of the script is executed -# as a normal Unix shell script - -# first we ensure a clean running environment: -module purge -# and load the module for the software we are using: -module load stata/17.0 - -# next we create a unique directory to run this job in. We will record its -# name in the shell variable "RUNDIR", for better readability. -# SLURM sets SLURM_JOB_ID to the job id, ${SLURM_JOB_ID/.*} expands to the job -# id up to the first '.' We make the run directory in our area under $SCRATCH, because at NYU HPC -# $SCRATCH is configured for the disk space and speed required by HPC jobs. -RUNDIR=$SCRATCH/my_project/run-${SLURM_JOB_ID/.*} -mkdir $RUNDIR - -# we will be reading data in from somewhere, so define that too: -DATADIR=$SCRATCH/my_project/data - -# the script will have started running in $HOME, so we need to move into the -# unique directory we just created -cd $RUNDIR - -# now start the Stata job: -stata -b do $DATADIR/data_0706.do -The second script has the same SBATCH directives, but this time we are using code we compiled ourselves. Starting after the SBATCH directives: -# first we ensure a clean running environment: -module purge - -# and ensure we can find the executable: -SRCDIR=$HOME/my_project/code - -# create a unique directory to run this job in, as per the script above -RUNDIR=$SCRATCH/my_project/run-${SLURM_JOB_ID/.*} -mkdir $RUNDIR - -# By default the script will have started running in the directory we ran sbatch from. -# Let's assume our input file is in the same directory in this example. SLURM -# sets some environment variables with information about the job, including -# SLURM_SUBMIT_DIR which is the directory the job was submitted from. So lets -# go there and copy the input file to the run directory on /scratch: -cd $SLURM_SUBMIT_DIR -cp my_input_params.inp $RUNDIR - -# go to the run directory to begin the run: -cd $RUNDIR - -# load whatever environment modules the executable needs: -module load fftw/intel/3.3.9 - -# run the executable (sending the contents of my_input_params.inp to stdin) -$SRCDIR/my_exec.exe < my_input_params.inp -``` - -## Batch Jobs - -Jobs are submitted with the sbatch command: - -```sh -sbatch options job-script -``` - -The options tell SLURM information about the job, such as what resources will be needed. **These can be specified in the job-script as SBATCH directives, or on the command line as options, or both** -:::note - When SBATCH options are provided in both the script and the command line, the command line options take precedence should the two contradict each other. - ::: - For each option there is a corresponding SBATCH directive with the syntax: - -```bash -#SBATCH option -``` - -For example, you can specify that a job needs 2 nodes and 4 cores on each node ( by default one CPU core per task ) on each node by adding to the script the directive: - -```bash -#!/bin/bash -#SBATCH --nodes=2 -#SBATCH --ntasks-per-node=4 -``` - -or as a command-line option to sbatch when you submit the job: - -```sh -[NetID@log-1 ~]$ sbatch --nodes=2 --ntasks-per-node=4 my_script.sh -``` - -### Job Output Options - -- `-J jobname` - - Give the job a name. The default is the filename of the job script. Within the job, `$SLURM_JOB_NAME` expands to the job name. - -- `-o path/for/stdout` - - Send `stdout` to `path/for/stdout`. The default filename is slurm-`${SLURM_JOB_ID}.out`, e.g. slurm-`12345.out`, in the directory from which the job was submitted. - -- `-e path/for/stderr` - - Send `stderr` to `path/for/stderr`. - -- `--mail-user=my_email_address@nyu.edu` - - Send mail to my_email_address@nyu.edu when certain events occur. - -- `--mail-type=type` - - Valid type values are NONE, BEGIN, END, FAIL, REQUIRE, ALL. - -### Job Environment Options - -- `--export=VAR1,VAR2="some value",VAR3` - - Pass variables to the job, either with a specific value (the `VAR=` form) or from the submitting environment ( without "`=`" ) - - - `--get-user-env`\[=timeout]\[mode] - - Run something like "su `-` \ -c /usr/bin/env" and parse the output. Default timeout is 8 seconds. The mode value can be "S", or "L" in which case "su" is executed with "`-`" option. - -### Resource Request Options - -- `-t, --time=time` - - `Set a limit on the total run time. Acceptable formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds"`. - -- `--mem=MB` - - Maximum memory per node the job will need in MegaBytes - -- `--mem-per-cpu=MB` - - `Memory required per allocated CPU in MegaBytes` - -- `-N, --node=num` - - Number of nodes are required. Default is 1 node. - - `-n, --ntasks=num` - - Maximum number tasks will be launched. Default is one task per node. - - `--ntasks-per-node=ntasks` - - Request that ntasks be invoked on each node. - - `-c, --cpus-per-task=ncpus` - - Require ncpus number of CPU cores per task. Without this option, allocate one core per task. - - Requesting the resources you need, as accurately as possible, allows your job to be started at the earliest opportunity as well as helping the system to schedule work efficiently to everyone's benefit. - -### srun & Interactive Job Options - -- `-nnum` - - `Specify the number of tasks to run, eg. -n4. Default is one CPU core per task.` Don't just submit the job, but also wait for it to start and connect `stdout`, `stderr`and `stdin` to the current terminal. - -- `-ttime` - - Request job running duration, eg. `-t1:30:00` - -- `--mem=MB` - - Specify the real memory required per node in MegaBytes, eg. `--mem=4000` - - `--pty` - - Execute the first task in pseudo terminal mode, eg. `--pty /bin/bash`, to start a bash command shell - -- `--x11` - - Enable X forwarding, so programs using a GUI can be used during the session (provided you have X forwarding to your workstation set up) - - To leave an interactive batch session, type `exit` at the command prompt - -### Delaying Jobs - -- `--begin=time` - - Delay starting this job until after the specified date and time, eg. `--begin=9:42:00`, to start the job at 9:42:00 am - -- `-d, --dependency=dependency_list` - - (More info here [https://slurm.schedmd.com/sbatch.html](https://slurm.schedmd.com/sbatch.html)) - - Example 1 - - `--dependency=afterok:12345`, to delay starting this job until the job 12345 has completed successfully - - Example 2 - - Let us say job 1 uses sbatch file job1.sh, and job 2 uses job2.sh - - Inside the batch file of the second job (job2.sh) add - - `#SBATCH --dependency=afterok:$job1` - - Start the first job and get id of the job - - `job1=$(echo $(sbatch job1.sh) | grep -Eo "[0-9]+")` - - Schedule second jobs to start when the first one ends - - `sbatch job2.sh` - -### Submitting Similar Jobs - -- `-a, --array=indexes` - - Submit an array of jobs with array ids as specified. Array ids can be specified as a numerical range, a comma-separated list of numbers, or as some combination of the two. Each job instance will have an environment variable `SLURM_ARRAY_JOB_ID` and `SLURM_ARRAY_TASK_ID`. For example: - - `--array=1-11`, to start an array job with index from 1 to 11 - - `--array=1-7:2`, to submit an array job with index step size 2 - - `--array=1-9%4`, to submit an array job with simultaneously running job elements set to 4 - - The srun command is similar to `pbsdsh`. It launches tasks on allocated resources - -## R Job Example - -Create a directory and an example R script - -```bash -[NetID@log-1 ~]$ mkdir /scratch/$USER/examples -[NetID@log-1 ~]$ cd /scratch/$USER/examples -``` - -Create `example.R` inside the examples directory: - -```R -df <- data.frame(x=c(1,2,3,1), y=c(7,19,2,2)) -df -indices <- order(df$x) -order(df$x) -df[indices,] -df[rev(order(df$y)),] -``` - -Create the following SBATCH script named `run-R.sbatch` : - -```bash -#!/bin/bash -# -#SBATCH --job-name=RTest -#SBATCH --nodes=1 -#SBATCH --tasks-per-node=1 -#SBATCH --mem=2GB -#SBATCH --time=01:00:00 - -module purge -module load r/intel/4.0.4 - -cd /scratch/$USER/examples -R --no-save -q -f example.R > example.out 2>&1 -``` - -Run the job using `sbatch`. - -```sh -[NetID@log-1 ~]$ sbatch run-R.sbatch -``` - -## Array Jobs - -Using job array you may submit many similar jobs with almost identical job requirement. This reduces loads on both shoulders of users and the scheduler system. Job array can only be used in batch jobs. Usually the only requirement difference among jobs in a job array is the input file or files. - -Please follow the recipe below to try the example. There are 5 input files named `sample-1.txt`, `sample-2.txt` to `sample-5.txt` in sequential order. Running one command `sbatch --array=1-5 run-jobarray.s`, you submit 5 jobs to process each of these input files individually. - -Prepare the data before submitting an array job: - -```sh -[NetID@log-1 ~]$ mkdir -p /scratch/$USER/myjarraytest -[NetID@log-1 ~]$ cd /scratch/$USER/myjarraytest -[NetID@log-1 ~]$ cp /share/apps/Tutorials/slurm/example/jobarray/* . -[NetID@log-1 ~]$ ls -``` - -Submit the array job: - -```sh -[NetID@log-1 ~]$ sbatch --array=1-5 run-jobarray.s -``` - -The content of the job script `run-jobarray.s` is copied below: +## Partitions -```bash -#!/bin/bash +`SLURM` partitions on Torch control stakeholder resource access. No physical nodes are tied to partitions — instead, equivalent compute resources are allocated via partition `QoS`([QualityOfService](https://slurm.schedmd.com/qos.html)). -#SBATCH --job-name=myJobarrayTest -#SBATCH --nodes=1 -#SBATCH --ntasks-per-node=1 -#SBATCH --time=5:00 -#SBATCH --mem=1GB -#SBATCH --output=wordcounts_%A_%a.out -#SBATCH --error=wordcounts_%A_%a.err - -module purge -module load python/intel/3.8.6 - -cd /scratch/$USER/myjarraytest -python wordcount.py sample-$SLURM_ARRAY_TASK_ID.txt -``` - -Job array submission introduces an environment variable, `SLURM_ARRAY_TASK_ID`, which is unique for each job array job. It is usually embedded somewhere so that when a job runs, its unique value is incorporated into producing a proper file name. - -Also as shown above: two additional options `%A` and `%a`, denoting the job ID and the task ID ( i.e. job array index ) respectively, are available for specifying a job's stdout, and stderr file names. - -## Additional Examples - -You can find more examples in the slurm jobarray examples directory: - -```sh -/scratch/work/public/examples/slurm/jobarry/ -``` - -## GPU Jobs - -To request one GPU card, use SBATCH directive in job script: - -```bash -#SBATCH --gres=gpu:1 -``` - -To request a specific card type, use eg. `--gres=gpu:v100:1`. As an example, let's submit an Amber job. Amber is a molecular dynamics software package. The recipe is: - -```sh -[NetID@log-1 ~]$ mkdir -p /scratch/$USER/myambertest -[NetID@log-1 ~]$ cd /scratch/$USER/myambertest -[NetID@log-1 ~]$ cp /share/apps/Tutorials/slurm/example/amberGPU/* . -[NetID@log-1 ~]$ sbatch run-amber.s -``` - -There are three NVIDIA GPU types and one AMD GPU type that can be used. - -:::warning -AMD GPUs require code to be compatible with ROCM drivers, not CUDA +:::tip Partitions +Do not specify partitions manually, except for preemption which is described later. ::: -**To request NVIDIA GPUs** - -- RTX8000 -```bash -#SBATCH --gres=gpu:rtx8000:1 -``` - -- V100 -```bash -#SBATCH --gres=gpu:v100:1 -``` - -- A100 -```bash -#SBATCH --gres=gpu:a100:1 -``` - -- H100 -```bash -#SBATCH --gres=gpu:h100:1 -``` - -**To request AMD GPUs** - -- MI100 -```bash -#SBATCH --gres=gpu:mi100:1 -``` - -- MI250 -```bash -#SBATCH --gres=gpu:mi250:1 -``` - -From the tutorial example directory we copy over Amber input data files "inpcrd", "prmtop" and "mdin", and the job script file "run-amber.s". The content of the job script "run-amber.s" is: - -```bash -#!/bin/bash -# -#SBATCH --job-name=myAmberJobGPU -#SBATCH --nodes=1 -#SBATCH --cpus-per-task=1 -#SBATCH --time=00:30:00 -#SBATCH --mem=3GB -#SBATCH --gres=gpu:1 -module purge -module load amber/openmpi/intel/20.06 -cd /scratch/$USER/myambertest -pmemd.cuda -O -``` - -The demo Amber job should take ~2 minutes to finish once it starts running. When the job is done, several output files are generated. Check the one named "mdout", which has a section most relevant here: - -```sh -|--------------------- INFORMATION ---------------------- -| GPU (CUDA) Version of PMEMD in use: NVIDIA GPU IN USE. -| Version 16.0.0 -| -| 02/25/2016 -[......] - -|------------------- GPU DEVICE INFO -------------------- -| -| CUDA_VISIBLE_DEVICES: 0 -| CUDA Capable Devices Detected: 1 -| CUDA Device ID in use: 0 -| CUDA Device Name: Tesla K80 -| CUDA Device Global Mem Size: 11439 MB -| CUDA Device Num Multiprocessors: 13 -| CUDA Device Core Freq: 0.82 GHz -| -|-------------------------------------------------------- -``` - -## Interactive Jobs - -### Bash Sessions - -The majority of the jobs on the NYU HPC cluster are submitted with the sbatch command, and executed in the background. These jobs' steps and workflows are predefined by users, and their executions are driven by the scheduler system. - -There are cases when users need to run applications interactively ( *interactive jobs* ). Interactive jobs allow the users to enter commands and data on the command line (or in a graphical interface ), providing an experience similar to working on a desktop or laptop. - -Examples of common interactive tasks are: +## Resource limits and restrictions +Jobs within the same partition cannot exceed their assigned resources (`QOSGrpGRES`). User GPU Quotas: Each user has a total GPU quota of 24 GPUs for jobs with wall time < 48 hours (`QOSMaxGRESPerUser`). -- Editing files +Non-stakeholders to temporarily use stakeholder resources (a stakeholder group to temporarily use another group’s resources). Stakeholders retain normal access to their own resources. If non-stakeholders (or other stakeholders) are using them, their jobs may be preempted (canceled) once stakeholders submit new jobs. Public users are allowed to use stakeholder resources only with preemption partitions. Refer to the section below for details on preemptible jobs. -- Compiling and debugging code +## Job Submission on Torch +As stated in the tuturial, always only request the compute resources (e.g., GPUs, CPUs, memory) needed for the job. Requesting too many resources can prevent your job from being scheduled within an adequate time. The `SLURM` scheduler will automatically dispatch jobs to all accessible GPU partitions that match resource requests. -- Exploring data, to insights - -- A graphical window to run visualization - -- etc - -To support interactive use in a batch environment, Slurm allows for interactive batch jobs. - -:::warning -Please do not run interactive jobs on the HPC Login nodes. Login nodes of the HPC cluster are shared between many users so running interactive jobs that require significant computing and IO resources on the login nodes will impact many users. For this reason running compute and IO intensive interactive jobs on the HPC login nodes is not allowed. - -**_Such jobs may be removed without notice!_** +:::danger Low GPU Utilization Policy +Jobs with low GPU utilization will be automatically canceled. The exact threshold is TBD, but enforcement will be very aggressive. ::: -::::tip -Instead of running interactive jobs on Login nodes, users can run interactive jobs on the HPC Compute nodes using SLURM's `srun` utility. Running interactive jobs on compute nodes does not impact many users and in addition provides access to resources that are not available on the login nodes, such as interactive access to GPUs, high memory, exclusive access to all the resources of a compute node, etc. -:::note -There is no partition on the HPC cluster that has been reserved for Interactive jobs. +## Preemptible jobs on Torch +On Torch, users may run "preemptible" jobs on stakeholder resources that their group does not own. This allows the stakeholder resources to be utilized by non-stakeholders which may otherwise be idle. To make the best use of these resources, you are encouraged to adopt checkpoint/restart to allow for resumption of the workload in subsequent jobs. + +:::warning Preemption Policy +Jobs become eligible for preemption after 1 hour of runtime. Jobs will not be canceled within the first hour. ::: -:::: -### Start an Interactive Job +The preemption order is: +- Stakeholder jobs (highest priority), these can preempt GPU jobs from public users or other stakeholders +- GPU jobs can preempt CPU-only jobs running on GPU nodes +- Partition Assignment Order +- PI stakeholder partitions +- School stakeholder partitions +- IT public partitions +- Preemption partitions +Applies separately to both GPU types: L40S first, then H200 -When you start an interactive batch job the command prompt is not immediately returned. Instead, you wait until the resource is available when the prompt is returned and you are on a compute node and in a batch job - much like the process of logging in to a host with ssh. **To end the session, type 'exit'**, again just like the process of logging in and out with ssh. -```sh -[NetID@log-1 ~]$ srun --pty /bin/bash -srun: job 58699789 queued and waiting for resources -srun: job 58699789 has been allocated resources -[NetID@cm034 ~]$ hostname -cm034.hpc.nyu.edu +To allow jobs in both normal and preemption partitions: ``` - -To use any GUI-based program within the interactive batch session you will need to extend X forwarding with the --x11 option. This of course still relies on you having X forwarding at your login session. To test if you have X forwarding running, you can try running the gnuplot test as shown: -```sh -[NetID@log-1 ~]$ module load gnuplot/gcc/5.4.1 -[NetID@log-1 ~]$ gnuplot -gnuplot> test +#SBATCH --comment="preemption=yes;requeue=true" ``` -If a window opens on your display with a gnuplot test window, you know that Xforwarding is working. Please see the [X11 Forwarding](../02_connecting_to_hpc/02_x11_forwarding.md) section for details. - -### Request Resources - -You can request resources for an interactive batch session just as you would for any other job, for example to request 4 processors with 4GB memory for 2 hours. - -If you do not request resources you will get the default settings. If after some directory navigation in your interactive session, you can jump back to the directory you submitted from with: - -```sh -[NetID@cm034 ~]$ cd $SLURM_SUBMIT_DIR +Jobs in stakeholder partitions will not be canceled, but those in preemption partitions may be. Canceled jobs will be re-queued automatically with `requeue=true`. To use only preemption partitions: ``` - -### Interactive Job Options - -(Don't just submit the job, but also wait for it to start and connect `stdout`, `stderr` and `stdin` to the current terminal) - -- `-nnum` - - Specify the number of tasks to run, eg. -n4. Default is one CPU core per task - -- `-ttime` - - Request job running duration, eg. `-t1:30:00` - -- `--mem=MB` - - Specify the real memory required per node in MegaBytes, eg. `--mem=4000` - - `--pty` - - Execute the first task in pseudo terminal mode, eg. `--pty /bin/bash`, to start a bash command shell - -- `--gres=gpu:N` - - To request `N` number of GPUs - -- `--x11` - - Enable X forwarding, so programs using a GUI can be used during the session (provided you have X forwarding to your workstation set up) - - To leave an interactive batch session, type `exit` at the command prompt - -Certain tasks need user interaction - such as debugging and some GUI-based applications. However the HPC clusters rely on batch job scheduling to efficiently allocate resources. Interactive batch jobs allow these apparently conflicting requirements to be met. - -### Interactive Bash Job Examples - -**Example (Without x11 forwarding)** - -Through `srun` SLURM provides rich command line options for users to request resources from the cluster, to allow interactive jobs. Please see some examples and short accompanying explanations in the code block below, which should cover many of the use cases. - -In the srun examples below, through `--pty /bin/bash` we request to start bash command shell session in pseudo terminal by default the resource allocated is single CPU core and 2GB memory for 1 hour: - -```sh -[NetID@log-1 ~]$ srun --pty /bin/bash +#SBATCH --comment="preemption=yes;preemption_partitions_only=yes;requeue=true" ``` +Jobs with preemption partitions only might be allowed to use more resources -To request 4 CPU cores, 4 GB memory, and 2 hour running duration: +## Advanced options -```sh -[NetID@log-1 ~]$ srun -c4 -t2:00:00 --mem=4000 --pty /bin/bash +### GPU MPS +Use GPU Multi-Process Service (MPS) to improve overall GPU utilization, as this allows multiple GPU jobs to share a single GPU concurrently by: ``` - -To request one GPU card, 3 GB memory, and 1.5 hour running duration: - -```sh -[NetID@log-1 ~]$ srun -t1:30:00 --mem=3000 --gres=gpu:1 --pty /bin/bash +#SBATCH --comment="gpu_mps=yes" ``` - -**Example (x11 forwarding)** - -In srun there is an option "–x11", which enables X forwarding, so programs using a GUI can be used during an interactive session (provided you have X forwarding to your workstation set up). - -To request computing resources, and export x11 display on allocated node(s) - -```sh -[NetID@log-1 ~]$ srun --x11 -c4 -t2:00:00 --mem=4000 --pty /bin/bash -[NetID@cm034 ~]$ module load gnuplot/gcc/5.4.1 -[NetID@cm034 ~]$ gnuplot -gnuplot> test +### RAM disk +A portion of the RAM available can be mounted as a disk for fast `I/O` operations: ``` - -To request GPU card etc, and export x11 display: - -```sh -[NetID@log-1 ~]$ srun --x11 -t1:30:00 --mem=3000 --gres=gpu:1 --pty /bin/bash +#SBATCH --comment="ram_disk=1GB" ``` - -### R interactive job - -The following example shows how to work with Interactive R session on a compute node: - -```sh -[NetID@log-1 ~]$ srun -c 1 --pty /bin/bash -[NetID@cm034 ~]$ module purge -[NetID@cm034 ~]$ module list - -No modules loaded -[NetID@cm034 ~]$ module load r/gcc/4.4.0 -[NetID@cm034 ~]$ module list - -Currently Loaded Modules: - 1) r/intel/4.4.0 - -[NetID@cm034 ~]$ R -R version 4.4.0 (2024-04-24) -- "Puppy Cup" -Copyright (C) 2024 The R Foundation for Statistical Computing -Platform: x86_64-pc-linux-gnu - -R is free software and comes with ABSOLUTELY NO WARRANTY. -You are welcome to redistribute it under certain conditions. -Type 'license()' or 'licence()' for distribution details. - -R is a collaborative project with many contributors. -Type 'contributors()' for more information and -'citation()' on how to cite R or R packages in publications. - -Type 'demo()' for some demos, 'help()' for on-line help, or -'help.start()' for an HTML browser interface to help. -Type 'q()' to quit R. -> 5 + 10 -[1] 15 -> 6 ** 2 -[1] 36 -> tan(45) -[1] 1.619775 -> -> q() -Save workspace image? [y/n/c]: n -[NetID@cm034 ~]$ exit -exit -[NetID@log-1 ~]$ +### GPU MPS & RAM Disk in a preemptible job +Both of these can be combined with preemption as shown: +``` +#SBATCH --comment="preemption=yes;preemption_partitions_only=yes;requeue=true;gpu_mps=yes;ram_disk=1GB" ``` diff --git a/docs/hpc/05_submitting_jobs/02_slurm_main_commands.md b/docs/hpc/05_submitting_jobs/02_slurm_main_commands.md index bc28a5deb8..039332c036 100644 --- a/docs/hpc/05_submitting_jobs/02_slurm_main_commands.md +++ b/docs/hpc/05_submitting_jobs/02_slurm_main_commands.md @@ -1,4 +1,4 @@ -# Slurm: Main Commands +# Slurm: Command reference Slurm offers many utility commands to work with, some of the most popularly used commands are: diff --git a/docs/hpc/05_submitting_jobs/03_slurm_tutorial.md b/docs/hpc/05_submitting_jobs/03_slurm_tutorial.md deleted file mode 100644 index 4c79123fc8..0000000000 --- a/docs/hpc/05_submitting_jobs/03_slurm_tutorial.md +++ /dev/null @@ -1,420 +0,0 @@ -# Slurm: Tutorial - -## Introduction to High Performance Computing Clusters - -In a High Performance Computing Cluster, such as the NYU-IT HPC Greene cluster, there are hundreds of computing nodes interconnected by high-speed networks. - -Linux operating system ( in our case Red Hat Enterprise Linux) runs on each of the nodes individually. The resources are shared among many users for their technical or scientific computing purposes. - -Slurm is a cluster software layer built on top of the interconnected nodes, aiming at orchestrating the nodes' computing activities, so that the cluster could be viewed as a unified, enhanced and scalable computing system by its users. - -In NYU HPC clusters the users coming from many departments with various disciplines and subjects, with their own computing projects, impose on us very diverse requirements regarding hardware, software resources, and processing parallelism. Users submit jobs, which compete for computing resources. - -The Slurm software system is a resource manager and a job scheduler, which is designed to allocate resources and schedule jobs. Slurm is an open-source software, with a large user community, and has been installed on many top 500 supercomputers. - -- This tutorial assumes you have a NYU HPC account. If not, you may find the steps to apply for an account on the [Getting and renewing an account page](../01_getting_started/02_getting_and_renewing_an_account.mdx). - -- It also assumes you are comfortable with Linux command-line environment. To learn about linux please read our [Linux Tutorial](../12_tutorial_intro_shell_hpc/01_intro.mdx). - -- Please review the [Hardware Specs page](../10_spec_sheet.md) for more information on Greene's hardware specifications. - -## Slurm Commands - -For an overview of useful Slurm commands, please read [Slurm Main Commands](./02_slurm_main_commands.md) page before continuing the tutorial. - -## Software and Environment Modules - -Lmod, an Environment Module system, is a tool for managing multiple versions and configurations of software packages and is used by many HPC centers around the world. With Environment Modules, software packages are installed away from the base system directories, and for each package, an associated modulefile describes what must be altered in a user's shell environment - such as the $PATH environment variable - in order to use the software package. The modulefile also describes dependencies and conflicts between this software package and other packages and versions. - -To use a given software package, you load the corresponding module. Unloading the module afterwards cleanly undoes the changes that loading the modules made to your environment, thus freeing you to use other software packages that might have conflicted with the first one. - -Below is a list of modules and their associated functions: - -- `module load ` : loads a module - - For example : `module load python` - -- `module unload ` : unloads a module - - For example : `module unload python` - -- `module show ` : see exactly what effect loading a module will have - -- `module purge` : remove all loaded modules from your environment - -- `module whatis ` : Find out more about a software package - -- `module list` : check which modules are currently loaded in your environment - -- `module avail` : check what software packages are available - -- `module help ` : A module file may include more detailed help for software package - -## Batch Job Example - -Batch jobs require a script file for the SLURM scheduler to interpret and execute. The SBATCH file contains both commands specific for SLURM to interpret as well as programs for it to execute. Below is a simple example of a batch job to run a Stata do file, the file is named myscript.sbatch : - -```sh -#!/bin/bash - -#SBATCH --nodes=1 -#SBATCH --ntasks-per-node=1 -#SBATCH --cpus-per-task=1 -#SBATCH --time=5:00:00 -#SBATCH --mem=2GB -#SBATCH --job-name=myTest -#SBATCH --mail-type=END -#SBATCH --mail-user=bob.smith@nyu.edu -#SBATCH --output=slurm_%j.out - -module purge -module load stata/14.2 - -RUNDIR=$SCRATCH/my_project/run-${SLURM_JOB_ID/.*} -mkdir -p $RUNDIR -DATADIR=$SCRATCH/my_project/data -cd $RUNDIR - -stata -b do $DATADIR/data_0706.do -``` - -Below we will break down each line of the SBATCH script. More options can be found on the [SchedMD website](https://slurm.schedmd.com/documentation.html). - -```sh -## This tells the shell how to execute the script -#!/bin/bash - -## The #SBATCH lines are read by SLURM for options. -## In the lines below we ask for a single node, -## one task for that node, and one cpu for each task. -#SBATCH --nodes=1 -#SBATCH --ntasks-per-node=1 -#SBATCH --cpus-per-task=1 - -## Time is the estimated time to complete, in this case 5 hours. -#SBATCH --time=5:00:00 - -## We expect no more than 2GB of memory to be needed -#SBATCH --mem=2GB - -## To make them easier to track, -## it's best to name jobs something recognizable. -## You can then use the name to look up reports with tools like squeue. -#SBATCH --job-name=myTest - -## These lines manage mail alerts for when the job ends, -## and who the email should be sent to. -#SBATCH --mail-type=END -#SBATCH --mail-user=bob.smith@nyu.edu - -## This places the standard output and standard error into the same file, -## in this case slurm_.out -#SBATCH --output=slurm_%j.out - -## First we ensure a clean environment by purging the current one -module purge - -## Load the desired software, in this case stata 14.2 -module load stata/14.2 - -## Create a unique directory to run the job in. -RUNDIR=$SCRATCH/my_project/run-${SLURM_JOB_ID/.*} -mkdir -p $RUNDIR - -## Set an environment variable for where the data is stored. -DATADIR=$SCRATCH/my_project/data - -## Change directories to the unique run directory -cd $RUNDIR - -## Execute the desired Stata do file script -stata -b do $DATADIR/data_0706.do -``` - -You can submit the job with the following command: - -```sh -sbatch myscript.sbatch -``` - -The command will result in the job queuing as it awaits resources to become available (which varies on the number of other jobs being run on the cluster and the resources requested). You can see the status of your jobs with the following command: - -```sh -squeue --me -``` - -> **_NOTE:_** Calling just squeue without passing the `--me` option will display all users' job queue status by default - -Lastly, you can read the output of your job in the ` slurm- `.out file produced by running your job. This is where logs regarding the execution of your job can be found, including errors or system messages. You can print the contents to the screen from the directory containing the output file with the following command: - -```sh -cat slurm-.out -``` - -## Interactive Job Example - -While the majority of the jobs on the cluster are submitted with the `sbatch` command, and executed in the background, there are also methods to run applications interactively through the `srun` command. Interactive jobs allow the users to enter commands and data on the command line (or in a graphical interface), providing an experience similar to working on a desktop or laptop. Examples of common interactive tasks are: - -- Editing files - -- Compiling and debugging code - -- Exploring data, to obtain a rough idea of characteristics on the topic - -- Getting graphical windows to run visualization - -- Running software tools in interactive sessions - -Interactive jobs also help avoid issues with the login nodes. If you are working on a login node and your job is too IO intensive, it may be removed without notice. Running interactive jobs on compute nodes does not impact many users and in addition provides access to resources that are not available on the login nodes, such as interactive access to GPUs, high memory, exclusive access to all the resources of a compute node, etc. - -In the `srun` example below, through `--pty /bin/bash` we request allocation of a `pseudo terminal` (pty) and start a `bash shell session`. By default the resource allocated is a single CPU core and 2GB memory for 1 hour time limit. - -```sh -srun --pty /bin/bash -``` - -To request resources such as 4 CPU cores, 4 GB memory for 2 hours of maximum duration, you can add the following arguments: - -```sh -srun --cpus-per-task=4 --time=2:00:00 --mem=4GB --pty /bin/bash -``` - -Similarly, to request one GPU card, 3 GB memory for a duration of 1.5 hours you can pass the following arguments to srun: - -```sh -srun --time=1:30:00 --mem=3GB --gres=gpu:1 --pty /bin/bash -``` - -Once the job begins you will notice your prompt change, for example: - -```shell-session -[mdw303@log-3 ~]$ srun --pty /bin/bash -srun: job 7864254 queued and waiting for resources -srun: job 7864254 has been allocated resources -[mdw303@cs080 ~]$ -``` - -You can see above that the prompt changed from log-3 ( one of the login nodes ) to cs080 ( one of the compute nodes ), meaning we have created a pseudo terminal and logged in with a bash shell on a compute node from our login node. - -You can now load modules, software and run them interactively on the compute node having the resources ( CPUs, memory, GPUs etc ) that we asked for. - -Below outlines the steps to start an interactive session and launch R: - -```sh -[sk6404@log-1 ~]$ srun --cpus-per-task=1 --pty /bin/bash -[sk6404@cs022 ~]$ module purge -[sk6404@cs022 ~]$ module load r/intel/4.0.3 -[sk6404@cs022 ~]$ module list -Currently Loaded Modules: - 1) intel/19.1.2 2) r/intel/4.0.3 -[sk6404@cs022 ~]$ R -R version 4.0.3 (2020-10-10) -- "Bunny-Wunnies Freak Out" -Copyright (C) 2020 The R Foundation for Statistical Computing -Platform: x86_64-centos-linux-gnu (64-bit) -R is free software and comes with ABSOLUTELY NO WARRANTY. -You are welcome to redistribute it under certain conditions. -Type 'license()' or 'licence()' for distribution details. - Natural language support but running in an English locale -R is a collaborative project with many contributors. -Type 'contributors()' for more information and -'citation()' on how to cite R or R packages in publications. -Type 'demo()' for some demos, 'help()' for on-line help, or -'help.start()' for an HTML browser interface to help. -Type 'q()' to quit R. -> 5 + 10 -[1] 15 -> q() -Save workspace image? [y/n/c]: n -[sk6404@cs022 ~]$ exit -exit -[sk6404@log-1 ~]$ -``` - -## MPI Job Example - -MPI stands for "Message Passing Interface" and is managed by a program, such as OpenMPI, to coordinate code and resources across the HPC cluster for your job to run workloads in parallel. You may have heard of HPC sometimes referred to as "parallel computing" because the ability to run many processes simultaneously - aka in parallel - is how the best efficiencies can be realized on the cluster. Users interested in MPI generally must compile the program they want to run using an MPI compiler. - -Greene supports many MPI compilers. We'll be using the OpenMPI GCC compiler in this tutorial. It can be loaded as a module: - -```sh -module load openmpi/gcc/4.1.6 -``` - -Below we will illustrate an example of how to compile a C script for MPI. Copy this into your working directory as ` hellompi.c ` : - -```C -#include -#include -#include - -int main(int argc, char *argv[], char *envp[]) { - int numprocs, rank, namelen; - char processor_name[MPI_MAX_PROCESSOR_NAME]; - - MPI_Init(&argc, &argv); - MPI_Comm_size(MPI_COMM_WORLD, &numprocs); - MPI_Comm_rank(MPI_COMM_WORLD, &rank); - MPI_Get_processor_name(processor_name, &namelen); - - printf("Process %d on %s out of %d\n", rank, processor_name, numprocs); - - MPI_Finalize(); -} -``` - -Once copied into your directory, load OpenMPI and compile it with the following: - -```sh -module load openmpi/gcc/4.1.6 -mpicc hellompi.c -o hellompi -``` - -Next, create a ` hellompi.sbatch ` script: - -```sh -#!/bin/bash - -#SBATCH --nodes=4 -#SBATCH --ntasks-per-node=1 -#SBATCH --cpus-per-task=1 -#SBATCH --time=1:00:00 -#SBATCH --mem=2GB -#SBATCH --job-name=hellompi -#SBATCH --output=hellompi.out - -# Load the default OpenMPI module. -module purge -module load openmpi/intel/4.1.1 - -# Run the hellompi program with mpirun. The -n flag is not required; -# mpirun will automatically figure out the best configuration from the -# Slurm environment variables. -mpirun ./hellompi -``` - -Run the job with the following command: - -```sh -sbatch hellompi.sbatch -``` - -After the job runs, cat the ` hellompi.out ` output file to see that your processes ran on multiple nodes. There may be some errors, but your output should contain something like the following, indicating the process was run in parallel on multiple nodes: - -``` -Process 0 on cs265.nyu.cluster out of 4 -Process 1 on cs266.nyu.cluster out of 4 -Process 2 on cs267.nyu.cluster out of 4 -Process 3 on cs268.nyu.cluster out of 4 -``` - -## GPU Job Example - -To request one GPU card, use SBATCH directives in job script: - -```sh -#SBATCH --gres=gpu:1 -``` - -To request a specific card type, use e.g. ` --gres=gpu:v100:1 `. The card types currently available are: -- NVIDIA - - RTX 8000 - - V100 - - A100 NVIDIA 8358 - - A100 NVIDIA 8380 - - H100 NVIDIA -- AMD - - MI100 - - MI250 - - As an example, let's submit an Amber job. Amber is a molecular dynamics software package. The recipe is: - -```sh -mkdir -p /scratch/$USER/myambertest -cd /scratch/$USER/myambertest -cp /share/apps/Tutorials/slurm/example/amberGPU/* . -sbatch run-amber.s -``` - -From the tutorial example directory we copy over Amber input data files "inpcrd", "prmtop" and "mdin", and the job script file "run-amber.s". - -> **_NOTE:_** At the time of writing this you may need to update the run-amber.s script to load amber version 20.06, rather than the default 16.06. - -The content of the job script "run-amber.s" should be as follows: - -```sh -#!/bin/bash - -#SBATCH --job-name=myAmberJobGPU -#SBATCH --nodes=1 -#SBATCH --cpus-per-task=1 -#SBATCH --time=00:30:00 -#SBATCH --mem=3GB -#SBATCH --gres=gpu:1 - -module purge -module load amber/openmpi/intel/20.06 - -cd /scratch/$USER/myambertest -pmemd.cuda -O -``` - -The demo Amber job should take ~2 minutes to finish once it starts running. When the job is done, several output files are generated. Check the one named `mdout`, which has a section most relevant here: - -``` -|--------------------- INFORMATION ---------------------- -| GPU (CUDA) Version of PMEMD in use: NVIDIA GPU IN USE. -| Version 16.0.0 -| -| 02/25/2016 -[......] - -|------------------- GPU DEVICE INFO -------------------- -| -| CUDA_VISIBLE_DEVICES: 0 -| CUDA Capable Devices Detected: 1 -| CUDA Device ID in use: 0 -| CUDA Device Name: Tesla V100 -| CUDA Device Global Mem Size: 11439 MB -| CUDA Device Num Multiprocessors: 13 -| CUDA Device Core Freq: 0.82 GHz -| -|-------------------------------------------------------- -``` - -## Array Job Example - -Using job array you may submit many similar jobs with almost identical job requirement. This reduces loads on both users and the scheduler system. Job arrays can only be used in batch jobs. Usually the only requirement difference among jobs in a job array is the input file or files. Please follow the recipe below to try the example. There are 5 input files named `sample-1.txt`, `sample-2.txt` to `sample-5.txt` in sequential order. Running one command ` sbatch run-jobarray.s `, you submit 5 jobs to process each of these input files individually. Run the following commands to create the directory and submit the array job: - -```sh -mkdir -p /scratch/$USER/myjarraytest -cd /scratch/$USER/myjarraytest -cp /share/apps/Tutorials/slurm/example/jobarray/* . -ls -``` - -> **_OUTPUT:_** run-jobarray.s sample-1.txt sample-2.txt sample-3.txt sample-4.txt sample-5.txt wordcount.py - -```sh -sbatch --array=1-5 run-jobarray.s -``` - -The content of the job script ` run-jobarray.s ` is copied below: - -```sh -#!/bin/bash - -#SBATCH --job-name=myJobarrayTest -#SBATCH --nodes=1 -#SBATCH --tasks-per-node=1 -#SBATCH --array=1-5 # this creates an array! -#SBATCH --time=5:00 -#SBATCH --mem=1GB -#SBATCH --output=wordcounts_%A_%a.out -#SBATCH --error=wordcounts_%A_%a.err - -module purge -module load python/intel/3.8.6 - -cd /scratch/$USER/myjarraytest -python2 wordcount.py sample-$SLURM_ARRAY_TASK_ID.txt -``` - -Job array submissions create an environment variable called ` SLURM_ARRAY_TASK_ID `, which is unique for each job in the array job. It is usually embedded somewhere so that at a job running time it's unique value is incorporated into producing a proper file name. Also as shown above: two additional options %A and %a, denoting the job ID and the task ID (i.e. job array index) respectively, are available for specifying a job's stdout, and stderr file names.