diff --git a/docs/hpc/03_storage/03_data_transfers.md b/docs/hpc/03_storage/03_data_transfers.md index cfc2a61309..9658f78fdb 100644 --- a/docs/hpc/03_storage/03_data_transfers.md +++ b/docs/hpc/03_storage/03_data_transfers.md @@ -1,15 +1,17 @@ # Data Transfers +:::tip Globus +Globus is the recommended tool to use for large-volume data transfers due to the efficiency, reliability, security and ease of use. Use other tools only if you really need to. Detailed instructions available at [Globus](./04_globus.md) +::: + ## Data-Transfer nodes Attached to the NYU HPC cluster Greene, the Greene Data Transfer Node (gDTN) are nodes optimized for transferring data between cluster file systems (e.g. scratch) and other endpoints outside the NYU HPC clusters, including user laptops and desktops. The gDTNs have 100-Gb/s Ethernet connections to the High Speed Research Network (HSRN) and are connected to the HDR Infiniband fabric of the HPC clusters. More information on the hardware characteristics is available at [Greene spec sheet](../10_spec_sheet.mdx). ### Data Transfer Node Access -The HPC cluster filesystems include `/home`, `/scratch`, `/archive` and the [HPC Research Project Space](./05_research_project_space.mdx) are available on the gDTN. - -The Data-Transfer Node (DTN) can be accessed in a variety of ways +The HPC cluster filesystems include `/home`, `/scratch`, `/archive` and the [HPC Research Project Space](./05_research_project_space.mdx) are available on the gDTN. The Data-Transfer Node (DTN) can be accessed in a variety of ways - From NYU-net and the High Speed Research Network: use SSH to the DTN hostname `gdtn.hpc.nyu.edu` - From the Greene cluster (e.g., the login nodes): the hostname can be shortened to `gdtn` -:::tip +:::info For example, to log in to a DTN from the Greene cluster, to carry out some copy operation, and to log back out, you can use a command sequence like: ```sh ssh gdtn @@ -19,6 +21,7 @@ logout ::: - Via specific tools like [Globus](#globus) + ## Tools for Data Transfer ### Linux & Mac Tools @@ -48,14 +51,10 @@ Windows 10 machines may have the Linux Subsystem installed, which will allow for ### Globus Globus is the recommended tool to use for large-volume data transfers. It features automatic performance tuning and automatic retries in cases of file-transfer failures. Data-transfer tasks can be submitted via a web portal. The Globus service will take care of the rest, to make sure files are copied efficiently, reliably, and securely. Globus is also a tool for you to share data with collaborators, for whom you only need to provide the email addresses. -The Globus endpoint for Greene is available at `nyu#greene`. The endpoint `nyu#prince` has been retired. - -Detailed instructions available at [Globus](./04_globus.md) +The Globus endpoint for Greene is available at `nyu#greene`. The endpoint `nyu#prince` has been retired. Detailed instructions available at [Globus](./04_globus.md) ### rclone -rclone - rsync for cloud storage, is a command line program to sync files and directories to and from cloud storage systems such as Google Drive, Amazon Drive, S3, B2 etc. rclone is available on DTNs. - -[Please see the documentation for how to use it.](https://rclone.org/) +rclone - rsync for cloud storage, is a command line program to sync files and directories to and from cloud storage systems such as Google Drive, Amazon Drive, S3, B2 etc. rclone is available on DTNs. [Please see the documentation for how to use it.](https://rclone.org/) ### Open OnDemand (OOD) One can use [Open OnDemand (OOD)](../09_ood/01_open_on_demand.md) interface to upload data. diff --git a/docs/hpc/03_storage/09_sharing_data_on_hpc.md b/docs/hpc/03_storage/09_sharing_data_on_hpc.md index f9cf24071c..0d3b7b647d 100644 --- a/docs/hpc/03_storage/09_sharing_data_on_hpc.md +++ b/docs/hpc/03_storage/09_sharing_data_on_hpc.md @@ -1,7 +1,7 @@ # Sharing Data on HPC ## Introduction -To share files on the cluster with other users, we recommend using file access control lists (FACL) for a user to share access to their data with others. FACL mechanism allows a fine-grained control access to any files by any users or groups of users. We discourage users from setting '777' permissions with `chmod`, because this can lead to data loss (by a malicious user or unintentionally, by accident). Linux commands `getfacl` and `setfacl` are used to view and set access. +To share files on the cluster with other users, we recommend using file access control lists (FACL) for a user to share access to their data with others. FACL mechanism allows a fine-grained control access to any files by any users or groups of users. We discourage users from setting `777` permissions with `chmod`, because this can lead to data loss (by a malicious user or unintentionally, by accident). Linux commands `getfacl` and `setfacl` are used to view and set access. ACL mechanism, just like regular Linux POSIX, allows three different levels of access control: diff --git a/docs/hpc/07_containers/01_intro.md b/docs/hpc/07_containers/01_intro.md index a8452b813c..ca5de13650 100644 --- a/docs/hpc/07_containers/01_intro.md +++ b/docs/hpc/07_containers/01_intro.md @@ -1,4 +1,4 @@ -# Singularity: Run Custom Applications with Containers +# Custom Applications with Containers ## What is Singularity Singularity is a container based Linux kernel workspace that works just like docker. You can run pre-built programs in containers without having to worry about the pre-install environment. @@ -7,14 +7,14 @@ For users who are familiar with Docker containers, Singularity works very simila For a detailed introduction on Singularity, visit their official site [here](https://apptainer.org/documentation/) -## why do we use Singularity +## Why do we use Singularity There are multiple reasons to use Singularity on the HPC clusters: 1. Security: Singularity provides a layer of security as it does not require any root access on our clusters. This makes it safer against malware and bad scripts that might jeopardize the outer system. Thus we only support Singularity on our clusters(there are not other options such as Kubernetes or Docker on our clusters right now) 2. Containerization: Singularity will run all your images(packaged and pre-built programs) inside of its containers, each container works like a small vm. They contain all the required environment and files of a single Linux kernel and you don't have to worry about any pre-installation nonsense 3. Inter-connectivity: Containers are able to talk to each other, as well as the home system, so while each container has its own small space, they are still a part of a big interconnected structure. Thus enabling you to connect your programs. 4. Accessibility: Probably the most important feature of all, Singularity allows you to run your program in 2 to 3 simple steps, as shown in the topic how to run a singularity container. -## how to run a singularity container +## How to run a singularity container There are 3 steps to run a Singularity container on our clusters: pulling a image from Singularity hub or Docker hub