Skip to content

Commit 38c4530

Browse files
s-sajid-alis-sajid-ali
committed
goodbye gpfs and minor reformat
 modified: docs/hpc/03_storage/01_intro_and_data_management.mdx modified: docs/hpc/03_storage/02_available_storage_systems.md modified: docs/hpc/03_storage/05_research_project_space.mdx
1 parent 17bb571 commit 38c4530

3 files changed

Lines changed: 46 additions & 91 deletions

File tree

‎docs/hpc/03_storage/01_intro_and_data_management.mdx‎

Lines changed: 41 additions & 69 deletions
Original file line numberDiff line numberDiff line change
@@ -1,40 +1,28 @@
11
# HPC Storage
22

3-
The NYU HPC clusters are served by a General Parallel File System (GPFS) cluster and an all Flash VAST storage cluster.
4-
5-
The NYU HPC team supports data storage, transfer, and archival needs on the HPC clusters, as well as collaborative research services like the [Research Project Space (RPS)](./05_research_project_space.mdx).
6-
7-
## Highlights
8-
- 9.5 PB Total GPFS Storage
9-
- Up to 78 GB per second read speeds
10-
- Up to 650k input/output operations per second (IOPS)
11-
- Research Project Space (RPS): RPS volumes provide working spaces for sharing data and code amongst project or lab members
3+
The Torch HPC cluster is served by an all Flash VAST storage cluster. The HPC team supports data storage, transfer, and archival needs on the HPC clusters, as well as collaborative research services like the [Research Project Space (RPS)](./05_research_project_space.mdx).
124

135
## Introduction to HPC Data Management
14-
The NYU HPC Environment provides access to a number of ***file systems*** to better serve the needs of researchers managing data during the various stages of the research data lifecycle (data capture, analysis, archiving, etc.). Each HPC file system comes with different features, policies, and availability.
15-
16-
In addition, a number of ***data management tools*** are available that enable data transfers and data sharing, recommended best practices, and various scenarios and use cases of managing data in the HPC Environment.
6+
The NYU HPC Environment provides access to a number of file-systems to better serve the needs of researchers managing data during the various stages of the research data life cycle (data capture, analysis, archiving, etc.). Each HPC file-system comes with different features, policies, and availability.
177

18-
Multiple ***public data sets*** are available to all users of the HPC environment, such as a subset of The Cancer Genome Atlas (TCGA), the Million Song Database, ImageNet, and Reference Genomes.
8+
In addition, a number of data management tools are available that enable data transfers and data sharing, recommended best practices, and various scenarios and use cases of managing data in the HPC Environment. We strongly recommend [`globus`](./04_globus.md) as the primary data transfer tool.
199

20-
Below is a list of file systems with their characteristics and a summary table. Reviewing the list of available file systems and the various Scenarios/Use cases that are presented below, can help select the right file systems for a research project. As always, if you have any questions about data storage in the HPC environment, you can request a consultation with the HPC team by sending email to [hpc@nyu.edu](mailto:hpc@nyu.edu).
10+
Multiple ***public data sets*** are available to all users of the HPC environment, such as a subset of The Cancer Genome Atlas (`TCGA`), the Million Song Database, `ImageNet`, and Reference Genomes. More information can be found in the [datasets section](../04_datasets/01).
2111

22-
### Data Security Warning
23-
::::warning
24-
#### Moderate Risk Data - HPC Approved
12+
:::warning[HPC is only approved for Moderate Risk Data]
2513
- The HPC Environment has been approved for storing and analyzing **Moderate Risk research data**, as defined in the [NYU Electronic Data and System Risk Classification Policy](https://www.nyu.edu/about/policies-guidelines-compliance/policies-and-guidelines/electronic-data-and-system-risk-classification.html).
26-
- **High Risk** research data, such as those that include Personal Identifiable Information (**PII**) or electronic Protected Health Information (**ePHI**) or Controlled Unclassified Information (**CUI**) **should NOT be stored in the HPC Environment**.
27-
:::note
28-
only the Office of Sponsored Projects (OSP) and Global Office of Information Security (GOIS) are empowered to classify the risk categories of data.
14+
- **High Risk** research data, such as those that include Personal Identifiable Information (**PII**) or electronic Protected Health Information (**ePHI**) or Controlled Unclassified Information (**CUI**) **should NOT be stored in the HPC Environment**.
15+
- Only the Office of Sponsored Projects (OSP) and Global Office of Information Security (GOIS) are empowered to classify the risk categories of data.
2916
:::
30-
:::tip
31-
#### High Risk Data - Secure Research Data Environments (SRDE) Approved
32-
Because the HPC system is not approved for High Risk data, we recommend using an approved system like the [Secure Research Data Environments (SRDE)](../../srde/01_getting_started/01_intro.md).
17+
:::info[SRDE for High Risk Data]
18+
Because the HPC system is not approved for High Risk data, we recommend using an approved system like the [Secure Research Data Environments (SRDE)](../../srde/01_getting_started/01_intro.md).
3319
:::
34-
::::
3520

36-
### Data Storage options in the HPC Environment
37-
#### User Home Directories
21+
## Data Storage options in the HPC Environment
22+
23+
Below is a list of file-systems with their characteristics and a summary table. Reviewing the list of available file-systems and the various Scenarios/Use cases that are presented below, can help select the right file-systems for a research project.
24+
25+
### User Home Directories
3826
Every individual user has a home directory (under **`/home/$USER`**, environment variable **`$HOME`**) for permanently storing code and important configuration files. Home Directories provide limited storage space (**50 GB**) and inodes (files) **30,000** per user. Users can check their quota utilization using the [myquota](http://www.info-ren.org/projects/ckp/tech/software/version/myquota.html) command.
3927

4028
User home directories are backed up daily and old files under **`$HOME`** are not purged.
@@ -44,79 +32,57 @@ The User home directories are available on all HPC clusters (Torch) and on every
4432
:::warning
4533
Avoid changing file and directory permissions in your home directory to allow other users to access files.
4634
:::
47-
User Home Directories are not ideal for sharing files and folders with other users. HPC Scratch or [Research Project Space (RPS)](./05_research_project_space.mdx) are better file systems for sharing data.
35+
User Home Directories are not ideal for sharing files and folders with other users. HPC Scratch or [Research Project Space (RPS)](./05_research_project_space.mdx) are better file-systems for sharing data.
4836

4937
:::warning
5038
**One of the common issues that users report regarding their home directories is running out of inodes,** i.e. the number of files stored under their home exceeds the inode limit, which by default is set to 30,000 files. This typically occurs when users install software under their home directories, for example, when working with Conda and Julia environments, that involve many small files.
5139
:::
5240

5341
:::tip
5442
- To find out the current space and inode quota utilization and the distribution of files under your home directory, please see: [Understanding user quota limits and the myquota command.](./06_best_practices.md#user-quota-limits-and-the-myquota-command)
55-
- **Working with Conda environments:** To avoid running out of inode limits in home directories, the HPC team recommends **setting up conda environments with Singularity overlay images**
43+
- Working with Conda environments: To avoid running out of inode limits in home directories, the HPC team recommends **setting up conda environments with Singularity overlay images**
5644
:::
5745

58-
#### HPC Scratch
59-
The HPC scratch file system is the HPC file system where most of the users store research data needed during the analysis phase of their research projects. The scratch file system provides ***temporary*** storage for datasets needed for running jobs.
46+
### HPC Scratch
47+
The HPC scratch an all flash (VAST) file-system where most of the users store research data needed during the analysis phase of their research projects. The scratch file-system provides ***temporary*** storage for datasets needed for running jobs.
6048

61-
Files stored in the HPC scratch file system are subject to the <ins>**HPC Scratch old file purging policy:** Files on the /scratch file system that have not been accessed for 60 or more days will be purged.</ins>
49+
Files stored in the HPC scratch file-system are subject to the <ins>**HPC Scratch old file purging policy:** Files on the /scratch file-system that have not been accessed for 60 or more days will be purged.</ins>
6250

63-
Every user has a dedicated scratch directory (**/scratch/$USER**) with **5 TB** disk quota and **1,000,000 inodes** (files) limit per user.
51+
Every user has a dedicated scratch directory (**/scratch/$USER**) with **5 TB** disk quota and **5,000,000 inodes** (files) limit per user.
6452

65-
The scratch file system is available on all nodes (compute, login, etc.) on Torch as well as Data Transfer Node (gDTN).
53+
The scratch file-system is available on all nodes (compute, login, etc.) on Torch as well as Data Transfer Node (gDTN).
6654

6755
:::warning
68-
There are **No Back ups of the scratch file system.** ***Files that were deleted accidentally or removed due to storage system failures CAN NOT be recovered.***
56+
There are no backups of the scratch file-system. Files that were deleted accidentally or removed due to storage system failures CANNOT be recovered.
6957
:::
7058

7159
:::tip
7260

73-
- Since there are ***no back ups of HPC Scratch file system***, users should not put important source code, scripts, libraries, executables in `/scratch`. These important files should be stored in file systems that are backed up, such as `/home` or [Research Project Space (RPS)](./05_research_project_space.mdx). Code can also be stored in a ***git*** repository.
74-
- ***Old file purging policy on HPC Scratch:*** <ins>All files on the HPC Scratch file system that have not been accessed ***for more than 60 days*** will be removed. It is a policy violation to use scripts to change the file access time.</ins> Any user found to be violating this policy will have their HPC account locked. A second violation may result in your HPC account being turned off.
61+
- Since there are ***no backups of HPC Scratch file-system***, users should not put important source code, scripts, libraries, executables in `/scratch`. These important files should be stored in file-systems that are backed up, such as `/home` or [Research Project Space (RPS)](./05_research_project_space.mdx). Code can also be stored in a ***git*** repository.
62+
- ***Old file purging policy on HPC Scratch:*** <ins>All files on the HPC Scratch file-system that have not been accessed ***for more than 60 days*** will be removed. It is a policy violation to use scripts to change the file access time.</ins> Any user found to be violating this policy will have their HPC account locked. A second violation may result in your HPC account being turned off.
7563
- To find out the user's current disk space and inode quota utilization and the distribution of files under your scratch directory, please see: [Understanding user quota Limits and the myquota command.](./06_best_practices.md#user-quota-limits-and-the-myquota-command)
76-
- Once a research project completes, users should archive their important files in the [HPC Archive file system](./01_intro_and_data_management.mdx#hpc-archive).
77-
:::
78-
79-
#### HPC Vast
80-
The HPC Vast all-flash file system is the HPC file system where users store research data needed during the analysis phase of their research projects, particularly for high I/O data that can bottleneck on the scratch file system. The Vast file system provides ***temporary*** storage for datasets needed for running jobs.
81-
82-
Files stored in the HPC vast file system are subject to the <ins>***HPC Vast old file purging policy:*** Files on the `/vast` file system that have not been accessed for **60 or more days** will be purged.</ins>
83-
84-
Every user has a dedicated vast directory (**`/vast/$USER`**) with **2 TB** disk quota and **5,000,000 inodes** (files) limit per user.
85-
86-
The vast file system is available on all nodes (compute, login, etc.) on Torch as well as Data Transfer Node (gDTN).
87-
88-
:::warning
89-
There are **No Back ups** of the vastsc file system. ***Files that were deleted accidentally or removed due to storage system failures CAN NOT be recovered.***
64+
- Once a research project completes, users should archive their important files in the [HPC Archive file-system](./01_intro_and_data_management.mdx#hpc-archive).
9065
:::
9166

92-
:::tip
93-
- Since there are ***no back ups of HPC Vast file system***, users should not put important source code, scripts, libraries, executables in `/vast`. These important files should be stored in file systems that are backed up, such as `/home` or [Research Project Space (RPS)](./05_research_project_space.mdx). Code can also be stored in a ***git*** repository.
94-
- ***Old file purging policy on HPC Vast:*** <ins>All files on the HPC Vast file system that have not been accessed ***for more than 60 days will be removed.*** It is a policy violation to use scripts to change the file access time.</ins> Any user found to be violating this policy will have their HPC account locked. A second violation may result in your HPC account being turned off.
95-
- To find out the user's current disk space and inode quota utilization and the distribution of files under your vast directory, please see: [Understanding user quota Limits and the myquota command.](./06_best_practices.md#user-quota-limits-and-the-myquota-command)
96-
- Once a research project completes, users should archive their important files in the [HPC Archive file system](./01_intro_and_data_management.mdx#hpc-archive).
97-
:::
98-
99-
#### HPC Research Project Space
100-
The HPC Research Project Space (RPS) provides data storage space for research projects that is easily shared amongst collaborators, ***backed up***, and ***not subject to the old file purging policy***. HPC RPS was introduced to ease data management in the HPC environment and eliminate the need of having to frequently copying files between Scratch and Archive file systems by having all projects files under one area. ***These benefits of the HPC RPS come at a cost***. The cost is determined by the allocated disk space and the number of files (inodes).
67+
### HPC Research Project Space
68+
The HPC Research Project Space (RPS) provides data storage space for research projects that is easily shared amongst collaborators, ***backed up***, and ***not subject to the old file purging policy***. HPC RPS was introduced to ease data management in the HPC environment and eliminate the need of having to frequently copying files between Scratch and Archive file-systems by having all projects files under one area. ***These benefits of the HPC RPS come at a cost***. The cost is determined by the allocated disk space and the number of files (inodes).
10169
- For detailed information about RPS see: [HPC Research Project Space](./05_research_project_space.mdx)
10270

103-
#### HPC Work
104-
The HPC team makes available a number of public datasets that are commonly used in analysis jobs. The data sets are available Read-Only under **`/scratch/work/public`**.
105-
106-
For some of the datasets users must provide a signed usage agreement before accessing.
71+
### HPC Work
72+
The HPC team makes available a number of public datasets that are commonly used in analysis jobs. The data sets are available Read-Only under **`/scratch/work/public`**. For some of the datasets users must provide a signed usage agreement before accessing.
10773

10874
Public datasets available on the HPC clusters can be viewed on the [Datasets page](../04_datasets/01_intro.md).
10975

110-
#### HPC Archive
111-
Once the Analysis stage of the research data lifecycle has completed, <ins>_HPC users should **tar** their data and code into a single tar.gz file and then copy the file to their archive directory (**`/archive/$USER`**_).</ins> The HPC Archive file system is not accessible by running jobs; it is suitable for long-term data storage. Each user has access to a default disk quota of **2TB** and ***20,000 inode (files) limit***. The rather low limit on the number of inodes per user is intentional. The archive file system is available only ***on login nodes*** of Torch. The archive file system is backed up daily.
76+
### HPC Archive
77+
Once the Analysis stage of the research data lifecycle has completed, <ins>HPC users should **tar** their data and code into a single tar.gz file and then copy the file to their archive directory (`/archive/$USER`).</ins> The HPC Archive file-system is not accessible by running jobs; it is suitable for long-term data storage. Each user has access to a default disk quota of **2TB** and ***20,000 inode (files) limit***. The rather low limit on the number of inodes per user is intentional. The archive file-system is available only ***on login nodes*** of Torch. The archive file-system is backed up daily.
11278

113-
- Here is an example ***tar*** command that combines the data in a directory named ***my_run_dir*** under ***`$SCRATCH`*** and outputs the tar file in the user's ***`$ARCHIVE`***:
79+
- Here is an example ***tar*** command that combines the data in a directory named `my_run_dir` under `$SCRATCH` and outputs the tar file in the user's `$ARCHIVE`:
11480
```sh
11581
# to archive `$SCRATCH/my_run_dir`
11682
tar cvf $ARCHIVE/simulation_01.tar -C $SCRATCH my_run_dir
11783
```
11884

119-
#### NYU (Google) Drive
85+
### NYU (Google) Drive
12086
Google Drive ([NYU Drive](https://www.nyu.edu/life/information-technology/communication-and-collaboration/document-collaboration-and-sharing/nyu-drive.html)) is accessible from the NYU HPC environment and provides an option to users who wish to archive data or share data with external collaborators who do not have access to the NYU HPC environment.
12187

12288
As of December 2023, storage limits were applied to all faculty, staff, and sutdent NYU Google accounts. Please see [Google Workspace Storage](https://www.nyu.edu/life/information-technology/about-nyu-it/key-projects-and-initiatives/google-workspace-storage.html) for details
@@ -125,7 +91,13 @@ There are also limits to the data transfer rate in moving to/from Google Drive.
12591

12692
Please read the [Instructions on how to use cloud storage within the NYU HPC Environment](./08_transferring_cloud_storage_data_with_rclone.md).
12793

128-
#### HPC Storage Mounts Comparison Table
129-
<iframe width="100%" height="300em" src="https://docs.google.com/spreadsheets/d/e/2PACX-1vT-q0rRueYg1Be_gcWSghB-GGFDonP8DaXNnm8Qi036w-Vi_l7CCOav4IPxi1yZy8TSnTRFy7S5dNTJ/pubhtml?widget=true&amp;headers=false"></iframe>
94+
### HPC Storage Comparison Table
95+
96+
| Space | Environment Variable | Purpose | Backed Up / Flushed | Quota Disk Space / # of Files |
97+
|-----------------------------|----------------------|-------------------------------------------------------|-------------------------------------|------------------------------------|
98+
| /home | $HOME | Personal user home space that is best for small files | YES / NO | 50 GB / 30 K |
99+
| /scratch | $SCRATCH | Best for large files | NO / Files not accessed for 60 days | 5 TB / 5 M |
100+
| /archive | $ARCHIVE | Long-term storage | YES / NO | 2 TB / 20 K |
101+
| HPC Research Project Space | NA | Shared disk space for research projects | YES / NO | Payment based TB-year/inodes-year |
130102

131103
Please see the next page for best practices for data management on NYU HPC systems.

0 commit comments

Comments
 (0)