You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/hpc/03_storage/01_intro_and_data_management.mdx
+13-28Lines changed: 13 additions & 28 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -23,42 +23,31 @@ Because the HPC system is not approved for High Risk data, we recommend using an
23
23
Below is a list of file-systems with their characteristics and a summary table. Reviewing the list of available file-systems and the various Scenarios/Use cases that are presented below, can help select the right file-systems for a research project.
24
24
25
25
### User Home Directories
26
-
Every individual user has a home directory (under **`/home/$USER`**, environment variable **`$HOME`**) for permanently storing code and important configuration files. Home Directories provide limited storage space (**50 GB**) and inodes (files) **30,000** per user. Users can check their quota utilization using the [myquota](http://www.info-ren.org/projects/ckp/tech/software/version/myquota.html) command.
27
-
28
-
User home directories are backed up daily and old files under **`$HOME`** are not purged.
29
-
30
-
The User home directories are available on all HPC clusters (Torch) and on every cluster node (login nodes, compute nodes) as well as and Data Transfer Node (gDTN).
26
+
Every individual user has a home directory (under **`/home/$USER`**, environment variable **`$HOME`**) for permanently storing code and important configuration files. Home Directories provide limited storage space (**50 GB**) and inodes (files) **30,000** per user. Users can check their quota utilization using the [myquota](http://www.info-ren.org/projects/ckp/tech/software/version/myquota.html) command. User home directories are backed up daily and old files under **`$HOME`** are not purged. The user home directories are available on every cluster node (login nodes, compute nodes) as well as and the Data Transfer Node (gDTN).
31
27
32
28
:::warning
33
29
Avoid changing file and directory permissions in your home directory to allow other users to access files.
34
30
:::
35
-
User Home Directories are not ideal for sharing files and folders with other users. HPC Scratch or [Research Project Space (RPS)](./05_research_project_space.mdx) are better file-systems for sharing data.
36
31
37
-
:::warning
38
-
**One of the common issues that users report regarding their home directories is running out of inodes,** i.e. the number of files stored under their home exceeds the inode limit, which by default is set to 30,000 files. This typically occurs when users install software under their home directories, for example, when working with Conda and Julia environments, that involve many small files.
39
-
:::
32
+
User Home Directories are not ideal for sharing files and folders with other users. HPC Scratch or [Research Project Space (RPS)](./05_research_project_space.mdx) are better file-systems for sharing data.
40
33
41
-
:::tip
34
+
:::tip[`inode` limits]
35
+
- One of the common issues that users report regarding their home directories is running out of inodes (i.e. the number of files stored under their home exceeds the inode limit), which by default is set to 30,000 files
42
36
- To find out the current space and inode quota utilization and the distribution of files under your home directory, please see: [Understanding user quota limits and the myquota command.](./06_best_practices.md#user-quota-limits-and-the-myquota-command)
43
-
- Working with Conda environments: To avoid running out of inode limits in home directories, the HPC team recommends **setting up conda environments with Singularity overlay images**
37
+
- Working with `conda` environments: To avoid running out of inode limits in home directories, the HPC team recommends **setting up `conda` environments with Singularity overlay images** as [described here](../07_containers/03_singularity_with_conda.md). Avoid creating `conda` environments in your `$HOME` directory.
44
38
:::
45
39
46
40
### HPC Scratch
47
-
The HPC scratch an all flash (VAST) file-system where most of the users store research data needed during the analysis phase of their research projects. The scratch file-system provides ***temporary*** storage for datasets needed for running jobs.
41
+
The HPC scratch an all flash (VAST) file-system where most of the users store research data needed during the analysis phase of their research projects. The scratch file-system provides ***temporary*** storage for datasets needed for running jobs. Every user has a dedicated scratch directory (**/scratch/$USER**) with **5 TB** disk quota and **5,000,000 inodes** (files) limit per user. The scratch file-system is available on all nodes (compute, login, etc.) on Torch as well as Data Transfer Node (gDTN).
48
42
49
-
Files stored in the HPC scratch file-system are subject to the <ins>**HPC Scratch old file purging policy:** Files on the /scratch file-system that have not been accessed for 60 or more days will be purged.</ins>
50
-
51
-
Every user has a dedicated scratch directory (**/scratch/$USER**) with **5 TB** disk quota and **5,000,000 inodes** (files) limit per user.
52
-
53
-
The scratch file-system is available on all nodes (compute, login, etc.) on Torch as well as Data Transfer Node (gDTN).
54
-
55
-
:::warning
56
-
There are no backups of the scratch file-system. Files that were deleted accidentally or removed due to storage system failures CANNOT be recovered.
43
+
:::warning[Scratch Purging Policy]
44
+
- Files on the /scratch file-system that have not been accessed for 60 or more days will be purged.
45
+
0 There are no backups of the scratch file-system. Files that were deleted accidentally or removed due to storage system failures CANNOT be recovered.
57
46
:::
58
47
59
48
:::tip
60
49
61
-
- Since there are ***no backups of HPC Scratch file-system***, users should not put important source code, scripts, libraries, executables in `/scratch`. These important files should be stored in file-systems that are backed up, such as `/home` or [Research Project Space (RPS)](./05_research_project_space.mdx). Code can also be stored in a ***git*** repository.
50
+
- Since there are ***no backups of HPC Scratch file-system***, users should not put important source code, scripts, libraries, executables in `/scratch`. These important files should be stored in file-systems that are backed up, such as `/home` or [Research Project Space (RPS)](./05_research_project_space.mdx). Code can also be stored in a `git` repository.
62
51
-***Old file purging policy on HPC Scratch:*** <ins>All files on the HPC Scratch file-system that have not been accessed ***for more than 60 days*** will be removed. It is a policy violation to use scripts to change the file access time.</ins> Any user found to be violating this policy will have their HPC account locked. A second violation may result in your HPC account being turned off.
63
52
- To find out the user's current disk space and inode quota utilization and the distribution of files under your scratch directory, please see: [Understanding user quota Limits and the myquota command.](./06_best_practices.md#user-quota-limits-and-the-myquota-command)
64
53
- Once a research project completes, users should archive their important files in the [HPC Archive file-system](./01_intro_and_data_management.mdx#hpc-archive).
@@ -69,9 +58,7 @@ The HPC Research Project Space (RPS) provides data storage space for research pr
69
58
- For detailed information about RPS see: [HPC Research Project Space](./05_research_project_space.mdx)
70
59
71
60
### HPC Work
72
-
The HPC team makes available a number of public datasets that are commonly used in analysis jobs. The data sets are available Read-Only under **`/scratch/work/public`**. For some of the datasets users must provide a signed usage agreement before accessing.
73
-
74
-
Public datasets available on the HPC clusters can be viewed on the [Datasets page](../04_datasets/01_intro.md).
61
+
The HPC team makes available a number of public datasets that are commonly used in analysis jobs. The data-sets are available Read-Only under `/scratch/work/public`. For some of the datasets users must provide a signed usage agreement before accessing. Public datasets available on the HPC clusters can be viewed on the [Datasets page](../04_datasets/01_intro.md).
75
62
76
63
### HPC Archive
77
64
Once the Analysis stage of the research data lifecycle has completed, <ins>HPC users should **tar** their data and code into a single tar.gz file and then copy the file to their archive directory (`/archive/$USER`).</ins> The HPC Archive file-system is not accessible by running jobs; it is suitable for long-term data storage. Each user has access to a default disk quota of **2TB** and ***20,000 inode (files) limit***. The rather low limit on the number of inodes per user is intentional. The archive file-system is available only ***on login nodes*** of Torch. The archive file-system is backed up daily.
@@ -85,11 +72,9 @@ tar cvf $ARCHIVE/simulation_01.tar -C $SCRATCH my_run_dir
85
72
### NYU (Google) Drive
86
73
Google Drive ([NYU Drive](https://www.nyu.edu/life/information-technology/communication-and-collaboration/document-collaboration-and-sharing/nyu-drive.html)) is accessible from the NYU HPC environment and provides an option to users who wish to archive data or share data with external collaborators who do not have access to the NYU HPC environment.
87
74
88
-
As of December 2023, storage limits were applied to all faculty, staff, and sutdent NYU Google accounts. Please see [Google Workspace Storage](https://www.nyu.edu/life/information-technology/about-nyu-it/key-projects-and-initiatives/google-workspace-storage.html) for details
89
-
90
-
There are also limits to the data transfer rate in moving to/from Google Drive. Thus, moving many small files to Google Drive is not going to be efficient.
75
+
As of December 2023, storage limits were applied to all faculty, staff, and student NYU Google accounts. Please see [Google Workspace Storage](https://www.nyu.edu/life/information-technology/about-nyu-it/key-projects-and-initiatives/google-workspace-storage.html) for details.
91
76
92
-
Please read the [Instructions on how to use cloud storage within the NYU HPC Environment](./08_transferring_cloud_storage_data_with_rclone.md).
77
+
There are also limits to the data transfer rate in moving to/from Google Drive. Thus, moving many small files to Google Drive is not going to be efficient. Please read the [Instructions on how to use cloud storage within the NYU HPC Environment](./08_transferring_cloud_storage_data_with_rclone.md).
0 commit comments