remove spurious page

Author: s-sajid-ali

docs/cloud/04_dataproc/02_data_management.md

@@ -4,7 +4,7 @@ HDFS stands for Hadoop Distributed File System. HDFS is a highly fault-tolerant
 
 ### File Permissions and Access Control Lists
 
-You can share files with others using [access control lists (ACLs)](../../hpc/03_storage/09_sharing_data_on_hpc.md). An ACL gives you per-file, per-directory and per-user control over who has permission to access files. You can see the ACL for a file or directory with the getfacl command:
+You can share files with others using [access control lists (ACLs)](../../hpc/03_storage/08_sharing_data_on_hpc.md). An ACL gives you per-file, per-directory and per-user control over who has permission to access files. You can see the ACL for a file or directory with the getfacl command:
 ```sh
 hdfs dfs -getfacl  /user/<net_id>_nyu_edu/testdir
 ```

docs/hpc/03_storage/01_intro_and_data_management.mdx

@@ -1,13 +1,10 @@
 # HPC Storage
 
-The Torch HPC cluster is served by an all Flash VAST storage cluster. The HPC team supports data storage, transfer, and archival needs on the HPC clusters, as well as collaborative research services like the [Research Project Space (RPS)](./05_research_project_space.mdx).
-
-## Introduction to HPC Data Management
 The NYU HPC Environment provides access to a number of file-systems to better serve the needs of researchers managing data during the various stages of the research data life cycle (data capture, analysis, archiving, etc.). Each HPC file-system comes with different features, policies, and availability.  
 
-In addition, a number of data management tools are available that enable data transfers and data sharing, recommended best practices, and various scenarios and use cases of managing data in the HPC Environment. We strongly recommend [`globus`](./04_globus.md) as the primary data transfer tool.
+Numerous data management tools are available that enable data transfers and data sharing, recommended best practices, and various scenarios and use cases of managing data in the HPC Environment. We strongly recommend [`globus`](./03_globus.md) as the primary data transfer tool.
 
-Multiple ***public data sets*** are available to all users of the HPC environment, such as a subset of The Cancer Genome Atlas (`TCGA`), the Million Song Database, `ImageNet`, and Reference Genomes. More information can be found in the [datasets section](../04_datasets/01_intro.md). 
+Selected public datasets are available to all HPC users, such as a subset of The Cancer Genome Atlas (`TCGA`), the Million Song Database, `ImageNet`, and Reference Genomes. More information can be found in the [datasets section](../04_datasets/01_intro.md). 
 
 :::warning[HPC is only approved for Moderate Risk Data]
 -   The HPC Environment has been approved for storing and analyzing **Moderate Risk research data**, as defined in the [NYU Electronic Data and System Risk Classification Policy](https://www.nyu.edu/about/policies-guidelines-compliance/policies-and-guidelines/electronic-data-and-system-risk-classification.html). 
@@ -29,11 +26,11 @@ Every individual user has a home directory (under **`/home/$USER`**, environment
 Avoid changing file and directory permissions in your home directory to allow other users to access files.
 :::
 
-User Home Directories are not ideal for sharing files and folders with other users. HPC Scratch or [Research Project Space (RPS)](./05_research_project_space.mdx) are better file-systems for sharing data.
+User Home Directories are not ideal for sharing files and folders with other users. HPC Scratch or [Research Project Space (RPS)](./04_research_project_space.mdx) are better file-systems for sharing data.
 
 :::tip[`inode` limits]
 -   One of the common issues that users report regarding their home directories is running out of inodes (i.e. the number of files stored under their home exceeds the inode limit), which by default is set to 30,000 files
--   To find out the current space and inode quota utilization and the distribution of files under your home directory, please see: [Understanding user quota limits and the myquota command.](./06_best_practices.md#user-quota-limits-and-the-myquota-command) 
+-   To find out the current space and inode quota utilization and the distribution of files under your home directory, please see: [Understanding user quota limits and the myquota command.](./05_best_practices.md#user-quota-limits-and-the-myquota-command) 
 -   Working with `conda` environments: To avoid running out of inode limits in home directories, the HPC team recommends **setting up `conda` environments with Singularity overlay images** as [described here](../07_containers/03_singularity_with_conda.md). Avoid creating `conda` environments in your `$HOME` directory.
 :::
 
@@ -47,15 +44,15 @@ The HPC scratch an all flash (VAST) file-system where most of the users store re
 
 :::tip
 
--   Since there are ***no backups of HPC Scratch file-system***, users should not put important source code, scripts, libraries, executables in `/scratch`. These important files should be stored in file-systems that are backed up, such as `/home` or [Research Project Space (RPS)](./05_research_project_space.mdx). Code can also be stored in a `git` repository.
+-   Since there are ***no backups of HPC Scratch file-system***, users should not put important source code, scripts, libraries, executables in `/scratch`. These important files should be stored in file-systems that are backed up, such as `/home` or [Research Project Space (RPS)](./04_research_project_space.mdx). Code can also be stored in a `git` repository.
 -   ***Old file purging policy on HPC Scratch:*** <ins>All files on the HPC Scratch file-system that have not been accessed ***for more than 60 days*** will be removed. It is a policy violation to use scripts to change the file access time.</ins> Any user found to be violating this policy will have their HPC account locked. A second violation may result in your HPC account being turned off.  
--   To find out the user's current disk space and inode quota utilization and the distribution of files under your scratch directory, please see: [Understanding user quota Limits and the myquota command.](./06_best_practices.md#user-quota-limits-and-the-myquota-command)
+-   To find out the user's current disk space and inode quota utilization and the distribution of files under your scratch directory, please see: [Understanding user quota Limits and the myquota command.](./05_best_practices.md#user-quota-limits-and-the-myquota-command)
 -   Once a research project completes, users should archive their important files in the [HPC Archive file-system](./01_intro_and_data_management.mdx#hpc-archive).
 :::
 
 ### HPC Research Project Space
 The HPC Research Project Space (RPS) provides data storage space for research projects that is easily shared amongst collaborators, ***backed up***, and ***not subject to the old file purging policy***. HPC RPS was introduced to ease data management in the HPC environment and eliminate the need of having to frequently copying files between Scratch and Archive file-systems by having all projects files under one area. ***These benefits of the HPC RPS come at a cost***. The cost is determined by the allocated disk space and the number of files (inodes). 
--   For detailed information about RPS see: [HPC Research Project Space](./05_research_project_space.mdx)
+-   For detailed information about RPS see: [HPC Research Project Space](./04_research_project_space.mdx)
 
 ### HPC Work
 The HPC team makes available a number of public datasets that are commonly used in analysis jobs. The data-sets are available Read-Only under `/scratch/work/public`. For some of the datasets users must provide a signed usage agreement before accessing. Public datasets available on the HPC clusters can be viewed on the [Datasets page](../04_datasets/01_intro.md).
@@ -74,7 +71,7 @@ Google Drive ([NYU Drive](https://www.nyu.edu/life/information-technology/commun
 
 As of December 2023, storage limits were applied to all faculty, staff, and student NYU Google accounts. Please see [Google Workspace Storage](https://www.nyu.edu/life/information-technology/about-nyu-it/key-projects-and-initiatives/google-workspace-storage.html) for details.
 
-There are also limits to the data transfer rate in moving to/from Google Drive. Thus, moving many small files to Google Drive is not going to be efficient. Please read the [Instructions on how to use cloud storage within the NYU HPC Environment](./08_transferring_cloud_storage_data_with_rclone.md). 
+There are also limits to the data transfer rate in moving to/from Google Drive. Thus, moving many small files to Google Drive is not going to be efficient. Please read the [Instructions on how to use cloud storage within the NYU HPC Environment](./07_transferring_cloud_storage_data_with_rclone.md). 
 
 ### HPC Storage Comparison Table
 

docs/hpc/03_storage/02_available_storage_systems.md

@@ -1,14 +1,14 @@
 # Data Transfers
 
 :::tip Globus
-Globus is the recommended tool to use for large-volume data transfers due to the efficiency, reliability, security and ease of use. Use other tools only if you really need to. Detailed instructions available at [Globus](./04_globus.md)
+Globus is the recommended tool to use for large-volume data transfers due to the efficiency, reliability, security and ease of use. Use other tools only if you really need to. Detailed instructions available at [Globus](./03_globus.md)
 :::
 
 ## Data-Transfer nodes
 Attached to the NYU HPC cluster Torch, the Torch Data Transfer Node (gDTN) are  nodes optimized for transferring data between cluster file systems (e.g. scratch)  and other endpoints outside the NYU HPC clusters, including user laptops and desktops. The gDTNs have 100-Gb/s Ethernet connections to the High Speed Research Network (HSRN) and are connected to the HDR Infiniband fabric of the HPC clusters. More information on the hardware characteristics is available at [Torch spec sheet](../10_spec_sheet.md).
 
 ### Data Transfer Node Access
-The HPC cluster filesystems include `/home`, `/scratch`, `/archive` and the [HPC Research Project Space](./05_research_project_space.mdx) are available on the gDTN. The Data-Transfer Node (DTN) can be accessed in a variety of ways
+The HPC cluster filesystems include `/home`, `/scratch`, `/archive` and the [HPC Research Project Space](./04_research_project_space.mdx) are available on the gDTN. The Data-Transfer Node (DTN) can be accessed in a variety of ways
 -   From NYU-net and the High Speed Research Network: use SSH to the DTN hostname `dtn011.hpc.nyu.edu` or `dtn012.hpc.nyu.edu` 
 
 :::info
@@ -42,12 +42,12 @@ where username would be your user name, project1 a directory to be copied to the
 
 ### Windows Tools
 #### File Transfer Clients
-Windows 10 machines may have the Linux Subsystem installed, which will allow for the use of Linux tools, as listed above, but generally it is recommended to use a client such as [WinSCP](https://winscp.net/eng/docs/tunneling) or [FileZilla](https://filezilla-project.org/) to transfer data. Additionally, Windows users may also take advantage of [Globus](./04_globus.md) to transfer files.
+Windows 10 machines may have the Linux Subsystem installed, which will allow for the use of Linux tools, as listed above, but generally it is recommended to use a client such as [WinSCP](https://winscp.net/eng/docs/tunneling) or [FileZilla](https://filezilla-project.org/) to transfer data. Additionally, Windows users may also take advantage of [Globus](./03_globus.md) to transfer files.
 
 ### Globus
 Globus is the recommended tool to use for large-volume data transfers. It features automatic performance tuning and automatic retries in cases of file-transfer failures. Data-transfer tasks can be submitted via a web portal. The Globus service will take care of the rest, to make sure files are copied efficiently, reliably, and securely. Globus is also a tool for you to share data with collaborators, for whom you only need to provide the email addresses.
 
-The Globus endpoint for Torch is available at `nyu#torch`. Detailed instructions available at [Globus](./04_globus.md)
+The Globus endpoint for Torch is available at `nyu#torch`. Detailed instructions available at [Globus](./03_globus.md)
 
 ### rclone
 rclone - rsync for cloud storage, is a command line program to sync files and directories to and from cloud storage systems such as Google Drive, Amazon Drive, S3, B2 etc. rclone is available on DTNs. [Please see the documentation for how to use it.](https://rclone.org/)

docs/hpc/03_storage/03_data_transfers.md docs/hpc/03_storage/02_data_transfers.mddocs/hpc/03_storage/03_data_transfers.md renamed to docs/hpc/03_storage/02_data_transfers.md

@@ -39,7 +39,7 @@ $ for d in $(find $(pwd) -maxdepth 1 -mindepth 1 -type d | sort -u); do n_files=
 ## Large number of small files
 In case your dataset or workflow requires to use large number of small files, this can create a bottleneck due to read/write rates. 
 
-Please refer to [our page on working with a large number of files](./07_large_number_of_small_files.md) to learn about some of the options we recommend to consider.
+Please refer to [our page on working with a large number of files](./06_large_number_of_small_files.md) to learn about some of the options we recommend to consider.
 
 ## Installing Python packages
 :::warning

docs/hpc/03_storage/04_globus.md docs/hpc/03_storage/03_globus.mddocs/hpc/03_storage/04_globus.md renamed to docs/hpc/03_storage/03_globus.md

@@ -1,7 +1,7 @@
 # Transferring Cloud Storage Data with rclone
 
 ## Transferring files to and from Google Drive with RCLONE
-Having access to Google Drive from the HPC environment provides an option to archive data and even share data with collaborators who have no access to the NYU HPC environment. Other options to archiving data include the HPC Archive file system and using [Globus](./04_globus.md) to share data with collaborators.
+Having access to Google Drive from the HPC environment provides an option to archive data and even share data with collaborators who have no access to the NYU HPC environment. Other options to archiving data include the HPC Archive file system and using [Globus](./03_globus.md) to share data with collaborators.
 
 Access to Google Drive is provided by [rclone](https://rclone.org/drive/) - rsync for cloud storage - a command line program to sync files and directories to and from cloud storage systems such as Google Drive, Amazon Drive, S3, B2 etc. [rclone](https://rclone.org/drive/) is available on Torch cluster as a module, the module versions currently available (March 2025) are:
 -   **rclone/1.68.2**
@@ -344,7 +344,7 @@ Please enter 'q' and we're done with configuration.
 
 ### Step 4: Transfer
 :::warning
-Please be sure to perform data transfers on a data transfer node (DTN).  It can degrade performance for other users to perform transfers on other types of nodes.  For more information please see [Data Transfers](./03_data_transfers.md)
+Please be sure to perform data transfers on a data transfer node (DTN).  It can degrade performance for other users to perform transfers on other types of nodes.  For more information please see [Data Transfers](./02_data_transfers.md)
 :::
 
 Sample commands: