diff --git a/docs/hpc/01_getting_started/01_intro.md b/docs/hpc/01_getting_started/01_intro.md deleted file mode 100644 index 6440f3c4d6..0000000000 --- a/docs/hpc/01_getting_started/01_intro.md +++ /dev/null @@ -1,5 +0,0 @@ -# Start here! - -Welcome to the Torch HPC documentation! If you do not have an HPC account, please proceed to the next section that explains how you may be able to get one. - -If you are an active user, you can proceed to one of the categories on the left. diff --git a/docs/hpc/01_getting_started/01_intro.mdx b/docs/hpc/01_getting_started/01_intro.mdx new file mode 100644 index 0000000000..d8eae6d5b1 --- /dev/null +++ b/docs/hpc/01_getting_started/01_intro.mdx @@ -0,0 +1,9 @@ +# Start here! + +Welcome to the Torch HPC documentation! If you do not have an HPC account, please proceed to the next section that explains how you may be able to get one. + +If you are an active user, you can proceed to one of the categories on the left. + +## Announcements + + diff --git a/docs/hpc/03_storage/01_intro_and_data_management.mdx b/docs/hpc/03_storage/01_intro_and_data_management.mdx index 9b6e8a7923..a8e278f1e9 100644 --- a/docs/hpc/03_storage/01_intro_and_data_management.mdx +++ b/docs/hpc/03_storage/01_intro_and_data_management.mdx @@ -105,7 +105,7 @@ The HPC team makes available a number of public datasets that are commonly used For some of the datasets users must provide a signed usage agreement before accessing. -Public datasets available on the HPC clusters can be viewed on the [Datasets page](../01_getting_started/01_intro.md). +Public datasets available on the HPC clusters can be viewed on the [Datasets page](../04_datasets/01_intro.md). #### HPC Archive Once the Analysis stage of the research data lifecycle has completed, _HPC users should **tar** their data and code into a single tar.gz file and then copy the file to their archive directory (**`/archive/$USER`**_). The HPC Archive file system is not accessible by running jobs; it is suitable for long-term data storage. Each user has access to a default disk quota of **2TB** and ***20,000 inode (files) limit***. The rather low limit on the number of inodes per user is intentional. The archive file system is available only ***on login nodes*** of Greene. The archive file system is backed up daily. diff --git a/docs/hpc/03_storage/03_data_transfers.md b/docs/hpc/03_storage/03_data_transfers.md index 66cf21ebc7..622e794c6a 100644 --- a/docs/hpc/03_storage/03_data_transfers.md +++ b/docs/hpc/03_storage/03_data_transfers.md @@ -5,7 +5,7 @@ Globus is the recommended tool to use for large-volume data transfers due to the ::: ## Data-Transfer nodes -Attached to the NYU HPC cluster Greene, the Greene Data Transfer Node (gDTN) are nodes optimized for transferring data between cluster file systems (e.g. scratch) and other endpoints outside the NYU HPC clusters, including user laptops and desktops. The gDTNs have 100-Gb/s Ethernet connections to the High Speed Research Network (HSRN) and are connected to the HDR Infiniband fabric of the HPC clusters. More information on the hardware characteristics is available at [Greene spec sheet](../10_spec_sheet.mdx). +Attached to the NYU HPC cluster Greene, the Greene Data Transfer Node (gDTN) are nodes optimized for transferring data between cluster file systems (e.g. scratch) and other endpoints outside the NYU HPC clusters, including user laptops and desktops. The gDTNs have 100-Gb/s Ethernet connections to the High Speed Research Network (HSRN) and are connected to the HDR Infiniband fabric of the HPC clusters. More information on the hardware characteristics is available at [Greene spec sheet](../10_spec_sheet.md). ### Data Transfer Node Access The HPC cluster filesystems include `/home`, `/scratch`, `/archive` and the [HPC Research Project Space](./05_research_project_space.mdx) are available on the gDTN. The Data-Transfer Node (DTN) can be accessed in a variety of ways diff --git a/docs/hpc/03_storage/08_transferring_cloud_storage_data_with_rclone.md b/docs/hpc/03_storage/08_transferring_cloud_storage_data_with_rclone.md index 74ba87801b..e3758c0066 100644 --- a/docs/hpc/03_storage/08_transferring_cloud_storage_data_with_rclone.md +++ b/docs/hpc/03_storage/08_transferring_cloud_storage_data_with_rclone.md @@ -344,7 +344,7 @@ Please enter 'q' and we're done with configuration. ### Step 4: Transfer :::warning -Please be sure to perform data transters on a data transfer node (DTN). It can degrade performance for other users to perform transfers on other types of nodes. For more information please see [Data Transfers](./03_data_transfers.md) +Please be sure to perform data transfers on a data transfer node (DTN). It can degrade performance for other users to perform transfers on other types of nodes. For more information please see [Data Transfers](./03_data_transfers.md) ::: Sample commands: diff --git a/docs/hpc/05_submitting_jobs/03_slurm_tutorial.md b/docs/hpc/05_submitting_jobs/03_slurm_tutorial.md index a8bb1a89c1..4c79123fc8 100644 --- a/docs/hpc/05_submitting_jobs/03_slurm_tutorial.md +++ b/docs/hpc/05_submitting_jobs/03_slurm_tutorial.md @@ -16,7 +16,7 @@ The Slurm software system is a resource manager and a job scheduler, which is de - It also assumes you are comfortable with Linux command-line environment. To learn about linux please read our [Linux Tutorial](../12_tutorial_intro_shell_hpc/01_intro.mdx). -- Please review the [Hardware Specs page](../10_spec_sheet.mdx) for more information on Greene's hardware specifications. +- Please review the [Hardware Specs page](../10_spec_sheet.md) for more information on Greene's hardware specifications. ## Slurm Commands diff --git a/docs/hpc/06_tools_and_software/01_intro.md b/docs/hpc/06_tools_and_software/01_intro.md index d430e7a831..28ba8028be 100644 --- a/docs/hpc/06_tools_and_software/01_intro.md +++ b/docs/hpc/06_tools_and_software/01_intro.md @@ -3,7 +3,7 @@ We encourage you to setup your own computational environment on Torch and to assist you in doing so, we allow you to run [Apptainer](../07_containers/01_intro.md) (formerly known as Singularity) containers, we manage licensed software suites and offer extensive documentation, training and support. :::tip -We stongly advise that you setup your own computational enviromments via Apptainer containers and overlay files. Detailed documentation is available in the [containers section](../07_containers/01_intro.md). +We strongly advise that you setup your own computational enviromments via Apptainer containers and overlay files. Detailed documentation is available in the [containers section](../07_containers/01_intro.md). ::: ## Package Management for R, Python, & Julia, and Conda in general diff --git a/docs/hpc/10_spec_sheet.md b/docs/hpc/10_spec_sheet.md new file mode 100644 index 0000000000..550c67c51a --- /dev/null +++ b/docs/hpc/10_spec_sheet.md @@ -0,0 +1,21 @@ + +# Torch Spec Sheet + +The Torch cluster has 518 [Intel "Xeon Platinum 8592+ 64C"](https://www.intel.com/content/www/us/en/products/sku/237261/intel-xeon-platinum-8592-processor-320m-cache-1-90-ghz/specifications.html) CPUs, 29 NVIDIA [H200](https://nvdam.widen.net/s/nb5zzzsjdf/hpc-datasheet-sc23-h200-datasheet-3002446) GPUs & 68 NVIDIA [L40S](https://resources.nvidia.com/en-us-l40s/l40s-datasheet-28413) GPUs connected together via Infiniband NDR400 interconnect. Further details on each kind of node is provided in the table below. + +| Type | Nodes | CPU Cores | GPUs | Memory (GB) | +|---|---|---|---|---| +| Standard Memory | 186 | 23,808 | N/A | 95,232 | +| Large memory | 7 | 896 | N/A | 21,504 | +| H200 GPU | 29 | 3,712 | 232 | 59,392 | +| L40S GPU | 68 | 8,704 | 272 | 34,816 | +| Login | 4 | 512 | N/A | 1024 | +| Data Transfer | 2 | 64 | N/A | 512 | +| Provisioning | 4 | 320 | N/A | 1024 | +| Scheduler | 2 | 64 | N/A | 1024 | +| Total | N/A | 38,080 | 504 | 209.5(TB) | + + +Torch was tested in June 2025 using the [LINPACK benchmark system](https://top500.org/project/linpack/), which is the basis for all HPC systems ranked on the Top500 list. It had a theoretical maximum performance of 12.25 PF/s thanks to its powerful GPU resources, of which LINPACK was able to use 10.79 PF/s, thus placing it at [#133 on the listed](https://top500.org/system/180363/). + + Torch was recently ranked [#40 on the Green 500 list](https://top500.org/lists/green500/list/2025/06/), a global list of the most energy efficient supercomputers in the world thanks to its advanced liquid cooling system. diff --git a/docs/hpc/10_spec_sheet.mdx b/docs/hpc/10_spec_sheet.mdx deleted file mode 100644 index d6274bc0e0..0000000000 --- a/docs/hpc/10_spec_sheet.mdx +++ /dev/null @@ -1,76 +0,0 @@ - -# Greene Spec Sheet - -[vast home page]: https://www.vastdata.com/ - -## Hardware Specs - -Please find Greene's hardware specification in detail at the [google sheets here](https://docs.google.com/spreadsheets/d/1czgPi6x8Qa5PNRIX_VLSt7mDpe8zgg5MRqdVmINJzxc?rm=minimal): - -:::tip -Hover a mouse over a cell with a black triangle to see more details_ -::: - - - -## Mounted Storage Systems - -Please find the details on Greene's available storage offerings at the [google sheets here](https://docs.google.com/spreadsheets/d/1pYZ0YtN1fhMN7kxcGcm6U-HZxMKLRBXXr2BwemxeS7Y?rm=minimal): - - - - -## General Parallel File System (GPFS) - -The NYU HPC Clusters are served by a `General Parallel File System (GPFS)` storage cluster. GPFS is a high-performance clustered file system software developed by IBM that provides concurrent high-speed file access to applications executing on multiple nodes of clusters. - -The cluster storage runs on Lenovo Distributed Storage Solution DSS-G hardware: - - -- **_2x DSS-G 202_** - - 116 Solid State Drives (SSDs) - - 464 TB raw storage -- **_2x DSS-G 240_** - - 668 Hard Disk Drives (HDDs) - - 9.1 PB raw storage - -### GPFS Performance - -| | | -| --- | --- | -| Read Bandwidth | **_78 GB_** per second reads | -| Write Bandwidth | **_42 GB_** per second writes | -| I/O Performance | **_~650k_** Input/Output operations per second (IOPS) | - -## Flash Tier Storage (VAST) - -An all flash file system, using [VAST Flash storage][vast home page] is now available on Greene. Flash storage is optimal for computational workloads with high I/O rates. For example, if you have jobs to run with huge number of _tiny files_, VAST may be a good candidate. - -Please contact the team hpc@nyu.edu for more information. - -- NVMe Interface -- _778 TB_ Total Storage -- Available to **all** users as **read only** -- **Write** access available to **approved** users only - -## Research Project Space (RPS) - -- Research Project Space (RPS) volumes provide working spaces for sharing data and work amongst project or lab members for long term research needs. -- RPS directories are available on the Greene HPC cluster. -- RPS is backed up. There is no file purging policy on RPS. -- There is a _cost per TB per year_, and _inodes per year_ for RPS volumes. - -Please find more information at \[Research Project Space page]. - - -## Data Transfer Nodes (gDTN) - -| | | -| --- | --- | -| Node Type | Lenovo SR630 | -| Number of Nodes | 2 | -| CPUs | 2x Intel Xeon Gold 6244 8C 150W 3.6 GHz Processor. -| Memory | 192 GB (total) - 12x 16GB DDR4, 2933 MHz -| Local Disk | 1x 1.92 TB SSD -| Infiniband Interconnect | 1x Mellanox ConnectX-6 HDR100/100GbE VPI 1-Port x16 PCIe 3.0 HCA -| Ethernet Connectivity to the NYU High-Speed Research Network (HSRN) | 200 Gbit - 1x Mellanox ConnectX-5 EDR IB/100GbE VPI Dual-Port x16 PCIe 3.0 HCA diff --git a/docs/srde/02_faq/01_basics.mdx b/docs/srde/02_faq/01_basics.mdx index 60bd052ea0..157a821da2 100644 --- a/docs/srde/02_faq/01_basics.mdx +++ b/docs/srde/02_faq/01_basics.mdx @@ -25,7 +25,7 @@ Fill out our intake form to provide more information about your project: [Secure The cost is dependent on the needs of the project, such as size of the data and the type of machine needed for the analysis. Google Cloud has a calculator that will help estimate the costs: https://cloud.google.com/products/calculator ## My data is not high risk, are there other options available? -There are several options available depending on the data risk classification and the needs of the project. There are resources provided by the university such as research project space, cloud computing, etc. You can check out many of these services on the [HPC Support Site](../../hpc/01_getting_started/01_intro.md). If you are unsure on how to proceed, a consultation with the SRDE team will help determine the best path forward. +There are several options available depending on the data risk classification and the needs of the project. There are resources provided by the university such as research project space, cloud computing, etc. You can check out many of these services on the [HPC Support Site](../../hpc/01_getting_started/01_intro.mdx). If you are unsure on how to proceed, a consultation with the SRDE team will help determine the best path forward. ## What does the SRDE team need from me for the consultation? To help get things started it would be beneficial to submit an [intake form](https://nyu.qualtrics.com/jfe/form/SV_3Vok9ax87Bxxdsy) with any related data governance documentation (files can be attached to the form), including, but not limited to, data use agreement, OSP/IRB documents, and project information.