torch is now open to all

Author: s-sajid-ali

docs/hpc/02_connecting_to_hpc/01_connecting_to_hpc.md

@@ -4,13 +4,13 @@
 This page gives an overview of connecting to the HPC cluster, for a tutorial on this topic, head to the [this section](../12_tutorial_intro_shell_hpc/02_connecting_to_hpc.mdx)! 
 :::
 
-There are several ways to interact with the Greene HPC cluster. Similar to other Linux clusters, the most common method of connection is via a Command Line Interface (CLI). A CLI is a program that allows you to create and delete files, run programs, and navigate through directories and files via a typed prompt. On Mac, the built-in CLI application is called Terminal. While Windows 11 machines support a Linux Subsystem, which allows for similar functionality, a popular tool used to connect to a Linux server is a free application called [MoabXterm](https://mobaxterm.mobatek.net/documentation.html) 
+There are several ways to interact with the Torch HPC cluster. Similar to other Linux clusters, the most common method of connection is via a Command Line Interface (CLI). A CLI is a program that allows you to create and delete files, run programs, and navigate through directories and files via a typed prompt. On Mac, the built-in CLI application is called Terminal. While Windows 11 machines support a Linux Subsystem, which allows for similar functionality, a popular tool used to connect to a Linux server is a free application called [MoabXterm](https://mobaxterm.mobatek.net/documentation.html) 
 
 :::tip
 If you'd prefer to access the HPC cluster via a web gateway instead, please refer to [the section describing Open OnDemand (OOD)](../09_ood/01_ood_intro.md).
 :::
 
-The following sections will outline basic ways to connect to the Greene cluster. Access to the clusters is primarily handled via the Secure Shell Protocol, or ssh. Below we outline ways to connect via ssh on Mac, Linux, and Windows machines.
+The following sections will outline basic ways to connect to the Torch cluster. Access to the clusters is primarily handled via the Secure Shell Protocol, or ssh. Below we outline ways to connect via ssh on Mac, Linux, and Windows machines.
 
 ## Configuring Your SSH Client
 
@@ -46,7 +46,7 @@ New-Item -Path $HOME\.ssh\config -ItemType File
 # Open config File with Notepad
 C:\WINDOWS\System32\notepad.exe $HOME\.ssh\config
 ```
-Configuring the file above will allow you to use the cmd terminal to connect to Greene.
+Configuring the file above will allow you to use the cmd terminal to connect to Torch.
 
 ### Common issues
 
@@ -60,7 +60,7 @@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
 Someone could be eavesdropping on you right now (man-in-the-middle attack)!
 It is also possible that a host key has just been changed.
 ```
-Do not be alarmed - this is an issue that occurs because the Greene and Torch clusters have multiple login nodes. For instance, `login.torch.hpc.nyu.edu` resolves to one of three possible login nodes (`log-1`, `log-2`, and `log-3`). The configuration directive `StrictHostKeyChecking no` referenced in the recommended config above will prevent this error. Ensure that you have not missed adding it.
+Do not be alarmed - this is an issue that occurs because the Torch and Torch clusters have multiple login nodes. For instance, `login.torch.hpc.nyu.edu` resolves to one of three possible login nodes (`log-1`, `log-2`, and `log-3`). The configuration directive `StrictHostKeyChecking no` referenced in the recommended config above will prevent this error. Ensure that you have not missed adding it.
 :::
 
 :::info SSH Timeout errors
@@ -119,7 +119,7 @@ There is an extra layer of security when accessing Torch which is described belo
 -   Return to terminal and hit Enter
 ![torch-ssh-2fa-9](static/torch-ssh-2fa-9.png)
 
-## SSH Keys for Greene
+## SSH Keys for Torch
 :::warning
 SSH keys are not supported on Torch!
 :::

docs/hpc/03_storage/01_intro_and_data_management.mdx

@@ -39,7 +39,7 @@ Every individual user has a home directory (under **`/home/$USER`**, environment
 
 User home directories are backed up daily and old files under **`$HOME`** are not purged.
 
-The User home directories are available on all HPC clusters (Greene) and on every cluster node (login nodes, compute nodes) as well as and Data Transfer Node (gDTN). 
+The User home directories are available on all HPC clusters (Torch) and on every cluster node (login nodes, compute nodes) as well as and Data Transfer Node (gDTN). 
 
 :::warning
 Avoid changing file and directory permissions in your home directory to allow other users to access files.
@@ -62,7 +62,7 @@ Files stored in the HPC scratch file system are subject to the <ins>**HPC Scratc
 
 Every user has a dedicated scratch directory (**/scratch/$USER**) with **5 TB** disk quota and **1,000,000 inodes** (files) limit per user. 
 
-The scratch file system is available on all nodes (compute, login, etc.) on Greene as well as Data Transfer Node (gDTN).
+The scratch file system is available on all nodes (compute, login, etc.) on Torch as well as Data Transfer Node (gDTN).
 
 :::warning
 There are **No Back ups of the scratch file system.** ***Files that were deleted accidentally or removed due to storage system failures CAN NOT be recovered.***
@@ -83,7 +83,7 @@ Files stored in the HPC vast file system are subject to the <ins>***HPC Vast old
 
 Every user has a dedicated vast directory (**`/vast/$USER`**) with **2 TB** disk quota and **5,000,000 inodes** (files) limit per user. 
 
-The vast file system is available on all nodes (compute, login, etc.) on Greene as well as Data Transfer Node (gDTN).
+The vast file system is available on all nodes (compute, login, etc.) on Torch as well as Data Transfer Node (gDTN).
 
 :::warning
 There are **No Back ups** of the vastsc file system. ***Files that were deleted accidentally or removed due to storage system failures CAN NOT be recovered.***
@@ -108,7 +108,7 @@ For some of the datasets users must provide a signed usage agreement before acce
 Public datasets available on the HPC clusters can be viewed on the [Datasets page](../04_datasets/01_intro.md).
 
 #### HPC Archive
-Once the Analysis stage of the research data lifecycle has completed, <ins>_HPC users should **tar** their data and code into a single tar.gz file and then copy the  file to their archive directory (**`/archive/$USER`**_).</ins> The HPC Archive file system is not accessible by running jobs; it is suitable for long-term data storage. Each user has access to a default disk quota of **2TB** and ***20,000 inode (files) limit***. The rather low limit on the number of inodes per user is intentional.  The archive file system is available only ***on login nodes*** of Greene. The archive file system is backed up daily.
+Once the Analysis stage of the research data lifecycle has completed, <ins>_HPC users should **tar** their data and code into a single tar.gz file and then copy the  file to their archive directory (**`/archive/$USER`**_).</ins> The HPC Archive file system is not accessible by running jobs; it is suitable for long-term data storage. Each user has access to a default disk quota of **2TB** and ***20,000 inode (files) limit***. The rather low limit on the number of inodes per user is intentional.  The archive file system is available only ***on login nodes*** of Torch. The archive file system is backed up daily.
 
 -   Here is an example ***tar*** command that combines the data in a directory named ***my_run_dir*** under ***`$SCRATCH`*** and outputs the tar file in the user's ***`$ARCHIVE`***: 
 ```sh

docs/hpc/03_storage/02_available_storage_systems.md

@@ -20,7 +20,7 @@ The NYU HPC cluster storage runs on Lenovo Distributed Storage Solution DSS-G ha
 -   I/O Performance: up to 650k input/output operations per second (IOPS)
 
 ## Flash Tier Storage (VAST)
-An all flash file system, using [VAST Flash storage](https://www.vastdata.com/), is now available on Greene. Flash storage is optimal for computational workloads with high I/O rates. For example, If you have jobs to run with huge amount of tiny files, VAST may be a good candidate.  If you and your lab members are interested, please reach out to [hpc@nyu.edu](mailto:hpc@nyu.edu) for more information.
+An all flash file system, using [VAST Flash storage](https://www.vastdata.com/), is now available on Torch. Flash storage is optimal for computational workloads with high I/O rates. For example, If you have jobs to run with huge amount of tiny files, VAST may be a good candidate.  If you and your lab members are interested, please reach out to [hpc@nyu.edu](mailto:hpc@nyu.edu) for more information.
 -   NVMe interface
 -   Total size: 778 TB
 :::note
@@ -29,7 +29,7 @@ An all flash file system, using [VAST Flash storage](https://www.vastdata.com/),
 
 ## Research Project Space (RPS)
 [Research Project Space (RPS)](./05_research_project_space.mdx) volumes provide working spaces for sharing data and code amongst project or lab members.
--   RPS directories are available on the Greene HPC cluster.
+-   RPS directories are available on the Torch HPC cluster.
 -   There is no old-file purging policy on RPS.
 -   RPS is backed up. 
 -   There is a cost per TB per year and inodes per year for RPS volumes.

docs/hpc/03_storage/03_data_transfers.md

@@ -5,14 +5,14 @@ Globus is the recommended tool to use for large-volume data transfers due to the
 :::
 
 ## Data-Transfer nodes
-Attached to the NYU HPC cluster Greene, the Greene Data Transfer Node (gDTN) are  nodes optimized for transferring data between cluster file systems (e.g. scratch)  and other endpoints outside the NYU HPC clusters, including user laptops and desktops. The gDTNs have 100-Gb/s Ethernet connections to the High Speed Research Network (HSRN) and are connected to the HDR Infiniband fabric of the HPC clusters. More information on the hardware characteristics is available at [Greene spec sheet](../10_spec_sheet.md).
+Attached to the NYU HPC cluster Torch, the Torch Data Transfer Node (gDTN) are  nodes optimized for transferring data between cluster file systems (e.g. scratch)  and other endpoints outside the NYU HPC clusters, including user laptops and desktops. The gDTNs have 100-Gb/s Ethernet connections to the High Speed Research Network (HSRN) and are connected to the HDR Infiniband fabric of the HPC clusters. More information on the hardware characteristics is available at [Torch spec sheet](../10_spec_sheet.md).
 
 ### Data Transfer Node Access
 The HPC cluster filesystems include `/home`, `/scratch`, `/archive` and the [HPC Research Project Space](./05_research_project_space.mdx) are available on the gDTN. The Data-Transfer Node (DTN) can be accessed in a variety of ways
 -   From NYU-net and the High Speed Research Network: use SSH to the DTN hostname `gdtn.hpc.nyu.edu`
--   From the Greene cluster (e.g., the login nodes): the hostname can be shortened to `gdtn`
+-   From the Torch cluster (e.g., the login nodes): the hostname can be shortened to `gdtn`
 :::info
-For example, to log in to a DTN from the Greene cluster, to carry out some copy operation, and to log back out, you can use a command sequence like:
+For example, to log in to a DTN from the Torch cluster, to carry out some copy operation, and to log back out, you can use a command sequence like:
 ```sh
 ssh gdtn
 rsync ...
@@ -48,7 +48,7 @@ Windows 10 machines may have the Linux Subsystem installed, which will allow for
 ### Globus
 Globus is the recommended tool to use for large-volume data transfers. It features automatic performance tuning and automatic retries in cases of file-transfer failures. Data-transfer tasks can be submitted via a web portal. The Globus service will take care of the rest, to make sure files are copied efficiently, reliably, and securely. Globus is also a tool for you to share data with collaborators, for whom you only need to provide the email addresses.
 
-The Globus endpoint for Greene is available at `nyu#torch`. The endpoint `nyu#prince` has been retired. Detailed instructions available at [Globus](./04_globus.md)
+The Globus endpoint for Torch is available at `nyu#torch`. The endpoint `nyu#prince` has been retired. Detailed instructions available at [Globus](./04_globus.md)
 
 ### rclone
 rclone - rsync for cloud storage, is a command line program to sync files and directories to and from cloud storage systems such as Google Drive, Amazon Drive, S3, B2 etc. rclone is available on DTNs. [Please see the documentation for how to use it.](https://rclone.org/)

docs/hpc/03_storage/04_globus.md

@@ -55,11 +55,11 @@ More information about **Globus Connect Personal** and download links for Linux,
 ![Globus Connect Personal](./static/globus_connect_personal.png)
 
 ### Transfer files between your Personal Endpoint and NYU nyu#torch
-To transfer files you need to specify two collections (endpoints). Specify one of them as **Greene scratch directory**, or **Greene archive directory** or **Greene home directory**. The other endpoint is the one created for your personal computer (e.g. My Mac Laptop) if it is involved in the transfer. When you first use the Greene directory collection, authentication/consent is required for the Globus web app to manage collections on this endpoint on your behalf.
+To transfer files you need to specify two collections (endpoints). Specify one of them as **Torch scratch directory**, or **Torch archive directory** or **Torch home directory**. The other endpoint is the one created for your personal computer (e.g. My Mac Laptop) if it is involved in the transfer. When you first use the Torch directory collection, authentication/consent is required for the Globus web app to manage collections on this endpoint on your behalf.
 
 ![Globus Start Transfer](./static/globus_start_transfer.png)
 
-When writing to your Greene archive directory, please pay attention that there is a default inode limit of 20K per user.  
+When writing to your Torch archive directory, please pay attention that there is a default inode limit of 20K per user.  
 
 When the second Endpoint is chosen to be your personal computer, your computer home directory content will show up. Now select directory and files (you may select multiple files when clicking on file names while pressing down "shift" key), click one of the two blue Start buttons to indicate the transfer direction. After clicking the blue Start button, you should see a message indicating a transfer request has been submitted successfully, and a transfer ID is generated. Globus file transfer service takes care of the actual copying.
 

docs/hpc/03_storage/06_best_practices.md

@@ -48,4 +48,4 @@ Your home directory has a relatively small number of inodes.
 If you create a conda or python environment in you home directory, this can eat up all the inodes. 
 :::
 
-Please review the [Package Management section](../06_tools_and_software/01_intro.md#package-management-for-r-python--julia-and-conda-in-general) of the [Greene Software Page](../06_tools_and_software/01_intro.md).
+Please review the [Package Management section](../06_tools_and_software/01_intro.md#package-management-for-r-python--julia-and-conda-in-general) of the [Torch Software Page](../06_tools_and_software/01_intro.md).

docs/hpc/03_storage/07_large_number_of_small_files.md

@@ -75,7 +75,7 @@ Essentially, this is a large file sitting on the disk that contains a lot of sma
 
 This is a memory-mapped database meaning, file can be larger than RAM. OS is responsible for managing the pages (like caching frequently uses pages).
 
-For practical use it means: say you have 10 GB of RAM, and LMDB file of 100 GB. When you connect to this file, OS may decide to load 5GB to RAM, and the rest 95GB will be attached as virtual memory. Greene does not have a limit for virtual memory. Of course, if your RAM is larger than LMDB file, this database will perform the best, as OS will have enough resources to keep what is needed directly in RAM.
+For practical use it means: say you have 10 GB of RAM, and LMDB file of 100 GB. When you connect to this file, OS may decide to load 5GB to RAM, and the rest 95GB will be attached as virtual memory. Torch does not have a limit for virtual memory. Of course, if your RAM is larger than LMDB file, this database will perform the best, as OS will have enough resources to keep what is needed directly in RAM.
 
 :::tip
 when you write key-value pairs to LMDB they have to be byte-encoded. For example, if you use Python you can use: for string `st.encode()`, for np.array use `ar.tobytes()`, or in general `pickle.dumps()`

docs/hpc/03_storage/08_transferring_cloud_storage_data_with_rclone.md

@@ -3,7 +3,7 @@
 ## Transferring files to and from Google Drive with RCLONE
 Having access to Google Drive from the HPC environment provides an option to archive data and even share data with collaborators who have no access to the NYU HPC environment. Other options to archiving data include the HPC Archive file system and using [Globus](./04_globus.md) to share data with collaborators.
 
-Access to Google Drive is provided by [rclone](https://rclone.org/drive/) - rsync for cloud storage - a command line program to sync files and directories to and from cloud storage systems such as Google Drive, Amazon Drive, S3, B2 etc. [rclone](https://rclone.org/drive/) is available on Greene cluster as a module, the module versions currently available (March 2025) are:
+Access to Google Drive is provided by [rclone](https://rclone.org/drive/) - rsync for cloud storage - a command line program to sync files and directories to and from cloud storage systems such as Google Drive, Amazon Drive, S3, B2 etc. [rclone](https://rclone.org/drive/) is available on Torch cluster as a module, the module versions currently available (March 2025) are:
 -   **rclone/1.68.2**
 -   **rclone/1.66.0**
 -   **rclone/1.60.1**
@@ -25,11 +25,11 @@ This option works great for file sizes 1Gb+ to 250GB. Keep in mind that there is
 ## rclone Configuration
 You need to configure rclone before you will be able to move files between the HPC Environment and Google Drive
 
-There are specific instruction on the [rclone web site](https://rclone.org/drive/) and here is an example of the process for configuring rclone for use on Greene:
+There are specific instruction on the [rclone web site](https://rclone.org/drive/) and here is an example of the process for configuring rclone for use on Torch:
 
-### Step 1: Login to Greene:
+### Step 1: Login to Torch:
 
-Follow [instructions](../02_connecting_to_hpc/01_connecting_to_hpc.md) to log into the Greene HPC cluster.
+Follow [instructions](../02_connecting_to_hpc/01_connecting_to_hpc.md) to log into the Torch HPC cluster.
 
 ### Step 2: Load the rclone module
 ```sh

docs/hpc/04_datasets/01_intro.md

@@ -92,7 +92,7 @@ Please open the ImageNet site, find the terms of use ([http://image-net.org/down
 
 NYU has a subscription to Twitter Decahose - 10% random sample of the realtime Twitter Firehose through a streaming connection
 
-*Datasets are stored* in GCP cloud (BigQuery) and on the HPC cluster Greene.
+*Datasets are stored* in GCP cloud (BigQuery) and on the HPC cluster Torch.
 
 Please contact Megan Brown at [The Center for Social Media & Politics](https://csmapnyu.org/) to get access to data and learn the tools available to work with it.
 

docs/hpc/06_tools_and_software/01_intro.md

@@ -11,7 +11,7 @@ We strongly advise that you setup your own computational enviromments via Apptai
 -   [Using virtual environments for Python](../06_tools_and_software/04_python_packages_with_virtual_environments.mdx)
 -   [Managing R packages with renv](../06_tools_and_software/05_r_packages_with_renv.mdx)
 
-## Examples of software usage on Greene 
+## Examples of software usage on Torch 
 Examples can be found under `/scratch/work/public/examples/` and include the following
 
 |                   |                       |                       |