From 8ee3fce12eb928e691ed80bb36446e6b56daa62d Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Mon, 6 Jul 2026 22:53:29 +0800 Subject: [PATCH 01/84] Fix typo in title from 'Publciations' to 'Publications' --- .../03_Slurm_Accounts/06_adding_publications_grants.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hpc/01_getting_started/03_Slurm_Accounts/06_adding_publications_grants.mdx b/docs/hpc/01_getting_started/03_Slurm_Accounts/06_adding_publications_grants.mdx index 5383efaed4..0bb0406f9e 100644 --- a/docs/hpc/01_getting_started/03_Slurm_Accounts/06_adding_publications_grants.mdx +++ b/docs/hpc/01_getting_started/03_Slurm_Accounts/06_adding_publications_grants.mdx @@ -1,4 +1,4 @@ -# Entering Grants and Publciations for Your Project +# Entering Grants and Publications for Your Project ## Acknowledgment Statement for Publications The following acknowledgment statement should appear in the publication of any material that resulted from using the NYU IT HPC resources, services, and staff expertise. From 2e7b1b6234055deedb9037dfe2b3f1504ed6f015 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Mon, 6 Jul 2026 22:55:13 +0800 Subject: [PATCH 02/84] Fix formatting in run_hf_model.md --- docs/hpc/08_ml_ai_hpc/08_LLM inference/02_run_hf_model.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hpc/08_ml_ai_hpc/08_LLM inference/02_run_hf_model.md b/docs/hpc/08_ml_ai_hpc/08_LLM inference/02_run_hf_model.md index 01cc653b0b..c71962af86 100644 --- a/docs/hpc/08_ml_ai_hpc/08_LLM inference/02_run_hf_model.md +++ b/docs/hpc/08_ml_ai_hpc/08_LLM inference/02_run_hf_model.md @@ -135,7 +135,7 @@ Create a python script using the following code from sections 1-9 and save it in ``` ## Prepare Sbatch file -After saving the above code in a script called `huggingface.py`, create a file called `run.SBATCH` with the the following code: +After saving the above code in a script called `huggingface.py`, create a file called `run.SBATCH` with the following code: ```batch #!/bin/bash From be95f45807f75aea3069d2ee667a006d37d4f03c Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Mon, 6 Jul 2026 22:57:18 +0800 Subject: [PATCH 03/84] Update 10_spec_sheet.md --- docs/hpc/10_spec_sheet.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hpc/10_spec_sheet.md b/docs/hpc/10_spec_sheet.md index eee053afaf..536566d4df 100644 --- a/docs/hpc/10_spec_sheet.md +++ b/docs/hpc/10_spec_sheet.md @@ -26,7 +26,7 @@ Torch was tested in June 2025 using the [LINPACK benchmark system](https://top50 Torch was recently ranked [#40 on the Green 500 list](https://top500.org/lists/green500/list/2025/06/), a global list of the most energy efficient supercomputers in the world thanks to its advanced liquid cooling system. ## Torch Stakeholder Resources -Many GPUs on Torch are owned by stakeholder groups who have prioirty access to the resources. The breakdown of these GPUs into stakeholder groups is as follows: +Many GPUs on Torch are owned by stakeholder groups who have priority access to the resources. The breakdown of these GPUs into stakeholder groups is as follows: | Stakeholders | H200 | L40S | H100 | A100 | RTX Pro 6000 | Total | | :--- | :---: | :---: | :---: | :---: | :---: | :---: | From a347b698fbe5ba2e21e95b04f6690270deda9025 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Mon, 6 Jul 2026 23:15:47 +0800 Subject: [PATCH 04/84] Update storage-finder-data.mdx --- src/pages/storage-finder-data.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/pages/storage-finder-data.mdx b/src/pages/storage-finder-data.mdx index 356a09053c..df39f971e4 100644 --- a/src/pages/storage-finder-data.mdx +++ b/src/pages/storage-finder-data.mdx @@ -478,7 +478,7 @@ During pilot phase, projects must be approved by RTS staff ### Permission Settings -Project dependent, role-based permissions. NeiID users added by PI request +Project dependent, role-based permissions. NetID users added by PI request ### Eligibility From 31fd9ca8017f7db2f48749238195629509a7a51c Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Mon, 6 Jul 2026 23:16:43 +0800 Subject: [PATCH 05/84] Fix typo in permission settings section --- src/pages/storage-finder-data.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/pages/storage-finder-data.mdx b/src/pages/storage-finder-data.mdx index df39f971e4..24d61b3b19 100644 --- a/src/pages/storage-finder-data.mdx +++ b/src/pages/storage-finder-data.mdx @@ -516,7 +516,7 @@ VPN needed when off-campus. ### Permission Settings -Done via Grouper; can give collaborators read access, read/write access, or administrator priveleges for a share. +Done via Grouper; can give collaborators read access, read/write access, or administrator privileges for a share. ### Eligibility From eb79b2e06ee27196365e232b3ec2620053522a54 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Mon, 6 Jul 2026 23:18:00 +0800 Subject: [PATCH 06/84] Fix typo in onboarding documentation --- docs/genai/02_onboarding/01_intro.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/genai/02_onboarding/01_intro.mdx b/docs/genai/02_onboarding/01_intro.mdx index d34d8738e0..39a0f22090 100644 --- a/docs/genai/02_onboarding/01_intro.mdx +++ b/docs/genai/02_onboarding/01_intro.mdx @@ -5,7 +5,7 @@ This section deals with the eligibility for getting access to Pythia. To learn more about affiliate access (students after graduation, collaborators from other institutions, please refer to [this section](../../hpc/01_getting_started/02_HPC_Accounts/01_getting_and_renewing_an_account.mdx)) :::tip[workspaces] -Access to Pythia is facilitated via workspaces. Beyond facilitating access, they also allow team members to collaboratively dvelop prompts. +Access to Pythia is facilitated via workspaces. Beyond facilitating access, they also allow team members to collaboratively develop prompts. ::: From 6dfd18cd40a8b539dfb6b5a55e9f9e327828d40f Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Mon, 6 Jul 2026 23:20:49 +0800 Subject: [PATCH 07/84] Update 01_temperature.md --- docs/genai/04_how_to_guides/01_temperature.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/genai/04_how_to_guides/01_temperature.md b/docs/genai/04_how_to_guides/01_temperature.md index b92cf211a5..da68f6a13e 100644 --- a/docs/genai/04_how_to_guides/01_temperature.md +++ b/docs/genai/04_how_to_guides/01_temperature.md @@ -34,7 +34,7 @@ completion = portkey.chat.completions.create( print(completion) ``` -At the temperature of 2.0, you might get an output along along the lines of: +At the temperature of 2.0, you might get an output along the lines of: - ``"listening to old radio static."`` - ``"... a really peculiar shade of chartreuse today."`` From c45ab1ea84fb03e9eead51bbcac316aa2fc0a7d0 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Tue, 14 Jul 2026 14:56:03 +0800 Subject: [PATCH 08/84] Update 06_large_number_of_small_files.md --- .../hpc/03_storage/06_large_number_of_small_files.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/hpc/03_storage/06_large_number_of_small_files.md b/docs/hpc/03_storage/06_large_number_of_small_files.md index 82729baf16..13efb97309 100644 --- a/docs/hpc/03_storage/06_large_number_of_small_files.md +++ b/docs/hpc/03_storage/06_large_number_of_small_files.md @@ -5,10 +5,10 @@ Many datasets contain a large number of files (for example [ImageNet](https://en Here are some ideas you can try and evaluate performance for your own project -## Squash file system with Singularity +## Squash File System With Singularity Please see [Squash File System and Singularity](../07_containers/04_squash_file_system_and_singularity.md) -## Use jpg/png files on disk +## Use JPG/PNG Files On Disk One option is to store image files (like png or jpg) on the disk and read from disk directly. :::warning @@ -36,12 +36,12 @@ You can also use the custom-made RAM mapped disk using `#SLURM_RAM_TMPDIR` while This basically requires you to use 2*(size of the data) size of RAM just to hold the data. ::: -## Binary files (pickle, etc) +## Binary Files (Pickle, Etc.) Store data in some binary file (say pickle in Python) which you load fully when you start a SLURM job. This option may require a lot of RAM - thus you may have to wait a long time for the scheduler to find resources for your job. Also this approach would not work on a regular PC without so much RAM, and thus your scripts are not transferable. -## Container files, one-file databases +## Container Files, One-File Databases Special containers, which allow to either load data fast fully or access chosen elements without loading the whole dataset into RAM. ### SQLite @@ -96,10 +96,10 @@ More info: LMDB supports reading by many readers and many parallel thread from the same file -#### Formats inside HDF5/LMDB: binary, numpy, other.. +#### Formats Inside HDF5/LMDB: Binary, NumPy, Other... One can store data in different way inside LMDB or HDF5. For example we can store binary representation of jpeg, or we can store python numpy array. In the first case file can be read from any language, in the second - only from Python. We can also store objects from other languages - for example tibble in R -#### Other formats +#### Other Formats There are other formats like [Bcolz](http://bcolz.blosc.org/), [Zarr](https://github.com/alimanfoo/zarr-python), and others. Some examples can be found [here](https://alimanfoo.github.io/2016/04/14/to-hdf5-and-beyond.html). ## Benchmarking Code From 204c2d793722c1bf45148f52406613304e0bfd7f Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Tue, 14 Jul 2026 14:59:43 +0800 Subject: [PATCH 09/84] Update 03_globus.md --- docs/hpc/03_storage/03_globus.md | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/docs/hpc/03_storage/03_globus.md b/docs/hpc/03_storage/03_globus.md index 033b0b7304..64edb018f7 100644 --- a/docs/hpc/03_storage/03_globus.md +++ b/docs/hpc/03_storage/03_globus.md @@ -2,7 +2,7 @@ The Globus project aims at providing powerful tools for scientific data management, to help researchers to focus on their domain subjects and solve data intensive research problems. Globus has been grown maturely to enable grid computing by connecting computing resources distributed globally across organizational boundary. Universities, national laboratories and computing facilities are using services of Globus. -## Transferring data between endpoints +## Transferring Data Between Endpoints ### Endpoint A globus **Endpoint** is a data transfer location, a location where data can be moved to or from using Globus transfer, sync and sharing service. An endpoint can either be a ***personal endpoint*** (on a user’s personal computer) or a ***server endpoint*** (located on a server, for use by multiple users). Please see [Data Transfer With Globus](https://www.globus.org/data-transfer) for details. @@ -40,12 +40,11 @@ The NYU HPC Server Endpoint: nyu#torch **Globus Connect Server** is already installed on the NYU HPC cluster creating a ***Server Endpoint*** named **nyu#torch**, that is available to authorized users (users with a valid HPC account) using Globus. If you want to move data to or from your computer and the NYU HPC cluster, you need to install **[Globus Connect Personal](https://www.globus.org/globus-connect-personal)** on your computer, thus creating a ***Personal Endpoint*** on your computer. -### Moving data between Server Endpoints +### Moving Data Between Server Endpoints If you plan to transfer data between ***Server Endpoints***, such as between the NYU server endpoint **nyu#torch** and a server endpoint at another institution, you do not need to install Globus Connect Personal on your computer. -#### Creating a Personal Endpoint on your computer - +#### Creating a Personal Endpoint On Your Computer This needs to be done only once on your personal computer. After clicking "Transfer or Sync to...", click "Search" on the upper right side. Then follow the link "Install Globus Connect Personal". @@ -54,7 +53,7 @@ More information about **Globus Connect Personal** and download links for Linux, ![Globus Connect Personal](./static/globus_connect_personal.png) -### Transfer files between your Personal Endpoint and NYU nyu#torch +### Transfer Files between Your Personal Endpoint and NYU nyu#torch To transfer files you need to specify two collections (endpoints). Specify one of them as **Torch scratch directory**, or **Torch archive directory** or **Torch home directory**. The other endpoint is the one created for your personal computer (e.g. My Mac Laptop) if it is involved in the transfer. When you first use the Torch directory collection, authentication/consent is required for the Globus web app to manage collections on this endpoint on your behalf. ![Globus Start Transfer](./static/globus_start_transfer.png) @@ -67,7 +66,7 @@ When the transfer is done, you should receive an email notification. Click "ACTI ![Globus Success](./static/globus_success.png) -### Small file download from web browsers +### Small File Download from Web Browsers Globus support HTTPS access to data. To download a small file from your web browser, select a file and right-click your mouse, then click 'Download' at the popup menu. From 79fa3392e3ef3a651a48f3b3cea204c749563b2a Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Tue, 14 Jul 2026 15:04:17 +0800 Subject: [PATCH 10/84] Update 07_transferring_cloud_storage_data_with_rclone.md --- .../07_transferring_cloud_storage_data_with_rclone.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/hpc/03_storage/07_transferring_cloud_storage_data_with_rclone.md b/docs/hpc/03_storage/07_transferring_cloud_storage_data_with_rclone.md index 79a6713d9a..0eca8f593e 100644 --- a/docs/hpc/03_storage/07_transferring_cloud_storage_data_with_rclone.md +++ b/docs/hpc/03_storage/07_transferring_cloud_storage_data_with_rclone.md @@ -1,10 +1,10 @@ -# Transferring Cloud Storage Data with rclone +# Transferring Cloud Storage Data with Rclone :::tip Globus Globus is the recommended tool to use for large-volume data transfers due to the efficiency, reliability, security and ease of use. Use other tools only if you really need to. Detailed instructions available at [Globus](./03_globus.md) ::: -## Transferring files to and from Google Drive with RCLONE +## Transferring Files to and from Google Drive with Rclone Having access to Google Drive from the HPC environment provides an option to archive data and even share data with collaborators who have no access to the NYU HPC environment. Other options to archiving data include the HPC Archive file system and using [Globus](./03_globus.md) to share data with collaborators. Access to Google Drive is provided by [rclone](https://rclone.org/drive/) - rsync for cloud storage - a command line program to sync files and directories to and from cloud storage systems such as Google Drive, Amazon Drive, S3, B2 etc. [rclone](https://rclone.org/drive/) is available on Torch cluster as a module, the module versions currently available (March 2025) are: @@ -26,7 +26,7 @@ rclone --transfers=32 --checkers=16 --drive-chunk-size=16384k --drive-upload-cut This option works great for file sizes 1Gb+ to 250GB. Keep in mind that there is a rate limiting of 2 files/sec for upload into Google Drive. Small file transfers don’t work that well. If you have many small jobs, please tar the parent directory of such folders and split the tar file into 100GB chunks and then uploads then into Google Drive. -## rclone Configuration +## Rclone Configuration You need to configure rclone before you will be able to move files between the HPC Environment and Google Drive There are specific instruction on the [rclone web site](https://rclone.org/drive/) and here is an example of the process for configuring rclone for use on Torch: @@ -35,12 +35,12 @@ There are specific instruction on the [rclone web site](https://rclone.org/drive Follow [instructions](../02_connecting_to_hpc/01_connecting_to_hpc.mdx) to log into the Torch HPC cluster. -### Step 2: Load the rclone module +### Step 2: Load the Rclone Module ```sh $ module load rclone/1.68.2 ``` -### Step 3: Configure rclone +### Step 3: Configure Rclone Configuring rclone and setting up remote access to your Google Drive, using the command: ```sh From 8b04527e15561d92e72099457d59c1f27499da4a Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Tue, 14 Jul 2026 15:05:29 +0800 Subject: [PATCH 11/84] Update 02_data_transfers.md --- docs/hpc/03_storage/02_data_transfers.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/hpc/03_storage/02_data_transfers.md b/docs/hpc/03_storage/02_data_transfers.md index a3e9796dd6..baec227c71 100644 --- a/docs/hpc/03_storage/02_data_transfers.md +++ b/docs/hpc/03_storage/02_data_transfers.md @@ -4,7 +4,7 @@ Globus is the recommended tool to use for large-volume data transfers due to the efficiency, reliability, security and ease of use. Use other tools only if you really need to. Detailed instructions available at [Globus](./03_globus.md) ::: -## Data-Transfer nodes +## Data-Transfer Nodes Attached to the NYU HPC cluster Torch, the Torch Data Transfer Node (DTN) are nodes optimized for transferring data between cluster file systems (e.g. scratch) and other endpoints outside the NYU HPC clusters, including user laptops and desktops. The gDTNs have 100-Gb/s Ethernet connections to the High Speed Research Network (HSRN) and are connected to the HDR Infiniband fabric of the HPC clusters. More information on the hardware characteristics is available at [Torch spec sheet](../10_spec_sheet.md). ### Data Transfer Node Access @@ -25,7 +25,7 @@ logout ## Tools for Data Transfer ### Linux & Mac Tools -#### scp and rsync +#### Scp and Rsync :::warning Please use Data Transfer Nodes (DTNs) with these tools. While one can transfer data while on login nodes, it is considered a bad practice because it can degrade the node's performance. ::: @@ -49,7 +49,7 @@ Globus is the recommended tool to use for large-volume data transfers. It featur The Globus endpoint for Torch is available at `nyu#torch`. Detailed instructions available at [Globus](./03_globus.md) -### rclone +### Rclone rclone - rsync for cloud storage, is a command line program to sync files and directories to and from cloud storage systems such as Google Drive, Amazon Drive, S3, B2 etc. rclone is available on DTNs. [Please see the documentation for how to use it.](https://rclone.org/) ### Open OnDemand (OOD) From a1f18919c8e449fb46725357a53cfe72313d6123 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Tue, 14 Jul 2026 15:57:45 +0800 Subject: [PATCH 12/84] Update 04_walkthrough_renew_hpc_account.md --- .../02_HPC_Accounts/04_walkthrough_renew_hpc_account.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hpc/01_getting_started/02_HPC_Accounts/04_walkthrough_renew_hpc_account.md b/docs/hpc/01_getting_started/02_HPC_Accounts/04_walkthrough_renew_hpc_account.md index 5f193a53a8..6156579ee0 100644 --- a/docs/hpc/01_getting_started/02_HPC_Accounts/04_walkthrough_renew_hpc_account.md +++ b/docs/hpc/01_getting_started/02_HPC_Accounts/04_walkthrough_renew_hpc_account.md @@ -1,4 +1,4 @@ -# Renewing your HPC Account +# Renewing Your HPC Account :::info VPN Needed You need to be on the NYU VPN to perform this task! From 9b0d4f701ce630d28dd028b51d9fb860b1a309a5 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Tue, 14 Jul 2026 15:59:39 +0800 Subject: [PATCH 13/84] Update 01_intro.md --- docs/hpc/04_datasets/01_intro.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/hpc/04_datasets/01_intro.md b/docs/hpc/04_datasets/01_intro.md index 3c3ec6eada..c724456134 100644 --- a/docs/hpc/04_datasets/01_intro.md +++ b/docs/hpc/04_datasets/01_intro.md @@ -49,7 +49,7 @@ ImageNet is an image dataset organized according to the [WordNet](https://wordne WordNet® is a large lexical database of English. Nouns, verbs, adjectives and adverbs are grouped into sets of cognitive synonyms (synsets), each expressing a distinct concept ([https://wordnet.princeton.edu/](https://wordnet.princeton.edu/)) ::: -#### ILSVRC (subset of ImageNet) +#### ILSVRC (Subset of ImageNet) ILSVRC uses a subset of ImageNet images for training the algorithms and some of ImageNet’s image collection protocols for annotating additional images for testing the algorithms ([https://arxiv.org/abs/1409.0575](https://arxiv.org/abs/1409.0575)). The name comes from 'ImageNet Large Scale Visual Recognition Challenge ([ILSVRC](https://image-net.org/challenges/LSVRC/2017/))'. Competition was moved to Kaggle ([http://image-net.org/challenges/LSVRC/2017/](http://image-net.org/challenges/LSVRC/2017/)) @@ -63,7 +63,7 @@ ILSVRC uses a subset of ImageNet images for training the algorithms and some of *Dataset is available under* - `/projects/work/public/ml-datasets/imagenet` -##### Get access to Data +##### Get Access to Data New York University does not own this dataset. @@ -110,7 +110,7 @@ This dataset consists of 9 million images covering 90k English words, and includ *Dataset is available under* - `/projects/work/public/ml-datasets/mjsynth` -### open-images-dataset +### Open-images-dataset *About data set*: [Open Images Dataset – opensource.google](https://storage.googleapis.com/openimages/web/index.html) A dataset of ~9 million varied images with rich annotations @@ -128,7 +128,7 @@ The Pile is a 825 GiB diverse, open source language modeling data set that consi *Dataset is available under* - `/projects/work/public/ml-datasets/pile` -### Waymo open dataset +### Waymo Open Dataset *About data set*: [Open Dataset – Waymo](https://waymo.com/open/) The field of machine learning is changing rapidly. Waymo is in a unique position to contribute to the research community with some of the largest and most diverse autonomous driving datasets ever released. From dda0fb9a889ab5acfe3b1f23d4726eebe0421160 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Tue, 14 Jul 2026 16:03:26 +0800 Subject: [PATCH 14/84] Update 02_run_hf_model.md --- .../08_LLM inference/02_run_hf_model.md | 28 +++++++++---------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/docs/hpc/08_ml_ai_hpc/08_LLM inference/02_run_hf_model.md b/docs/hpc/08_ml_ai_hpc/08_LLM inference/02_run_hf_model.md index c71962af86..b432027443 100644 --- a/docs/hpc/08_ml_ai_hpc/08_LLM inference/02_run_hf_model.md +++ b/docs/hpc/08_ml_ai_hpc/08_LLM inference/02_run_hf_model.md @@ -1,9 +1,9 @@ -# Basic LLM Inference with Hugging Face transformers +# Basic LLM Inference with Hugging Face Transformers Here we provide an example of how one can run a Hugging Face Large-language model (LLM) on the NYU Torch cluster -## Prepare environment -### Create project directory +## Prepare Environment +### Create Project Directory After [logging on to a Torch login node](../../02_connecting_to_hpc/01_connecting_to_hpc.mdx), make a directory for this project: ```bash @@ -14,30 +14,30 @@ After [logging on to a Torch login node](../../02_connecting_to_hpc/01_connectin You'll need to replace NetID above with your NetID ::: -### Move to a compute node +### Move to a Compute Node Some of the following steps can require significant resources, so we'll move to a compute node. This way we won't overload the login node we're on. ```bash [NetID@log-1 llm_example]$ srun --cpus-per-task=2 --mem=10GB --time=04:00:00 --pty /bin/bash ``` -### Copy appropriate overlay file to the project directory +### Copy Appropriate Overlay File to the Project Directory ```bash [NetID@cm001 llm_example]$ cp -rp /share/apps/overlay-fs-ext3/overlay-50G-10M.ext3.gz . [NetID@cm001 llm_example]$ gunzip overlay-50G-10M.ext3.gz ``` -### Launch Singularity container in read/write mode +### Launch Singularity container in Read/write Mode ```bash [NetID@cm001 llm_example]$ apptainer exec --fakeroot --overlay overlay-50G-10M.ext3:rw /share/apps/images/cuda12.1.1-cudnn8.9.0-devel-ubuntu22.04.2.sif /bin/bash ``` -### Install miniconda in the container +### Install Miniconda in the Container ```bash Singularity> wget --no-check-certificate https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh Singularity> bash Miniforge3-Linux-x86_64.sh -b -p /ext3/miniforge3 ``` -### Create environment script +### Create Environment Script Use an editor like nano or vim to create the file `/ext3/env.sh`. The contents should be: ```bash #!/bin/bash @@ -49,12 +49,12 @@ export PATH=/ext3/miniforge3/bin:$PATH export PYTHONPATH=/ext3/miniforge3/bin:$PATH ``` -### Activate the environment +### Activate the Environment ```bash Singularity> source /ext3/env.sh ``` -### Install packages in environment +### Install Packages in Environment ```bash Singularity> conda config --remove channels defaults Singularity> conda update -n base conda -y @@ -63,7 +63,7 @@ Singularity> conda install pip -y Singularity> pip install torch numpy transformers ``` -### Exit from Singularity and the compute node +### Exit from Singularity and the Compute Node ```bash Singularity> exit [NetID@cm001 llm_example]$ exit @@ -73,7 +73,7 @@ Singularity> exit You can find more information about using Singularity and Conda on our HPC systems in our documentation [Singularity with Conda](https://sites.google.com/nyu.edu/nyu-hpc/hpc-systems/torch/software/singularity-with-miniconda). ::: -## Prepare script +## Prepare Script Create a python script using the following code from sections 1-9 and save it in a file called `huggingface.py`: 1. Import necessary modules: @@ -134,7 +134,7 @@ Create a python script using the following code from sections 1-9 and save it in print("Shape of the batch embedding: {}".format(sentence_embedding.shape)) ``` -## Prepare Sbatch file +## Prepare Sbatch File After saving the above code in a script called `huggingface.py`, create a file called `run.SBATCH` with the following code: ```batch @@ -162,7 +162,7 @@ You'll need to change `NetID` in the script above to your NetID. If you're using a different directory name and/or path you'll also need to update that in the script above. ::: -## Run the run.SBATCH file +## Run the run.SBATCH File ```batch [NetID@log-1 llm_example]$ sbatch run.SBATCH ``` From a56f4fdb7d7ba2ab35fe0346629ed738c36dcb46 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Tue, 14 Jul 2026 16:07:16 +0800 Subject: [PATCH 15/84] Update 03_vLLM.md --- docs/hpc/08_ml_ai_hpc/08_LLM inference/03_vLLM.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/hpc/08_ml_ai_hpc/08_LLM inference/03_vLLM.md b/docs/hpc/08_ml_ai_hpc/08_LLM inference/03_vLLM.md index 0af63cdcf1..b4264e3837 100644 --- a/docs/hpc/08_ml_ai_hpc/08_LLM inference/03_vLLM.md +++ b/docs/hpc/08_ml_ai_hpc/08_LLM inference/03_vLLM.md @@ -1,4 +1,4 @@ -# High-performance LLM inference with `vLLM` +# High-performance LLM Inference with `vLLM` ## What is vLLM? [`vLLM`](https://docs.vllm.ai/en/latest/) is an easy-to-use library for LLM inference and serving which support a wide variety of models with optimized kernels ensuring optimal utilization of GPUs. @@ -29,7 +29,7 @@ Create a `vLLM` directory in your /scratch directory, then install the vLLM imag ``` apptainer pull docker://vllm/vllm-openai:latest ``` -### Avoid filling up your `$HOME` directory +### Avoid Filling up your `$HOME` Directory To avoid exceeding your `$HOME` quota (50GB) and inode limits (30,000 files), you should redirect `vLLM`'s cache and Hugging Face's model downloads to your scratch space: ```sh export HF_HOME=/scratch/$USER/hf_cache From c61aae744519219bba74afe1f7e841debf7abc19 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Tue, 14 Jul 2026 16:39:56 +0800 Subject: [PATCH 16/84] Update 03_pytorch_dpp.md --- docs/hpc/08_ml_ai_hpc/03_pytorch_dpp.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/hpc/08_ml_ai_hpc/03_pytorch_dpp.md b/docs/hpc/08_ml_ai_hpc/03_pytorch_dpp.md index 8440b19c09..c5b44cb961 100644 --- a/docs/hpc/08_ml_ai_hpc/03_pytorch_dpp.md +++ b/docs/hpc/08_ml_ai_hpc/03_pytorch_dpp.md @@ -36,7 +36,7 @@ Here are some webpages and videos: Do not use `DataParallel` in PyTorch for anything since it gives poor performance relative to `DistributedDataParallel`. ::: -## Main changes needed in going from single-GPU to multi-GPU training with DDP +## Main Changes Needed in Going from Single-GPU to Multi-GPU Training with DDP This completely new piece is needed to form the process group: @@ -192,7 +192,7 @@ The `local_rank` should be used everywhere in your script except when initializi # DDP and Slurm -## Total number of tasks equals total number of GPUs +## Total Number of Tasks Equals Total Number of GPUs When using DDP, the total number of tasks must equal the total number of allocated GPUs. Therefore, if `--ntasks-per-node=` then you must have `--gres=gpu:`. Here are two examples: @@ -429,6 +429,6 @@ Execute the commands below to run the example above: [NetID@log-1 full-ddp-test]$ sbatch run-full.SBATCH ``` -## Memory issues +## Memory Issues Use `gradient_as_bucket_view=True` when making the DDP model to decrease the required memory by 1/3. From 076ee1e13195e351cc4ce2d08f533a6c5f5b80b0 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Tue, 14 Jul 2026 16:41:41 +0800 Subject: [PATCH 17/84] Update 02_pytorch_intro.md --- docs/hpc/08_ml_ai_hpc/02_pytorch_intro.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hpc/08_ml_ai_hpc/02_pytorch_intro.md b/docs/hpc/08_ml_ai_hpc/02_pytorch_intro.md index e021706289..7500f0f70d 100644 --- a/docs/hpc/08_ml_ai_hpc/02_pytorch_intro.md +++ b/docs/hpc/08_ml_ai_hpc/02_pytorch_intro.md @@ -315,7 +315,7 @@ The slowest line is number 42 which consumes 84.5% of the time in the training f Note that GPU utilization as measured using nvidia-smi is only a measure of the fraction of the time that a GPU kernel is running on the GPU. It says nothing about how many CUDA cores are being used or how efficiently the GPU kernels have been written. However, for codes used by large communities, one can generally associate GPU utilization with overall GPU efficiency. For a more accurate measure of GPU utilization, use [Nsight Systems or Nsight Compute](https://researchcomputing.princeton.edu/support/knowledge-base/gpu-computing#profiling) to measure the occupancy. -## Step 6: Work through the Performance Tuning Guide from Princeton +## Step 6: Work Through the Performance Tuning Guide from Princeton Make sure you optimize the single GPU case before going to multiple GPUs by working through the [Performance Tuning Guide](https://pytorch.org/tutorials/recipes/recipes/tuning_guide.html). From 2f44cc3056d97b4ce64c8d1959ee23bd8ca6f8c7 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Tue, 14 Jul 2026 16:48:08 +0800 Subject: [PATCH 18/84] Update 02_slurm_main_commands.md --- docs/hpc/05_submitting_jobs/02_slurm_main_commands.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hpc/05_submitting_jobs/02_slurm_main_commands.md b/docs/hpc/05_submitting_jobs/02_slurm_main_commands.md index 039332c036..c442cdbcb3 100644 --- a/docs/hpc/05_submitting_jobs/02_slurm_main_commands.md +++ b/docs/hpc/05_submitting_jobs/02_slurm_main_commands.md @@ -1,4 +1,4 @@ -# Slurm: Command reference +# Slurm: Command Reference Slurm offers many utility commands to work with, some of the most popularly used commands are: From 4e7b7aa149677b4b6342ec836c5353b846b71ce1 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Tue, 14 Jul 2026 16:50:02 +0800 Subject: [PATCH 19/84] Update 01_slurm_submitting_jobs.md --- .../hpc/05_submitting_jobs/01_slurm_submitting_jobs.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/hpc/05_submitting_jobs/01_slurm_submitting_jobs.md b/docs/hpc/05_submitting_jobs/01_slurm_submitting_jobs.md index 9128bfa9ac..3fcf25f0f8 100644 --- a/docs/hpc/05_submitting_jobs/01_slurm_submitting_jobs.md +++ b/docs/hpc/05_submitting_jobs/01_slurm_submitting_jobs.md @@ -53,7 +53,7 @@ You can find the available types of GPUs at [Torch Spec Sheet](../10_spec_sheet. Do not specify partitions manually, except for preemption which is described later. ::: -## Resource limits and restrictions +## Resource Limits and Restrictions Jobs within the same partition cannot exceed their assigned resources (`QOSGrpGRES`). User GPU Quotas: Each user has a total GPU quota of 24 GPUs for jobs with wall time < 48 hours (`QOSMaxGRESPerUser`). Non-stakeholders to temporarily use stakeholder resources (a stakeholder group to temporarily use another group’s resources). Stakeholders retain normal access to their own resources. If non-stakeholders (or other stakeholders) are using them, their jobs may be preempted (canceled) once stakeholders submit new jobs. Public users are allowed to use stakeholder resources only with preemption partitions. Refer to the section below for details on preemptible jobs. @@ -81,7 +81,7 @@ Jobs with low GPU utilization will be automatically canceled: Enforcement will be very aggressive. ::: -## Preemptible jobs on Torch +## Preemptible Jobs on Torch On Torch, users may run "preemptible" jobs on stakeholder resources that their group does not own. This allows the stakeholder resources to be utilized by non-stakeholders which may otherwise be idle. To make the best use of these resources, you are encouraged to adopt checkpoint/restart to allow for resumption of the workload in subsequent jobs. :::warning Preemption Policy @@ -109,19 +109,19 @@ Jobs in stakeholder partitions will not be canceled, but those in preemption par ``` Jobs with preemption partitions only might be allowed to use more resources -## Advanced options +## Advanced Options ### GPU MPS Use GPU Multi-Process Service (MPS) to improve overall GPU utilization, as this allows multiple GPU jobs to share a single GPU concurrently by: ``` #SBATCH --comment="gpu_mps=yes" ``` -### RAM disk +### RAM Disk A portion of the RAM available can be mounted as a disk for fast `I/O` operations: ``` #SBATCH --comment="ram_disk=1GB" ``` -### GPU MPS & RAM Disk in a preemptible job +### GPU MPS & RAM Disk in a Preemptible Job Both of these can be combined with preemption as shown: ``` #SBATCH --comment="preemption=yes;preemption_partitions_only=yes;requeue=true;gpu_mps=yes;ram_disk=1GB" From 9ceaebb10880e54d4ee058c11d43239791fec081 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Tue, 14 Jul 2026 17:19:22 +0800 Subject: [PATCH 20/84] Update 03_singularity_with_conda.md --- docs/hpc/07_containers/03_singularity_with_conda.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/hpc/07_containers/03_singularity_with_conda.md b/docs/hpc/07_containers/03_singularity_with_conda.md index 8fd5c65649..e48ac439a6 100644 --- a/docs/hpc/07_containers/03_singularity_with_conda.md +++ b/docs/hpc/07_containers/03_singularity_with_conda.md @@ -147,7 +147,7 @@ exit # exit Singularity ``` -#### Install packages +#### Install Packages You may now install packages into the environment with either the `pip install` or `conda install` commands. @@ -202,7 +202,7 @@ Now, exit the Singularity container and then rename the overlay image. Typing `e exit mv overlay-15GB-500K.ext3 my_pytorch.ext3 ``` -#### Test your PyTorch Singularity Image +#### Test Your PyTorch Singularity Image ```sh singularity exec --overlay /scratch//pytorch-example/my_pytorch.ext3:ro /share/apps/images/cuda12.1.1-cudnn8.9.0-devel-ubuntu22.04.2.sif /bin/bash -c 'source /ext3/env.sh; python -c "import torch; print(torch.__file__); print(torch.__version__)"' @@ -213,7 +213,7 @@ singularity exec --overlay /scratch//pytorch-example/my_pytorch.ext3:ro / the end `:ro` addition at the end of the pytorch ext3 image starts the image in read-only mode. To add packages you will need to use `:rw` to launch it in read-write mode. ::: -### Using your Singularity Container in a SLURM Batch Job +### Using Your Singularity Container in a SLURM Batch Job Below is an example script of how to call a python script, in this case `torch-test.py`, from a SLURM batch job using your new Singularity image torch-test.py: @@ -276,7 +276,7 @@ cat slurm-3752662.out # True ``` -### Optional: Convert ext3 to a compressed, read-only squashfs filesystem +### Optional: Convert ext3 to a Compressed, Read-only Squashfs Filesystem Singularity images can be compressed into read-only squashfs filesystems to conserve space in your environment. Use the following steps to convert your ext3 Singularity image into a smaller squashfs filesystem. ```sh srun -N1 -c4 singularity exec --overlay my_pytorch.ext3:ro /share/apps/images/centos-8.2.2004.sif mksquashfs /ext3 /scratch//pytorch-example/my_pytorch.sqf -keep-as-directory -processors 4 -noappend From f1f01d4849683e6451ca7154b15299e53cd09ea5 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 11:53:03 +0800 Subject: [PATCH 21/84] Update 02_data_transfers.md --- docs/hpc/03_storage/02_data_transfers.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/hpc/03_storage/02_data_transfers.md b/docs/hpc/03_storage/02_data_transfers.md index baec227c71..e04c07ad62 100644 --- a/docs/hpc/03_storage/02_data_transfers.md +++ b/docs/hpc/03_storage/02_data_transfers.md @@ -25,7 +25,7 @@ logout ## Tools for Data Transfer ### Linux & Mac Tools -#### Scp and Rsync +#### `scp` and `rsync` :::warning Please use Data Transfer Nodes (DTNs) with these tools. While one can transfer data while on login nodes, it is considered a bad practice because it can degrade the node's performance. ::: @@ -49,7 +49,7 @@ Globus is the recommended tool to use for large-volume data transfers. It featur The Globus endpoint for Torch is available at `nyu#torch`. Detailed instructions available at [Globus](./03_globus.md) -### Rclone +### `rclone` rclone - rsync for cloud storage, is a command line program to sync files and directories to and from cloud storage systems such as Google Drive, Amazon Drive, S3, B2 etc. rclone is available on DTNs. [Please see the documentation for how to use it.](https://rclone.org/) ### Open OnDemand (OOD) From c3290e81acf74a812d0774b996c9270cafbdc15e Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 11:55:32 +0800 Subject: [PATCH 22/84] Update 06_large_number_of_small_files.md --- docs/hpc/03_storage/06_large_number_of_small_files.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/hpc/03_storage/06_large_number_of_small_files.md b/docs/hpc/03_storage/06_large_number_of_small_files.md index 13efb97309..f8ed6782a5 100644 --- a/docs/hpc/03_storage/06_large_number_of_small_files.md +++ b/docs/hpc/03_storage/06_large_number_of_small_files.md @@ -5,7 +5,7 @@ Many datasets contain a large number of files (for example [ImageNet](https://en Here are some ideas you can try and evaluate performance for your own project -## Squash File System With Singularity +## Squash File System with Singularity Please see [Squash File System and Singularity](../07_containers/04_squash_file_system_and_singularity.md) ## Use JPG/PNG Files On Disk @@ -41,7 +41,7 @@ Store data in some binary file (say pickle in Python) which you load fully when This option may require a lot of RAM - thus you may have to wait a long time for the scheduler to find resources for your job. Also this approach would not work on a regular PC without so much RAM, and thus your scripts are not transferable. -## Container Files, One-File Databases +## Container Files, One-file Databases Special containers, which allow to either load data fast fully or access chosen elements without loading the whole dataset into RAM. ### SQLite From c73c55b5f0dd63da8e544fa1cf173b54831c09a9 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 11:57:22 +0800 Subject: [PATCH 23/84] Refactor rclone documentation for clarity and consistency Updated formatting and headings for consistency, changed 'Rclone' to '`rclone`' for clarity, and improved instructions for better readability. --- .../07_transferring_cloud_storage_data_with_rclone.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/hpc/03_storage/07_transferring_cloud_storage_data_with_rclone.md b/docs/hpc/03_storage/07_transferring_cloud_storage_data_with_rclone.md index 0eca8f593e..89f15144dd 100644 --- a/docs/hpc/03_storage/07_transferring_cloud_storage_data_with_rclone.md +++ b/docs/hpc/03_storage/07_transferring_cloud_storage_data_with_rclone.md @@ -4,7 +4,7 @@ Globus is the recommended tool to use for large-volume data transfers due to the efficiency, reliability, security and ease of use. Use other tools only if you really need to. Detailed instructions available at [Globus](./03_globus.md) ::: -## Transferring Files to and from Google Drive with Rclone +## Transferring Files to and from Google Drive with `rclone` Having access to Google Drive from the HPC environment provides an option to archive data and even share data with collaborators who have no access to the NYU HPC environment. Other options to archiving data include the HPC Archive file system and using [Globus](./03_globus.md) to share data with collaborators. Access to Google Drive is provided by [rclone](https://rclone.org/drive/) - rsync for cloud storage - a command line program to sync files and directories to and from cloud storage systems such as Google Drive, Amazon Drive, S3, B2 etc. [rclone](https://rclone.org/drive/) is available on Torch cluster as a module, the module versions currently available (March 2025) are: @@ -26,7 +26,7 @@ rclone --transfers=32 --checkers=16 --drive-chunk-size=16384k --drive-upload-cut This option works great for file sizes 1Gb+ to 250GB. Keep in mind that there is a rate limiting of 2 files/sec for upload into Google Drive. Small file transfers don’t work that well. If you have many small jobs, please tar the parent directory of such folders and split the tar file into 100GB chunks and then uploads then into Google Drive. -## Rclone Configuration +## `rclone` Configuration You need to configure rclone before you will be able to move files between the HPC Environment and Google Drive There are specific instruction on the [rclone web site](https://rclone.org/drive/) and here is an example of the process for configuring rclone for use on Torch: @@ -35,12 +35,12 @@ There are specific instruction on the [rclone web site](https://rclone.org/drive Follow [instructions](../02_connecting_to_hpc/01_connecting_to_hpc.mdx) to log into the Torch HPC cluster. -### Step 2: Load the Rclone Module +### Step 2: Load the `rclone` Module ```sh $ module load rclone/1.68.2 ``` -### Step 3: Configure Rclone +### Step 3: Configure `rclone` Configuring rclone and setting up remote access to your Google Drive, using the command: ```sh From 013d6e015b8da02178e53a6a16c13b30dc36fab9 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 12:56:07 +0800 Subject: [PATCH 24/84] Update 01_intro.mdx --- docs/hpc/01_getting_started/01_intro.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hpc/01_getting_started/01_intro.mdx b/docs/hpc/01_getting_started/01_intro.mdx index 74bc3b26be..dff36bdda4 100644 --- a/docs/hpc/01_getting_started/01_intro.mdx +++ b/docs/hpc/01_getting_started/01_intro.mdx @@ -1,4 +1,4 @@ -# Start here! +# Start Here! Welcome to the Torch HPC documentation! If you do not have an HPC account, please proceed to the next section that explains how you may be able to get one. From 8c9bc489b29a1df4aa970fb6ad2bfddd40914572 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 12:58:42 +0800 Subject: [PATCH 25/84] Update 01_intro.md --- docs/cloud/04_dataproc/01_intro.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/cloud/04_dataproc/01_intro.md b/docs/cloud/04_dataproc/01_intro.md index dcf4525554..42196ad11c 100644 --- a/docs/cloud/04_dataproc/01_intro.md +++ b/docs/cloud/04_dataproc/01_intro.md @@ -9,7 +9,7 @@ Dataproc is a cloud-based Hadoop distribution that is managed by Google. Google administers updates to Dataproc so that it is kept current. Google also packages and maintains additional software that can be run on top of Hadoop. Additionally Dataproc includes other cloud-specific features, such as the ability to automatically add/remove nodes depending upon how busy the cluster is (autoscaling). It can also use object storage ([GCS][gcs]) or [BigQuery][bigquery] as an alternative to HDFS, and provides integration with [BigTable][bigtable] using HBase interfaces. -### What is Hadoop? +### What Is Hadoop? Hadoop is an open-source software framework for storing and processing big data in a distributed/parallel fashion on large clusters of commodity hardware. At its core, Hadoop strives to increase processing speed by increasing [data locality][data-locality] (i.e., it moves computation to servers where the data is located). There are three components to Hadoop: HDFS (the Hadoop Distributed File System), the Hadoop implementation of MapReduce, and YARN (Yet Another Resource Negotiator; a scheduler). From dba740983566b01c42a58c437a05b17e43e727bf Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 13:00:01 +0800 Subject: [PATCH 26/84] Fix capitalization in eligibility section header --- docs/genai/02_onboarding/01_intro.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/genai/02_onboarding/01_intro.mdx b/docs/genai/02_onboarding/01_intro.mdx index 39a0f22090..e0d298b8f3 100644 --- a/docs/genai/02_onboarding/01_intro.mdx +++ b/docs/genai/02_onboarding/01_intro.mdx @@ -9,7 +9,7 @@ Access to Pythia is facilitated via workspaces. Beyond facilitating access, they ::: -## Who is eligible for to access Pythia? +## Who Is Eligible for to Access Pythia? All full-time NYU faculty have the ability to request the creation of workspaces by filling out [the research intake form][research-workspace-request]. Once we create the workspace, members need to submit the [member on-boarding form][member-onboarding-form] to gain access to the workspace. Please be sure to indicate the models you'd like access to from the [catalouge](../03_external_llms/01_catalogue.md) From 8ec00224e26b26c9260dfcad7acda33dae0d772e Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 13:03:03 +0800 Subject: [PATCH 27/84] Update 03_quickstart.mdx --- docs/genai/02_onboarding/03_quickstart.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/genai/02_onboarding/03_quickstart.mdx b/docs/genai/02_onboarding/03_quickstart.mdx index 02008e608c..22189a15ad 100644 --- a/docs/genai/02_onboarding/03_quickstart.mdx +++ b/docs/genai/02_onboarding/03_quickstart.mdx @@ -7,7 +7,7 @@ If you are sending requests from a server like Colab notebook while your laptop ::: -## Getting started with the LLM gateway +## Getting Started with the LLM Gateway :::tip[Gateway URL] Whenever you instantiate a Portkey client, the `base_url` must be set to `base_url="https://ai-gateway.apps.cloud.rt.nyu.edu/v1/"`. If you miss this parameter you would be connecting to the vendor's SaaS platform and the API keys you created after signing on with SSO will not work. From feaa85ca196435125f43404d66a8f61c2a6ae512 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 13:08:43 +0800 Subject: [PATCH 28/84] Update 01_intro.md --- docs/cloud/02_gcp_self_managed/01_intro.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/cloud/02_gcp_self_managed/01_intro.md b/docs/cloud/02_gcp_self_managed/01_intro.md index e26c3d037f..7ac97bc85b 100644 --- a/docs/cloud/02_gcp_self_managed/01_intro.md +++ b/docs/cloud/02_gcp_self_managed/01_intro.md @@ -12,7 +12,7 @@ We facilitate access to Google Cloud Platform (GCP) for your research projects. In addition to the I2 Net+ GCP benefits, NYU scholars can enjoy significant discounts in using GCP resources in their research project through a 3 year commitment (started in 2019) NYU made in GCP. The NYU network connects with GCP via Partner Interconnect using [Internet2 Cloud Connect (I2CC)][internet2-cc] service. -## Why work with the NYU Research Cloud team to deploy your research project on GCP? +## Why Work with the NYU Research Cloud Team to Deploy Your Research Project on GCP? NYU researchers who work with the Research Cloud team to deploy projects on GCP may benefit from the following: - Discounted rates, lower GCP project cost: Through NYU's participation in the Internet2 Net+ agreement, as well as the 3-year commitment NYU made to using GCP, GCP projects enjoy discounted rates, lowering the cost of the project. The exact discounts depend on the GCP service used in the research project and can vary between 5% and 25%. Free data egress is usually included. @@ -36,12 +36,13 @@ The NYU Research Cloud team does not currently offer training on how to deploy a - Through the [Google For Education][google-for-edu] program, GCP offers training credits and discounts to Students, Faculty, and IT Admins. To apply for training credits and discounts, please [click here](https://services.google.com/fb/forms/googlecloudskillsbooststudenttrainingcreditsapplication/). - [Getting started with Google Cloud Platform][gcp-get-started] offers quick starts and sample projects on GCP. -## How can I fund my research project on GCP? +## How Can I Fund My Research Project on GCP? ### GCP Free Tier Apply for credits using your NYU account (https://cloud.google.com/free/). After credits expire, if you would like to switch to another type of funding and are approved to do so, we will modify your project so it can use other funds: - https://edu.google.com/programs/credits/research/?modal_active=none - https://edu.google.com/programs/?modal_active=none -### Sources of funding for GCP project + +### Sources of Funding for GCP Project Please consider options below and explore other options which may exist for your specific field. - [Google Cloud research credits][gcp-credits] - NIH STRIDES From 7f10d82a3156e89d3068ae80223e296fb3b40d33 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 13:09:37 +0800 Subject: [PATCH 29/84] Update 02_nih_strides.md --- docs/cloud/02_gcp_self_managed/02_nih_strides.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/cloud/02_gcp_self_managed/02_nih_strides.md b/docs/cloud/02_gcp_self_managed/02_nih_strides.md index 06150fc628..4db560eb63 100644 --- a/docs/cloud/02_gcp_self_managed/02_nih_strides.md +++ b/docs/cloud/02_gcp_self_managed/02_nih_strides.md @@ -13,7 +13,7 @@ The benefits of participating in NIH Strides program include: - Receive opportunities for professional service engagements to help drive success - Receive guidance for best practices in areas such as data storage, governance, and controlled access -## Enrolling to the NIH Strides initiative +## Enrolling to the NIH Strides Initiative NYU has enrolled in the NIH Strides initiative in December 2020 by signing an agreement with Carahsoft, GCP's billing and administrative partner. Thus NIH-funded NYU researchers with an active NIH award may take advantage of the STRIDES Initiative for their NIH-funded research projects. The NYU RTS team works closely with Burwood Group, a GCP reseller, to provide access to GCP resources for NYU researchers who are approved to participate in the NIH STRIDES initiative. NYU researchers who wish to participate must follow the steps outlined below. From 55c31e7944aec901cb74b98c86bca05e93b7b54a Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 13:10:11 +0800 Subject: [PATCH 30/84] Update section title for workspace access --- docs/genai/02_onboarding/02_setup.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/genai/02_onboarding/02_setup.mdx b/docs/genai/02_onboarding/02_setup.mdx index 71f2bb980e..59e6b3d5e3 100644 --- a/docs/genai/02_onboarding/02_setup.mdx +++ b/docs/genai/02_onboarding/02_setup.mdx @@ -1,6 +1,6 @@ # Setup -## Accessing your workspace +## Accessing Your Workspace Login to Portkey at [`app.portkey.ai`](https://app.portkey.ai/login) with the `Single-sign-on` option using your NYU NetID. Once you're in, you'll be placed in the "Shared Workspace" by default. You can navigate to your workspace by selecting it from the workspace drop-down list from the top of the left sidebar. ## API Keys From 31b3456f5739146a473d52dbc8aa65bb00c01443 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 13:13:32 +0800 Subject: [PATCH 31/84] Update 03_retrieval_augmented_generation.mdx --- .../04_how_to_guides/03_retrieval_augmented_generation.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/genai/04_how_to_guides/03_retrieval_augmented_generation.mdx b/docs/genai/04_how_to_guides/03_retrieval_augmented_generation.mdx index 444bf17ba1..e6a82c95fe 100644 --- a/docs/genai/04_how_to_guides/03_retrieval_augmented_generation.mdx +++ b/docs/genai/04_how_to_guides/03_retrieval_augmented_generation.mdx @@ -1,4 +1,4 @@ -# Retrieval-augmented generation +# Retrieval-augmented Generation :::tip[Foundations of Research Computing session on RAG] For an in-depth overview of RAG and Jupyter notebook examples, please access the source materials used for the 2025 FORC session on RAG at: https://github.com/NYU-RTS/rag-forc-2025 From d3990dd1726fdd889ef22fdfa215c7be59d0edc9 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 13:13:48 +0800 Subject: [PATCH 32/84] Update 02_embeddings.mdx --- docs/genai/04_how_to_guides/02_embeddings.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/genai/04_how_to_guides/02_embeddings.mdx b/docs/genai/04_how_to_guides/02_embeddings.mdx index a43dca4df0..f6f1617804 100644 --- a/docs/genai/04_how_to_guides/02_embeddings.mdx +++ b/docs/genai/04_how_to_guides/02_embeddings.mdx @@ -1,4 +1,4 @@ -# Generating embeddings +# Generating Embeddings While Decoder-only LLMs gained massive popularity via their usage in chatbots, Encoder-only LLMs can be used for a wider variety of tasks. Decoder-only LLMs "generate" tokens ("text") one at a time probabalisticsally. Encoder-only LLMs on the other hand take text as their input, tokenize it and generate "embeddings" as their output. Here, we shall walk through a task of generating embeddings from a text snippet. From 21e64b6aa8f2152feb7c37695bbed1155a2a51c1 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 13:15:08 +0800 Subject: [PATCH 33/84] Fix typos and formatting in embeddings guide --- docs/genai/04_how_to_guides/02_embeddings.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/genai/04_how_to_guides/02_embeddings.mdx b/docs/genai/04_how_to_guides/02_embeddings.mdx index f6f1617804..291e8fd2d9 100644 --- a/docs/genai/04_how_to_guides/02_embeddings.mdx +++ b/docs/genai/04_how_to_guides/02_embeddings.mdx @@ -1,6 +1,6 @@ # Generating Embeddings -While Decoder-only LLMs gained massive popularity via their usage in chatbots, Encoder-only LLMs can be used for a wider variety of tasks. Decoder-only LLMs "generate" tokens ("text") one at a time probabalisticsally. Encoder-only LLMs on the other hand take text as their input, tokenize it and generate "embeddings" as their output. Here, we shall walk through a task of generating embeddings from a text snippet. +While Decoder-only LLMs gained massive popularity via their usage in chatbots, Encoder-only LLMs can be used for a wider variety of tasks. Decoder-only LLMs "generate" tokens ("text") one at a time probabilistically. Encoder-only LLMs on the other hand take text as their input, tokenize it and generate "embeddings" as their output. Here, we shall walk through a task of generating embeddings from a text snippet. ```mermaid flowchart LR; @@ -40,7 +40,7 @@ and gives the following response: [0.052587852, 0.094195396, 0.24439038, 0.104940414, -0.028921358, -0.31591928, -0.1846261, 0.221018, 0.033215445, -0.1382735, -0.14776362, -0.15058714, 0.057725072, -0.23435123, 0.07956805, -0.32156628, -0.08454841, 0.04066637, -0.022215525, 0.19090058, -0.11160703, 0.22258662, -0.06843088, -0.22854735, 0.1033718, -0.38085997, 0.2933312, -0.023215517, 0.20768477, -0.039333045, 0.17192031, -0.14180289] ``` -## Applications of embeddings +## Applications of Embeddings Embeddings are typically used for: - retrieval-augmented generation From 5dfd9ae9ff148978b831558802162f7fb4a12153 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 13:18:32 +0800 Subject: [PATCH 34/84] Update 02_data_management.md --- docs/cloud/04_dataproc/02_data_management.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/cloud/04_dataproc/02_data_management.md b/docs/cloud/04_dataproc/02_data_management.md index ae74969859..e526d13b55 100644 --- a/docs/cloud/04_dataproc/02_data_management.md +++ b/docs/cloud/04_dataproc/02_data_management.md @@ -1,4 +1,4 @@ -# Data management +# Data Management HDFS stands for Hadoop Distributed File System. HDFS is a highly fault-tolerant file system and is designed to be deployed on low-cost hardware. HDFS provides high throughput access to application data and is suitable for analyses that use large datasets. From 4ad397dd8e611ae3949f7fd78c4322a2433e13f0 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 13:20:09 +0800 Subject: [PATCH 35/84] Update 02_walkthrough_request_hpc_account.mdx --- .../02_HPC_Accounts/02_walkthrough_request_hpc_account.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/hpc/01_getting_started/02_HPC_Accounts/02_walkthrough_request_hpc_account.mdx b/docs/hpc/01_getting_started/02_HPC_Accounts/02_walkthrough_request_hpc_account.mdx index 8b5f14c364..de224c55b0 100644 --- a/docs/hpc/01_getting_started/02_HPC_Accounts/02_walkthrough_request_hpc_account.mdx +++ b/docs/hpc/01_getting_started/02_HPC_Accounts/02_walkthrough_request_hpc_account.mdx @@ -1,7 +1,7 @@ import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; -# How to request an HPC account +# How to Request an HPC Account :::tip Make sure you don't already have an HPC account. You can check this by attempting to log in to the cluster, according to the instructions at [Connecting to the HPC Cluster](../../02_connecting_to_hpc/01_connecting_to_hpc.mdx). @@ -36,7 +36,7 @@ You need to be on the NYU VPN to perform this task! If the **Request HPC Account** QuickLink is clicked, the following form appears: ![Faculty request form](../static/faculty_request_form.png) - The user’s name will be pre populated, and the forms required fields must be completed (sponsor, reason for request, consent to terms of use). After clicking “Submit” the chosen sponsor will be notified of the request and provisioning will only occur after approval. + The user’s name will be prepopulated, and the forms required fields must be completed (sponsor, reason for request, consent to terms of use). After clicking “Submit” the chosen sponsor will be notified of the request and provisioning will only occur after approval. If the **Bulk HPC Account Request** QuickLink is clicked, the following form appears: ![Bulk account request form](../static/bulk_acct_req_form.png) From f35d0b28f3951699dc4be6cfe493796227c7c574 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 13:22:14 +0800 Subject: [PATCH 36/84] Update 01_intro_slurm_accounts.mdx --- .../03_Slurm_Accounts/01_intro_slurm_accounts.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hpc/01_getting_started/03_Slurm_Accounts/01_intro_slurm_accounts.mdx b/docs/hpc/01_getting_started/03_Slurm_Accounts/01_intro_slurm_accounts.mdx index e0d7901a6e..b9d4060aeb 100644 --- a/docs/hpc/01_getting_started/03_Slurm_Accounts/01_intro_slurm_accounts.mdx +++ b/docs/hpc/01_getting_started/03_Slurm_Accounts/01_intro_slurm_accounts.mdx @@ -1,3 +1,3 @@ # Slurm Accounts -Users are required to have at least one active Slurm acount to submit jobs on the cluster. You will specify this with the `--account=` flag. Accounts are associated with an active allocation within a project that PIs manage using the [HPC Project Portal](02_hpc_project_management_portal.mdx). \ No newline at end of file +Users are required to have at least one active Slurm account to submit jobs on the cluster. You will specify this with the `--account=` flag. Accounts are associated with an active allocation within a project that PIs manage using the [HPC Project Portal](02_hpc_project_management_portal.mdx). From 7f3d868a556f8e072042b0fd28b09c18e44814db Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 13:23:17 +0800 Subject: [PATCH 37/84] Correct capitalization in 'Fine Tuning' header --- docs/genai/04_how_to_guides/05_llm_fine_tuning.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/genai/04_how_to_guides/05_llm_fine_tuning.md b/docs/genai/04_how_to_guides/05_llm_fine_tuning.md index 88ca258b44..410025cc79 100644 --- a/docs/genai/04_how_to_guides/05_llm_fine_tuning.md +++ b/docs/genai/04_how_to_guides/05_llm_fine_tuning.md @@ -1,3 +1,3 @@ -# Fine tuning +# Fine Tuning Please look into harnessing RAG before attempting to fine-tune a model. For open-weight models, you can use the HPC cluster to perform LoRA fine-tuning as [described here](../../hpc/08_ml_ai_hpc/05_llm_fine_tuning.md). From 3b1fa9641ad5ad3dc8cdf4d4c99392e003875683 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 13:24:24 +0800 Subject: [PATCH 38/84] Update headings for consistency in batch inference guide --- docs/genai/04_how_to_guides/04_batch_inference.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/genai/04_how_to_guides/04_batch_inference.md b/docs/genai/04_how_to_guides/04_batch_inference.md index 4542406364..74e3a32c85 100644 --- a/docs/genai/04_how_to_guides/04_batch_inference.md +++ b/docs/genai/04_how_to_guides/04_batch_inference.md @@ -6,14 +6,14 @@ When immediate results are not needed, for instance in transforming large datase Batch processing is only supported for LLMs that can be accessed via the `@vertexai` provider. ::: -## Collect the prompts +## Collect the Prompts You'll collect the prompts you want to send to the LLM as a newline delimited JSON ([JSONLines](https://jsonlines.org/)) where each line contains a single prompt in the OpenAI format. Here's an example we will be using in this example: ```json {"custom_id": "request-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gemini-3-flash-preview", "messages": [{"role": "user", "content": "Where is NYU located?"}], "max_tokens": 2048}} {"custom_id": "request-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gemini-3-flash-preview", "messages": [{"role": "user", "content": "What resources are available for genAI research at nyu?"}], "max_tokens": 8196}} ``` -## Upload them to the GCS bucket +## Upload Them to the GCS Bucket We will upload this via the Portkey client to the GCS ([Google Cloud Storage](https://cloud.google.com/storage)) bucket via the following script: ```python from portkey_ai import Portkey @@ -50,7 +50,7 @@ This script will print to standard output the location of the uploaded file, lik ``` -## Submit a batch inference job +## Submit a Batch Inference Job We are now ready to submit the batch inference job. Here's a script to do so: ```python @@ -78,7 +78,7 @@ print(batch_job) Upon successful submission, you'll see an `id` field that refers to the job id. -## Query job status +## Query Job Status Using the id of the batch inference job, you can query the status by: @@ -111,7 +111,7 @@ The output for a pending job looks like: Once the job completes, the `status` field will change from `in_progress` to `completed`. -## Retrieving the output +## Retrieving the Output The output of the batch inference job can be obtained by: ```sh From 5f1c2d4e5e0a2b7a88d635fbb4d401ca212df106 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 13:26:00 +0800 Subject: [PATCH 39/84] Update 04_requesting_an_allocation.mdx --- .../03_Slurm_Accounts/04_requesting_an_allocation.mdx | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/hpc/01_getting_started/03_Slurm_Accounts/04_requesting_an_allocation.mdx b/docs/hpc/01_getting_started/03_Slurm_Accounts/04_requesting_an_allocation.mdx index cd6da4ffc4..054745564e 100644 --- a/docs/hpc/01_getting_started/03_Slurm_Accounts/04_requesting_an_allocation.mdx +++ b/docs/hpc/01_getting_started/03_Slurm_Accounts/04_requesting_an_allocation.mdx @@ -1,7 +1,7 @@ import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; -# Managing allocations for your project +# Managing Allocations for Your Project :::info[VPN Needed] You need to be connected to [NYU VPN](https://www.nyu.edu/life/information-technology/infrastructure/network-services/vpn.html) to access the HPC project management portal. @@ -13,10 +13,10 @@ Login to HPC projects portal at [projects.hpc.nyu.edu](https://projects.hpc.nyu. From the list of project, click on the project you'd like to submit an allocation request for and you'll see the current details: !["PI_One Project" section](../static/PI_one_project.png) -## Current allocations +## Current Allocations Scroll down and you'll reach the Allocations section. This section lists all the allocations associated with this project. All allocations have SLURM accounts associated with them. Allocations that are "Active" will allow you to submit jobs using the SLURM account associated with it. -## Requesting new allocations +## Requesting New Allocations If you scroll down, you'll see "+Request Resource Allocation" button. !["PI request allocation" section](../static/PI_request_allocation.png) From 361a43acc9b8f8ea2f16780db56e85afdc743183 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 13:34:50 +0800 Subject: [PATCH 40/84] Update 01_getting_and_renewing_an_account.mdx --- .../02_HPC_Accounts/01_getting_and_renewing_an_account.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hpc/01_getting_started/02_HPC_Accounts/01_getting_and_renewing_an_account.mdx b/docs/hpc/01_getting_started/02_HPC_Accounts/01_getting_and_renewing_an_account.mdx index ff2cc7bc7b..ae46e1e238 100644 --- a/docs/hpc/01_getting_started/02_HPC_Accounts/01_getting_and_renewing_an_account.mdx +++ b/docs/hpc/01_getting_started/02_HPC_Accounts/01_getting_and_renewing_an_account.mdx @@ -130,7 +130,7 @@ Linux clients are not officially supported, however we were able to successfully sudo openconnect -b vpn.nyu.edu ``` - When prompted follow the instructions and provide your netID, password, and authenticate with ('push', 'phone1' or 'sms') - This method was tested on few Linux distributions and settings however is not guaranteeed to work in future. + This method was tested on few Linux distributions and settings however is not guaranteed to work in future. From 8aa6a808a26d12a6330805c5f852d6b765808c6d Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 13:39:36 +0800 Subject: [PATCH 41/84] Update 01_connecting_to_hpc.mdx --- docs/hpc/02_connecting_to_hpc/01_connecting_to_hpc.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/hpc/02_connecting_to_hpc/01_connecting_to_hpc.mdx b/docs/hpc/02_connecting_to_hpc/01_connecting_to_hpc.mdx index b4a6428c5a..9f9560c943 100644 --- a/docs/hpc/02_connecting_to_hpc/01_connecting_to_hpc.mdx +++ b/docs/hpc/02_connecting_to_hpc/01_connecting_to_hpc.mdx @@ -35,7 +35,7 @@ Host torch login.torch.hpc.nyu.edu ``` :::warning[No SSH Keys on Torch] -SSH keys are not supported on Torch due to increased security resrtictions. +SSH keys are not supported on Torch due to increased security restrictions. ::: @@ -99,7 +99,7 @@ You can access the ssh client via Powershell, either by installing [Windows Term Alternatively, you can install WSL2, and then install Ubuntu or other Linux distribution (for example, from Microsoft Store). You will have a fully functional Ubuntu with terminal and can connect to cluster using instructions provided above for Linux/Mac users. Instructions on WSL installation can be found here: [https://docs.microsoft.com/en-us/windows/wsl/install-win10][wsl installation link] -With [Windowns Terminal](https://apps.microsoft.com/detail/9n0dx20hk701?hl=en-US&gl=US), you can access both the Linux WSL2 shell and the Windows Powershell from the same application. +With [Windows Terminal](https://apps.microsoft.com/detail/9n0dx20hk701?hl=en-US&gl=US), you can access both the Linux WSL2 shell and the Windows Powershell from the same application. :::tip - If you are using WSL 2 (Windows subsystem for Linux 2), you may not be able to access internet when Cisco AnyConnect VPN, installed from exe file, is activated. A potential solution: uninstall Cisco AnyConnect and install AnyConnect using Microsoft Store, and then setup new VPN connection using settings described on [IT webpage][install vpn on windows link]. From 7f23460e989dbe91779f2070ec423d609d6ce230 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 13:44:00 +0800 Subject: [PATCH 42/84] Update 05_best_practices.mdx --- docs/hpc/03_storage/05_best_practices.mdx | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/hpc/03_storage/05_best_practices.mdx b/docs/hpc/03_storage/05_best_practices.mdx index 0310efe5cf..98d476e77a 100644 --- a/docs/hpc/03_storage/05_best_practices.mdx +++ b/docs/hpc/03_storage/05_best_practices.mdx @@ -1,5 +1,5 @@ # Best Practices on HPC Storage -## User Quota Limits and the myquota command +## User Quota Limits and the Myquota Command All users have quote limits set on HPC fie systems. There are several types of quota limits, such as limits on the amount of disk space (disk quota), number of files (inode quota) etc. The default user quota limits on HPC file systems are listed [on our Data Management page](./01_intro_and_data_management.mdx#hpc-storage-comparison-table). :::warning[Home directory inode quotas] @@ -8,17 +8,17 @@ _One of the common issues users report is running out of inodes in their home di Users can check their current utilization of quota using the myquota command. The myquota command provides a report of the current quota limits on mounted file systems, the user's quota utilization, as well as the percentage of quota utilization. For details, please see [myquota](../06_tools_and_software/08_utils.mdx#myquota) -## Large number of small files +## Large number of Small Files In case your dataset or workflow requires to use large number of small files, this can create a bottleneck due to read/write rates. Please refer to [our page on working with a large number of files](./06_large_number_of_small_files.md) to learn about some of the options we recommend to consider. -## Installing Python packages +## Installing Python Packages :::warning Your home directory is limited to a relatively small number of inodes (30,000). Creating conda/python environments in you home directory, this can eat easily exhaust your inode quota. ::: Please review the [Package Management section](../06_tools_and_software/01_intro.md#package-management-for-r-python--julia-and-conda-in-general) of the [Torch Software Page](../06_tools_and_software/01_intro.md). -## Troubleshooting quota issues +## Troubleshooting Quota Issues :::warning Being over your quota can cause problems logging into OOD and other issues. If you find that you're suddenly unable to log in, one of the first things you should check is your quota with the [`myquota`](../06_tools_and_software/08_utils.mdx#myquota) command. @@ -35,7 +35,7 @@ and here's how you can see which ones most contribute to your disk usage: du -h -s -- * .[!.]* ..?* 2>/dev/null | sort -hr | head -n 5 ``` -### Moving common dot directories from `/home` to `/scratch` +### Moving Common Dot Directories from `/home` to `/scratch` As mentioned elsewhere in our documentation, a problem we often see for our users is that of filling their `/home` directories by using common tools like VSCode, Cursor, etc. These applications put a dot directory in your `/home` directory and it can quickly overwhelm your quota. From a87a6cbe4277a15a9b48ea2e98cb56b561be2016 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 13:55:35 +0800 Subject: [PATCH 43/84] Update 08_sharing_data_on_hpc.mdx --- docs/hpc/03_storage/08_sharing_data_on_hpc.mdx | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/hpc/03_storage/08_sharing_data_on_hpc.mdx b/docs/hpc/03_storage/08_sharing_data_on_hpc.mdx index 0ccf2e68fa..98433b4268 100644 --- a/docs/hpc/03_storage/08_sharing_data_on_hpc.mdx +++ b/docs/hpc/03_storage/08_sharing_data_on_hpc.mdx @@ -30,14 +30,14 @@ The following commands are available: - `nfs4_getfacl` to view ACLs with the usage described in the following examples. -#### Give someone access to read a particular file +#### Give Someone Access to Read a Particular File Append the ACL for that file by adding an ACE via ```sh nfs4_setfacl -a "A::NetID:R" filename ``` where the `-a` flag signifies "append". Since inheritance flags are only applicable to directories and the principal is not a group, no flags are needed. -#### Show current access properties +#### Show Current Access Properties Create an empty file and view the default ACEs it: ```sh @@ -75,7 +75,7 @@ The above grants kevin basic read/execute to /scratch susie, which is required t nfs4_setfacl -R -a "A:df:kevin:RWX" /scratch/susie/share ``` -The above would allow kevin to have have read, write, and execute access for *all directories* and files inside `/scratch/susie/share` +The above would allow kevin to have read, write, and execute access for *all directories* and files inside `/scratch/susie/share` ## Linux Groups for Group Permissions @@ -140,4 +140,4 @@ See instructions in the [Globus documentation](https://docs.globus.org/guides/tu Please be aware that it is possible when you select to create a collection that you will see an error message stating that you failed to authenticate. Please just select the 'Continue' button and you should be taken through the reauthentication process before proceeding to create the collection. Let us know if you have any questions or problems at [hpc@nyu.edu](mailto:hpc@nyu.edu). -::: \ No newline at end of file +::: From 3e702cf21aa4c2bbf60d233e0d983679a1977146 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 14:05:44 +0800 Subject: [PATCH 44/84] Update 01_intro.md --- docs/hpc/06_tools_and_software/01_intro.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/hpc/06_tools_and_software/01_intro.md b/docs/hpc/06_tools_and_software/01_intro.md index 86461a0eb0..66b45deacf 100644 --- a/docs/hpc/06_tools_and_software/01_intro.md +++ b/docs/hpc/06_tools_and_software/01_intro.md @@ -1,9 +1,9 @@ -# Start here! +# Start Here! We encourage you to setup your own computational environment on Torch and to assist you in doing so, we allow you to run [Apptainer](../07_containers/01_intro.md) (formerly known as Singularity) containers, we manage licensed software suites and offer extensive documentation, training and support. :::tip -We strongly advise that you setup your own computational enviromments via Apptainer containers and overlay files. Detailed documentation is available in the [containers section](../07_containers/01_intro.md). +We strongly advise that you setup your own computational environments via Apptainer containers and overlay files. Detailed documentation is available in the [containers section](../07_containers/01_intro.md). ::: ## Package Management for R, Python, & Julia, and Conda in general @@ -11,7 +11,7 @@ We strongly advise that you setup your own computational enviromments via Apptai - [Using virtual environments for Python](../06_tools_and_software/04_python_packages_with_virtual_environments.mdx) - [Managing R packages with renv](../06_tools_and_software/05_r_packages_with_renv.mdx) -## Examples of software usage on Torch +## Examples of Software Usage on Torch Examples can be found under `/scratch/work/public/examples/` and include the following | | | | From 271b6159e3701641769d0d27edc2bf1d4a1a9ce5 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 14:09:51 +0800 Subject: [PATCH 45/84] Update 04_python_packages_with_virtual_environments.mdx --- ...hon_packages_with_virtual_environments.mdx | 22 +++++++++---------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/docs/hpc/06_tools_and_software/04_python_packages_with_virtual_environments.mdx b/docs/hpc/06_tools_and_software/04_python_packages_with_virtual_environments.mdx index b8959a0857..6a0e6b06f6 100644 --- a/docs/hpc/06_tools_and_software/04_python_packages_with_virtual_environments.mdx +++ b/docs/hpc/06_tools_and_software/04_python_packages_with_virtual_environments.mdx @@ -8,18 +8,18 @@ This page describes how you can create virtual environments without containers i In order to be able to install new Python packages and make your work reproducible, please use virtual environments. There is more than one way to create a private environment in Python. -## Create project directory and load Python module +## Create Project Directory and Load Python Module ```sh -## Find python version you need +## Find Python Version You Need module avail python -## created directory for your project and cd there +## Created Directory for Your Project and cd There mkdir /scratch/$USER/my_project cd /scratch/$USER/my_project -## load python module (different versions available) +## Load Python Module (Different Versions Available) module load python/intel/3.8.6 ``` -## Automatic deletion of your files +## Automatic Deletion of Your Files This page describes the installation of packages on /scratch. One has to remember, though, that files stored in the HPC scratch file system are subject to the HPC Scratch old file purging policy:
@@ -37,10 +37,10 @@ Thus you can consider the following options: :::
-## Create virtual environment +## Create Virtual Environment It is advisable to create private environment inside the project directory. This boosts reproducibility and does not use space in `/home/$USER`: ```sh -## created directory for your project and cd there +## Created Directory for Your Project and cd There mkdir /scratch/$USER/my_project cd /scratch/$USER/my_project ``` @@ -49,12 +49,12 @@ cd /scratch/$USER/my_project [`virtualenv`](https://virtualenv.pypa.io/en/latest/) is a tool to create isolated Python environments. Since Python 3.3, a subset of it has been integrated into the standard library under the venv module. You can create new virtual environment in two ways: - empty ```sh -## Create an EMPTY virtual environment +## Create an EMPTY Virtual Environment virtualenv venv ``` - inherit all packages from those installed on HPC already (and available in PATH after you load python module) ```sh -## Create an virtual environment that inherits system packages +## Create an Virtual Environment that Inherits System Packages virtualenv venv --system-site-packages ``` @@ -73,11 +73,11 @@ python -m venv venv ``` - inherit all packages from those installed on HPC already (and available in PATH after you load python module) ```sh -## Inhering all packages +## Inhering All Packages python -m venv venv --system-site-packages ``` -## Install packages. Keep things reproducible +## Install Packages. Keep Things Reproducible ```sh ## activate source venv/bin/activate From 65def1b1986d06ecd8a8c9c3dae42a10cbfdb951 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 14:13:20 +0800 Subject: [PATCH 46/84] Update 07_sqlite_handling_large_structured_data.md --- .../07_sqlite_handling_large_structured_data.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/hpc/06_tools_and_software/07_sqlite_handling_large_structured_data.md b/docs/hpc/06_tools_and_software/07_sqlite_handling_large_structured_data.md index 6f50a45e40..a9f1a61799 100644 --- a/docs/hpc/06_tools_and_software/07_sqlite_handling_large_structured_data.md +++ b/docs/hpc/06_tools_and_software/07_sqlite_handling_large_structured_data.md @@ -7,7 +7,7 @@ Storing your data in the [SQLite](https://www.sqlite.org/index.html) format allo > > -- [SQLite website](https://www.sqlite.org/index.html): -### Some use-cases +### Some Use-cases - You think you need MySQL, PostreSQL, etc for your ML project. Usually you don't - You have to deal with hundreds of GB of table-structured data (or larger) and your script (for whatever reason) can't be made parallel. @@ -30,7 +30,7 @@ It is better in this case to request smaller amount of RAM and read data (effici - You can have multiple threads accessing an SQLite database simultaneously (for read operations. Writing is more tricky) - Merging/Joining datasets on disk -### Major benefits of SQLite compared to MySQL (PostgreSQL, etc) +### Major Benefits of SQLite Compared to MySQL (PostgreSQL, etc) - You control your own data (sqlite file). You don't depend on any service like MySQL - You can copy a file to your own laptop and work with it @@ -41,7 +41,7 @@ It is better in this case to request smaller amount of RAM and read data (effici - SQLite has some limitations in terms of concurrency, which usually don't apply for typical ML/AI jobs. - See [Four Different Ways To Handle SQLite Concurrency](https://medium.com/@gwendal.roue/four-different-ways-to-handle-sqlite-concurrency-db3bcc74d00e) for more information. -## Command line (CLI) example +## Command Line (CLI) Example Create environment ```sh $ mkdir projects/sqlite-test @@ -70,7 +70,7 @@ hello!|10 goodbye|20 ``` -## R example +## R Example ### Install Here we use conda, as a great way to keep everything isolated and reproducible. @@ -151,7 +151,7 @@ Save as feather feather::write_feather(df_temp, "my_data.feather") ``` -### Alternative: read csv file to SQLite directly +### Alternative: Read csv File to SQLite Directly If you already have a large csv file on disk, and you don't want to read it to RAM, you can read it to SQLite file directly ```R From 035bd064c985bd34a42859f809b0dc0126be691d Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 14:15:39 +0800 Subject: [PATCH 47/84] Update 05_r_packages_with_renv.mdx --- .../05_r_packages_with_renv.mdx | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/hpc/06_tools_and_software/05_r_packages_with_renv.mdx b/docs/hpc/06_tools_and_software/05_r_packages_with_renv.mdx index 87e5917519..7a283d50bd 100644 --- a/docs/hpc/06_tools_and_software/05_r_packages_with_renv.mdx +++ b/docs/hpc/06_tools_and_software/05_r_packages_with_renv.mdx @@ -1,6 +1,6 @@ # R Packages with renv -## Is this needed: +## Is this Needed: :::tip If you would like to use conda with `renv` you need to add following steps: @@ -42,7 +42,7 @@ Thus you can consider the following options: -### Cache directory setup +### Cache Directory Setup By default, renv will cache package installation files to your home directory (most likely either in `~/.local/share/renv` or `~/.cache/R/renv/` or something similar). To avoid filling up your home directory, we advise to set up path to alternative cache directory (otherwise your home directory may fill up quickly) @@ -51,7 +51,7 @@ To avoid filling up your home directory, we advise to set up path to alternative ```sh mkdir -p /scratch/$USER/.cache/R/renv ``` -- Create a file in you project directory named `.Renviron` and put the following in in the file. It is the R project directory (`/scratch/$USER/projects/project1`) in this example. +- Create a file in you project directory named `.Renviron` and put the following in the file. It is the R project directory (`/scratch/$USER/projects/project1`) in this example. ```sh RENV_PATHS_ROOT=/scratch//.cache/R/renv ``` @@ -87,7 +87,7 @@ renv::paths$cache() #[1] "/home/$USER/.cache/R/renv/cache/v5/R-4.1/x86_64-pc-linux-gnu" ``` -### Add/remove, etc. packages +### Add/remove, etc. Packages Install a package, such as reshape2. Below we can see it is not yet installed and then install it. ```R R @@ -136,7 +136,7 @@ No id variables; using all as measure variables [1] "hello again" ``` -### Clean up +### Clean Up Keep only the packages that you use in this particular project (not all the packages available on the system) ```R R # launch R @@ -201,6 +201,6 @@ In order to have your work reproducible by you and/or others: ### Migrating from Packrat The `renv` package has replaced the now deprecated `Packrat` package. The `renv::migrate()` function makes it possible to migrate projects from `Packrat` to `renv`. See the ?migrate documentation for more details. In essence, calling `renv::migrate("")` will be enough to migrate the `Packrat` library and lockfile such that they can then be used by `renv`. -## Useful links +## Useful Links - [Introduction to `renv`](https://rstudio.github.io/renv/articles/renv.html) From c21f0fc3f2fa6778222b55fc1dd408444ae0e152 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 14:21:19 +0800 Subject: [PATCH 48/84] Update 02_licensed_software.md --- docs/hpc/06_tools_and_software/02_licensed_software.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/hpc/06_tools_and_software/02_licensed_software.md b/docs/hpc/06_tools_and_software/02_licensed_software.md index 1bf5df6c94..0da068adcd 100644 --- a/docs/hpc/06_tools_and_software/02_licensed_software.md +++ b/docs/hpc/06_tools_and_software/02_licensed_software.md @@ -157,7 +157,7 @@ SAS is a software package which enables programmers to perform many tasks, inclu There are licenses for 2 CPUs on the HPC Cluster. -### Running a parallel SAS job on HPC cluster (Torch): +### Running a Parallel SAS Job on HPC Cluster (Torch): To submit a SAS job for running on multiple processing elements, follow below steps: @@ -180,7 +180,7 @@ sbatch run-sas.sbatch ## MATLAB [MATLAB](https://www.mathworks.com/products/matlab.html) is a technical computing environment for high performance numeric computation and visualization. MATLAB integrates numerical analysis, matrix computation, signal processing, and graphics in an easy to use environment without using traditional programming. -### MATLAB on personal computers and laptops +### MATLAB on Personal Computers and Laptops NYU has a Total Academic Headcount (TAH) license which provides campus-wide access to MATLAB, Simulink, and a variety of add-on products. All faculty, researchers, and students (on any NYU campus) can use MATLAB on their personal computers and laptops and may go to the following site to download the NYU site license software free of charge. @@ -199,7 +199,7 @@ Mathworks has provided a [Torch Matlab User Guide](https://drive.google.com/file ## STATA Stata is a command and menu-driven software package for statistical analysis. It is available for Windows, Mac, and Linux operating systems. Most of its users work in research. Stata's capabilities include data management, statistical analysis, graphics, simulations, regression and custom programming. -### Running a parallel STATA job on HPC cluster (Torch): +### Running a Parallel STATA Job on HPC Cluster (Torch): To submit a STATA job for running on multiple processing elements, follow below steps. @@ -223,7 +223,7 @@ Gaussian uses basic quantum mechanic electronic structure programs. This softwar The NYU Gaussian license only covers PIs at the Washington Square Park campus. We will grant access to you after verifying your WSP affiliation. For access, please email [hpc@nyu.edu](mailto:hpc.nyu.edu). -### Running a parallel Gaussian job on HPC cluster (Torch): +### Running a Parallel Gaussian Job on HPC Cluster (Torch): To submit a Gaussian job for running on multiple processing elements, follow below steps. @@ -242,7 +242,7 @@ sbatch run-gaussian.sbatch ## Knitro Knitro is a commercial software package for solving large scale mathematical optimization problems. Knitro is specialized for nonlinear optimization, but also solves linear programming problems, quadratic programming problems, systems of nonlinear equations, and problems with equilibrium constraints. The unknowns in these problems must be continuous variables in continuous functions; however, functions can be convex or nonconvex. Knitro computes a numerical solution to the problem—it does not find a symbolic mathematical solution. Knitro versions 9.0.1 and 10.1.1 are available. -### Running a parallel Knitro job on HPC cluster (Torch): +### Running a Parallel Knitro Job on HPC Cluster (Torch): To submit a Knitro job for running on multiple processing elements, follow below steps. From 4d8a3da9ecc178560192c6cdf93acd5ada33e3c5 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 14:21:34 +0800 Subject: [PATCH 49/84] Correct header casing for Apptainer section --- docs/hpc/07_containers/01_intro.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hpc/07_containers/01_intro.md b/docs/hpc/07_containers/01_intro.md index 116fe28fca..1eeeb1ed3b 100644 --- a/docs/hpc/07_containers/01_intro.md +++ b/docs/hpc/07_containers/01_intro.md @@ -1,6 +1,6 @@ # Custom Applications with Containers -## What is Apptainer? +## What Is Apptainer? Apptainer is a container based Linux kernel workspace that works just like Docker. You can run pre-built programs in containers without having to worry about the pre-install environment. You can even run Docker containers with Apptainer. Please see the [Apptainer and Docker](https://apptainer.org/docs/user/main/docker_and_oci.html) documentation by Syslabs for details about all the ways Apptainer supports Docker. For a detailed introduction on Apptainer, visit their [official site](https://apptainer.org/documentation/). Apptainer is the continuation of the Singularity project. When we transition to the new Torch cluster you will only see reference to Apptainer. The reason for this is that there were two container projects with the name Singularity. The original free and open-source (FOSS) project and a closed-source corporate fork. The FOSS version has renamed itself to Apptainer to avoid confusion and this is the version we use. From 343eed30dec694d763e7ed1fadb10986da080950 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 14:26:13 +0800 Subject: [PATCH 50/84] Update 03_singularity_with_conda.md --- docs/hpc/07_containers/03_singularity_with_conda.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/hpc/07_containers/03_singularity_with_conda.md b/docs/hpc/07_containers/03_singularity_with_conda.md index e48ac439a6..5c70610cf4 100644 --- a/docs/hpc/07_containers/03_singularity_with_conda.md +++ b/docs/hpc/07_containers/03_singularity_with_conda.md @@ -276,7 +276,7 @@ cat slurm-3752662.out # True ``` -### Optional: Convert ext3 to a Compressed, Read-only Squashfs Filesystem +### Optional: Convert `ext3` to a Compressed, Read-only `squashfs` Filesystem Singularity images can be compressed into read-only squashfs filesystems to conserve space in your environment. Use the following steps to convert your ext3 Singularity image into a smaller squashfs filesystem. ```sh srun -N1 -c4 singularity exec --overlay my_pytorch.ext3:ro /share/apps/images/centos-8.2.2004.sif mksquashfs /ext3 /scratch//pytorch-example/my_pytorch.sqf -keep-as-directory -processors 4 -noappend @@ -291,7 +291,7 @@ ls -ltrsh my_pytorch.* Notice that it saves over 3GB of storage in this case, though your results may vary. -#### Use a squashFS Image for Running Jobs +#### Use a `squashFS` Image for Running Jobs You can use squashFS images similarly to the ext3 images: ```sh @@ -301,7 +301,7 @@ singularity exec --overlay /scratch//pytorch-example/my_pytorch.sqf:ro /s #example output: 2.6.0+cu124 ``` -#### Adding Packages to a Full ext3 or squashFS Image +#### Adding Packages to a Full `ext3` or `squashFS` Image If the first ext3 overlay image runs out of space or you are using a squashFS conda environment, but need to install a new package inside, please copy another writable ext3 overlay image to work together. From 7e46f010a6e57fe9a80fc96727e77bc49d158477 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 14:39:07 +0800 Subject: [PATCH 51/84] Update 03_pytorch_dpp.md --- docs/hpc/08_ml_ai_hpc/03_pytorch_dpp.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/hpc/08_ml_ai_hpc/03_pytorch_dpp.md b/docs/hpc/08_ml_ai_hpc/03_pytorch_dpp.md index c5b44cb961..649228cdba 100644 --- a/docs/hpc/08_ml_ai_hpc/03_pytorch_dpp.md +++ b/docs/hpc/08_ml_ai_hpc/03_pytorch_dpp.md @@ -174,11 +174,11 @@ Here is one possibility: export MASTER_PORT=$(expr 10000 + $(echo -n $SLURM_JOBID | tail -c 4) + $SLURM_ARRAY_TASK_ID) ``` -### `--ntasks-per-node` versus `--ntasks` +### `--ntasks-per-node` Versus `--ntasks` Be sure to use `--ntasks-per-node` and not `--ntasks` in your Slurm script. -## What is `local_rank`? +## What Is `local_rank`? The indices of the GPUs on each node of your Slurm allocation begin at 0 and end at N - 1, where N is the total number of GPUs in your allocation on each node. Consider the case of 2 nodes and 8 tasks with 4 GPUs per node. The process ranks will be 0, 1, 2, 3 on the first node and 4, 5, 6, 7 on the second node while the GPU indices will be 0, 1, 2, 3 on the first and 0, 1, 2, 3 on the second. Thus, one cannot make calls such as `data.to(rank)` since this will fail on the second node where there is a mismatch between the process ranks and the GPU indices. To deal with this a local rank is introduced: From 604c5db10223ec3b9b71697d678ef7a96106b840 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 14:45:39 +0800 Subject: [PATCH 52/84] Update 06_conda_environments.mdx --- .../06_conda_environments.mdx | 24 +++++++++---------- 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/docs/hpc/06_tools_and_software/06_conda_environments.mdx b/docs/hpc/06_tools_and_software/06_conda_environments.mdx index 00290589f6..4aaa514ddf 100644 --- a/docs/hpc/06_tools_and_software/06_conda_environments.mdx +++ b/docs/hpc/06_tools_and_software/06_conda_environments.mdx @@ -5,7 +5,7 @@ This page describes how you can create conda environments without containers in ::: -## What is Conda? +## What Is Conda? Package, dependency and environment management for any language—Python, R, Ruby, Lua, Scala, Java, JavaScript, C/ C++, FORTRAN, and more. Please find more information at [the official documentation page](https://docs.conda.io/en/latest/) Conda provides a great way to install packages that are already compiled, so you don't need to go over the long compilation process. If a package you need is not available, you can install it (and compile it when needed) using `pip` (Python) or `install.packages` (R). @@ -15,7 +15,7 @@ Conda provides a great way to install packages that are already compiled, so you One of the ways to ensure the reproducibility of your results is to have an independent conda environment in the directory of each project (one of the options shown below). This will also keep conda environment files away from your `/home/$USER` directory. ::: -## Advantages/disadvantages of using Conda +## Advantages/disadvantages of Using Conda ### Advantages - A lot of pre-compiled packages (fast and easy to install) @@ -63,7 +63,7 @@ NYU HPC recommends using prefix environments as a best practice because it makes ::: ## Prefix Conda Environments -You can store your program/project in `/scratch` and keep the conda environment with it by using the `-p` parameter. This will keep all the files inside the project's directory, instead of putting in in your `/home/$USER`.
+You can store your program/project in `/scratch` and keep the conda environment with it by using the `-p` parameter. This will keep all the files inside the project's directory, instead of putting in your `/home/$USER`.
Here are details about how to do this for Python and R projects: ### Python @@ -203,7 +203,7 @@ Don't forget to change `NetID` above to your `NetID`. You should now see that the `package cache` and `envs directories` entries have changed to your `/scratch` space when you run `conda info` and packages and environments will be saved on your `/scratch` space for all named conda environments. ## Reproducibility -### Packages installed only using conda +### Packages Installed Only Using Conda Save a list of packages (so you are able to report the environment in a publication, and to restore/reproduce the environment on another machine at any time) @@ -218,7 +218,7 @@ This will not list packages installed by `pip` or `install.packages()` ::: -### Packages installed using conda and `pip` (Python) +### Packages Installed Using Conda and `pip` (Python) In this case you can use: ```bash @@ -239,7 +239,7 @@ However, this may fail if your conda environment is created as a sub-directory o ::: :::: -### Packages installed using `install.packages` (R) +### Packages Installed Using `install.packages` (R) The command `conda list --export` will not include packages installed by `install.packages`. So, only use `conda install` to install R and use `renv` to maintain information about packages installed by `install.packages`. @@ -247,11 +247,11 @@ The command `conda list --export` will not include packages installed by `instal Please see details of using `renv` with conda for reproducibilty on [R packages with `renv`](./05_r_packages_with_renv.mdx). -## Use conda env in a batch script +## Use Conda `env` in a Batch Script The part of the batch script that will call the command should look like: ### Python -#### Single node +#### Single Node ```bash #!/bin/bash #SBATCH --job-name=test @@ -272,7 +272,7 @@ python test.py ``` You'll need to replace `` with the full path to your penv directory. It's probably something like `/scratch/NetID/conda_tests/penv` -#### Multiple nodes, using MPI +#### Multiple Nodes, Using MPI ```sh #!/bin/bash #SBATCH --job-name=test @@ -294,7 +294,7 @@ python test.py" ``` Again, you'll need to replace `` above with the full path to your penv directory. -### R (conda packages only) +### R (Conda Packages Only) ```bash #!/bin/bash #SBATCH --job-name=test @@ -314,7 +314,7 @@ Rscript r_script.R ``` You'll again need to replace `` above with the full path to your renv directory. -#### Multiple nodes, using MPI +#### Multiple Nodes, Using MPI ```sh #!/bin/bash #SBATCH --job-name=test @@ -334,7 +334,7 @@ Rscript test.R ``` You'll again need to replace `` above with the full path to your renv directory. -### R (conda with renv combination) +### R (Conda with `renv` Combination) In this case, when you use sbatch you would activate conda in sbatch script, and R script will pickup packages installed in renv ```sh From c0a5db0a4e79209699e366b29cc2095aa95360db Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 14:47:21 +0800 Subject: [PATCH 53/84] Update 01_ood_intro.mdx --- docs/hpc/09_ood/01_ood_intro.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/hpc/09_ood/01_ood_intro.mdx b/docs/hpc/09_ood/01_ood_intro.mdx index ececdd6c91..a7e57a3c56 100644 --- a/docs/hpc/09_ood/01_ood_intro.mdx +++ b/docs/hpc/09_ood/01_ood_intro.mdx @@ -34,7 +34,7 @@ You must be on the NYU VPN to be able to access the Open On Demand gateway! For ### Connections to Open OnDemand -A common issue that can occur is receiving an error that the Open OnDemand page cannot be reached. Sometimes this can indicate that the service is down, but often this is an issue with the the local browser cache. You can test this by opening a private browser window and seeing if https://ood.torch.hpc.nyu.edu will load. If it does, try deleting the cache for https://ood.torch.hpc.nyu.edu in your browser history to resolve this issue. +A common issue that can occur is receiving an error that the Open OnDemand page cannot be reached. Sometimes this can indicate that the service is down, but often this is an issue with the local browser cache. You can test this by opening a private browser window and seeing if https://ood.torch.hpc.nyu.edu will load. If it does, try deleting the cache for https://ood.torch.hpc.nyu.edu in your browser history to resolve this issue. In Chrome, this can be done by navigating to this page in your settings: @@ -65,6 +65,6 @@ Just click on the `Session ID` link and a tab will open with the contents of the `output.log` is probably the most informative for debugging. Simply click on `output.log` and the contents will be displayed. -#### From terminal +#### From Terminal If your session is no longer visible from within OOD you may still be able to find your logs via the terminal. Simply [log into Torch](../02_connecting_to_hpc/01_connecting_to_hpc.mdx) and `cd` to `/home/$USER/ondemand/data/sys/dashboard/batch_connect/sys/` and then `cd` into the directory for the app that you're interested in. You should find the file `output.log` there. From dc18d45dd7214dba92e830f398816e5e4371dab1 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 14:49:28 +0800 Subject: [PATCH 54/84] Update 03_vLLM.md --- docs/hpc/08_ml_ai_hpc/08_LLM inference/03_vLLM.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/hpc/08_ml_ai_hpc/08_LLM inference/03_vLLM.md b/docs/hpc/08_ml_ai_hpc/08_LLM inference/03_vLLM.md index b4264e3837..44bb477641 100644 --- a/docs/hpc/08_ml_ai_hpc/08_LLM inference/03_vLLM.md +++ b/docs/hpc/08_ml_ai_hpc/08_LLM inference/03_vLLM.md @@ -1,6 +1,6 @@ # High-performance LLM Inference with `vLLM` -## What is vLLM? +## What Is vLLM? [`vLLM`](https://docs.vllm.ai/en/latest/) is an easy-to-use library for LLM inference and serving which support a wide variety of models with optimized kernels ensuring optimal utilization of GPUs. ## Why `vLLM`? @@ -24,12 +24,12 @@ Concurrency: 8 Max requests: 64 -## vLLM Installation Instructions +## `vLLM` Installation Instructions Create a `vLLM` directory in your /scratch directory, then install the vLLM image: ``` apptainer pull docker://vllm/vllm-openai:latest ``` -### Avoid Filling up your `$HOME` Directory +### Avoid Filling Up Your `$HOME` Directory To avoid exceeding your `$HOME` quota (50GB) and inode limits (30,000 files), you should redirect `vLLM`'s cache and Hugging Face's model downloads to your scratch space: ```sh export HF_HOME=/scratch/$USER/hf_cache @@ -45,7 +45,7 @@ echo "export VLLM_CACHE_ROOT=/scratch/\$USER/vllm_cache" >> ~/.bashrc Files on `$SCRATCH` are not backed up and will be deleted after 60 days of inactivity. Always keep your source code and .slurm scripts in `$HOME`! ::: -## Run vLLM +## Run `vLLM` ### Online Serving (OpenAI-Compatible API) `vLLM` implements the OpenAI API protocol, allowing it to be a drop-in replacement for applications using OpenAI's services. By default, it starts the server at `http://localhost:8000`. You can specify the address with `--host` and `--port` arguments. **In Terminal 1:** From c14816da12eda5214a4a663c41d7b6d4f77ca2bb Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 14:50:37 +0800 Subject: [PATCH 55/84] Fix typos and improve formatting in Desktop.md --- docs/hpc/09_ood/04_Desktop.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/hpc/09_ood/04_Desktop.mdx b/docs/hpc/09_ood/04_Desktop.mdx index b91d015a62..bf8925553e 100644 --- a/docs/hpc/09_ood/04_Desktop.mdx +++ b/docs/hpc/09_ood/04_Desktop.mdx @@ -15,7 +15,7 @@ You can select the number of cores, amount of memory, GPU type (if any), number ![OOD Desktop Configuration](./static/ood_desktop_config.png) -## Desktop running in OOD +## Desktop Running in OOD After you hit the `Launch` button you'll have to wait for the scheduler to find node(s) for you to run on: ![OOD Desktop in queue](./static/ood_desktop_in_queue.png) @@ -24,7 +24,7 @@ Then you'll have a short wait for the Desktop itself to start up.
Once that happens you'll get one last page that will give you links to: - open a terminal window on the compute node your Desktop session is running on - go to the directory associated with your Session ID that stores output, config and other related files for your session -- make changes to compression and image qualtiy +- make changes to compression and image quality - get a link that you can share that will allow others to view your Desktop session ![Pre-launch Desktop OOD](./static/ood_desktop_prelaunch.png) From 8e517105c0562df37650dc9b69beba5db54ee1cc Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 14:51:18 +0800 Subject: [PATCH 56/84] Update 03_matlab_proxy.mdx --- docs/hpc/09_ood/03_matlab_proxy.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hpc/09_ood/03_matlab_proxy.mdx b/docs/hpc/09_ood/03_matlab_proxy.mdx index 63ea8b8a9a..529f5df35a 100644 --- a/docs/hpc/09_ood/03_matlab_proxy.mdx +++ b/docs/hpc/09_ood/03_matlab_proxy.mdx @@ -39,7 +39,7 @@ ln -s /scratch/$USER/.MathWorks ``` ::: -## Matlab running in OOD +## Matlab Running in OOD After you hit the `Launch` button you'll have to wait for the scheduler to find you node(s) to run on: ![OOD Matlab in queue](./static/ood_matlab_in_queue.png) From a5901eb2a9197460f159a074dafafb501994412a Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 14:52:00 +0800 Subject: [PATCH 57/84] Update 05_RStudio.mdx --- docs/hpc/09_ood/05_RStudio.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hpc/09_ood/05_RStudio.mdx b/docs/hpc/09_ood/05_RStudio.mdx index a840970fbf..9a2f79e6e9 100644 --- a/docs/hpc/09_ood/05_RStudio.mdx +++ b/docs/hpc/09_ood/05_RStudio.mdx @@ -22,7 +22,7 @@ Be careful not to go over your quota for `/home`. You can find your current usa ::: :::: -## RStudio running in OOD +## RStudio Running in OOD After you hit the `Launch` button you'll have to wait for the scheduler to find you node(s) to run on: ![OOD RStudio in queue](./static/ood_rstudio_in_queue.png) From 749903238efc6d50bb59b591fefe72d0903cae56 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 14:53:44 +0800 Subject: [PATCH 58/84] Update section header for Singularity container --- docs/hpc/08_ml_ai_hpc/08_LLM inference/02_run_hf_model.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hpc/08_ml_ai_hpc/08_LLM inference/02_run_hf_model.md b/docs/hpc/08_ml_ai_hpc/08_LLM inference/02_run_hf_model.md index b432027443..f976790139 100644 --- a/docs/hpc/08_ml_ai_hpc/08_LLM inference/02_run_hf_model.md +++ b/docs/hpc/08_ml_ai_hpc/08_LLM inference/02_run_hf_model.md @@ -26,7 +26,7 @@ Some of the following steps can require significant resources, so we'll move to [NetID@cm001 llm_example]$ gunzip overlay-50G-10M.ext3.gz ``` -### Launch Singularity container in Read/write Mode +### Launch Singularity Container in Read/write Mode ```bash [NetID@cm001 llm_example]$ apptainer exec --fakeroot --overlay overlay-50G-10M.ext3:rw /share/apps/images/cuda12.1.1-cudnn8.9.0-devel-ubuntu22.04.2.sif /bin/bash ``` From e74a371f44547e25f7628a6b1e50351a0789b177 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 14:56:27 +0800 Subject: [PATCH 59/84] Correct section header from 'kernel' to 'Kernel' --- docs/hpc/09_ood/02_jupyter_with_conda_singularity.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hpc/09_ood/02_jupyter_with_conda_singularity.mdx b/docs/hpc/09_ood/02_jupyter_with_conda_singularity.mdx index 510ecce4b4..2d5569023c 100644 --- a/docs/hpc/09_ood/02_jupyter_with_conda_singularity.mdx +++ b/docs/hpc/09_ood/02_jupyter_with_conda_singularity.mdx @@ -230,7 +230,7 @@ and select Jupyter Notebook from the Interactive Apps pull-down menu: ![Configure Notebook](./static/configure_notebook.png) -### Select kernel +### Select Kernel Once configured and launched, kernels can be selected in the "New" dropdown or within the notebook under the kernel menu. Please note that your notebook view may look slightly different depending on available directories and environments, as well as if you choose the lab or traditional notebook view. ![Select Kernel](./static/select_kernel.png) From 1d85eb97ba77ae5fb1158de1496440c84138060c Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 14:57:45 +0800 Subject: [PATCH 60/84] Update 07_igv.mdx --- docs/hpc/09_ood/07_igv.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/hpc/09_ood/07_igv.mdx b/docs/hpc/09_ood/07_igv.mdx index be9c05b17e..c63f355439 100644 --- a/docs/hpc/09_ood/07_igv.mdx +++ b/docs/hpc/09_ood/07_igv.mdx @@ -19,14 +19,14 @@ You can select the number or cores, amount of memory, amount of time, and option ![OOD IGV Configuration](./static/ood_igv_config.png) -## IGV running in OOD +## IGV Running in OOD After you hit the `Launch` button you'll have to wait for the scheduler to find you node(s) to run on: ![OOD IGV in queue](./static/ood_igv_in_queue.png) Then you'll have a short wait for IGV itself to start up.
Once that happens you'll get one last form that will allow you to: -- make changes to compression and image qualtiy +- make changes to compression and image quality - open a terminal window on the compute node your IGV session is running on - get a link that you can share that will allow others to view your IGV session From 0b60af3e1e74469f85826bf4bad44082ef688bb3 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 14:58:38 +0800 Subject: [PATCH 61/84] Update 01_intro.mdx --- docs/hpc/12_tutorial_intro_shell_hpc/01_intro.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hpc/12_tutorial_intro_shell_hpc/01_intro.mdx b/docs/hpc/12_tutorial_intro_shell_hpc/01_intro.mdx index 4b07139a6b..595636b07f 100644 --- a/docs/hpc/12_tutorial_intro_shell_hpc/01_intro.mdx +++ b/docs/hpc/12_tutorial_intro_shell_hpc/01_intro.mdx @@ -58,7 +58,7 @@ Interacting with the shell is done via a command line interface (CLI) on most HP Learning to use Bash or any other shell sometimes feels more like programming than like using a mouse. Commands are terse (often only a couple of characters long), their names are frequently cryptic, and their output is lines of text rather than something visual like a graph. However, using a command line interface can be extremely powerful, and learning how to use one will allow you to reap the benefits described above. -### The rest of this lesson +### The Rest of this Lesson The only way to use these types of resources is by learning to use the command line. This introduction to HPC systems has two parts: From 2595e8f48217d8c68af73afb0b6a0c941112b059 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 14:59:31 +0800 Subject: [PATCH 62/84] Update 03_moving_looking.mdx --- docs/hpc/12_tutorial_intro_shell_hpc/03_moving_looking.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/hpc/12_tutorial_intro_shell_hpc/03_moving_looking.mdx b/docs/hpc/12_tutorial_intro_shell_hpc/03_moving_looking.mdx index f178b063f4..ef58722788 100644 --- a/docs/hpc/12_tutorial_intro_shell_hpc/03_moving_looking.mdx +++ b/docs/hpc/12_tutorial_intro_shell_hpc/03_moving_looking.mdx @@ -1,4 +1,4 @@ -# Moving around and looking at things +# Moving Around and Looking at Things :::note[Overview] Questions @@ -177,7 +177,7 @@ drwxr-sr-x 2 tc001 4096 Nov 28 09:58 . drwx--S--- 5 tc001 4096 Nov 28 09:58 .. ``` -## Where to go for help? +## Where to Go for Help? How did I know about the `-l` and `-a` options? Is there a manual we can look at for help when we need help? There is a very helpful manual for most UNIX commands: `man` (if you’ve ever heard of a “man page” for something, this is what it is). ```bash $ man ls From e96a31a6626fac723bb10409633763a2c6f636a1 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 15:15:40 +0800 Subject: [PATCH 63/84] Update 06_jbrowse_genome_browser.mdx --- docs/hpc/09_ood/06_jbrowse_genome_browser.mdx | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/hpc/09_ood/06_jbrowse_genome_browser.mdx b/docs/hpc/09_ood/06_jbrowse_genome_browser.mdx index 9450b31af8..b490f8a7a7 100644 --- a/docs/hpc/09_ood/06_jbrowse_genome_browser.mdx +++ b/docs/hpc/09_ood/06_jbrowse_genome_browser.mdx @@ -2,9 +2,9 @@ [JBrowse](https://jbrowse.org/jbrowse1.html) is a web-based genome browser for visualizing genomic features in common file formats, such as variants (VCF), genes (GFF3, BigBed) and gene expression (BigWig), and sequence alignments (BAM, CRAM, and GFF3). -## Getting started +## Getting Started -### 1. Place or link all files into one data folder +### 1. Place or Link All Files into One Data Folder ```bash $ cd /path/to/data_directory $ ls @@ -33,7 +33,7 @@ $ ln -s /scratch/work/cgsb/genomes/.../genes.gff3.tbi genes.gff3.tbi ``` ::: -### 2. Verify data files are in a supported format +### 2. Verify Data Files Are in a Supported Format For the commands listed below, `bgzip` and `tabix` are provided by the `htslib` module. `samtools` and `bcftools` are provided by the module of the same name. (use `module avail htslib samtools bcftools` to list the installed versions) #### FA @@ -76,7 +76,7 @@ tabix -p vcf variants.sorted.vcf.gz For large VCF files (above ~768M), the bcftools `--max-mem` option may be used to allocate extra memory for sorting VCF records, avoiding the use of the file system for an [external merge sort](https://en.wikipedia.org/wiki/External_sorting#External_merge_sort). -### 3. Create the configuration file, tracks.conf +### 3. Create the Configuration File, tracks.conf An example of a [minimal configuration](https://jbrowse.org/docs/minimal.html) ```ini @@ -122,7 +122,7 @@ Customizing Tracks: - [Wiggle/BigWig Tracks](https://jbrowse.org/docs/bigwig.html) - [VCF Tracks](https://jbrowse.org/docs/variants.html) -### 4. Making feature names searchable in JBrowse (Optional) +### 4. Making Feature Names Searchable in JBrowse (Optional) The JBrowse `generate-names.pl` script generates an index of feature names for searching and autocompletion when typed into the JBrowse search box, or the `View > search for features` menu option. ```bash @@ -135,6 +135,6 @@ generate-names.pl --compress --out . Enter the full path to the directory you created above in the `Data Folder Path` input, add any optional Slurm options and press the `Launch` button. -## JBrowse running in OOD +## JBrowse Running in OOD After a short pause while your job is in the Torch queue and then starts up, you'll see the JBrowse application. From 37fa18fd975f0fa268b6ae60296a1fac81bb7b85 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 15:17:05 +0800 Subject: [PATCH 64/84] Update 04_writing_reading_files.mdx --- .../12_tutorial_intro_shell_hpc/04_writing_reading_files.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/hpc/12_tutorial_intro_shell_hpc/04_writing_reading_files.mdx b/docs/hpc/12_tutorial_intro_shell_hpc/04_writing_reading_files.mdx index 20ca45497a..20c052f9c5 100644 --- a/docs/hpc/12_tutorial_intro_shell_hpc/04_writing_reading_files.mdx +++ b/docs/hpc/12_tutorial_intro_shell_hpc/04_writing_reading_files.mdx @@ -145,7 +145,7 @@ $ ls files documents newname.testfile ``` -## Removing files +## Removing Files We’ve begun to clutter up our workspace with all of the directories and files we’ve been making. Let’s learn how to get rid of them. One important note before we start… when you delete a file on UNIX systems, they are gone _forever_. There is no “recycle bin” or “trash”. Once a file is deleted, it is gone, never to return. So be ***very*** careful when deleting files. Files are deleted with `rm file [moreFiles]`. To delete the `newname.testfile` in our current directory: @@ -187,7 +187,7 @@ Steam is a major online sales platform for PC video games with over 125 million In January 2015, user kevyin on GitHub [reported that Steam’s Linux client had deleted every file on his computer](https://github.com/ValveSoftware/steam-for-linux/issues/3671). It turned out that one of the Steam programmers had added the following line: `rm -rf "$STEAMROOT/"*`. Due to the way that Steam was set up, the variable `$STEAMROOT` was never initialized, meaning the statement evaluated to `rm -rf /*`. This coding error in the Linux client meant that Steam deleted every single file on a computer when run in certain scenarios (including connected external hard drives). Moral of the story: be very careful when using `rm -rf`! ::: -## Looking at files +## Looking at Files Sometimes it’s not practical to read an entire file with `cat`. The file might be way too large, take a long time to open, or maybe we want to only look at a certain part of the file. As an example, we are going to look at a large and complex file type used in bioinformatics, a `.gtf` file. The GTF2 format is commonly used to describe the location of genetic features in a genome. Let’s grab and unpack a set of demo files for use later. To do this, we’ll use [`wget`](https://www.gnu.org/software/wget/). From 88e1d6c7bb46e03250b26a78295550ae37cd29e5 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 15:17:49 +0800 Subject: [PATCH 65/84] Update 05_wildcards_pipes.mdx --- docs/hpc/12_tutorial_intro_shell_hpc/05_wildcards_pipes.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/hpc/12_tutorial_intro_shell_hpc/05_wildcards_pipes.mdx b/docs/hpc/12_tutorial_intro_shell_hpc/05_wildcards_pipes.mdx index ef63996bc8..e7206c3e29 100644 --- a/docs/hpc/12_tutorial_intro_shell_hpc/05_wildcards_pipes.mdx +++ b/docs/hpc/12_tutorial_intro_shell_hpc/05_wildcards_pipes.mdx @@ -135,7 +135,7 @@ mv *.fastq fastq/ ::: -## Redirecting output +## Redirecting Output Each of the commands we’ve used so far does only a very small amount of work. However, we can chain these small UNIX commands together to perform otherwise complicated actions! For our first foray into *piping*, or redirecting output, we are going to use the `>` operator to write output to a file. When using `>`, whatever is on the left of the `>` is written to the filename you specify on the right of the arrow. The actual syntax looks like `command > filename`. @@ -208,7 +208,7 @@ $ cat word_counts.txt We want to add this sentence to the end of our file ``` -## Chaining commands together +## Chaining Commands Together We now know how to redirect `stdout` and `stderr` to files. We can actually take this a step further and redirect output (`stdout`) from one command to serve as the input (stdin) for the next. To do this, we use the `|` (pipe) operator. `grep` is an extremely useful command. It finds things for us within files. Basic usage (there are a lot of options for more clever things, see the `man` page) uses the syntax `grep whatToFind fileToSearch`. Let’s use `grep` to find all of the entries pertaining to the `Act5C` gene in Drosophila melanogaster. From 7fbd7863af8653a58ebae204a836c8bdba2da895 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 15:19:41 +0800 Subject: [PATCH 66/84] Update 02_why_use_cluster.mdx --- docs/hpc/13_tutorial_intro_hpc/02_why_use_cluster.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hpc/13_tutorial_intro_hpc/02_why_use_cluster.mdx b/docs/hpc/13_tutorial_intro_hpc/02_why_use_cluster.mdx index 47e7bbe5a6..b7ea6be583 100644 --- a/docs/hpc/13_tutorial_intro_hpc/02_why_use_cluster.mdx +++ b/docs/hpc/13_tutorial_intro_hpc/02_why_use_cluster.mdx @@ -44,7 +44,7 @@ Interacting with the shell is done via a command line interface (CLI) on most HP Learning to use Bash or any other shell sometimes feels more like programming than like using a mouse. Commands are terse (often only a couple of characters long), their names are frequently cryptic, and their output is lines of text rather than something visual like a graph. However, using a command line interface can be extremely powerful, and learning how to use one will allow you to reap the benefits described above. -## The rest of this lesson +## The Rest of this Lesson The only way to use these types of resources is by learning to use the command line. This introduction to HPC systems has two parts: - We will learn to use the UNIX command line (also known as the Bash shell). From 4161732e955ca27b9809cc9d891cf8007ab663d6 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 15:20:57 +0800 Subject: [PATCH 67/84] Fix typo in HPC tutorial regarding storage locations Corrected the spelling of 'Greene' from 'Geene' in the tutorial. --- .../hpc/13_tutorial_intro_hpc/03_exploring_remote_resources.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hpc/13_tutorial_intro_hpc/03_exploring_remote_resources.mdx b/docs/hpc/13_tutorial_intro_hpc/03_exploring_remote_resources.mdx index 4f03acb4be..cf7740c68b 100644 --- a/docs/hpc/13_tutorial_intro_hpc/03_exploring_remote_resources.mdx +++ b/docs/hpc/13_tutorial_intro_hpc/03_exploring_remote_resources.mdx @@ -52,7 +52,7 @@ bin etc lib64 mnt proc run share sys var The `/home/NetID`, `/scratch/NetID` and `/archive/NetID` directories are created for you by default and they are where you'll probably store most of your files, but there are other options as well. Please see the tip below and our [storage documentation](../03_storage/01_intro_and_data_management.mdx) for details about how these directories differ, as well as other storage options available. Other folders on a UNIX OS contain system files and change as you install new software or upgrade your OS. :::tip[Using the HPC filesystems] -On Geene, you have a number of places where you can store your files. These differ in both the amount of space allocated and whether or not they are backed up. +On Greene, you have a number of places where you can store your files. These differ in both the amount of space allocated and whether or not they are backed up. - **Home** – data stored here is available throughout the HPC system, and often backed up periodically. Please note the limit on the number of files (inodes) which can get used up easily. Use the `myquota` command to ensure that you are not running out of inodes! - **Scratch** – used for temporary file storage while running jobs. It is not backed up and files that are unused for over 60 days are purged. It should not be used for long term storage. From 696bbbb35b44328b2b1ecefa274ac1a708eaf934 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 15:22:11 +0800 Subject: [PATCH 68/84] Update 06_scripts_variables_loops.mdx --- .../06_scripts_variables_loops.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/hpc/12_tutorial_intro_shell_hpc/06_scripts_variables_loops.mdx b/docs/hpc/12_tutorial_intro_shell_hpc/06_scripts_variables_loops.mdx index 21cc4020a9..87085de9f6 100644 --- a/docs/hpc/12_tutorial_intro_shell_hpc/06_scripts_variables_loops.mdx +++ b/docs/hpc/12_tutorial_intro_shell_hpc/06_scripts_variables_loops.mdx @@ -2,7 +2,7 @@ import { Smile } from "lucide-react"; import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; -# Scripts, variables, and loops +# Scripts, Variables, and Loops :::note[Overview] Questions @@ -169,7 +169,7 @@ echo "Our script worked!" When we run our script, the output should be unchanged from before! -## Shell variables +## Shell Variables One important concept that we’ll need to cover are shell variables. Variables are a great way of saving information under a name you can access later. In programming languages like Python and R, variables can store pretty much anything you can think of. In the shell, they usually just store text. The best way to understand how they work is to see them in action. To set a variable, simply type in a name containing only letters, numbers, and underscores, followed by an `=` and whatever you want to put in the variable. Shell variable names are often uppercase by convention (but do not have to be). From da88506af1c299663f15a1dfe3aecdc41b7e11fd Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 15:23:24 +0800 Subject: [PATCH 69/84] Update 06_modules.mdx --- docs/hpc/13_tutorial_intro_hpc/06_modules.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hpc/13_tutorial_intro_hpc/06_modules.mdx b/docs/hpc/13_tutorial_intro_hpc/06_modules.mdx index ff384dc1d2..377d247a57 100644 --- a/docs/hpc/13_tutorial_intro_hpc/06_modules.mdx +++ b/docs/hpc/13_tutorial_intro_hpc/06_modules.mdx @@ -1,4 +1,4 @@ -# Accessing software via Modules +# Accessing Software via Modules :::info[Overview] Questions - How do we load and unload software packages? From e981e9a51b4ad600f0158313190d38541b27724c Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 15:24:16 +0800 Subject: [PATCH 70/84] Update 04_scheduler_fundamentals.mdx --- docs/hpc/13_tutorial_intro_hpc/04_scheduler_fundamentals.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hpc/13_tutorial_intro_hpc/04_scheduler_fundamentals.mdx b/docs/hpc/13_tutorial_intro_hpc/04_scheduler_fundamentals.mdx index 5f9b34adfc..24652899be 100644 --- a/docs/hpc/13_tutorial_intro_hpc/04_scheduler_fundamentals.mdx +++ b/docs/hpc/13_tutorial_intro_hpc/04_scheduler_fundamentals.mdx @@ -260,7 +260,7 @@ This job will use 2 CPUs. ``` Typically, the resulting shell environment will be the same as that for `sbatch`. -## Interactive jobs +## Interactive Jobs Sometimes, you will need a lot of resource for interactive use. Perhaps it’s our first time running an analysis or we are attempting to debug something that went wrong with a previous job. Fortunately, Slurm makes it easy to start an interactive job with `srun`: ```bash srun --pty bash From b1caecabfa1c8f5671b60fa3c01f1898fb7dd567 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 15:24:53 +0800 Subject: [PATCH 71/84] Update 05_environment_variables.mdx --- docs/hpc/13_tutorial_intro_hpc/05_environment_variables.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hpc/13_tutorial_intro_hpc/05_environment_variables.mdx b/docs/hpc/13_tutorial_intro_hpc/05_environment_variables.mdx index d63cb46858..5f3004afd5 100644 --- a/docs/hpc/13_tutorial_intro_hpc/05_environment_variables.mdx +++ b/docs/hpc/13_tutorial_intro_hpc/05_environment_variables.mdx @@ -87,7 +87,7 @@ $ echo $SECRET_IDENTITY Camilla ``` -## Environment variables +## Environment Variables When we ran the `printenv` command we saw there were a lot of variables whose names were in upper case. That’s because, by convention, variables that are also available to use by *other* programs are given upper-case names. Such variables are called *environment variables* as they are shell variables that are defined for the current shell and are inherited by any child shells or processes. To create an environment variable you need to `export` a shell variable. For example, to make our `SECRET_IDENTITY` available to other programs that we call from our shell we can do: From 0f0b1d17e26502a7e3c240224fbfbfc6993bd5dd Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 15:25:12 +0800 Subject: [PATCH 72/84] Update section title to 'Using Resources Effectively' --- .../13_tutorial_intro_hpc/09_using_resources_effectively.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hpc/13_tutorial_intro_hpc/09_using_resources_effectively.mdx b/docs/hpc/13_tutorial_intro_hpc/09_using_resources_effectively.mdx index aedd391070..6ee5c74db4 100644 --- a/docs/hpc/13_tutorial_intro_hpc/09_using_resources_effectively.mdx +++ b/docs/hpc/13_tutorial_intro_hpc/09_using_resources_effectively.mdx @@ -1,4 +1,4 @@ -# Using resources effectively +# Using Resources Effectively :::info[Overview] Questions - How can I review past jobs? From e3cebb503158ad30c54bf31f7dad6668a3e72915 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 15:32:59 +0800 Subject: [PATCH 73/84] Update 02_connecting_to_hpc.mdx --- .../02_connecting_to_hpc.mdx | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/hpc/12_tutorial_intro_shell_hpc/02_connecting_to_hpc.mdx b/docs/hpc/12_tutorial_intro_shell_hpc/02_connecting_to_hpc.mdx index 0bd66abb67..bf48d434ad 100644 --- a/docs/hpc/12_tutorial_intro_shell_hpc/02_connecting_to_hpc.mdx +++ b/docs/hpc/12_tutorial_intro_shell_hpc/02_connecting_to_hpc.mdx @@ -1,4 +1,4 @@ -# Connecting to the remote HPC system Torch +# Connecting to the Remote HPC System Torch :::note[Overview] Questions: @@ -25,14 +25,14 @@ SSH clients are usually command-line tools, where you provide the remote machine When logging in to a laptop, tablet, or other personal device, a username, password, or pattern are normally required to prevent unauthorized access. In these situations, the likelihood of somebody else intercepting your password is low, since logging your keystrokes requires a malicious exploit or physical access. For systems like log-1 running an SSH server, anybody on the network can log in, or try to. Since usernames are often public or easy to guess, your password is often the weakest link in the security chain. ## Remote Connections with the NYU VPN -If you are connecting from a remote location that is not on the NYU network (your home for example), you have have to set up your computer to use the NYU VPN. Once you’ve created a VPN connection, you can proceed as if you were connected to the NYU net. +If you are connecting from a remote location that is not on the NYU network (your home for example), you have to set up your computer to use the NYU VPN. Once you’ve created a VPN connection, you can proceed as if you were connected to the NYU net. ## Connect to the NYU VPN To connect to VPN, please see the [NYU instructions](https://www.nyu.edu/life/information-technology/infrastructure/network-services/vpn.html). ## Log into the Torch Cluster -### Inside the NYU network (non-Windows) +### Inside the NYU Network (non-Windows) From within the NYU network, that is, from an on-campus location, or after you VPN inside NYU’s network, you can log in to the HPC clusters directly. To log in to the HPC cluster (Torch), simply use: @@ -124,7 +124,7 @@ Under the clusters menu you can select the Torch Shell Access option to access t GUI based applications are accessible without the need for port or X11 forwarding. Select the Interactive Apps menu, select the desired application, and submit the job based on required resources and options. ### Troubleshooting Connections to Open OnDemand -A common issue that can occur is receiving an error that the Open OnDemand page cannot be reached. Sometimes this can indicate that the service is down, but often this is an issue with the the local browser cache. You can test this by opening a private browser window and seeing if [https://ood.torch.hpc.nyu.edu](https://ood.torch.hpc.nyu.edu) will load. If it does, try deleting the cache for [https://ood.torch.hpc.nyu.edu](https://ood.torch.hpc.nyu.edu) in your browser history to resolve this issue. +A common issue that can occur is receiving an error that the Open OnDemand page cannot be reached. Sometimes this can indicate that the service is down, but often this is an issue with the local browser cache. You can test this by opening a private browser window and seeing if [https://ood.torch.hpc.nyu.edu](https://ood.torch.hpc.nyu.edu) will load. If it does, try deleting the cache for [https://ood.torch.hpc.nyu.edu](https://ood.torch.hpc.nyu.edu) in your browser history to resolve this issue. In Chrome, this can be done by navigating to this page in your settings: @@ -167,7 +167,7 @@ You could run each line of the `ssh-agent` output yourself, and achieve the same If you are using PuTTY on Windows, download and use `pageant` as the SSH agent. See the [PuTTY documentation](./02_connecting_to_hpc.mdx#opening-a-terminal). -## Modifying your .ssh/config file +## Modifying Your .ssh/config File Please add the following lines to your `~/.ssh/config` file: ``` Host dtn.torch.hpc.nyu.edu @@ -189,7 +189,7 @@ Host torch login.torch.hpc.nyu.edu ``` You'll need to replace the sections above labelled `` with your NetID. You can find more details about this at the [Quickstart section of Accessing HPC at NYU](https://sites.google.com/nyu.edu/nyu-hpc/accessing-hpc#h.7t97br4zzvip) -## Logging onto the system +## Logging onto the System With all of this in mind, let’s connect to our remote HPC system. In this tutorial, we will connect to Torch — an HPC system located at New York University. Although it’s unlikely that every system will be exactly like Torch, it’s a very good example of what you can expect from an HPC installation. To connect to Torch we will use SSH (if you are using PuTTY see [details above](./02_connecting_to_hpc.mdx#opening-a-terminal)). SSH allows us to connect to UNIX computers remotely, and use them as if they were our own. The general syntax of the connection command follows the format: @@ -225,12 +225,12 @@ This change presents a small complication that we will need to navigate througho - `[NetID@glogin-1 ~]$` when the command is to be entered on a terminal connected to the remote system - `$` when it really doesn’t matter which system the terminal is connected to -### Being certain which system your terminal is connected to +### Being Certain Which System Your Terminal is Connected to If you ever need to be certain which system a terminal you are using is connected to then use the following command: ```bash $ hostname ``` -### Keep two terminal windows open +### Keep Two Terminal Windows Open It is strongly recommended that you have two terminals open, one connected to the local system and one connected to the remote system that you can switch back and forth between. If you only use one terminal window then you will need to reconnect to the remote system using one of the methods above when you see a change from `[local]$` to `[NetID@login-1 ~]$` and disconnect when you see the reverse. From 30558deaeade236b3dad5207b518db7507cc5ed4 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 15:40:16 +0800 Subject: [PATCH 74/84] Update 07_transferring_files_remote.mdx --- .../13_tutorial_intro_hpc/07_transferring_files_remote.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/hpc/13_tutorial_intro_hpc/07_transferring_files_remote.mdx b/docs/hpc/13_tutorial_intro_hpc/07_transferring_files_remote.mdx index 9fc3591fc3..2028378bc9 100644 --- a/docs/hpc/13_tutorial_intro_hpc/07_transferring_files_remote.mdx +++ b/docs/hpc/13_tutorial_intro_hpc/07_transferring_files_remote.mdx @@ -1,4 +1,4 @@ -# Transferring files with remote computers +# Transferring Files with Remote Computers :::info[Overview] Questions - How do I transfer files to (and from) the cluster? @@ -311,4 +311,4 @@ Say you have a “data” folder containing 10,000 or so files, a healthy mix of - `wget` and `curl -O` download a file from the internet. - `scp` and `rsync` transfer files to and from your computer. - You can use an SFTP client like FileZilla to transfer files through a GUI. -::: \ No newline at end of file +::: From 85dd830ebbe8433c8b73f594fd1f960d90bd8076 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 15:40:30 +0800 Subject: [PATCH 75/84] Update header to capitalize 'Parallel' in tutorial --- docs/hpc/13_tutorial_intro_hpc/08_running_parallel_job.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hpc/13_tutorial_intro_hpc/08_running_parallel_job.mdx b/docs/hpc/13_tutorial_intro_hpc/08_running_parallel_job.mdx index a217b9852b..accad476a2 100644 --- a/docs/hpc/13_tutorial_intro_hpc/08_running_parallel_job.mdx +++ b/docs/hpc/13_tutorial_intro_hpc/08_running_parallel_job.mdx @@ -1,4 +1,4 @@ -# Running a parallel job +# Running a Parallel job :::info[Overview] Questions - How do we execute a task in parallel? From 448e5b5ec9cb3aace07107fd4cb1a496edb02c01 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 15:41:02 +0800 Subject: [PATCH 76/84] Update 10_using_resources_responsibly.mdx --- .../13_tutorial_intro_hpc/10_using_resources_responsibly.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hpc/13_tutorial_intro_hpc/10_using_resources_responsibly.mdx b/docs/hpc/13_tutorial_intro_hpc/10_using_resources_responsibly.mdx index 4355fa78a1..f6b6dac8b3 100644 --- a/docs/hpc/13_tutorial_intro_hpc/10_using_resources_responsibly.mdx +++ b/docs/hpc/13_tutorial_intro_hpc/10_using_resources_responsibly.mdx @@ -1,4 +1,4 @@ -# Using shared resources responsibly +# Using Shared Resources Responsibly :::info[Overview] Questions - How can I be a responsible user? From 0d19636c66fa85a4cc1b0f646e86ade0f1a64b7a Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 15:42:08 +0800 Subject: [PATCH 77/84] Update header capitalization in intro.md --- docs/srde/01_getting_started/01_intro.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/srde/01_getting_started/01_intro.md b/docs/srde/01_getting_started/01_intro.md index 25193f1ffb..0723ce6a73 100644 --- a/docs/srde/01_getting_started/01_intro.md +++ b/docs/srde/01_getting_started/01_intro.md @@ -1,4 +1,4 @@ -# Start here! +# Start Here! Welcome to the Secure Research Data Environment documentation! If you do not have an active project, please proceed to the next section that explains the eligibility criteria and how you may request one. From c9a60feacb4e1a3424a12ff2dd7402ac036c8b3e Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 15:44:40 +0800 Subject: [PATCH 78/84] Update 02_env_roles.mdx --- docs/srde/02_faq/02_env_roles.mdx | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/srde/02_faq/02_env_roles.mdx b/docs/srde/02_faq/02_env_roles.mdx index 6517c3d587..90308518c2 100644 --- a/docs/srde/02_faq/02_env_roles.mdx +++ b/docs/srde/02_faq/02_env_roles.mdx @@ -8,12 +8,12 @@ If there is no such person, the project PI will need to assign the role to someo ::: -## Will other project users have access to my files? +## Will Other Project Users Have Access to My Files? Each user will have access to two drives within the SRDE workspace: home and scratch. The home drive is private and the scratch drive is shared. Any files for collaboration with other project team members on the workspace should be placed or copied over to the scratch drive. -## What kind of software is available on the SRDE? +## What Kind of Software is Available on the SRDE? Some statistical analysis software packages are preinstalled on the SRDE, such as Stata and MATLAB, other software is added on a case-by-case basis, dependent on the security and compatibility of the software with the SRDE. -## How long will the data stay in the SRDE? +## How Long Will the Data Stay in the SRDE? Project lifecycle will be determined between the PI and the SRDE Team during the intake interview. From ac8b2cc41b49ff5c8af2266939f7b305d469299a Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 15:46:14 +0800 Subject: [PATCH 79/84] Revise FAQ section headings to title case Updated section headings to use title case for consistency. --- docs/srde/02_faq/03_using_srde.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/srde/02_faq/03_using_srde.md b/docs/srde/02_faq/03_using_srde.md index 2f71e45244..7a5a8387ce 100644 --- a/docs/srde/02_faq/03_using_srde.md +++ b/docs/srde/02_faq/03_using_srde.md @@ -1,13 +1,13 @@ # Using the SRDE -## Will I receive training on how to use the SRDE? +## Will I Receive Training on How to Use the SRDE? Absolutely! Once your SRDE is set up the SRDE team will schedule onboarding sessions for the research workspace users and a separate one for the data steward. In addition, all users will have access to the User Guide and SRDE support team for troubleshooting and guidance. -## Why is the SRDE in a terminal, is there a screen layout? +## Why is the SRDE in a Terminal, is There a Screen Layout? At this time the SRDE is command-line only. We are working on an updated version with a graphical user interface (GUI) for projects with the need for it. -## How do I export a file? +## How Do I Export a File? All egress is done by the Data Steward. In order to have a file exported, please follow instructions in the SRDE User Guide to place the file in the export folder in the research workspace, then alert your Data Steward that there is a file ready for export. -## Can I upload my own (non-sensitive) files to the SRDE? +## Can I Upload My Own (non-sensitive) Files to the SRDE? In order to maintain the integrity of the data management flow into and out of the SRDE all data going into the environment needs to go through the Data Steward. This ensures both that the Data Steward is aware of all external data entering or leaving the environment and can confirm compliance with the DUA, as well as providing a single path for audit logging. To have your files uploaded to the environment, you can provide them to the Data Steward who can then upload them to the ingress bucket for your retrieval. From 5cb34be491fcc5167b49d22a3541362883839fe7 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 15:51:35 +0800 Subject: [PATCH 80/84] Update 01_basics.mdx --- docs/srde/02_faq/01_basics.mdx | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/srde/02_faq/01_basics.mdx b/docs/srde/02_faq/01_basics.mdx index 157a821da2..0c68b4824c 100644 --- a/docs/srde/02_faq/01_basics.mdx +++ b/docs/srde/02_faq/01_basics.mdx @@ -1,6 +1,6 @@ -# About SRDE, projects and getting started +# About SRDE, Projects and Getting Started -## What is the SRDE? +## What Is the SRDE? NYU Secure Research Data Environment (SRDE) is a centralized secure computing platform designed to support research projects that require storage, sharing and analysis of high risk datasets. The team provides researchers with consultations and resources to comply with security requirements of research grants and Data Use Agreements. SRDE resources intend to meet the security controls outlined in the NIST 800-171 to safeguard Controlled Unclassified Information (CUI). @@ -9,7 +9,7 @@ Please refer to [our technical description here](https://docs.google.com/documen ::: -## Who can have an SRDE project? +## Who Can Have an SRDE Project? Access to SRDE is available to NYU researchers and sponsored members of their research team (i.e. co-investigators, research assistants, external collaborators). @@ -18,16 +18,16 @@ A project must be reviewed and approved (or receive a waiver) by the University' ::: -## How do I sign up for an SRDE? +## How Do I Sign Up for an SRDE? Fill out our intake form to provide more information about your project: [Secure Research Data Environment intake form](https://nyu.qualtrics.com/jfe/form/SV_3Vok9ax87Bxxdsy). Once we have received your form, the team will review the information and will contact you to schedule the consultation. -## How much will the SRDE cost? +## How Much Will the SRDE Cost? The cost is dependent on the needs of the project, such as size of the data and the type of machine needed for the analysis. Google Cloud has a calculator that will help estimate the costs: https://cloud.google.com/products/calculator -## My data is not high risk, are there other options available? +## My Data Is Not High Risk, Are There Other Options Available? There are several options available depending on the data risk classification and the needs of the project. There are resources provided by the university such as research project space, cloud computing, etc. You can check out many of these services on the [HPC Support Site](../../hpc/01_getting_started/01_intro.mdx). If you are unsure on how to proceed, a consultation with the SRDE team will help determine the best path forward. -## What does the SRDE team need from me for the consultation? +## What Does the SRDE Team Need from Me for the Consultation? To help get things started it would be beneficial to submit an [intake form](https://nyu.qualtrics.com/jfe/form/SV_3Vok9ax87Bxxdsy) with any related data governance documentation (files can be attached to the form), including, but not limited to, data use agreement, OSP/IRB documents, and project information. From 571f82e01b842af378c03d3c00a79a3e8b5a4ea9 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 15:52:39 +0800 Subject: [PATCH 81/84] Update header to use title case for consistency --- src/pages/markdown-page.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/pages/markdown-page.md b/src/pages/markdown-page.md index 9756c5b668..015dc196d1 100644 --- a/src/pages/markdown-page.md +++ b/src/pages/markdown-page.md @@ -2,6 +2,6 @@ title: Markdown page example --- -# Markdown page example +# Markdown Page Example You don't need React to write simple standalone pages. From 9c9dabc5e1afa65cc8d9115a1732eaed4c227629 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 15:55:41 +0800 Subject: [PATCH 82/84] Update README.md --- src/pages/storage-finder/README.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/src/pages/storage-finder/README.md b/src/pages/storage-finder/README.md index 0ec28e069e..c0242185a5 100644 --- a/src/pages/storage-finder/README.md +++ b/src/pages/storage-finder/README.md @@ -42,7 +42,7 @@ This means **content editors only need to update the Google Sheet** — the next > [!NOTE] > The workflow only commits when the generated output differs from what is already in the repository. If the sheet has not changed since the last run, no commit is made. -### Triggering a manual sync +### Triggering a Manual `sync` If you need changes to appear immediately rather than waiting for Saturday's scheduled run: @@ -69,7 +69,7 @@ bun scripts/storage-finder-data-generator/generate.ts \ If you do not have Bun installed, get it from [https://bun.sh/](https://bun.sh/), or run the script with `pnpm dlx tsx` instead of `bun`. -### Environment variable +### Environment Variable The default Google Sheet URL is baked into `scripts/storage-finder-data-generator/constants.ts`. To override it without modifying the source, set the `STORAGE_FINDER_SHEET_URL` environment variable before running the script: @@ -78,7 +78,7 @@ STORAGE_FINDER_SHEET_URL="https://..." bun scripts/storage-finder-data-generator --output src/data/storage-finder ``` -### CLI flags summary +### CLI Flags Summary | Flag | Description | |------|-------------| @@ -126,18 +126,18 @@ When a service is selected, the following fields appear as rows in the compariso ## Making Changes -### Updating service information +### Updating Service Information Edit the corresponding row in the Google Sheet, then regenerate the data. Service IDs are automatically generated as URL-safe slugs from the `Title` column. If a title is duplicated, subsequent entries get a numeric suffix (e.g., `my-service-2`). -### Adding a new service +### Adding a New Service Add a new row to the Google Sheet with a unique value in the `Title` column and fill in all relevant columns. Then regenerate the data. > [!WARNING] > Every cell in a row should be filled in accurately. The generator uses regex patterns to map cell text to facet choices. A blank cell causes the generator to apply the fallback behavior for that question (usually matching all choices for that facet, keeping the service visible regardless of what users select for that question). -### Adding or changing a question +### Adding or Changing a Question 1. If the question requires a new signal, add a column to the Google Sheet and populate it for all services. 2. In `scripts/storage-finder-data-generator/config.ts`, add a new entry to the `FACET_CONFIGS` array (or modify an existing one): @@ -151,7 +151,7 @@ Add a new row to the Google Sheet with a unique value in the `Title` column and - `allowMultipleMatches: true` — required on radio facets that legitimately match more than one choice 3. Regenerate and verify in the app. -### Adding or changing a service detail field +### Adding or Changing a Service Detail Field 1. Ensure the column exists in the Google Sheet and is populated. 2. In `scripts/storage-finder-data-generator/config.ts`, add or modify an entry in `FIELD_DEFINITIONS`: From a3ff75bc72d8173169023d02fbc06e3ea9e9c5d0 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 16:14:08 +0800 Subject: [PATCH 83/84] Update 06_modules.mdx --- docs/hpc/13_tutorial_intro_hpc/06_modules.mdx | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/hpc/13_tutorial_intro_hpc/06_modules.mdx b/docs/hpc/13_tutorial_intro_hpc/06_modules.mdx index 377d247a57..2ec8ae8da3 100644 --- a/docs/hpc/13_tutorial_intro_hpc/06_modules.mdx +++ b/docs/hpc/13_tutorial_intro_hpc/06_modules.mdx @@ -202,12 +202,13 @@ Create a job that is able to run `R --version`. Remember, no software is loaded [NetID@log-1 ~]$ nano r-module.sh [NetID@log-1 ~]$ cat r-module.sh #!/bin/bash -#SBATCH #SBATCH -t 00:00:30 +#SBATCH -t 00:00:30 module load r/gcc/4.4.0 R --version [NetID@log-1 ~]$ sbatch r-module.sh +``` ::: @@ -215,3 +216,4 @@ R --version - Load software with `module load softwareName`. - Unload software with `module unload` - The module system handles software versioning and package conflicts for you automatically. +::: From bdd2d7cdca78903949ea5f66a20019248c8b89a7 Mon Sep 17 00:00:00 2001 From: Amanda-dong <159391549+Amanda-dong@users.noreply.github.com> Date: Wed, 22 Jul 2026 16:23:14 +0800 Subject: [PATCH 84/84] Update 08_running_parallel_job.mdx --- docs/hpc/13_tutorial_intro_hpc/08_running_parallel_job.mdx | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/hpc/13_tutorial_intro_hpc/08_running_parallel_job.mdx b/docs/hpc/13_tutorial_intro_hpc/08_running_parallel_job.mdx index accad476a2..bd2093592a 100644 --- a/docs/hpc/13_tutorial_intro_hpc/08_running_parallel_job.mdx +++ b/docs/hpc/13_tutorial_intro_hpc/08_running_parallel_job.mdx @@ -132,6 +132,7 @@ which should take 30.000 seconds with 0.850 parallel proportion of the workload. Hello, World! I am process 0 of 1 on cs. I will do parallel 'work' for 25.500 seconds. Total execution time (according to rank 0): 30.033 seconds +``` :::