diff --git a/docs/genai/02_onboarding/01_intro.mdx b/docs/genai/02_onboarding/01_intro.mdx index d34d8738e0..39a0f22090 100644 --- a/docs/genai/02_onboarding/01_intro.mdx +++ b/docs/genai/02_onboarding/01_intro.mdx @@ -5,7 +5,7 @@ This section deals with the eligibility for getting access to Pythia. To learn more about affiliate access (students after graduation, collaborators from other institutions, please refer to [this section](../../hpc/01_getting_started/02_HPC_Accounts/01_getting_and_renewing_an_account.mdx)) :::tip[workspaces] -Access to Pythia is facilitated via workspaces. Beyond facilitating access, they also allow team members to collaboratively dvelop prompts. +Access to Pythia is facilitated via workspaces. Beyond facilitating access, they also allow team members to collaboratively develop prompts. ::: diff --git a/docs/genai/04_how_to_guides/01_temperature.md b/docs/genai/04_how_to_guides/01_temperature.md index b92cf211a5..da68f6a13e 100644 --- a/docs/genai/04_how_to_guides/01_temperature.md +++ b/docs/genai/04_how_to_guides/01_temperature.md @@ -34,7 +34,7 @@ completion = portkey.chat.completions.create( print(completion) ``` -At the temperature of 2.0, you might get an output along along the lines of: +At the temperature of 2.0, you might get an output along the lines of: - ``"listening to old radio static."`` - ``"... a really peculiar shade of chartreuse today."`` diff --git a/docs/hpc/01_getting_started/02_HPC_Accounts/04_walkthrough_renew_hpc_account.md b/docs/hpc/01_getting_started/02_HPC_Accounts/04_walkthrough_renew_hpc_account.md index 5f193a53a8..6156579ee0 100644 --- a/docs/hpc/01_getting_started/02_HPC_Accounts/04_walkthrough_renew_hpc_account.md +++ b/docs/hpc/01_getting_started/02_HPC_Accounts/04_walkthrough_renew_hpc_account.md @@ -1,4 +1,4 @@ -# Renewing your HPC Account +# Renewing Your HPC Account :::info VPN Needed You need to be on the NYU VPN to perform this task! diff --git a/docs/hpc/01_getting_started/03_Slurm_Accounts/06_adding_publications_grants.mdx b/docs/hpc/01_getting_started/03_Slurm_Accounts/06_adding_publications_grants.mdx index 5383efaed4..0bb0406f9e 100644 --- a/docs/hpc/01_getting_started/03_Slurm_Accounts/06_adding_publications_grants.mdx +++ b/docs/hpc/01_getting_started/03_Slurm_Accounts/06_adding_publications_grants.mdx @@ -1,4 +1,4 @@ -# Entering Grants and Publciations for Your Project +# Entering Grants and Publications for Your Project ## Acknowledgment Statement for Publications The following acknowledgment statement should appear in the publication of any material that resulted from using the NYU IT HPC resources, services, and staff expertise. diff --git a/docs/hpc/03_storage/02_data_transfers.md b/docs/hpc/03_storage/02_data_transfers.md index a3e9796dd6..baec227c71 100644 --- a/docs/hpc/03_storage/02_data_transfers.md +++ b/docs/hpc/03_storage/02_data_transfers.md @@ -4,7 +4,7 @@ Globus is the recommended tool to use for large-volume data transfers due to the efficiency, reliability, security and ease of use. Use other tools only if you really need to. Detailed instructions available at [Globus](./03_globus.md) ::: -## Data-Transfer nodes +## Data-Transfer Nodes Attached to the NYU HPC cluster Torch, the Torch Data Transfer Node (DTN) are nodes optimized for transferring data between cluster file systems (e.g. scratch) and other endpoints outside the NYU HPC clusters, including user laptops and desktops. The gDTNs have 100-Gb/s Ethernet connections to the High Speed Research Network (HSRN) and are connected to the HDR Infiniband fabric of the HPC clusters. More information on the hardware characteristics is available at [Torch spec sheet](../10_spec_sheet.md). ### Data Transfer Node Access @@ -25,7 +25,7 @@ logout ## Tools for Data Transfer ### Linux & Mac Tools -#### scp and rsync +#### Scp and Rsync :::warning Please use Data Transfer Nodes (DTNs) with these tools. While one can transfer data while on login nodes, it is considered a bad practice because it can degrade the node's performance. ::: @@ -49,7 +49,7 @@ Globus is the recommended tool to use for large-volume data transfers. It featur The Globus endpoint for Torch is available at `nyu#torch`. Detailed instructions available at [Globus](./03_globus.md) -### rclone +### Rclone rclone - rsync for cloud storage, is a command line program to sync files and directories to and from cloud storage systems such as Google Drive, Amazon Drive, S3, B2 etc. rclone is available on DTNs. [Please see the documentation for how to use it.](https://rclone.org/) ### Open OnDemand (OOD) diff --git a/docs/hpc/03_storage/03_globus.md b/docs/hpc/03_storage/03_globus.md index 033b0b7304..64edb018f7 100644 --- a/docs/hpc/03_storage/03_globus.md +++ b/docs/hpc/03_storage/03_globus.md @@ -2,7 +2,7 @@ The Globus project aims at providing powerful tools for scientific data management, to help researchers to focus on their domain subjects and solve data intensive research problems. Globus has been grown maturely to enable grid computing by connecting computing resources distributed globally across organizational boundary. Universities, national laboratories and computing facilities are using services of Globus. -## Transferring data between endpoints +## Transferring Data Between Endpoints ### Endpoint A globus **Endpoint** is a data transfer location, a location where data can be moved to or from using Globus transfer, sync and sharing service. An endpoint can either be a ***personal endpoint*** (on a user’s personal computer) or a ***server endpoint*** (located on a server, for use by multiple users). Please see [Data Transfer With Globus](https://www.globus.org/data-transfer) for details. @@ -40,12 +40,11 @@ The NYU HPC Server Endpoint: nyu#torch **Globus Connect Server** is already installed on the NYU HPC cluster creating a ***Server Endpoint*** named **nyu#torch**, that is available to authorized users (users with a valid HPC account) using Globus. If you want to move data to or from your computer and the NYU HPC cluster, you need to install **[Globus Connect Personal](https://www.globus.org/globus-connect-personal)** on your computer, thus creating a ***Personal Endpoint*** on your computer. -### Moving data between Server Endpoints +### Moving Data Between Server Endpoints If you plan to transfer data between ***Server Endpoints***, such as between the NYU server endpoint **nyu#torch** and a server endpoint at another institution, you do not need to install Globus Connect Personal on your computer. -#### Creating a Personal Endpoint on your computer - +#### Creating a Personal Endpoint On Your Computer This needs to be done only once on your personal computer. After clicking "Transfer or Sync to...", click "Search" on the upper right side. Then follow the link "Install Globus Connect Personal". @@ -54,7 +53,7 @@ More information about **Globus Connect Personal** and download links for Linux, ![Globus Connect Personal](./static/globus_connect_personal.png) -### Transfer files between your Personal Endpoint and NYU nyu#torch +### Transfer Files between Your Personal Endpoint and NYU nyu#torch To transfer files you need to specify two collections (endpoints). Specify one of them as **Torch scratch directory**, or **Torch archive directory** or **Torch home directory**. The other endpoint is the one created for your personal computer (e.g. My Mac Laptop) if it is involved in the transfer. When you first use the Torch directory collection, authentication/consent is required for the Globus web app to manage collections on this endpoint on your behalf. ![Globus Start Transfer](./static/globus_start_transfer.png) @@ -67,7 +66,7 @@ When the transfer is done, you should receive an email notification. Click "ACTI ![Globus Success](./static/globus_success.png) -### Small file download from web browsers +### Small File Download from Web Browsers Globus support HTTPS access to data. To download a small file from your web browser, select a file and right-click your mouse, then click 'Download' at the popup menu. diff --git a/docs/hpc/03_storage/06_large_number_of_small_files.md b/docs/hpc/03_storage/06_large_number_of_small_files.md index 82729baf16..13efb97309 100644 --- a/docs/hpc/03_storage/06_large_number_of_small_files.md +++ b/docs/hpc/03_storage/06_large_number_of_small_files.md @@ -5,10 +5,10 @@ Many datasets contain a large number of files (for example [ImageNet](https://en Here are some ideas you can try and evaluate performance for your own project -## Squash file system with Singularity +## Squash File System With Singularity Please see [Squash File System and Singularity](../07_containers/04_squash_file_system_and_singularity.md) -## Use jpg/png files on disk +## Use JPG/PNG Files On Disk One option is to store image files (like png or jpg) on the disk and read from disk directly. :::warning @@ -36,12 +36,12 @@ You can also use the custom-made RAM mapped disk using `#SLURM_RAM_TMPDIR` while This basically requires you to use 2*(size of the data) size of RAM just to hold the data. ::: -## Binary files (pickle, etc) +## Binary Files (Pickle, Etc.) Store data in some binary file (say pickle in Python) which you load fully when you start a SLURM job. This option may require a lot of RAM - thus you may have to wait a long time for the scheduler to find resources for your job. Also this approach would not work on a regular PC without so much RAM, and thus your scripts are not transferable. -## Container files, one-file databases +## Container Files, One-File Databases Special containers, which allow to either load data fast fully or access chosen elements without loading the whole dataset into RAM. ### SQLite @@ -96,10 +96,10 @@ More info: LMDB supports reading by many readers and many parallel thread from the same file -#### Formats inside HDF5/LMDB: binary, numpy, other.. +#### Formats Inside HDF5/LMDB: Binary, NumPy, Other... One can store data in different way inside LMDB or HDF5. For example we can store binary representation of jpeg, or we can store python numpy array. In the first case file can be read from any language, in the second - only from Python. We can also store objects from other languages - for example tibble in R -#### Other formats +#### Other Formats There are other formats like [Bcolz](http://bcolz.blosc.org/), [Zarr](https://github.com/alimanfoo/zarr-python), and others. Some examples can be found [here](https://alimanfoo.github.io/2016/04/14/to-hdf5-and-beyond.html). ## Benchmarking Code diff --git a/docs/hpc/03_storage/07_transferring_cloud_storage_data_with_rclone.md b/docs/hpc/03_storage/07_transferring_cloud_storage_data_with_rclone.md index 79a6713d9a..0eca8f593e 100644 --- a/docs/hpc/03_storage/07_transferring_cloud_storage_data_with_rclone.md +++ b/docs/hpc/03_storage/07_transferring_cloud_storage_data_with_rclone.md @@ -1,10 +1,10 @@ -# Transferring Cloud Storage Data with rclone +# Transferring Cloud Storage Data with Rclone :::tip Globus Globus is the recommended tool to use for large-volume data transfers due to the efficiency, reliability, security and ease of use. Use other tools only if you really need to. Detailed instructions available at [Globus](./03_globus.md) ::: -## Transferring files to and from Google Drive with RCLONE +## Transferring Files to and from Google Drive with Rclone Having access to Google Drive from the HPC environment provides an option to archive data and even share data with collaborators who have no access to the NYU HPC environment. Other options to archiving data include the HPC Archive file system and using [Globus](./03_globus.md) to share data with collaborators. Access to Google Drive is provided by [rclone](https://rclone.org/drive/) - rsync for cloud storage - a command line program to sync files and directories to and from cloud storage systems such as Google Drive, Amazon Drive, S3, B2 etc. [rclone](https://rclone.org/drive/) is available on Torch cluster as a module, the module versions currently available (March 2025) are: @@ -26,7 +26,7 @@ rclone --transfers=32 --checkers=16 --drive-chunk-size=16384k --drive-upload-cut This option works great for file sizes 1Gb+ to 250GB. Keep in mind that there is a rate limiting of 2 files/sec for upload into Google Drive. Small file transfers don’t work that well. If you have many small jobs, please tar the parent directory of such folders and split the tar file into 100GB chunks and then uploads then into Google Drive. -## rclone Configuration +## Rclone Configuration You need to configure rclone before you will be able to move files between the HPC Environment and Google Drive There are specific instruction on the [rclone web site](https://rclone.org/drive/) and here is an example of the process for configuring rclone for use on Torch: @@ -35,12 +35,12 @@ There are specific instruction on the [rclone web site](https://rclone.org/drive Follow [instructions](../02_connecting_to_hpc/01_connecting_to_hpc.mdx) to log into the Torch HPC cluster. -### Step 2: Load the rclone module +### Step 2: Load the Rclone Module ```sh $ module load rclone/1.68.2 ``` -### Step 3: Configure rclone +### Step 3: Configure Rclone Configuring rclone and setting up remote access to your Google Drive, using the command: ```sh diff --git a/docs/hpc/04_datasets/01_intro.md b/docs/hpc/04_datasets/01_intro.md index 3c3ec6eada..c724456134 100644 --- a/docs/hpc/04_datasets/01_intro.md +++ b/docs/hpc/04_datasets/01_intro.md @@ -49,7 +49,7 @@ ImageNet is an image dataset organized according to the [WordNet](https://wordne WordNet® is a large lexical database of English. Nouns, verbs, adjectives and adverbs are grouped into sets of cognitive synonyms (synsets), each expressing a distinct concept ([https://wordnet.princeton.edu/](https://wordnet.princeton.edu/)) ::: -#### ILSVRC (subset of ImageNet) +#### ILSVRC (Subset of ImageNet) ILSVRC uses a subset of ImageNet images for training the algorithms and some of ImageNet’s image collection protocols for annotating additional images for testing the algorithms ([https://arxiv.org/abs/1409.0575](https://arxiv.org/abs/1409.0575)). The name comes from 'ImageNet Large Scale Visual Recognition Challenge ([ILSVRC](https://image-net.org/challenges/LSVRC/2017/))'. Competition was moved to Kaggle ([http://image-net.org/challenges/LSVRC/2017/](http://image-net.org/challenges/LSVRC/2017/)) @@ -63,7 +63,7 @@ ILSVRC uses a subset of ImageNet images for training the algorithms and some of *Dataset is available under* - `/projects/work/public/ml-datasets/imagenet` -##### Get access to Data +##### Get Access to Data New York University does not own this dataset. @@ -110,7 +110,7 @@ This dataset consists of 9 million images covering 90k English words, and includ *Dataset is available under* - `/projects/work/public/ml-datasets/mjsynth` -### open-images-dataset +### Open-images-dataset *About data set*: [Open Images Dataset – opensource.google](https://storage.googleapis.com/openimages/web/index.html) A dataset of ~9 million varied images with rich annotations @@ -128,7 +128,7 @@ The Pile is a 825 GiB diverse, open source language modeling data set that consi *Dataset is available under* - `/projects/work/public/ml-datasets/pile` -### Waymo open dataset +### Waymo Open Dataset *About data set*: [Open Dataset – Waymo](https://waymo.com/open/) The field of machine learning is changing rapidly. Waymo is in a unique position to contribute to the research community with some of the largest and most diverse autonomous driving datasets ever released. diff --git a/docs/hpc/05_submitting_jobs/01_slurm_submitting_jobs.md b/docs/hpc/05_submitting_jobs/01_slurm_submitting_jobs.md index 9128bfa9ac..3fcf25f0f8 100644 --- a/docs/hpc/05_submitting_jobs/01_slurm_submitting_jobs.md +++ b/docs/hpc/05_submitting_jobs/01_slurm_submitting_jobs.md @@ -53,7 +53,7 @@ You can find the available types of GPUs at [Torch Spec Sheet](../10_spec_sheet. Do not specify partitions manually, except for preemption which is described later. ::: -## Resource limits and restrictions +## Resource Limits and Restrictions Jobs within the same partition cannot exceed their assigned resources (`QOSGrpGRES`). User GPU Quotas: Each user has a total GPU quota of 24 GPUs for jobs with wall time < 48 hours (`QOSMaxGRESPerUser`). Non-stakeholders to temporarily use stakeholder resources (a stakeholder group to temporarily use another group’s resources). Stakeholders retain normal access to their own resources. If non-stakeholders (or other stakeholders) are using them, their jobs may be preempted (canceled) once stakeholders submit new jobs. Public users are allowed to use stakeholder resources only with preemption partitions. Refer to the section below for details on preemptible jobs. @@ -81,7 +81,7 @@ Jobs with low GPU utilization will be automatically canceled: Enforcement will be very aggressive. ::: -## Preemptible jobs on Torch +## Preemptible Jobs on Torch On Torch, users may run "preemptible" jobs on stakeholder resources that their group does not own. This allows the stakeholder resources to be utilized by non-stakeholders which may otherwise be idle. To make the best use of these resources, you are encouraged to adopt checkpoint/restart to allow for resumption of the workload in subsequent jobs. :::warning Preemption Policy @@ -109,19 +109,19 @@ Jobs in stakeholder partitions will not be canceled, but those in preemption par ``` Jobs with preemption partitions only might be allowed to use more resources -## Advanced options +## Advanced Options ### GPU MPS Use GPU Multi-Process Service (MPS) to improve overall GPU utilization, as this allows multiple GPU jobs to share a single GPU concurrently by: ``` #SBATCH --comment="gpu_mps=yes" ``` -### RAM disk +### RAM Disk A portion of the RAM available can be mounted as a disk for fast `I/O` operations: ``` #SBATCH --comment="ram_disk=1GB" ``` -### GPU MPS & RAM Disk in a preemptible job +### GPU MPS & RAM Disk in a Preemptible Job Both of these can be combined with preemption as shown: ``` #SBATCH --comment="preemption=yes;preemption_partitions_only=yes;requeue=true;gpu_mps=yes;ram_disk=1GB" diff --git a/docs/hpc/05_submitting_jobs/02_slurm_main_commands.md b/docs/hpc/05_submitting_jobs/02_slurm_main_commands.md index 039332c036..c442cdbcb3 100644 --- a/docs/hpc/05_submitting_jobs/02_slurm_main_commands.md +++ b/docs/hpc/05_submitting_jobs/02_slurm_main_commands.md @@ -1,4 +1,4 @@ -# Slurm: Command reference +# Slurm: Command Reference Slurm offers many utility commands to work with, some of the most popularly used commands are: diff --git a/docs/hpc/07_containers/03_singularity_with_conda.md b/docs/hpc/07_containers/03_singularity_with_conda.md index 8fd5c65649..e48ac439a6 100644 --- a/docs/hpc/07_containers/03_singularity_with_conda.md +++ b/docs/hpc/07_containers/03_singularity_with_conda.md @@ -147,7 +147,7 @@ exit # exit Singularity ``` -#### Install packages +#### Install Packages You may now install packages into the environment with either the `pip install` or `conda install` commands. @@ -202,7 +202,7 @@ Now, exit the Singularity container and then rename the overlay image. Typing `e exit mv overlay-15GB-500K.ext3 my_pytorch.ext3 ``` -#### Test your PyTorch Singularity Image +#### Test Your PyTorch Singularity Image ```sh singularity exec --overlay /scratch//pytorch-example/my_pytorch.ext3:ro /share/apps/images/cuda12.1.1-cudnn8.9.0-devel-ubuntu22.04.2.sif /bin/bash -c 'source /ext3/env.sh; python -c "import torch; print(torch.__file__); print(torch.__version__)"' @@ -213,7 +213,7 @@ singularity exec --overlay /scratch//pytorch-example/my_pytorch.ext3:ro / the end `:ro` addition at the end of the pytorch ext3 image starts the image in read-only mode. To add packages you will need to use `:rw` to launch it in read-write mode. ::: -### Using your Singularity Container in a SLURM Batch Job +### Using Your Singularity Container in a SLURM Batch Job Below is an example script of how to call a python script, in this case `torch-test.py`, from a SLURM batch job using your new Singularity image torch-test.py: @@ -276,7 +276,7 @@ cat slurm-3752662.out # True ``` -### Optional: Convert ext3 to a compressed, read-only squashfs filesystem +### Optional: Convert ext3 to a Compressed, Read-only Squashfs Filesystem Singularity images can be compressed into read-only squashfs filesystems to conserve space in your environment. Use the following steps to convert your ext3 Singularity image into a smaller squashfs filesystem. ```sh srun -N1 -c4 singularity exec --overlay my_pytorch.ext3:ro /share/apps/images/centos-8.2.2004.sif mksquashfs /ext3 /scratch//pytorch-example/my_pytorch.sqf -keep-as-directory -processors 4 -noappend diff --git a/docs/hpc/08_ml_ai_hpc/02_pytorch_intro.md b/docs/hpc/08_ml_ai_hpc/02_pytorch_intro.md index e021706289..7500f0f70d 100644 --- a/docs/hpc/08_ml_ai_hpc/02_pytorch_intro.md +++ b/docs/hpc/08_ml_ai_hpc/02_pytorch_intro.md @@ -315,7 +315,7 @@ The slowest line is number 42 which consumes 84.5% of the time in the training f Note that GPU utilization as measured using nvidia-smi is only a measure of the fraction of the time that a GPU kernel is running on the GPU. It says nothing about how many CUDA cores are being used or how efficiently the GPU kernels have been written. However, for codes used by large communities, one can generally associate GPU utilization with overall GPU efficiency. For a more accurate measure of GPU utilization, use [Nsight Systems or Nsight Compute](https://researchcomputing.princeton.edu/support/knowledge-base/gpu-computing#profiling) to measure the occupancy. -## Step 6: Work through the Performance Tuning Guide from Princeton +## Step 6: Work Through the Performance Tuning Guide from Princeton Make sure you optimize the single GPU case before going to multiple GPUs by working through the [Performance Tuning Guide](https://pytorch.org/tutorials/recipes/recipes/tuning_guide.html). diff --git a/docs/hpc/08_ml_ai_hpc/03_pytorch_dpp.md b/docs/hpc/08_ml_ai_hpc/03_pytorch_dpp.md index 8440b19c09..c5b44cb961 100644 --- a/docs/hpc/08_ml_ai_hpc/03_pytorch_dpp.md +++ b/docs/hpc/08_ml_ai_hpc/03_pytorch_dpp.md @@ -36,7 +36,7 @@ Here are some webpages and videos: Do not use `DataParallel` in PyTorch for anything since it gives poor performance relative to `DistributedDataParallel`. ::: -## Main changes needed in going from single-GPU to multi-GPU training with DDP +## Main Changes Needed in Going from Single-GPU to Multi-GPU Training with DDP This completely new piece is needed to form the process group: @@ -192,7 +192,7 @@ The `local_rank` should be used everywhere in your script except when initializi # DDP and Slurm -## Total number of tasks equals total number of GPUs +## Total Number of Tasks Equals Total Number of GPUs When using DDP, the total number of tasks must equal the total number of allocated GPUs. Therefore, if `--ntasks-per-node=` then you must have `--gres=gpu:`. Here are two examples: @@ -429,6 +429,6 @@ Execute the commands below to run the example above: [NetID@log-1 full-ddp-test]$ sbatch run-full.SBATCH ``` -## Memory issues +## Memory Issues Use `gradient_as_bucket_view=True` when making the DDP model to decrease the required memory by 1/3. diff --git a/docs/hpc/08_ml_ai_hpc/08_LLM inference/02_run_hf_model.md b/docs/hpc/08_ml_ai_hpc/08_LLM inference/02_run_hf_model.md index 01cc653b0b..b432027443 100644 --- a/docs/hpc/08_ml_ai_hpc/08_LLM inference/02_run_hf_model.md +++ b/docs/hpc/08_ml_ai_hpc/08_LLM inference/02_run_hf_model.md @@ -1,9 +1,9 @@ -# Basic LLM Inference with Hugging Face transformers +# Basic LLM Inference with Hugging Face Transformers Here we provide an example of how one can run a Hugging Face Large-language model (LLM) on the NYU Torch cluster -## Prepare environment -### Create project directory +## Prepare Environment +### Create Project Directory After [logging on to a Torch login node](../../02_connecting_to_hpc/01_connecting_to_hpc.mdx), make a directory for this project: ```bash @@ -14,30 +14,30 @@ After [logging on to a Torch login node](../../02_connecting_to_hpc/01_connectin You'll need to replace NetID above with your NetID ::: -### Move to a compute node +### Move to a Compute Node Some of the following steps can require significant resources, so we'll move to a compute node. This way we won't overload the login node we're on. ```bash [NetID@log-1 llm_example]$ srun --cpus-per-task=2 --mem=10GB --time=04:00:00 --pty /bin/bash ``` -### Copy appropriate overlay file to the project directory +### Copy Appropriate Overlay File to the Project Directory ```bash [NetID@cm001 llm_example]$ cp -rp /share/apps/overlay-fs-ext3/overlay-50G-10M.ext3.gz . [NetID@cm001 llm_example]$ gunzip overlay-50G-10M.ext3.gz ``` -### Launch Singularity container in read/write mode +### Launch Singularity container in Read/write Mode ```bash [NetID@cm001 llm_example]$ apptainer exec --fakeroot --overlay overlay-50G-10M.ext3:rw /share/apps/images/cuda12.1.1-cudnn8.9.0-devel-ubuntu22.04.2.sif /bin/bash ``` -### Install miniconda in the container +### Install Miniconda in the Container ```bash Singularity> wget --no-check-certificate https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh Singularity> bash Miniforge3-Linux-x86_64.sh -b -p /ext3/miniforge3 ``` -### Create environment script +### Create Environment Script Use an editor like nano or vim to create the file `/ext3/env.sh`. The contents should be: ```bash #!/bin/bash @@ -49,12 +49,12 @@ export PATH=/ext3/miniforge3/bin:$PATH export PYTHONPATH=/ext3/miniforge3/bin:$PATH ``` -### Activate the environment +### Activate the Environment ```bash Singularity> source /ext3/env.sh ``` -### Install packages in environment +### Install Packages in Environment ```bash Singularity> conda config --remove channels defaults Singularity> conda update -n base conda -y @@ -63,7 +63,7 @@ Singularity> conda install pip -y Singularity> pip install torch numpy transformers ``` -### Exit from Singularity and the compute node +### Exit from Singularity and the Compute Node ```bash Singularity> exit [NetID@cm001 llm_example]$ exit @@ -73,7 +73,7 @@ Singularity> exit You can find more information about using Singularity and Conda on our HPC systems in our documentation [Singularity with Conda](https://sites.google.com/nyu.edu/nyu-hpc/hpc-systems/torch/software/singularity-with-miniconda). ::: -## Prepare script +## Prepare Script Create a python script using the following code from sections 1-9 and save it in a file called `huggingface.py`: 1. Import necessary modules: @@ -134,8 +134,8 @@ Create a python script using the following code from sections 1-9 and save it in print("Shape of the batch embedding: {}".format(sentence_embedding.shape)) ``` -## Prepare Sbatch file -After saving the above code in a script called `huggingface.py`, create a file called `run.SBATCH` with the the following code: +## Prepare Sbatch File +After saving the above code in a script called `huggingface.py`, create a file called `run.SBATCH` with the following code: ```batch #!/bin/bash @@ -162,7 +162,7 @@ You'll need to change `NetID` in the script above to your NetID. If you're using a different directory name and/or path you'll also need to update that in the script above. ::: -## Run the run.SBATCH file +## Run the run.SBATCH File ```batch [NetID@log-1 llm_example]$ sbatch run.SBATCH ``` diff --git a/docs/hpc/08_ml_ai_hpc/08_LLM inference/03_vLLM.md b/docs/hpc/08_ml_ai_hpc/08_LLM inference/03_vLLM.md index 0af63cdcf1..b4264e3837 100644 --- a/docs/hpc/08_ml_ai_hpc/08_LLM inference/03_vLLM.md +++ b/docs/hpc/08_ml_ai_hpc/08_LLM inference/03_vLLM.md @@ -1,4 +1,4 @@ -# High-performance LLM inference with `vLLM` +# High-performance LLM Inference with `vLLM` ## What is vLLM? [`vLLM`](https://docs.vllm.ai/en/latest/) is an easy-to-use library for LLM inference and serving which support a wide variety of models with optimized kernels ensuring optimal utilization of GPUs. @@ -29,7 +29,7 @@ Create a `vLLM` directory in your /scratch directory, then install the vLLM imag ``` apptainer pull docker://vllm/vllm-openai:latest ``` -### Avoid filling up your `$HOME` directory +### Avoid Filling up your `$HOME` Directory To avoid exceeding your `$HOME` quota (50GB) and inode limits (30,000 files), you should redirect `vLLM`'s cache and Hugging Face's model downloads to your scratch space: ```sh export HF_HOME=/scratch/$USER/hf_cache diff --git a/docs/hpc/10_spec_sheet.md b/docs/hpc/10_spec_sheet.md index eee053afaf..536566d4df 100644 --- a/docs/hpc/10_spec_sheet.md +++ b/docs/hpc/10_spec_sheet.md @@ -26,7 +26,7 @@ Torch was tested in June 2025 using the [LINPACK benchmark system](https://top50 Torch was recently ranked [#40 on the Green 500 list](https://top500.org/lists/green500/list/2025/06/), a global list of the most energy efficient supercomputers in the world thanks to its advanced liquid cooling system. ## Torch Stakeholder Resources -Many GPUs on Torch are owned by stakeholder groups who have prioirty access to the resources. The breakdown of these GPUs into stakeholder groups is as follows: +Many GPUs on Torch are owned by stakeholder groups who have priority access to the resources. The breakdown of these GPUs into stakeholder groups is as follows: | Stakeholders | H200 | L40S | H100 | A100 | RTX Pro 6000 | Total | | :--- | :---: | :---: | :---: | :---: | :---: | :---: | diff --git a/src/pages/storage-finder-data.mdx b/src/pages/storage-finder-data.mdx index 356a09053c..24d61b3b19 100644 --- a/src/pages/storage-finder-data.mdx +++ b/src/pages/storage-finder-data.mdx @@ -478,7 +478,7 @@ During pilot phase, projects must be approved by RTS staff ### Permission Settings -Project dependent, role-based permissions. NeiID users added by PI request +Project dependent, role-based permissions. NetID users added by PI request ### Eligibility @@ -516,7 +516,7 @@ VPN needed when off-campus. ### Permission Settings -Done via Grouper; can give collaborators read access, read/write access, or administrator priveleges for a share. +Done via Grouper; can give collaborators read access, read/write access, or administrator privileges for a share. ### Eligibility